Application Security , Governance & Risk Management , IT Risk Management

Salesforce Says Permissions Bungle Almost Fixed

Salesforce says it has nearly recovered from a problematic database script that wiped out user permissions within its Pardot marketing management product on Friday.

See Also: ESG Research Report: Securing the API Attack Surface

In an update, Salesforce says it has now run "an automated provisioning fix to allow us to restore user permissions to where they were before the incident occurred." That was a relief for some adminstrators:

So, I was just about to haul my ass into the office on a Sunday of a long weekend to manually restore permissons on 20+ profiles and it appears you were able to restore overnight. You rock @SalesforceEng !!!

— Allie (@newsviewsbrews) May 19, 2019

But one instance, NA57 - which is served from data centers in the Chicago and Washington, D.C., areas - is still receiving the fix, Salesforce said on Sunday.

The snafu could have allowed employees to access data that they're not supposed to or modify it. It wasn't immediately clear if organizations would be able to audit access during the period when permissions were lifted. A Salesforce spokeswoman contacted in Sydney referred to the company's advisory

Shut It Down. Now.

On Saturday, it described the reason for the problem with a post on its website: "The deployment of a database script resulted in granting users broader data access than intended." But that was gone by Sunday.

The permission removal problem occurred with Salesforce's Pardot product, which is used for managing lead-generation tools, such as email campaigns for business-to-business marketing. But the quick fix was to disable access, which not only affected Pardot users but others across Salesforce.

"We have had to disable access to our service to customers affected in order to help resolve the issue," writes Salesforce CTO Parker Harris on Twitter. "We expect to be able to restore access soon as we continue to work through this issue."

'Holy Hell'

Salesforce is one of the most dominant CRM systems, so problems tend to have a quick and fierce response. As one user, FufuWarrior, who is based in London put it on Reddit: "Holy hell. Two beers at lunch & walk into this. What a day to be alive."

Another user on Reddit, palsy34, whose organization contacted Salesforce during the outage, gave a few more details on what went wrong: "On a bridge with Salesforce right now. Apparently they ran a script to adjust some permissions for community users that went off the chain and gave modify all to every profile. They reduced permissions to the most restrictive (no access) and are working on a fix."

Some users speculated that Salesforce may have not wanted to notify all of its customers of the permissions problem immediately because employees might have taken advantage of the deep access.

Another user, RTB4499, writes: "I think there is a reason for Salesforce's silence as the moment they publicly reveal the problem then it encourages users to explore and see what they have access too exacerbating an already significant data breach."

It may have been a prudent reaction, writes Reddit user going by the nickname St0rmborn.

"I would have probably done the same thing," the person writes. "I can only imagine how badly they were freaking out when they discovered it and kept peeling back the layers of how severe the impact was. I wouldn't want to cause a mass panic until I understood the issue and also made sure it didn't make the entire situation worse. There's no such thing as just notifying admins privately, because you know that would immediately go public and blown up on message boards worldwide."

Early on, Harris apologized.

"To all of our @salesforce customers, please be aware that we are experiencing a major issue with our service and apologize for the impact it is having on you," he writes. "Please know that we have all hands on this issue and are resolving as quickly as possible."