A permission change with Amazon Web Service prevented customers from accessing most Salesforce services for several hours.
There was a huge outage of Salesforce services today following an internal permission change. As a result, customers were not able to access services for several hours.
“This is about our public cloud access service,” Shanmugam Chinnasamy, SVP of Salesforce Sales Cloud Engineering, said during a video call with customers. “From Salesforce we call into AWS and we introduced a change in that permission that caused this error.”
Wide effects on Salesforce cloud services. A graphic displayed during the call showed most if not all of Salesforce’s cloud services were affected.
Chinnasamy said the company’s technology team became aware of the service disruption at 11:03 AM (EST). Because this was an issue with Salesforce communicating with AWS no customers’ data was ever at risk of being exposed or compromised, he added.
“We faced issues in our morning around the attachments section (files, notes, etc.) where the error message somehow pointed towards Amazon AWS bad gateway,” one customer wrote in the call’s chat. “Fortunately, it [was] resolved in a couple of hours.”
Customers on the call wanted to know why Salesforce’s trust site wasn’t updated until 11:53 AM. This was because the company first wanted to have a clear picture of what had happened, according to Cuthbert Langley, a member of Salesforce’s tech crisis communications team.
Business hours? The company wasn’t able to answer a question asked by many customers: Why was the update done during business hours?
“We will do root cause analysis and then understand why it was made, why it [happened] during business hours and then we will get back to you,” Chinnasamy said.
Langley said people with more specific questions and issues should contact their sales rep for further guidance.
Why we care. Most core applications used by most marketing organizations are now cloud-based. In the early days of cloud, there was great concern about security of data and about possible outages. Largely those concerns have been put to rest; cloud applications have generally come to be seen as more reliable than their on-prem predecessors. This incident shows that human error can still introduce problems.
The post Salesforce says an internal action caused service outage appeared first on MarTech.
About the author
Constantine von Hoffman is managing editor of MarTech. A veteran journalist, Con has covered business, finance, marketing and tech for CBSNews.com, Brandweek, CMO, and Inc. He has been city editor of the Boston Herald, news producer at NPR, and has written for Harvard Business Review, Boston Magazine, Sierra, and many other publications. He has also been a professional stand-up comedian, given talks at anime and gaming conventions on everything from My Neighbor Totoro to the history of dice and boardgames, and is author of the magical realist novel John Henry the Revelator. He lives in Boston with his wife, Jennifer, and either too many or too few dogs.
(7)
