The passwords of some people using sites monitored by popular analytics provider Mixpanel were mistakenly pulled into its software. Until TechCrunch’s inquiry, Mixpanel had made no public announcement about the embarrassing error beyond quietly emailing clients about the problem. Yet some need to update to a fixed Mixpanel SDK to prevent an ongoing privacy breach.
It’s unclear which clients were impacted due to confidentiality agreements, but Mixpanel lists Samsung, BMW, Intuit, US Bank and Fitbit as some of the companies it works with. “We can tell you that less than 25 percent of our customers were impacted,” the company’s spokesperson told me, but they noted approximately 4 percent of all Mixpanel Projects suffered from the privacy gap.
Mixpanel has raised $77 million in rounds led by prestigious investors like Andreessen Horowitz and Sequoia. But in early 2016 it laid off 10 percent of its 230-plus team, and has been dogged by a reputation for being expensive. Today’s news won’t help.
The password-harvesting bug stemmed from a March 2017 change to the open source React JavaScript library that clashed with how Mixpanel’s Autotrack feature, launched in 2016, works. It led Autotrack to pull in the values of hidden and password fields in ways it wasn’t supposed to. “We didn’t catch it, it’s that simple,” Mixpanel CEO Suhail Doshi tells me.
The problem persisted for nine months until a customer alerted Mixpanel on January 5th. By the 9th, the company had begun filtering out and securing passwords it accidentally scooped up, and it’s since destroyed any passwords it received. On February 1st, Mixpanel sent the email found at the end of this article to its clients informing them of the issue.
Clients that auto-update their Mixpanel SDK or load it straight from the startup have already gotten a patch to fix the issue. But some clients that manually update their Mixpanel SDK still need to download a new version to stop the flow of passwords. “Roughly 85 percent of affected customers have already updated their SDK to address this issue. We are actively working to contact remaining customers who have not yet updated their SDK,” according to the spokesperson.
In the meantime, “We’ve disabled Autotrack by default for all new projects created. We’ll be further evaluating Autotrack as a product in the future,” the spokesperson says, showing a mature level of contrition.
“To date, our forensics and security experts have not seen any indication that this data was downloaded or accessed by any Mixpanel employee or third party,” the company wrote in the email. That’s a relief, as there’s no way for an individual user of one of Mixpanel’s clients to know if their password got sucked in. Still, the possibility that end users’ privacy could have been breached is surely alarming to Mixpanel customers who trust it to watch how their sites and apps are used to optimize performance and monetization. The error could be a windfall for competitors like Google Analytics, KISSmetrics, Splunk, Flurry and Localytics.
Increasing reliance on open-source frameworks like React means engineering and security teams can’t just worry about their company’s own code. It has to mingle with changes to open-source projects that can cause unforeseen trouble. It’s like if the ingredients in one of your prescription drugs subtly changed, so your preferred over-the-counter pills suddenly caused a dangerous interaction.
The full email from Mixpanel is below:
EMAIL SENT TO CUSTOMERS ON FEBRUARY 1, 2018:
We are writing you today about a recently discovered data ingestion issue on the Mixpanel platform that affects your project(s) and requires that you update your SDK as soon as possible (unless your SDK is set to automatically update). Before we go into detail on what happened and how we’ve addressed the issue, we want to apologize for any difficulty this may cause your organization. Our team is committed to remedying this situation quickly, and we’re available to talk through any questions or concerns—just reply to this email, and we’ll be in touch.
What happened?
On January 5th, 2018, a customer informed us that they observed Autotrack sending the values of password fields in events. We confirmed that this was unexpected behavior; by design, Autotrack should not send the values of hidden and password form fields.
We immediately began investigating further and learned that the behavior the customer was observing was due to a change to the React JavaScript library made in March 2017. This change placed copies of the values of hidden and password fields into the input elements’ attributes, which Autotrack then inadvertently received. Upon investigating further, we realized that, because of the way we had implemented Autotrack when it launched in August 2016, this could happen in other scenarios where browser plugins (such as the 1Password password manager) and website frameworks place sensitive data into form element attributes.
To date, our forensics and security experts have not seen any indication that this data was downloaded or accessed by any Mixpanel employee or third party. It was a bug, plain and simple. Upon discovery, we took immediate steps to secure the data and shut down further receipt. As of today, all data that was inadvertently received has been destroyed. In order to be as transparent as possible, here is more detail on how we have addressed and will continue to address this issue.
How we’re addressing this issue
Since discovery, we have been actively working to resolve the issue for affected customers. The majority of projects were not impacted, but based on our findings, we believe that you may have project(s) that were impacted, which we list at the end of this email.
We took immediate steps when we discovered this data ingestion issue in the form of the following:
Limit further receipt of data: On January 9th, we implemented a server-side filter to securely discard this data as soon as we receive it, and soon thereafter refined the filter to solve for the last remaining edge cases.
Delete the inadvertently received data: We have cleared all data from our database that we inadvertently received and, upon request, we can provide you with fine-grained metadata about what data was inadvertently sent to Mixpanel servers. This will include a mapping of distinct IDs to property names (but not the data values themselves, which have been securely deleted using appropriate security measures).
Fix the Autotrack bug: We have implemented the Autotrack functionality fix in the Mixpanel SDK. You will, however, need to update your SDK as soon as possible to reflect this change. If your SDK is set to automatically update, or if your website loads the SDK directly from our content servers, then no action is required.
Review any access of this data: We do not believe this data was downloaded or accessed by any Mixpanel employee or third party. To the extent we discover otherwise, we will immediately notify you.
In addition to fixing the root cause of this issue, we’re taking proactive steps to identify and prevent similar issues from occurring in the future:
Incorporating formal privacy reviews as part of our design and development processes: Security and privacy have always been front of mind at Mixpanel, but we’re adding some additional explicit checkpoints in our product development processes to help ensure that we’ve considered all of the impacts of the changes we make.
In-depth security/privacy audits of key existing product areas: We’ve learned a lot from this issue, and our team has been diving in to look for similar cases where these same kinds of problems could arise.
Operationalizing our response tooling: We’ve built new tools in response to this issue to help us identify the scope of data collection, limit access to data, and to purge it from our systems quickly. We’re taking these tools and making them more general purpose so that we can respond more quickly in the unlikely event that a similar problem occurs in the future.
Data filtering and detection: We’re exploring capabilities that can detect something like this sooner including changes to the SDK to give us more insight into what data is being sent to us, integration with Data Loss Prevention (DLP) solutions, and even using our machine learning capabilities to detect anomalous ingestion.
We are conducting a thorough investigation of what happened and how we handled it. We believe that we have addressed the ingestion issue with the speed and accuracy required as your trusted partner. Below the signature, we have also listed your Project ID(s) and Project Name(s) that were affected.
If you have questions or for more information, please reply to this email for a response from your account team. Otherwise, as mentioned before, please update your SDK as soon as possible.
Sincerely,
The Mixpanel Security team
Comment