Is there anything worse than being hacked? How about a hacker that manages to acquire your customers’ contact information, emails them that the service is shutting down, and that the company has been lying to everyone about its security? That’s what happened to BrowserStack, the makers of a popular browser-testing service that allows customers to test websites on real desktop and mobile browsers in a VM environment.
The Mumbai-based company counts among its over 25,000 customers big names like Microsoft, jQuery, Xerox, Citrix, Github, eBay, Barclays, Adobe, Visa, Lockheed Martin, Bose, MIT, Johnson & Johnson, and Wikipedia, according to its website.
Over the weekend, some unknown number of BrowserStack customers received an odd email from the company.
It read:
Dear BrowserStack User,
We are unfortunately displeased to announce that BrowserStack will be shutting down. After much consideration on our part, we have realized we were negligent in the services we claimed to offer. In our terms of service, we state the following:
[…] after the restoration process is complete, the virtual machines are guaranteed to be tamper-proof.
[…] The machines themselves are in a secure network, and behind strong firewalls to present the safest environment possible.
[…] At any given time, you have sole access to a virtual machine. Your testing session cannot be seen or accessed by other users, including BrowserStack administrators. Once you release a virtual machine, it is taken off the grid, and restored to its initial settings. All your data is destroyed in this process.
Unfortunately, we have blatantly lied. Not only do all of our administrators have access, but so does the general public. We have no firewalls in place, and our password policies are atrocious. All virtual machines launched are open to the public, accessible to anyone with the alpha password \”nakula\” on port 5901, a password which is stored in plaintext on every VM. As well, our infrastructure uses the same root passwords on all machines, which is also stored in plaintext on every VM launched (\”c0stac0ff33\”).
Given the propensity for cyber criminals to target infrastructure services such as ours, it is almost certain all of your data has been compromised. These passwords take no less than 15 minutes to find for anyone who is looking.
We hope we have not caused you too much trouble, and to our enterprise customers who signed deals contracts based on a fabrication, we are equally sorry.
Sincerely,
The BrowserStack Team
Obviously, the tone of the email and its content raised a red flag. Even when companies are negligent about their security, they don’t tend to be quite this transparent when admitting it to their customers.
The email, not surprisingly, was not legit. At least, it wasn’t from the company itself. As for its claims? We’re still waiting to hear back if BrowserStack execs want to respond to these in more detail.
https://twitter.com/Chevex/status/531703412304257025
@browserstack Having a hard time believing this guys, to be honest. I think a formal response to the contents of the email is warranted ASAP
— Luke Rollans (@lukerollans) November 10, 2014
https://twitter.com/ceejayoz/status/531790240835719168
https://twitter.com/lojikil/status/531796341027266560
The hack itself doesn’t seem too involved. A company rep told TechCrunch that the attack was detected “immediately after the hacker gained access” and the damage was “contained to just a list of email addresses.” BrowserStack is advising customers to change their passwords as a precaution, however.
“We are still in the process of sanitisation, and making doubly sure this situation never re-occurs. We are on top of it, and will post updates as they happen. We will be sharing an entire post-mortem report in the next few days,” a company rep told us.
BrowserStack has also been keeping its Twitter account updated with news of the breach and the measures it’s taking to resolve the problem. Initially, it took the entire service down. Subsequent tweets have stated the service was coming back online, with the “Automate” and “Screenshot” services first to return and the “Live” service to follow soon. On Twitter, the company is also promising to email all users with the entire analysis after the services are restored.
“Currently efforts are focused on getting the service back on track, and protecting user interests,” one tweet read, contributing to concerns that the hacker’s claims have a grain of truth to them.
Indeed, the hacker’s statements have worried some BrowserStack users, who wonder now if there are security holes in the testing service — especially because the claims come across as those a disgruntled ex-employee might know about.
Additionally, on a thread on the developer-focused portal Hacker News, a number of commenters mention they’ve experienced oddities when using BrowserStack, noting they’ve started sessions where it appears they were able to see another user’s session in progress, including their internal URL, typing, and the other user’s mouse moving around on the screen.
Making matters worse, despite requests from users on Twitter, the company has mainly communicated about the status of its services, not whether or not the claims in the hacker’s email were true or false. When TechCrunch asked the company point-blank if it would respond to these claims, a rep responded: “I can assure you that the claims in the hacker’s email are false. The email was not sent by BrowserStack or anyone working with us.”
That said, our responses came in through a support email address, and not from the company execs directly. We wouldn’t characterize this response as an “official” company statement at this time. We reached out to BrowserStack founders via both email and LinkedIn for further clarification, and are still waiting to hear back. We’ll update if and when that occurs.
Update, Mon. 2:45 PM ET:
Co-founder Ritesh Arora has responded to our inquiry. “We will respond to all questions and will communicate in detail to all our users,” he says. “Our current priority is to get BrowserStack stand up and running. In short, We did get hacked. BrowserStack is technically huge and we’re checking each component strictly, adding extra precautions to prevent any further hack. We’re opening services one by one.”
We’ve asked Arora if he will provide a point-by-point analysis.
Update, Mon. 3:20 PM ET:
Arora: “All BrowserStack services are up and running. We continue scale up for spike in traffic. We’re keeping a strong check as well.”
Update, Nov. 12:
Here is the full email sent to customers:
As you may already know, BrowserStack experienced an attack on 9th November, 2014 at 23:30 GMT during which an individual was able to gain unauthorized access to some of our users’ registered email addresses. He then tried to send an email to all our registered users, but he was only able to reach less than 1% (our estimate is 5,000 users). The email contained inaccurate information, even claiming that BrowserStack would be shutting down.
When we realized this, our only concern was to protect our users. This involved temporarily taking down the service, as we scrutinized each component carefully. This inconvenienced our users for several hours, and for that we are truly sorry.
What happened?
BrowserStack application servers run using Amazon Web Services. The configuration is vast, consisting of thousands of servers. One of these was an old prototype machine, which was the target of the breach.
The machine had been running since before 2012, and was not in active use. It was penetrated using the shellshock vulnerability, and since it was no longer in active use, it did not have the appropriate patch installed.
The old prototype machine had our AWS API access key and secret key. Once the hacker gained access to the keys, he created an IAM user, and generated a key-pair. He was then able to run an instance inside our AWS account using these credentials, and mount one of our backup disks. This backup was of one of our component services, used for production environment, and contained a config file with our database password. He also whitelisted his IP on our database security group, which is the AWS firewall.
He began to copy one of our tables, which contained partial user information, including email IDs, hashed passwords, and last tested URL. His copy operation locked the database table, which raised alerts on our monitoring system. On receiving the alerts, we checked the logs, saw an unrecognized IP, and blocked it right away. In that time, the hacker had been able to retrieve only a portion of the data. Finally, using this data and the SES credentials, he was able to send an email to some of our users.
What was the extent of the damage?
Our database logs confirmed that user data was partially copied, but no user test history was compromised. Therefore all user data remains wholly intact. Most crucially, credit card details were not compromised, as we only store the last 4 digits of the credit card number, and all payment processing takes place through our payment processing partner. All user passwords are salted, and encrypted with the powerful bcrypt algorithm, which creates an irreversible hash which cannot be cracked. However, as an added precaution, we suggest that users change their BrowserStack account passwords.
We were able to verify the actions of the hacker using AWS Cloud Trail, which confirmed that no other services were compromised, no other machines were booted, and our AMIs and other data stores were not copied.
In addition, our production web server logs indicate that we were experiencing shellshock attempts, but they failed because the production web server has the necessary patches to foil all such attempts.
Points in the email
We would now like to address the points raised in the email. The hacker quoted three paragraphs from our Security documentation, as follows:
- after the restoration process is complete, the virtual machines are guaranteed to be tamper-proof. → Our restoration process is indeed tamper-proof. When we create a test machine from scratch, we take a snapshot. After every user session, the test machine is restored to its original state using that snapshot. Even if a previous user manages to install a malicious software, it is always erased due to the restoration process.
- The machines themselves are in a secure network, and behind strong firewalls to present the safest environment possible. → Every single machine has an OS firewall, in addition to the hardware network firewalls we use. On EC2, we use security groups as an equivalent safety measure. We also use industry-standard brute force-throttling measures.
- At any given time, you have sole access to a virtual machine. Your testing session cannot be seen or accessed by other users, including BrowserStack administrators. Once you release a virtual machine, it is taken off the grid, and restored to its initial settings. All your data is destroyed in this process. → The application ensures that a machine is allocated to only one person at a time, and VNC passwords are randomly generated for each session. Thus, even our administrators cannot see your test session.
With respect to the plaintext passwords on the VMs, this is certainly not the case, as we moved to key-based authentication years ago. Moreover root login is disabled in our SSH configuration.
Both the passwords mentioned, ‘nakula’ and ‘c0stac0ff33’, were indeed in use a couple of years ago during our prototyping phase, and thus were present in the old prototype machine that was hacked. ‘nakula’ was previously our VNC password, and was hashed. However, unlike the hash used for the user passwords, this hash is much weaker. This was due to a limitation in VNC protocol, and we had overcome this liability by regenerating a new password for every session, and thus ‘nakula’ has not been in use for years. ‘c0stac0ff33’ was one of our system user passwords on the prototype machine, before we moved to key-based authentication.
It is true that we still run our VNC server on port 5901, but we do not believe that it is a security vulnerability because a current password is still required for access. As mentioned before, the passwords are changed every test session.
Where did we go wrong?
All our servers, running or not, whether in active use or not, should have been patched with the latest security upgrades and updates including the shellshock one. Moreover, servers not in active use should have been stopped and the server shouldn’t have had the AWS keys.
Additionally, our communication could have been better. Instead of intermittent updates, we preferred to present a complete, honest picture of the attack to our users once our analysis was done.
Security measures taken to mitigate and prevent further incidents
- After taking down the service, we revoked all the existing AWS keys and passwords, and generated new ones immediately, as an added security measure.
- Subsequently, we went through all the SSH logs, web server logs, as well as AWS Cloud Trail logs, to ensure that no more damage was done.
- We are migrating all backups to encrypted backups, and removing all unencrypted ones.
- We have also put in several additional checks and alerts, which are triggered on specified AWS actions. As a precautionary measure we have also created new VM snapshots and have replaced all the existing ones.
- To prevent further incidents, we are in the process of evaluating certain VPC/VPN options to enhance our security measures.
- We’re going to have a security audit conducted by an external, independent agency.
Once again we apologise for the inconvenience. BrowserStack is deeply committed to providing the best and most secure testing infrastructure for our users. We will be forging ahead with exciting new releases in the next few weeks and look forward to continue serving you.
We have a trace and the IP of the hacker. We will be in touch with authorities soon to register an official complaint. Thank you for the support and understanding we have received over the last few days.
Sincerely,
Ritesh and Nakul
Founders, BrowserStack
Comment