NoSQL Database hosting service MongoHQ, a Y Combinator alum, has suffered a major security breach that appears to be a major factor in an attack over the weekend on Buffer, the social media scheduling service. The MongoHQ intrusion is affecting customers of the hosting service and potentially also their S3 storage accounts on Amazon Web Services (AWS).
MongoHQ Co-Founder Jason McKay, in an open letter on the company web site, wrote that they discovered the breach yesterday when they detected “unauthorized access to an internal support application using a password that was shared with a compromised personal account.” In other words, an employee was fooled into giving up their account credentials. To MongoHQ’s detriment, the internal support application was exposed to the Internet. There was no virtual private network (VPN) to fully protect the back-end of the service. MongoHQ has now taken steps to put a VPN into place.
On Hacker News, Buffer CEO Joel Gascoigne still took full blame, saying the tokens they used were not encrypted.
If access tokens were encrypted (which they are now) then this would have been avoided. In addition, MongoHQ have provided great insights and have much more logging in place than we have ourselves. We’re also increasing logging significantly as a result.
The attack looks like it could have been a lot worse, said JumpCloud CEO David Campbell in a phone interview today. JumpCloud protects users through its management platform, which stores cloud server keys for administrators. The platform abstracts the password process, preventing attacks by dropping a small piece of software on the customer’s cloud server. It is an agent-based approach similar to the way companies such as New Relic provide application performance management. The agent records the data from the server, monitoring it for unusual spikes in network loads and other unusual events.
Luckily, MongoHQ used bcrypt, which is designed to slow down brute-force attacks using powerful computers or corrupted server clusters, Campbell said. In these attacks, there is a brute force attempt to crack passwords. These attacks can access systems fairly easily if there is not something to slow them down. That’s why the LinkedIn attack last year was so significant. There were not the protections in place to slow the attacks. It is believed about six million passwords were stolen.
The fact that MongoHQ appears to have been using bcrypt for customer passwords makes it exponentially more difficult for the attackers to exfiltrate the hashes and crack those passwords offline.Using a popular open source GPU based password cracking tool, we see that an average machine can only try 3788 possible passwords per second against bcrypt. This same machine can try 2 billion possible passwords per second against unsalted SHA1, which is what LinkedIn was using when it suffered its breach last year.Their big failing here appears to have been putting their administrative application on the public Internet, rather than putting it behind the VPN. They also appear to have failed to require 2FA for access to the app. It sounds like they’re learning their lessons the hard way.
Still, whoever accessed the accounts systematically went through the system, accessing customer accounts:
Our support tool includes an “impersonate” feature that enables MongoHQ employees to access our primary web UI as if they were a logged in customer, for use in troubleshooting customer problems. This feature was used with a small number of customer web UI accounts. Our primary web UI allows customers to browse data and manage their databases. We are contacting affected customers directly.
We have additionally determined that an unauthorized user to our support system would have had some access to our account database, which includes connection info for customer MongoDB instances.
We’ve conducted an audit of direct access to customer databases and determined that several databases may have been accessed using information stored in our account database. We are contacting affected customers directly. If you have not heard from us individually, there is no evidence that your DB was accessed by an unauthorized user.
MongoHQ has also taken l steps to invalidate the AWS credentials it stored for customers as part of backing up to S3. Customers that use the same credentials on AWS as they do on MongoHQ are particularly vulnerable. AWS has created “Premium Support” cases for all affected accounts, to assist customers with establishing new credentials, as needed.
This is a major attack that reflects on the poor state of security in the startup community. It shows the need for services like JumpCloud and more emphasis on how companies enforce user management.
Comment