Khosla Ventures and Y Combinator-backed open source Git repository GitLab is currently offline after suffering what appears to be a major backup restoration failure after accidentally deleting production data.
Initially the service looks to have been suffering load time and stability problems, which led to an outage yesterday.
But the issue escalated into emergency database maintenance after data was deleted accidentally — followed by an apparent inability to restore the data from backups, according to a series of tweets from the @GitLabStatus account.
The affected database includes issues and merge requests but not repositories and wikis, according to these updates.
We are performing emergency database maintenance, https://t.co/r11UmmDLDE will be taken offline
— GitLab.com Status (@gitlabstatus) January 31, 2017
we are experiencing issues with our production database and are working to recover
— GitLab.com Status (@gitlabstatus) February 1, 2017
We accidentally deleted production data and might have to restore from backup. Google Doc with live notes https://t.co/EVRbHzYlk8
— GitLab.com Status (@gitlabstatus) February 1, 2017
The incident affected the database (including issues and merge requests) but not the git repo's (repositories and wikis).
— GitLab.com Status (@gitlabstatus) February 1, 2017
The company also been updating a Google Doc with info on the ongoing incident.
A section of the Google Document document detailing “problems encountered” notes that none of GitLab’s five deployed backup/replication techniques “are working reliably or set up in the first place”.
Which sounds sub-optimal, to put it politely. The Register has a blunter assessment saying the service is in “meltdown”. “The world doesn’t contain enough faces and palms to even begin to offer a reaction to that sentence,” it writes.
CHECK YOUR BACKUPS TODAY. Don’t do a @gitlabstatus https://t.co/q6tj8aiVfK
— The Register (@TheRegister) February 1, 2017
We’ve reached out to GitLab for comment and will update this story with any response.
One positive thing that can be said is the company is at least providing a high level of detail about the incident — however cold comfort that is if you’re facing the loss of a bunch of data.
According to The Register a folder containing 300GB of live production data was erroneously deleted by a GitLab sysadmin — with just 4.5GB remaining by the time the delete command was cancelled.
GitLab’s most recent tweet says the recovery of the database can be tracked here.
The code collaboration platform, which was founded in 2014, is headquartered in San Francisco and has raised around $25.6M to date, according to CrunchBase — including a $20M Series B round last September, led by August Capital — when founder and CEO Sid Sijbrandij said it wanted to “keep accelerating and not have to pull up”. It has some 100 employees at this point, many of whom work remotely.
GitLab competes with various other platforms for managing code for development teams — such as GitHub and Atlassian — but attracted investors owing to its open source twist on the code repository and collaboration tool model, which allows users to adapt the platform to better fit their needs by building on top of it.
As well as free community and SaaS offerings, GitLab lets enterprise customers run its platform on their own servers, via its paid Enterprise Edition. Customers for the paid product include IBM, Macy’s, ING, NASA and VMWare.
The company has reached out to confirm that the outage only affects GitLab.com — meaning that customers using its platform on premise are not affected.
This article was updated with additional detail about which GitLab users are affected
Comment