Monday.com launched more than a decade ago looking to help companies build a highly flexible set of business tools like CRM, marketing, operations and HR, built in customized ways you simply couldn’t find out of the box. It learned that companies not only liked that flexibility, but they also pushed the boundaries so far that it was going to outpace the ability of the underlying database technology to continue to handle all of the use cases.
So the company began looking for a replacement. With myriad database choices available off-the-shelf, you would think finding a suitable one would only be a matter of time and testing, but after reviewing several options, and talking to some experts, Monday concluded it needed something more than what was available in the market.
One of the main issues was flexibility; Monday.com had no idea how its customers would fashion the building blocks into applications. That meant it needed a schemaless database to handle whatever customers decided to build. That’s when it decided to build its own, but with a twist: It wouldn’t build a single database to take over all the future capabilities. Instead, it would layer on some other databases to handle particular tasks. It called the new solution MondayDB.
The new database has been in place since July, but as the company makes the transition to the new database, the old one still exists as another layer in the complexity that is Monday.com architecture.
Startups have to realize that no matter how careful they may be with technology choices — and often it’s impossible, as they struggle to get a product to market — there is no way to anticipate how products will grow and develop over time. That said, at some point, companies will have to pay off their technical debt by starting with some entirely new architecture, just as Monday.com has had to do.
We sat down with chief product and technology officer Daniel Lereya to learn how the team decided to build this solution and the challenges it faced in finding a database technology to meet these unique requirements.
Just another manic Monday
The process that led to building a database has been ongoing for a couple of years. In fact, in January 2021, the company began exploring ideas for a new database with a completely open mind. Lereya says the reason customers value Monday.com is the flexibility it brings, and the company needed a solution that could manage an adaptable approach.
The basic building block of Monday is called the “board,” which is a table designed to hold a variety of information types. The company described it in a blog post announcing the new database:
We offer over 40 types of columns, and our users enjoy the freedom to filter, sort, or aggregate just about any column combination, each with its own logic for such operations. For instance, if you filter by person, you can do so by their email, name, or even the team they’re a part of.
“When we built Monday.com, we put a lot of emphasis on giving people the building blocks to [essentially] build their own software themselves,” Lereya told TechCrunch+. These building blocks are boards that users can build in a dizzying array of ways.
When the company began thinking about this at the beginning of 2021, it started with the idea that it wanted to enable companies to scale to basically whatever they wanted to do, which would require a new database approach.
“We felt that we didn’t want to do just incremental improvements, but we thought that strategically, knowing also that we were going to continue and go up market in the future, we needed a much more robust solution that will support all these specific and unique needs,” Lereya said.
Monday morning, you sure look fine
The original thinking was that with so many databases on the market, surely there would be one that would meet its requirements, and in June 2021, the company formed a team to begin exploring the available options.
The group did a comprehensive search that included the likes of Elasticsearch, Apache Pinot, ClickHouse, Apache Druid and a wide range of NoSQL databases like CockroachDB, Couchbase, and others, according to the company.
“We assembled a document with all the requirements for this database . . . and we actually came to the realization that there is no one specific database that answers our needs. And with that, we decided to go for the MondayDB project,” he said.
Even after talking to a number of companies and experts in the space and building some detailed PoCs, in the end, Monday.com found that none of them really met a list of requirements.
In August 2021, it began building the new solution with a small team of just five engineers. It would take more than a year to get to beta. That was partly because of the complexity, partly because it was a small team and partly because the team wanted to build for the long haul and take time to get it right.
As time went on, the team grew, but each of the core Monday components like boards, dashboards and others needed to change to work with the new database. So ultimately the project touched just about every part of the engineering organization.
Interestingly, what Monday.com came up with was not a pure build-versus-buy decision. “We didn’t build the database technology by itself. We leveraged existing databases, so it was a combination of technologies together with our own layers that became the MondayDB solution,” Lereya said.
The final idea includes this entirely new database on top of the old database along with Cassandra, Redis and MySQL. That’s a lot of database technology, but it needed every bit of it to make the entire package work for each customer use case.
Come Monday, it’ll be alright
The company began testing the new database internally in September 2022 with the plan to run both the old database and the new MondayDB simultaneously and act as a kind of check against each other. That way Monday.com could make sure that MondayDB was behaving as expected against all of the components and all of the ways that customers use the product.
“First of all, we got to a phase where we wanted MondayDB in terms of the data to run side by side with our current architecture. We thought about it like a shadow mode where we do everything on both the old architecture and the new architecture,” Lereya said.
When Monday.com gave it to initial customers to test, the company made sure that the new database wouldn’t break anything because it had the ability to roll back if needed. Over time, Monday.com merged in more and more customers before making the official release in July 2023.
But the old database didn’t get replaced. “It’s running concurrently. And actually if you remember, the original database is MySQL, and there are cases in which it’s better to query that database,” Lereya said. “So actually it’s part of the solution and MondayDB is an additive layer and not a replacement.”
With the new database, the company was able to increase capacity 10x, which translates into 500 boards per dashboard, 1 million items per board and 5 million items per dashboard. What’s more, they built this with five nines of data integrity (meaning it is correct 99.999% of the time) with 5x the speed from the previous database and the ability to build in automations that move data to the right place automatically.
And the release of MondayDB 1.0 isn’t an ending, it’s a beginning. The company is looking to continue layering on functionality and make the database as responsive as possible to whatever use cases customers throw at it.
Comment