Tracking the explosive growth of open-source software


Image Credits: David Paul Morris/Bloomberg / Getty Images

Dharmesh Thakker


Dharmesh Thakker is a general partner at Battery Ventures and a former managing director at Intel Capital.

More posts from Dharmesh Thakker

Many of today’s hottest new enterprise technologies are centered around free, “open-source” technology. As a result, many big companies — from financial giants to retailers to services firms — are building their businesses around new, community-based technology that represents a sea change from the IT practices of the past.

But how can corporate customers — and investors — evaluate all these new open-source offerings? How can they tell which projects (often strangely named: Ansible, Vagrant, Gradle) are generating the most customer traction? Which ones have the biggest followings among software developers, and the most potential to capture market share?

These questions are especially tough to answer because most open-source companies are still private, and don’t have to disclose key user and financial metrics. (Though that’s changing — open-source giant Cloudera recently announced plans to go public, increasing the market’s focus on open-source technology.)

That’s why we decided to create a new, detailed index to track popular open-source software projects, and gain some insights into the new companies powered by these technologies. It is the Battery Open-Source Software Index (BOSS Index), which we’ve spent months putting together with publicly available information and are introducing here. We hope to update it quarterly — and the index should gain more relevance as more open-source companies using some of these projects grow and go public.

The index contains 40 open-source projects, gleaned from an initial scouring of projects listed on the GitHub source-code repository site, as well as Datamation, an enterprise-IT publication that also tracks open-source projects. The top 25 are listed below, and the full list can be found on our website.

We focused on projects in enterprise IT-related areas like IT operations, including technology powering operating and provisioning systems; data and analytics, including tools for artificial intelligence and machine learning as well as databases; and DevOps, which includes projects focused on the hot new trend of “containers,” which help people develop software quickly in a sort of self-contained environment.

Rank Project Name Overall Project Rating Category Sample of Related Companies
1 Linux 100.00 IT Operations Red Hat, Ubuntu
2 Git 31.10 DevOps GitHub, GitLab
3 MySQL 25.23 Data & Analytics Oracle
4 Node.js 22.75 DevOps NodeSource, Rising Stack
5 Docker 22.61 DevOps Docker
6 Hadoop 16.19 Data & Analytics Cloudera, Hortonworks
7 Elasticsearch 15.72 Data & Analytics Elastic
8 Spark 14.99 Data & Analytics Databricks
9 MongoDB 14.68 Data & Analytics MongoDB
10 Selenium 12.81 DevOps Sauce Labs, BrowserStack
11 NPM 12.31 DevOps NPM
12 Redis 11.61 Data & Analytics Redis Labs
13 Tomcat 11.04 IT Operations NA
14 Jenkins 10.47 DevOps CloudBees
15 Vagrant 8.15 IT Operations HashiCorp
16 Postgres 8.02 Data & Analytics EnterpriseDB
17 Gradle 7.68 DevOps Gradle
18 Nginx 7.57 IT Operations Nginx
19 Ansible 7.42 IT Operations Ansible
20 Kafka 7.22 Data & Analytics Confluent
21 GitLab 6.42 DevOps GitLab
22 Hbase 6.41 Data & Analytics Cloudera, Hortonworks
23 Chef 6.37 IT Operations Chef*
24 TensorFlow 5.97 Data & Analytics Google
25 Cassandra 5.74 Data & Analytics DataStax

Companies ranked according to four factors. Overall project rating represents the geometric mean of two of the four individual scores, which reflect online discussion activity; search activity; jobs impact; and GitHub activity. All data is as of February 9, 2017.

There are some very well-known names on the list, including projects that have spawned big companies. They include Linux, which underlies Red Hat; MySQL, which powers the company of the same name and was bought by Sun Microsystems (now part of Oracle) for $1 billion in 2008; and Hadoop, which brought us Cloudera and Hortonworks.

But some more-obscure names, like Selenium, also rank highly, indicating there is plenty of grassroots innovation happening in the open-source sector — and many new projects out there that are spawning valuable companies. Still, our research also found that having lots of users for your open-source project does not automatically translate into creating a commercially viable company.

We ranked the projects according to four factors, which included:

  • Public interest in the project, as measured by Google search activity;
  • User activity, gauged by mentions of the projects on the popular tech-discussion board Stack Overflow;
  • Jobs impact, measured by the number of job postings citing each open-source project listed on the job boards Indeed and Simply Hired; and
  • Impact in the open-source community, tracked by measuring a project’s influence on GitHub. Specifically we tracked the number of “forks,” or extensions, built on each project; the number of GitHub “stars” a company received, indicating its popularity; and the number of “watches,” another popularity indicator, all as of February 9, 2017.

Because some projects may have done extremely well, or poorly, on certain criteria — perhaps one had an off-the-charts Google search number, but a sub-par job-postings score — we threw out the top and bottom individual-criteria scores for each project. This is a methodology called “trimmed mean,” and it’s similar to what happens in Olympic gymnastics, where officials throw out an athlete’s highest and lowest scores from each judge and average the remaining ones. (We had no East German judges in our competition, but we are being careful.)

Even so, there’s always room for improvement. Some adoption and popularity criteria, such as download metrics, are obviously somewhat difficult to measure, and surely we haven’t captured all the hottest, new emerging tools. Though with our planned quarterly updates, we should be able to track new leaders as they emerge. So we’d love feedback from the community to improve our numbers, and the index, over time. Please let us know at opensource@battery.com if you have further insights into any of these metrics.

Here are some other key takeaways from our research.

Linux, Git, MySQL lead the pack

Perhaps not surprisingly, the open-source project leading our index is Linux, technology that was first released in 1991 and is one of the most widely adopted open-source projects in the world. It is commercialized by companies, including Red Hat, one of the few open-source companies to trade publicly, as well as Ubuntu and SUSE.

Git, which came in at No. 2 on the list, inspired GitHub and GitLab. The wildly popular open-source project serves as a “version-control system” for tracking changes and coordinating work between software developers.

Also high on the list, at No. 3, was MySQL, the database technology first developed in 1995. MySQL currently helps run huge, Web-scale companies like Google, Facebook and Twitter. But it’s also worth noting that several “NoSQL” database technologies — which are non-relational databases, unlike MySQL, and are often better suited for parsing the unstructured data being thrown off by many companies today — also ranked highly.

These NoSQL technologies include MongoDB, which came in at No. 9 in our index; Redis, which is being commercialized by company Redis Labs, at No. 12; Cassandra, which came in at No. 25 and is behind the database company DataStax; and Elasticsearch at No. 7, which is being commercialized by Elastic.

MongoDB raised a new round of financing in late 2015 — the company is estimated to be valued at around $1.5 billion — and now competes against established database players like Oracle, IBM and Microsoft. Overall, several of these NoSQL vendors are growing independently of one another — quite quickly in some cases — rather than converging into one giant system. This points to further fragmentation in the broader data-infrastructure sector, and could lead to the creation of several strong NoSQL players that could be public companies in the future.

Big data fuels open source

As many organizations struggle to manage huge volumes of structured and unstructured data today — generated by everything from security software to tweets to Web-enabled sensors in manufacturing plants — they are increasingly looking for new data-management and storage solutions. That trend is reflected in our index, as more than a dozen (15) of the 40 projects included are open-source technologies powering databases and data processing.

Hadoop, mentioned previously, is one such technology. But Spark, which is commercialized by companies like Databricks, is another, and ranked eighth on our list.

Other names to know

Docker, the container-technology darling that has helped make software development quicker and more efficient, came in at No. 5 in our index. Many view Docker as a possible replacement for technology from public giant VMware, and the fact that Docker can be easily and cheaply accessed through the open-source community has fueled its adoption.

Docker is also competing with open-source platform technologies like Google’s Kubernetes (No. 33) and Mesos to control the “orchestration” layer in software development, or the ability to manage containers across different software environments.

Another hot area for open source is “continuous integration and continuous delivery,” or the ability to write software with code that continuously and seamlessly integrates with other platforms. Some tools in this area include Jenkins (No. 14), which is commercially supported by CloudBees, and TravisCI. In the related DevOps area are technologies such as Maven (No. 30), as well as the fast-growing Artifactory binary repository, a software tool designed to optimize the download and storage of binary files, a platform commercialized by JFrog.*

Making it rain

But as we noted earlier, having lots of users — while essential to eventually gaining commercial traction — does not guarantee that an open-source project will make a good business. That takes a lot of hard work and creativity, particularly in structuring new types of business models; leveraging complicated open-source licenses; and tweaking traditional enterprise-sales practices to fit an open-source product, as we discussed in TechCrunch last year.

What’s more, we’ve found that the chances for commercial success for IT companies leveraging open source can sometimes increase if they offer several open-source technologies that can be used together in a sort of “stack.” Elastic has the “ELK” stack, for example, consisting of the open-source Elasticsearch (No. 7 in our index), Kibana (No. 36) and Logstash (No. 29) projects.

Time-series database company InfluxData,* similarly, sells versions of the “TICK” stack — that stands for Telegraf, InfluxDB, Chronograf and Kapacitor. DevOps company HashiCorp, in one final example, has commercialized many open-source projects, including two that made our list, Vagrant (No. 15) and Vault (No. 40). Software developers like to be able to pick their favorite components from these stacks, in keeping with the “best of breed” product mentality that dominates software development today. 

These open-source projects are not your father’s Sun or Oracle. But clearly, CIOs at major global companies now rely on open-source technologies — including many of the ones highlighted in our index — to run key segments of their infrastructure, and many of these projects are going to be around for the long haul. Indeed, at an open-source summit that we hosted last year, IT executives from large companies ranging from Goldman Sachs to Capital One discussed their “open-source first” attitude when it comes to deploying new software and infrastructure. And as more open-source-based companies go public in the coming years, we’ll get even more information about how these top projects are performing, and will continue to track their progress and influence on the industry.

*Denotes a Battery portfolio company. 

Nothing here is, or should be considered, investment advice or any recommendation or solicitation to buy or sell any security. Certain information in this article has been obtained from third-party sources and, although believed to be reliable, has not been independently verified as to its accuracy, and its completeness cannot be guaranteed.

More TechCrunch

“When I heard the released demo, I was shocked, angered and in disbelief that Mr. Altman would pursue a voice that sounded so eerily similar to mine.”

Scarlett Johansson says that OpenAI approached her to use her voice

The European venture capital firm raised its fourth fund as fund as climate tech “comes of age.”

ETF Partners raises €284M for climate startups that will be effective quickly — not 20 years down the road

Copilot, Microsoft’s brand of generative AI, will soon be far more deeply integrated into the Windows 11 experience.

Microsoft wants to make Windows an AI operating system, launches Copilot+ PCs

Hello and welcome back to TechCrunch Space. For those who haven’t heard, the first crewed launch of Boeing’s Starliner capsule has been pushed back yet again to no earlier than…

TechCrunch Space: Star(side)liner

When I attended Automate in Chicago a few weeks back, multiple people thanked me for TechCrunch’s semi-regular robotics job report. It’s always edifying to get that feedback in person. While…

These 81 robotics companies are hiring

The top vehicle safety regulator in the U.S. has launched a formal probe into an April crash involving the all-electric VinFast VF8 SUV that claimed the lives of a family…

VinFast crash that killed family of four now under federal investigation

When putting a video portal in a public park in the middle of New York City, some inappropriate behavior will likely occur. The Portal, the vision of Lithuanian artist and…

NYC-Dublin real-time video portal reopens with some fixes to prevent inappropriate behavior

Longtime New York-based seed investor, Contour Venture Partners, is making progress on its latest flagship fund after lowering its target. The firm closed on $42 million, raised from 64 backers,…

Contour Venture Partners, an early investor in Datadog and Movable Ink, lowers the target for its fifth fund

Meta’s Oversight Board has now extended its scope to include the company’s newest platform, Instagram Threads, and has begun hearing cases from Threads.

Meta’s Oversight Board takes its first Threads case

The company says it’s refocusing and prioritizing fewer initiatives that will have the biggest impact on customers and add value to the business.

SeekOut, a recruiting startup last valued at $1.2 billion, lays off 30% of its workforce

The U.K.’s self-proclaimed “world-leading” regulations for self-driving cars are now official, after the Automated Vehicles (AV) Act received royal assent — the final rubber stamp any legislation must go through…

UK’s autonomous vehicle legislation becomes law, paving the way for first driverless cars by 2026

ChatGPT, OpenAI’s text-generating AI chatbot, has taken the world by storm. What started as a tool to hyper-charge productivity through writing essays and code with short text prompts has evolved…

ChatGPT: Everything you need to know about the AI-powered chatbot

SoLo Funds CEO Travis Holoway: “Regulators seem driven by press releases when they should be motivated by true consumer protection and empowering equitable solutions.”

Fintech lender SoLo Funds is being sued again by the government over its lending practices

Hard tech startups generate a lot of buzz, but there’s a growing cohort of companies building digital tools squarely focused on making hard tech development faster, more efficient and —…

Rollup wants to be the hardware engineer’s workhorse

TechCrunch Disrupt 2024 is not just about groundbreaking innovations, insightful panels, and visionary speakers — it’s also about listening to YOU, the audience, and what you feel is top of…

Disrupt Audience Choice vote closes Friday

Google says the new SDK would help Google expand on its core mission of connecting the right audience to the right content at the right time.

Google is launching a new Android feature to drive users back into their installed apps

Jolla has taken the official wraps off the first version of its personal server-based AI assistant in the making. The reborn startup is building a privacy-focused AI device — aka…

Jolla debuts privacy-focused AI hardware

The ChatGPT mobile app’s net revenue first jumped 22% on the day of the GPT-4o launch and continued to grow in the following days.

ChatGPT’s mobile app revenue saw its biggest spike yet following GPT-4o launch

Dating app maker Bumble has acquired Geneva, an online platform built around forming real-world groups and clubs. The company said that the deal is designed to help it expand its…

Bumble buys community building app Geneva to expand further into friendships

CyberArk — one of the army of larger security companies founded out of Israel — is acquiring Venafi, a specialist in machine identity, for $1.54 billion. 

CyberArk snaps up Venafi for $1.54B to ramp up in machine-to-machine security

Founder-market fit is one of the most crucial factors in a startup’s success, and operators (someone involved in the day-to-day operations of a startup) turned founders have an almost unfair advantage…

OpenseedVC, which backs operators in Africa and Europe starting their companies, reaches first close of $10M fund

A Singapore High Court has effectively approved Pine Labs’ request to shift its operations to India.

Pine Labs gets Singapore court approval to shift base to India

The AI Safety Institute, a U.K. body that aims to assess and address risks in AI platforms, has said it will open a second location in San Francisco. 

UK opens office in San Francisco to tackle AI risk

Companies are always looking for an edge, and searching for ways to encourage their employees to innovate. One way to do that is by running an internal hackathon around a…

Why companies are turning to internal hackathons

Featured Article

I’m rooting for Melinda French Gates to fix tech’s broken ‘brilliant jerk’ culture

Women in tech still face a shocking level of mistreatment at work. Melinda French Gates is one of the few working to change that.

1 day ago
I’m rooting for Melinda French Gates to fix tech’s  broken ‘brilliant jerk’ culture

Blue Origin has successfully completed its NS-25 mission, resuming crewed flights for the first time in nearly two years. The mission brought six tourist crew members to the edge of…

Blue Origin successfully launches its first crewed mission since 2022

Creative Artists Agency (CAA), one of the top entertainment and sports talent agencies, is hoping to be at the forefront of AI protection services for celebrities in Hollywood. With many…

Hollywood agency CAA aims to help stars manage their own AI likenesses

Expedia says Rathi Murthy and Sreenivas Rachamadugu, respectively its CTO and senior vice president of core services product & engineering, are no longer employed at the travel booking company. In…

Expedia says two execs dismissed after ‘violation of company policy’

Welcome back to TechCrunch’s Week in Review. This week had two major events from OpenAI and Google. OpenAI’s spring update event saw the reveal of its new model, GPT-4o, which…

OpenAI and Google lay out their competing AI visions

When Jeffrey Wang posted to X asking if anyone wanted to go in on an order of fancy-but-affordable office nap pods, he didn’t expect the post to go viral.

With AI startups booming, nap pods and Silicon Valley hustle culture are back