Brett Stevens
How much more productive are developers using AI coding tools? Recently, there has been a lot of speculation that AI makes developers 2x, 3x, or even 5x more productive. One report predicts a tenfold increase in developer productivity by 2030.
The irony, however, is that the engineering community has, for the most part, not been able to agree upon a universal way to measure engineering productivity. Some have even rejected the idea altogether, arguing that most metrics are flawed or imperfect. Most of the claims around AI improving productivity today are qualitative — based on surveys and anecdotes, and not on quantitative data.
How can we make judgments about AI without first agreeing on how to measure productivity? If we learned anything from the remote work experiment, it’s that we floundered without data to inform our decisions — shifting back and forth between office, remote, and hybrid strategies based on dogma and ideology instead of data and measurement.
We’re on a path to repeat ourselves with AI. To move forward, we must first understand and quantify its impact.
The risk of falling behind
The current hype around AI may give some of us reason to pause — due to the unknown impact to quality, the potential risk of plagiarism and other factors. The most cautious companies have entered a holding pattern, waiting to see how it all plays out.
For tech-enabled businesses, however, the risk of falling behind is existential. AI is a double accelerant, impacting both what and how companies build. Companies that invest in AI today have the potential to double dip by bringing to market not only new AI-powered products, but also products to market faster and more cheaply.
Most companies have been focused on the what, but AI could be the driver for the how, creating the 10x or even 100x engineering team. Companies that figure out how to quickly cross the chasm — by optimizing AI tools in the most efficient and impactful way — and reach the plateau of productivity faster will benefit from a head start for years to come. The risk of doing nothing is too high.
Understanding the trade-offs
To someone with a hammer, everything looks like a nail. So, too, with AI.
According to a recent GitHub report, the top benefit of AI coding tools cited by developers was improving their coding language skills. Another key benefit is automating repetitive tasks, like writing boilerplate code. A recent experiment by Codecov showed that ChatGPT performs well at writing simple tests for trivial functions and relatively straightforward code paths.
But like any technology, AI has its trade-offs. For instance, generative AI and LLMs are not capable of highly complex or creative tasks, such as migrating a codebase from one architecture to another or incorporating business logic into a new feature. They might even open unforeseen security holes or legal questions if developers accept AI-generated suggestions without checks for plagiarism, licensing restrictions, or hallucinations.
But AI doesn’t need to be a panacea for it to change how software is written and shipped. Even with these limitations, when used in a few, precise ways to help with the right tasks, it can unlock new doors that make those tasks 10x easier, faster, or cheaper.
Quantifying the impact
Most developers are already using AI-powered tools. According to GitHub, 92% of users in a recent survey have used AI coding tools either at work or in their personal time. That makes it critical that companies establish baselines as early as possible to help compare productivity before and after rolling out new AI tools.
Even simple, proxy metrics can provide quantitative insights into the impact of new tools. For instance, in our research from our community of 400,000+ developers, we found that developers using GitHub Copilot are inserting 1.3x more characters per keystroke and 1.22x more lines of code in the same amount of time as developers not using the AI coding assistant. While changing more lines of code does not necessarily equate to improving productivity, the increasing rate at which developers using GitHub Copilot can write code — whether it’s unit tests, functions, or other boilerplate code — provides evidence that they are saving time and effort on repetitive tasks.
Similarly, measuring how many features your team is able to deliver before and after investing in AI tools can help you quantify their impact. Improvements in the number of features delivered per engineer (alongside a lower cost per feature delivered) make a stronger business case for continuing to invest in AI tools.
Comparing features delivered to churn and refactor can also help you understand the impact that AI has on quality. Companies experiencing quality issues as a result of rolling out new AI coding tools will spend more time on churn and refactor and less time developing new features.
While some amount of churn and refactor is necessary, an increasing amount may indicate that the ROI of those tools is less compelling.
Making the right investments
In order for us to answer whether AI improves developer productivity, we must first improve our visibility into its impact — where, when, and how in the development process AI is likely to have the biggest return on investment. With so much hype and excitement around generative AI, it is more important than ever to uncover the right data to get to the truth.
As companies invest in AI development tools, they can empower their teams by applying observability to software development. Observability not only helps you learn faster about the limits and opportunities with AI, but it also opens the door to conversations about other factors that impact productivity, like remote work, DevOps, and internal developer platforms.
There can be little doubt that AI has a role in improving productivity, but let’s examine the facts to understand how — and then make informed decisions.
Comment