Of all enterprise departments, product and engineering spend by far the most on AI technology. Doing so effectively stands to generate huge value — developers can complete certain tasks up to 50% faster with generative AI, according to McKinsey.
But that’s not as easy as just throwing money at AI and hoping for the best. Enterprises need to understand how much to budget into AI tools, how to weigh the benefits of AI versus new recruits, and how to ensure their training is on point. A recent study also found that who is using AI tools is a critical business decision, as less experienced developers get far more benefits out of AI than experienced ones.
Not making these calculations could lead to lackluster initiatives, a wasted budget and even a loss of staff.
At Waydev, we’ve spent the past year experimenting on the best way to use generative AI in our own software development processes, developing AI products, and measuring the success of AI tools in software teams. This is what we’ve learned on how enterprises need to prepare for a serious AI investment in software development.
Carry out a proof of concept
Many AI tools emerging today for engineering teams are based on completely new technology, so you will need to do much of the integration, onboarding and training work in-house.
When your CIO is deciding whether to spend your budget on more hires or on AI development tools, you first need to carry out a proof of concept. Our enterprise customers who are adding AI tools to their engineering teams are doing a proof of concept to establish whether the AI is generating tangible value — and how much. This step is important not only in justifying budget allocation but also in promoting acceptance across the team.
The first step is to specify what you’re looking to improve within the engineering team. Is it code security, velocity, or developer well-being? Then use an engineering management platform (EMP) or software engineering intelligence platform (SEIP) to track whether your adoption of AI is moving the needle on those variables. The metrics can vary: You may be tracking speed using cycle time, sprint time or the planned-to-done ratio. Did the number of failures or incidents decrease? Has developer experience been improving? Always include value tracking metrics to ensure that standards aren’t dropping.
Make sure you’re assessing outcomes across a variety of tasks. Don’t restrict the proof of concept to a specific coding stage or project; use it across diverse functions to see the AI tools perform better under different scenarios and with coders of different skills and job roles.
Hardware capabilities are an essential consideration in your proof of concept. You may even find that your computing power only just covers an experimental integration of AI, but wouldn’t be able to handle the load of a full implementation of the project. In which case, you have to factor in extra CPUs and other hardware into the hypothetical AI budget.
Now you can calculate the value of the AI project as it relates to gains — savings on employees’ salary, time reclaimed, extra productivity — and expenditure on software and hardware. Set benchmarks on how monetary savings and/or productivity gains would make the AI investment worthwhile. If those aren’t met, perhaps it would be more efficient for the company to explore an alternative AI strategy or simply meet their needs with extra staff.
Build a training and knowledge-sharing framework for your team
Whether you keep your core team or expand your team as you integrate AI tools, you need to make AI a pillar of the onboarding and upskilling process. Many AI tools emerging today for engineering teams are based on completely new technology, so you will need to do much of the integration, onboarding and training work in-house. Don’t underestimate how much effort will go into this framework.
Once you’ve decided on the tools you will be integrating, or developing in-house, build your own internal documents and guidelines on how to best use AI. These need to include when and where you can use the tools, what kind of data you can and can’t upload to the platform (e.g., what to do with sensitive or non-anonymized client data), risks to be aware of, and more.
When onboarding a new tool, make sure you give all recruits immediate access to the AI tools within their own sandbox so they can start experimenting with it without impacting workflow. This makes for faster training and also gives employees a chance to ask questions and flag issues.
Invest in knowledge sharing across your team, too. Create mechanisms and platforms for people to share not only internal developments regarding AI, but also what people are learning about different AI tools, and news that offers context on relevant AI technology. One mechanism is to have a regular team-pitching meeting. Remember that the whole company needs to be involved. For example, GitHub has a specific research team dedicated to exploring the future of software development, but they aren’t siloed. They communicate with people across teams from product to engineering, getting ideas and feedback from everyone.
Take inspiration from what others are doing
Especially when it comes to generative (or predictive) AI, this is a whole new world for enterprises, so it helps to consider what success looks like in other businesses. Follow similar companies that are talking openly about how they are leveraging AI in software development, from integrated developer environment (IDE) tools to general chatbots. Can you use them as examples?
Joe Welch, principal and technology leader of Launch Consulting, has given tried-and-true examples of how to use AI in software development — for example, using AI to create summaries of subsystems and modules to facilitate onboarding for new developers, with the developers able to ask AI specific questions on the code. Or to facilitate the migration of a codebase from an older language to a newer one, which is hard because it requires developers to be fluent in both languages.
GitHub also has a chatbot allowing users to write and understand code in any language, and is also available on mobile. Backstage built an open source chatbot into a local version of its developer portal.
Whatever your pathway to integrating AI into software development, it’s not a 0 to 1 process. Every step will require careful planning to make sure that your time and money are going toward better overall developer experience and performance, not wasted.