AI is poised to become a significant and ubiquitous presence in our lives. It holds tremendous potential value, but we cannot contribute meaningfully to a technology that we do not understand.
When a user sets out to buy a new piece of technology, they’re not particularly interested in what it might be able to do somewhere down the road. A potential user needs to understand what a solution will do for them today, how it will interact with their existing technology stack, and how the current iteration of that solution will provide ongoing value to their business.
But because this is an emerging space that changes seemingly by the day, it can be hard for these potential users to know what questions they should be asking, or how to evaluate products so early in their life cycles.
With that in mind, I’ve provided a high-level guide for evaluating an AI-based solution as a potential customer — an enterprise buyer scorecard, if you will. When evaluating AI, consider the following questions.
Does the solution fix a business problem, and do the builders truly understand that problem?
Chatbots, for example, perform a very specific function that helps promote individual productivity. But can the solution scale to the point where it is used effectively by 100 or 1,000 people?
The fundamentals of deploying enterprise software still apply — customer success, change management, and ability to innovate within the tool are foundational requirements for delivering continuous value to the business. Don’t think of AI as an incremental solution; think about it as a little piece of magic that completely removes a pain point from your experience.
But it will only feel like magic if you can literally make something disappear by making it autonomous, which all comes back to truly understanding the business problem.
What does the security stack look like?
Data security implications around AI are next level and far outstrip the requirements we are used to. You need built-in security measures that meet or exceed your own organizational standards out of the box.
Here’s a high-level guide for evaluating an AI-based solution as a potential customer — an enterprise buyer scorecard, if you will.
Today, data, compliance, and security are table stakes for any software and are even more important for AI solutions. The reason for this is twofold: First and foremost, machine learning models run against massive troves of data, and it can be an unforgiving experience if that data is not handled with strategic care.
With any AI-based solution, regardless of what it is meant to accomplish, the objective is to have a large impact. Therefore, the audience experiencing the solution will also be large. The way you leverage the data these expansive groups of users generate is very important, as is the type of data you use, when it comes to keeping that data secure.
Second, you need to ensure that whatever solution you have in place allows you to maintain control of that data to continually train the machine learning models over time. This isn’t just about creating a better experience; it’s also about ensuring that your data doesn’t leave your environment.
How do you protect and manage data, who has access to it, and how do you secure it? The ethical use of AI is already a hot topic and will continue to be with imminent regulations on the way. Any AI solution you deploy needs to have been built with an inherent understanding of this dynamic
Is the product truly something that can improve over time?
As ML models age, they begin to drift and start to make the wrong conclusions. For example, ChatGPT3 only took in data through November of 2021, meaning it couldn’t make sense of any events that occurred after that date.
Enterprise AI solutions must be optimized for change over time to keep up with new and valuable data. In the world of finance, a model may have been trained to spot a specific regulation that changes along with new legislation.
A security vendor may train its model to spot a specific threat, but then a new attack vector comes along. How are those changes reflected to maintain accurate results over time? When buying an AI solution, ask the vendor how they keep their models up to date, and how they think about model drift in general.
What does the technical team behind the product look like?
Good companies will be able to talk in great detail about the ML and AI models that underpin the technology. If they can’t speak in depth about the architecture or training models, that should be an immediate red flag.
Know the credentials of the people managing the models, and the infrastructure. Be sure they understand that the infrastructural needs for AI are vastly different than the last generation of software. It’s far more data-dependent and requires analysis of entirely different security angles.
The team needs to have enough collective experience to articulate how they are building their solution and why.
Does the company behind the solution truly understand your use case?
Make sure they can provide insight into similar use cases, and how they have benefited from specific features in unique ways. Ask if they can show you what your peers have accomplished by using the solution, and how that relates to your own organizational challenges.
The right vendors tend to be the ones that have learned and built through firsthand experience. Beware of vendors that appear to simply be chasing a problem versus those that can actually relate to your problem because they’ve been through it themselves.
A good AI vendor should reflect your problem back to you, talk about how AI can solve that problem, and ultimately show you how they’ve built a solution to do just that.
Can you build a tool that is scaled, supported, and secured at an enterprise level, or do you need help from a third-party solution?
There are many tools available that allow companies to build their own baseline models in-house. This “build vs. buy” question is one that comes up often when it’s time for CIOs to buy new solutions.
With AI, it’s really not something you should aim to build yourself unless you know it is something you can definitively maintain over time because you have the necessary resources in-house. Think of it like building a product, with internal customers rather than external.
They have feature requests, support requests, maintenance needs, feedback, complaints. It’s difficult to create a product in a new territory like AI. Instead, work with companies that spend day and night learning, building, and improving in this specific area.
Does the company have relevant and enthusiastic references in your space?
Find examples of other organizations like yours benefiting from the solution. Ask if they’re willing to speak with you about their experience.
For vendors, this can be a bit of a chicken-or-the-egg situation. Startups need first customers, but those customers want references. The good news is that there is a tremendous amount of excitement and enthusiasm around AI.
Most customers don’t know the right path forward, and because it is such a brand-new space, they are typically much more open to working with early-stage vendors.
But the hard work is not about getting that first check. It’s about making those early customers wildly successful so that they become champions of the cause. Risk-averse buyers will wait until others try it, and then eventually join in. Look for vendors that are heavily invested in making you successful, the kind that you would want to be a reference for down the line.