Monitoring is critical to successful AI

Companies often identify AI and ML performance issues after the damage has been done

As the world becomes more deeply connected through IoT devices and networks, consumer and business needs and expectations will soon only be sustainable through automation.

Recognizing this, artificial intelligence and machine learning are being rapidly adopted by critical industries such as finance, retail, healthcare, transportation and manufacturing to help them compete in an always-on and on-demand global culture. However, even as AI and ML provide endless benefits — such as increasing productivity while decreasing costs, reducing waste, improving efficiency and fostering innovation in outdated business models — there is tremendous potential for errors that result in unintended, biased results and, worse, abuse by bad actors.

The market for advanced technologies including AI and ML will continue its exponential growth, with market research firm IDC projecting that spending on AI systems will reach $98 billion in 2023, more than two and one-half times the $37.5 billion that was projected to be spent in 2019. Additionally, IDC foresees that retail and banking will drive much of this spending, as the industries invested more than $5 billion in 2019.

These findings underscore the importance for companies that are leveraging or plan to deploy advanced technologies for business operations to understand how and why it’s making certain decisions. Moreover, having a fundamental understanding of how AI and ML operate is even more crucial for conducting proper oversight in order to minimize the risk of undesired results.

Companies often realize AI and ML performance issues after the damage has been done, which in some cases has made headlines. Such instances of AI driving unintentional bias include the Apple Card allowing lower credit limits for women and Google’s AI algorithm for monitoring hate speech on social media being racially biased against African Americans. And there have been far worse examples of AI and ML being used to spread misinformation online through deepfakes, bots and more.

Through real-time monitoring, companies will be given visibility into the “black box” to see exactly how their AI and ML models operate. In other words, explainability will enable data scientists and engineers to know what to look for (a.k.a. transparency) so they can make the right decisions (a.k.a. insight) to improve their models and reduce potential risks (a.k.a. building trust).

But there are complex operational challenges that must first be addressed in order to achieve risk-free and reliable, or trustworthy, outcomes.

5 key operational challenges in AI and ML models

Model decay: Unlike traditional software, performance of AI or ML models can decay over time. This is because data is at the heart of any prediction and can become stale. By continuously monitoring model outcomes for decay, companies will experience immediate improvement.

For example, when online stores utilize ML for predicting product purchase recommendations, monitoring for when the shopper consistently does not add the recommended product to the cart enables the brand to identify shifts in purchasing behavior. Being aware of this is critical to online brands’ ability to keep their predictions fresh by continuously addressing changes in customer purchasing decisions.

Data drift: While each model is trained with specific data sets, they can encounter different data once in production. This can result in the models making inaccurate predictions that don’t apply to the desired outcome. Simply put, models that are trained on a specific data set often result in data drifting from its original inputs. And in many cases, it may not be possible to spot the accuracy of a model’s prediction until it’s too late.

For example, when a model is being used for a loan on a boat, the initial data set may be trained for that specific loan. However, once in production, the data drifts to include loans for a house or a car, etc. and may inaccurately assess a credit risk where it’s not warranted. Monitoring data distribution changes functions as a healthy checks and balances system to fine-tune and improve model performance.

Data integrity: Because data is dynamic, its composition is constantly changing, and as a result, such inconsistency can go unnoticed in deployed AI and ML systems. This can have an adverse performance impact on ML models, especially with automated data pipelines.

For example, online stores and airlines frequently experience data inconsistencies for their automated product recommendations. Without monitoring, newly cataloged items, discounts or even ZIP codes could be mislabeled or incorrectly attributed to shoppers, causing confusion for both the business and the customer. Explainability enables businesses to know how to look for these occurrences and understand why it’s happening so they can improve their models and instill trust.

Outliers: Deployed AI and ML models can encounter data from far outside the training parameters, potentially resulting in isolated performance issues that are difficult to detect and remove. There is even the possibility of models malfunctioning. Continuous monitoring allows businesses to pinpoint outliers in real time, and explainability provides those tasked with monitoring with key insights for addressing issues immediately.

Bias: Because AI and ML models typically use historical data as an input source and look for generic patterns of behavior, it doesn’t necessarily factor in gender and racial biases, thus generating potentially compromised output. Unfortunately, an AI or ML model has the potential to become biased after deployment. Even after monitoring for data changes, its true impact on minority groups such as women, LGBTQ+ and people of color might change despite model validation.

For example, an AI or ML model recommending job candidates could have a meaningfully different sensitivity or recall in male versus female applicants between training and production systems. Monitoring for bias can spot and correct these errors when a system deviates toward an unfair output.

Continuous monitoring through explainability is the single most important system that companies need to implement today. With visibility into AI and ML models, companies will be able to pinpoint underlying issues early. Coupled with the right insights through explainability, data scientists and engineers overseeing these models will be armed with the right information to take actionable steps that ensure they overcome these key operational challenges and drive risk-free and trustworthy business outcomes.