Splunk expands machine learning capabilities across platform

Splunk has always been data central for IT operations info, but as the logs fill up with ever-increasing amounts of data, it has become impossible for humans to keep up. Recognizing this, Splunk started building in machine learning and artificial intelligence last year, and this week they are enhancing those capabilities to make it easier to surface the data that’s most critical.

The company has been adding intelligence across the platform, in some cases enabling companies to build their own custom machine-learning powered applications on top of Splunk’s data store, while also introducing automation for those customers who would rather not get their hands dirty in the coding.

“With traditional monitoring you have alert fatigue [from too much information]. We want to use data training and pattern recognition to at least group alerts and surface things that matter most. We’ve taken a two-pronged approach in terms of machine learning. We have the manual stuff for folks who want to work their own models using [a tool like] Scala. We also have the machine learning capabilities baked into the solutions to quickly go in and do things with machine learning,” Jon Rooney, head of product marketing at Splunk told TechCrunch.

Specifically, Splunk wants to become the hub of data coming in and out from sources as varied as security, operations, continuous deployment and the emerging Internet of Things. Then ideally, it wants the software to take action in an automated fashion. Using machine learning takes humans out of the monitoring part of the equation and only alerts them when absolutely necessary, assuming the machine learning models have been tuned correctly.

The Splunk Machine Learning Toolkit has several new key features designed for the DIYers. First of all, there is a new data cleaning tool to get the data ready for the model. Next, there are machine learning APIs to import open source and proprietary algorithms and apply them in Splunk. Finally, there is a machine learning management component to integrate user permissions directly from Splunk into custom machine learning applications.

For those who would prefer Splunk to do some of the work for them, there are also new goodies including Splunk ITSI 3.0, which uses machine learning to help identify issues and prioritize them based on how critical a given operation is to the business. The machine learning components learn over time which functions are most important and which aren’t, and surfaces the crucial ones for operations to deal with.

Rooney says this kind of automation has become an imperative as customers can’t keep up with the amount of data and alerts and need tools to make sense of them. “Nobody is watching green, yellow and red lights and doing the job effectively,” Rooney explained.