Arun Kejariwal (Independent) and Ira Cohen (Anodot) share a novel two-step approach for building more reliable prediction models by integrating anomalies in them. The first step uses anomaly detection algorithms to discover anomalies in a time series in the training data. In the second, multiple prediction models, including time series models and deep networks, are trained, enriching the training data with the information about the anomalies discovered in the first step.

Anomaly detection for individual time series is a necessary but insufficient step due to the fact that anomaly detection over a set of live data streams may result in anomaly fatigue, thereby limiting effective decision making. One way to address the above is to carry out anomaly detection in a multidimensional space. However, this is typically very expensive computationally and hence not suitable for live data streams. Another approach is to carry out anomaly detection on individual data streams and then leverage correlation analysis to minimize false positives, which in turn helps in surfacing actionable insights faster.

They then walk you through marrying correlation analysis with anomaly detection, discuss how the topics are intertwined, and detail the challenges you may encounter based on production data. They also showcase how deep learning can be leveraged to learn nonlinear correlation, which in turn can be used to further contain the false positive rate of an anomaly detection system.

This session was recorded at the 2019 O'Reilly Artificial Intelligence Conference in New York.