Watch all the Transform 2020 sessions on-demand here.
I’ve been doing analytics, data science and statistics for a long time, and I remember when analytics was just called “math.” I walked around the RSA Conference show floor this year, and I would like to apologize on behalf of the entire analytics industry for all the noise out there on the subject. There’s no doubt that AI and analytics are revolutionizing many industries — especially the cybersecurity industry. But it’s not a silver bullet, because there are limitations to what math can actually accomplish. Let’s look at three, in the hopes that it will help you navigate the noise.
1. There is no One Algorithm to rule them all
There is no such thing as a single algorithm that will work in all cases. It’s true that some algorithms are fantastic and new and exciting, but you should be wary of solutions that focus on a single algorithm. The most recent example of this is deep learning. Deep learning is a legitimately exciting technique, and I am absolutely a fan of its potential, but that still doesn’t mean it is the right algorithm for everything.
Deep learning, a class of machine learning algorithms that learns by using large, layered collections of connected processes that rely on labels or examples, is often touted as the way of the future. But it is surprisingly easy to trick deep neural networks. It’s certainly not a silver bullet, but the reality is that there is no silver bullet in mathematical analytics. Deep learning — or any AI technique, for that matter — doesn’t work for every use case. It’s important to understand the strengths and limitations for every technique and choose the right tool for the problem at hand. Don’t get distracted by buzzwords or marketing spin that try to make everything look like the hammer for any and all nails. Just as there is a distinction between a hammer, a mallet and a sledgehammer, there are important differences between different algorithms.
For example, deep learning is pretty good at malware detection, primarily because these use cases involve large datasets of labels: decades’ worth of malware binaries. Most major antivirus and antimalware vendors tout the use of deep learning for their specific objectives, which is largely why it’s become such a prolific topic in our industry. But if your problem is not malware, deep learning may be less effective. When it comes to insider threat detection, for example, deep learning falls surprisingly short. Insider threat scenarios typically deal with limited, labelless datasets — deep learning cannot effectively make sense of this. Imagine trying to break up concrete with a mallet or hanging a picture frame on the wall with a sledgehammer. The right tool can lead to inadequate — or worse, disastrous — results.
June 5th: The AI Audit in NYC
Join us next week in NYC to engage with top executive leaders, delving into strategies for auditing AI models to ensure fairness, optimal performance, and ethical compliance across diverse organizations. Secure your attendance for this exclusive invite-only event.
Ultimately, it’s wise to be skeptical of any vendor that focuses on a single algorithm — it’s not a good way to define the problems you are trying to solve.
2. Math cannot predict for the future anything it hasn’t seen before
Predictive analytics depends on models that infer from existing information. This is similar to the human brain, which learns by observing and experience. If you have never experienced something before, you can’t know what to expect. At best, you may recognize that something new and unprecedented is happening, but something truly novel is unlikely to be something you would have predicted.
Similarly, math cannot predict anything it hasn’t seen before. For example, math would not have predicted speculative execution side-channel attacks, like Spectre, the first time it happened. It’s possible for math to predict it now because we now know what to look for and can potentially create models that can predict the next occurrence. The same is true for the very first instance of ransomware. At best, anomaly detection models would have detected unusual behaviors and alerted security teams to something strangely amiss. It would not, however, have identified the specific problem.
Note that this limitation does not mean predicting a zero-day attack is not possible. It can be done because nearly all zero days are variations on a theme that was seen before in the past, and this past experience makes it predictable.
3. Math cannot read your mind
Math cannot read your mind or infer anything that is not in the data. Ultimately, math is only as good as the data that you feed it. If there is business context that’s in your head or in the heads of your teams that would improve your results, that’s a dataset that needs to be provided and integrated. For example, if your behavioral analytics system detects an employee account who has VPN-ed in from China, takes an unusually large amount of data, and accesses file shares that that employee has never accessed before, the math is pointing out a legitimately concerning set of behaviors. But if you say it is a false positive because this employee took a vacation in China, changed job roles, and is downloading a new set of project files, there is no way for the math to figure that out without, for example, getting access to the HR database, vacation and job role information. The best math in the world is irrelevant without the right data, and the more comprehensive data you can feed a cybersecurity machine learning algorithm, the better it can provide you intelligence.
It’s important to understand what math can’t do so that you can take advantage of what it can do..
Math is magical, but it’s not magic.
Stephan Jou is CTO of Interset.
