Back in high school, I had to read the book “Innumeracy,” by John Allen Paulos. The subtitle of the book is “Mathematical Illiteracy and Its Consequences.” I guess since I became a math teacher and still remember the book all these years later, the book must have had some kind of an influence on me. I don’t teach math anymore, but it still annoys me when I see smart people make statements that don’t hold up to mathematical scrutiny. Usually, these people aren’t trying to be misleading. They just aren’t looking at what they’re saying through a mathematical lens. That happened a couple days ago while reading Politic’s Morning Cybersecurity newsletter.
The newsletter had a short section on FireEye and their trends report which found that companies have reduced the time needed to identify an attack. d gotten The report had gotten some press for finding that the median number of days attackers were present on a victim’s network had dropped from 205 days in 2014 down to 146 days in 2015. In talking to Politico, FireEye’s GM of Canadian operations Ajay Sood downplayed the finding, saying that, “the improvement was driven by a small number of businesses that discovered breaches themselves,” and that for attacks discovered by third-parties (i.e. not self-identified by the victim) the attacks remain undetected for just as long.
There are a couple things I see wrong with using that reason to downplay the findings. First of all, the report specifically says the median length of time had dropped significantly. Medians are not as influenced by a few outliers as the mean average. A quick refresher: the median is found by lining all the numbers up from shortest to longest and selecting the middle number. The mean is found by adding up every number and dividing by the total number of entries. I won’t go into the details, but this means that the mean is influenced a lot more by a few outliers than the median is. This is why home prices or incomes in a given neighborhood or city are normally given in terms of the median, so that a few really expensive homes or high-income earners don’t skew the results in a misleading way.
There’s a more subtle problem with the FireEye report, too. The number they use is often taken as the gospel truth. The problem is that the report is based on investigations that Mandian/FireEye have been a part of. There are a lot of incidents where an internal security team identifies an attempted breach and responds to it, using internal resources, before the incident is able to become a million dollar problem like Target experienced. (If there aren’t any such instances, then why are companies spending millions on creating Security Operations Centers?) None of these incidents will make it into the Mandiant report, because a third-party consultant wasn’t called in. But, if you wanted a real, mathematically accurate idea of the amount of time attackers are on a victim’s network then those incidents should also be included. That makes it a lot harder question to answer, though. It’s a lot easier just to use Mandiant’s number, which is why you keep seeing it.
The other problem, not related to math, I have with Sood’s comment is that if the median time to discover a breach is coming down because more companies are discovering breaches with their internal systems instead of using the Krebs IDS, that’s a good thing. It doesn’t make any sense to downplay that improvement just because some other companies didn’t improve as fast.
Workentin's Cybersecurity Blog 