I have a habit of tearing up the various reports and surveys that wander past my view in the world of information security. This is often really unkind of me, because we need to share more information on what works and what doesn’t if we are going to move forward in this struggle to protect whatever it is we’re trying to protect. Companies like Veracode, Verizon, Mandiant, Trustwave, and others put a lot of effort into sanitizing, organizing, and distributing the information they gather in their various endeavors, and they share it for free (or at least just an email address). In a desert largely devoid of data, these reports are oases of information.
And here I am being an ungrateful bastard, trying to x-ray the teeth of these gift horses, then complaining loudly about gingivitis, impacted molars, selection bias, confirmation bias, corporate agendas, and other things Crest™ and a good flossing will never fix.
The problem is that a lot of the data leaves me wanting more. More details on the data we get, just plain “more data”, and more context. I also want more honesty about the shortcomings of the reports and data. Let’s not even talk about some of the bizarre conclusions. And it makes me crazy (crazier) when I see contradictions in a single report, then one report contradicts another company’s report, then year over year reports appear random rather than additive or complementary.
When you read this year’s Report X from Company Y, ask yourself how the information presented made it into that dataset. In the case of the breach reports remember that they are about failures- organizations which were:
- Compromised
- Discovered it (probably not themselves)
- Called Company Y to help them solve it
- And could afford Company Y’s rates, and paid them
Suppose that skews things? Yeah, me too. Where are the success stories?
If you see me talk about any of the career studies I’m involved in you will generally hear me start talking about known flaws in the data, after the disclaimers and caveats we move into what we feel comfortable saying about what we have collected. Of course, I’m not trying to facilitate a transfer of funds from your organization to mine, so maybe its unfair of me to expect the same from those with a financial motive.
And for closing complaints: stop with the moronic USA Today-style “infographics” which tell me less than text would. Combine the graphics with mixed dark on light and light on dark type/background, add PDF format- and we can’t read them on anything but a large monitor (or in dead-tree mode). Just make the reports available in epub/mobi so I can read them on my terms and not be forced to read them in the deity-forsaken PDF format these always come in.
And, thanks for doing all that work. Just stop making me hate you for it.
Jack
