Tuesday, September 22, 2009

An insightful pair of reports

Where do you get information on products for your environment?  There are a lot of options- in fact there is no shortage of people with opinions who will happily share.  Now, the hard bit- Where do you get information you can rely on for products for your environment?  There are a lot of analyst firms, labs, websites, magazines, etc. that generate product comparisons, product evaluations, market analysis, and even the ubiquitous "shootouts" (which generally make we want to shoot something- or someone).  Sadly, most of it is crap for a variety of reasons: tainted (or at least made suspect) by vendor sponsorships, failure to define parameters and procedures, misconfiguration of tested systems, a fundamental misunderstanding of the products/market segment, and so on.  Some reports crank out a lot of data, but return very little information, and nothing which you can act on.  Sometimes you can parse the raw data yourself and come up with better conclusions, or at least more relevant to your needs.

There are very good resources available, but you need to do a lot of filtering to find reliable sources of useful information.  That is why personal recommendations are so valuable, and part of the appeal of real-world gatherings such as user groups, and virtual water coolers such as the Security Blogger's Network and the Security Twits- because real people are behind the answers.

These thoughts were triggered when I was fortunate enough to get previews of two reports released this week by NSS Labs. I hate to sound like a fanboy, but they have really put some thought into their analyses on endpoint security for web threats.  Actually, the full title is "Endpoint Security, Socially Engineered Malware Protection, Comparative Test Results.  That's a mouthful.  There are two similar reports, one for consumer products, and one for corporate products.  These are not "Anti-virus shootouts" or anything vague.  The reports define a specific problem (web-based malware downloads) and define their testing methodology, including steps to insure consistent testing.  The testing cycles were repeated, and they used live systems for testing, not canned data sets.  When the tests were complete and validated, the data showed some interesting things.  Both general and specific conclusions come from the tests.  Global observations include the increasing importance of reputation-based services in the cloud, and that no matter what anyone says, anti-malware packages are not "commodities", there are significant differences in performance between the tested systems.  That leads to the specific, the products which performed best dramatically outperformed the worst for the specific threats tested.  The consumer products report is available free (registration required), the corporate products report is not free- but depending on your environment, the $1800 price tag could be trivial compared to the cost of making a mistake in purchasing endpoint protection products.  You can extrapolate some things from the consumer report, but the corporate version includes some additional observations on the ease of management, and there are real differences in performance between corporate and consumer products.

When you need information, be skeptical, but keep looking- there is good information out there.