A few weeks ago David Rice, the author of Geekonomics, was on Pauldotcom episode 160. There is no denying that he's a smart guy, but David Rice said some things that just don't work for me, and I feel compelled to address a few of them. Mr. Rice is a big proponent of some kind of testing/certification framework for software to insure security. This sounds good, but the devil is in the details, especially his.
He suggests the testing will need to start out with a low set of standards, and improve/evolve over time, getting tougher until they are a real measurement and enforcement of security. Again, this sounds good, but it is where things start to unravel. He points to IIHS and NHTSA testing as models; according to him they started nearly useless and evolved over decades- which is true to an extent, but fundamentally unsafe cars still pass the tests, and there are many real dangers which 
are not considered in the tests and ratings. Only a specific (and well known to the manufacturers) set of tests are performed, this leads to building cars to pass the test, not necessarily actually be safe (hey, that sounds familiar, doesn't it?). Let's blow auto safety testing out of the water, shall we?
I love Jeeps, and have had several over the years. From the first CJs through the current Wranglers, and it's easy to make the case that they are all horribly "unsafe". The old ones were underpowered, but short, skinny, and prone to rollovers. As they have grown longer and wider, the improved stability has been offset by increased power so they are still rollover-prone. Driven properly, the risks can be minimized, but that means trusting the end-user (we're screwed). But, they "pass" the tests. Maybe not the highest safety ratings, but passing. (By the way, there are no more real Jeeps, the morons at Daimler-Benz and now the current owners of the Jeep name have killed them. They have even put V-6 engines in them. V-6s were a mistake in the '60s and are a heresy now. I-6s belong in Jeeps, an I-4 or V-8 if you must. Bastards).
You need look no further than the lists of safety recalls to see how often and how badly testing fails, and the myriad of things not tested which endanger you, me and our families (real world injured or dead on the road, no buffer overflow nuisances). Improper child seat mounting, mis-welded steering columns, overheating electronics, motorcycles helmets which fail to meet the standards, tow bars (for above mentioned Jeep Wranglers), leaking fuel filters- and that's just the first page and a half (of eighteen pages) in the report for the single month of July 2009.
Think about it, how can slamming a car into a barrier tell you that the lower control arms are formed so that they will trap sand, salt, and moisture- then rust out in two years and cause your suspension to separate when you hit a pothole "just right". That takes a level of inspection and review we haven't gotten to yet, and probably never will. And I can shine a light on the arm and hit it with a hammer- try that with software testing. 
One of my favorite examples of not testing for real-world safety is the moose. In limited parts of the world moose collisions are extremely common, and they are extremely dangerous wherever they happen. One of the primary problems is that cars are simply not designed to withstand an impact from the front against the a-pillars (the uprights between the front doors and the windshield), and they collapse easily. It is a known "defect", but it is not tested for, and the cost of correcting the defect is apparently not justified- and people die, avoidably, every year because of it.
Has that dampened your enthusiasm for automotive testing as a model for software assurance?
He also mentioned public pressure and demand for safer vehicles as supporting the improved safety, but the two highest profile news stories I recall about auto safety were both fraudulent hype. First we had the "runaway Audis", which supposedly accelerated out of control on their own. Audi maintained that the drivers were simply mistakenly stomping on the gas instead of the brake in all of the events, but 60 Minutes did a segment where they showed the engine of an Audi racing away "by itself" and drove public panic and outrage (costing Audi and their dealers untold millions). The "runaway" car 60 Minutes showed? It had its transmission rigged to drive the throttle linkage and artificially accelerate the engine; Audi fixed the operator error by retrofitting the brake/shift interlock system we all take for granted on cars now. The other big one involved GM trucks with fuel tanks outside of the frame rails, something that WAS unsafe, but Dateline NBC felt the need for a bit of drama to make the point. They staged side-impact collisions with the trucks, but didn't get much more than a little fuel leakage. Needing more drama to sell their schtick, they attached small rocket motors to the fuel tanks and rigged the test to guarantee spectacular fires. In the aftermath of the stunt, the real dangers of the design were overshadowed by the fraudulent reporting. I have no need for Fox News, CNN, MSNBC, or any other pack of screaming dimwits driving the discussion about software security. If you like the idea, ask them what a "hacker" is and get back to me.
Moving on...
When Rice said "cars aren't as complicated as software", I lost it. Clearly, he misspoke; surely he cannot believe a automobile where an entire Windows Media Center entertainment system is an afterthought tossed into the dashboard is somehow LESS complicated than the afterthought itself. Even the simplest of automotive components are expected to be functional at a bewildering array of temperatures under a variety of loads and with near-constant vibration. Cars have had microprocessors for decades, and hydraulic computers for decades longer than that. (If you think transistors are clever, try doing similar switching of forces in a high-temperature hydraulic environment which makes your car go down the road at varying speeds, that's what automatic transmissions have done for over half a century).
Finally (for this rantbuttal), how the !@#$ is the idea of a testing/certification framework which starts out merely enforcing current expectations and slowly evolves and becomes more strict over decades (with inevitable stumbles and false starts) until it is as [fundamentally flawed] as automotive crash testing a unicorn-inducing wonderful idea when he suggests it for software...
but the exact same thing is a horrible and destructive idea if it is applied to cardholder data and we call it PCI? Rice thoroughly trashed PCI, dragging out all of the same misguided drivel about it without offering a better alternative or seeing the correlation to his own proposals. Listen to the episode and hold a mirror up to all of his arguments before you answer that. He did a phenomenal job of rebutting himself when he launched into his PCI rant, so I won't try to top him.
Jack
