A few weeks ago David Rice, the author of Geekonomics, was on Pauldotcom episode 160. There is no denying that he's a smart guy, but David Rice said some things that just don't work for me, and I feel compelled to address a few of them. Mr. Rice is a big proponent of some kind of testing/certification framework for software to insure security. This sounds good, but the devil is in the details, especially his.
He suggests the testing will need to start out with a low set of standards, and improve/evolve over time, getting tougher until they are a real measurement and enforcement of security. Again, this sounds good, but it is where things start to unravel. He points to IIHS and NHTSA testing as models; according to him they started nearly useless and evolved over decades- which is true to an extent, but fundamentally unsafe cars still pass the tests, and there are many real dangers which 
are not considered in the tests and ratings. Only a specific (and well known to the manufacturers) set of tests are performed, this leads to building cars to pass the test, not necessarily actually be safe (hey, that sounds familiar, doesn't it?). Let's blow auto safety testing out of the water, shall we?
I love Jeeps, and have had several over the years. From the first CJs through the current Wranglers, and it's easy to make the case that they are all horribly "unsafe". The old ones were underpowered, but short, skinny, and prone to rollovers. As they have grown longer and wider, the improved stability has been offset by increased power so they are still rollover-prone. Driven properly, the risks can be minimized, but that means trusting the end-user (we're screwed). But, they "pass" the tests. Maybe not the highest safety ratings, but passing. (By the way, there are no more real Jeeps, the morons at Daimler-Benz and now the current owners of the Jeep name have killed them. They have even put V-6 engines in them. V-6s were a mistake in the '60s and are a heresy now. I-6s belong in Jeeps, an I-4 or V-8 if you must. Bastards).
You need look no further than the lists of safety recalls to see how often and how badly testing fails, and the myriad of things not tested which endanger you, me and our families (real world injured or dead on the road, no buffer overflow nuisances). Improper child seat mounting, mis-welded steering columns, overheating electronics, motorcycles helmets which fail to meet the standards, tow bars (for above mentioned Jeep Wranglers), leaking fuel filters- and that's just the first page and a half (of eighteen pages) in the report for the single month of July 2009.
Think about it, how can slamming a car into a barrier tell you that the lower control arms are formed so that they will trap sand, salt, and moisture- then rust out in two years and cause your suspension to separate when you hit a pothole "just right". That takes a level of inspection and review we haven't gotten to yet, and probably never will. And I can shine a light on the arm and hit it with a hammer- try that with software testing. 
One of my favorite examples of not testing for real-world safety is the moose. In limited parts of the world moose collisions are extremely common, and they are extremely dangerous wherever they happen. One of the primary problems is that cars are simply not designed to withstand an impact from the front against the a-pillars (the uprights between the front doors and the windshield), and they collapse easily. It is a known "defect", but it is not tested for, and the cost of correcting the defect is apparently not justified- and people die, avoidably, every year because of it.
Has that dampened your enthusiasm for automotive testing as a model for software assurance?
He also mentioned public pressure and demand for safer vehicles as supporting the improved safety, but the two highest profile news stories I recall about auto safety were both fraudulent hype. First we had the "runaway Audis", which supposedly accelerated out of control on their own. Audi maintained that the drivers were simply mistakenly stomping on the gas instead of the brake in all of the events, but 60 Minutes did a segment where they showed the engine of an Audi racing away "by itself" and drove public panic and outrage (costing Audi and their dealers untold millions). The "runaway" car 60 Minutes showed? It had its transmission rigged to drive the throttle linkage and artificially accelerate the engine; Audi fixed the operator error by retrofitting the brake/shift interlock system we all take for granted on cars now. The other big one involved GM trucks with fuel tanks outside of the frame rails, something that WAS unsafe, but Dateline NBC felt the need for a bit of drama to make the point. They staged side-impact collisions with the trucks, but didn't get much more than a little fuel leakage. Needing more drama to sell their schtick, they attached small rocket motors to the fuel tanks and rigged the test to guarantee spectacular fires. In the aftermath of the stunt, the real dangers of the design were overshadowed by the fraudulent reporting. I have no need for Fox News, CNN, MSNBC, or any other pack of screaming dimwits driving the discussion about software security. If you like the idea, ask them what a "hacker" is and get back to me.
Moving on...
When Rice said "cars aren't as complicated as software", I lost it. Clearly, he misspoke; surely he cannot believe a automobile where an entire Windows Media Center entertainment system is an afterthought tossed into the dashboard is somehow LESS complicated than the afterthought itself. Even the simplest of automotive components are expected to be functional at a bewildering array of temperatures under a variety of loads and with near-constant vibration. Cars have had microprocessors for decades, and hydraulic computers for decades longer than that. (If you think transistors are clever, try doing similar switching of forces in a high-temperature hydraulic environment which makes your car go down the road at varying speeds, that's what automatic transmissions have done for over half a century).
Finally (for this rantbuttal), how the !@#$ is the idea of a testing/certification framework which starts out merely enforcing current expectations and slowly evolves and becomes more strict over decades (with inevitable stumbles and false starts) until it is as [fundamentally flawed] as automotive crash testing a unicorn-inducing wonderful idea when he suggests it for software...
but the exact same thing is a horrible and destructive idea if it is applied to cardholder data and we call it PCI? Rice thoroughly trashed PCI, dragging out all of the same misguided drivel about it without offering a better alternative or seeing the correlation to his own proposals. Listen to the episode and hold a mirror up to all of his arguments before you answer that. He did a phenomenal job of rebutting himself when he launched into his PCI rant, so I won't try to top him.
Jack
7 comments:
Jack - Well. Said.
Last year I was speaking to a major exec at a Detroit auto firm who explained defects vs. security vulnerabilities like this... A defect is when you're driving down the street, you put your foot on th pedal to stop and the car does not stop... killing you. A "vulnerability" is a thief jimmying your door lock because of a flaw and stealing your Lexus without a sound.
The different is monumental - but only in the words. Anyway... the IIHS standards are a joke, and always have been... all they do is drive companies to build tanks which (when pitted against a smaller car) will kill the other vehicle's occupants. Software testing is just as bad these days.
C'est la vie.
Jack, I have to agree that testing for software, security and the auto industry are not perfect, and your points about the Jeep are well taken.
I've been doing a lot of thinking on this subject and do think we need to have some testing for at least a minimal threshold of security. I'd like to know that a web page passes a basic level of security. It's not that difficult to test for XSS or SQL injection, but I don't know if any site that I visit passes those tests. Of course, passing those tests doesn't mean it's not vulnerable to a different attack, but at least I know it has seat belts and air bags so to speak.
I'd love to read a post from you on what you think we should be doing proactively to improve security.
It seems to make sense that if the auto makers know which tests will be performed then they will build cars just to pass the tests. Is it possible that these tests could be updated or changed to become less predictable?
What I really mean to say is that it seems like the infosec industry has enough interest to generate a testing system that is constantly improving. This puts the onus on the governing body to maintain the excellence of the cert, not the test itself. We need more inspectors, not more tests.
I think of it like a second grade teacher. Everyone wants the children to be safe, but if she or he was given a list to follow of all the things they were supposed to prevent, that list could be a mile long and there would still be a kid that tapes his leg to the top of the jungle gym. A system must be able to adapt on the fly.
Kind of like the relatively painless software assessments we provide all the time. :)
Dunno why you've got a thing against V6's, but my real comment is this - Any vehicle capable of withstanding impact in it's windshield area with a 1000 pound moose wouldn't be commercially viable.... It's too much weight at exactly the right height.
Raf, thanks for the added perspective. I do see a PCI-like parallel, if we had no standards more people would probably die on the roads, but these standards are being used by many as the MOST they need to do instead of the LEAST.
Becki,
I'm not a developer, so I don't feel qualified to propose a specific set of solutions (not that a lack of qualifications silences too many people). I do think some basic automated scanning would be a decent baseline- as long as we don't let it become a ceiling instead of a floor as often happens with standards.
Marisa, making tests less predictable certainly seems like a good step forward, I think the trick would be in balancing increased complexity with manageability. That is a common tactic in standardized tests, you may get 100 questions out of 1000 possible, it makes it harder to just cram for the test. So, how to apply this, hmmm...
I agree that minimum standards (such as OWASP10/SANS20 issues) are needed, but companies need to follow through with more detailed assessments to assure that more detailed security reviews are performed. PCI is just a baseline to me, but we proscribe to do more than what this single industry regulation mandates.
It is up to us as security professionals to prioritize and budget for the more detailed reviews.
My teams have followed a risk-based approach to prioritize our applications, our services, and our products to focus on the highest risk projects and development efforts happening at my company.
It took us some time to assess our risks, so we took each request one at a time, built our risk assessment approach that have been custom-tailored for each of our types of business, deliver security guidance daily, and re-assess frequently that we're focused on the right efforts, provided the right guidance, and test our testing practices (people, process, technology) to ensure that we’re balanced and current in our approach. It isn’t perfect, but we’re fighting the good fight. We’re doing our job as best that we can.
I hope that this helps.
Regard,
Phil Agcaoili
Post a Comment