Prepare for sacrilege. But please read to the bottom before flaming me.
Penetration testing is a farce and largely a waste of time and money.
There, I feel better, I've said it. Come on, there are really only two possible outcomes to a penetration test:
- ONE: You confirm something you already know, that you are vulnerable to a sufficiently skilled and determined attacker.
- If you don't know this already, abandon hope.
- Really, just go into sales.
- If you are that good at lying to yourself it will be easy to lie to others.
- Just kidding.
- Mostly.
OR
- TWO: The Penetration Tester you hired isn't good enough.
- Or more likely, you prevented them from doing their job.
We don't even agree about what "Penetration Testing" is, and most people are totally 
clueless about it. Sure, you know what it means and so do I- but I bet it means different things to us. It is a term that is used for everything from a cursory Nessus or Saint scan, to a full-blown attempt to penetrate and compromise any or all aspects of a system or environment by any method. Whatever that means. And not enough people agree on the meanings for the phrase to be valuable. And, yes, you start every engagement by teaching the manager/customer/whoever exactly what *you* mean by "Penetration Test". Just like every other good Pen Tester, and you all contradict each other, at least a little. That doesn't help.
Another problem I have with the "Pen Testing Industry" is the offensive and ignorant term which accompanies it: Ethical Hacker. Or, heaven forbid, Certified Ethical Hacker. If what that implies about real hackers doesn't infuriate you, we probably really disagree about what hacker means. And you are wrong. This one isn't just my opinion, people you should respect feel the same way.
Possibly most damning, Penetration Testing has taken on a life of its own, independent of the greater business- always a mistake in the greater scheme of things.
Does this mean we shouldn't test things, challenge systems, push limits until things break? Of course not, but it needs to stop being a stand-alone, bolt-on afterthought.
The answer certainly isn't as simple as "building in security" as some people claim. First and foremost, I think it should be obvious that "building security in" from the beginning isn't that simple. That needs to be a goal, but people make mistakes and deploy things poorly, threats evolve and new ones emerge. We need to test and challenge systems to make sure they are secure, and to find holes so we can address them. The key is that we need to integrate the efforts to get things right the first time with continued testing and corresponding remediation- in alignment with and in support of the needs of the business. Consider that the Pen Tester often knows more about securing systems than the people trying to secure the information because they moved "up" from administration positions into their "security" roles- and so the people most able to secure the systems are only responsible for finding the problems. Don't even start with the "we write great reports detailing remediation best practice yada, yada, yada". Those reports are sitting right next to the audit reports and policies which haven't been seriously reviewed or updated since the last time something forced that. I know that it isn't always as ugly as I've presented it, but it often is. Maybe even "almost always" that ugly.
Pen Testing isn't going away anytime soon, nor should it. It isn't going to be absorbed into the larger security process quickly, either. But the sooner it is integrated into the overall security and business needs of the enterprise, the more secure our information will really be. And that's the goal, right? At least it is my goal.
Jack
6 comments:
Ok, let me start off with I think you have some extremely valid points BUT:
1. I think you are wrong about the outcomes. Penetration Tests, at leas my definition of them allows for a step above Vulnerability Assessment (another term that is loosely defined). Where VA simply does scans and reports back using automated tools, a PT will discover pivot points, 'low hanging fruit'. If I pop through Nessus or any other "Enterprise" VA software it isn't going to tell me that that Kiosk that we forgot about is vulnerable to attack. And when you are dealing with large organizations you don't have 'security' people at each site.
2. I am a C|EH and offended by the title, but it was job required. So I can be mad at E-Council all I want but I have to remember that their crappy cert has enabled me to do more fun stuff at work.
3. I completely agree and have first hand experience with PT reports sitting on desks without being read. As well as VA and policy write ups. That is something I will never quite understand, why physical defense is easy. A: Sir we need a lock on this door because the room contains things we don't want stolen. B: Great, go buy it immediately. When virtual / cyber defense is so much more of a struggle, and I think vendors play a huge role in that because of the prices their charge for their products.
That's all the flame I have for now, flame services will be restored in the next 24 to 65535 hours. Thank you.
I won't flame you Jack as you make some really good points and note some of the fundamental issues with pentesting. Being a pentester myself I have been on both sides like mubix and have seen the reports sit and collect dust. However, since I manage a internal corporate pentest team I have to deal with the remediation tracking of the issues that the team finds. What does the company do after the pentester leaves? Do they build the recommendations into their policies, procedures and programs? This is the hardest part for most companies...the simple holes are easy to fix but the greater issue is how can the pentest results help "change" the culture of the organization.
Like mubix said, a good pentest (not just a network pentest either) will find the "low hanging fruit". I would say that most of this "fruit" in my experience has lead to real "business damaging" findings..way more serious then getting admin and owning the domain.
I believe that the "pentest" has to evolve to test all aspects of security (people, process and technology). The only way I have found the pentest truly valuable is for the results to show the real business impact that an attacker (meaning everything from error prone internal user to dedicated attacker) could do to an organization or area of the business. This includes using rather unconventional pentest methods if needed (physical, social engineering, etc..). That is the challenge of the pentest as I see it.
Jack, like the others, I agree with some of your points but think you're missing others. One use of a pentest which most people don't pick up on is *enforcement*. For example, if you're responsible for another entity but don't have control over their security, you need regular pentests to demonstrate whether or not they're doing their basic job. This is an area where a traditional black-box pentest really does have its uses.
That said, if I order up a pentest, I do prefer it to be crystal-box because I don't like to pay for someone to tell me something I already know. I show them the results from previous pentests, tell them what else we know about, and then say, "Go for it, surprise me." And in any reasonably large enterprise, there will be *plenty* I don't know, especially if I don't have the time/staff/expertise to find it out myself.
A good pentester will tackle the weaknesses in business logic (see the excellent talk by Jeremiah Grossman). I don't settle for a pure tools-based "vulnerability assessment" either. Although again, if you haven't been able to do this yourself, you'll definitely get value even from that.
Finally, another reason why you will never be able to do away with the "bolt-on pentest" is that you can't "build security in" to legacy systems and applications (they're already there, duh). You're never starting with a clean slate; you have to work on learning and securing what's there. So by definition you need discovery work done, and it might as well be an objective 3rd party that does it.
Thanks, my spleen feels much lighter now.
I'm honored, insightful comments and many good points raised by three people I respect. I'm sure my SMB background skews my perspective towards the "quick and dirty" end of the PenTest spectrum, so maybe things aren't quite as bad as I make them out. But it looks like we all agree that there are still some significant hurdles to getting Pen Tests to where they need to be.
I always enjoy reading your work, Jack, and this entry certainly lives up to your blog's title!
I'd add that much of your context here seems to assume security happens between IT pros and security practitioners. But a lot of it (most of it?) has to be financed by non-technical execs. If you need to motivate them to spend money, what's the alternative to a pen-test report? Without it, we become Roy Scheider in Jaws, running up and down the beach screaming that the sharks are coming -- but the execs won't close the beach until somebody dies.
Sure, intrusions and data breaches are great motivators for getting better security budget -- but short of that, the humble pen test seems like the next best motivator. Is it not serviceable, despite the shortcomings you point out? Seriously... is there anything you like better for proving to non-security people that there is an urgent security need?
Nice try at controversy, but no.
Either you work with broken outfits who haven't already learned the lessons you outline years back, or a shop that does this sort of work outside a comprehensive ST&E type of umbrella -- either way, it's your own assumptions that are coloring the analysis here.
Just because a broken shop has the ability to negate the business value of almost any process, given the right lack of clue and ability, doesn't mean assessment work is lacking in business value.
That you even mention CEH speaks volumes about where you're coming at this from, as the people I know doing this work for real, have been doing it for so long, CEH is barely a blip on their screen.
There are plenty of shops who get results and value from this work; if yours isn't one of them, blame your people, don't cry "fallacy" on the technique.
Post a Comment