Prepare for sacrilege. But please read to the bottom before flaming me.
Penetration testing is a farce and largely a waste of time and money.
There, I feel better, I've said it. Come on, there are really only two possible outcomes to a penetration test:
- ONE: You confirm something you already know, that you are vulnerable to a sufficiently skilled and determined attacker.
- If you don't know this already, abandon hope.
- Really, just go into sales.
- If you are that good at lying to yourself it will be easy to lie to others.
- Just kidding.
- Mostly.
OR
- TWO: The Penetration Tester you hired isn't good enough.
- Or more likely, you prevented them from doing their job.
We don't even agree about what "Penetration Testing" is, and most people are totally 
clueless about it. Sure, you know what it means and so do I- but I bet it means different things to us. It is a term that is used for everything from a cursory Nessus or Saint scan, to a full-blown attempt to penetrate and compromise any or all aspects of a system or environment by any method. Whatever that means. And not enough people agree on the meanings for the phrase to be valuable. And, yes, you start every engagement by teaching the manager/customer/whoever exactly what *you* mean by "Penetration Test". Just like every other good Pen Tester, and you all contradict each other, at least a little. That doesn't help.
Another problem I have with the "Pen Testing Industry" is the offensive and ignorant term which accompanies it: Ethical Hacker. Or, heaven forbid, Certified Ethical Hacker. If what that implies about real hackers doesn't infuriate you, we probably really disagree about what hacker means. And you are wrong. This one isn't just my opinion, people you should respect feel the same way.
Possibly most damning, Penetration Testing has taken on a life of its own, independent of the greater business- always a mistake in the greater scheme of things.
Does this mean we shouldn't test things, challenge systems, push limits until things break? Of course not, but it needs to stop being a stand-alone, bolt-on afterthought.
The answer certainly isn't as simple as "building in security" as some people claim. First and foremost, I think it should be obvious that "building security in" from the beginning isn't that simple. That needs to be a goal, but people make mistakes and deploy things poorly, threats evolve and new ones emerge. We need to test and challenge systems to make sure they are secure, and to find holes so we can address them. The key is that we need to integrate the efforts to get things right the first time with continued testing and corresponding remediation- in alignment with and in support of the needs of the business. Consider that the Pen Tester often knows more about securing systems than the people trying to secure the information because they moved "up" from administration positions into their "security" roles- and so the people most able to secure the systems are only responsible for finding the problems. Don't even start with the "we write great reports detailing remediation best practice yada, yada, yada". Those reports are sitting right next to the audit reports and policies which haven't been seriously reviewed or updated since the last time something forced that. I know that it isn't always as ugly as I've presented it, but it often is. Maybe even "almost always" that ugly.
Pen Testing isn't going away anytime soon, nor should it. It isn't going to be absorbed into the larger security process quickly, either. But the sooner it is integrated into the overall security and business needs of the enterprise, the more secure our information will really be. And that's the goal, right? At least it is my goal.
Jack
