Stephen Northcutt responds to the SecurityFocus article...

In response to the SecurityFocus piece:
http://www.securityfocus.com/columnists/443
I have the following personal comments and opinions.

In terms of research into assessment and certification, we have learned a lot at GIAC through the years. And the process of applying for ISO certification has been really educational.

I think we all know and agree there are limits to what we can do with multiple choice tests. However, those of you that have been with the GIAC program for a long time have probably noticed measurable improvements in test quality. There are a large number of psychometric scripts running behind the scenes evaluating quality in a number of ways. A year ago, I would have dismissed psychometrics as voodoo, now we are paying for training to get a number of our staff educated to work at the practitioner level.

In addition, the Director of GIAC, Jeff Frisk, has led the charge to bring in scenario questions. We are monitoring these carefully, but they do allow you to test more than regurgitation. They allow you to test knowledge.

I believe GIAC is doing better than any other family of security certifications in actually testing the job someone might do. This is becoming known as role based training and role based assessment or certification and it is important.

What job does the CISSP test your qualification for, what job does CISSP training prepare you to do? Now, to be sure, it makes sense to have an exam for minimal competence, can you speak the language of security, do you understand the core concepts of security. The Common Body of Knowledge does a fine job of that and when they developed it they were far ahead of their time, there were clearly men and women of vision associated with the ISC2 to lead that charge.

We feel the same way about GIAC Security Essentials Certification, though it is not a job per se, we feel it establishes the minimum baseline for the knowledge, skills and abilities that a person with hands on responsibility for system should know from a security perspective.

However, the GSEC is only one of over 20 GIAC certifications and the overwhelming majority of GIAC certifications are based on a role, a set of tasks that one actually accomplishes in the workplace. For instance, when I was doing work for the Missile Defense Agency (BMDO) I was helping them with their perimeter and also with their intrusion detection. So I had two primary roles and there is a GIAC certification for both roles.

Further, there is a need for more senior people to demonstrate mastery and integration of multiple roles and that is the GSE, GSM, GSC and so forth.And those certifications are not just multiple choice, they are hands on,and require discussion of theory as well.

But you always want to do more, to push the envelope, that is why I am really excited that at Network Security 2007. In conjunction with White WolfSecurity, we will have our first Marathon Capstone. It is a cyber exercise, not a capture the flag event, but rather a multiple day hands on event where participants will have one role and they will be evaluated on their performance in their role. A seasoned ( all GIAC certified ) set of penetration testers will provide the attacks against the various stations.We have already formed an advisory board to determine what "passing" is and they will start to work as soon as we finish with all the legal paperwork.Participants that pass will be given a certificate with a joint GIAC/WhiteWolf Security trust mark.

Permission is granted to forward this note, post this note on blogs etc, so long as it is attributed to Stephen Northcutt, speaking as an individual, not speaking for GIAC, as I am no longer the Director or GIAC and that the note is not edited or modified.

Stephen Northcutt - Writer, speaker, security thought leader
(808) 823 1375