In a recent article by Reuters, "Survey: Google draws 64 percent of search queries" , the story confirmed what the majority of web surfers already knew; Google is the top search engine. Quoting from the March survey conducted by Hitwise, Reuters reported that the top three search engines account for 94.5% of all search queries on the web. Google, as the market leader, holds a commanding share of 64.1%, followed by Yahoo's 21.3%, and MSN Search with a 9.1% share of search queries. Now this may not be big news to most web users, but for the marketing folks, advertisers, and webmasters alike, it may mean the difference between a site’s success or failure and the report may also help them decide where to concentrate their resources.
As a webmaster, I am often concerned with conserving bandwidth on my web sites and each being available to the customers and clients that I hope will generate income for me. From the Reuters' story, I know that 94.5% of all web queries are conducted by only three search engines and as a result, I should get "more bang for my buck" if I focus more of my resources on just them. In terms of conserving bandwidth, I recently posted a comment on a site: "How I Invented the Free Lunch" , where I addressed a question (#57) that was raised by one of the readers: “How did you avoid the search engines from slurping up all your bandwidth?” One of the easiest ways to restrict the activities of search engines on a site, yet often overlooked by webmasters, is through the use of a Robots.txt file.
My response on line #82 was:
**********
Most webmasters use a Robots.txt file and a Meta tag to control the activities of search engines. For example, I use the following Meta tag on some sites:
META NAME="ROBOTS" CONTENT="NOINDEX, NOFOLLOW"
and a Robots.txt file in the root directory that limits the access of search engines to only the major ones I want, such as Yahoo and Google. For example:
User-agent: Googlebot
Allow: /
User-agent: Inktomi Slurp
Allow: /
User-agent: *
Disallow: /
I can also restrict the content that search engines may access on my web site by using the Robots.txt file. For example, to prevent Google or others from indexing web site images thus conserving bandwidth, I use the following lines in my robots file:
User-agent: Googlebot-Image
Disallow: /
User-agent: *
Disallow: /images/
If anyone is interested, I would be happy to post an example of a Robots.txt file that you could edit yourself to meet your own needs.
**********
Sound too simple to be true? I know from the use of web stats software that each site is crawled only by the search engines I want and they only access the content I allow. So I can only conclude that the Robots.txt file does indeed work. For more information and tips on using Robots.txt files, please visit: http://www.robotstxt.org/
Friday, May 4, 2007
Wednesday, May 2, 2007
Stephen Northcutt responds to the SecurityFocus article...
In response to the SecurityFocus piece:
http://www.securityfocus.com/columnists/443
I have the following personal comments and opinions.
In terms of research into assessment and certification, we have learned a lot at GIAC through the years. And the process of applying for ISO certification has been really educational.
I think we all know and agree there are limits to what we can do with multiple choice tests. However, those of you that have been with the GIAC program for a long time have probably noticed measurable improvements in test quality. There are a large number of psychometric scripts running behind the scenes evaluating quality in a number of ways. A year ago, I would have dismissed psychometrics as voodoo, now we are paying for training to get a number of our staff educated to work at the practitioner level.
In addition, the Director of GIAC, Jeff Frisk, has led the charge to bring in scenario questions. We are monitoring these carefully, but they do allow you to test more than regurgitation. They allow you to test knowledge.
I believe GIAC is doing better than any other family of security certifications in actually testing the job someone might do. This is becoming known as role based training and role based assessment or certification and it is important.
What job does the CISSP test your qualification for, what job does CISSP training prepare you to do? Now, to be sure, it makes sense to have an exam for minimal competence, can you speak the language of security, do you understand the core concepts of security. The Common Body of Knowledge does a fine job of that and when they developed it they were far ahead of their time, there were clearly men and women of vision associated with the ISC2 to lead that charge.
We feel the same way about GIAC Security Essentials Certification, though it is not a job per se, we feel it establishes the minimum baseline for the knowledge, skills and abilities that a person with hands on responsibility for system should know from a security perspective.
However, the GSEC is only one of over 20 GIAC certifications and the overwhelming majority of GIAC certifications are based on a role, a set of tasks that one actually accomplishes in the workplace. For instance, when I was doing work for the Missile Defense Agency (BMDO) I was helping them with their perimeter and also with their intrusion detection. So I had two primary roles and there is a GIAC certification for both roles.
Further, there is a need for more senior people to demonstrate mastery and integration of multiple roles and that is the GSE, GSM, GSC and so forth.And those certifications are not just multiple choice, they are hands on,and require discussion of theory as well.
But you always want to do more, to push the envelope, that is why I am really excited that at Network Security 2007. In conjunction with White WolfSecurity, we will have our first Marathon Capstone. It is a cyber exercise, not a capture the flag event, but rather a multiple day hands on event where participants will have one role and they will be evaluated on their performance in their role. A seasoned ( all GIAC certified ) set of penetration testers will provide the attacks against the various stations.We have already formed an advisory board to determine what "passing" is and they will start to work as soon as we finish with all the legal paperwork.Participants that pass will be given a certificate with a joint GIAC/WhiteWolf Security trust mark.
Permission is granted to forward this note, post this note on blogs etc, so long as it is attributed to Stephen Northcutt, speaking as an individual, not speaking for GIAC, as I am no longer the Director or GIAC and that the note is not edited or modified.
Stephen Northcutt - Writer, speaker, security thought leader
(808) 823 1375
http://www.securityfocus.com/columnists/443
I have the following personal comments and opinions.
In terms of research into assessment and certification, we have learned a lot at GIAC through the years. And the process of applying for ISO certification has been really educational.
I think we all know and agree there are limits to what we can do with multiple choice tests. However, those of you that have been with the GIAC program for a long time have probably noticed measurable improvements in test quality. There are a large number of psychometric scripts running behind the scenes evaluating quality in a number of ways. A year ago, I would have dismissed psychometrics as voodoo, now we are paying for training to get a number of our staff educated to work at the practitioner level.
In addition, the Director of GIAC, Jeff Frisk, has led the charge to bring in scenario questions. We are monitoring these carefully, but they do allow you to test more than regurgitation. They allow you to test knowledge.
I believe GIAC is doing better than any other family of security certifications in actually testing the job someone might do. This is becoming known as role based training and role based assessment or certification and it is important.
What job does the CISSP test your qualification for, what job does CISSP training prepare you to do? Now, to be sure, it makes sense to have an exam for minimal competence, can you speak the language of security, do you understand the core concepts of security. The Common Body of Knowledge does a fine job of that and when they developed it they were far ahead of their time, there were clearly men and women of vision associated with the ISC2 to lead that charge.
We feel the same way about GIAC Security Essentials Certification, though it is not a job per se, we feel it establishes the minimum baseline for the knowledge, skills and abilities that a person with hands on responsibility for system should know from a security perspective.
However, the GSEC is only one of over 20 GIAC certifications and the overwhelming majority of GIAC certifications are based on a role, a set of tasks that one actually accomplishes in the workplace. For instance, when I was doing work for the Missile Defense Agency (BMDO) I was helping them with their perimeter and also with their intrusion detection. So I had two primary roles and there is a GIAC certification for both roles.
Further, there is a need for more senior people to demonstrate mastery and integration of multiple roles and that is the GSE, GSM, GSC and so forth.And those certifications are not just multiple choice, they are hands on,and require discussion of theory as well.
But you always want to do more, to push the envelope, that is why I am really excited that at Network Security 2007. In conjunction with White WolfSecurity, we will have our first Marathon Capstone. It is a cyber exercise, not a capture the flag event, but rather a multiple day hands on event where participants will have one role and they will be evaluated on their performance in their role. A seasoned ( all GIAC certified ) set of penetration testers will provide the attacks against the various stations.We have already formed an advisory board to determine what "passing" is and they will start to work as soon as we finish with all the legal paperwork.Participants that pass will be given a certificate with a joint GIAC/WhiteWolf Security trust mark.
Permission is granted to forward this note, post this note on blogs etc, so long as it is attributed to Stephen Northcutt, speaking as an individual, not speaking for GIAC, as I am no longer the Director or GIAC and that the note is not edited or modified.
Stephen Northcutt - Writer, speaker, security thought leader
(808) 823 1375
Tuesday, May 1, 2007
Free virtual CD-ROM drive tool from Microsoft
Like most IT professionals, I often download large software distributions from vendors that are typically packaged as ISO files. Once downloaded, I use CD burning software to extract the data files from the ISO image and then to burn the package to a CD for use. Fortunately, Microsoft has released a free virtual CD-ROM tool for Windows XP users called the "Virtual CD-ROM Control Panel v2.0.1.1". This tool allows a user to view and work with the ISO data files directly on the hard drive without the need to extract and burn the entire package to a CD.
The Microsoft Virtual CD-ROM Control Panel download is a self-extracting archive file. The file can be downloaded directly from Microsoft at: http://download.microsoft.com/download/7/b/6/7b6abd84-7841-4978-96f5-bd58df02efa2/winxpvirtualcdcontrolpanel_21.exe
When the archive is extracted, a new folder is created that contains three files: the application's front end (VCdControlTool.exe), the virtual CD driver (VCdRom.sys), and a readme file with instructions for the installation and use of the tool. Please note, however, this virtual CD-ROM tool is unsupported by Microsoft.
The Microsoft Virtual CD-ROM Control Panel download is a self-extracting archive file. The file can be downloaded directly from Microsoft at: http://download.microsoft.com/download/7/b/6/7b6abd84-7841-4978-96f5-bd58df02efa2/winxpvirtualcdcontrolpanel_21.exe
When the archive is extracted, a new folder is created that contains three files: the application's front end (VCdControlTool.exe), the virtual CD driver (VCdRom.sys), and a readme file with instructions for the installation and use of the tool. Please note, however, this virtual CD-ROM tool is unsupported by Microsoft.
Subscribe to:
Posts (Atom)