How To Use A robots.txt File To Limit Web Crawlers Activities

Instructions can be found on Google's site at http://www.google.com/bot.html.

In May 2007 I saw Googlebot scanning my website from IP address 66.249.72.198. I looked up this IP address and found the following:

OrgName:    Google Inc.
OrgID:      GOGL
Address:    1600 Amphitheatre Parkway
City:       Mountain View
StateProv:  CA
PostalCode: 94043
Country:    US

NetRange:   66.249.64.0 - 66.249.95.255
CIDR:       66.249.64.0/19
NetName:    GOOGLE
NetHandle:  NET-66-249-64-0-1