User-agent: * Crawl-delay: 60 Request-rate: 1/60 # maximum rate is one page every 5 seconds Visit-time: 0300-0700 Disallow: /cgi-bin/