The robots.txt time makes it at times easier to scrape a target by the info that a website can reveal in it (e.g. allow a specific bot to scrape all). Their sitemaps are another gem.