All gists matching topic Robots

1	# To relieve servers
2
3	##Imagine a robots.txt file like this (Google understands this format):
4	#User-agent: *
5	#Disallow: /detailed
6	#Disallow: /?action=detailed
7	#Disallow: /*/detailed
8	#Crawl-delay: 20
9	##