Erreur32 / tips_httpd.conf_spiders.md
0 likes
0 forks
1 files
Last active 1 month ago
| 1 | # To relieve servers |
| 2 | |
| 3 | ##Imagine a robots.txt file like this (Google understands this format): |
| 4 | #User-agent: * |
| 5 | #Disallow: /detailed |
| 6 | #Disallow: /?action=detailed |
| 7 | #Disallow: /*/detailed |
| 8 | #Crawl-delay: 20 |
| 9 | ## |
Newer
Older