try to force delay 2 sec for crawling bots

rails5
Quentin Gibeaux 5 years ago committed by root
parent 61411b26de
commit d4c935a943

@ -3,3 +3,5 @@
# To ban all spiders from the entire site uncomment the next two lines:
# User-agent: *
# Disallow: /
User-agent: *
Crawl-delay: 3

Loading…
Cancel
Save