Browse Source

try to force delay 2 sec for crawling bots

rails5
Quentin Gibeaux 4 years ago
committed by root
parent
commit
d4c935a943
  1. 2
      public/robots.txt

2
public/robots.txt

@ -3,3 +3,5 @@
# To ban all spiders from the entire site uncomment the next two lines:
# User-agent: *
# Disallow: /
User-agent: *
Crawl-delay: 3

Loading…
Cancel
Save