agenda-libre/agenda-libre-ruby
agenda-libre/agenda-libre-ruby
2
1
Fork 0
Code Issues Pull Requests Releases Activity

try to force delay 2 sec for crawling bots

Browse Source
This commit is contained in:
Quentin Gibeaux 2017-10-03 10:02:28 +02:00 committed by root
parent 61411b26de
commit d4c935a943
1 changed files with 2 additions and 0 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

2
public/robots.txt
View File

@ -3,3 +3,5 @@
# To ban all spiders from the entire site uncomment the next two lines: # To ban all spiders from the entire site uncomment the next two lines:
# User-agent: * # User-agent: *
# Disallow: / # Disallow: /
User-agent: *
Crawl-delay: 3