Side by Side Diff: static/robots.txt
Issue 29673562:
Disallow / (all) for BUbiNG and bingbot (Closed)
Patch Set:
Created Jan. 18, 2018, 9:39 a.m.
| Left: | ||
| Right: |
Use n/p to move between diff chunks;
N/P to move between comments.
« no previous file with comments
|
« no previous file
|
no next file »
|
no next file with comments »
Toggle Intra-line Diffs ('i') | Expand Comments ('e') | Collapse Comments ('c') | Hide Comments ('s')
Toggle Intra-line Diffs ('i') | Expand Comments ('e') | Collapse Comments ('c') | Hide Comments ('s')
| OLD | NEW |
|---|---|
| 1 # Directions for web crawlers. | 1 # Directions for web crawlers. |
| 2 # See http://www.robotstxt.org/wc/norobots.html. | 2 # See http://www.robotstxt.org/wc/norobots.html. |
| 3 | 3 |
| 4 User-agent: HTTrack | 4 User-agent: HTTrack |
| 5 User-agent: puf | 5 User-agent: puf |
| 6 User-agent: MSIECrawler | 6 User-agent: MSIECrawler |
| 7 User-agent: Nutch | 7 User-agent: Nutch |
| 8 User-agent: BUbiNG | |
| 9 User-agent: bingbot | |
| 8 Disallow: / | 10 Disallow: / |
| 9 | 11 |
| 10 User-agent: * | 12 User-agent: * |
| 11 Disallow: /*/diff/ | 13 Disallow: /*/diff/ |
| 12 Disallow: /*/diff2/ | 14 Disallow: /*/diff2/ |
| 13 Disallow: /*/patch/ | 15 Disallow: /*/patch/ |
| 14 Disallow: /*/publish | 16 Disallow: /*/publish |
| 15 Disallow: /all | 17 Disallow: /all |
| 16 Disallow: /download/ | 18 Disallow: /download/ |
| 17 Disallow: /rss/ | 19 Disallow: /rss/ |
| 18 Disallow: /search | 20 Disallow: /search |
| 19 Disallow: /tarball/ | 21 Disallow: /tarball/ |
| 20 Disallow: /user/ | 22 Disallow: /user/ |
| OLD | NEW |
« no previous file with comments
|
« no previous file
|
no next file »
|
no next file with comments »
