Robots.txt tester, validator and generator

How Google decides which rule wins

A robots.txt is made of groups: each starts with User-agent and is followed by Allow and Disallow rules. Google applies them like this:

• The most specific group. Googlebot-Image uses the Googlebot-Image group if it exists; if not, the Googlebot one; if not, *. Only one: groups don't mix.
• The longest rule. With Disallow: / and Allow: /blog, the path /blog/my-post is allowed, because /blog is longer.
• On a tie, allow wins. Allow: /page and Disallow: /*.ph are the same length: /page.php is allowed.
• Wildcards. * means “anything” and $ “ends here”: Disallow: /*.pdf$ blocks PDFs, but not /doc.pdf?v=2.

The mistakes we see most

• A forgotten Disallow: / when a staging site goes live: Google can't see anything.
• Blocking CSS or JavaScript (/wp-includes/, *.js): Google sees a broken page and may rank it lower.
• Using robots.txt to remove pages from Google. It doesn't work: use noindex.
• A robots.txt that returns 5xx. Google treats it as “crawl nothing”.

Is everything allowed and your site still isn't on Google? Read why your website isn't showing up on Google, and check your redirects with the redirect checker.