Free tools that run in your browser. Use any tool without an account.

Favourites Cart
Sign in
Join free

Robots.txt Tester

  • Free
  • Nothing stored
  • No account needed

FreeToolr's Robots.txt Tester checks whether a chosen crawler may access a tested path under pasted rules. For example, Disallow: /private blocks /private/report unless a longer Allow rule matches.

Result
Allowed
Rule that decided
Line 4: Allow: /admin/help/
Group used
User-agent: *
Path tested
/admin/help/faq
Sitemaps listed
https://example.com/sitemap.xml
Runs in your browser. Your files and values are not sent to us. Report a problem
Advertisement

About Robots.txt Tester

FreeToolr's Robots.txt Tester helps SEO specialists, developers and site owners check crawl rules before publishing changes. Paste your robots.txt content, enter a URL or path, and add a crawler token such as Googlebot or Bingbot. The tool reports the result, deciding rule, user-agent group, tested path, listed sitemaps and notes. Results update instantly as you type, so there is no calculate button. The tester runs in your browser, and nothing is uploaded. It follows Google's matching behavior for user-agent groups, rule length, Allow and Disallow ties, wildcards and end markers. You can copy the result or share a link that reopens with your numbers. This gives you the exact rule and group behind the result, instead of showing only a general allowed or blocked message.

How to use it

  1. 1Paste the robots.txt content into the rules field.
  2. 2Enter the URL or path you want to test.
  3. 3Enter the crawler user-agent token, such as Googlebot.
  4. 4Review the instant result, deciding rule, group, path, sitemaps and notes.
  5. 5Copy the result or share a link that reopens the test.
Advertisement

Worked example

Input: User-agent: * Disallow: /private Allow: /private/report URL or path: https://example.com/private/report Crawler: Googlebot Exact result: Allowed. Rule that decided: Allow: /private/report. Group used: *. Path tested: /private/report. Sitemaps listed: None. The Allow path is longer than /private, so it wins.

How it works

Follows Google's robots.txt rules: the crawler obeys only the most specific User-agent group that names it (groups with the same name are merged; Googlebot-Image falls back to Googlebot, then to *)Within the group, the rule with the longest matching path wins; if an Allow and a Disallow are the same length, Allow wins* matches any characters and $ marks the end of the URLPaths are case sensitiveBlocking crawling does not remove a page from Google; use noindex for that

Source: Google Search Central: Introduction to robots.txtRFC 9309: Robots Exclusion Protocol. Last checked 11 November 2026.

What it is good for

  • See the exact rule that decided the result.
  • Compare crawler-specific groups with instant updates.
  • Test draft rules privately in your browser.
  • The tool tests the robots.txt content you provide. It does not fetch a live file.
  • Robots.txt controls crawling, not guaranteed search removal or private access.
Advertisement

Questions people ask

What does this robots.txt tester check?

It checks if the selected crawler may access the URL or path under the robots.txt content you enter. It also shows the deciding rule, group used, path tested, listed sitemaps and notes.

What should I enter for Crawler?

Enter the crawler's product token, such as Googlebot, Bingbot or GPTBot. Do not paste a full browser user-agent string unless its product token is the value you want to test.

What does Disallow: / mean?

For the matching user-agent group, Disallow: / blocks every path beginning at the site root. A longer matching Allow rule can win for a specific path.

Does robots.txt remove a page from Google?

No. Blocking crawling does not remove a page from Google. Use noindex on a crawlable page when you need search engines to avoid indexing it.

Which rule wins when Allow and Disallow both match?

The rule with the longest matching path wins. If Allow and Disallow match the same path length, Allow wins.

What do * and $ mean?

In this tester, * matches any characters and $ marks the end of the URL. Paths are case sensitive, so /Private and /private are different paths.

Can robots.txt protect private files?

No. Robots.txt gives crawl guidance, not access control. Use authentication or server-side access controls for private files.

Help keep every tool free

FreeToolr stays free because people help it grow. Share a tool you like, tell us what to build next, or support us on Ko-fi.

Advertisement
Advertisement