StickSEO
StickSEO › Free SEO tools › Robots.txt checker

Robots.txt checker

See what your robots.txt really tells search engines and AI crawlers, and get a corrected file if something is wrong.

Free, no sign up. The full report comes with it.

How it works

One wrong line in robots.txt can hide a whole website from search. The checker reads your file the way a crawler does, group by group, and tells you whether everything is blocked, which AI crawlers are named and blocked, and whether the file points to your sitemap.

When there is a fault, the report writes a clean robots.txt for your site that you can copy.

Questions

What does "Disallow: /" mean?

It tells the named crawler to stay out of the whole site. Under "User-agent: *" it applies to every crawler without its own rules.

Should robots.txt list the sitemap?

Yes. A "Sitemap:" line helps every search engine find all your pages.

Can robots.txt remove a page from Google?

No. It stops crawling, not indexing. Use a noindex tag to keep a page out of results.

More free SEO tools