Sitemap Checker
Validate sitemap XML, URL counts and structural issues.
SEO
Robots.txt Tester
Fetch and inspect a public robots.txt file, directives and sitemap references safely.
Your input is processed only as required by this tool.
Robots.txt Tester
robots.txt is a crawler-control file, not an access-control system. This tester retrieves the public file and surfaces its directives and sitemap references so you can diagnose accidental crawler blocks or missing discovery hints.
How to use it
- Enter the public URL or domain you want to inspect.
- Run the check and let Ciatto Tools collect only the signals required for this diagnostic.
- Review the result, warnings and related checks before making a security or configuration decision.
How to read the result
A Disallow rule tells compliant crawlers not to crawl matching paths, but it does not make those URLs private. Sensitive content needs real authentication and authorization.
Limitations
Different crawlers interpret rules according to their own documented behavior. This utility inspects the file and common directives; it is not a substitute for testing a specific crawler against a specific URL.
Frequently asked questions
Can robots.txt hide a page from everyone?
No. It is a crawler directive, not a security boundary. Anyone who knows the URL may still request it unless access is protected separately.
Should robots.txt contain my sitemap?
It can. A Sitemap directive is a useful discovery hint, although sitemaps can also be submitted through search-engine tools.