After launching a site
Make sure crawlers can reach it and find the sitemap.
Robots.txt and Sitemap Validator
Enter your site address and see what search engines will find. Our server fetches the two files once.
Your files stay with you. Unlike our other tools, where your files never leave your browser, this one sends the web address you type to our server so it can fetch the page for you. Nothing is stored or logged, and only the result is shown to you.
How to use it
- Type your site address, or the address of a sitemap file.
- Press the check button.
- Read the problems found in robots.txt, then in the first sitemap it lists.
Where it helps
Pages missing from Google
Rule out a robots.txt block or a broken sitemap.
After a migration
Check the new site's files point to the right URLs.
Good to know
Fetched once, not stored
Our server reads the public page or file once and discards it.
Blocked site warning
It flags when all crawlers are blocked from the whole site.
Compressed sitemaps
.gz sitemaps are valid but aren't read here.
Then check pages
Use the HTTP status checker for individual URLs.
Questions people ask
How do you check a website's robots.txt and sitemap?
Type your site address, or a sitemap's address, and press the check button. Our server fetches the two files once and reports problems in robots.txt, such as unknown rules or the whole site being blocked, then checks the first sitemap it lists for XML errors, bad URLs, duplicates and dates.
- 2 files robots.txt + sitemap. Fetched once.
- 50,000 URL limit checked. Per sitemap.
- 1 MB Read limit. Plain XML.
- No page visits Listed URLs not crawled.
What does it check in robots.txt?
That the file exists and isn't a web page in disguise, every line is a known rule, rules sit under a User-agent line, paths start with a slash, Sitemap lines are full URLs, and whether all crawlers are blocked from the whole site. Each problem comes with a short explanation.
What does it check in the sitemap?
That it's valid XML with a urlset or sitemapindex, entries have full URLs, there are at most 50,000 of them, there are no duplicates, dates are in the right format, and URLs are on the same host as the site. Those are the issues that most often stop sitemaps working.
Does it check every URL in the sitemap?
No. It reads the sitemap file and reports on its contents, but doesn't visit the listed pages, to avoid flooding your site with requests. Use the HTTP status checker for individual pages you want to test. Spot-check a few important pages. That keeps your server load light.
Why is a compressed or very large sitemap only partly checked?
To keep the tool fast and safe, it reads only the first 1 MB and plain XML. Compressed .gz sitemaps are valid for search engines but aren't read here. For very large sites, check one of the smaller child sitemaps instead. Smaller files help. Split huge sitemaps.
Why might my site be missing from Google?
Common causes include a robots.txt rule blocking the whole site, a noindex tag, a broken or missing sitemap, or a new site that hasn't been discovered yet. This validator rules out the first and third quickly; the meta tag analyzer checks for noindex. Search Console helps too.
What does 'web page in disguise' mean?
Some servers answer requests for robots.txt with an HTML page, such as a homepage or an error page, instead of a plain text file. Crawlers can't read rules from that. The validator spots this, which usually means the file doesn't really exist. Create a real file.
Is anything stored?
No. Our server fetches robots.txt and the sitemap once, checks them and discards them. Only the report is sent to you. Nothing is logged against you, and the pages listed in the sitemap aren't visited. Only the report is sent back to you, and no account is needed.
Why does it use your server?
Browsers block web pages from reading other sites' files for security reasons, so the check can't run in your browser. Our server fetches the two public files on your behalf and reports what it finds. The files are discarded straight after the check. Nothing is kept afterwards.
Can I check just a sitemap?
Yes. Enter the sitemap's full address instead of the site address, and it checks that file directly. That's useful when your sitemap isn't listed in robots.txt yet, or when you want to test one child sitemap of a large site. Child sitemaps work too. Paste its full address.
What if robots.txt blocks everything?
The validator flags it clearly, because Disallow: / for all user agents asks every crawler to skip the whole site. That's sometimes left over from a development site. Fix it straight away if the site should appear in search results. Recheck after fixing it. Then validate it again.
Why must sitemap URLs be on the same host?
Search engines only trust sitemaps listing pages on the same site, so example.com's sitemap shouldn't list pages on shop.example.com or another domain. Mixed hosts usually mean a copied or outdated sitemap that needs regenerating. Regenerate the sitemap from your live site's own pages. Regenerate it from the live site.
Which date format is correct?
W3C date format, such as 2026-10-24, optionally with a time. Other formats, such as 24/10/2026, are errors. If dates aren't accurate, it's better to leave lastmod out than to include wrong or constantly changing dates. Dates should reflect real changes to the page. Leave dates out if unsure.
Can I check someone else's site?
Yes, any public site. robots.txt and sitemaps are public files meant for crawlers. It's a quick way to check a client or prospect's setup before a meeting, without needing their login. Both files are meant to be public, so checking them is normal practice. Both files are public.
Does it work on a phone?
Yes. Enter the address on your phone and read the report. It's useful right after publishing changes to robots.txt or a sitemap, when you want to confirm they're live and correct. No app or account is needed, so you can check from anywhere. Results load quickly.
Is it free?
Yes. The robots.txt and sitemap validator is free, with no sign-up. It's part of the free SEO tools from Marketiism, alongside generators for both files and an HTTP status checker. No account is needed, and nothing is stored about the sites you check. No account is needed.
Want this inside your own account?
Marketiism keeps your invoice numbers, GST reports, UPI payment links and WhatsApp reminders in one place. Leave your details and we'll call to set it up for your business, or start the trial yourself.
More SEO tools
Robots.txt Generator
Build a robots.txt with blocked paths, a sitemap line and optional AI crawler blocks.
XML Sitemap Generator
Turn a list of page URLs into a valid sitemap.xml with optional dates, frequency and priority.
HTTP Status Checker
Check the status code of a URL and follow its redirect chain hop by hop.
Meta Tag Generator
Write the title, description, robots, canonical and viewport tags for a page, ready to paste.
Open Graph Generator
Create Open Graph and X (Twitter) card tags and preview how your link looks when shared.
SERP Snippet Preview
See how your title and description may look in search results, with character and pixel estimates.