New websites
Point crawlers to your sitemap from day one.
Robots.txt Generator
Tell crawlers where they may go. The file is built in your browser and you download it.
Your files stay with you. Your files never leave your browser. This tool runs on your device: nothing is uploaded to our servers, so there is nothing for us to store or see.
How to use it
- Choose whether to allow crawling and list any paths to block.
- Add your sitemap URL and tick any bots you want to block.
- Download robots.txt and upload it to the top level of your site.
Where it helps
Admin and test areas
Ask crawlers to stay out of login, cart and staging paths.
AI crawlers
Decide whether AI companies' bots may read your site.
Good to know
Stays in your browser
What you type is processed on your device and never sent to our servers.
Not a privacy tool
Use noindex or a password to keep pages out of results.
Root only
A robots.txt in a sub-folder is ignored.
Validate it
Check the live file with the robots and sitemap validator.
Questions people ask
How do you create a robots.txt file?
Choose whether to allow crawling, list any paths to block, add your sitemap URL and tick any bots you want to block, including AI crawlers. Download robots.txt and upload it to the top level of your site. Remember it's a request to crawlers, not a lock, and it doesn't reliably hide pages from Google.
- Root Where it goes. example.com/robots.txt.
- Sitemap Line included. Helps discovery.
- AI bots Optional blocks. A request, not a lock.
- 0 Data sent. Built in your browser.
What is robots.txt?
It's a small text file at the root of your website that tells search engine and other crawlers which parts of the site they may fetch. Well-behaved crawlers read it before crawling. It can also point them to your sitemap, which helps new pages get found.
Does robots.txt hide a page from Google?
Not reliably. It asks well-behaved crawlers not to fetch a page, but a blocked page can still be listed if other sites link to it. To keep a page out of search results, use a noindex tag or password protection instead of robots.txt. Never rely on it.
Where must the file go?
At the root of the site, for example https://example.com/robots.txt. A file in a sub-folder is ignored. Each subdomain needs its own file, so shop.example.com and example.com are treated separately by crawlers. The address must be exactly /robots.txt, all lower case, served as plain text. Open it to confirm.
Will blocking AI crawlers work?
It's a request, not a lock. Bots from major AI companies say they follow robots.txt, but not every crawler does. Think carefully before blocking them: if AI tools can't read your public pages, they may describe your business less accurately in their answers. Decide case by case.
Why does Google ignore my crawl delay?
Google doesn't support the Crawl-delay line. Bing and some other crawlers do. If Google is crawling your server too heavily, the usual fix is improving server capacity or caching, because Googlebot adjusts its rate automatically when a site responds slowly. Search Console offers crawl settings too.
What should a small business block in robots.txt?
Usually very little. Common choices are admin areas, login pages, cart and checkout steps, internal search results and test or staging folders. Never block your main pages, CSS or JavaScript files, because Google needs them to see your pages properly. Keep the file short. Fewer rules mean fewer mistakes.
Should I add my sitemap to robots.txt?
Yes. A Sitemap: line with the full address of your sitemap helps search engines find all your pages, especially new ones. Also submit the sitemap in Google Search Console and Bing Webmaster Tools for the best coverage. Use the full https:// address. Use the full https:// address.
Can a mistake in robots.txt hurt my site?
Yes. A single Disallow: / under User-agent: * asks every crawler to skip your whole site, which can remove you from search results over time. After uploading, check the live file with the robots and sitemap validator to catch mistakes like this. Check after every edit. Validate after every change.
Is anything sent to your server?
No. The file is built in your browser and you download it. Your paths and settings stay on your device. Uploading the file to your website is a separate step you do through your hosting panel or website builder. No account is needed. No account is needed.
How do I upload robots.txt?
Use your hosting control panel's file manager or FTP to place it in the site's root folder, where your homepage file lives. Many website builders, including WordPress SEO plugins, let you edit robots.txt from their settings instead. Then open the address to confirm it's live.
What does User-agent mean?
User-agent names the crawler a group of rules applies to. User-agent: * means all crawlers, while User-agent: Googlebot applies only to Google's crawler. Rules must sit under a User-agent line, or crawlers ignore them. Groups apply to the crawlers they name, and a crawler follows its most specific group.
Should I block AI crawlers?
It depends on your goals. Allowing them helps AI tools describe and recommend your business accurately, which is valuable for local businesses. Blocking them may make sense for original content you don't want used. Many small businesses choose to allow them. You can change it later.
Do I need robots.txt at all?
It isn't required; without one, crawlers assume they may fetch everything. But a simple file with your sitemap line is good practice and avoids error messages in some tools. The generator makes a correct basic file in seconds. It takes a minute to set up.
Does it work on a phone?
Yes. Fill in the options and download the file on your phone. Uploading it to your website is usually easier from a computer, through your hosting panel or website builder's settings. Nothing is uploaded, and you don't need an app or an account to use it on your phone.
Is it free?
Yes. The robots.txt generator is free, with no sign-up and no limit. It's part of a set of free SEO tools from Marketiism, including a sitemap generator and a validator that checks your live robots.txt and sitemap. Nothing is added to the output, and business use is fine.
Want this inside your own account?
Marketiism keeps your invoice numbers, GST reports, UPI payment links and WhatsApp reminders in one place. Leave your details and we'll call to set it up for your business, or start the trial yourself.
More SEO tools
XML Sitemap Generator
Turn a list of page URLs into a valid sitemap.xml with optional dates, frequency and priority.
.htaccess Redirect Generator
Create Apache redirect rules for moved pages, plus force HTTPS and www or non-www.
Meta Tag Generator
Write the title, description, robots, canonical and viewport tags for a page, ready to paste.
Open Graph Generator
Create Open Graph and X (Twitter) card tags and preview how your link looks when shared.
SERP Snippet Preview
See how your title and description may look in search results, with character and pixel estimates.
Keyword Density Checker
Count the most repeated words and 2 or 3 word phrases in your text, with percentages.