Robots.txt Generator
Generate robots.txt files with presets and custom rules
About Robots.txt Generator
robots.txt is a text file that instructs search engine crawlers and web robots which pages to crawl and which to ignore. It controls Google, Bing, AI training bots (GPTBot, CCBot, Claude) and hundreds of other crawlers. This free Robots.txt Generator includes presets for common use cases — including a "Block AI Bots" preset that prevents AI companies from scraping your content for training data — with sitemap URL inclusion and optional crawl delay.
How to Use Robots.txt Generator
- 1Choose a preset: Allow All (standard), Block All, Block AI Bots, E-Commerce or Blog/CMS.
- 2Enter your sitemap URL and optionally set a crawl delay.
- 3Toggle "Include Sitemap" to add or remove the Sitemap directive.
- 4Click "Generate robots.txt" — copy the output and upload it to your website's root folder.
Common Use Cases
Place your robots.txt at the root of your domain: https://yoursite.com/robots.txt. Google checks this file before crawling any page — an incorrectly configured robots.txt can accidentally block your entire site from Google.
⭐ Popular Free Tools
Frequently Asked Questions
What is robots.txt and why does my website need one?
robots.txt is a plain text file at the root of your domain that tells web crawlers which pages they are allowed to access. Without it, all crawlers can access all pages by default. You need robots.txt to protect admin areas, prevent duplicate content from query parameters, control crawl budget, and manage AI bot access to your content.
Does Disallow: / in robots.txt block Google from indexing my site?
Yes. "Disallow: /" tells ALL crawlers they cannot access ANY page on your site. This is a catastrophic SEO mistake that will remove your entire site from Google. Only use "Disallow: /" under a specific User-agent you want to fully block, and make sure Googlebot is not affected.
Can I block AI crawlers like GPTBot from scraping my content?
Yes, but only for bots that respect robots.txt. OpenAI's GPTBot, Anthropic's ClauseWeb, Common Crawl's CCBot and most major AI company crawlers do honor robots.txt. Add "User-agent: GPTBot\nDisallow: /" to prevent OpenAI from scraping your content for AI training.
Does robots.txt protect private pages from being accessed?
No. robots.txt is a public file that politely asks crawlers not to access certain pages. It does not enforce access control — the pages are still publicly accessible to anyone who knows the URL. For truly private pages, use server-side authentication. Never rely on robots.txt for security.