What is robots.txt and Where Does It Live?
robots.txt is a plain text file that must be placed at the root of your domain: https://yoursite.com/robots.txt
It uses the Robots Exclusion Standard — a simple protocol where you specify rules for different "user agents" (crawlers). A user agent is a string that identifies the crawler, such as "Googlebot" for Google's crawler, "Bingbot" for Bing, or "GPTBot" for OpenAI's crawler.
The file is publicly accessible — any bot (or human) can read it. This means robots.txt is not a security mechanism. It is a polite request that well-behaved crawlers follow voluntarily. Malicious bots ignore it entirely.
When Googlebot visits your site for the first time, it fetches robots.txt before crawling any other page. This is how it knows which areas of your site to avoid.
robots.txt Syntax and Directives
A robots.txt file is made up of groups, each starting with a User-agent directive:
User-agent: * (applies to all crawlers) Allow: / (allow crawling everything) Disallow: /admin/ (block the /admin/ section) Crawl-delay: 1 (wait 1 second between requests) Sitemap: https://yoursite.com/sitemap.xml
Key directives:
User-agent — specifies which crawler the following rules apply to. * means all crawlers.
Disallow — prevents the crawler from accessing the specified path. An empty Disallow line means "allow everything."
Allow — explicitly permits a path, overriding a broader Disallow. Useful for allowing /public/ within a blocked /private/ section.
Crawl-delay — asks the crawler to wait N seconds between requests. Note: Google ignores this directive. Use Google Search Console to adjust Googlebot's crawl rate.
Sitemap — tells crawlers where your XML sitemap is located.
Blocking AI Training Crawlers in 2026
One of the biggest changes in web crawling in recent years is the rise of AI company crawlers that scrape websites to train large language models. These are separate from search engine crawlers and have become a significant concern for many website owners.
Major AI crawlers and their User-agent strings: • GPTBot — OpenAI (ChatGPT, GPT-4) • ChatGPT-User — OpenAI browsing plugin • CCBot — Common Crawl (used by many AI companies) • anthropic-ai — Anthropic (Claude) • Claude-Web — Anthropic browsing • PerplexityBot — Perplexity AI • Bytespider — ByteDance (TikTok parent)
To block all AI training crawlers:
User-agent: GPTBot Disallow: /
User-agent: CCBot Disallow: /
User-agent: anthropic-ai Disallow: /
This prevents these crawlers from accessing your content for AI training while still allowing Googlebot and Bingbot to index your site normally.
Common robots.txt Mistakes to Avoid
These mistakes can seriously harm your site's SEO:
Accidentally blocking everything — Disallow: / blocks ALL crawlers from ALL pages. This is sometimes used intentionally during development but catastrophic if left on a production site.
Blocking CSS and JavaScript — some older guides recommend blocking /css/ and /js/ directories. Never do this. Google needs to render your JavaScript to understand your content. Blocking these files means Google sees a broken page.
Relying on robots.txt for security — as mentioned, robots.txt is a public file that malicious bots ignore. Use server-side authentication and access controls for genuinely private content.
Forgetting the trailing slash — Disallow: /admin blocks both /admin and /administrator. Disallow: /admin/ only blocks /admin/ and its subdirectories.
Multiple Sitemap directives — you can have multiple Sitemap lines if you have multiple sitemaps. This is valid and encouraged for large sites.
Case sensitivity — paths in robots.txt are case-sensitive on case-sensitive servers (most Linux servers). /Admin/ and /admin/ are different paths.