How Do You Write robots.txt, and Should You Block AI Bots Like GPTBot?

robots.txt is a plain-text file at the root of a site that tells bots which paths they may crawl, using User-agent, Disallow, Allow and Sitemap lines. For AI bots, separate training crawlers (GPTBot, ClaudeBot, CCBot, Google-Extended) from search and answer bots (OAI-SearchBot, PerplexityBot); blocking the search bots reduces your visibility in AI answers.

How do you write a robots.txt file?

A robots.txt file is published as UTF-8 plain text at the domain root (example.com/robots.txt) and is made of groups. Each group starts with one or more User-agent lines, followed by the Disallow and Allow rules for those bots. The file only applies to the exact protocol, host and subdomain it is served on.

• —"User-agent: *" targets all bots; "User-agent: Googlebot" targets only Google's crawler.
• —"Disallow: /panel/" blocks crawling of every URL starting with that path; an empty "Disallow:" allows everything.
• —"Allow: /panel/login/" opens an exception inside a blocked folder.
• —* matches any sequence of characters and $ marks the end of the URL: "Disallow: /*?sort=" or "Disallow: /*.pdf
  quot;.
• —When rules conflict, Google applies the longest (most specific) match; on a tie, the less restrictive Allow wins.
• —Google ignores anything beyond 500 KiB and does not support the Crawl-delay directive.

An often-missed rule: when a bot finds a group addressed to it by name, it follows only that group and ignores the "User-agent: *" group entirely. If you write a separate group for GPTBot, you must repeat any general restrictions inside it.

What is the difference between Disallow and noindex?

Disallow blocks crawling; noindex blocks indexing. A page disallowed in robots.txt can still be indexed by URL alone, without its content, if other sites link to it. To remove a page from search results, use a noindex meta tag or X-Robots-Tag header, and allow the page to be crawled so Google can actually see that tag.

Google has not supported a "Noindex:" line inside robots.txt since 2019. The practical split: areas that waste crawl budget, such as endless filter combinations, internal search results and admin paths, are handled with Disallow; pages that should be reachable but not in search, such as thank-you pages or legal text, get noindex. robots.txt is not a security tool; it is public, and listing secret paths in it exposes them.

How do you add a Sitemap line to robots.txt?

The Sitemap line can go anywhere in robots.txt, independent of groups, as an absolute URL in the form "Sitemap: https://example.com/sitemap.xml". If you have several sitemaps, each goes on its own line. The Sitemap line lets search engines you never submitted a sitemap to still discover it.

On 404 Ajans's site, robots.txt is a short file that allows all bots, and its sitemap line points to a sitemap.xml regenerated from the page list on every build. Opening the file in a browser after launch to confirm it is served as text/plain with the right encoding is a simple but important step.

Which AI bots exist and what do they do?

AI bots fall into two main groups: training crawlers that collect data for building models, and search or retrieval bots that fetch sources while answering a user's question. Each is controlled by its own user-agent name in robots.txt, and blocking a company's training crawler usually does not block that same company's search bot.

• —GPTBot (OpenAI): crawls for model training.
• —OAI-SearchBot (OpenAI): crawls so sites can appear as sources in ChatGPT search; not used for training.
• —ChatGPT-User (OpenAI): on-demand visits when a user asks ChatGPT to open a link.
• —ClaudeBot (Anthropic): crawls for model training. Anthropic also uses Claude-SearchBot for search and Claude-User for user-initiated fetches.
• —Google-Extended (Google): not a separate crawler but a robots.txt control token. It governs whether content is used to train Gemini models and for grounding in Gemini apps; it does not affect inclusion or ranking in Google Search.
• —PerplexityBot (Perplexity): indexes sites to show them in Perplexity's search results; Perplexity-User covers user-initiated visits.
• —CCBot (Common Crawl): builds an open web archive that many AI models use as training data.
• —Applebot-Extended (Apple): an extra token controlling whether content trains Apple's AI models.

One important detail: Google's AI Overviews and AI Mode are part of Google Search and rely on regular Googlebot crawling. Blocking Google-Extended does not remove your site from AI Overviews; that would require blocking Googlebot, which removes you from Search altogether.

Should you block AI bots?

Whether to block AI bots depends on what your content is and what appearing in AI answers is worth to you. For a business selling services, being cited in ChatGPT, Claude and Perplexity answers is a new acquisition channel, so blocking the search bots usually means reducing your own visibility.

• —Service and local business sites: allowing all bots usually makes sense; the content exists to attract customers and AEO visibility pays off directly.
• —Middle ground: block training crawlers (GPTBot, ClaudeBot, CCBot, Google-Extended, Applebot-Extended) and allow search bots (OAI-SearchBot, Claude-SearchBot, PerplexityBot).
• —Publishers of paid content, original research or licensed data: blocking training crawlers can be reasonable to protect the content's value.
• —In every case: document the decision and review robots.txt at least once a year; new bots and names keep appearing.

Know the limits too: robots.txt is voluntary and malicious bots ignore it, and it has no retroactive effect on models already trained. If you need hard blocking, you need server or CDN rules with user-agent and IP verification. Also check whether a CDN such as Cloudflare has a one-click AI-bot blocking setting switched on without you realising.

What do example robots.txt configurations look like?

Example robots.txt configurations turn the three strategies above into concrete lines. The examples below are described in plain text; each quoted item is a separate line in the file. After any change, use the robots.txt report in Search Console to confirm Google reads the file without errors.

• —Allow everything: "User-agent: *", an empty "Disallow:" below it, and "Sitemap: https://example.com/sitemap.xml" at the end.
• —Closing an admin area: add "Disallow: /admin/" and "Disallow: /*?sort=" to the "User-agent: *" group.
• —Blocking only training crawlers: list "User-agent: GPTBot", "User-agent: ClaudeBot", "User-agent: CCBot" and "User-agent: Google-Extended" one under another and close them with a single "Disallow: /"; the general group stays open.
• —Reminder: those bots will follow only their own group, so general rules like /admin/ do not apply to them; here that is fine because the whole site is already closed to them.

Do not confuse robots.txt with llms.txt: robots.txt grants or restricts access, while llms.txt is a proposed, not yet official, file that summarises a site's key pages for AI systems.