Robots.txt Explained: What It Does and How to Set It Up Correctly

What robots.txt actually controls

A robots.txt file sits at your site's root and tells crawlers which parts of your site they're allowed to request. It's the first thing most well-behaved bots check before crawling anything else — Googlebot, Bingbot, and increasingly AI crawlers like GPTBot and ClaudeBot all read it before touching a single page.

What it doesn't do is remove pages from search results or hide content from people who already have the link. It's a crawling instruction, not a privacy or security control — a disallowed page can still get indexed if enough other sites link to it, just without Google ever reading its content.

The two lines that matter most

Every rule in a robots.txt file comes down to two directives:

• Disallow tells a crawler not to request a given path.
• Allow carves out an exception within a broader disallowed section.

Both sit under a User-agent line, which specifies which bot the rules apply to — * for all bots, or a specific name like Googlebot for one crawler only. Rules are matched per user-agent group, so a general * group and a GPTBot-specific group are read independently.

Common mistakes that quietly break a site

• Disallowing / under User-agent: * by accident, usually left over from a staging environment, which blocks the entire site from every crawler.
• Blocking CSS and JS files, which used to be common practice but now actively hurts rankings — Google needs to render pages to evaluate them properly, and blocked resources can make a page look broken during rendering.
• Assuming Disallow guarantees de-indexing. If external links point to a disallowed page, Google can still index the URL without crawling its content, which often looks worse in search results than not blocking it at all.
• Forgetting the sitemap line. Including a Sitemap: entry costs nothing and gives crawlers a direct path to your full list of pages.
• One group blocking everything, including search engines, when the actual intent was only to block AI training crawlers. Search engines and AI bots need separate User-agent groups — a single blanket rule can't tell them apart.

Should you block AI crawlers?

This depends on what you're optimizing for. Blocking bots like GPTBot, Google-Extended, or CCBot prevents your content from being used to train AI models, but it doesn't affect your Google search rankings, since Google-Extended is a separate agent from Googlebot. Sites that rely on being cited or referenced by AI tools may prefer to allow these crawlers; sites focused purely on protecting original content may prefer to block them. There's no universally correct answer — it's a tradeoff between visibility in a newer channel and control over how your content gets reused.

How to set one up correctly

1. Start with a working baseline — a simple Allow: / under User-agent: * if you don't need to block anything yet.
2. Add specific Disallow rules only for paths you genuinely don't want crawled — admin areas, internal search results, checkout flows.
3. Never disallow CSS, JS, or image folders needed to render your pages properly.
4. Add separate groups for AI crawlers if you want different rules for them than for search engines.
5. Include your sitemap URL at the bottom of the file.
6. Test it before publishing — a syntax error or a stray / can silently block far more than intended.

Wrapping up

Robots.txt is a small file with an outsized ability to help or hurt your site, and the syntax is easy to get subtly wrong by hand. Build yours from templates instead of writing it from scratch, so the structure is right before it ever reaches your site's root.