kweri

Field Guide

Robots.txt and Sitemaps

robots.txt and sitemap.xml are the two files search engines look for first: one tells crawlers which parts of your site they're allowed to access, and the other hands them a map of every page worth indexing.

Why it matters for your site

A sitemap is how you make sure search engines find everything — especially newer pages, deeper pages, and pages that aren't linked prominently from your navigation. Without one, discovery is slower and some pages may sit unindexed for longer than they should.

robots.txt is powerful in the other direction: it can tell crawlers to stay out of certain areas. Used well, it keeps search engines away from admin pages and duplicate sections. Used carelessly, a single overly broad rule can accidentally hide your entire site from search — a surprisingly common and costly mistake.

The same robots.txt now also governs AI crawlers. The engines behind ChatGPT, Perplexity, and Google's AI answers use named crawlers, and a blanket block can quietly exclude your content from AI-generated answers — increasingly where people start their search.

How to check it yourself

  1. Visit yourdomain.com/robots.txt directly in your browser. Read the 'Disallow' lines — anything listed there is blocked from crawling. A line reading 'Disallow: /' under 'User-agent: *' blocks your whole site; make sure that's intentional.
  2. Visit yourdomain.com/sitemap.xml. You should see a list of your pages' URLs. If it's missing or only lists a handful of your pages, search engines may be working from an incomplete map.
  3. Check that your robots.txt points to your sitemap — a line like 'Sitemap: https://yourdomain.com/sitemap.xml' helps crawlers find it.
  4. Google Search Console's Sitemaps and Pages reports show what Google has actually discovered and indexed — the ground truth for whether these files are doing their job.
  5. The Kweri audit flags a missing robots.txt, a missing sitemap, navigation pages absent from your sitemap, and AI crawlers that are blocked in robots.txt.

Technical detail

robots.txt is a plain-text file at the root of your domain. It's organised into groups, each starting with a 'User-agent' line (which crawler the rules apply to) followed by 'Disallow' and 'Allow' rules. 'User-agent: *' applies to all crawlers. It can also carry one or more 'Sitemap:' directives.

robots.txt is advisory: well-behaved crawlers (Google, Bing, the major AI engines) honour it, but it is not a security control — it can't stop a determined or malicious bot, and it doesn't make a page private. Use proper authentication for anything that must stay hidden.

sitemap.xml is an XML file listing the URLs you want indexed, optionally with a 'lastmod' date for each so crawlers know what's changed. Large sites can split it into multiple sitemaps referenced by a sitemap index file.

AI crawlers identify themselves with their own user-agent names (for example GPTBot, ClaudeBot, PerplexityBot, Google-Extended). Blocking these — directly or via a wildcard 'Disallow: /' — removes your content from those engines' answers. Allowing or blocking them is a deliberate choice worth making consciously.

For the robots.txt specification and guidance, see Google — Introduction to robots.txt →

For the sitemap.xml specification and formats, see Google — Sitemaps overview →

For the full robots.txt syntax and crawler behaviour, see Google — robots.txt specification →

Related

Structured Data

Structured data is code you add to a page that tells search engines and AI answer engines exactly what your content is about — making it significantly more likely to be cited, featured in rich results, or extracted as a direct answer.

See Robots.txt and Sitemaps on your own site

Run a free Kweri audit — a plain-English review of your site’s speed, accessibility, SEO and design, ranked by what to fix first. No login, no jargon.

Run a free audit →