Robots.txt Explained: Complete Guide for SEO (2026)

Robots.txt is a plain text file at the root of your domain that tells search engine crawlers which URLs they may request. It controls crawling, not indexing. A URL blocked in robots.txt can still appear in Google if other sites link to it, so use a noindex tag when you need a page removed from search results.

What is a robots.txt file?

A robots.txt file sits at the root of a host and tells compliant crawlers which URLs they may request. It controls crawling, not indexing. Google’s robots.txt guidance explains that a blocked URL may still appear in Search if Google finds it through other pages. If you want to exclude a page from Google, keep it crawlable so Google can read a noindex directive.

When I review a WordPress site’s crawl setup, I check robots.txt alongside Search Console’s indexing reports. A rule can affect how Google crawls pages, but the file alone does not tell you whether those pages are indexed. This check belongs alongside the other technical SEO checks.

Quick facts

Why does robots.txt matter for SEO?

Robots.txt matters because it decides where crawlers spend their limited time on your site and which resources they are allowed to fetch. Used well, it keeps crawlers away from infinite filter URLs, internal search results, and admin paths. Used badly, it can hide your CSS and JavaScript, break rendering, or accidentally block your entire site.

What it does well

• Protects crawl efficiency. Blocking faceted navigation and internal search URLs stops crawlers from grinding through millions of near-duplicate combinations.
• Reduces server load. Fewer wasted requests means faster response times for real users, which supports your Core Web Vitals.
• Declares a sitemap location. Crawlers that support the Sitemap directive can use the URL to find your sitemap; submission does not guarantee crawling or indexing.

What it cannot do

• It cannot remove a page from search results.
• It cannot keep content private. The file is public; anyone can read yours right now.
• It cannot force compliance. Google’s own wording: the rules “cannot enforce crawler behavior to your site; it’s up to the crawler to obey them.”

Expert tip: If a URL is blocked in robots.txt and also carries a noindex tag, the noindex will never be seen, because the crawler never fetches the page. That conflict can explain why a blocked URL remains indexed: Google cannot fetch the page to read its noindex directive.

How does robots.txt work?

Before requesting URLs on a host, a well-behaved crawler fetches https://yourdomain.com/robots.txt, parses the groups inside, selects the group that matches its own user-agent name, and then applies those rules to every URL it considers. Google caches the file for roughly 24 hours, so edits are not instant.

1. Fetch. The crawler requests the file from the root of the exact host, protocol, and port. Rules on https://example.com do not apply to https://blog.example.com.
2. Group matching. The crawler uses the group that matches its user-agent. If no named group matches, it uses the User-agent: * group. Avoid duplicate groups for the same crawler; follow that crawler’s published rules when interpreting matches.
3. Path matching. Rules are matched by path prefix. Paths are case-sensitive, so /Admin/ and /admin/ are different rules.
4. Conflict resolution. Google uses the most specific rule by path length. When two rules of equal length conflict, Google uses the least restrictive one, meaning allow wins.

Example: which rule wins

User-agent: *
Disallow: /blog/
Allow: /blog/robots-txt-explained/

The Allow path is longer and therefore more specific, so that one article stays crawlable while the rest of /blog/ is blocked. Reverse the lengths and the outcome reverses with them.

Robots.txt directives explained

How to write a robots.txt file, step by step

Step 1: Check what you already have

Open https://yourdomain.com/robots.txt in a browser. If you get a 404, no rules are in force and everything is crawlable, which is a valid state for most small sites. If you see WordPress default text without a physical file, it is virtual output generated by WordPress or your SEO plugin.

Step 2: Decide what genuinely needs blocking

Block paths that create crawl waste and have no search value: internal search results, cart and checkout, session or tracking parameters, and endless filter combinations. Do not block pages simply because you do not want them ranking; that is a job for noindex.

Step 3: Write the groups

User-agent: *
Disallow: /wp-admin/
Allow: /wp-admin/admin-ajax.php
Disallow: /?s=
Disallow: /search/
Disallow: /cart/
Disallow: /checkout/

Sitemap: https://yourdomain.com/sitemap_index.xml

Keep the admin-ajax.php exception if your theme or plugins rely on it to render front-end content. Check the rendered page before changing this rule.

Step 4: Add your sitemap line

Use the full absolute URL of your sitemap index, not a relative path. You can list several sitemap lines. Adding the sitemap URL gives supporting crawlers a discovery path, but it does not guarantee crawling or indexing. See the XML sitemap guide for how to build and submit the file.

Step 5: Test before you ship

Google retired the robots.txt Tester in Search Console. First check that the file loads from the correct host. Then use URL Inspection to check whether a specific URL is allowed to be crawled. For syntax, use a current checker or inspect the rules against Google’s published specification. A successful check does not guarantee indexing.

Robots.txt for WordPress

WordPress serves a virtual robots.txt by default. Rank Math, Yoast, and similar plugins let you edit it from the dashboard, which writes to that virtual file. If a physical robots.txt exists in your web root, it overrides the virtual one, and plugin edits will appear to do nothing. Check for a real file first when edits are not taking effect.

A sensible WordPress baseline:

User-agent: *
Disallow: /wp-admin/
Allow: /wp-admin/admin-ajax.php

Sitemap: https://yourdomain.com/sitemap_index.xml

You do not need to block /wp-includes/, /wp-content/plugins/, or your uploads folder. Older guides recommend this; it is outdated advice that blocks CSS, JavaScript, and images Google needs in order to render your pages properly. If your host uses aggressive caching, confirm the file is not being served stale after edits; the same applies to LiteSpeed cache configurations.

Controlling AI crawlers with robots.txt

Robots.txt can express crawler-specific preferences, but each provider defines its own user-agent tokens and policies. Rules are voluntary, so they are not access control and do not guarantee how every AI system will use or cite your content.

Strategic caution: blocking these crawlers is a real trade-off, not a free safety measure. Blocking a provider’s search crawler can limit retrieval by that provider when it obeys the rule. Training crawlers and live search crawlers can be separate, so check each provider’s current documentation before choosing which user-agents to block. There is no universal trade-off: training use and search retrieval are different permissions, and each provider handles them differently. Check the provider’s documentation before making a choice. The llms.txt guide explains a separate proposal; it does not replace robots.txt or control crawler access.

Common robots.txt mistakes

Robots.txt vs noindex vs nofollow

The full breakdown, including when to combine them, is in noindex vs nofollow vs disallow.

Decision guide: which tool should you use?

• The page must not appear in Google: use noindex, and make sure robots.txt does not block it.
• The page wastes crawl budget and has no search value: use robots.txt disallow.
• The page moved: use a 301 redirect, not a block.
• The page is a duplicate of another: use a canonical tag, not a block.
• The content is confidential: use password protection. Robots.txt is not security.

Robots.txt audit checklist

• File returns HTTP 200 at the root of every host you own
• No stray Disallow: /
• Absolute sitemap URL present and returning 200
• CSS, JavaScript, fonts, and images all crawlable
• admin-ajax.php allowed on WordPress
• No noindex or crawl-delay directives relied on for Google
• Paths match live URL casing exactly
• No page carrying noindex is also disallowed here
• File under 500 KiB
• AI crawler policy is a deliberate decision, documented somewhere
• Verified in the Search Console robots.txt report

Frequently asked questions

Do I need a robots.txt file?

No. A site with no robots.txt is fully crawlable and that is perfectly valid. You need one when you have specific paths worth excluding, or when you want to declare your sitemap location to every crawler at once.

Does robots.txt stop a page from being indexed?

No. Google may index a disallowed URL if other pages link to it, showing the URL without a description. Only noindex, password protection, or removal reliably keeps a page out.

How long does a robots.txt change take to apply?

Google generally caches the file for up to 24 hours, and longer if it cannot refresh the cache. Plan for a day rather than minutes.

Does Google support crawl-delay?

No. Google ignores it. Bing and Yandex honour it. To slow Googlebot, adjust the crawl rate settings available in Search Console or fix the underlying server performance issue.

Can I block a specific URL parameter?

Yes, with a wildcard, for example Disallow: /*?sort=. Be careful: an over-broad parameter rule can block canonical pages that happen to carry that parameter.

Should I block AI crawlers?

Only if you have decided that withholding your content is worth losing citation visibility in AI answers. There is no universally correct answer; it depends on whether your content is the product or the marketing.

Where does robots.txt go on a subdomain?

At the root of that subdomain. Rules apply per host, protocol, and port, so blog.example.com needs its own file.

Key takeaways

• Robots.txt controls crawling, never indexing.
• Google supports four fields: user-agent, allow, disallow, sitemap.
• Most specific rule by path length wins; ties go to the least restrictive rule.
• Never block a page you want deindexed, because the noindex will go unseen.
• Never block CSS or JavaScript that pages need in order to render.
• Changes take up to 24 hours to take effect.
• AI crawler blocking is a visibility trade-off, not a default best practice.

Conclusion

Robots.txt is a small file with an outsized failure mode. Get the four supported directives right, point crawlers at your sitemap, keep render-critical assets open, and use noindex for anything that genuinely must leave the index. Audit it once a quarter and after every site migration, and it will quietly do its job for years.

Related articles

• Technical SEO guide
• XML sitemaps explained
• Noindex vs nofollow vs disallow
• Canonical tags and duplicate content
• How search engines work

External references

• Google Search Central: Introduction to robots.txt
• Google Search Central: robots.txt specification
• Google Search Central: Block indexing with noindex