Introduction

Robots Txt Explained

5 min read
View contents

Robots.txt: Controlling Search Engine Crawling

Introduction

The robots.txt file is one of the most fundamental tools for communicating with search engines. This small text file, located at the root of your website, acts as a set of instructions telling web crawlers which parts of your site they can explore and which they should avoid.

Although it may seem simple, robots.txt has a significant impact on how search engines discover and index your content. According to Google Search Central (March 2025), robots.txt has been in active use for over 30 years and is broadly supported by crawler operators, including tools for site owners, services, and search engines.

What is Robots.txt?

Robots.txt is a plain text file that follows the Robots Exclusion Protocol (REP), an industry standard that allows webmasters to communicate their crawling preferences to web bots. When a crawler visits your site, the first thing it does is look for this file at https://yourdomain.com/robots.txt.

The file contains simple directives that specify:

  • User-agent: Which crawler the rules apply to
  • Disallow: Which paths should NOT be crawled
  • Allow: Which paths CAN be crawled (overrides Disallow)
  • Sitemap: Location of your XML sitemaps

Basic example:

User-agent: *
Disallow: /admin/
Disallow: /private/
Allow: /admin/public/
Sitemap: https://example.com/sitemap.xml

Why is it Important for SEO?

Robots.txt is crucial for search engine optimization for several key reasons that directly affect how Google and other search engines interact with your site.

Key benefits:

  • Crawl budget optimization: Prevents bots from wasting resources crawling pages with no SEO value (admin areas, internal search pages, duplicate filters)
  • Sensitive content protection: Keeps private areas out of crawlers’ reach (though it’s NOT a security method)
  • AI crawler control: With the rise of AI crawlers (GPTBot, Claude-Web, Google-Extended), robots.txt lets you control which bots can access your content
  • Server overload prevention: Reduces unnecessary requests that could affect performance
  • Duplicate content management: Blocks URL parameter versions that create duplication

Basic Best Practices

1. Correct File Location

The robots.txt file MUST be at your domain’s root. It won’t work in subdirectories.

✅ Correct: https://example.com/robots.txt
❌ Incorrect: https://example.com/folder/robots.txt

Important: Each subdomain needs its own robots.txt. According to Bing Webmaster, Googlebot doesn’t recognize the root domain’s robots.txt for a subdomain.

2. Clear and Consistent Syntax

Use one directive per line and keep the format clean:

User-agent: Googlebot
Disallow: /search/
Disallow: /filter/
Allow: /search/important/

User-agent: *
Disallow: /admin/
Sitemap: https://example.com/sitemap.xml

3. Always Include Your Sitemap

Declaring your XML sitemap location in robots.txt helps search engines discover all your important pages:

Sitemap: https://example.com/sitemap.xml
Sitemap: https://example.com/sitemap-posts.xml

4. Manage AI Crawlers (2025)

With the expansion of generative AI, consider including directives for AI crawlers:

# AI Crawlers
User-agent: GPTBot
Disallow: /premium-content/

User-agent: Claude-Web
Disallow: /premium-content/

User-agent: Google-Extended
Disallow: /

Common Mistakes to Avoid

❌ Mistake 1: Using robots.txt to Hide Pages from Search Results

Problem: Blocking a page with robots.txt does NOT prevent it from appearing in search results. Google can index URLs without crawling their content if other sites link to them.

Solution: To prevent indexing, use the noindex meta tag or the X-Robots-Tag: noindex HTTP header.

<!-- This DOES prevent indexing -->
<meta name="robots" content="noindex">

❌ Mistake 2: Blocking CSS and JavaScript Files

Problem: Blocking CSS/JS prevents Googlebot from rendering your page correctly, negatively affecting your SEO.

Solution: Allow access to resources needed for rendering:

# ❌ Incorrect
Disallow: /css/
Disallow: /js/

# ✅ Correct - allows CSS/JS
User-agent: *
Allow: /css/
Allow: /js/
Disallow: /admin/

❌ Mistake 3: Forgetting that robots.txt is Public

Problem: Anyone can see your robots.txt by visiting yourdomain.com/robots.txt, potentially revealing sensitive paths.

Solution: Don’t use robots.txt as a security measure. Truly private areas should be protected with authentication.

❌ Mistake 4: Conflicting Rules

Problem: Contradictory rules can cause unexpected behavior.

Solution: Remember that the more specific rule takes priority:

User-agent: *
Disallow: /catalog/
Allow: /catalog/featured/  # This more specific rule wins

Practical Example

Here’s a complete, well-structured robots.txt for an e-commerce site:

# robots.txt for e-commerce site
# Last updated: 2025

# Rules for all bots
User-agent: *
Disallow: /admin/
Disallow: /cart/
Disallow: /checkout/
Disallow: /account/
Disallow: /search?
Disallow: /*?filter=
Disallow: /*?sort=
Allow: /

# Specific rules for Googlebot
User-agent: Googlebot
Disallow: /cart/
Disallow: /checkout/
Allow: /

# AI crawler control
User-agent: GPTBot
User-agent: ChatGPT-User
User-agent: Claude-Web
User-agent: anthropic-ai
Disallow: /exclusive-content/

# Sitemaps
Sitemap: https://example.com/sitemap.xml
Sitemap: https://example.com/sitemap-products.xml
Sitemap: https://example.com/sitemap-categories.xml

This example:

  • Blocks administrative and user areas
  • Prevents crawling of filter parameter URLs
  • Manages AI crawlers for premium content
  • Declares multiple sitemaps for better discovery

How to Check with UXR SEO Analyzer

The UXR SEO Analyzer extension helps you verify if your site has an accessible and well-configured robots.txt:

  1. Install the UXR SEO Analyzer extension in Chrome
  2. Navigate to any page on your website
  3. Open the extension
  4. Go to the “Basic SEO” tab
  5. Check the “Robots.txt” evaluator

The extension will verify:

  • If the robots.txt file exists and is accessible
  • If the syntax is valid
  • If there are directives that could affect crawling

Next Steps

Want to master all advanced robots.txt techniques? Read our complete robots.txt best practices guide where we cover:

  • Advanced syntax and wildcards
  • Crawl budget management for large sites
  • X-Robots-Tag as an HTTP alternative
  • Testing and validation tools
  • Case studies with measurable results

Additional Resources

Note: This article is part of our SEO analysis series. Explore all articles in the Basic SEO Hub.

Sources: Google Search Central (2025), Bing Webmaster Blog (2023-2025), robotstxt.org

Related articles

Related version

Detailed guide

Robots Txt Best Practices Guide

This technical guide delves into advanced robots.txt strategies to maximize crawl efficiency and protect your content

Category hub

Hub

Basic Seo Fundamentals Hub

Every successful website is built on a foundation of solid SEO fundamentals

In the same category

Introduction

Robots Meta Tag Explained

The robots meta tag is an HTML instruction that controls how search engines index and follow links on specific pages

Introduction

Xml Sitemaps Explained

XML Sitemaps are a fundamental tool for communicating directly with search engines

Introduction

Canonical Url Explained

The canonical URL (canonical tag) is an HTML element that tells Google which is the preferred version of a page when multiple URLs exist with similar or...

Other topics

Introduction

Language Declaration Explained

Language declaration (the attribute) is a fundamental HTML element that tells browsers, search engines, and assistive technologies the primary language of...