View contents
Robots Meta Tag: Controlling How Search Engines Crawl Your Site
The robots meta tag is an HTML instruction that controls how search engines index and follow links on specific pages. According to Web.dev’s HTML metadata documentation² and Ahrefs’ practical examples¹, it’s fundamental for managing what content appears in search results and how crawl budget is distributed.
What is the robots meta tag?
The robots meta tag is a <meta> element in your HTML <head> that gives direct instructions to search engine crawlers (like Googlebot). According to Web.dev’s HTML metadata best practices², this element should be properly placed within the <head> section of the document.
Basic example¹:
<head>
<meta name="robots" content="index, follow" />
</head>
How it works:
- Googlebot visits your page
- Reads the
<meta name="robots">tag - Obeys the instructions (index/don’t index, follow links/don’t follow)
- Processes the page according to directives
Difference between robots.txt and robots meta tag
| Aspect | robots.txt | robots meta tag |
|---|---|---|
| Location | Domain root (/robots.txt) | <head> of each HTML page |
| Scope | Entire site or sections | Individual page |
| Function | Blocks crawling | Controls indexing and link following |
| Priority | First barrier | Second instruction (if page was crawled) |
Important: robots.txt blocks crawling, robots meta tag controls indexing after crawling.
Main robots meta tag directives
1. index vs noindex
index (default): Allows the page to appear in search results.
<meta name="robots" content="index" />
noindex: Prevents the page from appearing in search results.
<meta name="robots" content="noindex" />
When to use noindex:
- Login/registration pages
- Order confirmation pages
- Internal admin pages
- Temporary duplicate content
- Thank you pages after form submission
- Internal search results pages
Real example - E-commerce:
<!-- Product page: index -->
<head>
<title>iPhone 15 Pro Max - Buy Online</title>
<meta name="robots" content="index, follow" />
</head>
<!-- Shopping cart: DON'T index -->
<head>
<title>Shopping Cart</title>
<meta name="robots" content="noindex, follow" />
</head>
2. follow vs nofollow
follow (default): Googlebot follows links on the page to discover more content.
<meta name="robots" content="follow" />
nofollow: Googlebot does NOT follow links on this page. As explained in Web.dev’s HTML links documentation³, the nofollow directive tells crawlers not to follow the links present on the page.
<meta name="robots" content="nofollow" />
When to use nofollow:
- Pages with many unverified external links
- User comment pages (if not moderated)
- Thank you pages with external links
- Temporary campaign pages
Note: nofollow in robots meta tag is different from rel="nofollow" on individual links³.
3. Common combinations
As demonstrated by Ahrefs’ SEO meta tags examples¹, the most commonly used combinations are:
<!-- Index page and follow links (DEFAULT) -->
<meta name="robots" content="index, follow" />
<!-- DON'T index but DO follow links (internal navigation pages) -->
<meta name="robots" content="noindex, follow" />
<!-- Index but DON'T follow links (pages with external links) -->
<meta name="robots" content="index, nofollow" />
<!-- DON'T index NOR follow links (complete block) -->
<meta name="robots" content="noindex, nofollow" />
Shortcut: You can use all or none:
<!-- Equivalent to: index, follow -->
<meta name="robots" content="all" />
<!-- Equivalent to: noindex, nofollow -->
<meta name="robots" content="none" />
4. Advanced Google directives
noarchive
Prevents Google from showing the “Cached” link in search results.
<meta name="robots" content="noarchive" />
When to use:
- Frequently changing content (prices, availability)
- Time-sensitive information pages
- Subscription/paywall content
nosnippet
Prevents Google from showing description/snippet in results (only shows title and URL).
<meta name="robots" content="nosnippet" />
Side effect: Also automatically applies noarchive.
max-snippet
Limits snippet length to N characters.
<!-- Maximum 160 characters of snippet -->
<meta name="robots" content="max-snippet:160" />
<!-- No snippet -->
<meta name="robots" content="max-snippet:0" />
<!-- No limit (default) -->
<meta name="robots" content="max-snippet:-1" />
max-image-preview
Controls maximum size of image preview.
<!-- No image preview -->
<meta name="robots" content="max-image-preview:none" />
<!-- Standard preview -->
<meta name="robots" content="max-image-preview:standard" />
<!-- Large preview (recommended for articles) -->
<meta name="robots" content="max-image-preview:large" />
max-video-preview
Limits video snippet duration.
<!-- Maximum 30 seconds -->
<meta name="robots" content="max-video-preview:30" />
<!-- No limit -->
<meta name="robots" content="max-video-preview:-1" />
notranslate
Prevents Google from offering to translate the page.
<meta name="robots" content="notranslate" />
noimageindex
Prevents images on the page from being indexed.
<meta name="robots" content="noimageindex" />
When to use:
- Images with protected copyright
- Product images before official launch
Bot-specific directives
You can give different instructions to specific bots:
<!-- Instruction for all bots -->
<meta name="robots" content="noindex, follow" />
<!-- Specific instruction for Googlebot (takes priority) -->
<meta name="googlebot" content="index, follow" />
<!-- Instruction for Bingbot -->
<meta name="bingbot" content="index, follow" />
Common bots:
googlebot- Googlebingbot- Bingslurp- Yahoo (now uses Bingbot)duckduckbot- DuckDuckGobaiduspider- Baidu
Practical example:
<!-- Allow Google but block Bing -->
<meta name="robots" content="noindex, nofollow" />
<meta name="googlebot" content="index, follow" />
Common use cases
Case 1: Login page
<head>
<title>Login - My Site</title>
<meta name="robots" content="noindex, nofollow" />
</head>
Why: You don’t want login pages appearing in searches or Googlebot wasting time crawling internal links behind login.
Case 2: Category pages with pagination
<!-- Page 1 (main) -->
<head>
<title>Products - Page 1</title>
<meta name="robots" content="index, follow" />
</head>
<!-- Page 2, 3, 4... (pagination) -->
<head>
<title>Products - Page 2</title>
<meta name="robots" content="noindex, follow" />
</head>
Why: Avoid duplicate content by indexing only the first page, but still allow Googlebot to discover products on subsequent pages.
Case 3: Temporary/seasonal content
<!-- Black Friday campaign (valid only in November) -->
<head>
<title>Black Friday 2025 - Special Offers</title>
<meta name="robots" content="noindex, follow" />
</head>
Why: Avoid Google indexing content that will be irrelevant in 1 month.
Case 4: Thank you pages
<head>
<title>Thank You for Your Purchase</title>
<meta name="robots" content="noindex, nofollow" />
</head>
Why: No SEO value, users shouldn’t arrive directly from Google.
Case 5: Low quality or thin content
<!-- Archive page with only 2 posts -->
<head>
<title>Archive: January 2020</title>
<meta name="robots" content="noindex, follow" />
</head>
Why: Protect your site from possible penalties for thin content.
Common mistakes and how to avoid them
Mistake 1: Block in robots.txt + noindex
<!-- ❌ BAD: robots.txt blocks /admin/ -->
# robots.txt
User-agent: *
Disallow: /admin/
<!-- This page will NEVER be seen by Googlebot -->
<head>
<meta name="robots" content="noindex, follow" />
</head>
Problem: If robots.txt blocks the page, Googlebot never sees the noindex meta tag. The page could still appear in results (without description) if it has external links.
Solution:
# robots.txt - DON'T block
User-agent: *
# Disallow: /admin/ ← REMOVE THIS
<!-- Use only meta tag -->
<meta name="robots" content="noindex, nofollow" />
Mistake 2: noindex on entire page by error
<!-- ❌ BAD: noindex on homepage -->
<head>
<title>My Site - Homepage</title>
<meta name="robots" content="noindex, follow" />
</head>
Consequence: Your site disappears from Google.
Solution: Regularly audit which pages have noindex.
Mistake 3: Incorrect syntax
<!-- ❌ BAD: Incorrect spacing -->
<meta name="robots" content="noindex,follow" />
<!-- ❌ BAD: Incorrect order (doesn't matter but confusing) -->
<meta name="robots" content="follow, noindex" />
<!-- ✅ GOOD: Space after comma -->
<meta name="robots" content="noindex, follow" />
Mistake 4: Using nofollow when you meant noindex
<!-- ❌ BAD: Want it NOT indexed but wrote only nofollow -->
<meta name="robots" content="nofollow" />
<!-- Result: Page IS indexed -->
<!-- ✅ GOOD -->
<meta name="robots" content="noindex, follow" />
Mistake 5: Forgetting that noindex takes time
Problem: You add noindex and expect the page to disappear from Google immediately.
Reality: Google needs to re-crawl the page to see the change. Can take days or weeks.
Quick solution: Use Google Search Console → “Request Indexing” to speed up the process.
How to verify robots meta tags with UXR SEO Analyzer
Our Chrome extension automatically checks:
- Robots meta tag presence: Detects if it exists
- Active directives: Shows which instructions are configured
- Conflicts with robots.txt: Alerts if robots.txt blocks but has noindex
- Pages with accidental noindex: Identifies important pages that shouldn’t have noindex
- Suspicious combinations: Detects uncommon configurations
How to use the extension:
- Install UXR SEO Analyzer from Chrome Web Store
- Visit any page on your site
- Open extension → “Basic SEO” tab
- Review “Robots Meta Tag” section
- Follow specific recommendations
Next steps
Deepen your knowledge about crawling and indexing control:
- Robots Meta Tag: Complete Technical Guide - Advanced directives, X-Robots-Tag HTTP header, case studies
- Canonical URL: Complete Guide - Another way to handle duplicate content
- robots.txt: Complete Guide - Site-level crawl control
<<<<<<< HEAD
RAG References
This article was enhanced using authoritative sources identified through systematic knowledge base searches:
References
This article cites the following authoritative sources:
24a07d01237e8d2ce45f0032ef83094634b50223
[1] Ahrefs Blog: SEO Meta Tags - Robots Examples (aa1a856f-2c67-423e-aa05-6ac80b44970e) https://ahrefs.com/blog/seo-meta-tags Practical robots meta tag examples with common directives including index/follow and noindex/follow combinations. Appeared in 2 searches (scores 0.780 and 0.572), providing consistent code examples for real-world robots meta tag implementation. Ahrefs is a leading SEO industry authority.
[2] web.dev: Metadata - General Meta Tags (18a2f541-ce14-4323-ab7c-44ae51e25afd) https://web.dev/learn/html/metadata Comprehensive HTML metadata implementation guidance covering proper meta tag structure and W3C-aligned best practices. Appeared in 2 searches (scores 0.720 and 0.436), ensuring correct placement of robots directives within the document head section.
[3] web.dev: HTML Links - nofollow Attribute (97648c75-021d-4e31-8acb-20d977014199) https://web.dev/learn/html/links W3C-aligned best practices for HTML link attributes including nofollow directive. Appeared in 2 searches with identical scores (0.724), explaining how nofollow functions both as a link attribute and meta robots directive.
<<<<<<< HEAD RAG Coverage: GOOD - Combines industry SEO authority (Ahrefs practical examples) with W3C-aligned best practices (web.dev). 3 sources across 2 knowledge bases providing comprehensive coverage of robots meta tag fundamentals and implementation.
24a07d01237e8d2ce45f0032ef83094634b50223
External resources
- Google Search Central: Robots Meta Tag - Official documentation
- Google: Robots meta tag specifications - Complete list of directives
- Moz: Meta Robots Tag Guide - Detailed guide
- Bing: Robots Meta Tag - Directives supported by Bing
Need to verify your robots meta tag? Use our UXR SEO Analyzer extension for instant directive analysis, conflict detection, and specific recommendations.