View contents
Meta Robots: How to Control Indexing and Crawling of Your Pages
Introduction
The meta robots tag tells search engines how to treat a specific page: whether they can index it, whether they should follow its links, and which snippets they're allowed to show in search results. The UXR SEO Analyzer checks each page's meta name="robots" tag to catch conflicting directives or accidental indexing blocks.
What the Meta Robots Tag Is
It sits inside the HTML <head> and accepts a comma-separated list of directives:
<meta name="robots" content="noindex, nofollow">There's also an equivalent HTTP header, X-Robots-Tag, which works on any resource type (PDF, images, JSON), not just HTML:
X-Robots-Tag: noindex, nofollowMost Common Directives
| Directive | Effect |
|---|---|
noindex | Prevents the page from appearing in search results |
nofollow | Tells the crawler not to follow the page's links |
noarchive | Prevents a cached copy of the page from being shown |
nosnippet | Prevents a text or video preview from being shown in results |
max-snippet:[number] | Limits the length of the displayed text snippet |
max-image-preview:[setting] | Controls the size of the image preview (none, standard, large) |
noimageindex | Prevents images on the page from being indexed |
The Most Common Mistake: Blocking in robots.txt AND Expecting noindex to Work
Google's developer blog has explained that if robots.txt blocks crawling of a URL, Googlebot can never fetch it to read the noindex tag—so the page can still show up indexed (without content) from external links. The correct sequence is: allow crawling in robots.txt, let Googlebot visit the page, and let it find the noindex directive there.
Conflict Resolution
When contradictory values exist (for example, an index directive in the HTML and a noindex in the HTTP header), Google always applies the most restrictive combination: noindex wins over index, and nofollow wins over follow.
Typical Use Cases
- Internal search result pages (avoiding low-quality duplicate content)
- Thank-you or checkout pages (no search value)
- Staging environments accidentally left public
- Filter pages with parameters that generate endless combinations
Meta Robots vs. robots.txt
These are distinct mechanisms that are frequently confused:
| Mechanism | Controls | Effect if blocked |
|---|---|---|
robots.txt | Crawling | Googlebot never visits the URL; the meta robots tag is never read |
meta robots | Indexing | Googlebot visits the URL, reads it, but doesn't index it |
That's why, to reliably remove a page from the index, the noindex directive must always be paired with allowed access in robots.txt.
Meta Robots for a Specific Bot
The name attribute also accepts a specific crawler's name instead of robots, which lets you give different instructions to each search engine:
<meta name="googlebot" content="noindex">
<meta name="bingbot" content="index, follow">This is useful when you want to exclude a page only from Google while keeping it visible in other search engines, though in practice most sites use the generic robots directive to simplify maintenance. Mixing bot-specific and generic directives on the same page is rarely necessary and increases the chance of a contradiction slipping through unnoticed.
What the UXR SEO Analyzer Checks
The tool flags when a key page on the site carries noindex with no apparent intent, when duplicate or contradictory directives exist across multiple meta robots tags, and when robots.txt blocks pages that also carry noindex—a pattern that prevents Google from ever seeing the directive.
References
- Google Search Central - Robots Meta Tags Specifications
- Google Search Central - Block Search Indexing with noindex
- Google Search Central Blog - Improving on Robots Exclusion Protocol
- MDN Web Docs - meta name="robots" attribute value