Noindex is a rule that removes pages from search engine results. You implement it as either a <meta> tag in your HTML <head> or an X-Robots-Tag HTTP response header. Marketers use it to hide staging environments, internal search results, and duplicate content while keeping the pages accessible to users.
What is Noindex?
Noindex is a directive that tells search engine crawlers not to add a specific page to their index. It functions as both a robots meta tag in your document head and as an HTTP header value. When Googlebot crawls a page and extracts this rule, Google drops the page entirely from Search results, regardless of whether other sites link to it (Google Search Central).
You cannot specify noindex rules in your robots.txt file. Google does not support this method (Google Search Central).
Why Noindex matters
- Protect sensitive content: Keep login pages, admin panels, and private databases out of public search results.
- Manage duplicate content: Prevent indexing of printer-friendly versions, mobile duplicates, or parameterized URLs that fragment your visibility.
- Optimize crawl budget: Stop search bots from wasting resources on thin content or transitory pages, ensuring they focus on high-value content.
- Clean up search results: Remove internal search result pages, which Google advises against indexing.
- Hide staging sites: Block development and test environments from appearing in search results before launch.
How Noindex works
When a crawler visits your page, it checks for indexing instructions in two locations:
- HTML
<head>: The crawler looks for a meta tag such as<meta name="robots" content="noindex">. - HTTP headers: The crawler checks the
X-Robots-Tagheader in the server response.
If either contains the directive, Google removes the page from results. This happens even if the page has inbound links (Google Search Central).
Implementation methods
Meta tag
Place this in the <head> section to block all supporting search engines:
<meta name="robots" content="noindex">
Target specific crawlers by changing the name value:
- Google only: <meta name="googlebot" content="noindex">
- Bing only: <meta name="bingbot" content="noindex">
- Baidu only: <meta name="baiduspider" content="noindex"> (Wikipedia)
HTTP response header
Return this header for non-HTML resources like PDFs, videos, and images:
X-Robots-Tag: noindex
This method works when you cannot insert HTML meta tags into the file (Google Search Central).
Combined directives
You can pair noindex with other rules. For example, noindex, nofollow tells crawlers not to index the page and not to follow links on it.
Best practices
Verify with Search Console Use the URL Inspection tool to confirm Googlebot sees your tag. If your robots.txt blocks the URL, Googlebot cannot access the page to see the noindex instruction, and the page will remain indexed (Google Search Central).
Don't mix blocking methods Never disallow a URL in robots.txt while also using noindex. The crawler cannot access the page to discover the tag. Allow crawling, or remove the disallow rule.
Be selective Apply noindex only to pages that truly require exclusion. Monitor the Page Indexing report in Search Console to track which pages Google has excluded based on your tags.
Use sitewide settings carefully Plugins like All in One SEO (AIOSEO), which has over 3 million active users (AIOSEO), allow global noindex settings. Use these only for staging sites or highly sensitive internal environments, then remove them before launch.
Consider alternatives for confidentiality Noindex hides pages from search results but does not block access. For sensitive data, use password protection instead.
Common mistakes
Mistake: Adding noindex to robots.txt You cannot specify noindex rules in robots.txt. Google ignores this syntax.
Fix: Use the meta tag or HTTP header method.
Mistake: Blocking the URL while using noindex If robots.txt prevents crawling, Googlebot cannot see the tag.
Fix: Allow crawling so Googlebot can discover the directive.
Mistake: Expecting immediate removal Google must recrawl the page to see the tag. Depending on the page's importance, this takes weeks or months.
Fix: Request recrawling via the URL Inspection tool for faster processing.
Mistake: Trying to noindex part of a page
Googlebot does not recognize partial-content techniques like the <noindex> HTML tag (used by Yandex) or class="robots-noindex" microformats. These methods do not work for Google (Wikipedia).
Fix: Remove sensitive content from the page entirely.
Mistake: Noindexing paginated archives Google improved handling of paginated content. Yoast SEO removed the feature to noindex subpages of archives in version 6.3 because it can prevent Google from understanding page relationships in a series (Yoast).
Noindex vs robots.txt Disallow
| Factor | Noindex | Robots.txt Disallow |
|---|---|---|
| Goal | Remove indexed page from results | Prevent crawler access |
| Crawler access | Allowed (to see the tag) | Blocked |
| Best for | Pages that should stay live but hidden | Files you don't want crawled (APIs, large media) |
| Index impact | Page removed from index over time | Page may remain indexed if previously crawled |
Examples
Example scenario: Staging environment
Your development site lives at staging.example.com. You add <meta name="robots" content="noindex"> to every page template. Google indexes the main site but ignores the staging environment.
Example scenario: Internal search results
Your site generates URLs like /search?q=widgets when users search. You set these to noindex to prevent Google from indexing thousands of low-value search result pages.
Example scenario: Author archives You run a single-author blog. You noindex the author archive pages to prevent duplicate content issues with your blog roll, using the method available in SEO plugins or theme settings.
FAQ
How long until a noindexed page disappears from results? Google must recrawl the page to discover the tag. This takes days for important pages, or months for less significant pages. You can request a recrawl via Search Console's URL Inspection tool to accelerate removal.
Can I noindex my entire website? Yes, by adding a global noindex tag to your header template or using a sitewide setting in an SEO plugin. Do this only for staging environments or sites with highly sensitive information. Remove it before launching public content (AIOSEO).
What's the difference between noindex and nofollow? Noindex keeps the page out of search results. Nofollow tells crawlers not to pass authority through links on that page. Since Yoast SEO version 14.4, you can control individual link attributes separately from page-level directives (Yoast).
Does noindex affect crawl budget? Yes. Noindexing low-value pages prevents search engines from wasting crawl resources on them, allowing deeper crawling of important content (AIOSEO).
Can I noindex part of a page?
No. Googlebot does not recognize techniques like <noindex> tags or CSS classes to exclude partial content. These methods work on other search engines like Yandex, but not Google. Remove the content entirely or use JavaScript to load it after render (Wikipedia).
Will noindex remove a page immediately if indexed? No. The page stays indexed until Googlebot recrawls it. For immediate removal, use Google's Removals tool in Search Console as a temporary measure while waiting for the recrawl.
