Using Robots.txt NoIndex to Prevent Search Engine Indexing

This is a follow up tutorial from " What is Robots.txt and How It Will Affects Your Website? "


Do you know that you can NoIndex a specific page by using robots.txt instead of using nofollow or noindex meta tags?

If you haven't heard of it. we shall show you how you can do this inside your robots.txt file.

Unlike disallowed pages, noindexed pages don’t end up in the index and therefore won’t show in search results. Combine both in robots.txt to optimize your crawl efficiency: the noindex will stop the page showing in search results, and the disallow will stop it being crawled:

Here's the actual syntax you can use to insert both "Disallow" and "Noindex" inside your robots.txt:

Disallow: /example-page-1/
Disallow: /example-page-2/

Noindex: /example-page-1/
Noindex: /example-page-2/

Adding the syntax above in the robots.txt is quicker, cleaner, easier to manage than getting a meta no index added to specific pages. It also means less confusion over which directives override which (because the robots.txt will override any directives that have been added on the page).

Updated on: 04/02/2019