Robots Meta Tag vs robots.txt: Which Tool for Which Job
28/07/2026
Two tools with similar names cause the most confusion in technical SEO: the robots.txt file and the robots meta tag. Both control search engine behavior, but at completely different levels. Mix them up and one of two classic accidents happens: a page you wanted to hide still shows in search, or a page you wanted out of the index can never be read by Google to see your instruction in the first place.
robots.txt controls crawling, not indexing
robots.txt is a text file at the root of the domain (for example example.com/robots.txt). It tells bots which parts of the site they may visit, that is, crawl. When you write Disallow: /admin/, you tell Google: do not spend resources visiting this folder. But here is the key thing many people miss: robots.txt does NOT remove a page from the index. If other sites link to a blocked URL, Google can still show it in results — just without a description, because it was not allowed to read the content.
The robots meta tag controls indexing
The robots meta tag sits in the <head> of an individual page: <meta name="robots" content="noindex">. It tells Google: you may read this page, but do not show it in results. Values combine — noindex, nofollow, noarchive and so on. The same instruction can also arrive as an X-Robots-Tag HTTP header, which is useful for non-HTML files such as PDFs.
- I want the bot not to spend crawl on a section → robots.txt Disallow.
- I want a page out of Google → robots meta noindex.
- I want a PDF out of the index → X-Robots-Tag: noindex in the header.
The most dangerous mistake
The most common and most expensive mistake is putting a noindex intention into robots.txt, or blocking a page in robots.txt while also giving it a noindex meta tag. In the second case Google may not enter the page, so it never reads the noindex — and the page stays in the index. The rule is simple: if you want something out of the index, you must ALLOW crawling (do not block it in robots.txt) so Google can see the noindex tag. Only once the page has dropped from the index can you block it in robots.txt.
Practical scenarios
Login pages, carts and internal search results usually get noindex. Test and staging environments should be password-protected, not just handled with robots.txt. Duplicate URLs with parameters are best solved with a canonical tag, not a block. If you run a larger site, these rules intertwine with crawl management — we cover that in detail in the guide to crawl budget optimization. The basics of setting up files and the sitemap are covered in sitemap.xml and robots.txt, and for solving duplicates see the guide to canonical tags.
Where hosting plays a role
X-Robots-Tag and correct serving of robots.txt depend on the server configuration, so good hosting with clear access to .htaccess or headers is a practical prerequisite. If you are choosing an environment that does not get in the way of technical SEO, look at our SEO hosting plans and packages and pricing.