Sitemap.xml and robots.txt: What They Do and How to Set Them Up

$body$

Every website has two files visitors never see, yet search engines read them before anything else: robots.txt and sitemap.xml. The first is a set of rules about what crawlers may visit; the second is a list of pages you want indexed. Configured correctly, they are invisible and just work. Configured wrong, they can erase your site from search results overnight.

Robots.txt: the rules at the front door

Robots.txt lives at yourdomain.com/robots.txt and crawlers read it before visiting a single page. A typical WordPress robots.txt is short:

• User-agent: * — the rules apply to all bots.
• Disallow: /wp-admin/ — the admin area is off limits.
• Allow: /wp-admin/admin-ajax.php — the exception WordPress needs.
• Sitemap: https://yourdomain.com/sitemap.xml — the pointer to your sitemap.

The most dangerous mistake on the internet is a single line: Disallow: /. It forbids crawling of the entire site, and it frequently gets left behind after a site launch, when the developer blocked indexing of the staging version. If your site does not show up on Google, check this first. It also matters to know what robots.txt is not: it is not protection. A page blocked in robots.txt can still appear in results if someone links to it — to truly keep a page out of the index you need a noindex tag.

Sitemap.xml: the map for search engines

A sitemap is an XML list of every page you want indexed, with last-modified dates. It does not guarantee indexing, but it helps Google discover new and updated pages faster — especially important for larger sites and online stores with hundreds of products. On WordPress you do not have to build anything by hand: SEO plugins like Yoast or Rank Math generate and update the sitemap automatically, usually at /sitemap_index.xml. Posts, pages, products and categories belong in the sitemap — tag archives, author archives and other zero-value pages do not.

Three steps to a correct setup

• Open yourdomain.com/robots.txt and confirm there is no Disallow: / and that the sitemap path is listed.
• Open the sitemap in your browser and make sure it loads without errors and contains the right URLs (https, no duplicates).
• Submit the sitemap in Google Search Console (Indexing → Sitemaps) and check the status occasionally — that is also where you see how many pages are actually indexed.

Small settings, big effect

These two files are a set-once-and-forget job — but worth rechecking after every major site change, migration or redesign. If you are just building your search presence, have a look at our SEO hosting, and for a solid foundation there are our hosting plans with free SSL — because the https in your sitemap URLs has to be right, too.

$body$