The robots.txt File: What It Does and How to Use It
The robots.txt file controls how bots crawl your site. Careful: it is not a way to prevent indexing.
· 1 min read
The robots.txt file is a simple text file placed at the root of a site (https://www.example.com/robots.txt). It tells crawlers which parts of the site they may or may not visit.
A crucial point: crawling is not indexing
robots.txt controls crawling, not indexing. A blocked URL can still appear in results (without a description) if other sites link to it. To prevent a page from being indexed, use the noindex tag and keep the page accessible to bots, because they must be able to read it to see that instruction.
The basic syntax
User-agent: *
Disallow: /cart/
Disallow: /search/
Sitemap: https://www.example.com/sitemap.xmlUser-agent: the targeted bot (*means all bots).Disallow: the path not to be crawled.Allow: an exception to a disallow rule.Sitemap: the address of your XML sitemap.
What is it used for?
- Preventing bots from wasting time on pages with no SEO value: cart, internal search pages, filters that generate many URL combinations.
- Reducing server load on large sites.
- Indicating the location of the sitemap.
Mistakes to avoid
- Blocking the whole site by mistake: the line
Disallow: /forbids crawling of everything. It is a classic error after going live from a staging site. - Blocking the CSS and JavaScript files needed for rendering: Google must be able to see the page like a visitor.
- Using robots.txt to hide sensitive information: the file is public, so it reveals the paths you want to hide. Protect them with authentication.
- Combining Disallow and noindex on the same page: the bot won't be able to read the noindex.
How to test it
Open /robots.txt in your browser and check the content. Google Search Console offers a robots.txt report to see how Google read it, and the URL inspection tool shows whether a page is blocked.
Keep this file short and clear, and recheck it at every redesign or migration.