Digital marketing term
Robots.txt
Robots.txt is a simple text file placed in a website's root directory that tells search engine bots which parts of the site they should and shouldn't crawl.
Detailed explanation
This file contains rules for search engine spiders and is typically used to block crawling of areas that shouldn't be indexed, such as hidden admin panels, filtered search results, or duplicate pages. If misconfigured, it can accidentally stop important pages from being indexed.
The file doesn't hide page content — it only directs crawling behavior — so it isn't a secure way to conceal sensitive data. A reference to the sitemap's location is also commonly included in this file.
During technical SEO audits, checking whether this file is configured correctly prevents crawl budget from being wasted on unnecessary pages and helps important pages get indexed faster.
Frequently asked questions
- Does robots.txt hide pages?
- No, it only directs bots' crawling behavior; pages linked to from elsewhere can still be indexed.
- How does robots.txt relate to the sitemap?
- Adding the sitemap's address inside this file lets search engines discover a site's full URL structure more quickly.
Related terms
Internal links for the topic cluster — read these concepts together.
- SEOSEO (Search Engine Optimization) is the set of technical and content practices that help a website rank more visibly in organic search results.
- SitemapAn XML sitemap is a file listing a site's URLs that helps search engines discover and crawl pages more efficiently, especially on large or complex sites.
- Crawl BudgetCrawl Budget refers to the amount of resource search engine bots allocate to crawling a website's pages within a given time period; it especially affects how promptly important pages get indexed on large-scale sites.
- Canonical URLA Canonical URL is an HTML tag that tells search engines which address should be treated as the "primary" version among multiple pages with identical or very similar content.
- IndexingThe step after crawling when a search engine stores a page and may show it for queries. Being crawled is not enough if the page never enters the index.