How does robots.txt affect crawling?

The robots.txt file, placed at the root of a site, provides crawling directives to search engine bots. It allows you to block certain sections (like /admin/ or /checkout/) from being crawled.

While it helps optimize crawl budget, it doesn’t prevent indexing if the blocked page is linked elsewhere.

To fully stop indexing, pair it with noindex or remove the page altogether.

A misconfigured robots.txt can harm SEO by unintentionally blocking important pages such as product categories or sitemaps.