SEO Glossary
What is Robots.txt?
Robots.txt is a text file placed at the root of a website (yoursite.com/robots.txt) that tells search engine crawlers which pages or sections of the site should or should not be crawled. It is part of the Robots Exclusion Protocol.
A properly configured robots.txt file ensures search engines can efficiently crawl and index your most important content.
1
How robots.txt works
When a search engine bot visits your site, it first checks for a robots.txt file at the root domain. The file contains directives telling the bot which URLs it can and cannot access.
Important: robots.txt controls crawling, not indexing. A page blocked by robots.txt can still appear in search results if other pages link to it. To prevent indexing, use the "noindex" meta tag instead.
2
Basic syntax
User-agent: * - Applies to all bots
User-agent: Googlebot - Applies only to Google
Disallow: /admin/ - Block the /admin/ directory
Disallow: /private.html - Block a specific page
Allow: /admin/public/ - Allow within a blocked directory
Sitemap: https://yoursite.com/sitemap.xml - Point bots to your sitemap
Each directive is on its own line. Blank lines separate rule groups for different user-agents.
3
Common mistakes
• Blocking CSS/JS files - Google needs these to render pages properly
• Blocking important content - Accidentally blocking pages you want ranked
• Using robots.txt to hide pages - It doesn't prevent indexing
• Missing sitemap reference - Always include your sitemap URL
• Blocking crawl of images - Prevents images from appearing in Google Image Search
• Using "Disallow: /" - Blocks the entire site from crawling
4
Robots.txt in SEO audits
PixlSEO checks robots.txt during audits for:
• Accessibility - Is the file reachable?
• Syntax errors - Invalid directives
• Important page blocks - Pages that shouldn't be blocked
• Sitemap reference - Is the sitemap declared?
• Crawl efficiency - Unnecessary blocks wasting crawl budget
FAQ
Frequently asked questions
Does robots.txt prevent pages from being indexed?
No. Robots.txt controls crawling, not indexing. Use the "noindex" meta tag to prevent indexing.
Is robots.txt required?
No, but it's strongly recommended. Without it, bots will crawl everything they can access.
Related Terms
Check your robots.txt configuration
Run an SEO audit to identify robots.txt issues and ensure proper crawl access.
$297/mo · No contracts · Cancel anytime