robots.txt explained: what to block and what never to block
TL;DR: robots.txt is a plain-text file at your site's root that tells well-behaved crawlers which parts of your site they should and shouldn't access, a voluntary convention that reputable crawlers (including Google's) honor but that provides no actual security or access enforcement, since a malicious actor can simply ignore it entirely. Its correct, valuable use is guiding crawl budget toward your genuinely important content and away from low-value or duplicate areas (admin pages, internal search results, certain parameter-based URL variations); its most damaging misuse is an overly broad rule that accidentally blocks your entire site, or genuinely important content, from being crawled at all.
What robots.txt actually is, and isn't
robots.txt implements the Robots Exclusion Protocol, a long-established, voluntary convention that well-behaved web crawlers check and respect before crawling a site, but it is fundamentally a request, not an enforcement mechanism. Reputable search engine crawlers (Google, Bing, and other major, legitimate crawlers) do genuinely honor it, which is why it is a real and useful tool for guiding their behavior, but nothing about robots.txt technically prevents a crawler, malicious or simply non-compliant, from accessing a "disallowed" path anyway; it relies entirely on voluntary compliance rather than any actual access control.
This means robots.txt should never be relied upon as a security measure to hide genuinely sensitive content; anything you do not want publicly accessible needs actual access control (authentication, IP restriction) rather than a robots.txt rule, which merely requests that compliant crawlers not index it, doing nothing to prevent a determined or non-compliant party from accessing it directly.
The basic syntax
A robots.txt file consists of one or more rule blocks, each specifying which crawler ("User-agent," with * meaning all crawlers) the following rules apply to, followed by Disallow lines (paths that crawler should not access) and optionally Allow lines (specific exceptions within a broader disallowed path). A Sitemap directive is also commonly included, pointing crawlers directly to your sitemap's location, covered in our XML sitemap guide.
A simple, common example:
User-agent: *
Disallow: /wp-admin/
Allow: /wp-admin/admin-ajax.php
Sitemap: https://yourdomain.com/sitemap.xml
This tells all crawlers not to access the WordPress admin area generally, but specifically allows the admin-ajax.php file within that otherwise-disallowed path, since this specific file is used by many legitimate frontend functions and blocking it entirely can break certain site features that rely on it, a genuinely common and instructive real-world example of why the more granular Allow exception exists.
What's genuinely worth blocking
Admin and login areas (/wp-admin/, with the admin-ajax.php exception noted above), since these have no value being crawled or indexed and represent pure crawl budget waste if a crawler spends time on them. Internal search result pages, since these are typically low-value, often near-duplicate content generated dynamically based on search terms, providing little unique value for indexing and potentially creating a large volume of thin, duplicate-content pages if left crawlable. Certain parameter-based URL variations (tracking parameters, session IDs, certain sorting/filtering parameters on ecommerce category pages) that generate many URLs pointing to essentially the same underlying content, a common and genuine source of crawl budget waste and duplicate content signals on larger sites, particularly WooCommerce stores with many filter combinations. Staging or development environments, if a staging site is ever inadvertently made publicly reachable, though the more robust protection here is genuine access restriction (password protection, IP allowlisting) rather than relying on robots.txt alone, given its non-enforcement nature discussed above.
What should never be blocked
Your CSS and JavaScript files. This was a more common mistake in the past (blocking /wp-content/ broadly, which inadvertently catches theme and plugin CSS/JS along with actual content), but it remains a real risk with any overly broad rule; Google specifically needs to be able to access and render your CSS and JavaScript to properly understand your page's actual layout and content as a real visitor would experience it, and blocking these resources can directly hurt how well your pages are understood and ranked. Your actual content pages, blog posts, product pages, any genuinely indexable content you want found in search, an obvious point but one worth stating directly since accidental over-broad blocking (covered next) most commonly and damagingly affects exactly this category. Your sitemap's referenced URLs. A page listed in your sitemap as something you want indexed, while simultaneously being blocked in robots.txt, creates a direct contradiction, covered as a specific error type in our sitemap errors guide, and should be checked for and resolved as a mismatch between two files that should be sending consistent signals.
The most damaging mistake: a wildcard disallow
The single most catastrophic robots.txt mistake, and one that happens more often than it should, particularly during site migrations or when moving from a staging to a live environment, is a rule reading:
User-agent: *
Disallow: /
This single line instructs every compliant crawler not to access anything on the entire site, effectively requesting complete deindexing. This is a completely reasonable, intentional setting for a genuine staging or development site not meant to be publicly indexed, but it becomes catastrophic when it accidentally carries over into a live production site, commonly after a migration where a staging site's robots.txt file was copied along with everything else and never updated for the live environment, or after a new site launch where a default "block everything during development" setting was never turned off.
The symptom is severe and often confusing when first noticed: previously well-ranking content can begin disappearing from search results, sometimes gradually as Google re-crawls and respects the new blanket disallow, which is why checking robots.txt directly should be one of the very first things verified any time indexing problems emerge unexpectedly, covered in our Search Console indexing guide.
robots.txt vs noindex: they do different things
This distinction is worth being explicit about since the two are frequently confused despite serving genuinely different purposes. robots.txt tells a crawler not to access (crawl) a given path at all; a noindex meta tag or HTTP header, by contrast, tells a crawler that has already accessed a page not to include it in the search index, a meaningfully different instruction. A page blocked by robots.txt cannot be crawled, which means Google cannot even see a noindex tag on that page if one exists, since it never accesses the page's content in the first place; conversely, a page you want genuinely excluded from indexing while still allowing it to be crawled (to pass along link authority through it, for example) should use noindex rather than a robots.txt block, since blocking prevents the crawl needed to even see and respect that more specific instruction.
A do/don't table
| Category | Block in robots.txt? |
|---|---|
| /wp-admin/ (with admin-ajax.php exception) | Yes |
| Internal search result pages | Yes, generally |
| Tracking/session parameter URL variants | Yes, generally |
| CSS and JavaScript files | No, never |
| Actual content pages (posts, products) | No |
| A genuine staging/dev site (not yet public) | Yes, but verify before going live |
| Pages you want noindexed but still crawled for link equity | Use noindex tag instead, not robots.txt |
How to check your current robots.txt safely
Load yourdomain.com/robots.txt directly in a browser to see the current live file, and cross-reference it against Search Console's robots.txt testing tool (or an equivalent standalone checker), which can confirm whether specific important URLs on your site are correctly allowed or accidentally blocked, a more reliable verification method than manually reading the raw rules and trying to reason through their effect, particularly for more complex rule sets with multiple user-agent blocks and wildcard patterns.
FAQs
Does every WordPress site need a custom robots.txt file, or is the default fine?
WordPress generates a basic virtual robots.txt automatically if no physical file exists, generally reasonable for a simple site; a custom file becomes worth creating specifically when you have identified genuine crawl budget waste (large parameter-based URL variations, for example) or specific paths worth explicitly excluding beyond WordPress's sensible default.
Can I use robots.txt to hide a page from competitors or the public?
No, as covered above, robots.txt is a voluntary convention honored by compliant crawlers, not an access control mechanism; anyone, including a competitor, can simply view the file directly and then access the "blocked" path anyway, since nothing technically prevents direct access, only crawling by compliant bots is affected.
Does blocking a page in robots.txt remove it from search results if it's already indexed?
Not directly or immediately; blocking prevents future crawling, but a page already indexed can remain in the index for some time since Google cannot re-crawl it to discover it should be removed. To reliably remove an already-indexed page, using noindex (which requires the page to remain crawlable so the instruction can be seen) or Search Console's removal tools is more direct and reliable than a robots.txt block alone.
How do I know if my robots.txt is accidentally blocking something important?
Use Search Console's URL Inspection tool on specific important URLs to directly confirm whether they are crawlable, or a dedicated robots.txt testing tool that lets you check specific paths against your current rules before assuming they are unaffected.
Does robots.txt affect AI crawlers the same way it affects search engine crawlers?
Many, though not all, AI-related crawlers do respect robots.txt as a voluntary convention similarly to traditional search crawlers, and some sites specifically add rules targeting named AI crawler user-agents to control their access separately from traditional search engines; this is an evolving area worth monitoring as AI crawling conventions continue to develop.
Is it safe to block WordPress's default /wp-content/uploads/ folder?
Generally no, this is where your actual media files, images used throughout your content, live, and blocking it can prevent those images from being properly indexed in image search and can interfere with how pages containing those images are rendered and understood; this folder should generally remain accessible to crawlers.
Can a plugin accidentally modify my robots.txt without me realizing?
Yes, some SEO and security plugins can programmatically modify robots.txt output (WordPress's virtual robots.txt can be filtered by plugin code), so if you notice unexpected robots.txt content you did not manually set, checking recently installed or updated plugins for this kind of behavior is a reasonable troubleshooting step.
Conclusion
robots.txt is a genuinely useful tool for directing crawl budget away from low-value areas of your site, but it provides no real access enforcement and, when misconfigured with an overly broad rule, can accidentally block crawlers from your entire site or from resources (CSS, JavaScript) they need to properly understand and rank your content. Checking your live robots.txt file directly, especially after any migration or site launch, is a fast, essential verification step, particularly given how severe and confusing the symptoms of an accidental blanket block can be.
HostCloud's free robots.txt tester checks specific URLs against your current rules and flags common dangerous patterns like an accidental full-site disallow before they cause real indexing damage. Try it at https://hostcloud.in/tools.