TL;DR: A meaningful share of traffic to a small business website is automated, and much of it is worthless to you: vulnerability scanners probing for known exploits, scrapers copying your content, credential stuffing attempts against your login, and crawlers hitting URLs that cannot be cached.
That traffic consumes real capacity. It is a recurring cause of resource-limit errors on shared hosting and of PHP worker exhaustion on a VPS. The principle that matters is filtering as early as possible: a request rejected at the edge costs nothing, while the same request reaching WordPress costs a full PHP execution and several database queries. And the thing to be careful about is not blocking search engines, AI search agents, monitoring, or payment gateway callbacks — the last of which fails silently.
What is actually hitting your site
Categories worth distinguishing, because they warrant different responses.
Search engine crawlers. Googlebot and equivalents. You want these, they respect robots directives, and blocking them removes you from search.
AI search agents. Fetch pages to answer live user questions and cite sources. You almost certainly want these, and the distinction between them and training crawlers is covered in the AI crawler guide. Blocking them removes you from AI answers.
Vulnerability scanners. Automated sweeps probing for known exploits: requests for wp-config.php.bak, admin paths from other CMSs, exploit paths for plugins you have never installed. These are indiscriminate, constant, and entirely unwanted.
Credential stuffing and brute force. Repeated authentication attempts against wp-login.php and xmlrpc.php, using credentials leaked from unrelated breaches.
Content scrapers. Copying your articles and product data, sometimes aggressively enough to affect performance.
SEO tool crawlers. Competitive analysis services. Harmless in intent, sometimes aggressive in volume, and of no benefit to you.
Spam bots. Hitting comment forms, contact forms and registration.
Uptime monitors and preview fetchers. Yours and other people's, plus link previews from messaging apps.
The proportions are surprising. On a small site with modest human traffic, automated requests frequently outnumber real visitors, and the great majority of that automation is in the unwanted categories.
Why bot traffic is expensive
The cost is not bandwidth, which is what people assume. It is compute on the uncacheable path.
Cacheable requests are cheap. A crawler fetching a cached blog post is served from cache without PHP running. Volume here is largely harmless.
Uncacheable requests are expensive. And bots disproportionately hit uncacheable paths:
Search result URLs, which cannot be cached because the query string varies infinitely. A scraper walking search parameters generates unlimited unique uncacheable requests.
Login endpoints, which are never cached and which run authentication logic.
xmlrpc.php, which permits many authentication attempts in a single request and is a favoured brute-force target.
Admin paths, which are uncacheable by definition.
Filtered and faceted URLs on a store, which multiply into an enormous space of unique combinations.
What this consumes: PHP workers, which are finite. A worker occupied by a scanner is unavailable to a customer, which is the mechanism behind the 502 and 503 errors that appear under load. Database connections. And on shared hosting, entry process allocation, which is what produces throttling.
The compounding problem on a store: search and filter URLs on WooCommerce are both uncacheable and query-expensive, so a scraper walking your filter combinations generates the most expensive possible requests, at volume, continuously.
The practical consequence: many sites that appear to need a bigger hosting plan are spending a substantial share of their capacity on traffic that will never buy anything. Filtering it is frequently cheaper than upgrading.
Reading your logs
Before filtering anything, find out what is actually happening. Guessing produces rules that block customers.
Where to look: access logs, available in most control panels or under /var/log on a VPS. This is one of several reasons log retention matters, as the breach response guide covers for different reasons.
What to extract:
Requests grouped by user agent, which immediately shows the proportion of automated traffic and which agents dominate.
Requests grouped by source address, which reveals single sources generating disproportionate volume.
The most-requested paths, which shows whether bots are hitting search, login or filter URLs.
Requests returning 404, which is where vulnerability scanning shows up clearly, since scanners probe paths that do not exist.
Requests to wp-login.php and xmlrpc.php specifically, with their volume and sources.
What you are looking for:
A single address making hundreds or thousands of requests in a short window.
A user agent you do not recognise appearing at high volume.
Repeated requests for paths belonging to software you do not run, which is unambiguous scanning.
Sustained login attempts.
Requests walking search or filter parameters systematically.
Do this before configuring anything. An hour of log review produces targeted rules; skipping it produces generic rules that block legitimate traffic and miss the actual problem.
Then re-check monthly, since the pattern changes and a rule set that made sense a year ago may be blocking something you now need.
The layers you can filter at
The cost of a request depends entirely on how far it travels, so where you filter matters enormously.
At the edge, via a CDN or proxy. Cheapest possible, since the request never reaches your infrastructure. Provides the largest rule sets and the best bot identification. The trade-offs are covered in the Cloudflare guide, including the real IP problem you must fix first.
At the network layer, via firewall. Blocking addresses or ranges before the web server sees them. Effective and blunt, since addresses change and shared addresses affect innocent users.
At the web server. Rules in nginx or .htaccess rejecting requests by user agent, path or rate before PHP runs. Cheap, since no PHP execution occurs, and this is the best available layer for anyone without an edge provider.
At the application, via a WordPress plugin. WordPress has already loaded, plugins have initialised, and database queries have run before the plugin decides to block. This is the most expensive layer and it is where most small sites do all their filtering.
Security plugins are useful for what only they can see — failed logins, form spam, file integrity — but blocking bulk bot traffic at this layer means paying full PHP cost to reject a request.
The ordering principle: push filtering as far up as you can. If you have an edge layer, use it. If not, use web server rules for volume and reserve the plugin for application-level concerns.
A note on fail2ban on a VPS: it watches logs and blocks offending addresses at the firewall, which is a good middle layer. Effective for repeated login failures and it reacts after the fact rather than preventing the first requests.
The endpoints that attract the most junk
Knowing where the traffic concentrates makes targeted filtering easy.
xmlrpc.php. A remote publishing interface that permits many authentication attempts per request, making it a preferred brute-force target. Most sites do not use it. Check whether anything you rely on needs it, then disable or restrict it.
wp-login.php. Constant automated attempts. Rate limiting here is high-value.
Search URLs. ?s= parameters generate infinite unique uncacheable requests. Rate limiting search specifically, or requiring a minimum query length, cuts a surprising amount of load.
Filtered and faceted URLs on stores. Combinatorially enormous and expensive. Consider limiting which combinations are crawlable via robots directives and canonical tags.
Paths for software you do not run. Requests for other CMS admin paths, database admin tools, or environment files. Any request for these is unambiguously hostile, since no legitimate visitor asks for them. These make excellent blocking rules because false positives are essentially impossible.
Author enumeration. ?author=1 style requests attempting to discover usernames, which then feed credential attacks.
REST API endpoints exposing user listings. Consider restricting unauthenticated access to endpoints that enumerate users.
Feed URLs, scraped aggressively by content thieves.
| Endpoint or pattern | Why bots target it | Cacheable | Suggested action |
|---|---|---|---|
xmlrpc.php |
Many auth attempts per request | No | Disable if unused |
wp-login.php |
Credential attacks | No | Rate limit, limit attempts |
?s= search URLs |
Infinite unique URLs | No | Rate limit, minimum length |
| Store filter combinations | Combinatorial, expensive | No | Restrict crawling |
| Other-CMS admin paths | Vulnerability scanning | N/A | Block outright, no false positives |
| Environment and backup files | Credential harvesting | N/A | Block outright |
?author= enumeration |
Username discovery | Sometimes | Block or redirect |
| REST user endpoints | Username discovery | No | Restrict unauthenticated access |
| Feeds | Content scraping | Yes | Rate limit if aggressive |
| Cached posts and pages | Normal crawling | Yes | Leave alone |
What to filter and what to leave alone
Filter aggressively:
Requests for paths belonging to software you do not run. Zero false positive risk.
Requests for backup files, environment files and configuration files.
Sources generating sustained high-volume requests far outside human patterns.
Repeated failed authentication from a single source.
User agents that identify themselves as scanning tools.
Filter with care:
SEO tool crawlers, which are harmless but useless to you. Blocking is reasonable and occasionally affects your own tooling.
Aggressive but legitimate crawlers, where rate limiting is better than blocking.
Traffic from specific countries, which is a blunt instrument that blocks real customers and is trivially bypassed. Rarely worth the collateral damage.
Leave alone:
Search engine crawlers.
AI search agents that fetch to answer live queries and cite sources.
Your own monitoring.
Link preview fetchers from messaging apps, which are genuine users sharing your content.
Payment gateway callbacks, discussed below.
The general test: would blocking this cost me anything? For a request probing for a vulnerability in software you do not run, plainly not. For an unrecognised agent fetching normal pages at reasonable rates, possibly yes, so rate limit rather than block.
Rate limiting without breaking things
Rate limiting is gentler than blocking and gets misconfigured constantly.
Set limits per endpoint, not globally. A global limit tuned for login attempts will block someone browsing a product catalogue quickly. Login, search, API and general browsing warrant different thresholds.
Base thresholds on observed behaviour. Look at your logs and determine what a genuine active user does, then set limits comfortably above it. Guessing produces limits that trip on real customers.
Account for shared addresses. Corporate networks, mobile carriers and institutions put many users behind one address, which is why per-IP limits can block an entire office. This is the most common cause of legitimate users being blocked.
Prefer graduated responses. A short delay, then a challenge, then a temporary block, rather than immediate hard blocking. Errors are recoverable rather than final.
Set sensible block durations. Minutes to an hour handles automated abuse; permanent blocks accumulate and eventually block someone who was assigned that address later.
Exclude your own infrastructure. Monitoring, payment callbacks, API integrations and your own office address.
Log what you block. Without a log, you cannot tell whether a customer complaint is a rate limiting problem, and you will not discover a bad rule until someone tells you.
Test after configuring. Browse your own site quickly, submit a form repeatedly, and confirm you are not blocking yourself. Then check the block log after a day for anything that looks legitimate.
Watch for the interaction with a proxy layer. If you have a CDN in front and have not restored real client addresses, every visitor looks like the same source and your rate limits will fire on everyone at once.
What you must never block
The list worth being deliberate about, because these failures are silent.
Payment gateway callbacks. Gateways send server-to-server notifications confirming payment, arriving from automated systems that look exactly like bot traffic to a rule set. Blocking them means payments succeed at the gateway while orders never record on your site.
This is the most damaging item on this list because it produces no error. You discover it from customer complaints about missing orders.
Allowlist your gateway's callback addresses explicitly and test a transaction after any security change.
Search engine crawlers, unless you want to leave search results.
AI search agents, unless you want to leave AI answers.
Your monitoring, or you lose visibility precisely when you need it.
API integrations your business depends on: shipping providers, accounting sync, CRM, SMS.
Your own team, particularly if they work from varied locations.
Webhook receivers from any service you integrate with.
The habit that prevents most of this: after any change to security rules, run through a checklist — place a test order end to end, confirm the callback recorded, check monitoring is still reporting, and verify a search console fetch succeeds. Ten minutes, and it catches the silent failures before customers do.
A proportionate setup
What a small business site should actually do, in order.
Review logs first. An hour, identifying what is actually hitting you and where.
Block the unambiguous. Paths for software you do not run, backup and environment files, known scanning agents. Zero false positive risk, immediate benefit.
Handle xmlrpc.php. Check whether anything needs it, then disable or restrict.
Rate limit login. Either at the web server, at the edge, or via a security plugin.
Rate limit search and consider a minimum query length.
Restrict crawling of filter combinations on a store via robots directives and canonical tags, which reduces load without blocking anything.
Ensure caching is working, since a cached page served to a crawler costs almost nothing. Verified caching does more for bot load than most filtering.
If you have an edge layer, move filtering there and fix real IP restoration first.
Allowlist everything you depend on before enabling anything aggressive.
Test the payment path after every change.
Review monthly, checking the block log for false positives and adjusting.
What not to do: install three security plugins, enable every rule, and hope. Overlapping rule sets conflict, duplicate load, and produce blocks nobody can trace, which is the opposite of the intended outcome.
FAQs
How much of my website traffic is bots?
On a small business site with modest human traffic, automated requests frequently outnumber real visitors, and most of that automation is unwanted: vulnerability scanners, scrapers, credential stuffing and aggressive crawlers. Reviewing your access logs grouped by user agent gives you the actual proportion for your site rather than a general estimate.
Why does bot traffic slow down my site?
Because bots disproportionately hit uncacheable paths — search URLs, login endpoints, filter combinations — which run PHP and query the database on every request. Those requests occupy the finite PHP workers your real customers need, which is the mechanism behind resource-limit errors on shared hosting and gateway errors on a VPS.
Should I disable xmlrpc.php?
Usually yes, if nothing you use requires it. It permits many authentication attempts in a single request, which makes it a preferred brute-force target, and most modern WordPress sites do not need it. Check first, since some mobile apps and remote publishing tools depend on it and disabling it produces confusing failures.
Where is the best place to block bots?
As early as possible. A request rejected at an edge or CDN layer never touches your infrastructure. Web server rules are next best, since they reject before PHP runs. Blocking inside a WordPress plugin is the most expensive layer, because WordPress has already loaded and queried the database before the decision is made.
Will blocking bots hurt my SEO?
Only if you block the wrong ones. Search engine crawlers must be allowed, and so should AI search agents that fetch pages to answer live queries, since blocking them removes you from AI answers. Blocking vulnerability scanners, scrapers and credential stuffing has no SEO effect whatsoever.
Why did rate limiting block my real customers?
Most often because many users share one IP address, as happens on corporate networks and mobile carriers, so a per-IP limit blocks an entire office at once. It also happens when a proxy layer is in front and real client addresses have not been restored, making every visitor appear to be the same source.
Can bot filtering break my payment gateway?
Yes, and this is the most damaging failure because it is silent. Gateways send server-to-server callbacks that look exactly like automated traffic to a rule set, so blocking them means payment succeeds at the gateway while the order never records. Allowlist callback addresses explicitly and test a transaction after every security change.
Should I block traffic from other countries?
Rarely worth it. Country blocking is trivially bypassed with a proxy, blocks legitimate customers and travellers, and can block services you depend on. It is a blunt instrument whose collateral damage usually exceeds its benefit. Target behaviour rather than geography.
What is fail2ban and should I use it?
A tool that watches log files for repeated failures and blocks offending addresses at the firewall, which is a good middle layer on a VPS. It reacts after the fact rather than preventing the first requests, and it substantially reduces sustained abuse and log noise. It is standard practice on a properly configured server.
Do I need a security plugin if I filter at the server?
They serve different purposes. Server or edge filtering handles bulk unwanted traffic cheaply. A security plugin sees things only the application can: failed login patterns, form spam, file integrity changes. Use one of each rather than several plugins, since overlapping rule sets conflict and duplicate load.
How do I know what to block?
Read your access logs before configuring anything. Group requests by user agent and by source address, look at the most-requested paths and at 404 responses, and identify sustained volume from single sources. Rules built from your own logs are targeted; rules copied from an article block things you need.
How often should I review my blocking rules?
Monthly is reasonable. Check the block log for legitimate traffic being caught, and re-examine what is hitting your site, since the pattern changes. Rules added during an incident tend to persist unexamined for years and quietly block something you later came to depend on.
Conclusion
Bot traffic is a capacity problem before it is a security problem. Most of what hits a small business site is automated, most of that automation is worthless to you, and it disproportionately lands on exactly the requests that cost the most — search URLs, login endpoints, and store filter combinations that can never be cached.
Which means a site that appears to need a bigger plan is frequently a site spending its capacity on traffic that will never buy anything.
The principle that decides everything is where you filter. A request rejected at the edge costs nothing; the same request reaching a WordPress security plugin has already loaded the application and queried the database. Push filtering upward wherever you can.
Start with the unambiguous wins, because they have no downside: requests for software you do not run, backup and environment files, and known scanning agents. Then rate limit login and search, which is where the expensive volume concentrates.
And be careful in one specific direction. The failure that costs real money is not blocking too little — it is blocking a payment gateway callback, which produces no error at all. Payment succeeds, the order never records, and you find out from a customer. Allowlist your integrations explicitly and test a transaction after every rule change.
Read your logs first. An hour of reading produces rules that fit your site; an hour of copying rules from an article produces blocked customers.
HostCloud gives you access logs from the control panel, server-level rule support and extended retention on Indian infrastructure, which is what targeted filtering requires. Plans from ₹99 a month at https://www.hostcloud.in.