Log File Analysis: Seeing What Crawlers Actually Do

Log File Analysis SEO: Seeing What Crawlers Actually Do

Most crawl reports describe what a crawler could access, not what crawlers actually requested. A sitemap, crawl tool, or index coverage report can show that a URL exists or returned a particular response; only server logs show which bots requested which URLs, how often they came back, and how much crawl capacity was spent on low-value pages.

That distinction matters when a site has faceted navigation, parameter URLs, infinite scroll, session paths, old redirects, soft 404s, or thousands of thin pages. Log file analysis SEO turns those requests into an operational picture: which bots are crawling, what they ignore, where they waste time, and what engineering or SEO change should happen next.

Why server logs are the evidence that matters

A server log records a request that reached the web server or an upstream layer such as a CDN, depending on the logging setup. A useful row normally includes the timestamp, client IP address, user-agent string, requested URL, HTTP method, status code, response size, referrer, and response time. With those fields, you can reconstruct crawler behaviour rather than infer it.

Other SEO data sources answer different questions:

• A crawler: what one tool discovered and how it behaved under its own crawl settings.
• Search Console or an equivalent search platform: how a particular search engine reports crawling, indexing, and selected examples.
• XML sitemaps: which URLs you recommend for discovery.
• Analytics: visits and events that execute or are measured in the browser.
• Server logs: which clients made requests to your infrastructure, including bots that never render JavaScript or appear in analytics.

“Only source” needs one practical qualification: logs are the only direct record of requests that reached the logging layer. A bot blocked before the server, served entirely from a cache without origin logging, or hidden by incomplete retention will not appear in the origin file. Therefore, confirm whether the data comes from the CDN edge, load balancer, web server, application server, or several layers, and whether sampling is enabled.

Do not treat a sitemap as proof of crawling. A URL can sit in a sitemap for months without a search bot requesting it. Do not treat a third-party crawler’s report as proof of Googlebot or Bingbot behaviour. And do not assume an increase in crawl requests means an increase in useful crawling: a bot can spend 90% of its requests on parameters and still appear highly active.

What to extract before interpreting the data

Start with a representative period. Seven days can reveal an acute incident, but 14 to 30 days is usually better for sites with weekly publishing cycles, international sections, or irregular crawl patterns. Retain the raw files and document the timezone, log source, collection period, and whether requests are sampled.

At minimum, group requests by:

• bot identity and verified bot family;
• URL path, query-string pattern, and canonical URL where available;
• HTTP status code, including 2xx, 3xx, 4xx, and 5xx responses;
• host, protocol, and language or country subdirectory;
• response time and response bytes;
• first-party versus external IP ranges; and
• date and hour, to expose bursts or repeated retry loops.

Calculate more than request volume. For each bot, calculate the percentage of requests to indexable HTML, non-HTML assets, redirects, errors, blocked paths, parameters, search results, internal search pages, and other low-value URL classes. Also calculate wasted crawl requests as a percentage of total bot requests. A practical first-pass formula is:

wasted crawl percentage = requests to excluded, duplicate, error, parameter, or low-value URLs / total verified search-bot requests × 100

These are prioritisation metrics, not universal ranking thresholds. On a small brochure site, 200 useless requests in 30 days may be noise. On a large ecommerce site, 200,000 parameter requests can consume meaningful server capacity and delay discovery of products that matter.

From log finding to SEO action

The table below connects observable behaviour with its likely meaning and the next action. It is deliberately action-oriented: a log review that produces only charts is not an SEO improvement.

Finding in the logs	What it means	What to do next
Repeated requests to URLs with sort, filter, or tracking parameters	The bot is spending requests on URL combinations that may duplicate one page or create an effectively unlimited crawl space.	Map valid parameter combinations. Canonicalise or redirect useful variants, remove internal links to junk variants, and use robots.txt only where blocking supports the intended strategy.
Large share of 3xx requests, especially chains	The bot is spending requests following redirects rather than fetching final content. Chains also add latency.	Update internal links and sitemaps to the final URL. Replace chains with one-step redirects and remove obsolete redirect targets where safe.
Frequent 4xx or soft-404-like URLs	Internal links, old sitemaps, external links, or templates are repeatedly exposing dead or empty paths.	Fix internal references, clean sitemaps, return an accurate status, and decide whether a genuinely removed page needs a relevant replacement or a 410 response.
5xx responses, timeouts, or slow responses concentrated in bot sessions	Crawling is colliding with application, database, hosting, or rate-limit problems.	Give the development team timestamps, paths, status codes, and latency. Check capacity and bot controls before changing crawl directives.
High request volume for old content but little activity on newly published URLs	Discovery and recrawl demand may be concentrated in stale sections, internal links may be weak, or the new URLs may be difficult to find.	Strengthen relevant internal links, update sitemaps, review publication architecture, and compare request patterns after deployment.
Bot requests to pages disallowed in robots.txt	A compliant crawler should not fetch those paths, so the rule may be new, misplaced, conflicting, or being ignored by an unverified client.	Compare the exact user-agent token and robots response. Verify the client before attributing the requests to a major search engine.
Only one locale or hostname receives repeated crawling	Internal links, hreflang implementation, redirects, or sitemap coverage may be directing discovery unevenly.	Segment logs by host and locale, then inspect links, hreflang references, canonical signals, and sitemap ownership.

Set thresholds that trigger investigation

Use thresholds as alarms, not laws. A sensible starting dashboard might flag more than 20% of verified search-bot requests to non-canonical parameter URLs, more than 10% to 4xx or 5xx responses, any redirect chain longer than one hop, or more than 5% of requests returning server errors in a rolling seven-day period. For response time, investigate bot-facing HTML endpoints when the server-side response time exceeds 1 second consistently, and escalate urgently when timeouts or 5xx responses cluster.

For a very large site, the absolute number of wasted requests may be more useful than the percentage. A 5% waste rate on 2 million requests is 100,000 avoidable requests. Conversely, a 40% rate on 50 requests may not justify an engineering project. Always report both the percentage and the count.

Verify the bot: user-agent strings are not credentials

Anyone can send a request claiming to be Googlebot, Bingbot, an AI crawler, or a browser. The user-agent string is a label, not proof. Acting on it without verification can lead to the wrong robots rules, false security conclusions, or an accidental block of a legitimate crawler.

Use a two-part verification process for important bot traffic:

1. Take the source IP address from the log and perform a reverse DNS lookup. The hostname should belong to the claimed organisation’s documented domain, rather than a generic cloud-hosting or residential ISP domain.
2. Perform a forward DNS lookup on that hostname. It should resolve back to the original IP address. This forward-confirmed reverse DNS check helps prevent a hostname from being fabricated to look legitimate.

Do not rely on reverse DNS alone. Check the organisation’s current published verification guidance where available, compare the user-agent and request pattern, and record the verification date. DNS ownership can change, and infrastructure may use multiple domains or ranges. For a high-impact block, validate a sample of IPs rather than one request.

Unverified traffic should be reported as “claims to be” a bot, not as that bot. In analysis, separate verified search bots, verified vendor bots, unverified claimed bots, and ordinary clients. This prevents a noisy scraper from inflating the apparent crawl rate of a search engine.

AI crawlers belong in the same analysis

AI-related bots can be operationally important even when they are not search crawlers. Some collect content for training or retrieval, some fetch pages when a user asks a question, and some belong to search products that generate summaries. Their goals, controls, and commercial implications differ, so grouping all of them as “AI bots” is too vague.

Identify each bot by verified ownership, user-agent, IP evidence, request frequency, and URL distribution. Look for:

• high-volume fetching of full HTML and article pages;
• requests for pages that search bots rarely visit;
• bursts that ignore normal user demand patterns;
• repeated retrieval of the same content without asset requests; and
• requests to unpublished, private, or parameterised paths.

Do not assume that blocking every AI crawler improves SEO. A training crawler may have no direct role in search visibility, while a retrieval bot associated with a search or assistant product may affect how your content is surfaced. The correct action depends on your legal position, commercial policy, server cost, and distribution strategy. Use verified identity and documented policy decisions, then monitor the logs after any robots.txt or access-control change.

Also do not assume robots.txt is a security boundary. It expresses crawl preferences; it does not protect confidential material. Private content needs authentication and access controls. Logs can reveal that a supposedly hidden path is being requested, but they cannot make that path private.

What not to do with log file analysis

• Do not block every parameter. Some parameters create valuable, indexable landing pages. First classify combinations and understand internal linking.
• Do not block a bot because it crawls too much. Verify it, identify the expensive URL patterns, and fix the crawl trap or server problem first. Blanket blocking can hide symptoms rather than solve them.
• Do not use analytics to estimate bot crawling. Many bots do not execute analytics scripts, and browser visits are not server requests.
• Do not delete raw logs after producing a chart. Keep enough history to compare before and after a release, redirect change, robots update, or template migration. A 90-day retention window is a practical minimum for many sites; high-change sites may need 180 days or more.
• Do not optimise for request count alone. A bot fetching fewer URLs is not necessarily a win if it stopped reaching important new pages. Pair crawl volume with coverage of valuable, indexable URLs and response health.

The popular advice that “more crawling is always better” is wrong. More requests to duplicate filters, error pages, or slow endpoints can consume infrastructure capacity while contributing nothing to discovery or indexing. The better objective is efficient crawling of the URLs that deserve attention.

Turn the analysis into a repeatable operating process

Run a baseline over 14 or 30 days, classify URL patterns, verify bot identities, and record the three largest sources of waste. Assign each finding to an owner: SEO for canonical and internal-link decisions, development for routing and performance, infrastructure for capacity and logging, and legal or commercial stakeholders for AI crawler policy.

After a change, wait long enough to capture comparable behaviour. Use at least 7 days for a high-volume site with daily crawling and 14 to 28 days for a lower-volume site. Compare request share, absolute requests, status codes, response time, and valuable URL coverage. A successful fix might reduce parameter requests by 35% while leaving total requests unchanged because the bot has shifted towards product pages; that is better than celebrating a simple volume decline.

Finally, preserve a change log alongside the data. Note the release date, robots.txt edits, redirect migrations, sitemap changes, CDN configuration, and major template launches. Log file analysis becomes strategically useful when it explains cause and effect: what crawlers actually did, what changed, and whether the next crawl budget was spent on pages that can earn search visibility.

Related reading

• Pricing — scope, method and what we measure.
• Our services
• Comparing Domains: Which Metrics Are Worth Looking At
• MCP for SEO: Connecting Your Data to an AI Assistant