Site Audit: What a Crawl Actually Checks, and What It Misses

A site audit can report a clean crawl while the business still has pages that rank for the wrong searches, give inaccurate advice, or should not exist at all. That is because a crawler can inspect what a URL exposes technically; it cannot reliably judge whether the page is useful, correct, commercially sensible, or aligned with search intent.

The practical distinction is simple: a crawler checks what it can crawl. It can find broken links, redirect chains, missing metadata, duplicate signals, and broken structured data. It cannot establish whether the content is true, whether the page answers the right question, or whether the site needs that page in the first place.

What a site audit crawler actually checks

A technical crawl requests URLs in a way that resembles a search engine bot. It follows internal links, records server responses, reads HTML, detects directives, and compares patterns across pages. The output is valuable, but narrower than many audit reports imply.

Most crawlers are good at identifying problems that leave an observable technical trace, including:

• Broken links: internal links leading to a 4xx or 5xx response, or links pointing to an irrelevant destination.
• Redirect chains: a URL that redirects through two or more intermediate URLs before reaching the final page.
• Missing or weak metadata: absent title tags, missing meta descriptions, duplicated titles, and titles that exceed a practical display length.
• Duplicate signals: identical or near-identical pages, repeated titles, canonical tags that conflict with indexability, and parameter URLs that expose the same content.
• Structured data errors: invalid JSON-LD syntax, missing required properties, and markup that does not match the visible page.
• Access and indexation signals: status codes, robots directives, canonical declarations, blocked resources, and links that the crawler cannot follow.

Those checks answer questions such as, “Can the crawler reach this URL?” and “What instructions does the page publish?” They do not answer, “Should this page rank?” or “Is this the best result for the searcher?”

What a crawler misses, even when the report says zero errors

A crawler can confirm that a page returns 200 OK, has a unique title, and contains valid schema. None of those facts proves that the page deserves organic traffic.

It cannot judge whether the content is right or wrong

Technical tools can flag thin content by word count, but word count is not a truth test. A 300-word pricing page may be more useful than a 2,000-word article filled with unsupported claims. A crawler cannot verify whether a product specification is accurate, whether a legal statement is outdated, or whether a medical explanation is safe and current.

Human review is required for factual accuracy. Use a subject-matter reviewer, source claims against primary documentation, and record the review date for pages where information changes. For commercially important pages, set a review cycle of 3 to 6 months; highly changeable topics may require monthly checks.

It cannot reliably identify search intent mismatch

A page can target the phrase “best accounting software” while mainly explaining how accounting software works. It may be technically excellent and still fail because the searcher wants a comparison, shortlist, pricing context, or a buying recommendation.

Intent review requires examining the current search results and the business objective. Classify the dominant need as informational, navigational, commercial investigation, or transactional, then compare the page’s format, depth, evidence, and next step with that need. A crawl cannot make that strategic judgement from HTML patterns alone.

It cannot decide whether a page should exist

Sites often accumulate location pages, old campaigns, filtered URLs, supplier pages, and lightly rewritten articles. A crawler may report no technical errors on any of them. The underlying problem is page inventory, not page code.

Decide whether each page has a distinct audience, search demand, business purpose, or internal linking role. If it has none, consider consolidation, redirection, exclusion from indexation, or removal. “Every page must be optimised” is poor advice when the correct action is to reduce the number of pages.

The boundary: what crawlers detect and what needs another method

The table below separates observable crawl findings from questions that require analysis, testing, or stakeholder input. The “crawler” column refers to a properly configured crawl that can access the relevant URLs; blocked, authenticated, JavaScript-dependent, or orphaned pages may require additional collection methods.

Issue or question	Crawler detects it?	If not, or not fully, check it with
Internal link returns 404 or 500	Yes	Validate the destination, then restore, redirect, or remove the link
Redirect chain or redirect loop	Yes	Review server or application rules and test the final destination manually
Missing, duplicate, or excessively long title	Yes	Map the title to the target query and rewrite for relevance, not just length
Duplicate or near-duplicate pages	Usually, as a pattern	Review unique value, intent, backlinks, conversions, and whether consolidation is appropriate
Invalid or incomplete structured data	Often	Compare markup with visible content and validate eligibility in the relevant search feature
Page gives inaccurate or outdated advice	No	Subject-matter review, source checking, and a documented update process
Page targets the wrong search intent	No	Search-result comparison, query classification, and review of conversions and engagement
Page should not exist	No	Content inventory, stakeholder interviews, traffic and revenue analysis, and editorial judgement
Orphan page with no internal links	Not from a link-only crawl	Compare the crawl with XML sitemaps, analytics, server logs, and known URL lists

The important qualification is “usually” or “often”. A crawler detects patterns based on rules and thresholds. It does not understand the business reason behind an exception. A duplicate page may be intentional, such as a print version, regional variant, or product configuration. The report is a prompt for a decision, not the decision itself.

How to interpret the most common crawl findings

Broken links

Not every reported broken link has the same priority. A navigation link returning a 404 affects many users and deserves faster attention than an old reference in a low-traffic article. Set a practical target of zero broken internal links in primary navigation, templates, conversion paths, and high-traffic content. For the rest, prioritise by sessions, backlinks, revenue contribution, and the number of pages linking to the error.

Do not automatically redirect every deleted URL to the homepage. That popular advice is often wrong. A redirect should lead to the closest useful replacement. If no relevant replacement exists, returning a proper 410 or 404 may be clearer for users and search engines than sending them to an unrelated page.

Redirect chains

A chain such as /old to /intermediate to /current creates unnecessary requests and makes migration logic harder to maintain. Aim for one redirect from the old URL directly to the final, relevant URL. Fix chains in templates, internal links, XML sitemaps, and canonical tags rather than relying indefinitely on server redirects.

Pay particular attention after a redesign or domain migration. A first-pass audit should happen before launch, a second within 24 to 48 hours after launch, and a third after roughly 2 to 4 weeks of real crawling and traffic data.

Missing metadata

A missing title is a real technical defect, but a title that is merely “the right length” is not necessarily good. Treat common display guidance as a quality check, not a law. A title of around 50 to 60 characters may fit many results, but pixel width, device, query, and rewriting by the search engine vary.

Write titles around the page’s primary topic and differentiator. Keep one clear topic per page, avoid repeated boilerplate, and review whether the title accurately describes the visible content. Meta descriptions can influence click behaviour, but they are not a guaranteed ranking control; do not spend a week polishing them while the page itself answers the wrong question.

Duplicate content and canonical tags

Duplicate detection is useful for finding faceted navigation, tracking parameters, printer-friendly URLs, and copied templates. It does not mean every similar page should be deleted. Two pages can share substantial wording and still serve different audiences or intents.

Review page purpose, internal links, organic landing data, external references, and conversion performance before consolidating. A canonical tag is a signal, not a substitute for a coherent URL structure. Test whether the chosen canonical is accessible, relevant, self-consistent, and linked internally as the preferred version.

Structured data

Valid schema syntax does not guarantee a rich result, better rankings, or accurate presentation. Markup must describe content that users can actually see. Do not add review, product, event, or other properties simply because a template allows them. The markup should be checked against the page, the visible offer, and the eligibility rules that apply at the time of review.

A site audit workflow that avoids false confidence

1. Define the crawl set. Include the main domain, subdomains where relevant, staging exclusions, XML sitemaps, known campaign URLs, and authenticated areas if they affect the public journey. A link-only crawl is incomplete when important pages are orphaned.
2. Run the technical crawl. Capture status codes, redirect paths, indexability directives, canonicals, titles, headings, duplicate clusters, internal links, resources, and structured data. Preserve the crawl date and configuration so later audits are comparable.
3. Prioritise by impact. Fix issues blocking access, indexation, navigation, or conversion before cosmetic metadata improvements. A broken checkout path outranks a duplicated meta description on a low-value page.
4. Perform content and intent review. Sample every important template and the highest-value landing pages. Compare them with the search results, customer questions, sales objections, and current source material.
5. Audit the page inventory. Identify pages with no traffic, no links, no conversions, no distinct intent, or no clear owner. Decide whether to improve, merge, redirect, noindex, or remove them.
6. Verify changes. Re-crawl after implementation, test critical journeys manually, and monitor organic landing pages, indexation, leads, revenue, and errors for at least 2 to 6 weeks.

For a small site, a monthly crawl plus quarterly content review may be sufficient. A site publishing daily, changing prices frequently, or deploying continuously may need weekly technical checks and a content review every 1 to 3 months. The right schedule depends on change rate and business risk, not on a universal SEO calendar.

What “zero errors” really means

“Zero errors” usually means zero findings under one tool’s rules, at one point in time, for one set of URLs. It does not mean the site is healthy. The crawl may have missed orphan pages, blocked sections, rendered content, external links, post-login journeys, or URLs discovered only through logs and analytics.

More importantly, a technically clean site can still have:

• accurate-looking pages built around the wrong intent;
• thin location pages created only to capture variations of a keyword;
• outdated prices, policies, or product claims;
• strong traffic to pages that produce no qualified business outcome;
• content that answers a question but gives the reader no credible next step;
• too many URLs competing with one another for the same topic.

Do not use a single audit score as a performance target. Scores are useful for tracking obvious technical debt, but they can reward easy fixes while ignoring strategic failures. A better report separates technical defects from content judgements, shows affected URLs and business impact, assigns an owner, and records a deadline.

How to brief and buy a site audit

Before approving an audit, ask what URL sources will be used, whether JavaScript rendering is required, how orphan pages will be found, and which issues are checked manually. Ask for examples of findings that the tool cannot validate. If the provider promises that a crawl alone will reveal content quality, intent, or whether pages should exist, the scope is overstated.

For budgeting, treat costs as market-dependent rather than fixed rules. A basic automated review for a small brochure site may be priced in the low hundreds of pounds or dollars; a combined technical, content, analytics, and migration audit for a larger site can run into several thousand. The meaningful comparison is not the headline price. Check the number of URLs reviewed, the depth of manual analysis, the quality of prioritisation, implementation guidance, and whether revalidation is included.

A useful deliverable should distinguish crawler evidence from editorial or commercial judgement. That distinction prevents a clean report from becoming false reassurance. A crawler can tell you what it could reach and what the code declared. Your team still has to decide whether the page is correct, useful, relevant, and worth keeping.

Related reading

• Content strategy — scope, method and what we measure.
• AI search optimisation
• XML Sitemaps: What Belongs in One and What Does Not
• Search Console Data Limits Everyone Hits