A website crawler solves this by following your links from the start page and checking every page it finds. This article explains what a crawl finds that a single audit cannot, what it cannot find and how to use the results without drowning in a list of hundreds of rows.
How a crawler works
A crawler behaves a little like a search engine. It opens your start page, reads the links on it, follows each one to a new page and repeats until it has seen the pages it can reach or hit a limit. For every page it records what came back: the status, the title, the description, the headings, how much text there is, how long it took and how many clicks it took to reach.
That is why a crawl shows patterns. One page with a missing title is a fix. Forty pages with a missing title is a template problem, and it tells you where to look.
Errors you would never notice by hand
The first thing a crawl reveals is the status of every page. Pages returning a 404 mean a link points at something that no longer exists. Pages returning a 5xx error mean the server failed for that page. Redirects, shown as 3xx, are normal in small numbers, but a large number of redirected links inside your own site means your menus and content still point at old addresses.
Broken links waste the authority your pages pass to each other and frustrate visitors. A crawl lists each broken link and where it appears, so you can fix the source. See how to find and fix broken links for the repair steps.
Titles, descriptions and headings across the whole site
Duplicate titles are one of the most common site wide faults and one of the easiest to miss. If twenty pages share the title Products, search engines cannot tell them apart and visitors cannot either. A crawl groups the pages that share a title so you can see the biggest groups first.
The same applies to the length of titles and descriptions, to pages with no H1 and to pages with several H1 headings. These are quick fixes in bulk, and they tend to affect how your pages look in search results more than almost anything else on the page.
Thin pages, slow pages and buried pages
A crawl counts the words on each page. Pages with very little text are often thin, such as empty tag pages, near empty category pages or leftover test pages. They give a search engine little to work with, and many of them can be merged, expanded or removed.
It also shows which pages are slowest and largest, which is where speed work should start. And it measures click depth, how many clicks it takes to reach a page from the start page. Important pages buried six clicks deep are harder for visitors and search engines to find, so link to them from higher up.
Pages in your sitemap and pages not in it
Comparing what the crawler found with what your sitemap lists is a useful check. Pages that are crawled but not in the sitemap may be missing from it by mistake. Pages in the sitemap that the crawler never reached may be orphans, which means no page links to them.
There is one honest limit to know about. A crawler that follows links cannot find a page that nothing links to, and it only covers as many pages as its limit allows. If your site is larger than the limit, the report says so, and you can crawl again with a higher limit or start from another section.
How to use the results without getting overwhelmed
Do not read the report row by row. Start with the summary and the biggest groups. Fix errors first, because they hurt visitors and waste link strength. Then fix the faults that repeat across many pages, since one template change can fix hundreds of rows. Leave one off wording changes until last.
Export the page list if you want to filter it in a spreadsheet, and keep it. Next month, crawl again and compare. A shrinking list shows that the work is paying off, and a growing one shows that new problems are arriving as fast as you fix old ones.
Running a crawl with Dynamic Growth
The Whole-Site Crawl follows your links across up to 100 pages and costs five credits for one report. It lists broken pages and redirects, duplicate and too short or too long titles and descriptions, H1 problems, thin pages, the slowest and largest pages, click depth and how the pages compare with your sitemap, with a page list you can download. Each issue comes with what to do next.
Use the SEO Audit for one important page in depth, and the crawl for the site as a whole. Once the main problems are fixed, weekly Monitoring keeps watching your main page. For the checks to run by hand, read the technical SEO audit checklist.
Frequently asked questions
What is a website crawler?
A tool that starts at one page, follows the links it finds and checks every page it reaches. It reports the status, title, headings and speed of each page, so you can see site wide problems.
What does a website crawl find that an audit of one page does not?
Broken links and errors on other pages, duplicate titles, thin or very slow pages, pages buried deep in the site and differences between your pages and your sitemap.
How many pages can the crawl check?
Up to 100 pages per run in our Whole-Site Crawl, which costs five credits. If your site is larger, the report says so and you can crawl another section.
Can a crawler find pages that nothing links to?
No. It follows links, so a page that no other page links to will not be found. Comparing the crawl with your sitemap helps you spot those pages.
How often should I crawl my site?
After any redesign, migration or large content change, and then every month or two for a site that changes often.