Technical SEO is the infrastructure layer that lets search engines find, crawl and correctly index your site, no matter how good your content is. You can write the best article in the world, but if Googlebot can't reach the page or chooses not to index it, that page simply doesn't exist in search results. In this guide we walk through how crawling and indexing actually work, what crawl budget is, what robots.txt does, and which indexing controls you actually have, with practical examples.
Crawling and indexing: two separate stages
For a page to appear in results it goes through two steps, and confusing them is the most common mistake:
- Crawling: Googlebot visits a URL, downloads the HTML, and follows the links on the page to discover new URLs.
- Indexing: the crawled content is processed, understood and added to the search index. Only an indexed page becomes eligible to show in results.
The key point: a page being crawled does not mean it will be indexed. Likewise, if you block crawling, you can't guarantee Google will drop the page, because it can no longer see the content to act on. This distinction is the foundation of the difference between robots.txt and noindex.
What is crawl budget, and when does it matter?
Crawl budget is the practical number of URLs Google will crawl from your site within a given window. It has two components: crawl capacity, which reflects how much load your server can handle, and crawl demand, which reflects how much Google actually wants to crawl your content.
For most small sites (a few hundred pages) crawl budget is a non-issue; Google crawls them without trouble. It starts to matter when you have:
- Large sites with tens of thousands or millions of URLs (e-commerce, classifieds).
- Filter/sort combinations that multiply through URL parameters (faceted navigation).
- Slow servers or frequent 5xx errors — when Google sees strain, it lowers the crawl rate.
The way to spend your budget well is to keep low-value URLs out of Google's path and steer crawl power toward the pages that actually matter. Choosing the right tool for that job is essential.
robots.txt: directing the crawl
robots.txt is a plain-text file at the site root (https://yoursite.com/robots.txt) that advises bots which paths not to crawl. It is not an access barrier but an etiquette; well-behaved bots (Googlebot included) honour it. A basic example:
User-agent: *
Disallow: /admin/
Disallow: /cart/
Disallow: /*?sort=
Sitemap: https://yoursite.com/sitemap.xmlThe most critical conceptual error here is this: Disallow blocks crawling, not indexing. If a page receives many external links, Google may still index that URL despite the robots.txt block — but because it can't read the content, it shows up as "No information is available for this page." If you want a page to stay out of the index, do not block crawling; on the contrary, let Google crawl the page so it can see the noindex tag.
noindex and indexing controls
To keep a page out of the index, use the meta robots tag or an HTTP header. The tag added to the page's <head>:
<meta name="robots" content="noindex, follow">For non-HTML files (PDFs, images) you can achieve the same effect with an HTTP header:
X-Robots-Tag: noindexOther tools commonly used in indexing management:
- Canonical tag:
<link rel="canonical" href="...">points to the "primary" version among several similar URLs, consolidating duplicate-content signals onto a single URL. - 301 redirect: sends permanently moved pages to a single target and passes link equity along.
- XML sitemap: a list of the canonical URLs you want indexed; it speeds up discovery but does not guarantee indexing.
Sitemaps and internal linking
An XML sitemap is the most direct way to tell Google "these URLs matter to me." It should contain only indexable, 200-returning, canonical URLs; it must not list noindex'd or redirected pages. A simple entry looks like this:
<url>
<loc>https://yoursite.com/blog/technical-seo</loc>
<lastmod>2026-06-27</lastmod>
</url>Still, the primary way Google discovers URLs is internal links. "Orphan" pages that receive no link from any page are rarely crawled. Give your important pages meaningful links from the navigation, the body content and related posts; this makes discovery easier for both users and bots.
Rendering and JavaScript
Modern sites generate part of their content client-side with JavaScript. Google can process this, but rendering is a two-stage process: the HTML is crawled first, then JavaScript is executed in a render queue. Leaving your critical content to client-side rendering alone can delay indexing or cause some links never to be discovered. Use server-side rendering (SSR) or static generation where you can, so the content is visible in the raw HTML.
Verification: what is being crawled, what is being indexed?
Measure instead of guessing. The URL Inspection tool in Google Search Console shows a single page's crawl and index status, its canonical choice, and the rendered HTML. The Pages (Indexing) report lists which URLs are excluded and why. By examining your server logs you can also see which URLs Googlebot actually visits and how often — the most precise way to diagnose crawl budget problems.
Frequently Asked Questions
If I block a page in robots.txt, will it leave the index?
No, there is no guarantee. Disallow only stops crawling. A URL already in the index can stay there for a while even after being blocked. To remove a page from the index for certain, allow crawling and add a noindex tag.
Should I worry about crawl budget on a small blog?
Usually not. On a site with a few hundred pages Google crawls every URL comfortably. Budget starts to matter when your page count reaches the tens of thousands or parameterised URLs multiply.
Will adding a sitemap boost my rankings?
Not directly. A sitemap is a discovery aid, not a ranking factor; it helps Google find your URLs faster but is no substitute for content quality and links.
Let's review your site's crawling and indexing health together. I can examine your Search Console data, your robots.txt and sitemap setup, and produce a concrete improvement plan. Get in touch with me.