A few weeks ago I published a small tool page — a color picker widget — and waited for it to show up in Google. Nothing. Two days passed, then five. I checked Search Console and saw the phrase that trips up almost every beginner: “Discovered — currently not indexed.” Not an error, not a penalty, just a page sitting in a waiting room that Google had no obligation to enter.

That experience sent me down a rabbit hole I should have explored years ago. I’ve written plenty about advanced search operators and Google dorking, but I never sat down and properly explained the machinery underneath all of it. So let’s fix that. This is the article I wish someone had handed me when I first typed site:mysite.com and got zero results.

The three-stage pipeline, in plain terms

Search engines don’t “find” pages in real time the way you might imagine. There’s no librarian walking the shelves every time you type a query. Instead, there’s a three-stage pipeline that runs asynchronously, on the engine’s schedule, not yours.

Discovery is when the engine learns a URL exists. It could come from a link on another page, a sitemap file, a manual submission, or a redirect. Discovery costs nothing and guarantees nothing.

Crawling is when a bot actually fetches the URL. Googlebot sends an HTTP request, gets back HTML, and reads it. This is where things like robots.txt and meta tags come into play — they can stop the process cold.

Indexing is when the engine parses the fetched content, extracts signals, and files it into a massive structured database so it can be retrieved later. Only indexed pages are eligible to appear in search results. A crawled page is not necessarily an indexed page, and this distinction causes more confusion than almost anything else.

The practical upshot: getting indexed is a precondition for ranking. You can have the best-optimized page on the internet and it will never appear in search if it isn’t in the index. It’s the same fundamental gap I ran into when comparing search engines versus databases — one tool retrieves from a live index, the other requires the data to exist in a curated store first.

What actually lives inside a search index

The word “index” makes people think of an alphabetical list. That undersells it badly. A modern search index is closer to a set of enormous lookup tables, and the most important one is called an inverted index.

Here’s the idea. A normal book index maps a word to a page number. An inverted index maps a term to every document that contains it. So the entry for “espresso” might point to document IDs 4471, 9082, 12004, and so on — millions of rows, each one a postings list.

That third column matters more than beginners expect. Position data tells the engine whether “espresso grinder” appears as a phrase or whether those two words are just scattered across the page. It also feeds proximity scoring, which is why reading the same two words in different orders gives you different results — a phenomenon I dug into when testing whether keywords are still enough in semantic search.

Beyond the postings, the index stores an enormous number of auxiliary attributes per document: crawl timestamps, language, mobile-friendliness signals, PageRank-style link equity, canonical URL, structured data extracted from JSON-LD, and a hash of the content so the engine can tell if the page changed since last crawl.

How a crawler decides what to fetch and re-fetch

Crawling is budgeted. Google can’t crawl the entire web every day, so it allocates a crawl budget per site based on host load, response times, and how frequently content appears to change.

In my experience, crawl frequency tracks change frequency surprisingly tightly. When I was updating a WordPress blog daily in 2023, Googlebot hit the homepage within about twenty minutes of each publish. When I let the same site sit dormant for two months, revisit intervals stretched past a week. The crawler isn’t lazy — it’s being efficient with a finite resource.

If you want to watch this behavior on your own site, your server logs are the ground truth:

Count Googlebot hits by hour over the last 7 days (nginx)

grep “Googlebot” /var/log/nginx/access.log
| awk ‘{print $4}’
| cut -d: -f1-2
| sort | uniq -c
| sort -rn | head -20

That output tells you when the bot is most active on your property, which is genuinely useful if you’re timing a content push. If you see zero Googlebot hits in the log, you have a discovery or blocking problem, not an indexing problem — those are different diagnoses with different fixes.

Blocked from index: the four ways it happens

“Blocked from index” is an umbrella phrase I see thrown around loosely. In practice there are four distinct mechanisms, and conflating them wastes a lot of troubleshooting time.

robots.txt disallow. A rule like Disallow: /private/ tells crawlers not to fetch those URLs at all. Crucially, this does not remove a page from the index — if the URL was already indexed and you add a disallow, Google may keep the bare URL in results with no snippet, because it can’t re-crawl to see the noindex you also added. This is the single most common self-inflicted wound I encounter.

Meta robots noindex. The tag <meta name="robots" content="noindex"> in the <head> is the correct way to remove a page. The crawler fetches the page, reads the directive, and drops it from the index on the next pass. This pairs with the disallow problem above: robots.txt blocks the fetch, so the noindex never gets read, so the page lingers.

HTTP status codes. A 404 or 410 will eventually drop a page from the index. A 503 is interpreted as temporary, and a sustained 5xx can throttle crawling sitewide. A 200 that returns a soft-404 page — “Sorry, product not found” with a success status — confuses the engine badly.

Canonical tags pointing elsewhere. If page B declares <link rel="canonical" href="page-A">, Google will index page A and treat page B as a duplicate. B is effectively blocked without any blocking directive existing.

The ordering lesson here is the thing that took me embarrassingly long to internalize: you must allow crawling in order to instruct de-indexing. The instinct to lock a page down with robots.txt actively prevents the removal you’re trying to achieve.

Checking whether a page is actually indexed

The most direct check is the site: operator. site:example.com/my-page returns the page if it’s in the index. I spent a weekend mapping out the quirks of this command and the short version is that it’s a filter, not a precise diagnostic — it can report near-misses, and it doesn’t show why something is missing. For the full set of tricks, my piece on the Google site command covers the edge cases.

Three more reliable checks:

1. Google Search Console URL Inspection. Paste the URL, hit “Test Live URL,” and it tells you the indexed status, the canonical Google selected, and whether rendering succeeded.
2. Direct URL query. Paste the full URL into the search box. If the exact page appears, it’s indexed. If you get a “no results” message with a suggestion, it isn’t.
3. Bing Webmaster Tools. Bing has its own index and its own quirks. A page can be indexed in Google and invisible in Bing for weeks.

I noticed that URL Inspection and the site: operator disagreed on two of my test URLs in May 2025 — Search Console said “indexed,” site: said nothing. Search Console won. It reads directly off Google’s internal state, while site: goes through the query pipeline, which is lossy.

The patience problem nobody warns you about

Here’s the honest limitation, and it’s the one that frustrates people most: you cannot force indexation. “Request Indexing” in Search Console queues a URL for priority crawling, but Google’s own documentation is explicit that it’s a hint, not a command. In my testing across roughly forty URLs on three different domains, the median time from request to index was around 26 hours for pages on a site with healthy crawl activity, and eleven days for a page on a domain that had been dormant for months.

That variance is the whole story. Indexation speed is a function of your site’s overall crawl health, not the merit of any single page. A brand-new domain with one brilliant article will wait longer than a well-established domain with one mediocre page. It’s not fair and there’s no shortcut around it.

A second caveat: indexing is continuous, not one-and-done. Pages fall out of the index. I’ve watched articles that ranked for eighteen months quietly drop after a site migration removed an internal link. If you want to know what a page looked like the last time it was properly indexed, the Wayback Machine is the closest thing to a time machine you’ll get.

Common misconceptions, corrected

“If it’s on my site, it’s indexed.” No. Sitemaps list URLs you’d like crawled. They’re suggestions. Google ignores sitemaps that contain noindexed URLs, redirects, or non-canonical pages.

“Indexed means ranked.” Being in the index makes you eligible. It doesn’t make you visible. There’s a whole chapter between eligibility and position on page one.

“I can see it in a private browser, so it’s live.” Live and indexed are orthogonal. A page can be perfectly live and completely absent from the index for months.

“Adding noindex will remove it fast.” It’ll remove it on the next crawl, which could be days or weeks out. There’s no instant button.

When I tested this directly in July 2026 on a staging subdomain, I added noindex to six pages. Five dropped within nine days. The sixth took twenty-three — and it turned out the delay was caused by a leftover canonical tag that pointed the page at a different host, which meant Google kept re-evaluating rather than simply dropping it.

What this means when you’re the searcher, not the publisher

Most readers here aren’t site owners. You’re researchers, students, and people trying to find something specific. The indexing model still shapes your results in ways worth knowing.

Because the index is a snapshot, you sometimes hit stale pages. A product page from 2021 can rank above the current one because the current one hasn’t been crawled since a redesign. If you’re comparing prices or products across sites, always check the visible publish or update date, and cross-reference with the merchant’s own site rather than trusting the snippet.

The index also skews toward what’s crawlable. Content behind a login, a paywall, or a robots.txt block simply isn’t there. That’s the mechanical reason deep web and database search are different disciplines from web search — you’re not searching a worse index, you’re searching a different collection entirely.

One practical habit that follows from all this: when a search returns nothing, don’t assume the information doesn’t exist. Assume it’s not indexed for your query, and change the query. Use different phrasing, widen the date range, or try a phrasing that matches how the page’s owner would have written it — a technique that overlaps heavily with Boolean query construction.

A short checklist for the blocked-from-index problem

If you’re the one publishing and something’s not showing up, work through this in order rather than guessing:

1. Confirm the URL returns HTTP 200 — not a soft-404, not a redirect chain.
2. Confirm robots.txt permits the path. Fetch /robots.txt directly and read it.
3. Check the <head> for noindex and for a stray canonical pointing to a different URL.
4. Verify the page is discoverable — is it linked from your navigation, or only from the sitemap?
5. Inspect the URL in Search Console and read the coverage detail, not just the headline status.
6. Request indexing once, then leave it alone for at least a week before worrying.

Step six is genuinely hard to follow. I refreshed Search Console roughly a dozen times a day for the first week of that color picker page, which accomplished precisely nothing except wasting my own afternoon. The index doesn’t care about your refresh button.

The mental model worth keeping

Think of the web as a library that keeps adding books, a crawler as a reader who visits at intervals based on how busy the shelves seem, and the index as the card catalog. The catalog is what you actually search. The books are what exist. They are never quite the same thing, and the gap between them is where all the frustration lives.

Once that clicks, a lot of search behavior stops feeling arbitrary. Why did that result vanish? Probably dropped from the index. Why does that page rank above a newer one? Probably a crawl-frequency mismatch. Why can’t I find something I know exists? Probably it was never crawled, or it was blocked, or it’s behind something the crawler can’t see.

Understanding the index doesn’t give you a magic search operator. It gives you something more useful: the ability to diagnose why a search failed, which is usually the difference between finding what you need and giving up on a query that was never going to work.