Discovered — Currently Not Indexed: Why Google Skips Directory Pages at Scale

Almost every explanation of "Discovered — currently not indexed" you will read tells you your content is thin. That is a guess, and Google's own definition says something else entirely:

The page was found by Google, but not crawled yet. Typically, Google wanted to crawl the URL but this was expected to overload the site; therefore Google rescheduled the crawl. This is why the last crawl date is empty on the report.

Read the second sentence again. Google never looked at the page. It cannot have an opinion about the copy, the schema, the listing count or the internal links, because it never fetched the HTML. This is a capacity decision made before any quality assessment happens.

That single distinction changes what you should do about it, and most directory operators get it wrong because they confuse this status with the one directly above it in the Page indexing report.

The two statuses mean opposite things

They sit next to each other in Search Console and they are routinely used interchangeably in SEO write-ups. They are not interchangeable.

If your unindexed bucket is mostly Crawled — currently not indexed, none of the crawl-budget advice below applies to you. Google saw the page and passed. That is a content and demand problem, and the honest answer is usually that you generated a page for a query nobody makes.

If it is mostly Discovered, keep reading, because the fix is structural and has nothing to do with rewriting listing descriptions.

Why directories hit this earlier than other sites

Google's crawl budget guide names three types of site the document is written for:

• Large sites (1 million+ unique pages) with content that changes moderately often
• Medium or larger sites (10,000+ unique pages) with very rapidly changing content
• "Sites with a large portion of their total URLs classified by Search Console as Discovered - currently not indexed"

Note that the third bullet carries no page-count threshold at all. A 4,000-listing directory with a large "Discovered" bucket is in scope for the advanced guide, even though it is nowhere near a million pages. That is unusual, and it exists because of what sites like yours do to a crawler.

A directory does not have a page count. It has a page formula. Listings, times categories, times cities, times filter combinations, times sort orders, times pagination depth. Five thousand real businesses routinely become several hundred thousand fetchable URLs without anyone deciding that should happen. Google describes the consequence directly in its faceted navigation guidance:

Because the URLs created for the faceted navigation seem to be novel and crawlers can't determine whether the URLs are going to be useful without crawling first, the crawlers will typically access a very large number of faceted navigation URLs before the crawlers' processes determine the URLs are in fact useless.

And then the part that produces your "Discovered" bucket: "if crawling is spent on useless URLs, the crawlers have less time to spend on new, useful URLs."

Two more mechanics from the same guide are worth internalising. First, crawl budget is set per hostname — example.com and listings.example.com are separate sites with separate budgets, which matters if you have split your directory across subdomains. Second, "every site starts with the same default, conservative crawl capacity limit," and it moves based on how your server responds. Slow responses, 5xx errors and 429s push it down.

Of the levers Google lists, it is explicit about which one is yours: perceived inventory is "the factor that you can positively control the most." Not speed. Not links. The number of URLs you ask Google to consider.

Four fixes that don't work

Before the things that do, here is what the directory community reaches for first, checked against Google's own documentation. Three of these are addressed directly on Google's myths and facts about crawling page.

Adding noindex to your thin pages. The crawl budget guide is unambiguous: "Don't use noindex, as Google will still request, but then drop the page when it sees a noindex meta tag or header in the HTTP response, wasting crawling time." The myths page adds the nuance that removing URLs from the index can indirectly free budget "in the long run" — but that is a slow second-order effect, not a fix for a crawl queue that is backed up today.

Blocking a section in robots.txt to push crawlers toward another one. Also addressed head-on: "Don't use robots.txt to temporarily reallocate crawl budget for other pages... Google won't shift this newly available crawl budget to other pages unless Google is already hitting your site's crawl capacity limit." robots.txt is for URLs you never want crawled, not for traffic shaping.

Gzipping your sitemaps. From the myths page: "Zipped sitemaps still have to be fetched from the server, so you're not really saving much crawling time or effort on Google's part by sending compressed sitemaps."

The Indexing API. This one is worth spelling out because it circulates as a growth hack in directory communities. Google's Indexing API quickstart states that it "can only be used to crawl pages with either JobPosting or BroadcastEvent embedded in a VideoObject," and that "any attempts to abuse the Indexing API, including the use of multiple accounts or other means to exceed usage quotas, may result in access being revoked."

There is a real exception buried in that. If you are building a job board, the Indexing API is not a hack — it is the documented, supported path, and Google recommends it over sitemaps for short-lived pages while still asking you to submit a sitemap for whole-site coverage. If you are building anything else, it is not available to you.

Two smaller myths worth retiring while you are here: 4xx responses (except 429) do not waste crawl budget, and crawling is not a ranking signal. Fixing this problem makes pages eligible. It does not make them rank.

Facets are where the budget actually goes

Google gives three ways to keep filter URLs out of the crawl queue, ordered by how well they work.

The strongest is robots.txt. Google's own example, adapted to a directory's parameter names:

user-agent: Googlebot
disallow: /*?*sort=
disallow: /*?*price=
disallow: /*?*rating=
disallow: /*?*page=
allow: /*?category=all$

The second is to move filters into the URL fragment, because Google generally doesn't support fragments in crawling and indexing. A filter state at /restaurants#cuisine=thai&open=now is invisible to the crawler by construction. This is a genuine architectural option, not a workaround — though it costs you the ability to ever rank a filtered view.

rel="canonical" and rel="nofollow" are the third option, and Google is lukewarm about both: they "are generally less effective in the long term than the previously mentioned methods." nofollow in particular only works if every anchor pointing at that URL carries the attribute — one unmarked link anywhere on the web reopens the door.

Then there is the rule that directories break most often, and it is stated as a requirement rather than a suggestion:

Return an HTTP 404 status code when a filter combination doesn't return results... Similarly, if a filter combination has no results, don't redirect to a common "not found" error page. Instead, serve a "not found" error with the 404 HTTP status code under the URL where it was encountered.

Nearly every directory ships the opposite. Ask for vegan restaurants in a town that has none and you get HTTP 200 with a friendly "No results found — try adjusting your filters." That is a soft 404, and per the crawl budget guide soft 404s "will continue to be crawled, and waste your budget." An infinite space of empty filter combinations, each returning 200, is the single most expensive thing a directory can do to its crawl queue.

One last detail from the same page: use & as your parameter separator. Commas, semicolons and brackets "are hard for crawlers to detect as parameter separators."

Diagnose it in about twenty minutes

None of this is worth acting on until you know which of your URLs Google is actually spending its budget on. Four checks, in order.

1. Count what you are asking for. Your sitemap is the bill you hand Google.

curl -s https://your-directory.com/sitemap.xml | grep -c "<loc>"

If that number is wildly larger than your listing count, find out what is inflating it before you touch anything else. We ran exactly this audit across five live directory sites in a URL-level teardown if you want a worked comparison.

2. Get the ratio, and know its limit. In the Page indexing report, compare indexed against the "Discovered" and "Crawled" buckets. Be aware of a cap that trips people up: the examples table "does not necessarily show all URLs with that issue, and is limited to 1,000 rows." Use the counts for diagnosis, not the sample list.

3. Check whether the problem is your server. Open the Crawl Stats report and look at host status and average response time. If Google is seeing timeouts or 5xx codes, your crawl capacity limit is being pushed down and no amount of URL pruning will help until that is fixed.

4. Test one empty filter combination.

curl -s -o /dev/null -w "%{http_code}\n" \
  "https://your-directory.com/browse?category=nonexistent&city=nowhere"

A 200 here means you are running a soft 404 factory. This is usually a ten-line fix in one route handler and it is the highest-leverage change on this page.

Deciding what to cut

Once you have the inventory, the decision is per page class, not global. This is the shape that works for most directories:

That last row matters more than it looks. Google's guidance: a 404 "is a strong signal not to crawl that URL again. Blocked URLs, however, will stay part of your crawl queue much longer, and will be recrawled when the block is removed." If you have a directory with churn — expired jobs, closed restaurants, delisted tools — returning proper 404s is how those URLs eventually leave your crawl queue. Redirecting them all to the homepage keeps them alive indefinitely.

Worth being straight about our own defaults here. DirectoryLaunch's local SEO module ships a minListingsToIndex threshold (3 by default) that emits noindex on city-category combinations too thin to deserve a page, and prerenders only the top combinations at build — the configuration is in the local SEO docs. That guard does its job: it keeps doorway-shaped pages out of the index. But by Google's own myths page, noindex does not save you crawl budget in the short term — Google still has to fetch the page to see the directive. If your bottleneck is crawl capacity rather than index quality, the threshold is not the lever you need. robots.txt and a smaller URL surface are.

When this is not your problem

Two cases where everything above is a waste of your afternoon.

Your site is small. Google's guidance on the Page indexing report: "If your site has fewer than 500 pages, you probably don't need to use this report." A 300-listing directory with 40 unindexed pages does not have a crawl budget problem. It has a demand problem, and the crawl budget guide opens by telling sites whose pages "seem to be crawled the same day that they are published" to stop reading.

Your pages are getting crawled and rejected. If the bucket is "Crawled — currently not indexed," you are past the crawl stage. The fix is fewer, better pages, which is the argument we made about choosing programmatic page shapes, not a robots.txt file.

And a limit that applies to everyone: "Google doesn't guarantee that all pages everywhere will make it into the Google index." One hundred percent coverage is not a target. Getting the canonical version of every page you actually want ranked into the index is.

What to do tomorrow morning

1. Read the split before anything else. Open the Page indexing report and write down two numbers: Discovered, and Crawled. If Crawled is the bigger one, close this article and go audit page quality instead.
2. Curl an empty filter combination. If it returns 200, fix that route today. It is the cheapest large win available and it stops the bleeding while you plan the rest.
3. Diff your sitemap against your listing count. Anything in the sitemap that is not a destination someone could plausibly search for comes out.
4. Write the robots.txt block from your route map, not from a template. Disallow sort, view and deep pagination parameters. Leave the canonical category and city pages alone.
5. Check host status in the Crawl Stats report last. If Google is recording availability problems, everything above is theoretical until the server responds reliably.

Then wait. Validation of a fix in Search Console "typically takes up to about two weeks, but in some cases can take much longer." Requesting indexing on individual URLs in the meantime will not accelerate a crawl queue that is capacity-bound, and for the Crawled bucket Google explicitly says there is "no need to resubmit."