Skip to content

Correct the zone-map explanation, and the reasoning that produced it (#391) - #406

Merged
jdatcmd merged 1 commit into
commandprompt:mainfrom
ChronicallyJD:fix/391-zonemap
Aug 5, 2026
Merged

Correct the zone-map explanation, and the reasoning that produced it (#391)#406
jdatcmd merged 1 commit into
commandprompt:mainfrom
ChronicallyJD:fix/391-zonemap

Conversation

@ChronicallyJD

Copy link
Copy Markdown
Collaborator

Closes #391. Follows your plan on that issue, including the three verification steps.

The correction

The page said q2 reads all 667 groups because neither key is sorted. Re-measured on the
same table, one predicate at a time:

predicate groups read groups removed time
hostname only 667 of 667 0 10,791 ms
time only 118 of 667 549 2,415 ms
both, as q2 118 of 667 549 2,238 ms

Time pruning already removes 82 percent of the groups. Hostname removes none. The
zone maps work. The gap is a hostname clustering failure, which is the second of
TimescaleDB's two mechanisms, not the first.

Verification, per your three steps

1. Decomposition with EXPLAIN (ANALYZE, BUFFERS). Above. Buffers came out at
shared hit=70995 for the hostname arm against 20491 for the time arm, which is the
same story in a second currency.

2. Physical order from the zone maps, not pg_stats. Decoded the bytea minima and
maxima out of pgcolumnar.zone_map. The decode is checked before anything rests on it:
the minimum across all groups' minima is 2024-01-01 00:00:00+00, which equals
min(time) on the table.

 group_number |           lo           |           hi           |   span
--------------+------------------------+------------------------+----------
            1 | 2024-01-01 14:00:10+00 | 2024-01-01 14:06:30+00 | 00:06:20
            2 | 2024-01-01 14:06:30+00 | 2024-01-01 14:12:40+00 | 00:06:10
          666 | 2024-01-01 13:49:50+00 | 2024-01-01 13:56:00+00 | 00:06:10
          667 | 2024-01-01 13:56:00+00 | 2024-01-01 14:00:10+00 | 00:04:10

 groups |   table_span    | avg_group_span  | pct_of_table
--------+-----------------+-----------------+--------------
    667 | 2 days 21:26:30 | 00:12:29.190405 |         0.30

And for hostname, the half that does not prune:

 groups | distinct_minima | distinct_maxima
--------+-----------------+-----------------
    667 |               1 |               1

All 667 groups share one minimum and one maximum. Every group holds all 4,000 hosts.
That is the clearest statement of the problem I could find, and it comes from the catalog
rather than from a plan.

3. Environment. work_mem 256 MB, shared_buffers 16 GB, serial, bloom on. No spill:
every arm reports shared hit only, no read and no temp.

The 667 that could not be reproduced

Your call, taken. 667 / 667 is exactly the hostname-only figure, and the page paired
it with full q2's row counts. That pairing does not occur in any arm I can produce now, so
the number is replaced with a fresh measurement rather than explained.

The methodological section, which you said matters more

Added, because the next person will otherwise repeat it. This table is a clean separator:
groups individually tight on time at about 6 minutes, but rotated rather than
ascending
, group 1 starting 14:00:10 and group 663 starting 13:31:00. A whole-relation
correlation is near zero. Skipping still removes 549 of 667.

So correlation answers "does value order track physical row order across the relation",
and skipping asks "is each group's range narrow", which is local. The page now says to use
Columnar Chunk Groups Removed by Filter, one predicate at a time, since a conjunction
hides which half is working.

The setup list also no longer cites correlation as evidence that the load order is
unsorted, since that is the claim this corrects.

Docs only. ste_check clean.

The page said q2 reads all 667 row groups because neither key is sorted. Half
of that is wrong, and the wrong half is mine: I wrote it in commandprompt#381 and jdatcmd
approved it.

Re-measured on the same table, one predicate at a time:

  hostname only   667 of 667 read, 0 removed    10,791 ms
  time only       118 of 667 read, 549 removed   2,415 ms
  both, as q2     118 of 667 read, 549 removed   2,238 ms

Time pruning already removes 82 percent of the groups. Hostname removes none.
The zone maps work. The gap is a hostname clustering failure.

From the zone_map catalog, decoded and checked against min(time) first: all
667 groups record the same minimum and maximum for hostname, so no group can
be ruled out. A group's time span averages about 6 minutes, which is 0.30
percent of the table's range.

The old 667 of 667 figure is exactly what the hostname-only predicate gives,
and the page attributed it to full q2. Nobody could reproduce that pairing, so
per jdatcmd's call on commandprompt#391 it is replaced with a fresh measurement rather than
explained.

A new section covers the reasoning error, because it will recur otherwise.
pg_stats.correlation measures value order against physical row order across
the whole relation. Skipping depends on each group's minimum and maximum being
narrow, which is local. This table separates the two: groups tight to about 6
minutes, but rotated rather than ascending, so correlation is near zero while
skipping still removes 549 of 667 groups. The instrument that answers the
question is Columnar Chunk Groups Removed by Filter, one predicate at a time.

Closes commandprompt#391.

@jdatcmd jdatcmd left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Approved. This corrects a claim that contradicted the page's own timing table.

The correction is right, and there is a sharper argument for it than the one the PR
makes. The old text was not merely arguing from the wrong instrument. It was
inconsistent with the numbers printed eighty lines above it.

The old block claimed q2 reads all 667 groups and filters 99,995,680 rows. This PR
measures a 667-group read of that table at 10,791 ms (the hostname-only arm). The
headline table has always reported q2 at 2,211 ms. A query cannot read 667 groups in
2.2 seconds when reading 667 groups takes 10.8. The old EXPLAIN block was stale or
came from a different query, and nobody caught it because it agreed with the story the
prose was telling.

Worth a sentence to that effect, because it is the durable lesson: the page held the
evidence to catch itself and did not, since the two numbers were never put side by side.

On the substance

The correlation point is correct and the distinction is the one that matters. Whole
relation correlation is a global statistic; zone-map skipping is a per-group property.
The rotated-but-locally-ordered layout is exactly the case that separates them, and
0.0133 correlation against 549 of 667 groups pruned is as clean a counterexample as you
could ask for.

"Run one predicate at a time, because a conjunction hides which half is doing the work"
is the most useful sentence in the diff.

Consistency checks I ran

  • q2 at 2,238 ms in the new table against 2,211 ms in the headline table: the same
    measurement inside noise. Good.
  • The correlation mention further down says ANALYZE collects it and the planner uses
    it for locality. That is the planner's cost model, not zone-map skipping, so it does
    not contradict the new "do not infer pruning from correlation" rule. It sits far
    enough away that I doubt a reader connects them, but a clause tying the two would not
    hurt.
  • No other 667 on the page outside this section.

One question, not blocking

hostname only 10,791 ms, time only 2,415 ms, both 2,238 ms. The conjunction beating
its more selective half is fine as noise, but if these are single runs the table should
say so. It reads as authoritative to four significant figures and it should say whether
it earned that.

The conclusion this leaves is the useful one: our zone maps work, and the q1 through q3
gap is a hostname clustering problem. That reframes what we would build to close it.

@jdatcmd
jdatcmd merged commit eee8fb3 into commandprompt:main Aug 5, 2026
11 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

benchmarks.md: the zone-map explanation is half wrong, and the correlation argument behind it does not hold

2 participants