Correct the zone-map explanation, and the reasoning that produced it (#391) - #406
Conversation
The page said q2 reads all 667 row groups because neither key is sorted. Half of that is wrong, and the wrong half is mine: I wrote it in commandprompt#381 and jdatcmd approved it. Re-measured on the same table, one predicate at a time: hostname only 667 of 667 read, 0 removed 10,791 ms time only 118 of 667 read, 549 removed 2,415 ms both, as q2 118 of 667 read, 549 removed 2,238 ms Time pruning already removes 82 percent of the groups. Hostname removes none. The zone maps work. The gap is a hostname clustering failure. From the zone_map catalog, decoded and checked against min(time) first: all 667 groups record the same minimum and maximum for hostname, so no group can be ruled out. A group's time span averages about 6 minutes, which is 0.30 percent of the table's range. The old 667 of 667 figure is exactly what the hostname-only predicate gives, and the page attributed it to full q2. Nobody could reproduce that pairing, so per jdatcmd's call on commandprompt#391 it is replaced with a fresh measurement rather than explained. A new section covers the reasoning error, because it will recur otherwise. pg_stats.correlation measures value order against physical row order across the whole relation. Skipping depends on each group's minimum and maximum being narrow, which is local. This table separates the two: groups tight to about 6 minutes, but rotated rather than ascending, so correlation is near zero while skipping still removes 549 of 667 groups. The instrument that answers the question is Columnar Chunk Groups Removed by Filter, one predicate at a time. Closes commandprompt#391.
jdatcmd
left a comment
There was a problem hiding this comment.
Approved. This corrects a claim that contradicted the page's own timing table.
The correction is right, and there is a sharper argument for it than the one the PR
makes. The old text was not merely arguing from the wrong instrument. It was
inconsistent with the numbers printed eighty lines above it.
The old block claimed q2 reads all 667 groups and filters 99,995,680 rows. This PR
measures a 667-group read of that table at 10,791 ms (the hostname-only arm). The
headline table has always reported q2 at 2,211 ms. A query cannot read 667 groups in
2.2 seconds when reading 667 groups takes 10.8. The old EXPLAIN block was stale or
came from a different query, and nobody caught it because it agreed with the story the
prose was telling.
Worth a sentence to that effect, because it is the durable lesson: the page held the
evidence to catch itself and did not, since the two numbers were never put side by side.
On the substance
The correlation point is correct and the distinction is the one that matters. Whole
relation correlation is a global statistic; zone-map skipping is a per-group property.
The rotated-but-locally-ordered layout is exactly the case that separates them, and
0.0133 correlation against 549 of 667 groups pruned is as clean a counterexample as you
could ask for.
"Run one predicate at a time, because a conjunction hides which half is doing the work"
is the most useful sentence in the diff.
Consistency checks I ran
- q2 at 2,238 ms in the new table against 2,211 ms in the headline table: the same
measurement inside noise. Good. - The
correlationmention further down saysANALYZEcollects it and the planner uses
it for locality. That is the planner's cost model, not zone-map skipping, so it does
not contradict the new "do not infer pruning fromcorrelation" rule. It sits far
enough away that I doubt a reader connects them, but a clause tying the two would not
hurt. - No other
667on the page outside this section.
One question, not blocking
hostname only 10,791 ms, time only 2,415 ms, both 2,238 ms. The conjunction beating
its more selective half is fine as noise, but if these are single runs the table should
say so. It reads as authoritative to four significant figures and it should say whether
it earned that.
The conclusion this leaves is the useful one: our zone maps work, and the q1 through q3
gap is a hostname clustering problem. That reframes what we would build to close it.
Closes #391. Follows your plan on that issue, including the three verification steps.
The correction
The page said q2 reads all 667 groups because neither key is sorted. Re-measured on the
same table, one predicate at a time:
hostnameonlytimeonlyTime pruning already removes 82 percent of the groups. Hostname removes none. The
zone maps work. The gap is a
hostnameclustering failure, which is the second ofTimescaleDB's two mechanisms, not the first.
Verification, per your three steps
1. Decomposition with
EXPLAIN (ANALYZE, BUFFERS). Above. Buffers came out atshared hit=70995for the hostname arm against20491for the time arm, which is thesame story in a second currency.
2. Physical order from the zone maps, not
pg_stats. Decoded thebyteaminima andmaxima out of
pgcolumnar.zone_map. The decode is checked before anything rests on it:the minimum across all groups' minima is
2024-01-01 00:00:00+00, which equalsmin(time)on the table.And for
hostname, the half that does not prune:All 667 groups share one minimum and one maximum. Every group holds all 4,000 hosts.
That is the clearest statement of the problem I could find, and it comes from the catalog
rather than from a plan.
3. Environment.
work_mem256 MB,shared_buffers16 GB, serial, bloom on. No spill:every arm reports
shared hitonly, no read and no temp.The 667 that could not be reproduced
Your call, taken.
667 / 667is exactly the hostname-only figure, and the page pairedit with full q2's row counts. That pairing does not occur in any arm I can produce now, so
the number is replaced with a fresh measurement rather than explained.
The methodological section, which you said matters more
Added, because the next person will otherwise repeat it. This table is a clean separator:
groups individually tight on
timeat about 6 minutes, but rotated rather thanascending, group 1 starting 14:00:10 and group 663 starting 13:31:00. A whole-relation
correlation is near zero. Skipping still removes 549 of 667.
So
correlationanswers "does value order track physical row order across the relation",and skipping asks "is each group's range narrow", which is local. The page now says to use
Columnar Chunk Groups Removed by Filter, one predicate at a time, since a conjunctionhides which half is working.
The setup list also no longer cites correlation as evidence that the load order is
unsorted, since that is the claim this corrects.
Docs only.
ste_checkclean.