Skip to content

v5: Radically improve memory footprint - #191

Merged
mourner merged 37 commits into
mainfrom
v5-new
Sep 3, 2026
Merged

v5: Radically improve memory footprint#191
mourner merged 37 commits into
mainfrom
v5-new

Conversation

@mourner

@mourner mourner commented May 18, 2026

Copy link
Copy Markdown
Member

TL;DR: 45–75% less retained memory and up to 79% lower peak across the bench corpus, with init build time within ~10% of v4 (and 50–70% faster on point datasets). Tile output is the same; the public API gets a class export, a trimmed getTile envelope, a new zero-copy getTileRaw, and shipped TypeScript types.

A focused rework of geojson-vt's internal data shape to slash memory footprint on large datasets — the workloads that have caused browser tabs to OOM. The headline case: building the county index (205 MB GeoJSON of US counties) now holds 103 MB instead of 241 MB on default settings, and the transient working set during a heavy drilldown build peaks at ~430 MB instead of 1.9 GB — a workload that previously crashed browser tabs now fits comfortably.

Breaking changes

  • Class export. The default export is the GeoJSONVT class; geojsonvt(data, options) becomes new GeoJSONVT(data, options). The UMD global is renamed from geojsonvt to GeoJSONVT.
  • Trimmed getTile result. getTile returns {features} only. Internal bookkeeping fields that used to leak through (numPoints, numSimplified, numFeatures, source, x, y, z, minXmaxY, transformed) are gone, and each call builds a fresh envelope instead of returning the mutated internal tile.
  • Sub-pixel output differences. Source coordinates are now stored as integer quanta, so tile coordinates can differ from v4 by one pixel in rare cases (see the two updated fixtures). Anyone snapshot-testing tile output should expect some churn.

New

  • getTileRaw(z, x, y) returns the tile as stored: flat [x, y, x, y, …] typed arrays (Int16Array, or Int32Array for large extents), one per ring, no per-coordinate objects. Lone points come back as type: 4 with x/y inline. Zero-copy — the arrays belong to the index and are read-only.
  • TypeScript types are hand-authored in src/index.d.ts and shipped with the package; @types/geojson-vt is no longer needed. @types/geojson becomes a runtime dependency for the input type.

Fixes

  • A point lying exactly on a shared tile edge now lands in exactly one tile (half-open clip interval) instead of both.
  • Polygons straddling tile boundaries no longer produce degenerate single-vertex rings at high zooms (e.g. 44 phantom [[4096, 4096]] rings for a polygon over (0, 0) at z14 on defaults).
  • Empty parts in a MultiPolygon are skipped instead of breaking the feature.
  • Geometries with null/missing coordinates throw the standard "Input data is not a valid GeoJSON object." error instead of an opaque TypeError.

What changed internally

Int32 source coords with a Float64 fallback. Source coordinates flow through convert → wrap → clip → tile.source as Int32Array, encoded as maxZoom-pixel quanta centered on zero. Halves the per-coord byte cost on the retained source slab (8 B → 4 B). Centering keeps every value inside V8's SMI range so reads stay on the fast integer path. A runtime gate falls back to Float64Array when (extent + 2*buffer) * 2^maxZoom > 2^32, covering extreme-extent configs.

Flat coords with inline ring headers. One buffer per feature in shape [ringLen, ringSize, x, y, z, …] instead of nested per-ring arrays. ringSize is signed area for polygons (encodes outer/hole) and stored sqrt-linear to fit Int32. Buffers are pre-sized exact-fit in convert, wrap and clip (two-pass clip: exact precount, then write).

Typed committed-tile coords. Every committed ring is retained as Int16Array (or Int32Array when extent+buffer > 32767), exact-sized via a pre-count pass over simplification z-values.

Specialized single-Point feature shape. Single-Point features use a 5-slot {id, type, x, y, tags} wrapper — no geometry array, no bbox slots. MultiPoint unchanged. Drives the point-dataset wins.

Foundation cleanup. Canonical winding enforced once at convert (GeoJSON structural nesting → signed-area outer/hole encoding), with the separate rewind pass removed. Eager extent projection moved inline into createTile; retained tile is immutable. Three clip implementations collapsed onto unified internal types (POINT=1, LINE=2, POLYGON=3). Public types now hand-authored via .d.ts + JSDoc with tsc --noEmit --checkJs in CI.

Benchmarks (v5-new vs v4 main)

Each dataset is built in two configurations, both measured:

  • init — default options (indexMaxZoom: 5, indexMaxPoints: 100k). The normal usage pattern: build a small pre-tiled index, generate the rest on demand.
  • deepindexMaxZoom: 10, indexMaxPoints: 1000. Proxy for heavy drilldown across the whole dataset. Not a typical user config, but a useful stress test for the transient working set during recursive clip+split.

Median of 3 iterations on the same machine, node --expose-gc, v8.GCProfiler for schedule-immune allocation tracking. held is post-build forced-GC retention (heap + external); peak is in-build high-water; alloc is total bytes allocated during build. Numbers re-measured on the final branch (2026-09-02).

Held memory — what stays in RAM after the index is built

dataset init v4 init v5 Δ deep v4 deep v5 Δ
earthquakes 3.8 MB 1.1 MB −71% 10.6 MB 3.8 MB −64%
places 4.5 MB 1.2 MB −73% 9.3 MB 3.3 MB −65%
route 1.4 MB 658 KB −54% 2.4 MB 1.1 MB −54%
hrr 15.2 MB 6.4 MB −58% 42.0 MB 16.1 MB −62%
us-2010 38.0 MB 21.0 MB −45% 87.0 MB 45.5 MB −48%
county 241 MB 103 MB −57% 323 MB 147 MB −54%

Peak memory — does this OOM on a constrained device?

dataset init v4 init v5 Δ deep v4 deep v5 Δ
earthquakes 12.0 MB 3.2 MB −73% 25.3 MB 11.8 MB −53%
places 14.0 MB 3.6 MB −74% 23.5 MB 9.8 MB −58%
route 13.3 MB 5.2 MB −61% 51.8 MB 12.6 MB −76%
hrr 79.3 MB 32.2 MB −59% 191 MB 83.7 MB −56%
us-2010 111 MB 83.1 MB −25% 305 MB 152 MB −50%
county 550 MB 278 MB −49% 1886 MB 456 MB −76%

Alloc — total bytes churned during build (GC pressure)

dataset init v4 init v5 Δ deep v4 deep v5 Δ
earthquakes 8.2 MB 1.2 MB −85% 19.6 MB 8.5 MB −57%
places 8.9 MB 1.2 MB −87% 18.0 MB 3.3 MB −82%
route 11.9 MB 12.6 MB +6% 93.6 MB 4.8 MB −95%
hrr 64.0 MB 6.4 MB −90% 269 MB 16.1 MB −94%
us-2010 169 MB 21.0 MB −88% 426 MB 90.9 MB −79%
county 1440 MB 419 MB −71% 3886 MB 506 MB −87%

Wall-clock — CPU is within ±35% of v4

dataset init v4 init v5 Δ deep v4 deep v5 Δ
earthquakes 2.3 ms 1.0 ms −57% 4.0 ms 3.9 ms
places 4.3 ms 1.4 ms −67% 5.0 ms 2.7 ms −46%
route 11.0 ms 9.4 ms −15% 32.6 ms 21.6 ms −34%
hrr 54.4 ms 56.6 ms +4% 130 ms 155 ms +19%
us-2010 105 ms 112 ms +7% 225 ms 265 ms +18%
county 938 ms 1011 ms +8% 2118 ms 2797 ms +32%

On the build users actually run (init), point-dominated workloads get 50–70% faster and coord-heavy workloads stay within ~10% of v4 (hrr +4%, us-2010 +7%, county +8%) — a small CPU cost in exchange for 45–75% less memory. The deep-drilldown stress test pays more CPU (up to +32% on county) for the same memory wins, but typical usage doesn't hit that path.

@mourner mourner added performance ai AI coding agents co-authored the code labels May 18, 2026
@mourner
mourner requested a review from a team as a code owner September 2, 2026 14:51
@mourner
mourner merged commit 41842fc into main Sep 3, 2026
6 checks passed
@mourner
mourner deleted the v5-new branch September 3, 2026 07:36
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

ai AI coding agents co-authored the code performance

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant