Skip to content

Query WAL/Buffer + Timestamps - #55

Merged
adubovikov merged 63 commits into
masterfrom
buffer-fix
Oct 15, 2026
Merged

adubovikov merged 63 commits into
masterfrom
buffer-fix

Conversation

@lmangani

Copy link
Copy Markdown
Member

No description provided.

@CLAassistant

CLAassistant commented Feb 11, 2025 •

Copy link
Copy Markdown

CLA assistant check
Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you all sign our Contributor License Agreement before we can accept your contribution.
1 out of 2 committers have signed the CLA.

✅ adubovikov
❌ Lorenzo Mangani


Lorenzo Mangani seems not to be a GitHub user. You need a GitHub account to be able to sign the CLA. If you have already a GitHub account, please add the email address used for this commit to your account.
You have signed the CLA already but the status is still pending? Let us recheck it.

@lmangani
lmangani marked this pull request as ready for review February 11, 2025 13:37

@adubovikov adubovikov left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Summary

This PR adds in-memory buffer visibility in SQL queries (UNION with parquet), improves line-protocol timestamp parsing, serializes metadata writes, and ships a bundled query UI (index.html). The direction is useful for “recent data not yet flushed” workflows, but several correctness and operational issues should be fixed before merge.

Correctness

  1. Unsynchronized buffer reads during queries (query.js): query() reads this.buffer.buffers.get(type).rows and builds a VALUES temp table while ingest/flush paths in ParquetBufferManager mutate the same arrays (push, flush clearing rows). There is no lock or snapshot copy, so queries can see torn/partial batches or miss rows mid-flush. Please snapshot buffer rows under the same synchronization used for flush (or copy rows before building SQL).

  2. WHERE clause stripping is too broad (parseQuery): splitting on AND and dropping any fragment where cond.toLowerCase().includes('time') will remove legitimate predicates (e.g. columns containing time, or expressions like runtime > 0 if ever matched). It also breaks OR groupings. Prefer parsing/removing only the known time comparison(s) from the original regex match, not substring filtering.

  3. timeRange with open-ended bounds: for < / <= only, start is null; for > / >= only, end can be null. The generated SQL uses new Date(parsed.timeRange.start / 1000000) — in JS null / 1e6 becomes 0, so open-ended queries silently become “since epoch” instead of unbounded. Handle null start/end explicitly in SQL.

  4. Schema drift for line protocol: addLineProtocolBulk fixes the parquet schema from the first row when a measurement buffer is created; later batches can add new fields via allFields in one bulk call, but not after the buffer already exists with a narrower schema. Document or widen schema when new columns appear.

  5. Dead compact(type) on ParquetBufferManager: new method is never called (compaction still goes through CompactionManager), builds SQL with interpolated paths, and uses next_db_id / next_table_id paths that may not match live layout. Remove or wire it up; as-is it is misleading and risky if invoked later.

  6. QueryClient CLI example (query.js main()): still constructs new QueryClient() without a buffer manager; query() now throws No buffer manager available. Update or remove the example.

Security

  1. Query surface unchanged but UI increases exposure: /query remains unauthenticated; serving index.html on / makes arbitrary SQL easier for anyone who can reach the HTTP port. If that matches deployment intent, call it out in README; otherwise consider binding the UI to localhost or reusing any existing auth story.

  2. SQL concatenation (pre-existing, worsened by buffer VALUES): user-supplied fragments (parsed.columns, parsed.conditions, parsed.orderBy, parsed.limit) are still stitched into DuckDB SQL. Buffer string escaping only doubles single quotes; object keys become column names without quoting/validation. A malicious line-protocol field name could break out of the VALUES list. Validate/quote identifiers and keep treating /query as a trusted-admin endpoint.

  3. DoS: materializing the entire in-memory buffer into one VALUES (...), (...), ... statement can blow DuckDB memory/parse limits under load. Cap rows included from the buffer or use parameterized inserts / Arrow append APIs.

Tests

  • No automated coverage for buffer+parquet UNION, timestamp units (ns/us/ms/s), or WHERE parsing changes. Only loopback_test.sh comment changed.
  • Recommend at least a small Bun test (or script) that: ingests a few rows, queries before flush, asserts rows visible; and cases for time >= / time < bounds.

Process / hygiene

  • CLA check is still pending (license/cla).
  • Very large vendored UI (index.html, ~1.1k lines, ClickHouse-play-style). Please confirm license/attribution requirements and whether this belongs in-repo vs examples/.
  • CompactionManager is constructed with debug hard-coded true (// Always show compaction logs) instead of this.debug.

What looks good

  • Metadata write queue per type reduces concurrent rename races on metadata.json.
  • union_by_name=true on parquet reads helps heterogeneous files.
  • Line-protocol bulk normalization (null for missing fields, optional schema fields) is a step toward consistent parquet rows.
  • Per-line parse errors on /write no longer fail the whole batch.

Happy to re-review after buffer snapshotting, WHERE/time-range fixes, and basic tests are in place.

Snapshot in-memory rows for DuckDB queries, drain buffers atomically on
flush, widen line-protocol schemas when new fields appear, and fix open-ended
time bounds plus WHERE stripping. Add unit tests and cap buffer SQL size.
@adubovikov

Copy link
Copy Markdown
Member

Pushed follow-up commit 4d7dedc on buffer-fix addressing the earlier review:

  • Buffer/query race: flush() drains rows synchronously before I/O; failed flushes re-queue rows; queries use getBufferSnapshot() + capped VALUES temp table (MAX_BUFFER_QUERY_ROWS=5000).
  • WHERE/time: stripTimePredicates() removes only time <op> '…' fragments; buildTimestampWhereClause() handles open start/end without coercing null to epoch.
  • Line protocol: optional fields evolve via schemaFields when new columns appear in later batches.
  • Hygiene: removed unused ParquetBufferManager.compact(), compaction debug follows server flag, added bun test (query.test.js).

Still open: CLA check, broader /query auth/SQL-injection hardening (pre-existing), and running tests in CI (bun test).

@adubovikov
adubovikov merged commit 5e36e1e into master Oct 15, 2026
4 of 5 checks passed
@adubovikov
adubovikov deleted the buffer-fix branch October 15, 2026 22:41
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants