Repository navigation
Query WAL/Buffer + Timestamps - #55
Conversation
* Create loopback_test.sh * Update loopback_test.sh
|
Lorenzo Mangani seems not to be a GitHub user. You need a GitHub account to be able to sign the CLA. If you have already a GitHub account, please add the email address used for this commit to your account. You have signed the CLA already but the status is still pending? Let us recheck it. |
adubovikov
left a comment
There was a problem hiding this comment.
Summary
This PR adds in-memory buffer visibility in SQL queries (UNION with parquet), improves line-protocol timestamp parsing, serializes metadata writes, and ships a bundled query UI (index.html). The direction is useful for “recent data not yet flushed” workflows, but several correctness and operational issues should be fixed before merge.
Correctness
-
Unsynchronized buffer reads during queries (
query.js):query()readsthis.buffer.buffers.get(type).rowsand builds aVALUEStemp table while ingest/flush paths inParquetBufferManagermutate the same arrays (push,flushclearing rows). There is no lock or snapshot copy, so queries can see torn/partial batches or miss rows mid-flush. Please snapshot buffer rows under the same synchronization used for flush (or copy rows before building SQL). -
WHERE clause stripping is too broad (
parseQuery): splitting onANDand dropping any fragment wherecond.toLowerCase().includes('time')will remove legitimate predicates (e.g. columns containingtime, or expressions likeruntime > 0if ever matched). It also breaksORgroupings. Prefer parsing/removing only the knowntimecomparison(s) from the original regex match, not substring filtering. -
timeRangewith open-ended bounds: for</<=only,startisnull; for>/>=only,endcan benull. The generated SQL usesnew Date(parsed.timeRange.start / 1000000)— in JSnull / 1e6becomes0, so open-ended queries silently become “since epoch” instead of unbounded. Handlenullstart/end explicitly in SQL. -
Schema drift for line protocol:
addLineProtocolBulkfixes the parquet schema from the first row when a measurement buffer is created; later batches can add new fields viaallFieldsin one bulk call, but not after the buffer already exists with a narrower schema. Document or widen schema when new columns appear. -
Dead
compact(type)onParquetBufferManager: new method is never called (compaction still goes throughCompactionManager), builds SQL with interpolated paths, and usesnext_db_id/next_table_idpaths that may not match live layout. Remove or wire it up; as-is it is misleading and risky if invoked later. -
QueryClientCLI example (query.jsmain()): still constructsnew QueryClient()without a buffer manager;query()now throwsNo buffer manager available. Update or remove the example.
Security
-
Query surface unchanged but UI increases exposure:
/queryremains unauthenticated; servingindex.htmlon/makes arbitrary SQL easier for anyone who can reach the HTTP port. If that matches deployment intent, call it out in README; otherwise consider binding the UI to localhost or reusing any existing auth story. -
SQL concatenation (pre-existing, worsened by buffer
VALUES): user-supplied fragments (parsed.columns,parsed.conditions,parsed.orderBy,parsed.limit) are still stitched into DuckDB SQL. Buffer string escaping only doubles single quotes; object keys become column names without quoting/validation. A malicious line-protocol field name could break out of theVALUESlist. Validate/quote identifiers and keep treating/queryas a trusted-admin endpoint. -
DoS: materializing the entire in-memory buffer into one
VALUES (...), (...), ...statement can blow DuckDB memory/parse limits under load. Cap rows included from the buffer or use parameterized inserts / Arrow append APIs.
Tests
- No automated coverage for buffer+parquet UNION, timestamp units (ns/us/ms/s), or WHERE parsing changes. Only
loopback_test.shcomment changed. - Recommend at least a small Bun test (or script) that: ingests a few rows, queries before flush, asserts rows visible; and cases for
time >=/time <bounds.
Process / hygiene
- CLA check is still pending (
license/cla). - Very large vendored UI (
index.html, ~1.1k lines, ClickHouse-play-style). Please confirm license/attribution requirements and whether this belongs in-repo vsexamples/. CompactionManageris constructed withdebughard-codedtrue(// Always show compaction logs) instead ofthis.debug.
What looks good
- Metadata write queue per type reduces concurrent rename races on
metadata.json. union_by_name=trueon parquet reads helps heterogeneous files.- Line-protocol bulk normalization (
nullfor missing fields, optional schema fields) is a step toward consistent parquet rows. - Per-line parse errors on
/writeno longer fail the whole batch.
Happy to re-review after buffer snapshotting, WHERE/time-range fixes, and basic tests are in place.
Snapshot in-memory rows for DuckDB queries, drain buffers atomically on flush, widen line-protocol schemas when new fields appear, and fix open-ended time bounds plus WHERE stripping. Add unit tests and cap buffer SQL size.
|
Pushed follow-up commit
Still open: CLA check, broader |
No description provided.