perf: make the append-only elementary tables actually append-only

[haritamar]

Summary

dbt_run_results, elementary_test_results, dbt_source_freshness_results, data_monitoring_metrics and schema_columns_snapshot never get rows from their model body — it is an empty select ... where 1=0, and the actual rows are inserted by the on-run-end hooks. But each was configured materialized="incremental" with a unique_key, so every dbt run emitted a merge with a when matched then update clause against an empty source:

merge into dbt_run_results DBT_INTERNAL_DEST
using (<0 rows>) DBT_INTERNAL_SOURCE
    on DBT_INTERNAL_SOURCE.model_execution_id = DBT_INTERNAL_DEST.model_execution_id
when matched then update set ...
when not matched then insert ...

The when matched clause plus an ON predicate with no partition filter makes BigQuery scan the entire target table — cost grows linearly with history and is paid on every run, for zero rows written. This is the most expensive statement in some users' projects (reported by an OSS user; also option 3 in elementary-data/elementary#1959, which was closed as stale).

Per model:

-        unique_key="<key>",
         meta={
+            "dedup_by_column": "<key>",
             ...
         },
-        incremental_strategy=elementary.get_default_incremental_strategy(),
+        incremental_strategy=elementary.get_append_only_incremental_strategy(),

Without a unique_key, dbt generates on (FALSE) ... when not matched then insert, which doesn't read the target at all (same as test_result_rows today). Nothing is lost: uniqueness was never enforced by the materialization (rows bypass it via direct inserts) and dedup happens at read/dump time via meta.dedup_by_column, which is now set on all five models.

One behavior change to be aware of: dump_table dedups by meta.dedup_by_column, defaulting to unique_id. For dbt_source_freshness_results (the only newly-annotated model that has both unique_id and generated_at) dumps now dedup by source_freshness_execution_id, i.e. every freshness execution is kept rather than the latest per source. The other models have no unique_id/generated_at, so no dedup was happening and none starts.

The snapshot-style artifact tables (dbt_models, dbt_tests, dbt_sources, ...) are deliberately untouched: they are written with delete+insert / full replace and are bounded by project size rather than history, so the merge there is cheap and its upsert semantics are meaningful.

Elementary Cloud's synced-schema loader must read the upsert key from meta.dedup_by_column (elementary-data/elementary-internal#7249) and be deployed before this is released, otherwise these tables fall back to a time-range replace on sync.

Note for existing users: tables also need a --full-refresh to pick up the created_at partitioning added in #939/#972.

Link to Devin session: https://app.devin.ai/sessions/28a642e68cf64b94950aea4e21610310
Requested by: @haritamar

Summary by CodeRabbit

• Bug Fixes
  • Improved incremental processing across monitoring, snapshot, freshness, test, and run-result data to reliably append new records without replacing existing data.
  • Added record-level deduplication to maintain accurate results while preserving historical information.

[devin-ai-integration]

🤖 Devin AI Engineer

I'll be helping with this pull request! Here's what you should know:

✅ I will automatically:

• Address comments on this PR. Add '(aside)' to your comment to have me ignore it.
• Look at CI failures and help fix them

Note: I can only respond to comments from users who have write access to this repository.

⚙️ Control Options:

• Disable automatic comment, CI, and merge conflict monitoring

[github-actions]

👋 @haritamar
Thank you for raising your pull request.
Please make sure to add tests and document all user-facing changes.
You can do this by editing the docs files in the elementary repository.

[coderabbitai]

📝 Walkthrough

Walkthrough

Five incremental models remove explicit unique keys, add deduplication metadata, and use Elementary’s append-only incremental strategy.

Changes

Append-only incremental model configuration

Layer / File(s)	Summary
Configure append-only incremental loading models/edr/dbt_artifacts/dbt_run_results.sql, models/edr/data_monitoring/data_monitoring/data_monitoring_metrics.sql, models/edr/data_monitoring/schema_changes/schema_columns_snapshot.sql, models/edr/run_results/dbt_source_freshness_results.sql, models/edr/run_results/elementary_test_results.sql	The models remove explicit unique keys, define deduplication columns through metadata, and use elementary.get_append_only_incremental_strategy().

Estimated code review effort: 1 (Trivial) | ~2 minutes

🚥 Pre-merge checks | ✅ 5

✅ Passed checks (5 passed)

Check name	Status	Explanation
Docstring Coverage	✅ Passed	No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Description Check	✅ Passed	Check skipped - CodeRabbit’s high-level summary is enabled.
Title check	✅ Passed	The title clearly summarizes the main change: configuring Elementary tables to use append-only behavior.

✨ Finishing Touches

🧪 Generate unit tests (beta)

• Create PR with unit tests
• Commit unit tests in branch devin/1786391046-dbt-run-results-append-only

---

_{Comment @coderabbitai help to get the list of available commands.}

[haritamar]

I think you mentioned we should also do the same change to other append only models?

(I'm wondering if it can also be in other artifacts as anyway we only modify them in the post hook / on run end hook)

[devin-ai-integration]

Yes — no elementary table's rows come from the model body (all writes are hook-side inserts/replaces), so the merge is dead weight everywhere. But the tables split into two groups:

Append-only history tables — elementary_test_results, dbt_source_freshness_results, data_monitoring_metrics, schema_columns_snapshot, plus test_result_rows (already has no unique_key). Same cost profile as dbt_run_results (they grow with history, so the scan gets more expensive over time). Blocker: none of them set meta.dedup_by_column, so dropping unique_key makes sync_elementary_schema flip them from upsert to replace_time_range — exactly the case you flagged with customers pruning old rows. Adding the meta is not free either: dump_table uses meta.dedup_by_column for its dedup and it currently defaults to unique_id, so setting it changes what a dedup=true dump collapses (e.g. elementary_test_results would go from deduping by unique_id to id) — needs to be checked per table rather than done blind.

Snapshot artifact tables — dbt_models, dbt_tests, dbt_sources, dbt_columns, etc. Written via delete+insert / full replace, so they're bounded by project size, not history — the wasted merge is small and constant. Removing unique_key there also flips internal to replace_time_range, which for full-replace semantics is arguably wrong.

So I'd keep this PR to dbt_run_results (the actual reported cost) and do the append-only group as a follow-up where each table gets the right meta.dedup_by_column plus a dump-behavior check. Want me to open that follow-up now?