Fix Redshift↔Databricks row-hash serialization

[ameersalman-db]

Changes

What does this PR do?

Fixes four pre-existing correctness bugs in the row-hash reconcile path for Redshift → Databricks reconciles, and extends the shared per-type transform table so the affected types serialize identically across the two engines.

None of these are specific to any new feature — each one corrupts the existing row-hash compare pipeline (bugs 1–3) or its result reporting (bug 4) on real Redshift schemas. They surfaced while integrating a Redshift source (see the Redshift connector work in #2339), but they live in shared reconcile code and affect every Redshift → Databricks row/data/all reconcile today. This PR is intentionally scoped to just these fixes so it can be reviewed on its own; a follow-up fingerprint pre-check PR builds on top of it.

Relevant implementation details

Bugs 1–3 add or repair a dialect handler in the shared per-type transform table _DATATYPE_TRANSFORM_MAPPING, which lives in reconcile/query_builder/column_transformer.py (the ColumnTransformer service introduced by #2601). The row-hash path resolves each column's serializer through one shared helper, get_transform_for_type(datatype, source, counterpart), so there is a single definition of "how is this column type turned into text before hashing", consumed by RuleBasedColumnTransformer._normalize_by_type.

#	Fix	What was broken
1	TIMESTAMP / TIMESTAMPTZ handler on the Databricks (target) dialect	The target had no override and fell through to TRIM(CAST(ts AS string)). Spark emits a variable-length fraction (dropped entirely at zero microseconds → '2023-10-02 18:08:43') while the Redshift source emits fixed 6-digit microseconds ('…43.000000'). The byte-width drift made every identical timestamp row false-mismatch. Fixed with COALESCE(DATE_FORMAT(_, 'yyyy-MM-dd HH:mm:ss.SSSSSS'), '_null_recon_'); Redshift TIMESTAMPTZ is pinned to UTC via AT TIME ZONE 'UTC' so it does not depend on the Redshift session TIMEZONE. Counterpart-gated (see below): only selected when the source is Redshift, so non-Redshift → Databricks reconciles keep their previous serialization.
2	BOOLEAN handler on the Redshift (source) dialect	Redshift had no boolean override and no dialect default, so BOOLEAN fell through to the universal TRIM(...), which Redshift rejects at output-schema resolution (before any rows are read) with function pg_catalog.btrim(boolean) does not exist. Any schema with a single boolean column crashed the reconcile. Fixed with COALESCE(CASE WHEN col THEN 'true' WHEN NOT col THEN 'false' ELSE NULL END, '_null_recon_'), matching Spark's cast(boolean AS string).
3	Floating-point (DOUBLE/float8 and FLOAT/real/float4) handlers on the Redshift and Databricks dialects	Neither floating-point type had an override on either side, so both fell through to TRIM(CAST(col AS string)) — but Redshift renders full precision (0.28999999999999998) while Spark emits the shortest round-trip (0.29), so every float-bearing row false-mismatched. Fixed by pinning both sides to a fixed-scale COALESCE(CAST(CAST(_ AS DECIMAL(38,10)) AS STRING/VARCHAR), '_null_recon_'). FLOAT reuses the same DECIMAL(38,10) scale as DOUBLE on purpose: sqlglot resolves Redshift float→DOUBLE but Databricks float→FLOAT, so a mismatched scale would diverge a float/double column pair. NaN / ±Infinity, and any finite magnitude ≥ 1e28 (which overflows DECIMAL(38,10) and hard-fails — Redshift aborts the whole recon, Spark returns NULL), bypass the DECIMAL cast and fall back to a direct string cast so the recon does not crash. Residual (documented, not fixed): that native fallback is not byte-identical across engines for a finite magnitude ≥ 1e28 — Spark renders 1.0E30 where Redshift renders 1e+30 (empirically confirmed on DBR 18 + live Redshift), so those rows false-positive (over-report; never hide a real diff). NaN / ±Infinity render identically on both engines (NaN / Infinity / -Infinity) and do not diverge. A finite magnitude ≥ 1e28 is vanishingly rare (and a double carries only ~15–16 significant digits there), so this is accepted as a fail-safe edge — see the reviewer note below.
4	success_count inflation in trigger_recon_service.verify_successful_reconciliation	The succeeded-table count was computed as total − exceptions + mismatches, adding mismatched tables instead of subtracting them, so any run with a mismatch reported more succeeded tables than were reconciled (success_count > total_count). Pre-existing since #2259; unrelated to serialization but grouped here as another pre-existing correctness fix. Fixed to total − exceptions − mismatches. The existing tests only asserted the log wording, never the numbers — which is why it slipped through; a regression test now asserts the counts.

Counterpart gating: get_transform_for_type(datatype, source, counterpart) gates the _COUNTERPART_PINNED_TYPES handlers — DOUBLE, FLOAT, TIMESTAMP, TIMESTAMPTZ — whose pinned forms (fixed-scale DECIMAL(38,10); …HH:mm:ss.SSSSSS microseconds) are byte-identical only when both engines pin (currently Redshift ↔ Databricks). RuleBasedColumnTransformer already holds both the source and target ReconcileLayer, so when it normalizes one layer it passes the other layer's dialect as the counterpart. When the counterpart does not pin (e.g. a BigQuery/Snowflake/Oracle/TSQL source into a Databricks target), the pinned types fall back to the universal default so the two sides still agree — so this PR does not change serialization for any non-Redshift → Databricks reconcile.

Caveats/things to watch out for when reviewing:

• Timestamp determinism relies on the reconcile cluster session being UTC (the Databricks default). The Databricks target renders timestamps via DATE_FORMAT, which reads spark.sql.session.timeZone; the Redshift source renders in UTC. This PR does not force-pin the Spark session timezone — that is left to the operator. On a non-UTC session, every timestamp / timestamptz row diverges and false-mismatches. Decision for reviewers: enforce this at runtime (assert / pin the session TZ), or accept it as a documented hard prerequisite (recon clusters run in UTC)? Nothing enforces it today.

• Floating-point precision floor — decision for reviewers. Making the two engines agree requires pinning floats to one fixed decimal representation, so some scale must be hardcoded. DECIMAL(38,10) (10 fractional digits) is inherited from the hand-written Teradata recon fixture (tests/integration/reconcile/conftest.py), not derived from a precision requirement. Consequence: two values differing only beyond the 10th fractional digit (e.g. 0.12345678901 vs 0.12345678902) hash identically and are reported as a MATCH. Fine for money / most data; a blind spot for high-precision data (scientific, sub-1e-10 financial). Confirm 10 digits is acceptable, or specify a different scale.

• The floating-point fix changes the serialized form of DOUBLE and FLOAT columns on both engines (to DECIMAL(38,10)). This is a behavior change to the row hash for those columns — intentional, since the previous behavior always false-mismatched them.

• ≥ 1e28 overflow residual — documented limitation, both paths, fail-safe. The overflow guard routes a finite DOUBLE/FLOAT magnitude ≥ 1e28 (which overflows DECIMAL(38,10)), plus NaN/±Infinity, to a native string cast so the recon doesn't crash. That native cast is byte-identical across engines for NaN/±Infinity (NaN / Infinity / -Infinity on both) but not for a finite ≥ 1e28: Spark emits 1.0E30, Redshift 1e+30 (confirmed on DBR 18 + live Redshift). Effect: such rows false-positive (over-report), never a false MATCH. It is not limited to the fingerprint pre-check — the pin lives in the shared get_transform_for_type, gated by dialect (Redshift ↔ Databricks), so the same residual applies to the legacy row-hash path. It did not exist before this PR (previously all doubles false-mismatched under the universal default); ≥ 1e28 never occurs in practical recon data, so we accept it as a fail-safe edge. If we ever need it closed: route the overflow branch through CAST(_ AS DECIMAL(38,0)) on both engines (plain integer, byte-identical), which narrows the divergence window to ≥ 1e38.

Linked issues

Relates to the Redshift connector work in #2339 (these bugs sit in that integration path).
Bug 4 (success_count) is pre-existing since #2259.

Functionality

• internal row-hash correctness fix — no CLI or config surface change
• added new CLI command
• modified existing command

Tests

• added unit tests in tests/unit/reconcile/query_builder/test_expression_generator.py:
  exact-rendered-SQL assertions for each handler (TIMESTAMP/TIMESTAMPTZ/BOOLEAN/DOUBLE
  on both dialects), a source↔target byte-alignment check for timestamps, tests that the
  DOUBLE/FLOAT/timestamp pins are only emitted when the counterpart also pins, a
  FLOAT/real/float4 regression test (pins like DOUBLE, gated on the counterpart), and
  a ≥ 1e28 overflow-guard test (routes to the native cast instead of crashing).
• added a success_count regression test in tests/unit/reconcile/test_verify_reconcile.py
  asserting the summary counts (not just the log wording) so an inflated count fails CI.
• existing reconcile/ unit suite passes (row-hash query-builder tests unchanged in behavior
  for the dialects/types they cover).

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 71.33%. Comparing base (fec265c) to head (0a76410).

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #2664      +/-   ##
==========================================
+ Coverage   71.31%   71.33%   +0.02%     
==========================================
  Files         112      112              
  Lines       10122    10131       +9     
  Branches     1111     1114       +3     
==========================================
+ Hits         7218     7227       +9     
  Misses       2690     2690              
  Partials      214      214              

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[github-actions]

✅ 182/182 passed, 9 flaky, 2 skipped, 2h16m48s total

Flaky tests:

• 🤪 test_installs_and_runs_local_bladebridge (14.66s)
• 🤪 test_installs_and_runs_pypi_bladebridge (23.419s)
• 🤪 test_recon_for_report_type_is_data (49.755s)
• 🤪 test_transpiles_informatica_to_sparksql_non_interactive[True] (21.631s)
• 🤪 test_transpiles_informatica_to_sparksql (24.02s)
• 🤪 test_transpiles_informatica_to_sparksql_non_interactive[False] (3.974s)
• 🤪 test_transpile_teradata_sql_non_interactive[True] (5.864s)
• 🤪 test_transpile_teradata_sql (26.074s)
• 🤪 test_transpile_teradata_sql_non_interactive[False] (5.642s)

_{Running from acceptance #5473}