MW-1482: requisition-grain submission date, and Skipped as its own reporting outcome - #6
Open
denys1204 wants to merge 5 commits into
Open
MW-1482: requisition-grain submission date, and Skipped as its own reporting outcome#6denys1204 wants to merge 5 commits into
denys1204 wants to merge 5 commits into
Conversation
mart_stock_status had no submission date of its own, so the order-timeliness charts joined one on (facility, programme, period) and a requisition could inherit a date belonging to a different requisition for the same key. An INITIATED requisition showed up as submitted on time. The mart now carries submitted_date at requisition grain, taken from the requisition's own earliest SUBMITTED status change. 3,109 dates change on the Malawi replica, and emergency-order timeliness moves from 84% to 43% on time, which is the real figure. The join key is aliased to sub_requisition_id on purpose: the ClickHouse analyser keeps a join key's output column qualified when the name exists on both sides, which would emit a column literally called "li.requisition_id" and break this table's ORDER BY.
Two dimensions the dashboards had no way to offer: emergency, so regular and emergency requisitions stop being read as one population (2,651 of 169,412 follow a different process), and schedule_type through the shared macro so cadence is defined in one place across the marts.
The newest month in the data is structurally incomplete - about 7% of the usual facility coverage - so it read as a collapse in the report. The mart now carries in_complete_month from the shared completeness mart and the report filters on it. Both join keys are aliased (tp_program_name, tp_product_name) because the ClickHouse analyser keeps a join key's output column qualified when the name exists on both sides, which broke this chart once already.
Did-not-report was mixing four different things: nothing filed, filed and skipped, filed and still open, and filed under a status we do not recognise. A facility that legitimately skipped a period read as delinquent, and every programme in the data has periods_skippable set, so a skip is a legitimate outcome rather than a data error. reporting_status is now three-valued, Reported winning over Skipped, resolved through the SKIPPED status change for symmetry with the existing submitted CTE. On the Malawi replica HIV reads 1.8% skipped against 5.6% actually silent instead of one 7.4% block. reporting_rate is deliberately unchanged: it counts only Reported over all obligations, so a skip still reads as a miss and no existing figure moves. mart_non_reporting_facilities gains was_skipped. Those rows stay in the table, because its purpose is stock visibility and a skipped period is the same blind spot, but the flag lets accountability be separated from visibility. It also picks up an explicit alias on facility_id, without which the ClickHouse analyser emitted a column called "e.facility_id". mart_month_completeness reads the same column to decide which months every reporting chart draws. It counted a skip as zero before and still does; the comment now says it measures data arriving rather than compliance, so a future change to the rate policy does not drag it along by accident.
Month completeness compared a month's activity against the preceding three, which catches late data but not a month whose population changed. The weekly reporting schedule stops generating periods in April 2026, so May carried 6,072 obligations against 9,284 the month before while reported volume held - and the pooled reporting rate jumped 51.4% to 75.8% on the final point of every trend, on the latest complete month, which is the point everyone reads. A composition change was reading as improvement. A month must now also carry a comparable base: obligations for the reporting family, and the activity itself for stock and adjustments, where the two are the same measure so the test is a no-op there by construction. Measured: exactly one month changes state, May 2026, and the pooled trend then ends at 51.4% in line with the months before it. Stock and adjustments keep 33 of 34 complete months.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What this is
The core dbt marts for MW-1482 round 4, split from the Superset assets so the
data layer can be read on its own. The assets sit in a follow-up PR based on this
branch. Five commits, one per change.
Every requisition gets its own submission date.
mart_stock_statushad none,so the round-4 dashboard work had to join one in at the dataset layer on
(facility, programme, period) - a key coarser than a requisition, so a
requisition could inherit a date belonging to a different one and an INITIATED
requisition could land in an on-time bucket. The mart now carries
submitted_dateat requisition grain, from the requisition's own earliestSUBMITTED status change.
Reporting cadence on the requisition mart.
schedule_typethrough the sharedmacro, so cadence is defined once across the marts rather than derived per chart.
The completeness column reaches the logistics mart. The newest month is
structurally incomplete, around 7% of the usual facility coverage, so it reads as
a collapse. The mart now carries
in_complete_month; the chart-side filter thatuses it is in the asset PR.
Skipped becomes its own reporting outcome. Did-not-report was mixing four
things: nothing filed, filed and skipped, filed and still open, and filed under a
status we do not recognise.
reporting_statusis now three-valued, Reportedwinning over Skipped, resolved through the SKIPPED status change for symmetry
with the existing submitted CTE. On the dev replica HIV reads 1.8% skipped
against 5.6% actually silent instead of one 7.4% block.
reporting_rateis deliberately unchanged - it counts only Reported over allobligations, so a skip still reads as a miss there and that metric does not move.
mart_non_reporting_facilitiesgainswas_skipped; those rows stay in the tablebecause its purpose is stock visibility and a skipped period is the same blind
spot, but the flag lets accountability be told apart from visibility.
A changed cohort stops reading as improvement. Month completeness compared a
month's activity against the preceding three, which catches late data but not a
month whose population changed. The weekly reporting schedule stops generating
periods in April 2026, so May carried 6,072 obligations against 9,284 the month
before while reported volume held - and the pooled reporting rate jumped 51.4% to
75.8% on the final point of every trend, on the latest complete month, which is
the point everyone reads. A month must now also carry a comparable base:
obligations for the reporting family, and the activity itself for stock and
adjustments, where the two are the same measure so the test is a no-op there by
construction. Exactly one month changes state, May 2026.
One thing that moves the moment this merges
main's Reporting Rate by Program groups byreporting_status, so it gains athird Skipped band as soon as this branch lands, before the asset PR renames its
count or gives Skipped its own colour. Same for the Malawi twin. Nothing breaks;
it just looks half-finished until the asset PRs follow.
Three ClickHouse aliases that are not cosmetic
The analyser keeps a join key's output column qualified when the name exists on
both sides, which emits a column literally called
li.requisition_idore.facility_idand breaks the table's ORDER BY. It cost real debugging timethree times in this work, so each of those joins aliases its key and says why in
a comment.
Verification
dbt buildon this branch alone: PASS=184, ERROR=0.make verify-dbt17/0.The branch is deliberately checkable on its own: with only this merged, importing
main's Superset assets against the new marts leavesmake verify-supersetat6/0. That proves Superset still boots and the assets still import - it does not
check chart output, so read it as "nothing crashes", not "everything renders
right".
Deploying this
mart_stock_statusismaterialized='incremental'withon_schema_change='append_new_columns', so a normal run addssubmitted_dateasa column and leaves already-materialised rows without a value - the timeliness
charts would then read almost every requisition as having no submission date,
with no error anywhere. It needs
dbt run --full-refresh --select mart_stock_status+. The other four marts hereare
tablematerialisations and rebuild themselves.Not in this PR, on purpose
Almost no EPI requisition carries a SUBMITTED status change: they go INITIATED
straight to AUTHORIZED. That is verified against the source database and its
JaVers audit trail rather than inferred, and it means EPI reads as barely
reporting at all. Treating AUTHORIZED as a filing would move EPI from 0.2% to
4.3% reported across 18 charts and lands in core for every country, so it belongs
in its own ticket. Same for whether the reporting rate should drop skipped
periods from its denominator: measured at around 0.9pp on aggregates, and the
harm it would fix is already handled by
was_skipped.