New approach for partial dispatch

[tsmbland]

Description

The current approach for partial dispatch (dispatch on intermediate systems during investment, which we perform successively after investments in each market to generate demands for upstream markets) is to run dispatch on the cumulative system. For example, consider:

A (SED) -> B (SED) -> C (SVD) -> Demand

We begin by doing investments for C, then we dispatch the system passing in the exogenous C demands and only balancing C. This generates demands for B. We then do investments for B, then we dispatch the system again passing in the exogenous demands for C, this time balancing C and B. Then we invest for A and dispatch balancing C, B and A.

In the new approach, we would do the following, in order:

• invest for C
• dispatch on exogenous demands for C, balancing only C
• invest for B
• dispatch on internal demands for B, balancing only B
• invest for A
• dispatch on internal demands for A, balancing only A

Each round we only dispatch for a single commodity (i.e. the commodity we've just invested in), rather than re-dispatching every commodity we've already visited. This obviously avoids wasted computation, but it's also helps with the circularity problem, as a big problem with circularities using the previous partial dispatch approach was that cumulative systems could be unresolvable (hence the need to do "cycle balancing", which I'm moving away from).

Circularities are currently temporarily broken (as of #1539), but I have a plan for fixing them which follows on from this.

One implication is that this could lead to excess capacity for processes with multiple output commodities (see two_outputs), which was previously "optimised away" in the cumulative approach. Not too concerned about this as I think the current approach for dealing with multiple outputs is very imperfect anyway and the previous approach was just a plaster.

There may also be some unpredictable effects of this, for example see missing_commodity. Overall capacities haven't changed, but the tranche size for BIOPRO has changed considerably. This appears to be due to changing shape of the BIOPRD demand profile seen at the BIOPRD investment stage. I asked Claude to explain for me exactly what's going on here, as it's a bit complicated, and here's what it said:

Installed capacity and tranche size are two different aggregations of the same demand profile: capacity is set by the peak time-slice selection (the binding season sizes the plant), while tranche size is set by the annual total (calculate_candidate_asset_capacity_scale sums demand over all selections). So if the shape of the propagated demand changes but its peak doesn't, tranche size moves while capacity stays put — which is exactly what we see (capacity ≈ unchanged, num_tranches up).

The shape change is specific to this model because RSHEAT is contested by three competing technologies (gas boiler, heat pump, and the new biomass boiler), and the biomass chain has no base-year capacity so its price is generated from scratch. The new partial-dispatch approach changes the internal-vs-external pricing in the post-heat dispatch, which nudges the seasonal split between those competing heat technologies. That reshapes RBIOBL's output across seasons, which propagates down the new BIOPEL → BIOPRD chain, changing the annual total (→ BIOPRO tranche size) without changing the peak season (→ BIOPRO capacity).

This is expected behaviour, not a bug: tranche size is a granularity heuristic that is legitimately sensitive to demand shape, whereas installed capacity is peak-driven. Models where each demand has a single dominant supplier keep the same split and are unaffected.

Maybe that makes sense? I think actually the root cause is that in the previous approach we were effectively using new shadow prices for all downstream commodities (as we were balancing them in the cumulative dispatch), whereas now we're consistently using previous year shadow prices. I think that's better?

Fixes # (issue)

Type of change

• Bug fix (non-breaking change to fix an issue)
• New feature (non-breaking change to add functionality)
• Refactoring (non-breaking, non-functional change to improve maintainability)
• Optimization (non-breaking change to speed up the code)
• Breaking change (whatever its nature)
• Documentation (improve or add documentation)

Key checklist

• All tests pass: $ cargo test
• The documentation builds and looks OK: $ cargo doc
• Update release notes for the latest release if this PR adds a new feature or fixes a bug
  present in the previous release

Further checks

• Code is commented, particularly in hard-to-understand areas
• Tests added that prove fix is effective or that feature works

[codecov]

Codecov Report

❌ Patch coverage is 98.38710% with 1 line in your changes missing coverage. Please review.
✅ Project coverage is 90.60%. Comparing base (892ee42) to head (769d469).
⚠️ Report is 1 commits behind head on circularity.

Files with missing lines	Patch %	Lines
src/simulation/investment.rs	80.00%	0 Missing and 1 partial ⚠️

Additional details and impacted files

@@               Coverage Diff               @@
##           circularity    #1542      +/-   ##
===============================================
- Coverage        90.63%   90.60%   -0.03%     
===============================================
  Files               61       62       +1     
  Lines             8870     8860      -10     
  Branches          8870     8860      -10     
===============================================
- Hits              8039     8028      -11     
  Misses             516      516              
- Partials           315      316       +1     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[Copilot]

🔵 Needs a closer look

It changes core investment/dispatch semantics and updates many golden outputs, so it needs careful domain validation beyond automated review confidence.

Pull request overview

Updates the simulation’s partial-dispatch workflow so that, during sequential investment, dispatch is re-run only for the currently-invested market (rather than cumulatively re-dispatching all downstream markets), with demand passed explicitly into the dispatch optimiser.

Changes:

• Introduces a shared simulation::demand module and threads an AllDemandMap into DispatchRun / dispatch constraints so partial dispatch can use internal (propagated) demands.
• Adjusts investment-time dispatch to balance only the current market set, using internal demand propagation between investment stages.
• Updates multiple golden-test CSV outputs to reflect the new dispatch/investment behaviour.

File summaries

File	Description
tests/data/two_outputs/commodity_prices.csv	Updated expected prices for the two_outputs fixture under the new partial-dispatch behaviour.
tests/data/two_outputs/asset_capacities.csv	Updated expected capacities/tranches for two_outputs.
tests/data/simple/debug_solver.csv	Updated expected objective values for intermediate dispatch runs.
tests/data/simple/debug_commodity_balance_duals.csv	Updated expected commodity balance duals for intermediate dispatch runs.
tests/data/simple/debug_appraisal_results.csv	Updated expected appraisal outputs due to changed dispatch inputs/duals.
tests/data/simple/debug_appraisal_results_time_slices.csv	Updated expected appraisal time-slice outputs.
tests/data/simple_tranche_size/commodity_flows.csv	Updated expected flows (minor floating-point differences).
tests/data/simple_tranche_size/asset_capacities.csv	Updated expected capacities (minor floating-point differences).
tests/data/missing_commodity/commodity_prices.csv	Updated expected prices for missing_commodity.
tests/data/missing_commodity/commodity_flows.csv	Updated expected flows for missing_commodity.
tests/data/missing_commodity/asset_capacities.csv	Updated expected capacities/tranches for missing_commodity.
tests/data/circularity/commodity_prices.csv	Updated expected prices for circularity.
tests/data/circularity/commodity_flows.csv	Updated expected flows for circularity.
tests/data/circularity/asset_capacities.csv	Updated expected capacities/tranches for circularity.
tests/data/circularity_npv/commodity_prices.csv	Updated expected prices for circularity_npv.
tests/data/circularity_npv/commodity_flows.csv	Updated expected flows for circularity_npv.
tests/data/circularity_npv/asset_capacities.csv	Updated expected capacities/tranches for circularity_npv.
src/simulation/optimisation/constraints.rs	Makes commodity-balance lower bounds come from an explicit demand map (supports internal demand in partial dispatch).
src/simulation/optimisation.rs	Threads AllDemandMap through DispatchRun and into constraint construction.
src/simulation/market.rs	Updates dispatch calls used in cycle selection to provide current demand map.
src/simulation/investment.rs	Switches post-investment dispatch to balance only the current market set and uses internal demand propagation.
src/simulation/demand.rs	New module defining demand map types and collecting preset demands for a year.
src/simulation.rs	Collects preset demands once per year and passes them into final dispatch runs.
benches/assets.rs	Updates benchmark setup dispatch calls to provide demand maps.

Review details

• Files reviewed: 26/26 changed files
• Comments generated: 1
• Review effort level: Lite

---

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.