Jobs merge

[nkrah]

Summary

This PR completes the multi-job merge work and consolidates the split-run-merge workflow into a coherent user and developer API.

The branch started from the original job-splitting work and extends it in four main directions:

1. It implements campaign-level merge orchestration, including rehydration of split simulations, run-index mapping, merge planning/execution/finalization, and error propagation.
2. It refactors actor output handling so image/statistics outputs and ROOT-backed outputs can participate in split-run merge with clearer responsibilities and less ad hoc logic.
3. It introduces a higher-level local workflow API around split execution and merge, together with CLI support and updated documentation.
4. It tightens many contracts that had grown historically loose: path handling, data item/container behavior, source runtime configuration, per-run statistics, and split-compatibility checks.

Main Functional Changes

• Added campaign merge support through jobs_merge(...) and the merge manager/controller workflow.
• Introduced explicit merge lifecycle stages: planning, execution, finalization.
• Added MergeContext-based orchestration and coordinator-based execution, with a dedicated ROOT merge path for outputs that cannot be merged actor-by-actor because multiple outputs may share one ROOT file.
• Implemented merge support for:
  • image/statistics-like actor outputs via data-item/container logic
  • ROOT output via metadata-driven rehydration and chunk-wise streaming/concatenation logic
• Added support for per-run statistics storage and merge validation.
• Fixed cumulative-vs-per-run merge behavior so cumulative output is merged directly from children rather than reconstructed from per-run data when that is not guaranteed to exist.

Actor Output / Data Model Refactor

• Continued the actor-output refactor around DataItem and DataItemContainer.
• Clarified the transparency contract:
  • data items should feel like enriched payloads
  • containers may expose convenience behavior in simple cases, but explicit item-aware APIs remain authoritative
• Added module-level and inline documentation in dataitems.py.
• Tightened container APIs, including item selection semantics and safeguards around sparse/out-of-order set_data(...).
• Added iteration helpers over data items.
• Introduced better handling of derived data items and their primary dependencies.
• Added deprecation-aware bridges for old convenience access patterns where needed.

ROOT Output Work

• Refactored ROOT output closer to the formal actor-output architecture.
• Added RootDataItem-style metadata handling rather than relying on loose actor-level special cases.
• Implemented persisted ROOT metadata files to support rehydrated merge workflows.
• Added EventID and RunID remapping during merge where needed.
• Added support for shared ROOT files with multiple trees contributed by different actors.
• Separated internal ROOT merge machinery from user-facing contrib helpers by moving internal logic toward dedicated core helpers.
• Added warnings and validation around problematic output configurations, including absolute output paths in split campaigns.
• Improved handling of missing/empty child ROOT output and metadata.

Job Split / Run / Merge API

• Refactored split-side orchestration around a dedicated JobsSplitManager.
• Refactored local high-level workflow around SplitRunController / split-run-merge controller semantics.
• Extended Simulation.run(...) so local multi-job execution can be driven through the high-level API.
• Added support for attached local execution (detach=False) to improve robustness in CI and scripted workflows.
• Reworked folder/path conventions around:
  • simulation directory
  • campaign directory
  • job output folders
• Added better summaries and result objects for split and merge.

Serialization / Rehydration / Path Handling

• Improved simulation packaging and resolved-config persistence.
• Added package_simulation(...).
• Cleaned up input-file transfer and path rewriting behavior.
• Improved Windows compatibility in path rewriting by allowing relative-path requests to fall back to absolute paths when no relative representation exists across drives.
• Preserved symlink semantics better in rehydrated split jobs.
• Added stronger compatibility checks between a merge campaign and an explicitly provided live target simulation.

Sources / Runtime Configuration

• Refactored source API from source.n to source.number_of_primaries.
• Moved multi-thread scaling of number_of_primaries and activity into runtime adaptation rather than requiring the user to scale manually.
• Added deprecation handling for the old n API.
• Normalized runtime user-info preparation for Geant4 source initialization.
• Fixed fallout in several specialized source types and tests.
• Clarified the current contract for activity, including visible warnings in multi-threaded runs.
• Added split-compatibility checks for configurations that are currently not safe to split, such as dose uncertainty target stopping logic.

Statistics / Per-Run Output

• Extended the statistics actor to expose counts per run in addition to cumulative counts.
• Adapted test helpers such as assert_stats_json(...) to stop relying on magic container forwarding in places where that was no longer appropriate.
• Preserved the useful “data item feels like payload” behavior for statistics data items.

Digitizers / End-of-Simulation Hooks

• Cleaned up part of the digitizer end-of-simulation flow so Python-side postprocessing is routed through helper methods rather than mixed directly into trampolined EndSimulationAction() methods.
• Added comments/FIXMEs to make the intended inheritance contract clearer for future refactors.

CLI and Documentation

• Added/updated CLI coverage for:
  • split
  • run
  • merge
  • clean
  • status
• Aligned CLI naming and defaults with the refactored Python API.
• Added user-guide documentation for multi-job workflows:
  • local split-run-merge via sim.run(...)
  • deferred merge workflows
  • split policies
  • server-oriented workflow guidance
• Added a developer-guide chapter on multi-job architecture and workflow concepts.

Tests

This PR adds and/or significantly updates split/merge coverage across:

• merge context inspection
• local controller-driven split-run-merge
• dose/statistics per-run and cumulative consistency
• ROOT merge consistency
• shared-file multi-tree ROOT output
• missing/corrupted campaign data error handling
• path rewriting and archived inputs
• Windows and CI robustness adjustments

Notable new/expanded coverage includes:

• test100_deadtime_jobs_merge.py
• test113_jobs_merge_context.py
• test113_phsp_jobs_merge.py
• test114_split_run_merge_controller.py
• split/merge variants around existing geometry/source/actor tests

Follow-up / Known Deferred Items

A few areas were intentionally left for later PRs:

• terminology cleanup from merged/merge_after_simulation toward clearer cumulative naming
• broader refactor of region/material/path architecture not specific to multi-job logic
• deeper refactor of digitizer/ROOT runtime ownership
• possible future move to always-spawn subprocess behavior on Linux
• more formal provenance/recovery metadata for partial merge scenarios
• further consolidation of source config validation hooks
• future refactor of biasing/splitting architecture and related source/process design notes