Repository navigation
Conversation
egparedes
marked this pull request as ready for review
July 30, 2026 17:58
egparedes
force-pushed
the
otf-split-3-observability
branch
from
July 30, 2026 17:58
e3d68d2 to
1c59da0
Compare
egparedes
force-pushed
the
otf-split-3-observability
branch
from
July 31, 2026 16:15
1c59da0 to
292deb1
Compare
egparedes
force-pushed
the
otf-split-3-observability
branch
from
August 17, 2026 15:07
292deb1 to
b79be9b
Compare
egparedes
force-pushed
the
otf-split-3-observability
branch
from
August 20, 2026 16:17
b79be9b to
573e320
Compare
egparedes
force-pushed
the
otf-split-3-observability
branch
from
August 20, 2026 16:52
573e320 to
5e0b9c1
Compare
egparedes
force-pushed
the
otf-split-3-observability
branch
from
August 20, 2026 20:28
5e0b9c1 to
05ddb45
Compare
egparedes
force-pushed
the
otf-split-3-observability
branch
from
August 26, 2026 11:05
05ddb45 to
658a51e
Compare
egparedes
force-pushed
the
otf-split-3-observability
branch
from
August 28, 2026 13:06
658a51e to
4b16df7
Compare
egparedes
force-pushed
the
otf-split-3-observability
branch
from
August 28, 2026 13:09
4b16df7 to
aa3bb68
Compare
egparedes
changed the title
egparedes
force-pushed
the
otf-split-3-observability
branch
from
September 1, 2026 18:30
aa3bb68 to
e53ca53
Compare
egparedes
force-pushed
the
otf-split-3-observability
branch
from
September 9, 2026 16:34
e53ca53 to
3799397
Compare
egparedes
force-pushed
the
otf-split-3-observability
branch
from
September 9, 2026 16:51
3799397 to
c95d6e7
Compare
egparedes
force-pushed
the
otf-split-3-observability
branch
from
September 10, 2026 11:54
c95d6e7 to
43b76e2
Compare
egparedes
force-pushed
the
otf-split-3-observability
branch
from
September 17, 2026 15:21
7556515 to
7d5d0d8
Compare
egparedes
force-pushed
the
otf-split-3-observability
branch
from
September 23, 2026 19:28
7d5d0d8 to
aade78d
Compare
egparedes
force-pushed
the
otf-split-3-observability
branch
from
September 26, 2026 07:49
aade78d to
636bc18
Compare
egparedes
force-pushed
the
otf-split-3-observability
branch
from
September 30, 2026 06:15
636bc18 to
644a54c
Compare
egparedes
force-pushed
the
otf-split-3-observability
branch
from
October 1, 2026 13:34
644a54c to
9614b82
Compare
egparedes
force-pushed
the
otf-split-3-observability
branch
from
October 1, 2026 14:58
9614b82 to
165bb51
Compare
egparedes
force-pushed
the
otf-split-3-observability
branch
from
October 1, 2026 15:32
165bb51 to
a050a91
Compare
…dace __sdfg__ reach-in Give the toolchain a sanctioned way to observe and partially run its pipelines, and remove the one consumer that had to reach into them: - `otf.workflow.stage_hook(name, artifact)`: an event hook emitted after each step by the two named pipelines (`backend.Transforms` and `recipes.OTFCompileWorkflow`, via the generic `MultiWorkflow` / `NamedStepSequence` loops) and by `Toolchain.translate`. Artifacts are passed opaquely and never formatted at the emit site; with nothing subscribed the hook body is empty, so emission is an empty callback loop (~160 ns). - `GT4PY_DUMP_STAGES=<dir>` (`config.DUMP_STAGES`) plus the `instrumentation.stage_dump` subscriber, which writes every stage artifact to `<dir>/<program>/<NNN>_<step>.<ext>`. Registered at import time when the variable is set, so a plain program run dumps its stages without any user code; `enable()` / `disable()` cover programmatic use. Exclusive-create with an index bump keeps concurrent compilation workers and same-named programs from overwriting each other. - `Toolchain.translate(definition, compile_time_args)`: the sanctioned partial run (frontend + translation only). It narrows to the standard `OTFCompileWorkflow` shape and raises a clear error on a monolithic backend. Per-call step options are deliberately not offered; a caller needing a variant step builds a variant pipeline with `dataclasses.replace`. - The dace `__sdfg__` reach-in is gone. It duck-typed through `backend.backend.translation`, unwrapped the `CachedStep` by hand, and used `object.__setattr__` to write into the *in-memory-cached* `past_to_itir` stage and its frozen `CompileTimeArgs`. It is replaced by a dace-owned translate-only toolchain variant built with `dataclasses.replace`, plus `Toolchain.translate`. That last change also fixes a real bug: because the mutated stage was the cached one, a second `__sdfg__` call re-transformed an already-transformed program, and any other consumer of that cache entry saw args whose `offset_provider` had been replaced by offset-provider *types*, losing the runtime connectivity tables. A regression test covers it. Removing the reach-in drops the last non-doc consumer of the workflow mixins' `.replace`, which unblocks the pipeline simplification. `GT4PY_DUMP_STAGES` is new opt-in functionality; nothing else changes behavior, and no fingerprinted dataclass gains a field, so no cache keys rotate. Claude-Session: https://claude.ai/code/session_01R8zRtFMhdJ8c96XJYCXkRk
egparedes
force-pushed
the
otf-split-3-observability
branch
from
October 19, 2026 15:18
a050a91 to
fd93a98
Compare
…ability Bring the reworked bottom of the stack (#2741 backend structure, #2956 embedded backends) under the stage-observability PR and adapt it: - `translate` moves onto the internal `Toolchain` (it needs the global frontend), reading the translation step from `backend.compiler`; errors name the backend and its compiler type. - dace `__sdfg__`: keeps `is_dace_backend` and the reach-in fix; `_translation_only_toolchain` becomes `_translation_only_backend`, which replaces the translation step inside `CompiledBackend.compiler`, used as `Toolchain(variant).translate(...)`. - Tests: the partial-run tests construct `Toolchain(CompiledBackend(...))`; the stage-hook and stage-dump tests run on the compiled backends only and use `gtx_typing.Backend`; gtfn/dace `translate` tests go through the internal `Toolchain`; the dace cache regression test reads the global frontend.
egparedes
added a commit
that referenced
this pull request
Oct 29, 2026
Bring the reworked stack (#2741 backend structure, #2956 embedded backends, #2742 observability on the new design) under the pipeline PR and adapt it: - `CompiledBackend.compiler` / `.loader` and `Toolchain.frontend` are typed as `workflow.Step`; `Toolchain.translate` and `is_dace_backend` narrow to `CompilePipeline`; the dace SDFG conversion replaces the translation step inside `CompiledBackend.compiler`. - The allocator-vs-compiler device-agreement check moves from `Toolchain.__post_init__` to `CompiledBackend.__post_init__` (after the abstract-base check); its tests and the pre-built-backend checks use `backend.compiler`. - `roundtrip.gtir` keeps the default frontend (no private `Transforms`); the `Transforms.foast_to_itir` comment no longer names `roundtrip`. - ADR 0029 edits of this PR (CachedStep without `.replace`/`.chain`, static-checking scope, "Cross-step agreement is checked on construction", phasing) are carried over into the rewritten, renamed ADR; the old file stays deleted. HackTheToolchain.md / WorkflowPatterns.md / AGENTS.md use the backend vocabulary.
egparedes
changed the base branch from
otf-split-2-toolchain-naming
to
otf-split-2b-embedded-backends
October 29, 2026 07:48
egparedes
marked this pull request as draft
October 29, 2026 08:10
…ability # Conflicts: # src/gt4py/next/program_processors/runners/dace/program.py # tests/next_tests/integration_tests/feature_tests/dace_tests/test_program.py
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Stack: #2741 → #2956 → #2742 → #2743. #2741 and #2956 were merged into this branch and the PR adapted to them.
Gives the compile route a sanctioned way to observe and partially run its pipelines, and removes the one consumer that reached into them:
otf.workflow.stage_hook(name, artifact), an event hook emitted after each step by the frontend (backend.Transforms), the standard compiler (recipes.OTFCompileWorkflow) andToolchain.translate. Artifacts are passed opaquely; with nothing subscribed, emission is an empty callback loop (~160 ns).GT4PY_DUMP_STAGES=<dir>registers theinstrumentation.stage_dumpsubscriber at import time, writing every stage artifact to<dir>/<program>/<NNN>_<step>.<ext>;enable()/disable()cover programmatic use. Exclusive-create keeps concurrent workers and same-named programs from overwriting each other.Toolchain.translate(definition, compile_time_args), the sanctioned partial run (frontend + translation). It lives on the internalToolchainbecause it needs the global frontend, and raises a clear error for compilers that are not the standard pipeline. Variants are built withdataclasses.replaceon the backend's compiler; there are no per-call step options.__sdfg__reach-in is gone. It duck-typed through the compiler'stranslationstep and unwrapped itsCachedStepby hand. It is replaced by a DaCe-owned translate-only backend variant used throughToolchain.translate. (refactor[next]!: make Backend abstract with a CompiledBackend kind and an internal Toolchain #2741 already stopped__sdfg__from mutating the cached frontend result; this PR keeps a test that the cache stays untouched.)Removing the reach-in drops the last non-doc consumer of the workflow mixins'
.replace, which unblocks the pipeline simplification in #2743.GT4PY_DUMP_STAGESis new and opt-in. Nothing else changes behavior, and no cache keys rotate.