Repository navigation
fix(openai-agents): run agents as written, support agent.asTool() and remove agentAsTool (breaking) - #270
fix(openai-agents): run agents as written, support agent.asTool() and remove agentAsTool (breaking)#270campoy wants to merge 10 commits into
Conversation
…agentAsTool (breaking) TemporalOpenAIRunner now gives its inner Runner a model provider that sends every model name to the model Activity, plus the run's model. Upstream passes the runner's config to nested agent.asTool() runs, so the framework's own asTool() works in a Workflow: its model calls are model Activities with the run's model and model settings and the runner's Activity options. agentAsTool and AgentAsToolOptions are removed. Use agent.asTool().
|
Let's take option 2: keep native For now, wrap The main benefit is the developer experience: applications moving from native execution to Temporal can keep their agent-tool declarations. We also reuse upstream's tool behavior rather than maintain a separate implementation. The plugin still needs its Temporal runner, Worker setup and Activity boundaries; native syntax does not remove those requirements. Existing plugin users would still migrate from The tradeoff is that the plugin owns a temporary patch to a shared upstream class in the Workflow runtime. That makes initialization order and SDK compatibility our responsibility. The implementation must install interception before tools are created, handle repeated initialization safely, and account for subclasses and duplicate SDK module copies. Tools whose source was not recorded must be rejected with a clear error. This choice does not accept the two review findings as documented limitations. Before merging, we need to:
In parallel, request a supported upstream way to inspect an agent tool's source and options and rebuild or rebind it while preserving identity and approval restoration. Once that exists, we can replace the interception inside the plugin while keeping the same native API and tested behavior. Callers should not need another migration for that internal change. |
TemporalOpenAIRunner no longer clones the agent graph to swap in Activity-backed models. Its inner Runner's model provider already sends every model name to the model Activity, so agents run as the caller wrote them. That also fixes two defects of the clones: agent.on(...) listeners on the caller's agents now fire (#235), and a custom Handoff now hands off to the agent its callback returns (#237). convertAgent becomes checkAgent, which runs the same checks, including the nested agent.asTool() checks, without changing the graph. asTool() tools are no longer rebuilt, so the asTool() record keeps only the agent and the runConfig and runOptions the tool was made with. The same-name check goes: without clones, handoff and nested agents keep their own string models. A ModelSummaryProvider still gets the running agent, tracked from the inner Runner's agent_start and agent_handoff events; a model call during a tool call belongs to a nested agent.asTool() run and gets none. As upstream, runConfig.model now applies only to agents that set no model of their own.
brianstrauch
left a comment
There was a problem hiding this comment.
Lint, build, and all 261 existing tests pass on Node 24. A focused Worker regression probe reproduces the nested-streaming restriction bypass noted inline.
| const innerRunner = new Runner({ | ||
| ...runnerConfigOverrides, | ||
| model: modelOverride, | ||
| modelProvider: new ActivityBackedModelProvider(this.modelParams, () => |
There was a problem hiding this comment.
[P2] Enforce the local Activity streaming restriction for nested tools
With useLocalActivity: true and a nested agent.asTool({ onStream }), the child run goes directly through upstream Runner, bypassing the StreamingLocalActivityUnsupported guard in TemporalOpenAIRunner.run(). This provider inherits the local Activity options and schedules invokeModelStreamActivity locally. I reproduced this by running the existing agentToolStreamWorkflow with modelParams.useLocalActivity: true: the provider's getStreamedResponse() runs once, both stream handlers receive the answer, and the parent succeeds instead of rejecting the unsupported combination. Please enforce the restriction in ActivityBackedModel.getStreamedResponse() or the shared run check, and add nested regression coverage.
There was a problem hiding this comment.
Good catch, thanks. Fixed in 3f11f6c: ActivityBackedModel.getStreamedResponse() now rejects useLocalActivity, next to its streaming-topic check, so every streamed model call has the check, including nested agent.asTool() runs. The runner's own check stays, so a top-level streamed run still fails before it starts.
One difference from the top-level case: as upstream, asTool() returns a nested run's failure to the parent model as the tool's output, so the parent run goes on. The nested model is never called. This matches the other checks that run inside a nested run. Rejecting the parent before it starts would need every on() handler up front, and handlers can be added after the run starts.
The new Worker test, A nested agent.asTool() run fails to stream when model calls are local Activities, runs agentToolStreamWorkflow with useLocalActivity: true. It fails without the check (the nested model streams) and passes with it.
…agent.asTool() runs A nested agent.asTool() run with onStream or on() handlers streams through upstream's Runner, so it skipped the runner's useLocalActivity check and ran the streamed model call as a local Activity. Check in ActivityBackedModel.getStreamedResponse(), which every streamed model call reaches. As upstream, the nested failure is the tool's output.
What was changed
TemporalOpenAIRunnerruns the caller's agents as they are written. It no longer clones the agent graph to swap in Activity-backed models. Its innerRunnergets a model provider,ActivityBackedModelProvider, that sends every model name to the plugin's model Activity with the runner's Activity options, and the run'smodel.agent.asTool()runs, so the framework's ownasTool()now works in a Workflow. Nested runs take the run'smodel(for agents that set none), itsmodelSettingswithouttoolChoiceandparallelToolCalls, itscontextand sandbox, and the runner's Activity options.agentAsTool,AgentAsToolOptionsand the hook that passed the sandbox to it are removed.agent.on(...)listeners on the caller's agents fire, and a customHandoffhands off to the agent its callback returns.convertAgentbecomescheckAgent, which checks the graph without changing it: raw functions intools, a missing model, andModelobjects, through handoffs andasTool()tools. It also rejectsasTool()options that would run model calls or sandbox operations in the Workflow:runConfig.modelProvider(evenundefined, which upstream spreads over the runner's provider), aModelobject asrunConfig.model, a sandbox client or live session not fromtemporalSandboxClient(), and aMemorySession.Agent.prototype.getSystemPrompt, which upstream calls once per model call, after it picks the model. This covers agents the check before the run can't see: one that a customHandoff's callback returns, or one that anagent.on(...)listener changes. Its model must be a name, and itsasTool()tools must pass the checks above. In the outer run, the run fails. In a nested run, the error goes back to the parent model as the tool's output, and the model isn't called. An agent from another copy of@openai/agents-core, such as one a dependency brings, is checked the same way: the plugin wraps that copy'sAgent.prototype.getSystemPromptbefore the agent's first model call, from the check before the run, from each run's start and from the runner'sagent_startevent.asTool()tool'srunConfigandrunOptionsonly when the nested run starts, after the tool'sparametersschema, itsinputBuilderand anyagent_tool_startlistener ran. So the plugin also checks eachRunnerrun as it starts, by wrappingRunner.prototype.runin the Workflow: the run's model provider must be the runner's, itsrunConfig.modela name, its session not aMemorySession, and its sandbox config must pass the checks above. A nested run that fails this doesn't start, and the error is the tool's output. The run then uses a frozen copy of its sandbox config, which upstream reads as it goes, so a later change to the caller's object, for example in anagent_startlistener, doesn't reach it.useLocalActivitynow fails inActivityBackedModel, not only inrun(), so a nestedasTool()run withonStreamoron()handlers is rejected too.asTool()tool returns a nested run's failure as its output, including a cancelled model Activity. If the run's cancellation scope was cancelled,run()now throws theCancelledFailureinstead of returning, and a streamed run fails with it, even when the run ended on that output, for example withtoolUseBehavior: 'stop_on_first_tool'.asTool()tool, which upstream keeps private, the plugin wrapsAgent.prototype.asToolin the Workflow before the Workflow module loads, and records each tool's agent and therunConfigandrunOptionsit was made with. The returned tool is not modified. A tool that looks like anasTool()tool but wasn't recorded, such as a copy, is rejected.ModelSummaryProviderstill gets the running agent, tracked from the innerRunner'sagent_startandagent_handoffevents. A model call made during a tool call belongs to a nestedasTool()run and getsagent: undefined, which its type allows.agent.asTool(), lists the rejected nested setups, the check before each model call (including agents from another copy of@openai/agents-core) and the check when each run starts, explains cancellation when a run ends on a tool's output, adds agent lifecycle hooks to the feature table, and saysrunConfig.modelapplies to agents that set none.Migration: replace
agentAsTool(agent, { toolName, toolDescription })withagent.asTool({ toolName, toolDescription }), created in Workflow code.Why?
Three defects had the same root, the plugin's copies of the framework's objects:
agentAsTool, the plugin's partial copy ofasTool(), dropped the run's model, model settings and runner defaults. The framework's ownasTool()didn't run the nested agent at all: the parent model gotNo default model provider setas the tool result.agent.on(...)listeners on the caller's agents never fired.Handoff's callback returns, so the run went to the wrong agent.With a provider on the runner, upstream's code does the work, including fields it passes to nested runs later.
Breaking changes
agentAsToolandAgentAsToolOptionsare gone.runConfig.modelnow applies only to agents that set no model of their own, as upstream. Before, it replaced every agent's model. Runs in progress andRunStatesaved before the upgrade follow the new rule from their next model call: an agent that sets its own model switches to it, without an error. Let those runs finish before upgrading, or use Worker Versioning.Nested runs behave like upstream
asTool(). A nested run's failure goes back to the parent model as the tool's output instead of failing the run. A nested approval pauses the parent run instead of failing it withNestedAgentInterruption. Nested model calls use the runner's Activity options.A
SandboxAgentreturned by a customHandoff's callback, in a run withoutrunConfig.sandbox, fails the run with upstream's error, anAgentsWorkflowErrorwithSandboxAgent execution requires RunConfig.sandbox.An agent that a custom
Handoff's callback returns, or that anagent.on(...)listener changes, now fails the run if its model is aModelobject or one of itsasTool()tools is rejected, including an agent from another copy of@openai/agents-core. AnasTool()tool whoserunConfigorrunOptionsa listener, itsinputBuilderor itsparametersschema changes to a rejected setup doesn't start its nested run; the error is the tool's output, as upstream.An upstream
Runner, orrun(), can't run agents in a Workflow: its run is rejected unless its model provider is the plugin's. It had no Activity-backed provider before, so it failed withNo default model provider setor ran models in the Workflow. UseTemporalOpenAIRunner.A
runConfig.sandbox.sessionthat isn't from a Temporal sandbox client is rejected withSandboxConfigurationError.asTool()tools must be created in Workflow code, so the plugin can record them.Replay: runs that started before the upgrade keep working after users switch from
agentAsTooltoasTool(), except five kinds, which fail replay with[TMPRL1100]:useLocalActivity: the nested model call was a regular Activity and is now local;agent.asTool()before this change: the nested agent never ran, and now it schedules a model Activity;agentAsToolruns whose nested run failed and whose Workflow recorded anything after the failure, such as recovery work in acatch: the failure now goes back to the parent model, which schedules a model Activity;agent.on(...)listeners, for example anagent_endlistener that schedules an Activity: the listeners didn't fire before;Handoffwhose callback returns a different agent from the one it was declared with, when the two agents call different Activities: the run went to the declared agent before.A run still waiting after such a failure, for example in its
catchhandler, replays without an error but changes its result: it continues through the parent model and skips its recovery. A replay check of old histories doesn't find these runs, or the model switch above. Let all of these runs finish before upgrading, or use Worker Versioning.Checklist
Closes [Bug] openai-agents (TypeScript): agentAsTool runs drop the run's model, model settings and runner defaults #243, closes [Bug] openai-agents (TypeScript): agent.on(...) lifecycle hooks never run through TemporalOpenAIRunner #235, closes [Bug] openai-agents (TypeScript): TemporalOpenAIRunner ignores the agent a custom Handoff's callback returns #237. Replaces fix(openai-agents): give agentAsTool runs the run model, model settings and runner defaults #244, Run agent.on(...) lifecycle hooks in TemporalOpenAIRunner #236 and fix(openai-agents): hand off to the agent a custom Handoff returns #238.
How was this tested:
New Worker tests:
agent.asTool() runs take the run model, model settings and runner defaults;agent.on listeners run and their context writes reach later turns;A custom Handoff hands off to the agent its callback returns;agentAsToolreplaced byasTool();runConfig.model leaves an agent that sets its own model on that model, as upstream.Each of these fails without the change.
A ModelSummaryProvider gets the running agent; nested agent.asTool() calls get nonepasses before and after the clone removal, with the same labels.Tests from the review rounds, which pass on this design:
Agents picked or changed during the run are checked before they call their model. It covers aModelobject from a customHandoff's callback, a callback-returned agent with anasTool()tool that setsrunConfig.modelProvider, andagent_startlisteners in the outer and a nested run. It also covers anasTool()tool whose capturedrunConfiganagent_tool_startlistener changes to amodelProvider, tomodelProvider: undefined, or to aModelobject for a model-less nested agent; aninputBuilderthat sets amodelProvider; andasTool({ runConfig: { modelProvider: undefined } }), with a default model provider registered. Each is rejected before the nested model is called, and each replays. Without the checks, the inline model runs in the Workflow. A control, a listener that changes the tool'srunConfig.modelto another name, schedules two model Activities and the nested call uses the new name. AparametersZod refinement that sets amodelProviderwas checked the same way in a local probe (the plugin has no directzoddependency): the inline model ran without the run check and was rejected with it;Agents from another copy of @openai/agents-core are checked before they call their modelbundles a second, unmodified copy of@openai/agents-core0.14.3. It covers three ways such an agent gets aModelobject: anagent_startlistener, a customHandoff's callback, and a tool between two turns. Each is rejected before the model is called (0, 1 and 1 model Activities), and each replays. Without the fix, all three ran the inline model in the Workflow. A control agent from that copy with a string model completes through one model Activity;A sandbox client that an agent_start listener replaces with a raw one never runs: the run completes through sandbox Activities; without the frozen copy, the raw client ran in the Workflow. Theagent_tool_startvariant, on anasTool()tool'srunOptions.sandbox, is rejected as the tool's output without calling the backend. Both replay. A rawrunConfig.sandbox.sessionis rejected; without the check it ran in the Workflow;stop_on_first_tool, plain and streamed. Without the fix, both completed withActivityFailure: Activity cancelledas output. Two controls, the default loop andstop_on_first_toolwithout cancelling, pass with and without the fix;continueAsNewwith an agent that is both a handoff target and anasTool()source;onStreamand toon()handlers, including a handler registered after the run starts;useLocalActivity: the nested model isn't called, and the tool's output is theStreamingLocalActivityUnsupportedmessage. Without the check, the nested model streamed through a local Activity;asTool()options object.check-agent.test.tscovers the checks, theasTool()record, the check before each model call (also on another copy'sAgentclass, installed once) and the check when each run starts (each installed once, inactive outside a Workflow).Locally, in a Linux container with Node 24 and pnpm 10:
make lint(the 8 warnings already onmain, none in changed code),make test(227 tests),make build, and Prettier with the SDK settings pass. The Node 22, macOS and Windows cells did not run locally.Replay check, with Workers with no Workflow cache, on runs with no run config, a run model and settings, a local-Activity runner and native
asTool():main(agentAsTool) and this change (asTool()), in both directions;asTool()runs fail with[TMPRL1100], as listed above;[TMPRL1100]in both directions; the same run held in itscatchhandler onmaincompleted on this change through the parent model, without its recovery;agent_endlistener that schedules an Activity, and a customHandoffthat picks an agent with a different Activity tool, failed with[TMPRL1100]in both directions, while their self-replays passed; they are listed above.The same histories, and runs held across versions, replay between the first commit and the clone-free code before the review-round merge. The merge doesn't change which commands are scheduled, and neither do the checks added since: they schedule nothing for valid setups.
Cancelling a Workflow during a nested model call fails it on
mainand cancels it with this change.Any docs updates needed?
The README is updated here. The docs page and the samples are updated in Show agent.asTool() instead of agentAsTool on the TypeScript OpenAI Agents page documentation#5597 and Use agent.asTool() in the OpenAI Agents agent-patterns sample samples-typescript#534, drafts until this is released.
Not covered:
getSystemPrompt. AnAgentsubclass that overrides it without callingsuperskips the check.Runner.prototype.run. Code that replacesrunon the prototype or on aRunnerinstance skips it.Handoffobject from another copy of@openai/agents-corein an agent'shandoffs: upstream's run loop doesn't recognize it (it checksinstanceof Handoff), so the run fails in upstream before any model call.