You have a list and a fuzzy criterion: grant proposals by expected impact, a backlog by user pain, ideas by upside. Direct 1–10 ratings cluster around 7 and one-prompt sorts can drop items. llmsort instead asks many small pairwise ratio questions, fits one consistent score scale, and reports the size and uncertainty of every gap.
It spends comparisons where they are most informative, stops when the requested top-k is certain enough or the budget runs out, and reports comparisons, tokens, dollars, and the stop reason.
$ curl -fsSL https://raw.githubusercontent.com/XyraSinclair/llmsort/main/install.sh | sh
$ export OPENROUTER_API_KEY=...
$ curl -fsSLO https://raw.githubusercontent.com/XyraSinclair/llmsort/main/examples/ideas.txt
$ llmsort sort ideas.txt --by "usefulness as startup advice" --scores
2.186±0.230 Talk to ten users before writing any code
1.883±0.178 Ship a rough version this week and fix it live
1.835±0.193 Charge money from day one
0.486±0.176 Rewrite the backend in a faster language
0.187±0.183 Add a dark mode
0.000±0.230 Design a logo before the product existsScores are natural-log magnitudes: a gap of 1.0 means about 2.7× as much of the attribute in the judge's reading; ± is one standard deviation.
Results go to stdout in the input shape; progress and cost go to stderr. JSONL and CSV rows keep every field and gain scores:
$ llmsort sort grants.jsonl --by "expected good done per dollar" > ranked.jsonl
$ jq -r '"\(.llmsort.rank) \(.title)"' ranked.jsonl
1 Deworming in Kenyan schools
2 Malaria nets for Kano State
3 Open-source vaccine cold-chain sensors
4 Office plants for a think tank
5 A podcast about the history of fonts
6 Luxury yacht refit for a donor retreat
$ llmsort sort backlog.csv --by "user pain if unfixed" --field title --field notes
id,title,notes,llmsort_rank,llmsort_latent_mean,llmsort_latent_std,llmsort_z_score,llmsort_percentile
1,Login fails on Safari,"Users cannot sign in; ""blank screen"" after submit",1,4.570811,0.454564,0.674491,0.900000
5,Payments double-charge,Some cards are charged twice,2,4.360191,0.433434,0.388191,0.700000
3,Export to CSV drops rows,Rows with commas vanish silently,3,4.074614,0.411624,0.000000,0.500000
4,Dark mode button misaligned,Off by 2px on settings,4,0.225386,0.436597,-5.232328,0.300000
2,Typo in footer,The word privacy is misspelled,5,0.000000,0.454564,-5.538700,0.100000
$ git log --format=%s -n 40 | llmsort sort --by "risk of breaking users" | head -n 5- A
.csv,.jsonl, or.ndjsonfilename selects that input; otherwise a leading{means JSONL,[means a JSON array, and anything else means one item per line.--input lines|json|jsonl|csvoverrides detection. --fieldis repeatable and limits which JSONL keys or CSV columns the judge reads. Without it, the judge sees the whole row asfield: valuelines.- JSONL gains an
llmsortobject. CSV gainsllmsort_rank,llmsort_latent_mean,llmsort_latent_std,llmsort_z_score, andllmsort_percentile. Re-sorting replaces old scores before judging. --format jsonl|csvconverts row shape,--format textprints what the judge read, and--format jsonprints the full run report.--top-k 10focuses the comparison budget on settling the top ten.
$ curl -fsSL https://raw.githubusercontent.com/XyraSinclair/llmsort/main/install.sh | sh
$ cargo binstall llmsort # the same prebuilt binary
$ cargo install llmsort # build from source
$ cargo add llmsort # use the libraryThe install script places a checksum-verified macOS or Linux binary in
~/.local/bin; Windows builds are on the
releases page.
LLMSORT_INSTALL_DIR and LLMSORT_VERSION override the destination and release.
The judge uses OpenRouter, so set
OPENROUTER_API_KEY. Pairwise sorting defaults to openai/gpt-5.6-terra;
setwise sorting defaults to openai/gpt-5.6-luna. --model accepts any
OpenRouter slug. Judgements are cached locally, so an identical rerun does not
pay for the same calls again.
These methods were run head-to-head on the same pools, models, and seeds. See PROGRAM.md E12–E16 and the linked evidence packs for denominators and limits.
| Method | What the measurements say | Use when |
|---|---|---|
| Pointwise 0–100 | Cheapest per item, but close pools collapse into tie blocks; one 16-item run produced three distinct scores and truth-ρ −0.19 | A rough full-list order where top-k and magnitudes do not matter |
| Single-call listwise | Fast for small lists, but one malformed answer loses the run and the implementation is capped at 26 items | A one-shot sort you will inspect manually |
Setwise (--setwise) |
At ring k=8 with two rounds, matched the pairwise path's test–retest band at about one-third the cost; the flip-rate gauge flags unstable pools | Adequate orders for reranking, triage, and queues |
| Funnel (setwise screen, pairwise top-k refinement) | Reached the pairwise path's own top-10 reproducibility at 0.3–0.6× its cost | Finding the best few of many |
| Pairwise ratio (default) | Returns cardinal scores ±σ, counterbalances presentation order, and targets the requested boundary | Magnitudes, error bars, or certification matter |
| Typed ten-level rating in a 24-item window | On hosted Jev, led 29 tested designs across four cohorts and every budget; it is not yet a crate instrument | Research with a typed-probability judge |
Setwise order is not evidence that near-duplicates differ: read adjacent error bars before trusting their relative positions. The general rule is to choose an instrument that measures its own trustworthiness and to treat any top-k claim without a stability number as unmeasured.
$ llmsort judge "plan A" "plan B" --by "execution risk"
$ llmsort judge @a.md @b.md --by "clarity" --spin
$ llmsort judge @a.md @b.md --by "clarity" --orbitjudge exposes one comparison and optional probes for presentation order,
polarity, wording, and requester framing.
use std::sync::Arc;
use llmsort::gateway::NoopUsageSink;
use llmsort::rerank::{sort_texts, RerankExecution, SortOptions};
use llmsort::{Attribution, ProviderGateway};
# async fn demo() -> Result<(), Box<dyn std::error::Error>> {
let gateway = ProviderGateway::from_env(Arc::new(NoopUsageSink))?;
let execution = RerankExecution::new(Arc::new(gateway), Attribution::new("app::sort"));
let sorted = sort_texts(
vec![
"First essay...".into(),
"Second essay...".into(),
"Third essay...".into(),
],
"clarity of explanation",
execution,
SortOptions::default(),
)
.await?;
for item in &sorted.items {
println!(
"{:>2}. {:.3} ± {:.3} {}",
item.rank, item.latent_mean, item.latent_std, item.text
);
}
println!("cost: ${:.4}", sorted.meta.provider_cost_nanodollars as f64 / 1e9);
# Ok(())
# }The promised surface is sort_texts, sort_documents, their setwise siblings,
the sort and judge CLI verbs, and the content-addressed judgement-packet
format. Other public modules support composition but may change before 1.0.
Three 20-item pools with public numeric truth (countries by population, rivers
by length, mountains by height), one judge (openai/gpt-5.6-terra), two seeds,
v0.15.0 binary. Mean Spearman ρ against truth:
| Method | Countries | Rivers | Mountains | $ per run | What went wrong |
|---|---|---|---|---|---|
| Pointwise 1–10 | 0.57 | 0.34 | −0.37 | 0.006 | 60–90% of items got the same score |
| One-prompt listwise | 0.99 | 0.99 | 0.99 | 0.002 | dropped 6–7 of 20 countries in both runs (counted as tied last) |
sort --setwise |
0.95 | 0.78 | 0.98 | 0.005 | — |
sort (pairwise ratio) |
0.99 | 0.62 | 0.93 | 0.25 | lost on rivers |
Only pairwise returns magnitudes: its scores correlate with log truth at Pearson r 0.99 (countries), 0.98 (mountains), and 0.69 (rivers). When a single prompt returns every item, it orders well-known facts as well as anything else at about 1/100 the cost. Pairwise is worth its price when you need the gaps, the error bars, or a guarantee that no item is lost. Rivers, whose lengths depend on measurement convention, is its open failure. Raw responses, outputs, and the rerun command are in the evidence pack ($1.53 total).
Each ratio answer becomes a noisy log-space measurement. llmsort fits the comparison graph with Huber IRLS, derives score uncertainty from the posterior, and chooses new pairs by their expected value near the top-k boundary.
The cost trade is measured rather than assumed. On 72,813 private production judgements, with public analysis scripts and no published rows, ratio magnitude needed 1.3–1.8× fewer calls than direction alone at small budgets; probability mass from answer logprobs added about 1.4× efficiency. At full measured budget, direction-only sorting plateaued at split-half Kendall τ 0.16 while PMF moments reached 0.49. Scope and method are in the analysis note.
ALGORITHM.md explains the design,
MODEL.md states the mathematical contract, and
WORKED_EXAMPLE.md walks through a complete run.
PROGRAM.md indexes every research claim by its replayable evidence
pack. experiments/ contains instruments that have not graduated into the
published crate; research/ holds dated notes, analysis, and evidence packs.
MIT.