Knowledge graph alignment at scale: fast, cheap, and (sometimes) right

Can a cheap, fast decision model stand in for a reasoning LLM? We tested two decision models (Jev and Strands Decider) against Claude Haiku and Claude Sonnet on a real entity-alignment pipeline: 8,524 pairs across two domains.

We rebuilt a knowledge-graph-alignment pipeline so that the judging step is a pluggable component, then ran the same 8,524 candidate pairs, from two domains, through four AI judges. Two are decision models: they return a structured probability directly, without writing any text. The other two are general-purpose reasoning models that generate their answer as text. The best decision model, Jev, matched the strongest reasoning model on the final merge decision on one dataset and came within a few points on the other, at more than 100 times lower cost. Against the cheapest reasoning model in this comparison, Jev was cheaper and about 5 times faster than Claude Haiku; the draft’s conflicting Haiku cost figures are noted below. Not every decision model is there yet: the open-weight Strands Decider scored barely above a baseline with no language model on one dataset, and below it on the other. And one reasoning model’s F1 collapsed on one dataset, not because it judged badly but because of how its decision threshold was chosen. Below are the results, what they mean for anyone running judgments at high volume, and where reasoning models still earn their keep.

The problem: merging two things that say almost the same thing#

Any organization that works with more than one knowledge source eventually ends up with multiple taxonomies, ontologies or product catalogs and no clean mapping between them. Is “Software Developer” in one system the same job as “Applications Programmer” in another? Is “DOID:0050741” the same disease concept as “NCIT:C34798”? At the scale of a real taxonomy, answering this takes thousands to millions of pairwise judgments. Each one needs more than a yes or no: is it an exact match, is one broader or narrower than the other, are they merely related, or are they unrelated?

The two standard approaches have limits. Lexical and rule-based matching (string similarity, shared keywords) are fast and free, but brittle. Both miss paraphrases and synonyms, and can’t separate an exact match from a related but distinct concept without a lot of hand-tuned logic. The newer default is to put an LLM in the loop: show it both entities and ask it to judge the relationship. That works well, but cost and latency grow linearly with the number of candidate pairs, and exhaustive alignment between two sources requires the product of their entity counts in candidate comparisons.

A newer category of model targets that gap. Decision models like TypeSafe’s Jev and the open-weight Strands Decider turn a prompt directly into a typed judgment: a probability distribution over relation types, returned in a single step, with no generated text to parse. If one of them can match a reasoning model’s judgment quality at a fraction of the cost and latency, high-volume judging pipelines get much cheaper to run.

So we tested it. We built kgalign, a knowledge-graph-alignment pipeline with a swappable judging stage, and ran the same entity pairs through Jev, Strands Decider, Claude Haiku and Claude Sonnet. The question was whether a decision model can do a reasoning model’s job on a real task, at a fraction of the cost. The short answer is yes for one of them, with caveats worth knowing.

Meet the four judges#

*Decider’s latency isn’t a fair comparison: it ran on a low-end consumer laptop GPU, while the others ran on dedicated cloud infrastructure. It’s included for completeness and shouldn’t be read as a ranking.

The table shows the cost gap before any accuracy numbers come in. Jev is more than 100 times cheaper than Sonnet on the draft’s reported figures. Decider has no API fees; local hardware and electricity still have a cost. The Haiku cost figures conflict, so we do not calculate a cost ratio for it. The question is what you give up for that.

We also ran a simple logistic-regression baseline, fit on lexical and embedding similarity scores. It isn’t a fifth contender. It shows how far you get with no language model at all.

We chose entity alignment as the testbed on purpose. Every answer is a typed label that can be checked against real gold data, so we can score accuracy and relation type directly. The task also needs calibrated confidence, because the downstream pipeline uses a probability threshold to decide whether to auto-merge a pair, send it to human review, or reject it. And it runs at a volume where differences in cost and latency matter.

Methodology: same pairs, same rules, two domains#

We used two public datasets whose text reads very differently:

• ESCO ↔ O*NET (occupations). ESCO is the European Commission’s multilingual classification of skills, competences and occupations. O*NET is the US Department of Labor’s occupational database. The European Commission publishes an official crosswalk between the two that labels each occupation pair as an exact, broader, narrower or close match. We used ESCO v1.2.1 (3,027 occupations) and O*NET 31.0 (1,016 occupations). The labels are descriptive and often paraphrased (“Software Developer” vs. “Applications Programmer”), so a judge has to understand what a job involves, not just compare titles.
• Bio-ML NCIT ↔ DOID (diseases). Bio-ML is the biomedical track of the Ontology Alignment Evaluation Initiative (OAEI), a long-running benchmark for ontology-matching systems. The NCIT ↔ DOID pair aligns the US National Cancer Institute Thesaurus with the Human Disease Ontology. We used the 2024 release (Zenodo, CC BY 4.0), restricted to the classes marked for alignment: 6,835 in NCIT and 8,448 in DOID. The vocabulary is dense, literal and hierarchical, and most DOID classes in this release have no text definition, only a name.

Both ship with official gold-standard alignments, allowing the same candidate pairs to be scored against reference labels. No confidence intervals or significance tests were supplied for this comparison.

Every judge saw the same candidate pairs and the same input text, so none got an easier or richer description than another. For each judge, we selected the decision threshold (the probability cutoff for auto-merging a pair) once on a held-out development split and then froze it. We scored the test split once against those frozen thresholds and never tuned on the data we report. Both Claude models ran with extended thinking turned off (their lowest thinking setting) and answered through the same structured tool call. Sonnet doesn’t accept a forced tool call, so it was told to answer with the tool, and it did every time.

The runs covered 2,913 pairs on ESCO↔O*NET and 5,611 on Bio-ML NCIT↔DOID, 8,524 in total, each scored by all four judges and the baseline. The draft reports total spend of $46 and a Jev cost of $0.30 across both datasets. Its itemized costs do not reconcile with that total; see the reporting note below. Decider and the baseline incur no API fees.

How we scored the judges#

For every pair, each judge returns a probability that the two entities are the same thing, plus its best guess at the relation type. Turning that probability into an action takes a threshold. The pipeline auto-merges a pair when the probability is at or above a merge cutoff, rejects it when it falls below a lower cutoff, and sends everything in between to human review. We set each judge’s merge cutoff with one rule: the lowest value that gives at least 99% precision on the development split, so auto-merges are almost never wrong. That cutoff is then frozen and applied unchanged to the test split.

For the match metrics, a pair counts as positive when its gold label is an exact match. Pairs the ESCO crosswalk labels “close” are left out, and candidate pairs with no gold label count as non-matches.

• F1 at the merge cutoff (decision quality). Precision is the share of auto-merged pairs that really are exact matches. Recall is the share of real exact matches that got auto-merged. F1 is their harmonic mean, so it’s only high when both are. It scores the decision the pipeline makes, so it depends on the judge and on the cutoff.
• AUPRC (ranking quality). Sort all pairs by the judge’s probability and lower the cutoff step by step, tracking precision and recall at each step. AUPRC is the area under the resulting precision-recall curve. A score of 1.0 means every true match is ranked above every non-match. Because it covers every possible cutoff, it measures how well the judge orders pairs, independent of the threshold.
• Relation-typing macro-F1. On pairs with a gold relation, we compare the judge’s predicted relation (exact, broader, narrower, related or unrelated) with the gold one, compute F1 for each relation type that occurs in the dataset’s gold labels, and average them with equal weight. Those types are exact and narrower on Bio-ML, plus broader on ESCO. Equal weighting stops a judge from scoring well by handling only the common type. A judge that predicts a type absent from the gold labels, such as “related”, is still penalized, because that pair counts as a miss for its true type.
• ECE and Brier score (calibration). These check whether the probabilities can be taken literally. ECE groups pairs into 15 bins by predicted probability and measures, on average, how far each bin’s predicted probability is from the share of true matches in it. Brier score is the mean squared difference between the predicted probability and the actual outcome (1 for a match, 0 otherwise). Lower is better for both.

Results: no single winner, and one surprise#

No judge wins everywhere. On ESCO↔O*NET, Jev and both reasoning models have close point estimates on the final merge decision (F1 0.811 for Sonnet, 0.808 for Jev, 0.797 for Haiku). Sonnet has the best ranking quality (AUPRC 0.887) and the best relation typing (macro-F1 0.693). On Bio-ML NCIT↔DOID, Sonnet leads Jev on F1 by 0.042 F1 (0.853 against 0.811). Strands Decider and the baseline trail far behind on every metric that involves getting the relationship type right.

Bold marks the best unambiguous value in each column for each dataset (lowest for ECE and Brier). Bio-ML AUPRC has no highlighted winner because the two supplied Haiku values imply different rankings.

Ranking quality and decision quality are different things#

Look at Claude Haiku on Bio-ML. Its F1 is 0.440, by far the lowest score either reasoning model produced, on a dataset where Sonnet and Jev both score above 0.81. One might incorrectly assume it’s a capability issue.

However, both reported values for Haiku’s AUPRC on the same dataset are high (see the reporting note). If you sort the pairs by Haiku’s score, the true matches sit almost entirely at the top. The ranking works, but the final selection failed.

The cause is how Haiku reports probabilities. A reasoning model writes its confidence as a number in its answer, and Haiku sticks to a limited set of round values. On the development split, its most common score for true matches was 0.93 (18 of the 50), but one non-match also scored 0.93. That puts precision at 0.93 at 96.9%, short of the 99% target, so the rule moved up to the next score Haiku gave: 0.95. Only 13 dev pairs scored that high. All 13 were correct, which satisfied the rule, but the cutoff now sat above most of Haiku’s true matches.

The test set repeated the pattern. 552 of the 1,402 true matches scored exactly 0.93, and, therefore, the 0.95 cutoff excluded them. Haiku auto-merged only 396 pairs. Every one of them was correct, but recall fell to 28% and F1 to 0.440. With the cutoff at 0.90, the same judgments would have scored an F1 of 0.859. The true matches it didn’t merge weren’t thrown away: below the merge cutoff, a pair goes to human review unless it scores under the lower unlink cutoff. 990 of Haiku’s missed matches landed in the review queue and only 16 were rejected. The cost of the bad cutoff was work for people: Haiku sent 1,880 pairs to review, against 604 for Jev.

The failure came from how the threshold was chosen, not from the model, and the lesson applies to any judge, decision model or reasoning model. When a judge’s scores cluster on a few values, a strict precision target on a small development set can land on a cutoff that sends most of the correct answers to human review instead of merging them. Looking only at F1, you’d conclude that Haiku can’t handle biomedical text. AUPRC, which ignores the cutoff, tells a different story: both supplied values indicate strong ranking quality, although their disagreement prevents a precise comparison with Jev and Sonnet. What failed was the one cutoff the rule picked from its clustered scores; with a cutoff of 0.90, the same answers score an F1 of 0.859.

It’s probably possible to get finer-grained probabilities out of Haiku, for example by changing the prompt or averaging several sampled answers, and the threshold rule itself could be tuned too. But the same kind of per-model tuning could be applied to every judge, Jev included, and once you start, the comparison stops being like for like. We ran every judge with the same setup and the same threshold rule, and left per-model optimization out of scope.

The rest of the picture#

Claude Sonnet has the lowest Brier score on both datasets. On ECE, the picture depends on the dataset. On the occupation data, Jev is clearly worse calibrated than both reasoning models (ECE 0.079, against 0.046 for Sonnet and 0.047 for Haiku). On the biomedical data, Jev is the best calibrated of the three (0.026, against 0.036 for Sonnet and 0.039 for Haiku). That matters if your pipeline needs probabilities it can trust on top of correct final labels.

Strands Decider is the only language-model judge in this comparison that runs locally without an API key, but zero API fees do not guarantee strong judgments. It has the weakest relation typing of all five on ESCO (macro-F1 0.204), and when it’s unsure it tends to fall back on “related” instead of committing to a specific relationship.

The baseline is a floor. With no LLM at all you get roughly 0.50 to 0.62 F1, which shows how much of the lift comes from language understanding on top of similarity scoring.

All of these numbers come from a single test-split run in October 2026, with no repeats or cross-validation.

When a decision model is enough#

Our results make a practical case for decision models. Jev’s merge F1 was close to Claude Sonnet’s on the occupation data and came within a few points on the disease data (F1 0.811 against 0.853), at more than 100 times lower cost and about 7 times lower latency. For a pipeline that has to judge hundreds of thousands of candidate pairs, a gap that size decides whether you can afford to judge every pair with the strongest model or only a sample.

The cheapest reasoning model also deserves consideration. Claude Haiku’s merge F1 was close to Jev’s on the occupation data, and it ran about 5 times slower. The conflicting cost figures prevent a precise cost comparison: a decision model is still cheaper and faster, but the case for it rests as much on fixed-label output, with nothing to parse, and on scores you can threshold reliably.

Reasoning models still earn their cost in some places. Sonnet made better merge decisions on the disease data, typed relationships better on the occupation data, and gave better-calibrated probabilities there. A pipeline that depends on the exact relationship (broader vs. narrower), or on probabilities it can read literally, may want Sonnet, at least for the pairs a decision model is unsure about. Of the judges tested here, the reasoning models can also generate explanations when a pipeline needs them.

Not every decision model is a drop-in replacement yet. Strands Decider was free to run, but on this task it scored about the same as a simple baseline with no language model, and worse on one dataset. Test a candidate model on your own data before you switch.

Whichever judge you use, measure ranking quality (AUPRC) separately from the F1 at your chosen threshold, and look at how the scores are spread. A judge whose scores cluster on a few round values can land on a cutoff that fails badly at production volume, as Haiku’s did here.

What’s next#

We’re open-sourcing the kgalign pipeline that produced these numbers. The results use only the two public datasets (ESCO/O*NET occupation data and Bio-ML’s OAEI 2024 NCIT↔DOID biomedical ontologies). No private or client data went into this benchmark.

Since we ran this benchmark, Jev has gone from a single product to the first of a new category, usually called decision models (TypeSafe’s own name is “System One” models). In the weeks after Jev’s launch, AWS released Strands Decider with open weights, Cloudflare launched its open-source Clef and Clef-flash models on Workers AI, and Ollama added a decision-model API so models like Nimble, Tev1 and Clef can run locally. Several of them deliberately copy Jev’s interface, so switching between them is mostly a matter of changing the endpoint.

They share one idea. Keep what large language models learned about language and the world, and drop the expensive part, which is generating text. A decision model reads the input once and returns probabilities over a fixed set of answers, so classification gets most of an LLM’s understanding at a small fraction of the cost and latency. Our own numbers show how large that gap is. Jev answered in about a quarter of a second, against roughly 1.3 seconds for Haiku and 1.8 seconds for Sonnet. Across all 8,524 pairs, the draft reports $0.30 for Jev and $35.50 for Sonnet. The Haiku totals remain unresolved. Against the strongest model in this lineup, a decision model can judge every candidate pair in a large alignment job for what a small sample would cost.

Our results suggest the idea holds for entity alignment, with a caveat that applies to every model in the category: as the Haiku result shows, a probability is only as useful as the threshold you apply to it. The open-weight models add one more advantage: weak spots like Decider’s relation typing can be fixed with a fine-tune, without per-call costs.