Model Difference Infrastructure?

While leaderboards tell us which models do best on certain benchmarks, they do not speak to model difference: i.e. whether two models behave alike, make similar mistakes, share common origins, or if they could substitute for one another in production.

Many research groups are exploring this, seeking to map and fingerprint these differences, using different methods and with varying degrees of success.

Five ways to compare models

1. Probability-distribution distance

Mapping 1,000+ Language Models via the Log-Likelihood Vector represents each model by its likelihoods of tokens over a collection of texts. Distances between these vectors approximate differences between the models' probability distributions.

This produces scalable model maps and can predict benchmark performance, but the method requires weight access and therefore can be applied only to open weight models. It can't necessarily determine whether two models are interchangeable for users.

2. Observable behavioural distance

LLM DNA instead embeds responses to fixed prompts and compresses these into compact functional fingerprints. This creates a 'DNA' sequence (a vector) for each model, which can be used to determine model relatedness. They found that models known to be in the same model family have similar DNAs, and that similar DNAs correspond to similar functional behaviours in domains such as maths, coding, etc.

This approach works with APIs, allowing it to catalogue closed models, detect drift and onboard new models. But their results depend on the prompts, decoding configuration and embedding model. A behavioural family tree is not proof of ancestry: independently developed models could have similar DNA simply because they were trained on similar datasets.

3. Capability and error distance

You could argue that success or failure on identical tests matters more than whether two models produce similar text.

CAPA measures agreement in model errors after adjusting for overall accuracy. EmbedLLM learns model representations from a large question-by-model correctness matrix, while UniRoute profiles models using their errors on representative prompts.

This is the most directly useful family for routing, portfolio construction and ensemble diversity. Its weakness is that diffs are highly specific to the evaluation tasks and scoring system that you use.

4. Identity, equality and provenance

Others are exploring whether an API is serving the advertised model or whether one model was derived from another.

Model Equality Testing statistically compares API output distributions. The newer KBF reports detecting model substitutions across production endpoints using cheap black-box probes. LLMmap identifies model versions from a handful of crafted interactions.

Where weights or activations are available, REEF, Independence Tests and LLMPrint target lineage and reused components. These methods support API assurance, licence enforcement and supply-chain investigations. It is important to remember that identity and ancestry are different from functional similarity.

5. Interpretable model diffing

Rather than producing one distance, Model-diff searches for regions where two models disagree. Diff Mining uses logit changes to identify what a fine-tuned model may have learned. Anthropic has also demonstrated a cross-architecture behavioural diff tool.

This points towards automated model release notes, fine-tuning audits, bias discovery and regression analysis.

Commercial opportunities

Given the landscape, our view on the most plausible products are:

• routing based on capability, cost and risk
• continuous API authenticity and monitoring of behavioural drift
• model migration and replacement recommendations