The open-weight leaders, measured
There is a conversation we keep having with banks and payment companies, and it always starts the same way: leadership wants the leverage everyone else is getting from the frontier APIs, and compliance has already said no. Cardholder data under PCI DSS. Transaction-level AML flags. Anything touched by GDPR, DORA, or national banking secrecy. It doesn't matter how good the vendor's data-processing agreement looks — for these data classes, "we send it to a third-party API" is not a sentence anyone wants to defend to the regulator.
The conversation usually stalls there, because both sides think the alternative is a science project. It isn't. As of this summer, the practical question for a bank is not whether capable open-weight models can be self-hosted inside the perimeter — on-premise, air-gapped if policy demands it — they can, on hardware that fits in one rack unit of budget discussion. The question is which model tier goes on which queue, and in what order.
This memo is that mapping. It is deliberately not a deployment guide — nobody on your board needs to hear about inference servers. It's about three numbers: how big the capability gap actually is, what each tier of model costs to own, and which of your queues each tier can honestly carry.
Open-weight vs commercial models in 2026: one generation behind, and holding
The chart above is the cleanest summary we know. On the Artificial Analysis Intelligence Index — the closest thing the industry has to a neutral composite score — the best open-weight models sit at 44–51 against roughly 60 for the closed frontier. That distance has held steady for over a year: about one model generation, six to twelve months. It is not widening.
Averages hide the texture, though, and the texture is what decides your rollout. Broken out by the work banks actually do:
- Document understanding — open weights win outright. On OmniDocBench, the benchmark closest to real KYC files, scanned invoices, and statements, the best open document models score around 94 against 90 for the best commercial flagship. The current leader is a specialized model under one billion parameters — it runs on a workstation GPU. Your document pipeline is the easiest workload to make fully sovereign, with zero quality sacrifice.
- Tool-using service agents — effectively tied. On tau2-bench, which simulates exactly the multi-turn, systems-connected work of a support or payments-ops desk, the best open model scores 99.1 against the best commercial 99.3. On function-calling benchmarks, open models have led outright since last year.
- Reasoning and knowledge — a few points. On MMLU-Pro-class evaluations the gap is three to four points. For drafting compliance answers, summarizing casework, and multilingual support, that difference rarely survives contact with a well-designed review step.
- Long-horizon autonomy — still closed territory. Agents that run unsupervised for hours across dozens of steps, and reliable retrieval across million-token contexts, remain clearly better on the frontier APIs. If a workflow genuinely needs this, it needs a different sovereignty answer — not a worse model.
One caveat belongs in every board deck: most open-model scores are vendor-reported, and results swing several points with test settings. Treat leaderboards as a shortlist, not a verdict — the verdict comes from a two-week evaluation on your own queue, inside your own perimeter, which is the one benchmark nobody can game.
The weights are just files. Once they are inside your perimeter, the sovereignty question is answered — everything that remains is an operations question.
The shortlist: GLM-5.2 vs DeepSeek V4 vs Mistral Large 3 vs Qwen3.5 vs gpt-oss-120b
Five models cover almost every sovereign deployment we would argue for this year. Not five at once — the whole point of the comparison is that you will deploy one, maybe two. Prices are street estimates for the server class, not quotes; the point is the order of magnitude.
| GLM-5.2 | DeepSeek V4 | Mistral Large 3 | Qwen3.5-397B | gpt-oss-120b | |
|---|---|---|---|---|---|
| Built by | Z.ai — China | DeepSeek — China | Mistral — France | Alibaba — China | OpenAI — US |
| Size | 744B40B active | 1.6T Pro · 284B Flash49B / 13B active | 675B41B active | 397B17B active | 117B5B active |
| License | MIT | MIT | Apache 2.0 | Apache 2.0 | Apache 2.0 |
| Context window | 1M | 1M | 256K | 262K | 128K |
| Intelligence IndexArtificial Analysis v4.1 | 51open-weight #1 | 44 | 45 | ||
| Known for | Long-horizon coding and agentic work | Reasoning and math at the lowest cost | European vendor, strong multilingual | Vision + documents, fine-tuning ecosystem | Near-frontier quality on one GPU |
| Runs on | One 8×H200 at FP8~$370k · a 4-bit build fits one 8×H100, ~$250–320k | 2×H200 (Flash) to a full node (Pro)~$90–420k | One 8×H100 server~$250–320k | One 8×H100 server~$250–320k | One H100 GPU~$35–50k |
| Weak spot | Text-only; token-hungry per answer | Text-only; prose weaker than technical work | A step behind the leaders on raw benchmarks | Trails on the hardest coding | Eleven months old — ancient in this market |
Two things about this table surprise most executives. First, the ceiling: the best open-weight models — a few points off the global frontier — each fit on a single eight-GPU server. Sovereignty at the top of the open leaderboard is a ~$370k line item, not a datacenter program. Second, the floor: gpt-oss-120b runs on one GPU, carries a license your counsel can read in an afternoon, and covers support drafting, compliance Q&A, and multilingual service at near-frontier quality — hardware a mid-size PSP approves without a committee. Below all of this sits the departmental class — Qwen3.6-27B, Gemma 4, Ministral 3, and sub-1B document extractors on a ~$20k server — which is where document pipelines belong anyway.
Self-hosting vs paying per token: the economics
The per-token price gap between open and commercial models is the least discussed number in this market, and it is not subtle: hosted DeepSeek V4-Pro output costs about $0.87 per million tokens against ~$25 for Claude-Opus-class commercial output — a factor of thirty before you negotiate anything. Running the model yourself comes down to one decision — buy the box or rent it — and the math by tier looks like this:
| Buy — one-time | Rent — per month | |
|---|---|---|
| DepartmentalQwen3.6-27B · documents, KYC triage | $18–30kone 96GB-GPU server | ~$1–1.5kone GPU, reserved |
| Workhorsegpt-oss-120b · support drafting, compliance Q&A | $35–50ksingle-H100 server | ~$1.5–2kone H100, reserved |
| Frontier-classGLM-5.2, DeepSeek V4 · AML narratives, agentic ops | $250–420k8×H100 (4-bit) → 8×H200 (FP8) | ~$12–14k8-GPU node, ~$16–20/hr reserved |
| On top, either way | Power, people, redundancy≈ +15–25% of hardware / year | People onlypower and cooling are in the rate |
The buy column assumes you already run your own racks — most banks do; if you don't, the rent column is the same sovereignty with a monthly bill instead of a capex line. Either way the contrast with a commercial API is the same: at bank volumes — a dispute desk alone can burn tens of millions of tokens a month — an API line item grows linearly forever, while owned or rented capacity is a flat number your CFO can plan around.
To be clear about scale: the $370k box is the ceiling of this market, not the ticket price. Most engagements we run start on a $18–50k server or a rented node at ~$13k a month, and never need more — the eight-GPU purchase is a decision you earn your way into with volume, not a prerequisite. Nobody should be buying frontier hardware before a single queue has proven it needs frontier capability.
Which model for which banking workload: match the queue, not the leaderboard
The mistake we see is choosing one model for the institution. Choose per queue instead:
- KYB and document intake go to the small specialists — a sub-1B extractor or Qwen3.6-27B on a workstation-class server. They beat the flagships here anyway; buying frontier capability for OCR is paying for reasoning you'll never invoke.
- Support, disputes, payment ops go to gpt-oss-120b on a single GPU. The benchmark that matters — multi-turn tool use — is at parity, and the human sign-off stays where your regulator expects it regardless of the logo on the model.
- AML investigations and representment narratives go to GLM-5.2 or DeepSeek V4. This is where the extra reasoning depth pays, and where the case for the 8×H200 server is made — one box, MIT-licensed weights, every token generated inside the building.
- Long-horizon autonomous work stays on closed models — via an EU sovereign-cloud arrangement with contractual zero retention, if your regulator accepts one, or it waits. A worse model running unsupervised for hours is not a sovereignty win; it's an incident report with extra steps.
For European institutions there is also a vendor-geography question that has nothing to do with capability. Mistral is the only top-tier European lab shipping permissive open weights, its flagship is Apache 2.0, and HSBC-class institutions already run it in production — that combination reads well in procurement and in front of EU supervisors. The strongest models on the leaderboard are Chinese-built and MIT-licensed; since self-hosted weights move no data anywhere, the residual concern is model-risk review, not data transfer. Just don't let anyone treat nationality as a proxy for license terms — the two costliest shortcuts we've seen were both a legal team assuming "open" meant "ours to use."
The on-premise LLM deployment strategy: five steps, in order
The sequence matters more than the shopping list. Ours, across every sovereign engagement this year:
- Classify queues before models. Split every AI-candidate workflow by data class: what could legally run on a sovereign-region API with zero retention, and what must never leave. Most institutions discover the "must never leave" pile is smaller than assumed — and that's your hardware budget, right-sized.
- Start departmental, on one queue. One document pipeline, one $20–30k server, one Apache-2.0 model, production data from the first week — with human review at the same gates as today. This is the overlay discipline: the queue's weekly numbers, not the model's benchmark, decide what happens next.
- Escalate tiers on evidence. Move a queue up a tier when its own metrics demand capability the current tier can't deliver — not when a new leaderboard drops. The 8×H200 purchase is easy to defend when it arrives with twelve weeks of queue data showing exactly which cases the workhorse tier dropped.
- Treat weights as cattle, not pets. The open-weight leader changed three times in the first half of this year. Buy hardware for the class, not the model; keep your evaluation harness ready to re-run quarterly; assume the model you deploy in Q3 is not the one you'll run next summer. The switching cost, done right, is a weekend.
- If capex is the blocker, rent the perimeter. On-prem-as-a-service (hardware in your datacenter, billed monthly) and EU sovereign clouds renting GPU capacity both keep data inside the compliance boundary without a balance-sheet event. They're also the honest middle path while the hardware order is in procurement.
When you don't need on-premise AI at all
Say the quiet part: if your regulator and your data classification genuinely accept a sovereign-region commercial API with contractual zero retention, you may not need to own a single GPU. We have told clients exactly that, and it was the right answer for a third of their queues. Sovereignty is a portfolio decision, revisited quarterly — not an identity.
But for the queues where the answer is "the data stays" — and every bank and PSP has them — the excuse expired sometime this spring. The gap to the frontier is one generation and stable. The best open models fit on one server. The licenses are readable. What's left is the part that was always the real work: picking the queue, naming the owner, and running it by the numbers.
The verdict: which model to deploy for which job
If you skip everything else in this memo, take the decision table. One row per plan, one winner per row — ties broken by license simplicity and hardware cost:
| Deploy | On | |
|---|---|---|
| Agentic ops workflows and serious codingdispute automation, internal tooling | GLM-5.2MIT | 8×H200 — or 8×H100 with a 4-bit build~$250–370k to buy · ~$16–20/hr to rent |
| Math-heavy casework at the lowest costAML scoring, reconciliation analysis | DeepSeek V4MIT · start with Flash, grow to Pro | 2×H200 and upfrom ~$90k |
| EU procurement and European languagessupervisors, works councils, optics | Mistral Large 3Apache 2.0 · French vendor | One 8×H100 server~$250–320k |
| Scanned documents, vision, fine-tuning plansKYB files, statements, invoices at scale | Qwen3.5-397BApache 2.0 | One 8×H100 server~$250–320k |
| A first production queue this quartersupport drafting, compliance Q&A | gpt-oss-120bApache 2.0 | One H100 GPU~$35–50k |
| Document extraction, and nothing else yetOCR, classification, triage | GLM-OCR or Qwen3.6-27BApache 2.0 | One workstation GPU~$18–30k |
Frequently asked questions
Can a bank use ChatGPT or Claude with customer data?
For restricted data classes — cardholder data under PCI DSS, AML case files, anything covered by banking secrecy — sending data to a commercial model API is usually not defensible to a regulator, regardless of the data-processing agreement. Those workloads need self-hosted open-weight models or a regulator-accepted sovereign-cloud arrangement with contractual zero retention.
Which open-weight LLM is best for on-premise deployment in 2026?
It depends on the workload tier. For document extraction and KYC triage: Qwen3.6-27B, Gemma 4, or Ministral 3 on a single-GPU server. For support drafting, compliance Q&A, and multilingual service: gpt-oss-120b — Apache 2.0, runs on one H100 GPU. For AML narratives and complex casework: GLM-5.2, DeepSeek V4, or Mistral Large 3, each of which fits on a single 8×H200 server.
What hardware do you need to self-host an LLM?
A 27–32B model runs on one 96GB-GPU server ($18–30k). gpt-oss-120b runs on a single H100 GPU ($35–50k per server). The strongest open-weight models — GLM-5.2, DeepSeek V4, Mistral Large 3 — run on one 8×H200 node (~$320–420k). Budget an extra 15–25% for networking, storage, power, and support.
How much worse are open-weight models than GPT or Claude?
About one model generation. On composite indexes the best open models score 44–51 vs ~60 for the closed frontier. On document extraction, open specialized models beat the commercial flagships; on multi-turn tool use they are at parity; on graduate-level reasoning the gap is 3–4 points. The gap is only decisive for hours-long autonomous agents and extreme long-context retrieval.