Banking77: which of 77 intents is this customer message?

Dataset

Name

banking77

Source

PolyAI-LDN/task-specific-datasets

Pinned revision

9d081458ff52e53cf7e848f414e6e9344e4e6696

Licence

CC-BY-4.0

Synthetic

No

Fine-tune split

9,003 items

Calibration split

1,000 items

Held-out split

1,000 items

Question

intent (choice, 77 answers): Which banking customer-service intent does this message express?

Calibrators and thresholds are fitted on the calibration split only and every result is judged on the held-out split, which nothing was tuned on. The splits are fixed by the dataset manifest (seed 42), and each file's sha256 is checked before use.

Accuracy and calibration, before and after

Held-out items only. ECE (expected calibration error) is the average gap between how confident a model is and how often it is right, over 15 confidence bins; lower is better and 0 is perfect. Confidence for ECE and thresholds is max(p), the probability of the chosen answer, not the contract's confidence field, which for choice and score questions is not a calibrated probability. The goal is a 50% cut in ECE. The largest cut is laya-en's (87.1%) and the smallest is laya-en-ft-banking77's (13.6%); a model whose confidence is still off after calibration should not be trusted to gate decisions on its own.

Reliability diagrams

Each point is one confidence bin on the held-out split: how confident the model was (across) against how often it was right (up). Points below the diagonal are overconfident, points above it underconfident. After calibration the points should sit close to the diagonal; where they don't, a threshold on that model's confidence will not deliver the accuracy it promises.

Calibrators fitted

Both methods are fitted on the calibration split's raw probabilities, and the one with the lower calibration-split log loss is written for the Runtime to load; the scores in this table are on the calibration split, so they show the fit, not the result (the held-out result is in the first table). Isotonic regression needs at least 200 items and falls back to temperature scaling below that.

• laya-en: A spec asks one question, so every item has 77 options and the 11+ bucket calibrator is fitted on the same items as the question-type one. The Runtime prefers the bucket file for 77-option requests; the type-level file covers other option counts.
• von-1.2.0: A spec asks one question, so every item has 77 options and the 11+ bucket calibrator is fitted on the same items as the question-type one. The Runtime prefers the bucket file for 77-option requests; the type-level file covers other option counts.
• laya-en-ft-banking77: A spec asks one question, so every item has 77 options and the 11+ bucket calibrator is fitted on the same items as the question-type one. The Runtime prefers the bucket file for 77-option requests; the type-level file covers other option counts.

How much can stay local

Each line shows, on held-out items, what happens as the confidence threshold τ rises: fewer decisions are kept local (moving left) and those kept are more often right (moving up). The dot marks the τ chosen on the calibration split for a 5.0% target error. A line that never reaches the target line means no threshold makes that model safe enough on its own at that target. Thresholds that keep fewer than 10 items are left off the chart because a handful of items says little; every point is in report.json. Dashed lines are baselines, not served through Tau, put through exactly the same threshold rule on their own calibration lines.

Cascade: local first, frontier for the rest

Accuracy is on held-out items, with the number of items each figure covers in brackets. The cascade keeps the local answer when its confidence is at least τ and uses the frontier model's cached answer otherwise; an escalated item with no cached answer is counted and left out, never guessed. If the cascade is close to frontier-only accuracy while keeping a large share local, most of the frontier cost can be avoided. Frontier latency is not measured: no API call was made, so the latency mix covers the local side only.

Cost for laya-en Estimates, not measured bills

No cascade: no threshold met the target, so there is nothing to price.

Every figure in this table is an estimate from list prices and estimated token counts, not a measured bill. It shows what the same decisions would cost through the API, and what share of that the cascade avoids. Basis:

• Frontier tokens are estimated, not counted: characters / 4 (Anthropic's rule of thumb) × 1.3 for the tokenizer, averaged over 1,000 cached answers: 1258.9 input and 6.5 output tokens per decision.
• List prices in US dollars per million tokens, checked 2026-09-27 (platform.claude.com/docs/en/about-claude/pricing (checked 2026-09-27)).
• USD to GBP at 0.75458 (ECB euro foreign exchange reference rates, 2026-09-25).
• Frontier answers for this benchmark came from an interactive Claude Code session, not the API; the costs price what the same work would cost through the API.
• Local energy: 337.8 W mean whole-GPU power × 0.02446 s per decision (phase duration / items) = 2.295e-6 kWh per decision, at £0.2632 per kWh (www.ofgem.gov.uk/news/changes-energy-price-cap-between-1-october-and-31-december-2026). Hardware purchase and depreciation are excluded.

Cost for von-1.2.0 Estimates, not measured bills

Every figure in this table is an estimate from list prices and estimated token counts, not a measured bill. It shows what the same decisions would cost through the API, and what share of that the cascade avoids. Basis:

• Frontier tokens are estimated, not counted: characters / 4 (Anthropic's rule of thumb) × 1.3 for the tokenizer, averaged over 1,000 cached answers: 1258.9 input and 6.5 output tokens per decision.
• List prices in US dollars per million tokens, checked 2026-09-27 (platform.claude.com/docs/en/about-claude/pricing (checked 2026-09-27)).
• USD to GBP at 0.75458 (ECB euro foreign exchange reference rates, 2026-09-25).
• Frontier answers for this benchmark came from an interactive Claude Code session, not the API; the costs price what the same work would cost through the API.
• Local energy: 338.2 W mean whole-GPU power × 0.02318 s per decision (phase duration / items) = 2.178e-6 kWh per decision, at £0.2632 per kWh (www.ofgem.gov.uk/news/changes-energy-price-cap-between-1-october-and-31-december-2026). Hardware purchase and depreciation are excluded.
• Cascade: every decision runs locally, and 95.5% are also sent to the frontier model.

Cost for laya-en-ft-banking77 Estimates, not measured bills

Every figure in this table is an estimate from list prices and estimated token counts, not a measured bill. It shows what the same decisions would cost through the API, and what share of that the cascade avoids. Basis:

• Frontier tokens are estimated, not counted: characters / 4 (Anthropic's rule of thumb) × 1.3 for the tokenizer, averaged over 1,000 cached answers: 1258.9 input and 6.5 output tokens per decision.
• List prices in US dollars per million tokens, checked 2026-09-27 (platform.claude.com/docs/en/about-claude/pricing (checked 2026-09-27)).
• USD to GBP at 0.75458 (ECB euro foreign exchange reference rates, 2026-09-25).
• Frontier answers for this benchmark came from an interactive Claude Code session, not the API; the costs price what the same work would cost through the API.
• Local energy: 338.9 W mean whole-GPU power × 0.06121 s per decision (phase duration / items) = 5.763e-6 kWh per decision, at £0.2632 per kWh (www.ofgem.gov.uk/news/changes-energy-price-cap-between-1-october-and-31-december-2026). Hardware purchase and depreciation are excluded.
• Cascade: every decision runs locally, and 26.4% are also sent to the frontier model.

Cost for minilm-l6-banking77 (baseline, not served through Tau: local latency and energy not measured) Estimates, not measured bills

Every figure in this table is an estimate from list prices and estimated token counts, not a measured bill. It shows what the same decisions would cost through the API, and what share of that the cascade avoids. Basis:

• Frontier tokens are estimated, not counted: characters / 4 (Anthropic's rule of thumb) × 1.3 for the tokenizer, averaged over 1,000 cached answers: 1258.9 input and 6.5 output tokens per decision.
• List prices in US dollars per million tokens, checked 2026-09-27 (platform.claude.com/docs/en/about-claude/pricing (checked 2026-09-27)).
• USD to GBP at 0.75458 (ECB euro foreign exchange reference rates, 2026-09-25).
• Frontier answers for this benchmark came from an interactive Claude Code session, not the API; the costs price what the same work would cost through the API.
• Local energy is not included: no GPU power was measured for the local model, so the cascade figures price the frontier calls only.
• Cascade: every decision runs locally, and 5.3% are also sent to the frontier model.

Frontier labels: noise and agreement

Label noise is how often claude-opus-5-5 disagrees with the dataset's gold label on the same item; it is measured before any human override. Some of that disagreement is the frontier model's error and some is the gold label's, so it bounds how precisely any accuracy here can be read. Agreement between two wordings of the prompt shows how much the frontier answers depend on phrasing. Answers came from claude-code-session: subagent answer sheet; the prompt's fixed part shown once per batch of up to 200 items, each item judged on its own; no API, dated 2026-09-27 to 2026-09-27; no paid API call was made.

Baselines

A classic small encoder, fine-tuned on the same training split, scored with exactly the same metric code as the Tau models. It is here to test the claim that a fixed task favours classic fine-tuning: where it beats a Tau model, that is listed under the misses, and it is a reason to prefer the classic model for this task.

Where the models go wrong

laya-en: most frequent confusions (calibrated, 1,000 items)

von-1.2.0: most frequent confusions (calibrated, 1,000 items)

laya-en-ft-banking77: most frequent confusions (calibrated, 1,000 items)

With 77 answers a full matrix is too large to read, so these tables list the 15 most frequent mistakes per model. Pairs that recur across models are usually answers whose descriptions overlap, which is where clearer option descriptions would help most.

The misses

• Calibration cut laya-en-ft-banking77's held-out ECE by only 13.6% (from 0.071 to 0.061), short of the 50% goal.
• Calibration helped laya-en-ft-banking77 least: 13.6% lower held-out ECE.
• Baseline minilm-l6-banking77 beats laya-en on held-out accuracy: 91.5% against 37.2%.
• After calibration, minilm-l6-banking77 is also better calibrated than laya-en: ECE 0.024 against 0.065.
• Baseline minilm-l6-banking77 beats von-1.2.0 on held-out accuracy: 91.5% against 77.1%.
• After calibration, minilm-l6-banking77 is also better calibrated than von-1.2.0: ECE 0.024 against 0.039.
• Baseline minilm-l6-banking77 beats laya-en-ft-banking77 on held-out accuracy: 91.5% against 87.3%.
• After calibration, minilm-l6-banking77 is also better calibrated than laya-en-ft-banking77: ECE 0.024 against 0.061.
• Baseline minilm-l6-banking77 keeps 94.7% local against laya-en-ft-banking77's 73.6%, at 94.2% against 93.2% blended accuracy.
• laya-en: No threshold meets the 5.0% target error on the calibration split. The lowest error any threshold reaches is 37.9%, at τ = 0.54, accepting 28.2% of items. No threshold is invented.
• Label noise: the frontier model disagrees with gold on 5.8% (58 of 1,000).

These are the results that don't flatter Tau: label noise, calibration that helped little, baselines that win, and anything excluded. They are published with the wins so the favourable numbers can be weighed against them.

Run metadata

Date (UTC)

2026-09-27T17:15:19Z

Command

tau report examples/banking77/decision.yaml

Git commit

8954e9b095b40f68ee34139ca7c413fad4386300 (clean, outputs excluded)

Workbench

0.1.0

GPU

NVIDIA GeForce RTX 3080 Ti (driver 610.47)

CPU

11th Gen Intel(R) Core(TM) i9-11900K @ 3.50GHz, 16 logical processors

OS and runtime

Microsoft Windows 10.0.19045; .NET 10.0.11

Endpoint

localhost:18093/ (Tau Runtime at localhost:18093/ (3 model(s) installed))

/v1/models: laya-en

sha256 866a05b244e47e96820660d18ee050c518c2de0f60c39e2e9a89dfeb056e1eec, revision 55cf4c4ebb4ebe31b2550e8bdf3bd21b99753851

/v1/models: von-1.2.0

sha256 0777bb988636663b770775ae0b4eb961d6fbee176c9bea1d1da823723b1eb3f3, revision 5df8185a4f2327ad0a7cd117cc4f701ac557b9ae

/v1/models: laya-en-ft-banking77

sha256 198e563957ce43348f944a10739bdf17929bc3d2175f22a72239b4601d421556, revision 469442c64b974d12d1c397c6fab2baeeb1cdd8cc

Measured model laya-en

866a05b244e47e96820660d18ee050c518c2de0f60c39e2e9a89dfeb056e1eec

Measured model von-1.2.0

0777bb988636663b770775ae0b4eb961d6fbee176c9bea1d1da823723b1eb3f3

Measured model laya-en-ft-banking77

198e563957ce43348f944a10739bdf17929bc3d2175f22a72239b4601d421556

Dataset manifest sha256

7ff6d7b0974afb66a781fccbeae9f58db38a37ebd2a0398792d5f2550ac6c79e

Dataset revision (cache keys)

9d081458ff52

Reference labels

gold: Scored against the dataset's gold labels (data.reference: gold).

Frontier model

claude-opus-5-5

Prompt versions

v1, v1-alt

Frontier cache

v1: 1,000 answers, v1-alt: 200 answers

Confidence

Confidence for ECE and thresholds is max(p), the probability of the chosen answer, not the contract's confidence field, which for choice and score questions is not a calibrated probability.

This block ties every figure above to a machine, a model hash, a dataset revision, a date and a command. Re-running the command on the same checkout and models should reproduce the numbers; anything that differs should be explained by what changed here.