Synthetic support tickets: how urgent is this ticket?

Dataset

Name

tickets

Source

Tobi-Bueck/customer-support-tickets

Pinned revision

ddf1c81a5475992c4fa6752bf1e8b4e31f07bbeb

Licence

CC-BY-NC-4.0

Synthetic

Yes: generated data

Fine-tune split

8,000 items

Calibration split

1,000 items

Held-out split

1,000 items

Question

urgency (score, 5 answers): How urgent is this customer support ticket?

Calibrators and thresholds are fitted on the calibration split only and every result is judged on the held-out split, which nothing was tuned on. The splits are fixed by the dataset manifest (seed 42), and each file's sha256 is checked before use.

Agreement with the frontier model and calibration, before and after

Held-out items only. ECE (expected calibration error) is the average gap between how confident a model is and how often it is right, over 15 confidence bins; lower is better and 0 is perfect. Confidence for ECE and thresholds is max(p), the probability of the chosen answer, not the contract's confidence field, which for choice and score questions is not a calibrated probability. Here a model is right when its answer matches claude-opus-5-5's, so the agreement columns are agreement with the frontier model, not accuracy, and ECE measures whether a model's confidence says when it agrees. The goal is a 50% cut in ECE. The largest cut is laya-typed-decisions's (76.3%) and the smallest is von-1.2.0's (6.2%); a model whose confidence is still off after calibration should not be trusted to gate decisions on its own.

Reliability diagrams

Each point is one confidence bin on the held-out split: how confident the model was (across) against how often it was right (up), where right means agreeing with the frontier model. Points below the diagonal are overconfident, points above it underconfident. After calibration the points should sit close to the diagonal; where they don't, a threshold on that model's confidence will not deliver the agreement it promises.

Calibrators fitted

Both methods are fitted on the calibration split's raw probabilities, and the one with the lower calibration-split log loss is written for the Runtime to load; the scores in this table are on the calibration split, so they show the fit, not the result (the held-out result is in the first table). Isotonic regression needs at least 200 items and falls back to temperature scaling below that.

• laya-en: A spec asks one question, so every item has 5 options and the 3-5 bucket calibrator is fitted on the same items as the question-type one. The Runtime prefers the bucket file for 5-option requests; the type-level file covers other option counts.
• laya-typed-decisions: A spec asks one question, so every item has 5 options and the 3-5 bucket calibrator is fitted on the same items as the question-type one. The Runtime prefers the bucket file for 5-option requests; the type-level file covers other option counts.
• von-1.2.0: A spec asks one question, so every item has 5 options and the 3-5 bucket calibrator is fitted on the same items as the question-type one. The Runtime prefers the bucket file for 5-option requests; the type-level file covers other option counts.
• laya-en-ft-tickets: A spec asks one question, so every item has 5 options and the 3-5 bucket calibrator is fitted on the same items as the question-type one. The Runtime prefers the bucket file for 5-option requests; the type-level file covers other option counts.
• jev-1.13: Offline only: a hosted endpoint can't load calibrators and has no model hash, so its calibrators are applied by the Workbench only, and its calibrated view is the offline one. The files end in .offline-calibrator.json, which the Runtime never loads.
• jev-1.13: Probabilities rounded to 2 dp by the endpoint; calibrators fitted on rounded values.
• jev-1.13: A spec asks one question, so every item has 5 options and the 3-5 bucket calibrator is fitted on the same items as the question-type one. The Runtime prefers the bucket file for 5-option requests; the type-level file covers other option counts.

How much can stay local

Each line shows, on held-out items, what happens as the confidence threshold τ rises: fewer decisions are kept local (moving left) and those kept are more often right (moving up), where right means agreeing with the frontier model. The dot marks the τ chosen on the calibration split for a 20.0% target disagreement. A line that never reaches the target line means no threshold makes that model safe enough on its own at that target. Thresholds that keep fewer than 10 items are left off the chart because a handful of items says little; every point is in report.json. Dashed lines are baselines, not served through Tau, put through exactly the same threshold rule on their own calibration lines.

Cascade: local first, frontier for the rest

Every rate here is agreement with the frontier model on held-out items, not accuracy, with the number of items it covers in brackets. The cascade serves the local answer when its confidence is at least τ and the frontier model's cached answer otherwise; the cascade figure is the share of items where the served answer equals the frontier model's. Frontier only agrees with the frontier model on 100% of items by construction: its answers are the reference, so this is not a result. The closer the cascade gets to 100% while keeping a large share local, the better the local model stands in for the frontier call. Frontier latency is not measured: no API call was made, so the latency mix covers the local side only.

A row marked hosted endpoint, measured over the network puts a hosted endpoint where the local model would be: its "local" decision is a call to that endpoint, its latency is wall clock with the network included, and its cost rows price that call at the spec's price rather than GPU energy.

Estimated cost per million decisions, by where escalations go Estimates, not measured bills

Every figure in this table is an estimate from list prices and estimated token counts, not a measured bill. Each column is a frontier model the escalated decisions could go to: the first row is that model answering everything, and every other row is a local, hosted or baseline model answering what it's confident about and escalating the rest. Only the first column's model produced the frontier answers; the other columns change the price, not the accuracy. Basis:

• Frontier tokens are estimated, not counted: characters / 4 (Anthropic's rule of thumb) × 1.3 for the tokenizer, averaged over 1,000 cached answers: 328.7 input and 0.3 output tokens per decision.
• List prices in US dollars per million tokens, checked 2026-09-27 (platform.claude.com/docs/en/about-claude/pricing (checked 2026-09-27)).
• USD to GBP at 0.75458 (ECB euro foreign exchange reference rates, 2026-09-25).
• Frontier answers for this benchmark came from an interactive Claude Code session, not the API; the costs price what the same work would cost through the API.
• laya-en: Local energy: 343.5 W mean whole-GPU power × 0.01328 s per decision (phase duration / items) = 1.267e-6 kWh per decision, at £0.2632 per kWh (www.ofgem.gov.uk/news/changes-energy-price-cap-between-1-october-and-31-december-2026). Hardware purchase and depreciation are excluded.
• laya-en: Cascade: every decision runs locally, and 99.9% are also sent to the frontier model.
• laya-typed-decisions: Local energy: 345.6 W mean whole-GPU power × 0.01338 s per decision (phase duration / items) = 1.284e-6 kWh per decision, at £0.2632 per kWh (www.ofgem.gov.uk/news/changes-energy-price-cap-between-1-october-and-31-december-2026). Hardware purchase and depreciation are excluded.
• laya-typed-decisions: Cascade: every decision runs locally, and 99.9% are also sent to the frontier model.
• von-1.2.0: Local energy: 346.6 W mean whole-GPU power × 0.01126 s per decision (phase duration / items) = 1.084e-6 kWh per decision, at £0.2632 per kWh (www.ofgem.gov.uk/news/changes-energy-price-cap-between-1-october-and-31-december-2026). Hardware purchase and depreciation are excluded.
• von-1.2.0: Cascade: every decision runs locally, and 99.9% are also sent to the frontier model.
• jev-1.13 (hosted endpoint, measured over the network): The first-line model here is the hosted endpoint jev-1.13, not a local GPU, so each of its decisions is priced as a call: 430.4 input and 18 output tokens per decision (its measured usage) at $0.042 and $0 per million (the spec's estimate), $0.0000 per decision. No GPU energy is counted.
• jev-1.13 (hosted endpoint, measured over the network): Cascade: every decision goes to jev-1.13 first, and 60.2% are also sent to the frontier model.
• minilm-l6-tickets (baseline, not served through Tau: local latency and energy not measured): Local energy is not included: no GPU power was measured for the local model, so the cascade figures price the frontier calls only.
• minilm-l6-tickets (baseline, not served through Tau: local latency and energy not measured): Cascade: every decision runs locally, and 99.6% are also sent to the frontier model.

Frontier labels: noise and agreement

Label noise is how often claude-opus-5-5 disagrees with the dataset's gold label on the same item; it is measured before any human override. Some of that disagreement is the frontier model's error and some is the gold label's, so it says how far the dataset's labels can be trusted; this report scores against the frontier model instead, so it does not bound the figures above. Agreement between two wordings of the prompt shows how much the frontier answers depend on phrasing. Answers came from claude-code-session: subagent answer sheet; the prompt's fixed part shown once per batch of up to 200 items, each item judged on its own; no API, dated 2026-09-27 to 2026-09-27; no paid API call was made.

Secondary view: against the dataset's own labels

The dataset's labels are synthetic and close to arbitrary; the frontier model agrees with them on 23.8%. That is no better than the 41.0% a model would get by always giving the most common label. This table is a secondary view: every other figure in the report is agreement with the frontier model, because this example asks whether a local model can stand in for the frontier call.

Baselines

A classic small encoder, fine-tuned on the same training split, scored with exactly the same metric code as the Tau models. It is here to test the claim that a fixed task favours classic fine-tuning: where it beats a Tau model, that is listed under the misses, and it is a reason to prefer the classic model for this task.

Where the models go wrong

Rows are the frontier model's answer and columns the model's answer, so the diagonal is correct and everything else is a mistake; darker cells hold a larger share of their row. For an ordered scale, mistakes next to the diagonal are near misses and those far from it are serious; a model whose errors sit far off the diagonal should not be trusted on the extreme levels.

The misses

• Calibration cut laya-en's held-out ECE by only 43.4% (from 0.273 to 0.155), short of the 50% goal.
• Why laya-en fell short: the calibrator is chosen by lower calibration-split log loss, fixed before any held-out result. That picked isotonic (log loss 1.422, ECE 0.216) over temperature (log loss 1.581, ECE 0.016). Log loss and max(p) ECE disagree here, and the rule was not changed after seeing the held-out result.
• Calibration cut von-1.2.0's held-out ECE by only 6.2% (from 0.045 to 0.042), short of the 50% goal.
• Calibration helped von-1.2.0 least: 6.2% lower held-out ECE.
• Baseline minilm-l6-tickets beats laya-en on held-out agreement with the frontier model: 27.4% against 23.2%.
• After calibration, minilm-l6-tickets is also better calibrated than laya-en: ECE 0.015 against 0.155.
• Baseline minilm-l6-tickets beats laya-typed-decisions on held-out agreement with the frontier model: 27.4% against 18.4%.
• After calibration, minilm-l6-tickets is also better calibrated than laya-typed-decisions: ECE 0.015 against 0.064.
• After calibration, minilm-l6-tickets is also better calibrated than von-1.2.0: ECE 0.015 against 0.042.
• After calibration, minilm-l6-tickets is also better calibrated than laya-en-ft-tickets: ECE 0.015 against 0.072.
• After calibration, minilm-l6-tickets is also better calibrated than jev-1.13: ECE 0.015 against 0.122.
• laya-en-ft-tickets: No threshold meets the 20.0% target disagreement on the calibration split. The lowest disagreement any threshold reaches is 43.8%, at τ = 0.45, accepting 1.6% of items. No threshold is invented.
• Label noise: the frontier model disagrees with gold on 76.2% (762 of 1,000). This example is scored against the frontier model's answers instead of the dataset's labels; the figures against those labels are kept in a secondary table.

These are the results that don't flatter Tau: label noise, calibration that helped little, baselines that win, and anything excluded. They are published with the wins so the favourable numbers can be weighed against them.

Run metadata

Date (UTC)

2026-09-28T06:53:22Z

Command

tau report examples/support-tickets/decision.yaml

Git commit

3f09e89ad55185fab4c7ea33fa868ab6c5e5cebd (uncommitted changes outside this run's outputs)

Workbench

0.1.0

GPU

NVIDIA GeForce RTX 3080 Ti (driver 610.47)

CPU

11th Gen Intel(R) Core(TM) i9-11900K @ 3.50GHz, 16 logical processors

OS and runtime

Microsoft Windows 10.0.19045; .NET 10.0.11

Endpoint

localhost:18093/ (Tau Runtime at localhost:18093/ (4 model(s) installed))

/v1/models: laya-en

sha256 866a05b244e47e96820660d18ee050c518c2de0f60c39e2e9a89dfeb056e1eec, revision 55cf4c4ebb4ebe31b2550e8bdf3bd21b99753851

/v1/models: laya-typed-decisions

sha256 2b7a961ac37157d1cfa8105e3283106baf1ba2a5cc30fb3a673253f06aa1e1c5, revision 55cf4c4ebb4ebe31b2550e8bdf3bd21b99753851

/v1/models: von-1.2.0

sha256 0777bb988636663b770775ae0b4eb961d6fbee176c9bea1d1da823723b1eb3f3, revision 5df8185a4f2327ad0a7cd117cc4f701ac557b9ae

/v1/models: laya-en-ft-tickets

sha256 9079fee9291e35dc4d3ed8c817a8f1592d97e4413ccbae0539b7a0ee6d9c950a, revision 03677900b97d57980f7c525a84031b9a43cb989e

Measured model laya-en

866a05b244e47e96820660d18ee050c518c2de0f60c39e2e9a89dfeb056e1eec

Measured model laya-typed-decisions

2b7a961ac37157d1cfa8105e3283106baf1ba2a5cc30fb3a673253f06aa1e1c5

Measured model von-1.2.0

0777bb988636663b770775ae0b4eb961d6fbee176c9bea1d1da823723b1eb3f3

Measured model laya-en-ft-tickets

9079fee9291e35dc4d3ed8c817a8f1592d97e4413ccbae0539b7a0ee6d9c950a

Hosted endpoint jev-1.13

api.typesafe.ai/, requested model jev-latest, answered as jev-1.13.0 (no model hash: a hosted endpoint has no /v1/models hash, so the returned model string is its identity)

Hosted usage jev-1.13

857,667 input and 36,000 output tokens; estimated spend $0.0360 at $0.042 and $0 per million tokens (the spec's estimate, not a bill), against a $1.00 budget

Hosted headers jev-1.13

x-tau-raw and x-tau-precision: full were sent on every request, as for a local model. The endpoint echoed neither and returned no x-tau-* header, so it did not honour them: its answers are its own output, rounded to 2 dp.

Hosted latency jev-1.13

Wall-clock latency per call, network included: the time from sending the request to reading the whole response.

Dataset manifest sha256

46c1ec565c6c8dda61044250c224fc9a691688c44dd95a6ec87922bc7ca64943

Dataset revision (cache keys)

ddf1c81a5475

Reference labels

frontier: Scored against the frontier model: every rate is agreement with claude-opus-5-5's v1 answers (after any human override), not accuracy against the dataset's labels (data.reference: frontier).

Frontier model

claude-opus-5-5

Prompt versions

v1, v1-alt

Frontier cache

v1: 2,000 answers, v1-alt: 200 answers

Confidence

Confidence for ECE and thresholds is max(p), the probability of the chosen answer, not the contract's confidence field, which for choice and score questions is not a calibrated probability.

This block ties every figure above to a machine, a model hash, a dataset revision, a date and a command. Re-running the command on the same checkout and models should reproduce the numbers; anything that differs should be explained by what changed here.