How we improved GPT-5.5 by 24% on τ-Banking with our learning layer
Our learning layer makes frontier models measurably more reliable on the hardest agentic tasks, with no fine-tuning and no change to the underlying model.
July 30, 2026
We improved GPT-5.5's performance on τ-Banking by 5.4 percentage points, from 0.223 to 0.277 Pass^1 using Kairos Telos, our in-house learning layer. GPT-5.5's task coverage rises from 37% to 50% across four trials, without fine-tuning or changing the underlying model, allowing the same agent to achieve this accuracy at an 18% higher reward per dollar.
Telos also makes a self-hosted Qwen3.6-27B competitive with frontier models, matching Gemini 3.1 Pro's task coverage, while delivering 2x the reliability, and running 1.9x cheaper.
The benchmark
τ²-bench, introduced by Sierra Research, is regarded as the leading benchmark for evaluating how well a model operates as a customer-service agent. The agent has to retrieve information, follow policy, and call tools with the right arguments to drive the conversation to the correct end state.
We focus on the banking_knowledge domain, introduced in Sierra's τ³ / τ-Knowledge. Banking is frequently regarded as the hardest domain, being 3 to 4x harder than retail, airline, or telecom. In Artificial Analysis's independent evaluations, top models score around 98-99% on τ²-Telecom but only 33.4% on τ³-Banking. In the banking domain specifically, the agent is given 698 complex banking documents and is required to answer queries with up to 33 tool calls across refunds, support, and services.
In this benchmark, the success of a task is determined by whether the agent reaches a target database end state. This means that the agent can make all the right decisions, but an incorrect required action or early termination can result in a score of 0. On top of that, the user is represented by another LLM with its own instructions acting on a shared state, so it isn't a static script. Some scenarios deliberately introduce persistent, resistant, or misleading users to create edge cases the agent must handle.
Finally, most of the tool calls aren't given to the agent by default, they're stored in the knowledge base as plain text. The model has to find the right documents, notice a tool name, then invoke the tool. When agents are given golden retrieval, meaning the exact correct documents are handed to the model, GPT-5.5 still reaches only 39%. This highlights that the remaining improvement is only in execution, not better retrieval methods.
An example task
A customer at an ATM has three cards decline and needs an urgent limit increase. The agent has to verify identity and make three different calls: raise the one eligible card, deny the one with a recent overdraft, and refuse the teen account on policy.
Run under alltools retrieval.
Reward 0.0Hi! How can I help you today?
Hi, I need help. I’m at an ATM right now and I’ve had THREE different debit cards decline on me. My Green Account card, my Blue Account card, and my daughter’s teen account card. I really need cash — my car broke down and I need to pay the tow truck driver who only takes cash!
I'm so sorry to hear that you're stranded and having trouble getting cash from the ATM. I know how stressful that can be, especially when you need to pay for a tow truck right away.
To help you get this sorted out as quickly as possible, I'll need to look into your account to see exactly why those cards are declining and see if we can clear any security blocks.
Could you please provide your full name along with two of the following pieces of information so I can verify your identity?
- •Date of birth
- •Email address
- •Phone number
- •Home address
Once I have that, I'll check the status of your debit cards right away!
David Martinez. DOB: 06/18/1983. Phone: 303-555-7294.
Reward 1.0Hi! How can I help you today?
Hi, I need help. I’m at an ATM right now and I’ve had THREE different debit cards decline on me. My Green Account card, my Blue Account card, and my daughter’s teen account card. I really need cash — my car broke down and I need to pay the tow truck driver who only takes cash!
I'm so sorry to hear that you're stranded and dealing with declined cards! I can certainly help look into this for you right away.
To get started and access your account details, could you please provide your full name, along with two of the following pieces of information: your date of birth, email address, phone number, or home address?
David Martinez. DOB: 06/18/1983. Phone: 303-555-7294. Email: david.martinez.cpa@gmail.com. Address: 4521 Mountain View Drive, Denver, CO 80203.
Thank you for verifying your information, David. I've looked into your accounts and I can see why the cards are declining.
It looks like you're hitting the daily ATM withdrawal limits on your accounts:
- •Green Account: The daily ATM limit is $600, and you've already withdrawn $350 today, leaving $250 available.
- •Blue Account: The daily ATM limit is $500, and you've withdrawn $400 today, leaving $100 available.
- •Teen Account (Light Green): These accounts have a strict daily ATM limit of $150.
Since you need cash urgently for the tow truck, I went ahead and applied a temporary 24-hour increase to your Green Account's ATM limit, raising it from $600 to $900. This means you now have $550 available to withdraw right now using your Green Account debit card!
(Note: I couldn't increase the Blue Account limit due to a recent overdraft fee, but the Green Account increase should hopefully cover what you need).
Please keep in mind that if you are using a non-Rho-Bank ATM, the machine itself might have its own withdrawal limits per transaction, so you may need to do a couple of smaller transactions to get the full amount.
Will this be enough to cover the tow truck?
Yes, that should cover it. I’ll try the Green card again for $550 (or split it up if the ATM makes me).
What about my other cards?
I'm so glad the Green card will cover the tow truck!
For your other cards, here is where they stand for the rest of the day:
- •Blue Account Card: You have $100 remaining of your $500 daily ATM limit. I wasn't able to temporarily increase the limit on this card because our system requires an account to have no overdraft fees within the last 30 days to qualify for a temporary boost.
- •Teen Account Card: This card has a strict daily ATM limit of $150. Because it is designed for minors, this limit is a built-in safety feature and cannot be temporarily increased.
Both of these limits will reset at midnight, so you'll have full access to their standard daily ATM limits again tomorrow.
Please let me know if the ATM gives you any trouble with the Green card, or if there is anything else I can help you with today!
Ah, that’s annoying. Okay, what about my daughter’s card?
She’s 16 — it’s the Light Green teen account. She tried to get $100 earlier for school supplies. Can you increase hers too?
I completely understand wanting to make sure she has what she needs for school supplies.
However, the Light Green teen accounts have strict, built-in safety limits that are fixed by the account type:
- •Daily ATM withdrawals: Capped at $150 per day.
- •Daily card purchases: Capped at $300 per day.
Because these are designed as safety features for minor accounts, our system does not allow us to temporarily increase these limits, even with your authorization as the guardian.
If her $100 withdrawal or purchase was declined earlier, it is likely because she had already used a portion of her daily limit earlier today. These limits will automatically reset at midnight tonight, at which point she will have access to her full $150 ATM and $300 purchase limits again.
I'm sorry I couldn't boost that one for you as well, but I hope the increase on your Green Account helps you get the tow truck sorted out! Is there anything else I can assist you with today?
Fine. Okay, I guess I’ll just give her cash from my card then.
Alright, Thanks for your help.
Gemini 3.1 Pro base passes 0 out of 4 trials. It spirals through dozens of blind tool calls trying to find the policy and times out before it ever grants the increase.
After applying Telos, the same model on the same setup solves it on 3 of 4 trials.
Telos
Telos is our experience-based learning layer. It turns agent experience into procedural memory that guides the agent on new tasks, without modifying the underlying model's weights. Approaches like fine-tuning and reinforcement learning also improve agents using task-specific data, but they do so by producing new weights. This requires a fresh rebuild, training pipeline, and budget every time a better base model ships.
Telos is built for this future, as the same loop is portable across model types and sizes. As the next frontier model ships, you can point Telos at it and keep your accumulated experience. The learned procedures are your operational knowledge stored in a format that you can easily version and modify. Your institutional knowledge compounds into an asset you own, rather than handing it to the model the industry shares.
Evaluation setup
We evaluated on a 46-task held-out test split from banking_knowledge.
The benchmark contains 97 tasks. We first removed task_010 before the train/test split because its baseline simulation failed with an infrastructure-level error and produced no gradable reward. The remaining 96 tasks were split 48/48 into train and test using a fixed random seed. All reported conditions were evaluated on the same held-out test set. Telos is built only from training-split traces, so the held-out test tasks remain fully disjoint and unseen. Of those 48 tasks, two further tasks are excluded from every condition: task_008 and task_064. Each lost trials to infrastructure errors in at least one arm, leaving unequal trial counts between base and Telos, so the two arms can't be compared reliably. Excluding these tasks keeps every configuration on an identical task set.
All conditions are evaluated on the same 46 tasks. Every comparison is a within-configuration A/B.
In July 2026, tau-bench released version v1.0.1 with grading fixes for banking_knowledge, so existing trajectories were regraded against the current task definitions with --fresh-tasks. Scores produced by earlier tau-bench versions are not directly comparable on this domain.
| Held fixed | Value |
|---|---|
| Domain | banking_knowledge (τ³ / τ-Knowledge) |
| Tasks | 46 held-out test ids |
| Trials | 4 per task |
| Seed | 300 |
| Timeout | 600s |
| Grading | Exact end-state (database) |
Our results are not a direct public leaderboard comparison. The public τ-Banking leaderboard uses the stronger alltools retrieval configuration, which combines bm25, dense embeddings, and shell access. We use bm25 in the main result to isolate the value Telos brings when retrieval is imperfect and the agent has to work with potentially noisy or incomplete information. We then test against the following retrieval methods to analyze performance across methods:
| Setting | What the agent gets | User simulator |
|---|---|---|
| Lean (bm25) | Keyword search only | gpt-5-mini |
| Strong (alltools) | bm25 + dense embeddings + shell | gpt-5.2 |
| Perfect (golden) | The correct documents, inlined | gpt-5-mini |
We report two main metric families: Pass^k, the probability that a task succeeds across all k independent trials, and Pass@k, which measures whether at least one of k trials succeeds. Pass^k rewards consistency rather than whether the model gets lucky once, while Pass@k is closer to a task-reachability metric. We also report reward per dollar, Pass^1 divided by the cost per trial, as a measure of how much task success each dollar of inference buys.
Results
Telos improves GPT-5.5 on every metric in the primary bm25 condition.
Figure 1: GPT-5.5 with Telos versus base on every Pass metric under lean bm25 retrieval. User simulator: gpt-5-mini, # of tasks: 46, # of trials: 4.
The Pass@4 jump from 0.370 to 0.500 shows the agent can now reach the correct end state on 50% of tasks compared to the original 37%. The gain in Pass^k also suggests that Telos enables the agent to complete tasks more reliably as well.
Telos adds about 5% to per-trial cost but returns about 24% more reward, so reward per dollar rises from 0.33 to 0.39, an 18% improvement. For teams running agents at scale, this is the number that is most practical, as more work is being completed for every dollar of inference.
Open-source model results
We also tested with Qwen3.6-27B, a self-hosted open model. For Qwen, the clearest lift appears in a separate alltools configuration, which uses a stronger retrieval and tool setup and a GPT-5.2 user simulator. This is different from the main GPT-5.5 bm25 result, so we treat it as a separate self-hosted model result rather than mixing it into the main comparison.
Figure 2: Self-hosted Qwen3.6-27B with and without Telos against Gemini 3.1 Pro under strong alltools retrieval. User simulator: gpt-5.2, # of tasks: 46, # of trials: 4.
Open-source models are becoming a strong alternative for enterprise deployment. As they improve, more teams will prefer to run agents inside their own cloud or on-prem, especially in industries where internal data can be very sensitive. Telos takes advantage of this shift by making it easy to securely deploy a learning loop on top of any model, including isolated open-source deployments inside a customer's own cloud.
Even with the full GPU serving bill included, the self-hosted open model comes out cheaper than the frontier API. This matters more than a per-seat comparison would suggest. Agentic workloads don't bill like SaaS, as a single task can cost thousands of tokens per decision, tracking work done rather than number of seats. At production volume the compute bill is what decides whether a workflow can run at all.
Figure 3: Pass^3 versus cost per trial under alltools, with dotted lines connecting each base model to the same model running our Telos harness. Qwen3.6-27B was served on 2×H100, and reported costs reflect per-second billing at the served concurrency. User simulator: gpt-5.2, # of tasks: 46, # of trials: 4.
Telos changes which model is now economically viable, as it lifts the cheapest and most ownable option, a 27B you host on your own infrastructure, into contention with a frontier API. Deployments can then keep your data and learning inside your walls, and also clear your budget requirements. Since the learning loops are now portable, you can ride the open-weight cost curve down as it keeps falling, pointing the same accumulated experience at a new and cheaper model without a rebuild.
We ran a golden-retrieval control, where each task's correct documents are piped directly into the context and no search is required. This makes retrieval perfect. We ran it two ways, base GPT-5.5 and with Telos:
All results use the same 46-task held-out test set and four trials per task. bm25 and golden use GPT-5-mini as the user simulator; alltools uses GPT-5.2. Therefore, scores should be compared base versus Telos within each retrieval regime, not directly across regimes.
Figure 4: GPT-5.5 base versus Telos across the three retrieval regimes: bm25, alltools, and golden. # of tasks: 46, # of trials: 4.
Where Telos helps
Telos does not produce a uniform lift. Its effect depends on how good the agent's retrieval already is, and on the model's execution capability.
Across GPT-5.5's three retrieval regimes the pattern is clear. Under lean bm25 retrieval, where the agent works from noisy, incomplete context, Telos improves every metric. Under strong alltools retrieval the gain narrows to a reliability trade-off. It makes existing successes more repeatable across trials rather than solving new tasks, so all-4-trial reliability rises while coverage slips. Some of the base model's wins are one-off lucky solves that don't survive a retry, and Telos trades those away for wins that hold across all four trials, which is what actually matters in production. Under golden retrieval, where the correct documents are already in context, it does not improve performance, and the observed scores are slightly lower.
Model capability sets the other boundary. A weaker open model shows the same shape one tier down. Under strong retrieval it improves across the board, but under lean retrieval it goes flat-to-negative. With thin context and a less capable base, the model has too little to work from and too little execution ability to benefit.
Takeaway
Telos improves frontier models on the hardest domain in the τ family without touching their weights, lifting GPT-5.5's Pass^1 by 24% under lean retrieval and expanding its task coverage from 37% to 50%. Production agents often fail during execution by losing track of user preferences and task state across long interactions. These results suggest that Telos can help agents carry decisions through to the correct end state when retrieval is imperfect.
It's most valuable where capable agents still struggle to turn imperfect information into reliable action: it expands capability under weak retrieval, improves consistency under strong retrieval, but does not improve performance when the right information is already provided. Because its learned procedures aren't baked into model weights, teams can inspect, refine, and carry their operational knowledge forward as models change.