Benchmark · Version 1.0.1 · 19 October 2026, updated 29 October

BudgetBench™

BudgetBench tests budgeting chat assistants. A person asks about their own transactions, and the assistant answers. BudgetBench checks every figure in every answer. It counts wrong figures separately from right answers.

What it measures

An assistant for money has two duties. First, it must never show a wrong figure. A wrong total that looks right is worse than no answer. Second, it must give the right answer.

Many tests count only right answers. Then a system that guesses can score well and still show wrong figures. BudgetBench gives each case one of six labels. It reports the six labels separately and never adds them into one score.

A case is one conversation of 1 to 4 questions. A case is right only when every turn is right. A seventh state, "needs review", is not a result. It means that the grader could not place a figure. A person then reads the answer and writes a ruling.

The data

Version 1 of BudgetBench has a public split and a first sealed split. Two more sealed splits came after it. The figures in the next five paragraphs are for version 1.

18 households. A household is a set of made-up transactions in the real shape of a bank or app export. The shapes are a connected bank, Monarch, YNAB, Chase, American Express, Capital One, Copilot, German bank and card files in euros, and a plain CSV. Two households have accounts in two currencies. One household is the ledger of an earlier test of ours. Each household covers four months, and the last month is still running.

1,060 cases. The questions ask about totals, shops, months and comparisons. Others ask for a figure that leaves something out, or follow up on an earlier answer. Some ask how to use the app, some ask for a decision that the assistant must decline, and some ask about currencies.

Right answers from an independent program. A program called the oracle computes every right answer from the true records of the household. It shares no code with Pulse Local. It shares only a CSV line reader with the tools of the stock-model rows. On the 176 oldest questions, an older, separate oracle gives the same 176 answers.

A grader with no AI judge. A grader made of written rules reads every turn. It checks every money figure in the answer against all true figures of the household, in the text and in the tables. Its own test has 5,031 checks on the public cases, and 7,236 with the sealed cases. A run counts only when that test passes.

Writers who never saw the app. Independent writers wrote 566 of the cases: 229 public and 337 in the first sealed split. Each writer was a separate Claude session in an empty folder. It saw one household with its true records and a persona. It also saw the writing rules, the public help pages of Pulse Local, and a validator that holds the oracle. It saw no app code. A second session then read each case and flagged doubtful ones. We removed 2 cases and edited none.

Three more sealed splits. v2 sealed has 340 cases on 6 households, v3 sealed has 387 cases on 8 households, and v4 sealed has 380 cases on 8 households. Their households use other months, and they add accounts in euros, pounds, Swedish kronor and Norwegian kroner. They were written in the same way: one new writer for each household, a packet with no app code, and a validator. One more session read every case. We removed 5 cases from v2, 7 from v3 and 14 from v4, and edited none. The v2, v3 and v4 writers were Claude agents that also got a general description of our company and products. The version 1 writers did not get it.

The sealed splits

A score means something only if the system did not learn the test. Thus BudgetBench has a public split and sealed splits.

Public: 723 cases on 12 households. Anyone can read them, run them and tune a system on them. A public score is a development score.

Sealed: four splits so far. The first has 337 cases on 6 households, v2 has 340 cases on 6 households, v3 has 387 cases on 8 households, and v4 has 380 cases on 8 households. The sealed households use made-up shop names and their own dates. Only the SHA-256 hash of each sealed file is public. When we release a sealed split, anyone can check its files against these hashes.

The rules for the sealed splits:

• Each new accuracy claim gets a new sealed split, with new households and new writers. Before the run, we write down what the test must show.
• We run each setting of Pulse Local once on a sealed split. We run it again only after a harness failure, such as a crash.
• After a claim, its sealed split is used up, and the next claim needs a new split. Later, a used split joins the public split.
• Nobody reads sealed answers to change the app. If somebody does, we mark the split as exposed.
• Claude sessions wrote and checked the sealed cases. On the first sealed split, a Claude session also started the runs, and it read some answers to rule on the rows that the grader could not place. No system under test got a sealed case through an API.
• Only counts, rates and intervals leave a sealed run. We publish no case, question or answer.

Leaderboard

We added the two Pulse Local 0.4.19 rows on 25 October 2026. The same grader, version 1.0.1, graded every row in these tables.

Each cell gives the share of cases with that label. Under it are the number of cases and a 95% interval. The interval comes from a bootstrap that picks households, then cases. We use households because the cases of one household share shops and wordings.

The next table keeps only the 229 public cases from the independent writers. We ran the 0.4.16 rows before anyone changed the chat code because of its answers to these cases. The 0.4.19 rows are different: while we fixed the app after 0.4.16, we read its answers to some of these cases. Thus the 0.4.19 rows are development scores too. Nobody tuned the stock models.

The sealed table holds the runs of Pulse Local 0.4.16 on the first sealed split. For a sealed split we publish only these counts, rates and intervals, never a case or an answer. The claim for Pulse Local 0.4.22 comes from a later sealed split, v4. See The chat in Pulse Local 0.4.22.

The tables show the labels of the grader, which anyone can compute again. A reviewer then read every "needs review" row and wrote a ruling. For version 1, the reviewer was a Claude session that works for us. The rulings are in each run folder. After that review:

• Pulse Local 0.4.16, rules, public split: 39 wrong shown, 605 right with no tap.
• Pulse Local 0.4.16, rules, writers split: 35 wrong shown, 124 right with no tap.
• Pulse Local 0.4.19, rules, public split: 31 wrong shown, 619 right with no tap.
• Pulse Local 0.4.19, rules, writers split: 26 wrong shown, 138 right with no tap.
• Pulse Local 0.4.19, rules + 0.6B checker (agree or not sure), public split: 5 wrong shown, 560 right with no tap.
• Pulse Local 0.4.19, rules + 0.6B checker (agree or not sure), writers split: 5 wrong shown, 117 right with no tap.
• Qwen3 0.6B (stock, Q4_K_M, thinking off) + reference tools, public split: 207 wrong shown, 55 right with no tap.
• Qwen3 0.6B (stock, Q4_K_M, thinking off) + reference tools, writers split: 51 wrong shown, 12 right with no tap.
• Qwen3 4B (stock, Q4_K_M, thinking off) + reference tools, public split: 321 wrong shown, 162 right with no tap.
• Qwen3 4B (stock, Q4_K_M, thinking off) + reference tools, writers split: 98 wrong shown, 43 right with no tap.
• Qwen3 8B (stock, Q4_K_M, thinking off) + reference tools, public split: 282 wrong shown, 289 right with no tap.
• Qwen3 8B (stock, Q4_K_M, thinking off) + reference tools, writers split: 116 wrong shown, 53 right with no tap.
• Pulse Local 0.4.16, rules, sealed split: 80 wrong shown, 181 right with no tap.

How to read the rows:

• In these row names, rules means the app's own built-in code, not rules that a person makes in the app. Pulse Local, rules is the app with the check off: the "Better understanding" setting is off. Its own code answers, and its small tuned model answers the questions that the code does not support.
• Pulse Local 0.4.16, rules + 4B checker is the app as 0.4.16 shipped it, with "Better understanding" on. The checker was a second model, Qwen3-4B-Instruct-2507. It also read every message. The app showed a figure only when the two readings agreed. If they did not agree, the app said that it was not sure. Since 0.4.19 a smaller checker does this job.
• Pulse Local 0.4.19, rules + 0.6B checker is the app as 0.4.19 shipped it, with "Better understanding" on, the default on every machine. The checker is a Qwen3-0.6B that we trained to read questions about money. It does the same job as the 4B did. Pulse Local 0.4.22 keeps this checker and fixes the app's own code. We developed those fixes on the public cases too, so we did not add a public row for 0.4.22.
• Stock models are Qwen3 0.6B, 4B and 8B, not tuned for this task. Each gets the same instructions and the same tools to read the household. These models read parsed rows. Pulse Local reads the raw bank files.
• Tuned on public tells if the system used the public split during its development.
• No hosted model. BudgetBench runs no paid hosted model and publishes no such row. You can run one with your own key and send us the row.

The chat in Pulse Local 0.4.22

After the v3 run, we opened the first three sealed splits as development material, and they now count as exposed. We used their conversations to find where the app's own code read a question wrong, and we fixed those rules in six rounds. The checker did not change. Before the first change, we built v4 sealed, wrote down the test and its bar, and froze both. Then we ran the test once on v4 sealed: 380 cases that no system had seen. The app ran in three settings: Pulse Local 0.4.21 as it shipped, the fixed code with the check on (Pulse Local 0.4.22), and the fixed code with the check off.

With the check on, about 5 conversations in 100 of these 380 still showed a wrong figure. With the check off, about 22 in 100 did. Pulse Local 0.4.21 showed a wrong figure in about 8 in 100. The 95% interval of the first figure is 2.3% to 7.4%. The interval comes from a bootstrap that picks households, then cases.

The bar was: fewer wrong cases than 0.4.21, and at least 95% of its right cases. Result: passed. Case by case, every conversation that 0.4.22 got wrong was also wrong for 0.4.21. After this run, v4 sealed is used up. The next claim needs new sealed cases.

The checker in Pulse Local 0.4.19

In Pulse Local 0.4.19, a Qwen3-0.6B that we trained to read questions about money replaced the 4B checker. Before the run, we wrote down the test and its bar. Then we ran the test once on a third sealed split, v3 sealed: 387 cases that no system had seen. The app ran in three settings on the same code: the check off (code only), the code with the 4B checker of 0.4.18, and the code with the new checker.

With the new checker on, about 6 conversations in 100 of these 387 still showed a wrong figure. With the check off, about 23 in 100 did. The 95% interval of the first figure is 2.6% to 9.4%. The interval comes from a bootstrap that picks households, then cases.

The bar was: no more wrong cases than the 4B, and at least 90% of its right cases. Result: passed. After this run, v3 sealed is used up: it now counts as exposed. The next claim about a checker needs new sealed cases.

Earlier results, kept as history:

• 20 October 2026, first sealed split, 337 cases, Pulse Local 0.4.16. With the 4B checker, 23 cases showed a wrong figure: about 7 in 100. With the check off, 79: about 23 in 100. As we reported them then, these two counts are wrong figures shown only; each run also had 1 case that offered only wrong choices. The sealed table above holds these runs.
• 22 October 2026, v2 sealed, 340 cases, the code of 0.4.18. With the check off, the app had 73 wrong cases and 165 right. The 4B checker had 17 wrong and 103 right. The 0.6B checker had 16 wrong and 140 right. It took about 1.1 seconds a turn, against 11.7 for the 4B. This result decided the switch.

Reproduce

The dataset on Hugging Face holds the public cases and households, the oracle, the grader and the runner. It also holds every public run, with each answer in full. From the folder that holds the dataset, run:

# First, check the files and the grader.
node budgetbench/check.mjs

# Run the three stock Qwen3 rows.
budgetbench/runner/serve-local.sh public

# Grade a saved run again.
node budgetbench/stats/report.mjs <run folder> --regrade

The Pulse Local rows run the app's own code, which is not public. You can still check them. Each run folder keeps every answer, and the last command above computes every label again from those answers.

Submit a system

• Write a system file. Or use the runner for any OpenAI-compatible endpoint, with your own key in your own environment. The runner never writes the key to a file.
• Run the check. Then run the public split.
• Send the run folder, the system file or the package version, your hardware, the time and, for a paid model, the cost.
• Tell us if the system, its prompts or its training data ever used the public split. We print the answer in the "tuned on public" line of the row.

We compute every label again from your answers, and a person reads every wrong row. We run the sealed split only on our own machine. Thus a system that needs a hosted API gets no sealed row. The full rules are in the submission guide of the dataset.

Limits

• The households are made up. A score here is not a score on real records.
• 9 of the 12 public households and 4 of the 6 sealed households are in US dollars only. All questions are in English.
• Claude sessions wrote the 566 cases of the independent writers, not real people. A system from the same model family may find those wordings easier. The other public cases come from our earlier tests.
• The grader reads text by rules. It can miss a wrong claim that has no figure.
• "Right after one tap" uses the answer key to choose the option. It is the best that a person could do, so it is an upper limit.
• The team that makes Pulse Local made BudgetBench. Read the Pulse Local rows with that in mind.

Cite

@misc{budgetbench2026,
  title        = {BudgetBench: Measuring Wrong Figures in Budgeting Chat Assistants},
  author       = {Grira, Nizar and {Pulse Local team}},
  year         = {2026},
  version      = {1.0.1},
  howpublished = {\url{https://huggingface.co/datasets/PulseOS/budgetbench}}
}

Licence

• The code (the household generator, the oracle, the grader, the runner and the reports): Apache License 2.0.
• The data (households, cases, right answers and results): Creative Commons Attribution 4.0 (CC BY 4.0).
• The copy of the Pulse Local help pages that the how-to cases use: we have not yet confirmed that CC BY 4.0 covers it. Ask us before you reuse it.
• The sealed splits are not released.

What the score says about Pulse Local, and what it does not say: read the article. Questions: niz@pulseos.finance.

BudgetBench is a trademark of Pulse Browser, Inc.