BudgetBench·Engineering · 20 October 2026, updated 27 October

How often does Pulse Local show a wrong figure?

Pulse Local checks every money answer before you see it. It reads your question in two separate ways, and it shows a figure only when the two readings agree. We tested this check on 380 conversations that Pulse Local had never seen. With the check on, about 5 conversations in 100 showed a wrong figure. With the check off, about 22 in 100 did. A conversation counts as wrong when any answer in it shows a wrong figure. When Pulse Local is not sure, it says so and asks you to name the shop, the category, the account or the month. We measure all of this with BudgetBench^™, a benchmark that we built and published. This article tells how the test works, what the numbers mean for you, and what they do not show.

What the numbers mean for you

The check is the setting Better understanding, which is on by default.

Out of every 100 conversations in that test:

• With the check on, about 43 were right, about 5 showed a wrong figure, and about 46 said "not sure" at some point.
• With the check off, about 48 were right, about 22 showed a wrong figure, and about 23 said "not sure".

So the check costs a few right answers, and it removes about three wrong figures in four. We think that is the right trade for money. You can act on "not sure": name the shop or the month, and ask again. A wrong total that looks right is much harder to catch.

The check uses a small model that we trained to read questions about money. On our third sealed test, on 25 October, we also ran our earlier and larger checker. The small model showed about as few wrong figures (23 against 24 of 387), it gave 70% more right answers (158 against 93), and it answered about ten times faster. It is also a 397 MB download, not 2.5 GB.

Why we built BudgetBench

Pulse Local answers questions about your money on your machine. The worst answer it can give is a wrong total that looks right. Thus we measure wrong figures first, and right answers second.

Our earlier tests had two problems. Some counted only right answers. A system that guesses can then score well and still show wrong figures. Other tests used questions that we read while we built the app. When we change the app until it passes a question, the score on that question goes up. The score on new questions often does not.

BudgetBench replaces our earlier tests with one benchmark that has rules against both problems. It is public. You can check our numbers and test your own assistant.

What it checks

A case is one conversation of 1 to 4 questions about one household. Each question and its answer is a turn. A household is a set of made-up transactions in the real shape of a bank or app export. BudgetBench gives each case one of six labels and reports them apart:

• Wrong shown: the answer shows a wrong figure, name or claim.
• Wrong offered: the assistant asks which reading you mean, and every option is wrong.
• Right, no tap: every turn is right.
• Right after one tap: every turn is right after one tap on an option that the assistant offers.
• Not sure: the assistant says that it is not sure, or asks a question back.
• Declined: the assistant declines, or answers something else with no figure.

A separate program computes every right answer from the true records. It shares no code with the app. A grader made of written rules, with no AI judge, checks every money figure in every answer. The BudgetBench page gives the full method.

How we keep the test honest

A score means something only if the app did not learn the test. Thus the number above comes from sealed cases: cases that we keep private, and that nobody reads to change the app. Independent writers wrote them. Each writer was a separate Claude session that saw one household and no app code. The sealed cases are not published, and nobody reads the app's answers to them to change the app.

Before the test, we wrote down what it must show and how we would say the result. Then we ran it once, on our machine, so no sealed case went to an outside service. After a test, its sealed cases count as used. The next claim needs new cases from new writers.

How the number changed in October

• 20 October 2026, the first sealed test, 337 conversations on 6 households. With our earlier and larger checker, 23 showed a wrong figure, about 7 in 100. With the check off, 79, about 23 in 100. As we reported them then, these two counts leave out the one conversation in each run that offered only wrong choices.
• 22 October 2026, a second sealed test, 340 new conversations. It compared the larger checker with the small one that we trained. The small one had 16 wrong and 140 right, against 17 and 103. It was also ten times faster, so it replaced the larger one.
• 25 October 2026, a third sealed test, 387 new conversations on 8 households. We wrote the test down first and ran it once. With the small checker on, about 6 in 100 showed a wrong figure. With the check off, about 23 in 100 did.
• 28 October 2026, a fourth sealed test, 380 new conversations on 8 households. After the third test, we opened the first three sealed tests and used their conversations to find where the app's own rules read a question wrong. We fixed those rules, and those tests now count as used. Then we ran the fourth test once. On its conversations, the app before the fixes showed a wrong figure in about 8 in 100, and the app with the fixes in about 5 in 100. Its result is the one at the top of this article.

Each test has new households and new writers. So a difference of a point or two between tests is not a finding. Two changes are clear because we measured them on the same conversations. On the third test, the small checker gave many more right answers than the larger one, much faster, with about as few wrong figures. On the fourth test, each conversation where the fixed app showed a wrong figure was also wrong for the app before the fixes, and the fixed app gave 6 more right answers.

When our earlier article first appeared, on 4 October, it reported about one wrong figure in a hundred on our own test households. The sealed tests found more, between about 5 and 8 in 100. They count whole conversations, not single questions, and their households and questions are harder.

Why the public cases show fewer wrong figures

On the public cases, with the check off, Pulse Local showed a wrong figure in 31 of 723 cases (4.3%; 95% interval 1.7% to 7.1%). With the check on: 5 of 723 cases (0.7%; 95% interval 0.0% to 1.9%). These rates are much lower than on the sealed tests. We see two reasons.

The first reason is exposure. 494 of the 723 public cases come from our own tests. 176 are the questions of an earlier test. 318 are template cases: the wordings of a release test of Pulse Local, asked on new households. We read the answers to these questions, or to the same wordings, while we changed the app. With the check off, Pulse Local got all 318 template cases right with no tap, and 163 of the 176 older questions.

The second reason is that the exposed cases are simpler. The template cases ask only about shops, months, totals and follow-ups. The writers' cases also ask about comparisons, exclusions, currencies and how to use the app.

The fairer part of the public cases is the 229 cases from the independent writers. On these cases, with the check off, Pulse Local showed a wrong figure in 26 of 229 cases (11.4%; 95% interval 6.3% to 17.1%). With the check on: 5 of 229 cases (2.2%; 95% interval 0.0% to 5.5%). We read some of the app's answers to these cases while we fixed it, so they are a development score too. They come from 4 households only, so the intervals are wide.

Thus we do not quote the public score as the score of the app. The sealed score is the release score.

Small open models with the same tools

We also ran three stock Qwen3 models, with 0.6, 4 and 8 billion parameters, on the public cases. Each model got the same instructions and the same tools to search and add the household's rows. Nobody tuned them for this task.

These rows show how hard the questions are for a general model of that size. They are not a contest with Pulse Local. BudgetBench runs no paid hosted model and publishes no such row. Anyone can run one with their own key and send us the row.

What the score does not show

• Your records. The households are made up. Each covers a few months, and the last month is still running. Real records have more shops, odder bank text and years of history.
• Every currency and language. 9 of the 12 public households are in US dollars only. The two latest sealed tests had 8 households each, in several currencies. All questions are in English.
• Real people. Claude sessions wrote the cases, each with a persona. Real people ask in other ways. A model from the same family may also find the wordings easier.
• Every kind of mistake. The grader reads text by rules. It can miss a wrong claim that has no figure. It can also call an answer wrong where a person may not agree. For example, it scores the latest Netflix charge as wrong when the question asks what a Netflix plan costs today. We do not change the label of such an answer in this version of BudgetBench. A later version can change the rule, and then we grade every row again.
• A small sample. Each sealed test has a few hundred conversations on 6 to 8 households. The interval of the 5 in 100 is 2.3% to 7.4%.
• An outside referee. We made BudgetBench, and we make Pulse Local. We publish every public answer in full, so that you can check every label.

What comes next

Each new claim gets a new sealed test, from new writers and new households. The old sealed cases then join the public cases, marked as exposed. We will report the trend over several tests, not the change from one test to the next.

Our next work is to win back right answers without new wrong figures. Today the app says "not sure" in about half of the conversations, and many of those questions had a clear answer. After that, we want questions from real people about their own money.

Every household in BudgetBench is made up, and nothing in it is anyone's real data. The cases, the grader and every public answer: BudgetBench. What to check before you rely on an answer: Accuracy and limits.

BudgetBench is a trademark of Pulse Browser, Inc.