How Pulse works·Engineering · 4 October 2026, updated 29 October
How a second reading cut wrong figures from about 22 to about 5 in 100 conversations
Pulse Local answers questions about your money with AI that runs on your machine. In test after test, about one answer in five was wrong, and nothing we tried moved that number. Then we changed what the app does when it is unsure. It now shows a figure only when two readings of your question agree. On 380 test conversations that it had never seen, about 5 conversations in 100 showed a wrong figure with this check on, and about 22 in 100 with the check off.
We did not get there with a bigger model or a cleverer prompt, though we tried both. We got there by reading every money question twice and showing a figure only when the two readings agree. This post covers the idea, what it costs and where the claim stops.
This post first appeared on 4 October 2026 and reported 5 wrong answers in 494 questions on nine test households. Larger tests since then, on conversations that nobody had read before the run, put the rate at about 5 conversations in 100. We keep the earlier numbers below, each with the date it was measured. The section Where the claim stops gives the current figures.

The problem
You ask "how much did we spend on groceries in August?" and expect a number you can act on. Pulse Local reads the question on your machine, with code we wrote by hand and a small AI model. Every figure it prints comes from a calculation over your own records. A separate guard, the figure guard, refuses any figure that calculation did not produce.
The figure guard makes sure a figure is real. It cannot make sure the figure answers your question. Ask about "the other card" and the app might take it to mean a spending category called Other. It prints a true total, for the wrong thing. Nearly every mistake in this post looks like that: the arithmetic is right and the question was misread.
How we test
We first tested the app on questions we wrote ourselves, and the scores crept toward 100%. Then a friend tried it and got seven wrong answers to seven questions. Our questions had measured how well the app fit them. So we built a test we could not tune the app to.
For each test, a fresh AI session that has never seen the app's code invents three households. It writes several months of transactions for each, then about 55 questions per household in the voice of the person who lives there. Typos, shorthand and follow-ups are all fair game. Here is a real two-question conversation from our eleventh test.
how many times did I eat at brindlecap in july? expected: 3 visits what did those visits cost me in total expected: $101.41
A separate program, which shares no code with the app, works out every expected answer from the transactions. We freeze the app, another session runs the test, a grader checks every figure in every reply, and a person reads every answer. A reply with one wrong figure in it is wrong. If the app asks "did you mean A or B?" and both are wrong, that is wrong too. Once a test's mistakes have shaped the app, it stops counting as evidence, and the next claim needs new households and a new author.
Eleven tests, one flat line
Between tests we read every wrong answer and fixed each pattern of mistakes in our code.
Wrong answers in each test, on households the app had never seen
The first two tests improved fast. Then the line went flat, between 16% and 20% wrong for eight tests, and the eleventh came in higher, at 23.5%. Each fix worked on the wordings it was written for, and each new author found new ways to ask. After one fix, every question about what a shop charges came out right. In the next test, written by someone new, one in seven came out wrong.

We also tried to read questions better: training our own small model, a larger model that rewrote each follow-up in full, and asking "did you mean A or B?" when readings differed. Each looked good on questions we had already seen. None held up on new households. The trained model scored 97% on questions we had generated and got a quarter wrong on new ones, because it had learned our phrasing. A model five times the download, on its own, answered 21 of 77 real questions correctly. Our small tuned model answered 35.
When we sorted 208 wrong answers from seven tests by cause, about 90% were reading mistakes: a follow-up that lost track of the conversation, a total where you asked for a price, money in taken for money out. There is no end to the ways a question can be phrased, so fixing one wording at a time never caught up.
The bug that hid the answer
Our ninth test tried a larger model that read every question alongside our code, and it showed no clear gain. In the logs, every wrong figure it let through came from a moment when the second reading had never been checked.
When the app asks "did you mean…?", it remembers the question so it can match your reply. That memory carried forward into every later answer, so the second-reading check thought the app was still asking and stood aside. After the first question back in a conversation, the check was off. The fix took a few lines.

With the check working, the same test showed us what to build. Our code and that model agreed on 137 answers. Of those, 135 were right, one declined to answer, one needed a person to grade it, and none was wrong. Every mistake came from what the app did when the two disagreed. So the app already had a signal for when it was right, and did nothing with it.
Agree or not sure
The new rule is short. Every money question is read twice, by two methods built in different ways. One is our own code. The other is a second reader, an open AI model that runs on your machine. The same code turns both readings into calculations. When they agree, you see the figure. When they differ, the app says "I am not sure which figure you mean, so I will not guess" and asks you to name the shop, category, account or month.

The second reader never writes an answer. It writes down what you asked for: what to count, about what, and over which dates. It can only name shops, categories and accounts that appear in your records, so it cannot invent one. It can confirm our code or stop it, nothing more. When you pick an option yourself, or tap a question the app suggested, the check steps aside, because the meaning is already settled.
Here is the kind of disagreement the check is for. Someone asked "if Cinemoth stays at this price, what's that a year?" Our code read it as a total so far and would have said $95.93. The second reader read it as a yearly cost and said $179.88. They disagreed, so the app said it was not sure. The second reader was right, and the disagreement showed us exactly what to fix in our code. Now both say $179.88.
We wrote the rule down before we looked at the ninth test's results, then applied it to them. For the tenth and eleventh tests we froze the app and ran it on three new households each.
| Test | Questions | Right | Wrong | Not sure |
|---|---|---|---|---|
| Ninth, without the check | 160 | 104 | 28 | 0 |
| Ninth, agree or not sure | 160 | 87 | 1 | 59 |
| Tenth, without the check | 168 | 103 | 33 | 0 |
| Tenth, agree or not sure | 168 | 74 | 1 | 87 |
| Eleventh, without the check | 166 | 93 | 39 | 0 |
| Eleventh, agree or not sure | 166 | 71 | 3 | 59 |
Across the three tests that is 5 wrong answers in 494, against 100 without the check. Of the 237 figures the app chose to show, 5 were wrong. From 5 in 494, the true rate is below about 2.4%, at 95% confidence. That was the rate on those nine households. The larger tests in Where the claim stops found a higher one: about 6 conversations in 100 showed a wrong figure. A conversation counts as wrong when any answer in it shows a wrong figure.
Two models are not two methods
The obvious next step was a referee. When our code and the second reader disagree, ask a third model and go with the majority. We set the bar first: a referee that settles about one disagreement in five the wrong way is not good enough. We tried a larger model from the same family, then a model from another company. They settled 22% and 20% of disagreements wrongly, and nearly every wrong call sided with the other model against our code.

Given the same instructions and examples, two models make the same mistakes, whoever built them. Our code and a model are built differently, so they go wrong in different places. That is why the pair works.
What it costs
The app answers less. With the check on, it answered 54%, 44% and 43% of questions correctly in the three tests. Without it, about 60%. The second reader never rescued a question our code got wrong; it only filtered. So every "not sure" on a question our code had right is an answer we want back. Conversations take the biggest hit, because once one answer is "not sure", the next question has nothing to build on.
It was also slower. When we wrote this, an answer took about 12 to 14 seconds on an M4 Mac with 24 GB of memory, against a tenth of a second without the check. The second model was a 2.5 GB download, once. Since 25 October 2026 the second reader is a small model that we trained for this job. It is a 397 MB download, and a checked answer takes about 1.4 seconds.
For money, we think that trade is right. You can act on "not sure, please name the account". A wrong total that looks right is much harder to catch.
What we learned
Questions you have used are questions you have fitted. Four designs in a row looked good on tests we had already read and made no difference on tests we had not. Even fixes to the new design raised right answers from 74 to 84 on households we had read, and did not move the share answered right on new ones: 43%, against 44%. Only a new author writing about new households gives a signal you can trust.
Read every answer. Our grader had blind spots, and the bug that hid our result showed up in the logs long before it showed up in a score.
Know when you are right before trying to be right more often. Test after test of better reading barely moved the error rate. One decision about what to do when unsure cut it by a factor of twenty on those households, and by about four on the larger tests since.
Where the claim stops
Nine households and 494 questions are enough to tell 1% apart from 20% with confidence, but not to promise a rate to the decimal. AI authors invented the households and questions, not people writing about their own money, and everything ran on one Mac. The result covers the app with the check on. In the app, the check is the setting Better understanding, in Settings. With it off, the app answers as it did before.
Agreement is not proof, either. In the eleventh test, two of the three wrong answers came from both readings making the same mistake: "now pull the groceries out of that" was read as groceries alone, and "per month" as a total. The third came from a question the app asked back, and the check does not cover those yet.
Those limits were real. On 20 October 2026 we ran a larger test that we keep sealed: nobody reads its conversations to change the app. On its 337 conversations, written by new authors, about 7 conversations in 100 showed a wrong figure with the check on, against about 23 in 100 with the check off. The check still cut wrong figures by more than three times, and when Pulse Local was not sure, it told you.
On 22 October 2026 a second sealed test, 340 new conversations, compared the second reader we shipped then, a larger 4-billion-parameter model, with a small model of 0.6 billion parameters that we trained to read questions about money. With the small checker, 16 conversations had a wrong figure and 140 were right. With the larger model, 17 and 103. With the check off, 73 had a wrong figure. The small checker also took about 1.1 seconds an answer, against 11.7 for the larger model. On 25 October 2026 the small checker replaced the larger model, on by default on every machine.
On 25 October 2026 we wrote the test down first and ran a third sealed test once. On these 387 new conversations, about 6 conversations in 100 still showed a wrong figure with the small checker, and about 23 in 100 with the check off.
After that test, we opened the first three sealed tests and used their conversations to find where the app's own rules read a question wrong. We fixed those rules, and those tests now count as used. On 28 October 2026 a fourth sealed test, written down first and run once, judged the fixes. On its 380 new conversations, about 5 conversations in 100 showed a wrong figure with the check on, and about 22 in 100 with the check off. The app before the fixes showed a wrong figure in about 8 in 100 of the same conversations. The BudgetBench page gives the full counts.
What comes next
First, winning answers back. When the two readings differ, the app could show both and let you pick. On the eleventh test's households that turned 24 "not sure" answers into right ones after one tap, but 6 times it offered only wrong readings, past the limit we had set in advance. So for now the app keeps saying it is not sure.
Update, 31 October 2026. Pulse Local 0.4.27 does this now. When the two readings differ, the app offers them, two as a choice or one to confirm, and shows a figure only after you tap one. Three filters drop the offers we had seen go wrong: a share question offers only shares, a question that names two things never gets a reading of just one of them, and a reading offered alone must use the question's own words. On the 723 public test conversations, "not sure" fell from 140 to 76, wrong figures shown stayed at 4, and 12 conversations were offered only wrong readings. The sealed figures in this post were measured before that change and still stand.
Speed is done. 12 seconds was too long to wait for a grocery total, and the small checker brought a checked answer to about 1.4 seconds. Next is the test we do not have yet: real people asking about their own money.
Every household in these tests is invented, and nothing in them is anyone's real data. For what to check before relying on an answer, see Accuracy and limits.