How Pulse works·Engineering · 4 October 2026, updated 6 October

How we cut wrong answers from one in five to about one in a hundred

Pulse Local answers questions about your money with AI that runs on your machine. Test after test found about one wrong answer in every five, and nothing we tried moved that number. Then we changed what the app does when it is unsure. Across our last three tests it showed a wrong figure 5 times in 494 questions.

What changed was a decision about what the app should do when it does not know. We did not get there with a bigger model or a cleverer prompt, and we tried both. This post describes how we test the chat, the designs that failed, the bug that hid the answer, and what the result costs you. It ends with the limits of the claim.

The new design ships in Pulse Local 0.4.0 as Better understanding in Settings. It is on by default.

The problem

You type "how much did we spend on groceries in August?" and you expect a number you can act on. Your transactions never leave your machine, so two things read the question locally. One is code we wrote by hand. The other is a small AI model. Every figure the app prints comes from a calculation over your own records, and a separate check refuses to print any figure that calculation did not produce.

That check makes sure a figure is real. It cannot make sure the figure answers your question. Suppose you ask about "the other card" and the app takes that to mean a spending category called Other. It will print a true total, for the wrong thing. Nearly every error in this post looks like that. The arithmetic is right and the question was misread.

How we test

We started by testing the chat on questions we wrote ourselves, and the scores crept toward 100%. Then a friend tried the app and got seven wrong answers to seven questions. Our own questions had been measuring how well the app fit them. So we built a test we could not tune the app to.

A household nobody has seen

Every test starts with a new author, a fresh AI session that has never seen the app's code, an earlier test, or the examples the app learns from. The author invents three households. In our latest test they were a potter in Asheville, a retiree in Missoula and a family in Wichita. For each one it writes several months of transactions, between 210 and 290 of them across four to six accounts, plus budgets and a date the household treats as today. The result looks like any bank export.

Date,Description,Amount,Account,Category
2026-02-01,MARLPIT PROPERTIES RENT,-1150.00,Gorsefield Checking,Rent/Mortgage
2026-02-02,MARROWFIELD GROCERS,-81.90,Kestrelwood Rewards Visa,Groceries
2026-02-02,TRANSFER TO CAIRN SAVINGS,-300.00,Gorsefield Checking,Account Transfer

The author then writes about 55 questions per household, in the voice of the person who lives there. About half are short conversations of two to four questions, since follow-ups are where reading gets hard. Typos, shorthand and "that one" are all fair game.

An answer key the app never touches

Every question has an expected answer before the app sees it. Here is a real two-question conversation from our latest test.

how many times did I eat at brindlecap in july?      expected: 3 visits
what did those visits cost me in total               expected: $101.41

A short separate program computes the expected figures straight from the household's transactions. It shares no code with the app, so the app is never graded by its own arithmetic. A checker then tries to break the answer key before we accept it. It builds wrong answers that look right, with the wrong month or the wrong shop, and every one has to fail. It also compares each question with every earlier one, so nothing gets reused. For our latest test that came to 1,678 checks and more than 16,000 comparisons, and none of them turned up a repeat.

A frozen app, run by someone else

Before a test, we freeze the app by taking a fingerprint of every file in it. The test is run by a separate session that never sees the app's code. It checks the fingerprint before it starts and again when it finishes, and if anything changed, the test stops. To be frozen at all, the app first has to replay 52,651 recorded questions from earlier testing and give the same answer to each one, down to the figure.

Each household runs twice on the same questions, once with the app as it was and once with the change we are testing. Because the questions are identical, we can count the ones whose outcome changed. That is a far more sensitive measure than comparing two totals, and it is what lets a test of 170 questions tell us anything.

Every figure checked, every answer read

A grader checks every figure in every reply against the answer key. If a reply has the right total and a wrong comparison next to it, the reply is wrong, because you would read both. We test the grader too. Before each test it has to reject 1,433 replies we built to look right while being wrong.

Then a person reads every answer and records a decision on each case the grader could not settle. It became a rule after one version of our grader passed any reply that contained the right figure somewhere, even inside a wrong comparison, , which only reading showed us.

Counting honestly

• Four outcomes, kept apart. We report right, wrong, declined and not sure separately. A declined question never counts as a success.
• A question back is judged by its options. If the app asks "did you mean A or B?" and both are wrong, we count a wrong answer.
• A conversation is only as good as its worst answer. One wrong answer anywhere makes the whole conversation wrong.
• Paired results with a margin. We report the questions that only one version got wrong, and how confident we can be in the difference.
• Rules written before results. When we change how the app decides, we write the new rule down before we read the test it will be judged on.
• Each test is used once. As soon as a test's mistakes shape the app, we stop counting it as evidence. Its questions become examples for the app, and the next claim needs new households and a new author.

Eleven tests, one flat line

Between tests we did what anyone would. We read every wrong answer, grouped the mistakes into patterns such as "a count followed by how much" or "a yearly price", and fixed each pattern in our code. Each fix came with checks over many generated wordings, so it would cover more than the one sentence we had seen.

The first two tests improved fast. After that the line went flat, between 16% and 20% wrong for eight tests, and our latest came in higher still, at 23.5%. The fixes were real. Each new author simply found new ways to say the same things. After one fix, every question about what a shop charges came out right. In the next test, written by someone new, one in seven did.

What did not work

We tried five ways of reading questions better. Each looked good on questions we had already seen. None of them held up on households they had not seen.

The trained model taught us the most. On questions we had generated it scored 97%, because those questions were written the same way as the material it trained on. On real authors it got a quarter of its answers wrong. It had learned our phrasing, and people do not phrase things our way.

Asking "did you mean A or B?" changed the shape of the mistakes more than their number. Fewer wrong figures reached the screen, but the questions back filled the gap. In one test, 21 of them offered only wrong choices.

We also tried a much bigger model on its own, five times the download. It answered 21 of 77 real questions correctly. Our small tuned model, also on its own, answered 35. A bigger model was clearly not what we were missing.

Reading the mistakes instead of the score

So we stopped staring at the headline number and sorted 208 wrong answers, collected over seven tests, by cause.

About 90% were reading mistakes. The arithmetic was fine and the question had been misread. That explained why our fixes never settled the matter, since there is no end to the ways people can phrase a question.

The bug that hid the answer

Our ninth test tried a larger model that read every question alongside our code, and it showed no clear gain. When we looked through the logs, something was off. Every wrong figure it let through came from a moment when the second reading had never been checked.

When the app asks "did you mean…?", it remembers the question so it can match your reply. Every answer after that carried the memory forward. The check saw the old question, concluded the app was still asking, and stood aside. So after the first question back in a conversation, the check stayed off. Our own earlier measurements had the same flaw, which meant some numbers we had trusted came from a broken check.

The fix took a few lines. Since then we treat any safety check that a conversation can switch off as a bug waiting to happen.

Agreement is almost never wrong

The same test held the result that changed our direction. Our code and the second reader agreed on 137 answers. Of those, 135 were right and none was wrong. Every mistake came from what the app did when the two disagreed, which was to show our code's figure anyway or to offer readings to pick from. In other words, the app could already tell when it was right. It just had no way to act on it.

Agree or not sure

The new rule is short. Every question is read twice, by two methods built in different ways. One is our own code. The other is a second reader, an open AI model that runs on your machine. The same code turns both readings into calculations. When they agree, you see the figure. When they differ, the app says "I am not sure which figure you mean, so I will not guess" and asks you to name the shop, category, account or month.

The second reader cannot answer on its own. It can only confirm our code or stop it. Anything that states a fact from your records goes through this check, counts and "there is nothing there" included. If you pick an option yourself, the check steps aside, because the reading is yours. The same goes for the questions the app suggests and the buttons under an answer: the app wrote those, so it already knows what they mean.

We wrote the rule down before we looked at the ninth test's results, then applied it to them. For the tenth and eleventh tests we froze the app and ran it on three new households each.

Across the three tests that is 5 wrong answers in 494, compared with 100 without the check. With five mistakes in 494 questions, the true rate is below about 2.4%, at 95% confidence. The app chose to show 237 figures, and 5 of them were wrong.

Inside the second reader

The second reader never writes an answer. It writes down what you asked for, meaning what to count, about what, and over which dates. For "what did those visits cost me in total", the correct reading is spending at Brindlecap Noodle Bar in July 2026. It can only name shops, categories and accounts that appear in your records, so it has no way to invent one.

Before it reads your question, it sees the eight past questions closest in meaning to yours, each with the reading its author intended. They come from a pool of more than 2,200 questions written by ten different authors. If its reading names a shop or account you never mentioned, we set the reading aside. That keeps "the other card" from turning into whichever card the model happens to prefer.

We judge agreement on the two calculations, never on the wording. They need the same total, compared with the same thing, worked out the same way. An amount you kept and a percentage you kept come from the same numbers, and they still never count as agreeing.

Here is the kind of disagreement the check is for. Someone asked "if Cinemoth stays at this price, what's that a year?" Our code read it as a total so far and would have said $95.93. The second reader read it as a yearly cost and said $179.88. They disagreed, so the app said it was not sure. The second reader was right. Our code only recognized "a year" in a handful of fixed phrasings, and the disagreement showed us exactly where to fix it. Now both say $179.88.

Two models are not two methods

The obvious next step was a referee. When our code and the second reader disagree, ask a third model and go with the majority. Before we tried it, we set a bar. A referee that settles one disagreement in five the wrong way is not good enough.

We tried a larger model from the same family, then a model from another company trained on different data. The first settled 58 disagreements correctly and 16 wrongly. The second got 49 right and 12 wrong. Both missed the bar. Nearly every wrong settlement sided with the other model against our code, and 14 were the same questions for both referees.

The company behind a model made little difference. What mattered was what the models shared. They had the same instructions, the same examples and the same way of turning a reading into a calculation, so they made the same mistakes. Our code and a model work well together for the opposite reason. They are built differently, so they go wrong in different places.

What it costs

With the check on, the app answers fewer than half of all questions correctly: 54%, 44% and 43% in the three tests. Without it, the figure is about 60%. On new households the second reader never rescued a question our code got wrong. It only filtered. So every "not sure" on a question our code had right is an answer we want back. Conversations take the biggest hit, because once one answer is "not sure", the next question has nothing to build on.

It is also slower and bigger. An answer takes about 12 to 14 seconds on an M4 Mac with 24 GB of memory, compared with a tenth of a second without the check. The second model is a 2.5 GB download, once.

For money, we think that trade is right. You can act on "not sure, please name the account". A wrong total that looks right is much harder to catch.

Winning answers back, safely

The design also changed how we make fixes. A bad fix used to create a new wrong answer. Now the worst it can do is turn an answer into "not sure", because nothing reaches the screen without agreement. That turns the list of disagreements into a list of work we can do without fear.

Going through that list after the tenth test, we found a surprise. In 15 of the 25 cases where our code was right and the app said "not sure", the second reader had not disagreed at all. It had produced no reading. Three things we had given it were to blame. Its examples never showed several accounts treated as one, as in "savings balances altogether". It rejected things you describe rather than name, such as "the house loan" for a mortgage. And each example carried its own household's calendar, so the reader copied their dates and wrote August for "this month" in every household. Each of these took one fix in code the whole app shares.

Those fixes, and a handful more, raised the answers right from 74 to 84 on the tenth test's households, which we had already read. On the eleventh test's new households the share answered right did not move: 43%, against 44%. The first lesson below applies to us too.

Freezing has paid for itself, too. When we froze the app for the tenth test, the replay of recorded questions found 8 answers where a new fix had mistaken a fresh question for a follow-up. We withdrew the freeze, narrowed the fix, and froze again.

What we learned

Questions you have used are questions you have fitted. Four designs in a row looked good on tests we had already read and made no difference on tests we had not. The only reliable signal came from a new author writing about new households.

Write the rule down before you look. We wrote down agree or not sure before reading the results it would be judged on. A good number on material you have already seen is hard to trust, and that includes for us.

Count questions back honestly. If the app asks "did you mean A or B?" and both are wrong, that is a wrong answer. Counted any other way, two of our designs would have looked like progress.

Read every answer. Our grader had blind spots, and the bug that hid our result showed up in the logs long before it showed up in a score.

Know when you are right before trying to be right more often. Test after test of better reading barely moved the error rate. One decision about what to do when unsure cut it by a factor of twenty.

Independence comes from method, not from vendor. Two models given the same context tend to agree, wrong answers included. A model and hand-written code go wrong in different places.

Close the paths, not the cases. Nearly every wrong answer the checked design let through in development took a path that skipped the check. An answer was never read twice, or it was a count, or a "there is nothing there", or a note that added a figure for scale. Each one was a single gap to close.

Where the claim stops

Nine households and 494 questions are enough to tell 1% apart from 20% with confidence, though not enough to promise a rate to the decimal. The households and questions are invented by AI authors, not written by people about their own money, and everything ran on one Mac. The result also only covers the app with Better understanding on. With it off, the app answers as it did before.

Agreement is not proof, either. In the eleventh test, two of the three wrong answers came from both readings making the same mistake: "now pull the groceries out of that" was read as groceries alone, and "per month" as a total. The third came from a question the app asked back, and the check does not cover those yet.

Here is the claim we are comfortable making. When Pulse Local shows you a figure from your records, it is almost always right, and when it is not sure, it tells you.

What comes next

We have already tried one way of winning answers back. When the two readings differ, the app could show both and let you pick. We wrote down a limit first, then ran it on the eleventh test's households. It turned 24 "not sure" answers into right ones after one tap, but 6 times it offered only wrong readings, past the limit we had set, so the app keeps saying it is not sure.

Next, speed, because 12 seconds is too long to wait for a grocery total. Then the test we do not have yet, which is real people asking about their own money.

Every household in these tests is invented, and nothing in them is anyone's real data. For what to check before relying on an answer, see Accuracy and limits.