Why AI should write code, not do math

In my last post, I shared that I built an AI analyst using my own bank statements. This week: I share the iterations it took to make it work

Version 1: AI Does Math Directly

My initial approach seemed logical: give the AI access to the data (more on the data in another post) and ask it questions directly.

The question: "What was my total spending in 2024?"
Claude's answer: $980.93
Actual answer: $938.92

This is the uncanny valley of data. The answer looked perfect - it was precise, formatted correctly, and delivered with total confidence. But beneath the surface, it was ‘off’ in a way that felt almost eerie once I looked at the math. It had the form of a professional analyst but the substance of a hallucination.

When I asked what happened, Claude admitted

This wasn’t a one-off failure mode but something that happened A LOT. It had hallucinated entire quarters of data. In some cases, it invented transactions that never happened. In others, it compared completely wrong time periods and reported bizarre growth rates.

Version 2: AI Writes Code

I attended an AI meetup where a team demoed using AI to write code (specifically SQL) instead of having it calculate directly. This was the unlock.

Don’t ask the AI to calculate. Ask it to write SQL.

The workflow became:

1. I asked question in plain English: “What was my shopping spend in 2024?”

2. AI generates SQL query: SELECT SUM(amount) FROM transactions WHERE category=’Shopping’ AND year = 2024

3. Database executes the query against REAL data

4. AI translates results back to plain English: “Your shopping spend last quarter was $2,847”

Why this works

The AI isn’t doing math anymore. It’s doing what LLMs are actually good at: understanding language and writing code.

The actual calculation happens in the database, where it’s deterministic and verifiable.

The AI’s job is just translation:

• English → SQL (which LLMs are great at)

• SQL results → English (which LLMs are also great at)

Validating accuracy

I built a test suite of pre-calculated answers:

• Total spend for 2024: $X

• Grocery spend Q4 2024: $Y

• YoY restaurant spending comparison: Z%

I ran these benchmark queries through the system and verified every single answer matched my manual calculations.

Accuracy went from “wildly wrong” to “reliably correct.”

Version 3: Add Context

Version 2 worked well for simple questions like “What was my spend in 2024?” but failed on questions like “How much did I spend on fuel?”

Here’s a snapshot of the data:

The AI didn’t know whether to filter by Category or Subcategory and inconsistently switched between the two fields.

The fix: Add context via a semantic layer

I provided AI a documentation describing:

• What each column means

• Hierarchy of categories and subcategories

• Business rules (e.g., “Refunds are categorized with original merchant category” → Amazon refund = Shopping, not Refunds)

  • Why this matters: Most automated systems fail because they treat a refund as a separate income event. By forcing the AI to map it back to the original category, the system finally understood net cash flow i.e. the difference between "I bought it" and "I kept it."

I also provided 15-20 example SQL queries for common questions to show the AI what correct outputs look like.

Now AI was working for questions like “How much did I spend on fuel?”

Version 4: Add Evaluation & Feedback Loop

I felt good about the system, but I still didn’t have objective confidence it was working correctly every time. “Working correctly” meant:

1. Getting the math right (table stakes)

2. Understanding the data the way I intended

Next post: The evaluation framework that ensures we never slip back into the uncanny valley, and why building it was more complex (and fun!) than the agent itself.