I almost replaced my AI analyst with Claude. Here's why I didn't.
Claude is a powerful component. But a component is not a system.
Last week, I wrote how I gave Claude an analysis and got a high-quality output in under an hour with just 4-5 back and forth conversations. This finding that reversed my long held view that AI is not good for math.
This recent capability also got me thinking about past AI Analyst agents I had built and I started questioning whether they were even necessary.
After a day of research, I concluded that an AI analyst agent doesn’t need to die. There is a different place for AI Analyst agent vs. Claude.
A quick note before we dive in. The system I'm walking through is a personal finance analyst. It pulls data from multiple bank accounts and lets me (& my family) query and report on my spending. I'm using this as the example because it's concrete and I can share the code. But the same architecture applies to the analytics agents I've built too. The principles don't change.
Part 1: Data Foundation
This is the step where raw bank exports from Chase, Citibank and Bank of America (each in a completely different format) get converted into a single source of truth.
Two things happen here:
Standardization. Format differences get resolved. Dates, amounts, transaction types all get normalized into a unified schema.
Categorization. This is where Claude comes in. I use the Claude API to classify transactions by merchant i.e. grouping all 43 “UBER TRIP” entries and categorizing them as Transportation in a single API call. Claude is genuinely good at this and I couldn’t imagine doing it any other way.
But here’s the thing: Claude categorizes, and then I audit. Some categorizations need manual correction. Those corrections get codified as overrides and fed back into the system for future runs. Claude is a powerful component inside this step. It is not the step itself.
The output I care about is a dataset with a full audit trail on what was ingested, what was AI-categorized, what was manually overridden, and what the final classification was. That lineage matters. This process runs the same boring way every time I add new data. It can’t be a self-discovery moment each time.
Verdict: keeping it. AI-assisted, but wrapped in deterministic repeatable steps. A 1M token context window gives you an answer. A database gives you a record of every decision that produced it.
Part 2: Recurring Reporting
A codified monthly report that runs the same deterministic logic every time. Same output and same structure, every month.
Claude helped me build it. But once it was built, Claude stepped back. This is not a place for interactive AI, it’s a place for consistency. In my experience, stakeholders need to orient to the same view every month to get to decisions quickly. A different structure every month means your audience is re-learning the report instead of reading it.
Verdict: never in question. Built with AI, runs without it.
Part 3: The Ad-Hoc Question Layer
This is the part I most seriously considered replacing. The idea is simple: ask a question in plain English, the system figures out the answer (by writing SQL code in the background), and returns it. “How much did I spend on dining in Q1?” “What’s my biggest expense category this year?” A non-technical user should be able to get answers without writing a line of code or asking someone else to pull the number.
Could I just do this directly in Claude?
The distinction that settled it: Claude Code is a developer tool, not a product. It runs analysis in an interactive session where a human is in the loop, reading output, course-correcting, re-prompting. That’s exactly what I was doing in last week’s post, and it’s powerful for that use case.
This layer is for something different. It has to work reliably without a developer in the room, every time, not just the first time you set it up. And if you’re building this for a team, that reliability requirement only increases.
That reliability comes from infrastructure, not from AI itself:
A semantic layer: This step is critical for making business rules explicit. This is different from the data definitions in Part 1. Data definitions are about what the data means. The semantic layer is about how the system should reason about it e.g. standard query patterns, valid filter values, how to handle edge cases like refunds or transfers. These rules live in config files, not in a prompt that gets re-interpreted every session.
Curated examples: In this case 25+ question/answer pairs that teach the system how to handle the queries that come up most often and the ones most likely to go wrong. In a real-world setting this library grows fast. Edge cases like "how do we treat intercompany transfers?" multiply quickly once real users start asking questions.
An eval suite: A set of test cases with expected values and tolerances. Before any change to the system, the evals run and tell me whether anything that was working before has broken. It’s a regression suite, not a live monitor but it’s what gives me confidence to ship changes without guessing.
Logging: Every question asked by the user and the corresponding answer is logged, creating a perfect trail for future product improvements. This self-learning doesn’t come out of the box in Claude Code.
All of this sounds like a lot of work. A friend at Snowflake asked me whether Claude's skills feature addresses these gaps. I tested it. Skills are essentially well-crafted prompts and that’s the honest limitation. A prompt is a suggestion, not a contract. Better instructions improve the odds but don’t guarantee consistent behavior across every query. With any LLM-based system, these rules can be “forgotten” mid-reasoning. Rigid infrastructure after every step is what gives you a failsafe. The eval suite is what tells you when the failsafe isn't holding.
Verdict: keeping it. AI without scaffolding is a demo, not a product.
The through-line
Claude is a powerful component inside each of these steps. I genuinely can’t imagine the productivity without it. The right mental model is not Claude as a chatbot you converse with, but Claude as a tool inside infrastructure you control. The categorization step is a good example: Claude does the classification, but the pipeline handles the ingestion, the audit trail, the overrides, and the evals. Claude is one function call in a larger system. And that’s the case for keeping my AI agent alive. Not because Claude isn’t good enough. Because a system is what I am advocating for.
Last post answered “Can AI do math?” This post answers “What does it take to make AI a reliable part of your analytics workflow?”
This really clarified the difference between AI as a tool vs a system. The “component vs infrastructure” point hits hard.