· 7 min read

Hundreds of Prompts Later, Still No Working System: Why an AI Agent Project Needs a Business Analyst

Why businesses burn AI requests after data preparation and still get no reliable result — and why the missing piece is a person who owns the process, the edge cases, and the acceptance tests.

TL;DR

QuestionShort answer
We prepared our data properly — why is the agent still unreliable?Because data preparation fixes the inputs. Specifying the process — every exception, every edge case — is a separate job the tool assumes someone already did
So the models aren’t good enough?They’re excellent and improving (Claude Opus 5, GPT-5.x). The bottleneck moved from the model to the specification
Who does OpenAI build these tools for?Its AgentKit launch says it plainly: “developers and enterprises” — people who can already specify a system
What does the missing person do?Arbitrates contradictions, decides edge cases, builds the acceptance tests, owns the change process
When is DIY enough?Personal productivity, one-off documents, throwaway prototypes
What does “done” look like?A system with roles, audit trail, tests, and a scoreboard — see the live demo at demo.kamensky.dev

This is the second half of a story. In the first half, we cleaned the data: one canonical price list, controlled statuses, documented process, a test set. That work is real and it pays off. And yet a remarkable number of teams then stall for months in the same place: prompting, re-prompting, paying for agents that demo beautifully and fail on Monday.

The failure isn’t in the preparation. It’s in what no prompt can supply.

First, credit where it’s due

The tools genuinely got good. OpenAI’s AgentKit ships a visual Agent Builder; custom GPTs connect to your Drive; Claude’s projects and workspaces ingest documents and build working apps from them; frontier models keep climbing — Claude Opus 5 arrived in July 2026 explicitly tuned for long-running agents. For a technical person, the distance from idea to working prototype has collapsed to hours.

Read the AgentKit announcement again, though, and notice the audience: “a complete set of tools for developers and enterprises to build, deploy, and optimize agents.” Developers and enterprises. The tooling assumes someone on your side can specify the system: what the process is, where the exceptions live, what “correct” means, what happens when things fail. If nobody on your side can — and in a typical SMB, nobody can — the tool doesn’t remove that need. It just leaves it unfilled.

Seven failure modes that more prompting cannot fix

1. Somebody must arbitrate contradictions. Your sales director says returns are 14 days; your operations manager says 30; both are “right” for different client tiers. An agent given both rules doesn’t escalate the conflict — it picks one, invisibly. A business analyst’s core skill is turning “both, depends” into one written rule with named exceptions. That decision is business work, not prompt work.

2. The edge cases are the system. The happy path is 20% of the process and 100% of the demo. The value is in the long tail: the client with two legal entities, the order placed at 23:58 on December 31st, the partial delivery, the currency revaluation mid-invoice. These aren’t in any file — they live in the heads of your two most experienced people. Extracting them is interviews, whiteboards, and decisions. Requests to the model don’t surface them; questions to your people do.

3. Reading is easy; writing safely is engineering. Chat agents read your Sheets fine. Writing back — without creating duplicate rows when two people edit simultaneously, without losing an update when the network drops mid-save, without the agent “helpfully” overwriting a price — requires transactions, locking, and conflict handling. No chat product does this because it isn’t a chat problem; it’s a data-platform problem. (This is the same wall Sheets hits at scale, wearing a new hat.)

4. Without an evaluation loop, you can’t tell “better” from “different”. You tweak the prompt; the answers change. Are they more correct? Nobody knows, because there’s no frozen test set and nobody whose job is to measure. So the loop becomes taste-driven: prompt, squint, prompt again. That’s how “hundreds of requests” happen — not because the model is weak, but because nothing defines success. The test set from the preparation phase is half the fix; the other half is a person who runs it after every change and signs the result.

5. A business process must run the same way every time. Agents are probabilistic; your order desk is not allowed to be. “It usually approves the right discount” is not an operational standard — it’s an incident report waiting for a date. Reliability comes from wrapping the agent in deterministic rails: validation, retries, idempotency, an audit trail of who changed what. Building those rails is engineering work around the agent, not conversation with it.

6. Someone must own the change process. The price list changes Tuesday. If the reference data has an owner and an update rule, the agent is correct again in minutes. If not, it stays confidently wrong until a customer notices. Every real system needs that owner — a person accountable for keeping the ground truth true. Vendors sell you the agent; the owner is not included in any plan.

7. Trust has to be earned, then kept. Teams typically abandon a system after a couple of bad answers, and they’re right to. Rebuilding trust requires exactly what an analyst provides: the incident reviewed, the rule fixed in one place, the test extended with that case, and the fix demonstrated — not another hopeful prompt.

The economics of iterating in prompts

Iterating on prompts feels free because each request is cheap. It isn’t: at chat-tool metered pricing, a 40-tab workbook re-read per session plus exploratory prompting scales with activity, not results — and the real cost is the months your operations team spends half-on-the-old-process, half-on-the-agent. The alternative isn’t a better prompt. It’s a specification: the process document, the edge-case register, the test set, and the acceptance criteria from the preparation checklist — produced once, by a person, in weeks — after which agents become cheap and reliable.

Why the missing role is a business analyst first

Notice that of the seven failure modes above, only #3 and #5 are substantially engineering. The other five are business analysis: arbitrating rules, extracting unwritten knowledge, defining measurable correctness, owning reference data, running acceptance. This is why “hire a prompt engineer” so rarely rescues these projects — the bottleneck was never the prompts.

It’s also the honest description of what I do. I spent 17 years inside business operations before I built software: economics degree, industrial analytics, business analysis, SAP and Excel automation in real companies. Now I run that discovery discipline as step one of every automation project — then build agents and the platform around them so the result has roles, permissions, an audit trail, backups, and a test suite. The same process, applied to your data, is visible end-to-end at demo.kamensky.dev: the migration pipeline, the authenticated operations platform your team would actually use, and the scoreboard that shows the hours and dollars.

If your team is stuck in the prompting loop, the cheapest next step is not another request. Book a 20-minute fit call — bring the process that hurts; you’ll leave knowing whether it’s a data problem, a specification problem, or a platform problem, and what each costs to fix.

When doing it yourself is genuinely enough

The honest boundary: personal productivity (summarize, draft, translate), one-off analytical questions over a single clean file, and throwaway prototypes that validate an idea before you invest. For those, the built-in tools are superb and this article doesn’t apply. The moment a process runs weekly, touches money or customers, or needs two people to agree on the numbers — the missing role appears, whether anyone planned for it or not.

Sources & further reading