Why AI Agents Get Stuck in POC
With Brandon Lei, Tannvi Banerjee, Daniel He, Vidit Agrawal, Johnny Kim
Two things are going wrong in agent development. Teams are not building credible evals, and they are not thinking holistically about the levers available to improve an agent once they have them.
Your engineering or product team builds an agent that reaches roughly 90% accuracy on its eval suite (or worse, they aren’t using an eval suite at all), but it still does not feel ready for production. It hallucinates in cases nobody predicted. It still needs a person watching it, correcting it, and cleaning up after it. So the team rewrites the prompt again, but the agent still never reaches meaningful production volume.
We see this across our customer engagements, and it almost always comes down to two failure modes.
First, the agent evals don’t credibly define what good looks like. They were generated by an engineer, or by an AI model an engineer prompted, without enough input from the domain experts or end users who actually do the work. A 90% on an eval suite like that does not tell you the agent is 90% good. It tells you the agent agrees with a rough guess at what good looks like. That is why the score reads 90% and your gut says otherwise.
Second, the team is only optimizing the prompt. Even with a credible suite, hill climbing on one dimension of a system that has many will stall. Everything in the agent is tunable: the model, the skills, the context, the harness. In practice, teams often spend far more time iterating on the prompt than on the rest of the system.
Credible evals come from people who do the work
Reliable software development depends on tests. The analog for agents is evals. Test-driven development becomes eval-driven development.
The difference is where the spec comes from. When an engineer writes a unit test, they already know the right answer, because they decided what the function was supposed to do. When an engineer writes an agent eval, the right answer lives in a domain they may not practice.
The engineer building a Telco internet customer service agent knows Python. They do not necessarily know what separates a good customer service call from a bad one.
Take that internet customer service agent. An expert will tell you:
- Call verify_identity before any tool that reads account data
- Call check_outage before troubleshooting the individual line
- Run get_modem_diagnostics and try reset_modem before asking the customer to unplug anything
The engineer may not have thought to check any of that. Those checks become part of the eval suite. The eval suite is the set of tasks, environments, verifiers, and criteria used to measure the agent. The broader platform is what turns that measurement into an optimization loop e.g., running the evals, identifying where failures originate, and routing that feedback to the part of the agent stack that needs to change.
Most engineers building the agent have experienced customer support as users, not operators. The gap is even wider in domains like finance, auditing, or underwriting.
The hard part of building a credible eval suite is knowing what to test. That judgment comes from people who understand the work.
The experts who will build your agent evals come from one of two places:
- Your own employees: The right choice when the agent runs an internal process specific to how your company works. These employees understand the workflow, company-specific policies, and edge cases the agent needs to handle. These employees should be well versed in how to write agent evaluations and prepared to dedicate significant time to writing them.
- External practitioners: The right choice when the agent is customer or product-facing, like a shopping agent or a support agent, or when the work takes a general professional skill rather than company-specific knowledge e.g., performing an audit, or underwriting a risk, or reviewing a contract.
For the second case, Mercor sources domain experts from its 7M+ contractor network who have done the specific work the agent is being built to perform. Their judgment helps define the criteria, edge cases, and failure modes the evaluation system needs to capture.
Everything in the agent is hill climbable, not just the prompt
Ask most teams what they can change about an agent and you get prompts and instructions. That is a fraction of what is available.
Everything in the agent configuration is hill climbable:
- The prompt
- The skills
- The context
- The tool definitions
- The model
- The harness
- The deterministic logic
Each can be tuned against the same evaluation standard.
Fig 1: The levers of agent optimization
Take the example mentioned earlier: a Telco internet customer service agent.
The eval says the agent should try reset_modem before asking the customer to touch anything. Instead the production agent spends six minutes walking them through unplugging the router, and never calls the tool. There are at least five ways to fix that, and one of them is the prompt.
| Lever | What you would change |
|---|---|
| Prompt instructions | Add a rule to try the remote reset first. This may work, but it adds another instruction the model has to balance against the rest of the prompt. |
| Tool definitions | reset_modem is described as "resets customer equipment." Say instead that it is the first-line remedy and takes thirty seconds. |
| Harness | Run check_outage and get_modem_diagnostics concurrently as soon as the account is verified, so the results are available before troubleshooting begins. |
| Model selection | Move to a model that holds a multi-step procedure over a long call, and let the eval tell you whether it was worth the cost. |
| Deterministic logic | Run reset_modem on any no-sync fault before the call reaches troubleshooting. |
A change that improves one behavior can degrade another, so each meaningful change should be evaluated against the full suite before it ships.
This is the development loop we build at Mercor. Domain experts help define the criteria, edge cases, and failure modes that matter. Those standards become the eval suite. Our platform then runs and audits those evals, surfaces where the agent is failing, and routes that feedback to the relevant layer of the agent stack. Our forward-deployed engineers can use that signal to hill climb across the prompt, model, skills, context, tools, harness, and deterministic logic, evaluating each change against the same standard.
Build the eval suite alongside the agent
The eval suite should not be a one-time test added at the end of development. It should evolve alongside the agent.
For a new agent, that means defining what good looks like early and expanding the suite as new edge cases and failure modes emerge. For an agent already in production or stuck in POC, it is not too late. Production failures are often some of the best inputs for strengthening the eval suite. Capture what went wrong, turn it into a repeatable test, and use it to prevent the same failure from recurring.
The eval suite sets the bar. The platform turns that bar into a development loop. As the agent encounters new failure modes, the suite gets stronger; as the suite gets stronger, teams get better signals on what to improve across the prompt, model, tools, skills, context, harness, and deterministic logic.
Without a credible standard, optimization is guesswork. With one, teams can identify what is failing, change the right part of the system, and measure whether the agent actually got better.