Why AI Agents Fail After the Demo—and What CARS24 Gets Right

Watch (8:20)

---

Overview

CARS24 reports more than a million agent conversation minutes a month while most agent demos collapse the moment the task leaves a clean prompt. The difference is a five-part contract around the model: accepted outcome, bounded permissions, evidence-based routing, trajectory evaluation, and a fallback.

---

Full transcript (from the video)

Here is the useful number. Open AAI published a customer case study on July 16. In that study, Cars 24 says its agents handle more than 1 million conversation minutes per month. It also reports a 50% increase in customer support resolution, an 80% reduction in turnaround time across key service workflows, and 12% of previously lost seller leads recovered through re-engagement.

Those are customer reported results on a vendor site, not an independent audit. But they are specific enough to ask the right question. Why can this system operate across real customer journeys while so many agent demos collapse when the task leaves a clean prompt? The answer is not a smarter chatbot.

It is a production system wrapped around the model. Together, four stories this week form an agent autopsy. A production case study, roing economics, a hands-on failure test, and investment guidance. Start with the shape of the work.

Cars 24 describes car buying in India as a journey rather than a transaction. A customer may compare vehicles, arrange financing, book a test drive, change preferences, prepare documents, miss an appointments, return days later, and need after sales support. Sellers move through a parallel chain of vehicle details, inspections, reminders, rescheduling, and renewed intent. That means the agent is not answering one question.

It is carrying state across time, choosing tools, updating business systems, and knowing when the next action is appropriate. Cars 24 says its voice and chat agents cover buying, selling, financing, follow-up, and support. Internally, Codeex also moves work from tickets and bug reports into implementation and status updates. The lesson is simple.

The workflow is the product. A demo proves that a model can perform one attractive step. Production requires every handoff around that step to be owned, observable, and recoverable. The first mechanism is routing.

And this week, IBM research made the trap unusually concrete. in a hugging face article. Across 417 AppWorld tasks using the same Kodak agent, the authors report that Claude Sonet cost $79 in total, while GPT4.1 costs >> >> $155. Sticker pricing suggested the opposite.

The difference was caching. Agent trajectories reuse large blocks of context, so cash read economics change the effective cost of the entire run. The authors also point out that task difficulty is partly invisible before execution. A request that sounds simple can trigger retrieval, compliance checks, tools, and several rounds of refinement.

Latency is equally systemic. Endpoint load, warm caches, hardware, and the number of routing decisions can dominate raw model speed. So routing cannot be a one-time label that says easy or hard. It must optimize the operating point across quality, cost, latency, reliability, privacy, and approved model policy.

The model is one variable inside the system, not the system itself. The smallest production path is not prompt, model, tools, done. It is a five-part contract. First, define the accepted outcome in business terms.

a resolved support case, a booked inspection, a tested code change, or a correctly requalified lead. Second, bound the context and permissions, which customer records may be read, which systems may be changed, which actions need approval. Third, choose the route using real workload evidence, including cache behavior, latency, reliability, and policy constraints. Fourth, evaluate the whole trajectory instead of judging only the final answer.

Test representative tasks and edge cases. Record attempts, tool results, human corrections, and the cost of reaching an acceptable result. Fifth, create a fallback. Ask a clarifying question, hand off to a person, stop safely, or retry from a known state.

CARS 24's case study is persuasive because it names repeatable workflows and measurable outcomes. The durable pattern is the contract around the agent, not any one model name. Now look at the failure case. In an independent TechCrunch review published this week, Vert's Hermes agent was tested on executive style workflows rather than a staged benchmark.

In one request, it sent a message and enabled do not disturb, but created a reminder for the wrong time and did not start navigation. During a request to plan a trip, it created a calendar entry for the wrong dates and left the workflow incomplete. It also summarized a spreadsheet correctly during the original test, but days later no longer recognized that same document in the conversation. The comparison is revealing.

Hermes was more willing to act across applications while Gemini asked follow-up questions about ambiguous details. Hermes attempted more of the workflow. Gemini sometimes produced the more accurate outcome. That is the production paradox.

Autonomy increases both useful reach and the blast radius of a mistaken assumption. This week's secondary reporting also framed evaluation and credential handling as enterprise agent gaps. Those reports are context, not our primary proof. The direct lesson from the hands-on test is enough.

Clarification, scoped credentials, durable state, and human escalation are capabilities. The fourth mechanism is investment discipline. Open AI's own guidance this week says the lowest token price may not create the lowest total cost. A cheaper model can fail, retry, or produce work that needs correction.

A more capable model can cost more per token and still reach the quality bar with fewer attempts and less review. Their recommended unit is cost per accepted outcome paired with business impact, time saved, cycle time reduced, revenue protected, risk avoided, or capacity created. That framing matters because agent demos optimize the visible moment while hiding the expensive loop around it. Funding should follow maturity.

Exploration asks whether the model can perform the task. Validation tests representative cases against a clear quality bar. Production pays for integrations, permissions, evaluations, observability, reliability, and change management. Cars 24 fits that pattern.

Repeated high volume conversations, defined operational owners, and measurable results. The decision boundary is not whether an agent looks impressive. It is whether useful demand and accepted outcomes are repeatable enough to justify the production system. So here is the operating rule.

Before calling an agent production ready, answer five questions. What exact outcome counts as accepted? What context, tools, and actions are permitted? How is the model and infrastructure route chosen?

Which real tasks and failure cases evaluate the full trajectory? And what happens when the agent is uncertain, wrong, interrupted, or unauthorized? If any answer is vague, you do not have a production agent yet. You have a promising demo.

Cars 24's reported results are not proof that every agent deployment will work, and a vendor case study should never be treated as universal evidence. What the case does show is the right unit of design. A specific workflow instrumented at real scale connected to systems the operator controls, measured by outcomes and governed by safeguards. The model creates capability.

The operating system around it creates reliability.