The week that decides the deal
In an API sale, the demo opens the door and the evaluation closes it. At some point the customer's engineers take a sample of their own work, run it through the product, and come back with three numbers: how often it was right, what it cost, and whether it broke. Those numbers decide the contract, and they usually arrive without a seller in the room.
I wanted those numbers before a customer had them. The product is the Agent API: send a question, and it searches the web, reads what it finds and writes a cited answer, priced per request, with a dial for how hard it works that runs from fast to xhigh. Perplexity's published benchmarks show accuracy climbing steeply up that dial on very hard research questions.
Most business work isn't very hard research, and that is where the dial is easiest to set wrong. So I built four test sets that look like everyday work: single-fact lookups, questions that combine several facts, company records for a CRM, and questions built on events that never happened. Two of those are a seller's own job, researching accounts and keeping records accurate. Every answer was graded against a key, and where an answer could change by the hour, such as a stock price, the key was pulled from the original source at the same moment.
Along the way I logged everything a new developer trips over. That list, in section 06, is what I would carry back to the product team.
Match the effort to the job
The first set was 15 single-fact lookups with one right answer: the latest release of nine widely used software projects, and the closing price of six stocks on November 19. Every effort level got all 15 right. The cheapest did it in 1.7 seconds.
The second set was 10 questions that need several facts combined. Which of six stocks moved most on a given day? How many releases did a project ship in October? Here the cheapest level got 6 right, and the next level up got all 10.
Accuracy and cost by effort level
Agent API, answers graded against a key pulled from the original sourcesSingle-fact lookupscorrect, of 15
Multi-step questionscorrect, of 10
Cost per correct multi-step answerUS cents, lower is better
The fast level's mistakes were confident and plausible. Asked which of six stocks had the largest move on November 19, it answered CrowdStrike, up 2.71%. That figure is real: CrowdStrike was the biggest gainer. The right answer was NVIDIA, which fell 3.30%. Fast ran two searches and answered the easier question it found. Low ran more searches and caught it.
Because fast missed so often, low was the cheapest level per correct answer, at about half a cent. High was right every time too, at five times the price.
Effort level is a pricing decision the customer makes on day one, usually without data. Set it too low and answers go quietly wrong. Set it too high and the bill is five to seven times what the job needs. Helping a customer pick the level from a sample of their own questions is the most valuable first meeting a seller can offer.
Filling in records, and refusing to make things up
A common paid use is enrichment: hand the agent a company name and get back a filled-in record for a CRM. I asked for seven fields on 20 public companies (ticker, headquarters, CEO, headcount, fiscal year, revenue and security chief), returned as structured data in a fixed format.
Company records, fast vs medium effort
20 public companies, 7 fields each, structured output| Measure | Fast | Medium |
|---|---|---|
| Valid, complete records | 20 of 20 | 20 of 20 |
| Ticker matches | 20 of 20 | 20 of 20 |
| Revenue and CEO | Matched across both levels | |
| Security chief named | 12 of 20 | 16 of 20 |
| Median time per record | 3.3 s | 12.0 s |
| Cost per 1,000 records | $4.10 | $7.10 |
The public facts were settled at the cheap level: medium returned the same revenue and chief executive for every company. Its extra time went into the one field that is hard to find, the head of security, where it named four more people. For one company the two levels named different people, so that field still needs a human check before anyone emails it.
The last set tested trust. I asked 12 questions built on events that never happened, such as how much Apple paid to acquire Netflix, or who won the 2024 Nobel Prize in Mathematics. At three effort levels, all 36 answers rejected the premise, and most explained the likely mix-up with a source.
"The Philadelphia Eagles won Super Bowl LIX, defeating the Kansas City Chiefs 40-22. It was played in New Orleans, not San Diego."Fast effort, asked "Which team won the 2025 Super Bowl played in San Diego?"
For enrichment, the cheap level is the product and the higher level is an upsell for the hard fields. The false-premise result is the one an enterprise buyer's risk team will ask about first, and here it held at every price point.
What depth costs on a real research job
The jobs above were small, and even the high level stayed near two cents. A full research brief is different. I asked for a cited one-page brief on three public companies at four levels. At fast, the agent leaned on each company's homepage. At high it found the annual report, current job postings and trade press, and cited 11 to 13 sources instead of 3 to 6.
| Effort level | Time per brief | Cost per brief | What it found |
|---|---|---|---|
| Fast | 8-9 s | $0.004-0.005 | Company homepage, a few news items |
| Low | 20-77 s | $0.006-0.024 | Similar to fast; time varied widely |
| Medium | 71-111 s | $0.028-0.034 | Adds full-page reads of key sources |
| High | 61-85 s | $0.10-0.13 | Annual report, job postings, trade press |
Every response states its own cost in dollars, broken down by step, so a developer can model spend without talking to anyone. Two lines run above the price list. Reading a full page is listed at $0.0005 but was billed at $0.0137, because the model's work on that page is charged to the same step. And at high effort about 60% of each bill was a caching charge on requests that never reused the cache.
Underneath the agent: search and models
The same key also sells the two parts the agent is built from, for teams that want to assemble their own. I tested both against what I run at home.
Search. To get readable text from the top five web pages for a question, my own setup makes six calls, and sites refused a quarter of them. The Search API does it in one call, in a fifth of the time, with usable text from almost every source.
Getting usable web text: my stack vs the Search API
32 questions, top 5 sources eachMedian time to usable textseconds, lower is better
Sources with usable textpercent, higher is better
Questions left shortunder 3 usable sources, of 32
Models. Orchestrator puts models from several labs behind one endpoint. I gave seven of them the same job: label 30 real headlines positive, negative or neutral. They disagreed on only 7 headlines, yet the most expensive cost 217 times as much as the cheapest. Defaults drove much of it. Many models "reason" in hidden notes before answering, and those notes are billed. One model wrote 116 tokens to deliver a one-word answer, and two models ignored the setting meant to turn that off.
Agreement vs cost, seven models at two settings
Share of 30 headlines where a model matched the majority label, against cost per 1,000 (log scale)The agent is the easy entry point, and the parts underneath are where a growing account goes when it wants control over cost. A customer who outgrows the agent's defaults doesn't need to leave for another vendor. They can move down a layer on the same key and the same bill.
What a new developer runs into
None of these is serious alone. Together they are what a self-serve developer meets before anyone at the company knows they exist.
| Finding | Why it matters commercially |
|---|---|
| No published accuracy or cost guide per effort level for everyday jobs | Customers guess the level, and guessing wrong costs either accuracy or 5-7x the bill. |
| Add-on charges run above list price | Page reads billed at 27x list, and caching charges on one-shot calls. The first large invoice surprises. |
| Docs and live API disagree | Citations arrive in a different format than documented, so code written from the docs finds none. |
| New accounts start at 1 request per second | The first real evaluation hits the wall, and some models are locked until $50 is spent. |
| Model "reasoning" is on by default and not always switchable | A simple job ported to a new model can cost 8x what it needs to. |
What this means
- Lead with accuracy per dollar, not price per call.The cheapest effort level was the most expensive way to get a right answer on anything multi-step. Cost per correct answer is the number that wins the comparison against a do-it-yourself stack.
- The effort level is the first sale.Running a customer's own questions across levels in week one sets their accuracy and their bill. It's a short, concrete exercise with an obvious next step.
- Trust is a feature to sell, with evidence.36 of 36 false premises corrected, zero failed calls, valid structured data every time. Those are the questions a risk team asks before a contract, answered before they ask.
Method and limits
All tests ran November 19-20, 2026, from one server on home fibre in San Diego, with plain HTTPS calls, no SDK, on a new account's starting tier. Agent costs are what each response reported; other costs are tokens multiplied by list price. About $3.60 in total, not yet reconciled against the billing portal.
- Small samples: 15 lookups, 10 multi-step questions, 20 companies, 12 false premises, 12 research briefs.
- The test sets are mine, not a standard benchmark. The lookups were easy enough that every level scored 100%.
- Security-chief names and research briefs were not fact-checked line by line.
- Not tested: the xhigh level, behaviour when out of credit or rate-limited, and load above one request per second.