Go to Groundby Jaidev Karthickeyan
← All notes Infrastructure · January 2026

QuarryCan enterprises afford to run AI at scale, under control, for long enough to matter?

Token bills grow with every step an agent takes. A cluster you own doesn't.

On the same workload, metered inference costs 2.4¢ a case and owned capacity 8.7¢. Once agents use about 3.6 times the tokens, owning is cheaper, and agents typically use four times.

What happened

A dispute-resolution agent that costs about $24,000 a month to run can, with a few sensible upgrades, cost over $1m a year. Nobody changed the model. The agent just started doing its job properly.

Enterprise AI has crossed a quiet threshold. Whether large language models work is settled; they do. The open question is whether companies can afford to run them at scale, under control, for long enough to matter. So the strategic choice, as I see it, is not which model. It is which economic and governance posture comes with it: hyperscaler AI, sovereign AI or private AI.

Hyperscaler AI is built for speed. Infrastructure arrives as APIs and managed services, and you pay for what you use. The catch is that what you use doesn't sit still. As AI systems become more agentic, more multi-step and more persistent, their behaviour changes faster than anyone's budget.

The arithmetic

Dispute resolution, hyperscaler
Steps per case10 inference, 2 retrieval, 1 evaluation
Cases per month1 million
Tokens per case, end to end12,000
Tokens per month12 billion
Blended rate, already optimised$0.002 per 1,000 tokens
Inference cost per monthabout $24,000
With longer context, a QA reasoning loop and continuous evaluation (tokens double)approaching $600,000 a year
Adding vector storage, observability, API surcharges and downstream callsover $1m a year
Same workload, private cluster
Hardware32 modern GPUs, amortised over 4 years
Annualised cost, including infrastructure, power and platform software$1.3m
Cases per year15 million
Inference cost per caseunder 9¢

Illustrative figures from the original article. Metered cost per case: $24,000 ÷ 1 million cases = 2.4¢ before the extra loops.

Three postures

HyperscalerSovereignPrivate
You pay forEach token and service callEach token, in-countryCapacity you own
Cost as workloads get more complexConvex: rises faster than useConvex: same mathematicsConcave: marginal cost barely moves
Governance costScales with activityScales with activityMostly fixed once built
LatencyGood on average, variable at the tailAs hyperscalerSimilar median, far less variance
Best forPilots, bursts, frontier accessRegulated data that must stay localStable, inference-heavy, governed work

Going to ground

First order

Every step an agent takes is on the meter. Extend the context window, add a second reasoning loop for quality, add continuous evaluation for compliance, and the bill doubles before storage, observability and surcharges are counted. Better capability invites more use, and more use invites more capability, and the meter runs the whole time. Sovereign AI changes where the data sits, not how the cost behaves: it is still metered, so it still punishes complexity.

Second order

Owning turns a billing problem into an engineering problem. The question shifts from "how many tokens did we burn?" to "how well are we using what we own?" Governance splits the same way. Rented, every extra interaction means more logs, more evaluations and more spend; owned, new workloads inherit controls at little extra cost, and in regulated industries that alone can justify the move. Latency too: shared infrastructure optimises the average across tenants, while dedicated paths cut the variance, which raises how much can be automated. Even spiky work like the month-end close can favour owning. Sizing for the peak once is often cheaper than paying premium burst rates twelve times a year, especially when the work can't tolerate throttling.

Third order

The winners will run a split estate. Exploration, pilots and volatile workloads stay in the cloud, where hyperscaler and sovereign AI are indispensable for frontier access and uncertain demand. Stable, inference-heavy, governed workloads move to private platforms, where cost, latency and compliance can be engineered rather than rented. For executives, this is a capital allocation decision dressed up as an architecture decision: token pricing turns behavioural uncertainty into financial uncertainty, and capacity pricing turns financial commitment into operating discipline.

Ears up: what to watch

  • Tokens per completed case in your own production workloads, quarter on quarter.
  • Utilisation of owned clusters. The private line only holds if the capacity is actually used.
  • The pace of token price cuts, which pushes the crossover further out.
  • Sovereign cloud pricing: whether any provider offers capacity pricing instead of a meter.

Bone of contention

My crossover is only as good as its assumptions. Anthropic's 4× and 15× compare agents with chat; applying them to a pipeline that is already multi-step is illustrative. The private line stays flat only while the 32-GPU cluster has headroom: at fifteen times the tokens, it would need more hardware. And if token prices keep falling faster than hardware costs, the metered line flattens and the crossover moves right. Owning is cheaper only if you keep the thing busy.

J. & the researcher

Sources
  1. Anthropic, "How we built our multi-agent research system" (13 June 2025): agents use about 4× the tokens of chat; multi-agent systems about 15×confirmed
  2. Workload, token, rate and cluster figures: scenarios from the original articlescenario

Data for the chart: inference-cost-per-case.csv · Originally published on Medium.