Index of Plates

Plate VIII

Cloud AI or Local AI? Why I Stopped Having This Debate

AI Systems

14 NOVEMBER 20265 Min Read

Running models on your own machine is fun, and sometimes it's the right call. But you can't out-GPU the frontier labs, so for most real work the answer is simple: use the best model you can reach, and save local for the few cases where it actually wins.

Every few weeks the same argument comes back around. Should you use cloud AI, the frontier models from Anthropic, OpenAI and Google, or run models locally on your own laptop or GPU?

People get surprisingly passionate about it. And for a while I went back and forth too: downloading open-weight models, comparing quantisations, tuning settings to get a few more tokens a second out of my machine.

Then I asked myself a simpler question. What am I actually trying to get done? Once I answered that honestly, the debate mostly went away.

The Uncomfortable Truth About Local

You can't beat the top models on your own hardware. It's not a matter of skill or settings. It's physics and money.

The frontier models run on data centres with tens of thousands of top-end GPUs, connected by networking most of us will never touch. The labs train them for months, on budgets measured in billions, then serve them on hardware tuned for exactly that job.

A good laptop has one GPU and a fixed amount of memory. That limits which models you can even load. A model with 70 billion parameters needs roughly 40 GB of memory just to hold its weights when compressed to 4 bits. A small 8-billion-parameter model fits in about 5 GB and runs nicely, but it's in a different league from the frontier.

So when you run locally, you're usually picking from two options:

  • A small model that runs fast, but makes more mistakes, reasons less deeply and handles long, messy tasks worse.
  • A bigger model that runs slowly, if your machine can load it at all, and still sits behind the best cloud models.

Open-weight models have improved a lot, and the gap is smaller than it used to be. But by the time a model that matches today's frontier runs well on a laptop, the frontier has moved on.

What Actually Matters for Real Work

When I'm building something, three things matter more than anything else.

Quality of the answer. In real work, a wrong answer isn't cheap just because it was free. A subtle bug in generated code or a missed detail in a document costs far more time than the API call saved. The best model makes fewer of those mistakes.

Speed. Cloud providers serve models on hardware built for it. For most tasks a frontier model in the cloud answers faster than a comparable model on my machine, and it doesn't turn my laptop into a space heater while it does it.

My own time. Running models locally means downloading, updating, picking quantisations, managing memory and debugging drivers. That's fun as a hobby. As part of shipping a product, it's a tax.

On all three, the cloud usually wins. And the price is no longer the problem it used to be. Prices for a given level of capability fall steeply every year, and for an individual developer the monthly bill for serious use is often less than the cost of the hardware upgrade you'd need to run a decent model locally.

So Why Does the Debate Keep Coming Back?

Because local AI isn't useless. There are real cases where it's the right choice, and I don't want to pretend otherwise. They're just narrower than the debate suggests.

Data that can't leave the building. Medical records, legal documents, some company code. If the rules or the contract say the data stays on your hardware, that decides it. Even then, check first: many cloud providers now offer zero data retention and enterprise agreements that cover this.

No internet, or a bad one. On a plane, in the field, on a robot or a drone. When the network isn't guaranteed, the model has to run on the device. This is where small models really shine, and it's a big part of why physical AI needs efficient on-device models.

Huge volumes of simple work. If you're classifying millions of short messages a day, a small model on your own GPU can be far cheaper than paying per token. That's a scale problem, not a typical developer problem.

Learning how it works. Running a model yourself teaches you a lot: what tokens and context windows really are, why quantisation changes behaviour, where the memory goes. I recommend every engineer try it once with something like Ollama or llama.cpp. Just don't confuse the learning project with the production choice.

Notice what's missing from that list: "because I want the best results." If that's the goal, local is the wrong tool.

The Rule I Use Now

My default is simple:

  1. Start with the best cloud model you can access. Get the thing working, with the strongest model available.
  2. Only go local when a specific constraint forces you to. Privacy rules, offline use, or a cost problem you've actually measured, not one you imagine.
  3. When you do go local, use it for the narrow part. Keep the small model on the simple, high-volume or sensitive step, and let the frontier model handle the hard thinking.

That last point matters. It's rarely all or nothing. The best systems I've seen use both: a small model where it's good enough, and the top model where quality decides the outcome.

Stop Arguing, Start Building

The cloud-versus-local debate often feels like an identity question: are you the kind of engineer who runs their own stack? But the people getting the most out of AI right now aren't the ones with the most elaborate local setup. They're the ones solving real problems with the best tools available.

You're not going to out-GPU the companies spending billions on data centres. You don't need to. Their scale is something you can rent by the token.

Use it. Spend your energy on the problem, not the infrastructure. And keep local AI for the cases where it clearly earns its place.

Written & engraved by Mayank Bairagi