Why on-device inference matters
A short primer on why moving inference to the device — not the cloud — is the unlock for sustainable, universal AI.
Most production AI today runs in a data center hundreds of milliseconds away from the user. That is a deliberate choice, not a law of physics — and it has costs.
The hidden costs of cloud-first AI
Three forces push intelligence off the device and into the cloud, and each one carries a price:
- Latency. A round trip to a data center is fine for chat. It is not fine for a robot picking up a fragile object, a hearing aid filtering speech in real time, or a car deciding whether to brake.
- Privacy. Every cloud inference is a copy of user data leaving the device. Some categories of users — patients, lawyers, defence operators — cannot accept that copy at any price.
- Cost and energy. A single GPU inference can cost more than the action it powers. Run that at billions-per-day scale and the math stops working.
What “on device” actually requires
Running a useful model on a phone or an embedded chip is not just about quantising weights to 4 bits and hoping. It requires the model to be small enough to fit, fast enough to feel instant, and accurate enough to be worth shipping. That trifecta is what our compression work is aimed at.
We will go deeper on each of these in future posts. Subscribe via RSS if you want them in your reader.