Intelligence at the Edge

The test we use

When we evaluate whether a system is fit for an operational environment, we ask one question: what happens if the connection goes away for four minutes at the busiest moment of the day?

The answers divide sharply. Some systems queue the work locally, keep accepting input, and reconcile when the link returns — the user may not notice at all. Others stop, and the business stops with them: orders taken on paper, a backlog to key in afterwards, and mistakes that surface on the bill.

This is not an edge case in Bangladesh, and it is not an edge case anywhere with a busy kitchen and a router behind a stack of crates. It is a Thursday. Any system whose correctness depends on continuous connectivity has been designed for a demonstration rather than for the room.

What happens if the connection goes away for four minutes at the busiest moment of the day?

Latency is a product decision, not an engineering one

There is a threshold, somewhere around a tenth of a second, below which a response feels like a consequence of your action and above which it feels like a request you have made. Cross it and people change how they use the software. They stop trusting that the tap registered. They tap again. On a point-of-sale system, tapping again is a duplicate order.

A round trip to a server — even a fast one, even a nearby one — spends most of that budget before any work is done. So the question is not how to make the round trip faster. It is which decisions have to happen locally, and the answer is: every decision the user is waiting on.

This reframes the architecture. The network is not where the system lives; it is how instances of the system agree with each other afterwards. That is a different design, and it is considerably more work, which is why most software does not do it.

Where inference belongs

The same argument applies, with more force, to anything predictive.

Consider the useful things a kitchen display could know. Which orders are about to be late, given what is already on the pass. Whether a table's courses are going to arrive out of order. Which item is being sold faster than the prep can sustain. None of these are hard predictions. All of them are worthless if they arrive thirty seconds after the decision had to be made, and all of them require knowing the state of the room right now.

Sending that state to a remote model and waiting for an answer is the wrong shape twice over: it is too slow to act on, and it fails exactly when the kitchen is busiest and the network is worst. A small model running on the device, on data that never leaves the building, is both faster and more robust — and it is a genuinely smaller engineering problem than it sounds, because the questions are narrow.

The heavy work still belongs centrally. Training, aggregate analysis across branches, anything that needs history rather than the present moment. The division is not cloud versus edge; it is that the edge decides and the centre learns.

The edge decides. The centre learns.

The cost of doing it this way

Being fair about the trade-off: local-first systems are harder to build and meaningfully harder to reason about. Two devices that both accepted input while disconnected will eventually disagree, and resolving that disagreement is real design work that cannot be skipped or automated away. You need to decide, per kind of data, what wins.

Deployment is harder too — software on a hundred devices in twelve branches is not software on one server. And there is a real temptation to over-apply the principle: not everything needs to work offline, and building it that way where it is not needed is expense without return.

The test earns its keep here as well. If the user is waiting on the answer to do their job, it belongs locally. If they are not, it does not.

Why this is becoming easier

Two trends are moving in the right direction. Models capable of doing narrowly useful work have become small enough to run on ordinary hardware — the kind of tablet already on a restaurant floor — rather than requiring anything exotic. And the tooling for local-first data, the unglamorous problem of syncing and reconciling, has improved considerably.

The result is that an architecture which was recently the preserve of companies with infrastructure teams is now a reasonable default for a system serving a dozen branches. We think that is where operational software is going, for the plain reason that it is where the users already are.