Machine learning at the edge

What is machine learning at the edge?#

Machine learning at the edge means running ML models (most often inference, the stage where a trained model makes predictions) on infrastructure located close to where the data is generated and the results are consumed, rather than in a distant central cloud region.

Training usually stays in large clusters, because it needs sustained access to big datasets and lots of parallel compute. Inference is different: it happens on every user request, its output is needed immediately, and it benefits enormously from being nearby. That split (train centrally, infer at the edge) is the pattern behind most edge ML deployments, and it mirrors how CDNs and edge computing divide work more generally.

Why latency matters for AI workloads#

An inference request’s response time has two parts: the time the model spends computing, and the time the request and response spend travelling over the network. You can buy faster hardware to attack the first part, but the second is governed by distance. A user in Singapore calling a model hosted in Virginia pays a transcontinental round trip on every single request, before the model does any work at all.

For interactive AI, such as chat interfaces, voice assistants, live translation, recommendations and fraud checks in a payment flow, that network tax is the difference between an experience that feels instant and one that feels sluggish. Streaming LLM responses makes it worse, not better: every token round trip inherits the same latency floor.

Moving inference to nodes near the user removes most of the travel time. It also reduces how much raw data has to cross the internet: a computer-vision system that analyses video at the edge sends back results, not footage.

GPU compute at the edge#

Modern models, particularly LLMs and vision models, run far faster on GPUs than CPUs, so edge ML at scale means putting GPU capacity into the distributed network itself. This is a natural fit for a decentralised platform: the same model that distributes nodes for content delivery can distribute GPU nodes for inference.

Edge is bringing GPU instances to its network, with support for multiple vendors including NVIDIA, AMD, and Intel, billed hourly with no long-term commitments. In the meantime, Edge Compute VMs run CPU-based inference today, and smaller quantised models are genuinely practical on CPU.

The economics matter as much as the hardware. Model serving moves a lot of data, and on traditional clouds that means egress charges on every response. Edge charges zero egress fees, so serving predictions or shipping model weights between environments carries no bandwidth cost. See AI & Machine Learning on Edge for the full picture.

Deploying models: Ollama, vLLM, and llama.cpp#

You don’t need a bespoke ML platform to serve models on edge infrastructure. The open-source ecosystem has settled on a few excellent serving tools, each of which deploys cleanly to an Edge VM:

• Ollama: the simplest way to run open models like Llama and Mistral on your own VM, with a clean local API.
• vLLM: high-throughput serving for production LLM workloads, with efficient batching and an OpenAI-compatible API.
• llama.cpp: lightweight, CPU-friendly inference for quantised models. Useful where GPU capacity is unnecessary or unavailable.

The workflow is the same in each case: provision a VM sized for your model, install the serving tool, load your weights, and put the endpoint behind Edge CDN or Edge DNS for routing and TLS.

Benefits of ML at the edge#

The gains compound: faster responses, lower bills, and less data in transit.

• Lower latency: inference happens close to the user, cutting the round trip that dominates response time for real-time AI features.
• Lower cost: moving less data to and from a central cloud means less bandwidth spend. On Edge, egress is free entirely.
• Better privacy: data can be processed near where it is generated rather than shipped to a distant region, reducing exposure in transit.
• Resilience: distributed inference keeps working when any single location fails, because there is no central endpoint to go down.

Run your models on Edge#

Deploy inference on Edge Compute today, and join the waitlist for GPU instances.