vLLM on Edge: build the serving stack now

Deploy

vLLM on CPU in five steps

Follows vLLM's build-from-source steps for x86 CPUs. Check docs.vllm.ai for the current release before you pin a version.

~/inference — zsh

$ edge compute create --name inference \

--size s-8vcpu-16gb --disk 160 \

--image ubuntu-24 --region london \

--script vllm-cpu

✓ vm-1f6c3a running · vLLM CPU build ready

$ vllm serve Qwen/Qwen2.5-1.5B-Instruct \

--port 8000 --api-key $VLLM_KEY \

--max-model-len 8192

✓ Application startup complete.

✓ OpenAI-compatible · /v1/chat/completions

Egress on weight pulls$0

inference.acme.comHealthy

Servervllm serve · :8000

ModelQwen2.5-1.5B · bf16

1. 01

   Write a bootstrap script

   Builds vLLM with the CPU backend into a virtual environment, and installs the Edge CLI for weight syncs.

   vllm-cpu.shCopy

   #!/bin/bash
   set -e
   apt-get update
   apt-get install -y build-essential libnuma-dev python3-dev git
   curl -fsSL https://edge.network/install.sh | sh
   curl -LsSf https://astral.sh/uv/install.sh | sh
   . $HOME/.local/bin/env
   uv venv /opt/vllm --python 3.12 && . /opt/vllm/bin/activate
   git clone https://github.com/vllm-project/vllm.git /opt/vllm-src
   cd /opt/vllm-src
   uv pip install -r requirements/cpu-build.txt --torch-backend cpu
   uv pip install -r requirements/cpu.txt --torch-backend cpu
   VLLM_TARGET_DEVICE=cpu uv pip install . --no-build-isolation

2. 02

   Create the VM and mirror the model

   Upload weights to your bucket once. For multi-gigabyte files the docs recommend an S3 tool with multipart uploads.

   shellCopy

   $ edge compute scripts create --name vllm-cpu --file vllm-cpu.sh
   $ edge compute create --name inference --size s-8vcpu-16gb \
       --disk 160 --image ubuntu-24 --region london --script vllm-cpu
   
   $ hf download Qwen/Qwen2.5-1.5B-Instruct --local-dir qwen2.5-1.5b
   $ aws s3 sync qwen2.5-1.5b s3://llm-weights/qwen2.5-1.5b \
       --endpoint-url https://storage.edge.network

3. 03

   Run it as a service

   Weights sync from the bucket before each start. VLLM_CPU_KVCACHE_SPACE sets the KV cache size in GiB.

   /etc/systemd/system/vllm.serviceCopy

   [Unit]
   Description=vLLM OpenAI-compatible server
   After=network-online.target
   
   [Service]
   EnvironmentFile=/etc/vllm.env   # EDGE_API_KEY, VLLM_API_KEY
   Environment=VLLM_CPU_KVCACHE_SPACE=4
   ExecStartPre=/usr/local/bin/edge storage sync \
     llm-weights/qwen2.5-1.5b/ /opt/models/qwen2.5-1.5b/
   ExecStart=/opt/vllm/bin/vllm serve /opt/models/qwen2.5-1.5b \
     --served-model-name qwen2.5-1.5b \
     --host 127.0.0.1 --port 8000 \
     --api-key ${VLLM_API_KEY} --max-model-len 8192
   Restart=always
   
   [Install]
   WantedBy=multi-user.target

4. 04

   Put the endpoint behind the CDN

   Proxy :443 on the VM to port 8000, and add a bypassCache rule for /v1/** in the deployment's configuration.

   shellCopy

   $ edge cdn create --name inference
   $ edge cdn domains add cdn-a1b2c3 \
       --domain inference.acme.com \
       --origin https://<vm-ip>

5. 05

   Call it from your app

   Any OpenAI SDK works. The same client code will point at a GPU-backed server later.

   client.tsCopy

   import OpenAI from 'openai'
   
   const client = new OpenAI({
     baseURL: 'https://inference.acme.com/v1',
     apiKey: process.env.VLLM_API_KEY,
   })
   
   const res = await client.chat.completions.create({
     model: 'qwen2.5-1.5b',
     messages: [{ role: 'user', content: 'Summarise this ticket…' }],
   })

Prefer to hand it off? Give the job to your AI agent or have our engineers do it.