llama.cpp on Edge, the leanest LLM you can run

Deploy

llama-server live in five steps

A CPU build, one GGUF file, a systemd unit and Caddy. Everything else is optional.

~/llm — zsh

$ edge compute create --name llm \

--size s-4vcpu-8gb --image ubuntu-24 \

--region london --script llamacpp-build

✓ vm-8d20c1 running · llama.cpp built

$ M=Llama-3.2-3B-Instruct-Q4_K_M.gguf

$ edge storage cp llm-weights/$M .

$ llama-server -m $M \

--port 8080 --api-key $LLM_KEY

✓ server is listening on 127.0.0.1:8080

✓ Live at https://llm.acme.com/v1

Tokens/s · 3B Q4, 4 vCPU~8

llm.acme.comRunning

Serverllama-server · :8080

ModelLlama 3.2 3B · Q4_K_M

1. 01

   Build llama-server on first boot

   A plain CPU build. CMake picks up the instruction sets the VM supports.

   llamacpp-build.shCopy

   #!/bin/bash
   set -e
   apt-get update
   apt-get install -y build-essential cmake git libcurl4-openssl-dev caddy
   curl -fsSL https://edge.network/install.sh | sh
   git clone https://github.com/ggml-org/llama.cpp /opt/llama.cpp
   cd /opt/llama.cpp
   cmake -B build -DCMAKE_BUILD_TYPE=Release
   cmake --build build -j"$(nproc)" --target llama-server
   ln -s /opt/llama.cpp/build/bin/llama-server /usr/local/bin/

2. 02

   Create the VM and point DNS at it

   A Performance VM handles 3B models comfortably and fits an 8B Q4 in memory.

   shellCopy

   $ edge compute scripts create --name llamacpp-build \
       --file llamacpp-build.sh
   $ edge compute create --name llm --size s-4vcpu-8gb \
       --image ubuntu-24 --region london --script llamacpp-build
   $ edge dns records add zone-abc123 --type A --name llm --data <vm-ip>

3. 03

   Cache the model in Edge Storage

   Download once from Hugging Face, keep it in your bucket, and copy it onto any VM that needs it.

   shellCopy

   $ hf download bartowski/Llama-3.2-3B-Instruct-GGUF \
       --include "*Q4_K_M.gguf" --local-dir .
   $ edge storage cp Llama-3.2-3B-Instruct-Q4_K_M.gguf llm-weights/
   
   # on the VM
   $ edge storage cp \
       llm-weights/Llama-3.2-3B-Instruct-Q4_K_M.gguf /opt/models/

4. 04

   Run it as a service

   llama-server listens on localhost and requires the API key on every request.

   /etc/systemd/system/llama.serviceCopy

   [Unit]
   Description=llama.cpp server
   After=network-online.target
   
   [Service]
   EnvironmentFile=/etc/llama.env   # LLAMA_API_KEY=...
   ExecStart=/usr/local/bin/llama-server \
     -m /opt/models/Llama-3.2-3B-Instruct-Q4_K_M.gguf \
     --host 127.0.0.1 --port 8080 -c 8192 \
     --api-key ${LLAMA_API_KEY}
   Restart=always
   
   [Install]
   WantedBy=multi-user.target

5. 05

   Add TLS with Caddy

   Caddy fetches a certificate for llm.acme.com and streams tokens as they're generated.

   /etc/caddy/CaddyfileCopy

   llm.acme.com {
     reverse_proxy 127.0.0.1:8080 {
       flush_interval -1
     }
   }

Prefer to hand it off? Give the job to your AI agent or have our engineers do it.