Ollama on Edge, open models on your own VM

1. 01

   Write a bootstrap script

   Installs Ollama and the Edge CLI, keeps Ollama on localhost and stores models where the sync step expects them.

   ollama-setup.shCopy

   #!/bin/bash
   set -e
   curl -fsSL https://ollama.com/install.sh | sh
   curl -fsSL https://edge.network/install.sh | sh
   mkdir -p /var/lib/ollama/models
   chown -R ollama:ollama /var/lib/ollama
   mkdir -p /etc/systemd/system/ollama.service.d
   cat > /etc/systemd/system/ollama.service.d/edge.conf <<'EOF'
   [Service]
   Environment="OLLAMA_HOST=127.0.0.1:11434"
   Environment="OLLAMA_MODELS=/var/lib/ollama/models"
   Environment="OLLAMA_KEEP_ALIVE=24h"
   EOF
   systemctl daemon-reload && systemctl restart ollama
   apt-get install -y nginx

2. 02

   Create the VM

   8 vCPU and 16 GiB leave room for an 8B Q4 model plus an embedding model. --disk 160 gives space for several models.

   shellCopy

   $ edge compute scripts create --name ollama-setup \
       --file ollama-setup.sh
   $ edge compute create --name llm --size s-8vcpu-16gb --disk 160 \
       --image ubuntu-24 --region london --script ollama-setup

3. 03

   Pull models and cache them

   Pull once, then sync to your bucket. A replacement VM syncs back before Ollama starts.

   shellCopy

   $ ollama pull llama3.2:3b
   $ ollama pull nomic-embed-text
   $ edge storage sync /var/lib/ollama/models llm-weights/ollama/
   
   # on a new VM, before starting Ollama
   $ edge storage sync llm-weights/ollama/ /var/lib/ollama/models

4. 04

   Check the key in Nginx

   OpenAI SDKs send the key as a Bearer token. Buffering is off so tokens stream as they're generated.

   /etc/nginx/sites-enabled/ollamaCopy

   server {
     listen 443 ssl;
     server_name ai.acme.com;
     ssl_certificate     /etc/ssl/origin.pem;   # origin certificate
     ssl_certificate_key /etc/ssl/origin.key;
   
     location /v1/ {
       if ($http_authorization != "Bearer sk-acme-change-me") {
         return 401;
       }
       proxy_pass http://127.0.0.1:11434;
       proxy_buffering off;
       proxy_read_timeout 300s;
     }
   }

5. 05

   Put the endpoint behind the CDN

   Add a bypassCache rule for /v1/** in the deployment's configuration, then call it like any OpenAI endpoint.

   shellCopy

   $ edge cdn create --name ai
   $ edge cdn domains add cdn-a1b2c3 \
       --domain ai.acme.com --origin https://<vm-ip>
   
   $ curl https://ai.acme.com/v1/chat/completions \
       -H "Authorization: Bearer $OLLAMA_KEY" \
       -H "Content-Type: application/json" \
       -d '{"model": "llama3.2:3b",
            "messages": [{"role": "user", "content": "Hi"}]}'

Prefer to hand it off? Give the job to your AI agent or have our engineers do it.