Model weights download finished or failed

The weights of modern models are tens, sometimes hundreds, of gigabytes. Downloading a 70B model over a normal link takes hours, and all that time the terminal is busy with a progress bar there is no point in watching. Worse — the download can silently drop at 90% (a network break, disk out of space, hub rate-limit), and you discover it only when you come back. A push on completion/failure frees you.

The pattern is like “completing a long task”: wrap the command and send the result by its return code.

Option 1: a bash wrapper around huggingface-cli download

We time it and get the final size, distinguishing success from failure by the exit code:

#!/usr/bin/env bash
# dl_model.sh REPO [DEST]
REPO="$1"
DEST="${2:-./models/$(basename "$REPO")}"

start=$(date +%s)
huggingface-cli download "$REPO" --local-dir "$DEST" --local-dir-use-symlinks False
code=$?
dur=$(( $(date +%s) - start ))

if [[ $code -eq 0 ]]; then
  size=$(du -sh "$DEST" | cut -f1)
  title="✅ Downloaded: $(basename "$REPO")"
  msg="Size: $size, time: $((dur/60)) min $((dur%60)) s"
  prio=5
else
  title="❌ Download failed: $(basename "$REPO")"
  msg="Exit $code after $((dur/60)) min. Check the network / disk space."
  prio=8
fi

curl -fsS "$NOTIFLY_URL/message?token=$NOTIFLY_TOKEN" \
  -H "Content-Type: application/json" \
  -d "$(jq -n --arg t "$title" --arg m "$msg" --argjson p "$prio" \
        '{title:$t, message:$m, priority:$p}')"
exit $code

Success is priority=5 (a normal push, “you may get back to work”), failure is priority=8 (you need to step in, download again). Run it with: ./dl_model.sh meta-llama/Llama-3.1-70B ./weights.

Option 2: ollama pull with the same pattern

ollama pull also returns a non-zero code on failure:

#!/usr/bin/env bash
MODEL="$1"
start=$(date +%s)
ollama pull "$MODEL"
code=$?
dur=$(( $(date +%s) - start ))

if [[ $code -eq 0 ]]; then
  size=$(ollama list | awk -v m="$MODEL" '$1==m {print $3, $4}')
  msg="Model $MODEL is ready (${size:-see ollama list for size}) in $((dur/60)) min"
  prio=5; title="✅ ollama pull done"
else
  msg="ollama pull $MODEL failed (exit $code) after $((dur/60)) min"
  prio=8; title="❌ ollama pull failed"
fi
curl -fsS "$NOTIFLY_URL/message?token=$NOTIFLY_TOKEN" \
  -H "Content-Type: application/json" \
  -d "$(jq -n --arg t "$title" --arg m "$msg" --argjson p "$prio" '{title:$t,message:$m,priority:$p}')"

Option 3: Python with progress and an ETA in an interim push

If you download from Python (huggingface_hub.snapshot_download with a callback), you can send one quiet interim push at the halfway point, to know the process is alive:

import os, time, requests
from huggingface_hub import snapshot_download

def notify(title, msg, prio):
    requests.post(f"{os.environ['NOTIFLY_URL']}/message",
                  params={"token": os.environ["NOTIFLY_TOKEN"]},
                  json={"title": title, "message": msg, "priority": prio}, timeout=5)

t0 = time.time()
try:
    path = snapshot_download(repo_id="meta-llama/Llama-3.1-8B")
    size_gb = sum(os.path.getsize(os.path.join(dp, f))
                  for dp, _, fs in os.walk(path) for f in fs) / 1e9
    notify("✅ Weights downloaded",
           f"{size_gb:.1f} GB in {int(time.time()-t0)//60} min\n{path}", prio=5)
except Exception as e:
    notify("❌ Download failed", f"{type(e).__name__}: {e}", prio=8)
    raise

For very long downloads, add a heartbeat into the progress callback — then a download stuck at 60% (not crashed, but exactly frozen) will also report itself on timeout.

What to put in the alert text

• the repo/model name and the destination path;
• the final size and total time (for success);
• the exit code and the cause, if known: network / disk / rate-limit (for failure);
• how much space is left on the disk — a common cause of a failure at 90%.