Skip to content
Notifly Notifly

Model fine-tuning completed

A fine-tune can take from 30 minutes to several days. Opening the dashboard every hour is torture. A polling function that runs every N minutes to check the job status:

import os, json, requests, openai


JOB_ID = os.environ["FT_JOB_ID"]
STATE  = "/tmp/ft-state.json"


def handler(event, context):
    job = openai.fine_tuning.jobs.retrieve(JOB_ID)
    cur = job.status                     # validating_files / running / succeeded / failed / cancelled
    prev = (json.load(open(STATE)) if os.path.exists(STATE) else {}).get("status")
    if cur != prev:
        push(f"🎓 Fine-tune: {cur}",
             f"Job: {JOB_ID}\nModel: {job.fine_tuned_model or '—'}\n"
             f"Trained tokens: {getattr(job,'trained_tokens',None)}",
             10 if cur == "failed" else 5 if cur == "succeeded" else 4)
        json.dump({"status": cur}, open(STATE, "w"))
    return {"statusCode": 200}


def push(t, m, p):
    requests.post(f"{os.environ['NOTIFLY_URL']}/message",
                  params={"token": os.environ["NOTIFLY_TOKEN"]},
                  json={"title": t, "message": m, "priority": p}, timeout=5)

Set the timer to */5 * * * ? *. After succeeded, immediately run eval-runner and compare with the base model.

Section titled “Related recipes”