Solo development with AI
In solo development with AI you do the work as a trio: you, the AI assistant in the IDE and autonomous agents that iterate on things while you switch to another task. This mode has its own set of “blind spots”:
- Long-running autonomous tasks — an agent spends half an hour dealing with a DB migration, and you don’t want to check the terminal every 30 seconds.
- External dependencies — LLM APIs, vector DBs, embeddings service: their status changes without warning, and you don’t have an SRE team.
- Cost — a lot of tokens get burned; it’s easy to lose control of the bill.
- Model quality — a new model release can break your pipeline right in the middle of the night.
- MCP tools — if your AI calls its MCP server, its outage looks like “the model is acting up”, not like an alert.
Notifly covers all of this in a few lines: heartbeats, active monitors, webhooks and custom cloud functions for synthetic checks send push to your device exactly when you need to intervene.
Ready-made recipes
Section titled “Ready-made recipes”| Scenario | What it covers |
|---|---|
| LLM API spending | Exceeded token budget |
| LLM provider availability | OpenAI/Anthropic/etc. not responding, 5xx, rate limits |
| Model latency degradation | Sudden “model has slowed down” |
| Long AI agent task completion | ”Agent finished refactoring — go review it” |
| Stuck agent / loop | Heartbeat protection against silent hangs |
| Human-in-the-loop needed | Agent paused a task — your approval is required |
| Vector DB / RAG infrastructure | Qdrant/Pinecone/Weaviate are down |
| Eval / model quality drop | Regression after changing prompt or model |
| Safety / prompt injection triggered | Suspicious user input |
| MCP server health | Own or third-party MCP stopped responding |
| Custom cloud function integrity check | Server-side “synthetic user” that sends an alert when something breaks |
Additional recipes
Section titled “Additional recipes”Cost and quotas
Section titled “Cost and quotas”- Model token quotas
- Cost per request
- Voice API quotas (TTS / STT)
- Infrastructure costs (YC / AWS)
- GPU utilization (idle or saturated)
- Reserved capacity under-utilized
- Provider free-tier exhausted
- Batch API job finished
- Embedding cost spike
Availability and latency
Section titled “Availability and latency”- Cold-start serverless
- SLO p95 breach
- Traffic anomaly
- LLM task queue growing
- Provider rate-limit 429
- Token streaming stall (TTFT)
- Fallback to a backup model
- LLM endpoint region failover
- Realtime/voice websocket drop
- Inference-server queue backlog
AI agents
Section titled “AI agents”- Browser agent blocked (CAPTCHA / 403)
- Context window near limit
- Agent cost runaway in a single run
- Agent hammering a failing tool
- Agent tool-call timeout
- Multi-agent orchestration deadlock
- Approval stalled and expired
- Scheduled (nightly) agent run report
- Agent touched protected files / git
Response quality
Section titled “Response quality”- Hallucination spike
- User thumbs-down
- Drop in prompt-cache hit rate
- Dataset drift
- Fine-tune completed
- Spike in model refusals (“I can’t help with that”)
- JSON structured-output parse failures
- Empty and truncated responses
- Repetition and looping in generation
- Answer in the wrong language
- RAG answer without sources
- Toxic app output
- Nightly eval finished — scorecard
RAG and data
Section titled “RAG and data”- Embeddings version change
- Schema drift in JSON response
- Vector index rebuild finished
- Empty RAG retrieval (0 chunks)
- Document ingestion failures
- Stale knowledge index
- Chunking and index config drift
Security
Section titled “Security”- Secrets in LLM logs
- New CVE in dependencies
- Moderation during image generation
- API keys and license expiry
- Moderation queue backlog
- PII in a prompt or response
- API-key usage spike (leak)
- Request blocked by provider content policy
- Data exfiltration via tool call
Infrastructure and development
Section titled “Infrastructure and development”- AI does PR review
- Flaky tests found
- Canary deployment errors
- Progress of long DB migration
- Webhook retry storm
- Memory leak in serverless
- Mobile client crash
- Model deprecation / sunset
- New model shipped — worth benchmarking
- Training checkpoint / run progress
- CUDA out-of-memory in training/inference
- Model weights download done/failed
Personal
Section titled “Personal”Why push, not email
Section titled “Why push, not email”LLM incidents are often short and noisy: 10 minutes of 500s → fixed. Email alerts arrive in batches with delay and drown in email; a push notification to your phone via Notifly is instant, free, and easy to silently dismiss if you see the incident has already resolved itself.
All recipes in this section use the same basic stack:
POST /message— ad-hoc alerts from code;- Heartbeat — “agent is alive”;
- Active monitors — external HTTP/TCP/TLS checks;
- Webhooks — receipt of events from billing/CI;
- Email Inbox — alerts from providers that don’t have webhooks;
- custom cloud functions on YC — for your own checks.