Skip to main content

Build an actionable anomaly watcher

A scheduled Agent Automation can compare the last hour of production activity with a seven-day baseline and notify your on-call channel only when somebody has a concrete next step.

The goal is not to report every statistical change. A useful notification must answer five questions:

  1. What is failing now?
  2. Who or what is affected?
  3. What evidence makes it urgent?
  4. What should the owner do next?
  5. How will the owner know it recovered?

Before you start​

Create these account-level resources:

  • A read-only MCP key. The watcher only needs quave:read.
  • An AI provider key.
  • A Slack, PagerDuty, email, or webhook contact point.

Avoid giving a monitoring automation deploy, configuration, secret-read, or delete scopes. Its job is to observe and hand off an action, not mutate production.

Create the automation​

Open Agent Automations for the account, select New automation, and use:

  • Name: Actionable production anomaly watcher
  • Cron expression: 0 * * * *
  • Timezone: your team's IANA timezone, or UTC
  • Delivery mode: On trigger
  • Send results to: your on-call contact point

Replace ACCOUNT_ID in the prompt below. If your production environments do not end in -prod or -production, also replace that scope rule with your own naming convention.

Prompt​

You are the production anomaly triage watcher for account ACCOUNT_ID. Write all output in English.

OUTCOME
Notify only when a human should act before the next hourly run. The delivered text must be a compact incident handoff, not an analysis diary. Never alert merely because a percentage changed or a log signature is new.

SCOPE AND BUDGET
- Analyze only environments whose summary name ends in -prod or -production. Ignore beta, staging, dev, test, and standalone databases.
- Use the tools' default windows: recent 1h versus the preceding 7d baseline.
- Do not call list-app-envs. Make at most 7 tool calls and one list-agent-automation-runs call.

1. SCREEN
- Call get-account-observability-summary with accountId ACCOUNT_ID and limit 50.
- An abnormal production status is a candidate.
- CPU or memory is a candidate only when recent p95 is at least 75% AND either absolute zScore is at least 3 or absolute percentDelta is at least 50%. A relative delta with low absolute utilization is noise.
- Select at most the two strongest production candidates. Call get-app-env-log-error-summary for them.
- A log pattern is actionable only when it is plausibly tied to failed user or operational work AND either:
  a) it is absent from baseline with at least 10 occurrences in the last hour, or
  b) it is at least 3x baseline with an absolute increase of at least 10.
  A low-volume or harmless isNew signature is noise.

2. CONFIRM IMPACT AND CAUSE
For the single strongest candidate:
- Call get-app-env-history to check for a deploy/config correlation.
- If the summary does not expose enough detail for a concrete action, call get-logs once for the relevant appEnvId, recent error stream, and a focused search from the signature.
- Suppress expected short-lived deploy churn when the environment is healthy again.
- Do not recommend restart, scale, rollback, or credential rotation unless the evidence specifically supports it.
- If you cannot state a concrete immediate action AND how to verify recovery, do not trigger yet.

3. DEDUPE
- Call list-agent-automation-runs once with your automationId from self-context, status triggered, limit 30.
- Use a stable fingerprint: appEnvId:signal:normalized-cause.
- Suppress the same fingerprint for 24h unless severity rises, the recent count/absolute utilization at least doubles, or impact expands to another production environment.

4. SEVERITY
- warning: confirmed degradation or repeated failed work that the owning on-call should check within 30 minutes.
- critical: current availability, data integrity, security, or broad customer impact; or sustained CPU/memory p95 at least 90%.
- Do not trigger info findings; keep monitoring them.

5. NOTIFICATION OUTPUT
When triggering, output ONLY this structure before the final JSON. Keep it under 900 characters. Use the environment name; do not lead with internal IDs. Do not mention suppressed candidates, tool calls, thresholds, or your decision process.

*Action required — ENVIRONMENT_NAME: SHORT_PROBLEM*
*Impact:* who/what is failing now.
*Evidence:* last-hour absolute count or p95, the 7d comparison, and first/last seen when available.
*Likely cause:* the strongest evidence-backed cause, explicitly marked uncertain when needed.
*Do now (owner: application on-call or platform on-call; within 15/30 min):*
1. First exact action using the observed endpoint, error, deploy, or resource.
2. Second exact action or escalation path.
*Verify:* a measurable recovery condition for the next 15–30 minutes.
*Escalate if:* a measurable condition that changes urgency.

Then append exactly one single-line JSON object:
{"trigger":true,"severity":"warning|critical","fingerprint":"stable-id","affectedResources":["env:APP_ENV_ID"],"summary":"impact plus evidence in 2 sentences","reason":"the same short actionable headline"}

If nothing meets every actionability rule, reply in at most 3 lines with what is being watched and why it is not actionable yet. Do not include JSON or TRIGGER.

Trigger condition​

Trigger only for a new or escalated production anomaly that crosses an absolute impact floor and produces a concrete owner, immediate action, and measurable verification step. Do not trigger on relative delta alone, low-volume new signatures, expected deploy churn, or non-actionable findings. Deduplicate the same fingerprint for 24 hours unless severity rises or impact at least doubles.

What a useful notification looks like​

Action required — api-prod: payment-provider calls are failing
Impact: Checkout requests that need a new payment session are failing.
Evidence: 47 timeout errors in the last hour versus 2 in the 7-day baseline; first seen 14:08 UTC and still occurring at 14:56 UTC.
Likely cause: The provider endpoint is timing out; no application deploy overlaps the first error.
Do now (owner: application on-call; within 15 min):
1. Check the provider status page and the api-prod error log for the affected endpoint.
2. If the provider is healthy, escalate with the request IDs from the latest failures.
Verify: Fewer than 2 matching errors and successful payment-session creation for 15 minutes.
Escalate if: Errors continue for 15 minutes or checkout 5xx exceeds 2%.

This is actionable because it names the failing workflow, quantifies current impact, assigns an owner, gives evidence-based steps, and defines recovery.

Test before relying on it​

  1. Save the automation and select Run now.
  2. Inspect the run in Quave ONE even if it does not trigger.
  3. Confirm that low absolute CPU or memory usage does not alert only because the relative change is large.
  4. Confirm that a trigger contains Impact, Evidence, Do now, Verify, and Escalate if.
  5. Keep delivery set to On trigger after validation so normal hourly checks do not create notification noise.

Tune the absolute floors to the workload after observing several days of runs. Keep the actionability contract even when you change the numbers.