Lab & Completion

3 Signals, 1 Incident Exercise

For one controlled incident (e.g., triggering a /panic or /status/500 route), capture all three artifacts:

  1. Metrics Symptom: Latency or error-rate spike in Grafana.
  2. Trace Path: Showing the failing route and span chain in Uptrace.
  3. Log Evidence: Matching backend log with the correct trace_id.

Success Condition: All three artifacts must point to the same causal path.

Core Exercises (Required)

  1. Metrics: Trigger 40× /status/500 through the frontend Chaos page. Find the error-rate spike in Grafana and the BackendHighErrorRate alert in Prometheus (kubectl -n observability port-forward svc/kube-prometheus-stack-prometheus 9090).
  2. Logs: Use kubectl -n develop logs deploy/backend to find the matching structured error entries for the same window.
  3. Traces (Cloud track): In Uptrace find the end-to-end trace and correlate the backend log entry by trace_id; verify the alert is captured by k8s-ai-monitor. On the local cluster: explain what is missing from the operational contract when logs carry no trace_id.
  4. SLO Check: Identify the backend availability SLI/SLO in your Prometheus rules and explain the burn-rate alert.

Challenge Exercise (Optional)

End-to-End Signal Correlation: Trigger a controlled backend error, then trace it end-to-end through all three signals: find the metric spike in Grafana, locate the trace in Uptrace, and correlate the log entry using only the trace_id.

Done When

You have completed this chapter when:

  • You can find an end-to-end trace from frontend to backend.
  • You can match a backend log entry by trace_id.
  • You can explain why the current alert path goes through k8s-ai-monitor.
  • You have successfully run the incident workflow: metrics -> traces -> logs.
  • You understand why metrics alone are not enough for root cause analysis.

Knowledge Check

Before finishing this chapter, complete the Quiz to verify your understanding of the guardrail principles.