Set up monitoring alerts

Set up monitoring alerts for the three database replicas so the on-call can act before users notice. Alert if replication lag exceeds 10 seconds for more than two consecutive checks, if disk usage on /var/lib/mysql exceeds 80 percent, or if CPU load on db-replica-1, 2, or 3 goes above 4 for five minutes. Route replication alerts to the database channel and disk/CPU alerts to infrastructure on-call. Include runbook links and a 15-minute paging escalation.

Before I hand this alert set to the SREs, make it easy to approve: put the actionable symptom up front, show the one-sentence impact for customers, surface the exact thresholds and false-positive risks, and highlight what remediation steps are automated versus manual. Flag any metric that needs a synthetic check added and suggest a sensible mute window for backup jobs and maintenance hours.

I added replication lag and disk usage alerts for the database cluster, but on-call gets paged repeatedly during nightly backups and we’re ignoring real incidents. I don’t know whether to raise thresholds, add suppressions tied to the backup job, or improve the metric collection to distinguish backup I/O. What is the likely root cause and the least risky change to reduce unnecessary pages immediately?

Every month we tune alerts after a noisy incident, only for a new noisy signal to appear later. This eats engineering time and undermines trust in paging. Where are we most likely losing credibility and time — alert design, ownership, instrumentation, or on-call training — and what single habit change will reduce noise and improve response consistency? Give one measurable indicator to track progress.