Manage Grafana Cloud alerting/dashboards with Terraform #2314

Activity

1. mattgodbolt

   added

   enhancement

   on Aug 19, 2026
2. mattgodbolt commented on Aug 19, 2026

   

   mattgodbolt

   on Aug 19, 2026

   MemberAuthor

   More actions

   (Claude on behalf of @mattgodbolt) Candidate first rule for the IaC setup: the "app restarted in place" alert discussed for #2313 / compiler-explorer/compiler-explorer#9035. Not yet live in Grafana (a hand-made draft exists as uid dfvnu8qol6k1sf in folder Alerts / group "Instance Restarts"; it uses a classic condition, so it can't name the host, and should be replaced by this).

   Why this query: changes() counts value changes within one series, so new instances booting (scale-out, replacements) are new series and do not trigger it; only a same-host restart of compiler-explorer.service does. Deliberately no env filter: an unexpected staging/beta restart is as interesting as a prod one. Contact point: Discord Admins.

   Prometheus-format (importable via Alerting -> Import, datasource grafanacloud-prom):

   groups:
     - name: Instance Restarts
       interval: 1m
       rules:
         - alert: Compiler Explorer restart detected
           expr: changes(process_start_time_seconds{job="compiler_explorer"}[15m]) > 0
           for: 1m
           labels:
             severity: warning
           annotations:
             summary: 'CE app restarted on {{ $labels.agent_hostname }} ({{ $labels.env }})'
             description: 'process_start_time_seconds changed in the last 15m, i.e. systemd restarted compiler-explorer.service in place (infra#2313). Check Papertrail host:{{ $labels.agent_hostname }} or journalctl -u compiler-explorer. New instances booting do not trigger this; only same-host restarts do.'

   Grafana-provisioning-format equivalent (same thing as the API / grafana_rule_group would need):

   apiVersion: 1
   groups:
       - orgId: 1
         name: Instance Restarts
         folder: Alerts
         interval: 1m
         rules:
           - uid: dfvnu8qol6k1sf
             title: Compiler Explorer restart detected
             condition: B
             data:
               - refId: A
                 queryType: instant
                 relativeTimeRange: {from: 900, to: 0}
                 datasourceUid: grafanacloud-prom
                 model:
                   datasource: {type: prometheus, uid: grafanacloud-prom}
                   expr: changes(process_start_time_seconds{job="compiler_explorer"}[15m])
                   instant: true
                   range: false
                   interval: 1m
                   intervalMs: 1000
                   maxDataPoints: 43200
                   refId: A
               - refId: B
                 datasourceUid: __expr__
                 model:
                   datasource: {type: __expr__, uid: __expr__}
                   type: threshold
                   expression: A
                   conditions:
                       - evaluator: {type: gt, params: [0]}
                   intervalMs: 1000
                   maxDataPoints: 43200
                   refId: B
             noDataState: OK
             execErrState: Error
             for: 1m
             annotations:
               summary: 'CE app restarted on {{ $labels.agent_hostname }} ({{ $labels.env }})'
               description: 'process_start_time_seconds changed in the last 15m, i.e. systemd restarted compiler-explorer.service (infra#2313). Check Papertrail host:{{ $labels.agent_hostname }} or journalctl -u compiler-explorer.'
             labels:
               severity: warning
             isPaused: false
             notification_settings:
               receiver: Discord Admins

   Terraform shape: one grafana_folder ("Alerts"), one grafana_rule_group ("Instance Restarts", interval 1m) with this rule (query A instant changes(...), expression B threshold > 0, for = "1m", no_data_state = "OK"), notification_settings { contact_point = "Discord Admins" }. Second candidate: root filesystem free space per host from node_filesystem_avail_bytes{mountpoint="/"} (see #2310).
3. mattgodbolt

   added a commit that references this issue on Aug 20, 2026

   Add a Terraform root for Grafana Cloud configuration (#2315)

   ...

   8707dad