Monitoring, Health Probes & Alerting

RUSEON Core provides production-grade monitoring capabilities via a dedicated metrics endpoint (http://<host>:9090/metrics) and strict Kubernetes-compliant health probes.

---

1. Truthful Health & Readiness Probes

RUSEON Core exposes two distinct probe endpoints on port 9090:

Liveness Probe (GET /livez)

• Verifies that the Go server process is running, responsive, and not deadlocked.
• Returns HTTP 200 OK with {"status": "healthy"}.
• Used by container orchestrators (Kubernetes, Docker) to trigger automatic process restarts if the server hangs.

Readiness Probe (GET /readyz)

• Performs deep operational checks:
  1. BadgerDB Durability: Confirms the key-value store is writable and not in read-only recovery.
  2. Storage Volume: Confirms the recording directory is mounted and has write permissions.
  3. Stream Engine: Confirms the stream router is initialized.
• Returns HTTP 200 OK when ready to serve traffic, or HTTP 503 Service Unavailable if an essential dependency is failing.

---

2. Prometheus Scrape Configuration

Add RUSEON Core to your prometheus.yml configuration:

scrape_configs:
  - job_name: "ruseon"
    scrape_interval: 10s
    scrape_timeout: 5s
    static_configs:
      - targets: ["ruseon.internal:9090"]
    relabel_configs:
      - source_labels: [__address__]
        target_label: instance
        regex: "([^:]+)(?::\\d+)?"
        replacement: "${1}"

---

3. Recommended Prometheus Alerting Rules

Create ruseon_alerts.yml in Prometheus:

groups:
  - name: ruseon_core_alerts
    rules:
      - alert: RuseonCameraStreamOffline
        expr: ruseon_stream_online == 0
        for: 1m
        labels:
          severity: warning
        annotations:
          summary: "Camera stream offline: {{ $labels.camera_id }}"
          description: "Camera {{ $labels.camera_id }} has been disconnected for more than 1 minute."

      - alert: RuseonBufferFrameDrops
        expr: rate(ruseon_buffer_drops_total[5m]) > 0
        for: 2m
        labels:
          severity: critical
        annotations:
          summary: "Frame drops detected on buffer: {{ $labels.camera_id }}"
          description: "Slow consumer detected. Buffer is dropping frames to protect ingest pipeline."

      - alert: RuseonStorageSpaceLow
        expr: ruseon_storage_free_bytes / ruseon_storage_total_bytes < 0.10
        for: 5m
        labels:
          severity: critical
        annotations:
          summary: "Archive storage partition under 10% free space"
          description: "Storage space is critically low. Retention policy should be verified."

Related Documentation

• Operations Overview
• Prometheus Metrics Reference
• Server Configuration
• Troubleshooting Streaming Issues