Observability Setup¶

Compliance mapping: ISO 27001 A.8.16 · SOC 2 CC7.2 · NIST DE.CM · FedRAMP AU-6/SI-4

Overview¶

Opencomplai emits OpenTelemetry (OTel) traces and Prometheus metrics from every service. The Docker Compose stack bundles a full observability pipeline:

Logs¶

The Python services write logs to standard output as one JSON object per line. Each line carries ts, level, logger, service, msg and, for logged exceptions, exc_info. trace_id appears only when an OpenTelemetry span is active, for example with external OpenTelemetry instrumentation, so you can match a log line to its trace. OTEL_SERVICE_NAME sets the service field. uvicorn's own startup and access lines keep its plain-text format.

Configuration¶

Quick Start¶

1. Copy the env template and enable OTel:

cp infra/compose/.env.example infra/compose/.env
# OTEL_EXPORTER_OTLP_ENDPOINT and OTEL_SERVICE_NAME are enabled by default

1. Start the stack:

docker compose -f infra/compose/docker-compose.yml up -d

1. Open Grafana at http://localhost:3001. Anonymous Viewer access is on, so no login is needed to see the dashboards. Grafana and Prometheus listen on this machine's loopback only by default; see Network exposure.

Environment Variables¶

Leave OTEL_EXPORTER_OTLP_ENDPOINT unset to disable trace export. Prometheus metrics are always exposed via each service's /metrics endpoint regardless.

Network exposure¶

Prometheus and Grafana are operator tools, not part of the API surface, so their host ports are published on 127.0.0.1 unless you say otherwise. From the Docker host http://localhost:3001 and http://localhost:9090 work as usual; other machines cannot reach them (see the Docker Engine note below). The gateway (GATEWAY_PORT) is unaffected: it is the stack's entry point and stays published on all interfaces behind its own authentication.

To reach them from other hosts, set OBSERVABILITY_BIND_ADDR=0.0.0.0 in infra/compose/.env (every IPv4 interface; :: is the IPv6 equivalent) or the address of a single interface, and recreate the containers. The setting covers both ports.

Instrumented Events¶

All services emit the following canonical events (PRD Section 11.1):

Grafana Dashboard Panels¶

The provisioned Opencomplai — Compliance Health dashboard includes:

• Time to first scan (P95 ms) — latency gauge
• Control pass rate — percentage of scans that pass all controls
• Trap detection frequency — rate of trap_detected events by system
• Override rate — rate of override_submitted events
• Egress blocked events — total EGRESS_BLOCKED count (red alert threshold: ≥10)
• BREAK_GLASS_ACTIVATED count — total override activations (red alert threshold: ≥1)
• Audit Events Rate — rate of audit events entering the ledger
• Auth Failure Rate — rate of verification/auth failures (brute-force indicator)

Health Endpoints¶

Three distinct endpoints answer three different questions. Pointing a monitor at the wrong one is the most common way to get misleading signals here.

GET /v1/status¶

Probes risk-engine, evidence-vault, doc-generator, and egress-proxy concurrently and reports each one. Results are cached briefly (STATUS_CACHE_TTL_MS, default 5000) so polling does not fan out to the backend on every request.

{
  "status": "degraded",
  "service": "gateway-api",
  "version": "0.13.1",
  "checked_at": "2026-07-30T20:35:00Z",
  "services": {
    "risk-engine":    { "status": "ok", "latency_ms": 12 },
    "evidence-vault": { "status": "ok", "latency_ms": 9 },
    "doc-generator":  { "status": "unreachable", "latency_ms": 2000, "reason": "timeout" },
    "egress-proxy":   { "status": "ok", "latency_ms": 7 }
  }
}

Per-service status is one of ok, degraded (answered, but reported itself unhealthy), or unreachable (no answer). reason is one of timeout, connection_error, or http_error. A service entry carries a version only when that service's health response includes one; the four internal services' health responses currently do not, so none appears above.

Monitor this URL, with the strict flag:

GET /v1/status?strict=1

By default /v1/status returns 200 even when degraded — the request succeeded and the body is an accurate answer, and many clients discard the body of a non-2xx response, which would throw away the per-service detail exactly when it matters. ?strict=1 returns 503 whenever anything is not ok, so a monitor that only inspects the status code still sees the problem. The body is identical either way.

The failure mode of the default is silent green: a monitor configured without ?strict=1 reports healthy while the system is degraded. Use the strict URL for automated monitoring and the plain URL for dashboards that render the detail.

Alert Routing¶

For production deployments, configure alert routing in Grafana or your SIEM:

Air-Gapped Deployments¶

In air-gapped environments where OTel export is not possible:

1. Leave OTEL_EXPORTER_OTLP_ENDPOINT unset.
2. Prometheus still scrapes each service's /metrics endpoint directly.
3. Traces are emitted locally but not forwarded to the collector.

See Air-Gap Deployment for full configuration.