Metrics & health #

Husaria splits liveness, readiness, and metadata into separate endpoints so you can wire each into the right place in your observability stack.

Health endpoints #

HEAD handlers are registered alongside GET on all probe endpoints — sidecars that do HEAD-only checks (graphql-proxy, some Istio configurations) work out of the box.

Both /healthz and /version payloads carry the read_only flag. A typical LB pattern is to route mutations to whichever instance has read_only:false and reads to whichever is closer / cheaper.

Prometheus metrics #

/metrics is exposed when HUSARIA_METRICS_ENABLE=true (default). The path can be moved with HUSARIA_METRICS_PATH.

Authentication required. /metrics was relocated to the admin group during a security pass. Prometheus needs an X-Husaria-Admin-Secret header in its scrape config — previously the endpoint was open on the engine root and leaked goroutine counts, DB pool stats, and pg_stat_statements labels (admin-issued SQL text) to any pod on the cluster network. Add to your scrape job:

scrape_configs:
  - job_name: husaria
    static_configs:
      - targets: ['husaria.husaria.svc.cluster.local:8080']
    metrics_path: /metrics
    authorization:
      credentials_file: /etc/prometheus/secrets/husaria-admin-secret
    # OR, if your Prometheus build doesn't honour `credentials` with
    # a non-Bearer auth scheme, use static headers:
    # http_config:
    #   headers:
    #     X-Husaria-Admin-Secret: ${HUSARIA_ADMIN_SECRET}

Surfaced series #

• husaria_executor_* — query / mutation / aggregate latency + counts, broken down by operation type.
• husaria_subs_* — subscription cohort + tick metrics:
  • husaria_subs_cohorts_total (counter, kind={live,stream})
  • husaria_subs_cohorts_active (gauge)
  • husaria_subs_subscribers_active (gauge)
  • husaria_subs_ticks_total (counter, outcome={success,error})
  • husaria_subs_emits_total (counter)
  • husaria_subs_tick_duration_seconds (histogram, 5ms–10s)
  • husaria_subs_dropped_subscribers_total (counter, reason={slow,client_error})
• husaria_cache_* — response-cache hit / miss / size + per-op latency. Latency samples are keyed by a monotonic counter rather than wall-clock nanoseconds, so the histogram count is accurate even on hot goroutines that record multiple samples in the same nanosecond.
• Standard process_* and go_* — exposed by the default Prometheus registry.

Label cardinality is bounded by design — no per-query, per-cohort, or per-subscriber labels that would blow up the time-series store. Cohort identity surfaces in structured logs only (first 16 hex chars of the SHA-256 cohort key). Full GraphQL queries and variables are never logged.

Logs #

Log output is JSON in production (ENVIRONMENT=production) and console-formatted in development. Each request emits a single line per surface:

{
  "level": "info",
  "ts": "...",
  "msg": "REQ",
  "trace_id": "4bf92f3577b34da6a3ce929d0e0e4736",
  "status": 200,
  "duration_ms": 1.96
}

Variants: REQ 4xx, REQ 5xx, REQ slow. GraphQL paths only emit the post-handler completion line (with operation complexity / depth fields) — the duplicate pre-handler line was removed. At info level the monitoring middleware skips request/response body capture entirely.

The trace_id field correlates with the traceparent header on the response — see Tracing.

What's not there #

There is no built-in distributed-tracing exporter. Husaria propagates trace IDs but does not emit spans of its own — run an OpenTelemetry collector in front of (or beside) Husaria to capture spans at the ingress layer. See Tracing for the full propagation contract and a wiring example.