Observability

Traced at commit ffe4e48 · apps/api/src/app/observability/db-gauges.service.ts · libs/telemetry/src/lib/instruments.ts · tools/grafana/*.py · tools/observability/pilot-spec-manifest.psv

The three sources

1. The application. The code emits a counter or a timer as each event happens.
2. These arrive continuously and need no polling.
3. The poller. A service asks the database a fixed set of questions every 60 seconds.
4. The poller publishes each answer as a gauge.
5. Use a gauge for a question about the current state, such as how many leads are in each state.
6. A direct query. Some panels ask the database when a person opens the board.
7. Use a direct query for an expensive question that no alert needs.

The rehearsal label

1. Every metric that can distinguish traffic carries a rehearsal label.
2. The boards have a Traffic control with three settings: real only, simulated only, and both.
3. The default is real only.
4. The platform has placed 4 real dials in total.
5. At the default setting almost every board is therefore empty.
6. That is the correct behaviour. It is not a fault.
7. Three panels answer their question without a metric behind them. No alert can read them.

Failure mode 1 — a poller query grows slow

1. The poller has a budget of 5 seconds for each cycle.
2. A query that outgrows the budget fails.
3. One slow query stops a whole family of numbers at once.
4. The panels show no error. They stop moving.
5. A panel that stopped moving and a panel reading zero look the same.
6. Add an expensive question as a direct query, not as a gauge.

Failure mode 2 — a name stops matching reality

1. A metric name, a label or a database key can stop matching what the code writes.
2. The query still runs. It returns nothing, or it returns the wrong value.
3. The board reports that value with full confidence.
4. This platform has produced that failure several times.
5. An index listed states that cannot occur, so it was never used.
6. A gauge read d2_party while the data uses d2Party.
7. A check queried a metric name that the exporter had renamed.
8. A nightly job now compares every metric in the specification against what the platform emits.

The campaign audit uses the same definitions

Traced at commit cf34e2f · libs/temporal/src/activities/campaign-audit{,-rules}.ts · libs/temporal/src/activities/call-review.ts

1. The main worker audits each running campaign of a real organization every 15 minutes.
2. It posts to the private Slack channel #campaign-audits. It changes nothing on the platform.
3. The audit reads the database, the Temporal workflows and a review of each answered call.
4. Four of its checks copy four Grafana rules exactly, with the same severity.
5. calls.disclosure copies va-disclosure-failures. Both are P1, and the audit gives a FAIL.
6. calls.prohibited_claim copies va-prohibited-claim. Both are P1, and the audit gives a FAIL.
7. calls.slow_engagement copies va-safe-harbour. Both are P2, and the audit gives a WARN.
8. That check counts calls with more than 2 s before the first agent words. It fires above 5 calls.
9. calls.recording_missing copies va-recording-missing. Both are P2, and the audit gives a WARN.
10. Each finding names its Grafana rule.
11. The disclosure check counts only calls of 20 s or more, the same as the gauge.
12. The audit counts only the calls that carry its campaign id.
13. The audit does not read Grafana alert history or the metrics store.