Grafana Dashboards¶

Importable Grafana dashboard definitions live in docs/deployment/grafana-dashboards/. This page indexes what each dashboard shows and documents the Vercel Drains → Loki pipeline that feeds the web-performance dashboard.

Dashboard index¶

Dashboard	File	What it shows
Vercel Web Analytics & Speed Insights	vercel-drains-web.json	Core Web Vitals (p75 LCP/INP/CLS/FCP/TTFB with Google thresholds, trends, worst routes, country/device breakdowns) and Web Analytics (pageviews, custom events, top pages/referrers/countries, device/browser split). Fed by the Vercel drain pipeline below.
Production - Build & Deployment / Staging - Build & Deployment	Production · Staging	GitHub Actions health: build/deployment success rate, average workflow duration, deployments today, deployment frequency by service.
Production - Logs / Staging - Logs	Production · Staging	Error log volume over time and per-app error streams (Frontend Client, Frontend Server, aiHub, aiHub SQS Processor, Sana). The `frontend-server`, `sana`, and aiHub warn/error streams are fed by the per-event in-process Loki push below (one mechanism for all packages); the browser's client errors ride `/api/log-error`.
Production - Database / Staging - Database	Production · Staging	Full Supabase host + Postgres observability (~137 panels): node-exporter system metrics (CPU, memory, disk, network), Postgres status/size/connections/replication, pgbouncer pools, query and bgwriter stats.
Production - Testing / Staging - Testing	Production · Staging	CI quality metrics: overall quality score, unit/E2E pass rates and trends, failures by category, coverage by type, E2E success by service and browser.

To import: Grafana → Dashboards → New → Import → upload the JSON. vercel-drains-web.json is a classic-schema export and will prompt you to map its Loki datasource input; the production/staging dashboards are Grafana v2-schema exports tied to the datasources of our Grafana Cloud stack. When you change a dashboard in Grafana, re-export it and update the JSON here so the definitions stay versioned.

Vercel Drains → Loki pipeline¶

Visualizes Vercel Speed Insights (Core Web Vitals) and Vercel Web Analytics (pageviews + custom events) forwarded to Grafana Loki via Vercel Drains. Vercel Drains for these two data types only support custom HTTP endpoints (no native Grafana integration), so a small translator endpoint sits in between:

@vercel/speed-insights ─┐
                        ├─> Vercel Drain (NDJSON POST) ─> /api/drains/grafana ─> Loki push API ─> Grafana
@vercel/analytics ──────┘

Each drain event is stored as one Loki log line with two low-cardinality labels the dashboard filters on:

Label	Values
`service_name`	`vercel-speed-insights` \| `vercel-analytics`
`environment`	`staging` \| `production` \| `preview` \| `unknown`

environment is resolved host-first by the receiver (resolveAnalyticsEnvironment in the route): the event's serving host (origin / url / vercelUrl / host, whichever are present) is checked before its vercelEnvironment. A host containing staging → staging; any other host on the farmcove.co.uk suffix → production (the staging check runs first, since staging.farmcove.co.uk matches both). This is required because staging is a Vercel preview deployment behind the custom domain, so its events arrive with vercelEnvironment: "preview" — labelling by that field alone would mislabel every staging pageview as preview. Only when no host resolves does it fall back to the allowlisted vercelEnvironment (production / preview), then unknown.

Everything else (metricType, value, route, path, country, deviceType, clientName, eventName, referrer, …) is queried at read time with | json. Never promote those to Loki labels — the cardinality would explode the index.

The receiver endpoint¶

Implemented at packages/app/src/app/api/drains/grafana/route.ts. It:

Verifies the x-vercel-signature header as an HMAC-SHA1 hex digest of the raw body, keyed on VERCEL_DRAIN_SECRET (constant-time comparison; 403 on mismatch, 500 if the secret is unset).
Parses either NDJSON (one JSON object per line) or a JSON array — whichever format the drain is configured for. Malformed input is dropped with a warning rather than failing the delivery (a 500 would make Vercel retry a payload that can never parse).
Sanitizes every event with removeSensitiveFields from @delta/common before it is written — the same "nothing reaches Loki unsanitized" rule every other Loki-writing path follows. Redaction is by field name (email, phone, token, …), so the documented Vercel schema fields the dashboard queries all pass through untouched.
Transforms each event into one Loki stream entry and pushes them all in a single Loki request, reusing the app's existing LOKI_URL / LOKI_USER / LOKI_PASSWORD credentials.
Self-authenticates via the drain signature, so it is registered in serviceApiRoutes in packages/app/src/lib/database/middleware.ts — otherwise updateSession would 401 the drain deliveries before the handler runs.

Setup¶

Prerequisites: Vercel Pro or Enterprise (Drains are unavailable on Hobby, billed at $0.50/GB of drained volume), @vercel/speed-insights and/or @vercel/analytics already collecting data, and a Loki instance.

1. Loki credentials¶

This repo already has them. The common logger's per-event Loki push transport pushes to Grafana Loki using LOKI_URL / LOKI_USER / LOKI_PASSWORD, which both deploy-app-staging.yml and deploy-app-production.yml already thread into the Vercel build from GitHub Actions vars/secrets. The drain endpoint reuses those same three variables — no new Loki credentials needed.

For a fresh setup elsewhere: Grafana Cloud portal → your stack → Loki → Details for the push URL (e.g. https://logs-prod-012.grafana.net) and numeric user/tenant ID, then Access Policies → Create access policy (logs:write scope) → create a token to use as the password.

2. Environment variables¶

Variable	Status
`LOKI_URL`	Already set — GitHub Actions var, threaded by both `deploy-app-*.yml` workflows
`LOKI_USER`	Already set — GitHub Actions var, threaded by both `deploy-app-*.yml` workflows
`LOKI_PASSWORD`	Already set — GitHub Actions secret, threaded by both `deploy-app-*.yml` workflows
`VERCEL_DRAIN_SECRET`	Threaded by both `deploy-app-*.yml` workflows — the secret value must exist in GitHub Actions secrets (generate any strong random string; it is also entered on the Vercel drains in step 3)

Deployment env vars in this repo live in GitHub Actions vars/secrets, not in the Vercel project settings.

3. Create the drains in Vercel¶

For each of Speed Insights and Web Analytics (one drain per data type):

Vercel dashboard → Team Settings → Drains → Add Drain.
Data type: Speed Insights (first drain) / Web Analytics (second).
Select projects and a sampling rate (start at 100%; lower it if drain volume costs grow).
Destination → Custom Endpoint: URL https://<your-app>/api/drains/grafana, format NDJSON. Set the signature secret to the VERCEL_DRAIN_SECRET value (same secret for both drains).
Create Drain — Vercel tests the endpoint immediately; the Test button re-sends a sample anytime.

Privacy: Team Settings → Security & Privacy → IP Address Visibility can strip client IPs from drain payloads team-wide.

4. Import the dashboard¶

Grafana → Dashboards → New → Import → upload grafana-dashboards/vercel-drains-web.json → map the Loki datasource input to the Loki instance receiving the drain data.

Dashboard variables¶

Environment (visible): populated from the environment label; multi-select, defaults to All.
si_service / wa_service (hidden constants): the service_name label values, vercel-speed-insights / vercel-analytics. If the endpoint's stream names ever change, edit these two constants in dashboard Settings → Variables instead of touching every query.

Gotchas¶

Verify metric units after the first real events arrive. The dashboard assumes time-based vitals (LCP/FCP/TTFB/INP) arrive in milliseconds (as emitted by the web-vitals library). Vercel's docs example shows LCP: 2.5, which would be seconds — if the stats look 1000× off, change the panel units from ms to s (thresholds too: 2.5/4 for LCP, etc.).
Loki rejects entries with timestamps too far in the past relative to already-ingested data on the same stream; drain deliveries are near-real-time so this rarely matters, but replayed/test events with old timestamps may 400.
If no data appears: check the drain's status on the Vercel Drains page (errored drains are flagged and emailed after sustained failures), and query {service_name=~"vercel-.+"} in Grafana Explore to see raw lines.
The Value #A column overrides in table panels depend on Grafana's default field naming for instant queries; if a table shows an unstyled Value column after a Grafana upgrade, re-point the override at the new field name.
On the Grafana Cloud free tier: log retention is 14 days (dashboards show at most two weeks of history) and drain events share the 50 GB/month logs allowance with the app's own logs — the drain sampling rate on the Vercel side is the volume dial.

App / service logs → Loki (per-event in-process push)¶

Sends warn + error logs from every package (frontend-server, sana, aihub, the lambda functions) to Grafana Loki via a single mechanism: a per-event push transport in the common logger factory. There is no logs drain and no batched winston-loki transport — as each warn/error is logged, the transport pushes that one line straight to Loki:

logger.warn() / logger.error() ─> LokiPushTransport (structured JSON, one line) ─> Loki push API ─> Grafana

Only warn + error ship. The transport runs at level: 'warn', so info/debug never reach Loki. Console still carries every level (structured JSON on the app for Vercel's log UI; human-readable in aiHub/lambdas). The dashboards query level="error"; warn is now available too.
One mechanism, all packages. createLogger(appName, options) in packages/common/src/lib/logger/index.ts registers the LokiPushTransport whenever Loki delivery is enabled. It replaced BOTH the old batched winston-loki transport AND the app-only error-drain path. The app passes { structuredConsole: true, schedulePush }; aiHub/lambdas pass no options (defaults enable the transport). lokiTransport: false disables Loki delivery entirely.
Per-event, no long-lived socket. Each event builds its structured line via the shared buildStructuredLine helper (byte-identical to the structured-console line, same cleanLogData/removeSensitiveFields sanitisation) and pushes ONE stream. One retry on failure.
Scheduling / flush. Every push promise is tracked in an in-flight Set; logger.flushLogs() awaits Promise.allSettled of that set (aiHub ECS shutdown + lambdas call it before freeze so no push is lost). On Vercel the app additionally passes a schedulePush hook backed by @vercel/functions' waitUntil, keeping the Function alive until the push settles; outside a request context (local dev, scripts, tests) waitUntil throws and the push simply runs fire-and-forget (still tracked).
Push primitives (getLokiConfig, pushToLoki, groupIntoLokiStreams, msToLokiTimestampNs, logLokiPushFailure, LokiPushError) live in @delta/common (packages/common/src/lib/logger/loki; wire types in packages/common/src/types/loki.ts) and are fetch-based (no app HTTP client). They are also reused by the app's analytics drain receiver (/api/drains/grafana) and client-error receiver (/api/log-error). Reuses the existing LOKI_URL / LOKI_USER / LOKI_PASSWORD credentials; no VERCEL_DRAIN_SECRET (that secret is only for the analytics drain).

Label mapping contract¶

The label contract the Logs dashboards filter on is unchanged in shape. Each pushed line carries three low-cardinality labels:

Label	Value
`app`	the logger's own app name (`frontend-server`, `sana`, `aihub`, …).
`environment`	`getEnvironment()` from `@delta/common` (`production` / `staging` / `development`), resolved in-process.
`level`	the event's level — `warn` or `error` (info/debug are never pushed).

The full structured line (all metadata, message, stack) is stored in the log line and queried at read time, never promoted to a label. Sanitisation runs inside the logger (cleanLogData / removeSensitiveFields) before the line is built, so nothing reaches Loki unsanitized.

Anti-recursion invariant¶

A failure of the Loki push must never be reported through the winston logger — the push transport is what a logger.warn/logger.error invokes, so logging the push failure would re-enter the transport and push again, forever. The transport reports push failures with a plain console.warn (prefix [loki-push], captured by the platform log UI but never re-fed into Loki) and never through logLokiPushFailure (which takes the logger; that helper is only for the app's route handlers, which log through a different surface). The transport also swallows any build/scheduling throw and always calls next(), so logging can never throw.

One-time cleanup¶

This mechanism replaced the old Vercel Logs drain (which POSTed captured runtime logs to a now-deleted /api/drains/grafana-logs receiver). Once this change is deployed, delete the redundant Vercel Logs drain so nothing keeps hitting the removed route:

Vercel dashboard → Team Settings → Drains.
Find the "Logs - Grafana" drain (id drn_PQxL9tZY2pDf0Qa6).
Delete it — nothing consumes it anymore. Leave the Speed Insights and Web Analytics drains (which still POST to /api/drains/grafana) untouched.

To confirm error logs are flowing, query {app=~"frontend-server|sana", level="error"} in Grafana Explore.