Built first‑party error monitoring and tracing
Built first‑party error monitoring and OpenTelemetry tracing rather
than buying them — payload sanitising, spike and regression
detection, symbolication and a synthetic heartbeat — behind 11
operator views.
2025

Situation. The difference between a platform that is up and a
platform that is working is whether anyone would know. An error a
user hits at eleven at night, on a page nobody tests, is invisible
unless something goes and collects it. The usual answer is to buy a
hosted error tracker, which is a good answer — and it also means the
platform's own errors, stack traces and user context leave for a
third party.

Task. Errors and traces had to be collected, grouped and made
actionable, without the platform's internals leaving the platform.

Action. Two pieces got built. OpenTelemetry handles tracing over
OTLP, so a slow request can be followed across the frontend, the API
and the database rather than guessed at. Alongside it sits an
in‑house error pipeline — an errmon service and an ingest service —
that sanitises payloads before storage, groups errors into recurring
problems rather than a flat list, detects spikes and regressions
with a cooldown so one bad deploy does not page anyone forty times,
symbolicates minified frontend stack traces back into readable code,
and runs a synthetic heartbeat to prove that the pipeline itself is
alive. It all lands in the database as error events, error groups,
an inbox, API latency and stack samples, and it surfaces through 11
operator views, including one for service level objectives.

Result. Errors turn into a queue that someone can work through, and
a regression announces itself instead of being discovered by a user.
Building rather than buying cost real time and means this is one
more thing to maintain — a bought tracker would have been running
the same afternoon. What it bought was that nothing sensitive
leaves, and that the alerting rules fit this platform rather than a
generic one.

Categories:
1Backend Engineering	/categories/backend/	platform.engineer.company	70
1DevOps	/categories/devops/	platform.engineer.company	70
1Infrastructure	/categories/infrastructure/	platform.engineer.company	70
1Reliability & Backups	/categories/reliability/	platform.engineer.company	70
1Monitoring & Observability	/categories/observability/	platform.engineer.company	70
1Security	/categories/security/	platform.engineer.company	70

Services:
1Backend & API Development	/services/backend-development/	platform.engineer.company	70
1DevOps & CI/CD Automation	/services/devops-cicd/	platform.engineer.company	70
1Site Reliability & Monitoring	/services/site-reliability/	platform.engineer.company	70
1Custom AI Agent Development	/services/ai-agents/	platform.engineer.company	70

hThis page on the web	URL:https://platform.engineer.company/portfolio/built-first-party-error-monitoring-and-tracing/	platform.engineer.company	70
