Engineer / Platform / Services
Knowing before your customers do
Monitoring and alerts that speak up while a fault is still small, and the figures that show how often it stays that way.
Built the company's first observability dashboard
Developed and launched the company's first observability dashboard, providing real‑time system performance insights and data visualization on the large office TV.
Built the company's first observability dashboard for real-time system insight, helping teams detect and resolve incidents 40% faster.
Ran 10 Postgre and MS SQL servers on Ubuntu
Designed, deployed, and maintained 10 PostgreSQL and MS SQL servers on Ubuntu Linux VPS, ensuring optimal server performance and reliability.
Designed and maintained 10 PostgreSQL and MS SQL servers on Ubuntu Linux VPS at 99.9% uptime, with 30% faster queries via tuning.
Cut operational risk with Grafana and Prome
Mitigated operational risks by implementing a monitoring dashboard using Grafana and Prometheus, improving system reliability.
Built a Grafana + Prometheus monitoring dashboard that catches problems before they escalate, shifting the team from firefighting to prevention.
Administered 40 websites on Apache and Nginx
Administered 40 websites on Ubuntu Linux hosting servers with Apache and Nginx, ensuring high availability and performance.
Administered 40 websites on Ubuntu Linux with Apache and Nginx at high availability and performance — hosting clients never had to think about.
A mapping product, carried end to end
Architected, developed and supported infrastructure, data processing and a map application over 2 years, covering both software delivery and operational maintenance.
Architected, built and supported infrastructure, data processing and a map application over two years, covering delivery and operational maintenance.
Built email with failover and delivery logging
Built email as a platform capability — three providers with failover, delivery webhooks, send and delivery logging, templating and campaigns — behind a startup check that will not boot without one.
Built email as a platform capability: three providers with failover, delivery webhooks, send and delivery logging, templating and campaign broadcasts.
Owned end‑ platform deployments to Azure
Owned end‑to‑end deployments of the platform to Azure, managing releases across development, staging and production environments.
Owned end-to-end Azure deployments, moving releases through dev, staging and production along a defined, repeatable path.
Ran 24/7 support for ~1,000 IPTV/OTT servers
Provided round‑the‑clock 24/7 infrastructure support for an IPTV/OTT streaming platform, administering ~1,000 servers plus client‑owned systems for global customers in China, the US and Germany.
Provided 24/7 infrastructure support for an IPTV/OTT streaming platform — ~1,000 servers plus client systems across China, the US and Germany.
Kept IPTV streaming signals running non‑stop
Ensured uninterrupted delivery of IPTV streaming signals between suppliers and clients, monitoring and maintaining the streaming network and IP telephony around the clock.
Ensured uninterrupted IPTV streaming between suppliers and clients, monitoring and maintaining the streaming network and IP telephony 24/7.
Built infrastructure that still runs years later
Planned and implemented new infrastructure functionality for internal and external systems, building solutions durable enough to still run years later with minimal change.
Planned and built infrastructure for internal and external systems durable enough to still run years later with minimal change.
Moved slow work onto a River job queue
Moved slow work off the request path onto a River job queue — 15 worker modules, 8 scheduled tasks and 20 pg_cron jobs — so a request returns while the work behind it carries on.
Moved slow work off the request path onto a River job queue: 15 worker modules, 8 scheduled tasks and 20 pg_cron maintenance jobs.
Built first‑party error monitoring and tracing
Built first‑party error monitoring and OpenTelemetry tracing rather than buying them — payload sanitising, spike and regression detection, symbolication and a synthetic heartbeat — behind 11 operator views.
Built first-party error monitoring and OpenTelemetry tracing — sanitising, spike detection, symbolication and a heartbeat — behind 11 operator views.
Built fail‑closed rate limiting and anti‑abuse
Built fail‑closed abuse controls — 22 Redis‑backed rate limiters, Cloudflare Turnstile, request idempotency and an origin lock — so the platform sheds bots and floods instead of trusting its callers.
Built fail-closed abuse controls: 22 Redis-backed rate limiters, Cloudflare Turnstile, request idempotency and an origin lock on the front door.
Three locks on the door, none of them locked
Found and closed three SSH brute‑force protections that had never worked: a ban jail watching port 22 while the daemon listened on 1986, a rate limit shadowed by a broader rule above it, and a ban action whose binary never resolved, so no ban had ever applied.
Three protections that had never once fired now do, and the class of defect they belong to — a control whose failure mode is that it keeps reporting healthy —…
Proved the intrusion ban works every run
Proved the intrusion‑banning path end to end on every hardening run by banning a reserved test address, reading the resulting kernel rule and unbanning in a guaranteed cleanup block, so a jail that stops working fails a run instead of reporting healthy.
A banning path that stops working now fails a converge instead of continuing to report healthy, which is the only property that mattered.
Backup automation with an explicit restore method
Implemented restic backup automation with retention, integrity checks and a monthly restore‑test workflow; documented storage activation and live‑restore verification as separate operational steps.
The repository contains the backup and restore-test implementation and documents its activation prerequisites.
Built dead‑ host monitoring
Built dead‑man's‑switch monitoring that pings only while memory and disk are healthy, so a degraded host raises an alert by going silent — and caught six variable names saying "free" where the check correctly measured "available", an order of magnitude apart on a 464 MB box.
The host gained an alarm that fires when the expected signal is absent, and the misleading names were corrected.
Made Ansible check mode tell the truth
Made check mode tell the truth across the whole platform after finding six probes deciding on a value the host never gave, because Ansible's command module reports success under --check while skipping the command entirely.
A dry run now either measures something or says it did not, and neither of those is the third option it used to have.
Built 7 read‑only host reporting roles
Built seven read‑only reporting roles that render a live host to Markdown — facts, access, git, metrics, traffic, security and provider inventory — under a rule that no number is printed the run did not measure.
The estate can be described from the repository on demand, and the reports have found real things: the two dead security controls, an undeclared listening…
Split every operational secret per host
Split the four per‑host operational secrets after establishing that two hosts sharing one dead‑man's switch alert less than two switches, not more, and that a shared backup passphrase can expose both hosts' backups, even in separate repositories, if someone obtains it and can read both repositories' encrypted data and key files.
Secrets are per host by construction, and the two failure modes that would have followed from sharing them are written down where the next person edits the…
A bounded event pipeline for a macOS app
Bounded event buffers and logging after a recorded macOS incident with 41 MB per second growth and 111 GB of compressed pages on a 36 GB machine; a recorded full test run reduced core log lines from 610,996 to 1,411.
The incident record documents a full test-run comparison: core log lines fell from 610,996 to 1,411, and captured log size from 63 MB to 388 KB.
Measured 13 addresses with no tracking
Measured thirteen addresses from the web server's own access log alone — no cookie, no tracker and no third‑party service — masking the client network as each line is written, and folding the result into a permanent archive of 21 daily counts and 20 monthly facets.
Thirteen addresses are measured this way, and the site ships no cookie, no tracker and no third-party request, so there is nothing to consent to and nothing…
Made six collectors state what they skipped
Made six reporting collectors state the part of their input they never read — files that stayed shut, probes not installed, journal lines unrecognised — and fitted thresholds to 116 measured samples, after establishing that a blind collection and a healthy host render the same page.
A page of this platform's numbers now carries its own blind spots beside its findings. None of the self-measurement is a gate, and that is stated rather than…