Reliability & Backups

• Developed and launched the company's first observability dashboard, providing real‑time system performance insights and data visualization on the large office TV.
• Designed, deployed, and maintained 10 PostgreSQL and MS SQL servers on Ubuntu Linux VPS, ensuring optimal server performance and reliability.
• Automated GIS SaaS application deployment, data processing, and reporting system using GitHub Actions CI/CD, Python, Bash, and SQL.
• Automated delivery of 20 GIS data pipelines and app data ETL processes, streamlining infrastructure automation and reporting.
• Automated 100 critical data backups using Barman, Google Cloud, Bash, and Python, ensuring data integrity across databases.
• Deployed and maintained 20 Docker containerized applications, troubleshooting with Podman and Kubernetes, and managing R‑based apps on Google Cloud and AWS.
• Managed 30 Ubuntu Linux VPS instances, implementing disaster recovery strategies and ensuring optimal network configurations.
• Mitigated operational risks by implementing a monitoring dashboard using Grafana and Prometheus, improving system reliability.
• Managed and troubleshooted 8 WireGuard VPN and IPSEC VPN connections, ensuring secure communication across Google Cloud and Linux systems.
• Administered network infrastructure for over 1,000 servers, ensuring optimal system deployment, security, and troubleshooting.
• Automated SSL/TLS certificate creation for 100 Docker applications, ensuring secure connections across Ubuntu Linux hosts.
• Streamlined CI/CD processes, saving 4,000 hours by introducing automation in software development pipelines.
• Configured and deployed 1,000 Wi‑Fi routers, improving network accessibility and performance for clients.
• Administered 100 Bare Bone servers, physical networks, and IP telephony systems, ensuring robust infrastructure for company growth.
• Administered 40 websites on Ubuntu Linux hosting servers with Apache and Nginx, ensuring high availability and performance.
• Architected, developed, implemented, supported infrastructure, data processing, and the map application for 2 years non‑stop without any weekends, holidays, or vacations, 10–14 hours a day.
• Generated the API contract outward from the database — OpenAPI, a 44,076‑line typed TypeScript client, 61 mock handlers and the limits the UI enforces — with a guard at every hop that fails on drift.
• Built email as a platform capability — three providers with failover, delivery webhooks, send and delivery logging, templating and campaigns — behind a startup check that will not boot without one.
• Provisioned Azure infrastructure as code with Bicep — Container Apps, PostgreSQL Flexible Server, Front Door/WAF and networking — across the development, staging and production environments.
• Built GitHub Actions CI/CD pipelines with a distroless production frontend image and multi‑environment promotion.
• Owned end‑to‑end deployments of the platform to Azure, managing releases across development, staging and production environments.
• Configured database backup retention as infrastructure‑as‑code, then audited the recovery position and documented the restore procedure — naming the remaining gaps rather than leaving them to be found during an incident.
• Provided round‑the‑clock 24/7 infrastructure support for an IPTV/OTT streaming platform, administering ~1,000 servers plus client‑owned systems for global customers in China, the US and Germany.
• Ensured uninterrupted delivery of IPTV streaming signals between suppliers and clients, monitoring and maintaining the streaming network and IP telephony around the clock.
• Planned and implemented new infrastructure functionality for internal and external systems, building solutions durable enough to still run years later with minimal change.
• Built a layered automated test suite — 981 Go tests, 543 frontend and browser specs, 494 SQL behavioural tests — with mutation testing, property‑based tests and an accessibility gate.
• Moved slow work off the request path onto a River job queue — 15 worker modules, 8 scheduled tasks and 20 pg_cron jobs — so a request returns while the work behind it carries on.
• Built first‑party error monitoring and OpenTelemetry tracing rather than buying them — payload sanitising, spike and regression detection, symbolication and a synthetic heartbeat — behind 11 operator views.
• Kept the schema honest across 1,022 migrations with a CI gate that builds the database both ways — a fresh install, and an install plus every migration — and fails when the two disagree.
• Built fail‑closed abuse controls — 22 Redis‑backed rate limiters, Cloudflare Turnstile, request idempotency and an origin lock — so the platform sheds bots and floods instead of trusting its callers.
• Established a continuous security programme — code scanning, DAST, dependency and vulnerability checks, SBOM generation, secret scanning and SHA‑pinned actions — alongside 21 written security audits.
• Built the company's own infrastructure as 19 Ansible playbooks and 34 roles across 12,065 lines of YAML, converging a live host to a declared state with every play idempotent.
• Ran the whole company on one 512 MB single‑core host — a git forge, a web server serving seven domains, Tor, two alternate‑protocol servers, backups and intrusion banning — by treating 464 MB of usable memory as the binding architectural constraint.
• Proved the intrusion‑banning path end to end on every hardening run by banning a reserved test address, reading the resulting kernel rule and unbanning in a guaranteed cleanup block, so a jail that stops working fails a run instead of reporting healthy.
• Built encrypted off‑host backups on restic with retention pruning, an integrity check and a monthly automated restore drill, then audited the recovery position and wrote down the gaps rather than leaving them to be found during an incident.
• Built dead‑man's‑switch monitoring that pings only while memory and disk are healthy, so a degraded host raises an alert by going silent — and caught six variable names saying "free" where the check correctly measured "available", an order of magnitude apart on a 464 MB box.
• Made check mode tell the truth across the whole platform after finding six probes deciding on a value the host never gave, because Ansible's command module reports success under --check while skipping the command entirely.
• Added a preflight play that runs the same code as the converge against operator‑local secrets in about a second, after a half‑applied production run died on its ninth task with the swap settings already written to the live host.
• Split the four per‑host operational secrets after establishing that two hosts sharing one dead‑man's switch alert less than two switches, not more, and that a shared backup passphrase makes two hosts one repository.
• Held the generator to 981 test cases at a 92% branch‑coverage floor with warnings treated as failures, and asserted idempotence by running the whole build pipeline twice from an empty file and requiring the second pass to change nothing.
• Stopped an application filling memory at 41 MB a second — a recorded 111 GB of compressed pages on a 36 GB machine — by bounding every event stream, subscribing by event type and putting a rate budget on logging, taking 610,996 log lines down to 1,411.
• Reached the half of the messaging core the application had never used — backup transfer, disappearing messages, message editing and resending, verified invitations, proxies and encryption policy — driving every test against the real library with no mocks.

This work is part of what we offer as DevOps & CI/CD Automation, Site Reliability & Monitoring, System Administration, Security & Access Management, Infrastructure as Code, Backend & API Development, Networking & VPN Setup and Backup & Disaster Recovery.

Building something that needs this? Get in touch.