Engineer — Platform — Portfolio and notes

Developed and launched the company's first observability dashboard, providing real‑time system performance insights and data visualization on the large office TV.

2026-09-13T01:45:50+02:00

Situation. The company was facing challenges in monitoring real‑time system performance, which often led to delayed incident responses and reduced visibility into infrastructure health. There was no centralized solution in place for teams to gain insights into operational metrics.

Task. The task was to develop a solution that would enable technical and non‑technical stakeholders to monitor key system metrics in real time, with a focus on accessibility, clarity, and proactive issue detection.

Action. The company’s first observability dashboard was designed and implemented, collecting all essential system metrics — such as CPU, RAM, HDD, temperature, and more — from remote Linux servers via SSH. Even Docker containers were monitored using this method. Later, a second version was designed and implemented using Grafana and Prometheus for more advanced visualization and monitoring capabilities. Collaboration with DevOps and engineering teams identified the critical metrics, such as CPU utilization, memory usage, service uptime, and API latency. Data pipelines were configured to ingest and process performance metrics from various systems, and the dashboard deployed on a large office TV screen for maximum visibility. Alerting mechanisms for threshold breaches were also integrated to enable immediate action.

Result. The dashboard significantly improved system transparency and response time to operational issues. Teams were able to detect and resolve incidents 40% faster. It also fostered a culture of shared ownership over system health by making performance data accessible to everyone in the office, ultimately contributing to a more stable and efficient production environment.

Designed, deployed, and maintained 10 PostgreSQL and MS SQL servers on Ubuntu Linux VPS, ensuring optimal server performance and reliability.

2026-09-13T01:45:50+02:00

Situation. As a DevOps engineer at a mid‑sized tech company, the task was managing and optimizing database infrastructure to support a growing user base and critical business applications. The organization relied heavily on PostgreSQL and Microsoft SQL Server for data storage and analytics, requiring high availability, scalability, and security.

Task. The primary responsibility was to architect, deploy, configure, and maintain 10 PostgreSQL and MS SQL Server instances on Ubuntu Linux VPS environments. This included ensuring optimal performance, implementing robust security protocols, and establishing proactive monitoring to prevent outages. Additionally, the task included scaling the infrastructure to accommodate future growth while minimizing costs and maintaining compliance with industry standards.

Action. 1. Deployment & Configuration:

Installed and configured PostgreSQL 14 and MS SQL Server 2019 on Ubuntu 20.04 LTS VPS instances, ensuring compatibility with the company’s applications.
Set up automated backups using pg_dump for PostgreSQL and SQL Server Agent jobs for MS SQL, with retention policies and offsite storage.
Optimized server configurations (e.g., memory allocation, query caching, and connection pooling) to improve query performance and reduce latency.

Monitoring & Maintenance:
- Implemented monitoring tools like Prometheus, Grafana, to track CPU, memory, disk I/O, and query performance metrics in real time.
- Conducted regular patching and updates for both databases and the Ubuntu OS to address security vulnerabilities and ensure compliance.
- Created custom scripts for log analysis.
Security & Scalability:
- Configured firewalls (UFW), enforced role‑based access control (RBAC) to protect sensitive data.
- Documented procedures for disaster recovery, including point‑in‑time restores and failover protocols.

Result. - Achieved 99.9% uptime across all 10 database servers.

Improved query response times by 30% through configuration tuning and index optimization, enhancing application performance.
Reduced manual maintenance tasks by 50% via automation, freeing up 10+ hours per month for strategic projects.
Successfully scaled the infrastructure to support a 40% increase in user traffic without service degradation, contributing to a 20% revenue growth in the following quarter.
Received recognition from the CTO for implementing security best practices that prevented potential data breaches.

This experience solidified deep expertise in database management, DevOps automation, and infrastructure optimization, delivering reliable, secure, and scalable solutions for complex enterprise environments.

Enhanced data security by implementing 1,000 RBAC rules for developers, application instances, PostgreSQL, MS SQL, and other Linux servers, preventing unauthorized access; documented with Ansible automation.

2026-09-13T01:45:50+02:00

Situation. The organization needed to strengthen access controls across multiple systems, including developer environments, application instances, PostgreSQL, MS SQL, and Linux servers. While the underlying RBAC model was straightforward, the challenge lay in managing over 1,000 individual rules to cover diverse user roles and system requirements. The goal was to ensure strict access restrictions without introducing complexity.

Task. Implement a scalable RBAC solution by defining and enforcing 1,000+ rules for access control. This involved mapping permissions to specific roles (e.g., developers, application instances, database admins) and ensuring rules were applied consistently across all systems. The task also required documenting the rules and automating their deployment to avoid manual errors.

Action. The approach focused on creating a simple, modular RBAC structure, breaking down permissions into clear, reusable categories (e.g., “read‑only access to production databases”). Using Ansible, the configuration of each rule was automated, ensuring consistency across environments. For example, developers were granted access only to their designated servers, while application instances had limited permissions to prevent lateral movement. The process prioritized clarity over complexity, with each rule explicitly tied to a specific role and system.

Result. The implementation secured over 1,000 rules without introducing unnecessary complexity, reducing unauthorized access risks by 85%. Automation streamlined deployment, cutting setup time by 60% compared to manual methods. The documented framework allowed teams to quickly audit or modify rules, ensuring scalability as the infrastructure grew. By focusing on simplicity and volume, the solution achieved robust security while maintaining operational efficiency.

Automated GIS SaaS application deployment, data processing, and reporting system using GitHub Actions CI/CD, Python, Bash, and SQL.

2026-09-13T01:45:50+02:00

Situation. At Utiligize, getting the GIS SaaS app deployed, processing its data and producing the reports were all manual steps — and manual steps are both slow and quietly dangerous. Every release ate engineering time and carried the chance of a slip, and the recurring data and reporting work sat there eating capacity week after week.

Task. Automating the whole path from code to production, plus the recurring data‑processing and reporting, was the task — the goal being releases that were fast, safe and repeatable rather than a careful manual ritual each time.

Action. The whole path got automated. GitHub Actions pipelines took over the test‑build‑deploy cycle, so a release stopped depending on someone remembering the steps. The recurring data processing and the reports moved into scheduled Python, Bash and SQL jobs, so they just ran instead of being someone’s chore. And the configuration and secrets were standardised so every environment behaved identically — which is what kills the “works on my machine” surprises, because there stops being a “my machine” that’s different from production.

Result. Deployment, data processing and reporting all became automated and reliable, the manual toil came off the team’s plate, and the release cycle got shorter. The team could put its attention on the product instead of the operations wrapped around it.

Automated delivery of 20 GIS data pipelines and app data ETL processes, streamlining infrastructure automation and reporting.

2026-09-13T01:45:50+02:00

Situation. The platform ran on a lot of GIS data pipelines and application‑data ETL processes, and they were being delivered and watched by hand. Hand‑run pipelines create bottlenecks, they drift out of consistency, and worst of all they carry a constant low risk that one quietly fails and nobody notices until the data’s already wrong downstream.

Task. Automating the delivery of those pipelines and ETL processes — so the data flowed reliably and predictably without someone shepherding it — was the task.

Action. Twenty GIS data pipelines and the application‑data ETL came under automated delivery, end to end. The scheduling, the logging and the failure handling got standardised, so every pipeline behaved the same way and, crucially, you could see when one didn’t — a silent failure is only silent if nothing’s watching. And they were folded into the existing infrastructure automation and reporting, so they were part of one coherent system rather than a drawer full of scripts someone had to remember to run.

Result. All twenty pipelines and their ETL ran automatically and predictably, and the whole infrastructure‑automation and reporting picture got tidier for it. The business got dependable, current data without anyone having to walk it through by hand — and without the quiet‑failure risk hanging over it.

Automated 100 critical data backups using Barman, Google Cloud, Bash, and Python, ensuring data integrity across databases.

2026-09-13T01:45:50+02:00

Situation. The critical GIS data, the application instances and the databases were scattered across systems with backups that were inconsistent and partly manual. For a product that lives on its data, that’s not a risk you can leave sitting — and the day you actually need a backup is precisely the worst day to find out it was incomplete.

Task. The task was to guarantee that all the critical data could be recovered, which meant automating comprehensive, verified backups across the whole estate — verified being the word that matters.

Action. The backup regime was built end to end. A hundred critical data backups got automated, with Barman handling the PostgreSQL side and Google Cloud holding the offsite copies, and the whole thing was orchestrated and validated with Bash and Python — because a backup you’ve taken but never checked isn’t really a backup, it’s a hope. So there were retention policies to keep them current and integrity checks to confirm each one was actually good, not just present.

Result. Backups ran automatically and were verifiable across every database, which turned data recoverability from an assumption into something tested. A major operational risk came off the business and got replaced with a recovery path you could actually trust — the difference being that this one had been checked, not just configured.

Deployed and maintained 20 Docker containerized applications, troubleshooting with Podman and Kubernetes, and managing R‑based apps on Google Cloud and AWS.

2026-09-13T01:45:50+02:00

Situation. There was a growing set of containerised applications — including some R‑based analytics apps — running across both Google Cloud and AWS. Spread over two clouds and a few container runtimes, they needed to deploy consistently and to be diagnosable quickly when something went wrong, which is harder than it sounds when no two environments are quite the same.

Task. Deploying and maintaining those workloads reliably, and being able to diagnose problems fast across the runtimes and clouds, was the job.

Action. The containerised estate was managed across both clouds — twenty Docker applications deployed and maintained with consistent configuration and monitoring, so they weren’t each their own snowflake. When things went wrong, troubleshooting went through Podman and the orchestration debugging through Kubernetes. The R‑based analytics apps got dedicated attention in the Google Cloud and AWS production environments, kept stable and reproducible, which for analytics matters — a result you can’t reproduce isn’t much of a result.

Result. The containerised estate ran reliably across both clouds, problems got diagnosed faster, and deployments stayed stable and reproducible. The applications the product leaned on stayed dependable no matter which cloud they happened to be running in.

Managed 30 Ubuntu Linux VPS instances, implementing disaster recovery strategies and ensuring optimal network configurations.

2026-09-13T01:45:50+02:00

Situation. Utiligize ran on a fleet of Ubuntu Linux VPS instances whose setup had grown organically over time — which is a polite way of saying it had accumulated rather than been designed. That left gaps: inconsistent configurations, and recovery and networking arrangements that were more historical accident than plan.

Task. The task was to get the fleet under proper management, harden the disaster‑recovery side, and make the network configuration consistent and sensible across every instance.

Action. The thirty instances came under deliberate management — treated as one coherent fleet rather than thirty individual pets. Real disaster‑recovery went in: backups, and restore procedures that were actually tested, because an untested restore is just a theory. And the network configuration was standardised for security and performance, so every instance followed the same hardened baseline instead of whatever it had happened to end up with.

Result. The infrastructure became resilient and consistent, with recovery paths that had been tested and networking you could rely on. The downtime risk dropped, and the business ended up with a dependable foundation to grow on rather than a patchwork it had to keep nursing along.

Prevented security breaches by leading access management initiatives, utilizing M365, 1Password, Red Hat SSO, and OKTA SSO.

2026-09-13T01:45:50+02:00

Situation. Access to systems and services across Utiligize was managed inconsistently — permissions granted ad hoc, over time, by different people. That’s a double problem: it opens the door to access nobody intended, and it makes auditing nearly impossible, because nobody can actually say who can reach what, or why they can.

Task. The goal was to close off the security exposure by centralising and tightening access management across the whole organisation.

Action. The overhaul consolidated the identities and access across M365, 1Password, Red Hat SSO and OKTA SSO, so there was a coherent picture instead of scattered per‑system permissions. Least privilege was enforced — people and systems having exactly what they needed and nothing spare — and the onboarding and offboarding standardised, so access got granted and, just as importantly, revoked promptly and consistently rather than lingering after someone had moved on.

Result. The unauthorised‑access risk dropped sharply and access became auditable and consistent — you could finally answer “who can reach this, and why.” The odd part is it also made daily life simpler for the team: the right doors opened easily and the wrong ones stayed shut, which is what good access management actually feels like.

Mitigated operational risks by implementing a monitoring dashboard using Grafana and Prometheus, improving system reliability.

2026-09-13T01:45:50+02:00

Situation. Problems at Utiligize usually got noticed after they’d already hit users, because there was no single view of how the systems were doing. Without that visibility the team was permanently on the back foot — reacting to things that had already gone wrong instead of seeing them coming.

Task. The aim was to cut the operational risk by giving the team real‑time visibility into the systems they depended on.

Action. The observability layer was built out. A monitoring dashboard on Grafana and Prometheus, the key services instrumented, and — the part that actually matters — metrics that meant something rather than vanity numbers that look busy and tell you nothing. Then alert thresholds set on those, surfaced where the team would actually see them and could act while there was still time to act.

Result. Problems started getting caught and dealt with before they escalated, and system reliability improved for it. The team shifted from reactive firefighting to something calmer and more proactive — catching issues while they were still small enough to be boring.

Managed and troubleshooted 8 WireGuard VPN and IPSEC VPN connections, ensuring secure communication across Google Cloud and Linux systems.

2026-09-13T01:45:50+02:00

Situation. Secure connectivity between the cloud and the on‑premises Linux systems ran over several VPN tunnels, and they were fragile and awkward to diagnose when they dropped. A dead tunnel could cut communication between environments, and troubleshooting one was slow and uncertain — you were never quite sure you’d found the real cause.

Task. Managing and troubleshooting those connections, to guarantee secure and uninterrupted communication, was the job.

Action. The VPN estate came under control — eight WireGuard and IPSec tunnels across Google Cloud and the Linux systems, managed and troubleshot as a set rather than eight separate mysteries. Their configuration was standardised so they were consistent and understandable instead of each being its own special case, and their health was monitored, with the recurring routing and key‑exchange problems tackled at the root rather than papered over with a restart.

Result. All eight tunnels ran securely and reliably, communication between environments stayed protected, and the recurring connectivity incidents that used to interrupt work stopped happening. Fixing the root causes rather than nursing the symptoms is what turned them from a recurring headache into something that just worked.

Improved team communication and collaboration by implementing Slack, Mattermost, 1Password, and Jira, saving 8,000 hours of labor.

2026-09-13T01:45:50+02:00

Situation. As the team got bigger, the communication and the tooling hadn’t kept up, and it showed. Things got said in one place and missed by the people who needed them, work got duplicated because nobody could see what someone else had already done, and coordinating anything took longer than the work itself. That kind of friction is invisible day to day, but it adds up to a lot of lost time.

Task. The aim was to fix how the team communicated and worked together, and claw back the time being quietly bled to all that friction.

Action. The tools got brought in and standardised, and — this is the part that actually matters — the practices for using them were set so they didn’t just become another place to check. Slack and Mattermost for communication, 1Password so shared secrets weren’t being passed around in ways nobody could track, Jira so work was tracked in one place instead of living in people’s heads and inboxes. The tools were the easy bit; getting everyone to actually use them the same way was the real work.

Result. Communication and collaboration got noticeably better, and the streamlined setup saved something on the order of 8,000 hours of labour — time that had been going into chasing information and redoing work, now going into actual delivery.

Overhauled internal processes, saving 8,000 hours by improving software architecture, systems, and scheduling efficiency.

2026-09-13T01:45:50+02:00

Situation. The way things were done internally had accumulated the usual cruft — software architecture that had grown by accretion rather than design, systems that worked but not efficiently, scheduling that left people either waiting or slammed. None of it was on fire, which is exactly why it had been left alone, but it was quietly costing a lot of time.

Task. The aim was to overhaul those processes — to go find the waste and take it out rather than keep paying for it.

Action. The internal processes got reworked across three fronts: the software architecture, so it was something you could reason about and build on instead of work around; the systems, streamlined so the routine work stopped taking longer than it should; and the scheduling, so capacity was actually matched to the work. And the changes were made to stick — embedded in how the team operated rather than left as a memo everyone nodded at and forgot — because process improvements that aren’t made permanent just decay back to the old way.

Result. The overhaul saved roughly 8,000 hours by making the architecture, systems and scheduling meaningfully more efficient. That’s capacity that went straight back into higher‑value work instead of into overhead nobody had thought to question.

Administered network infrastructure for over 1,000 servers, ensuring optimal system deployment, security, and troubleshooting.

2026-09-13T01:45:50+02:00

Situation. The company’s operations sat on top of a big server estate — over a thousand of them — and an estate that size doesn’t stay reliable on its own. Deployment, security and the steady stream of things that go wrong all need someone running them with actual discipline, or the whole thing gets flaky and nobody’s quite sure why.

Task. Administering that network infrastructure was the job — keeping it secure, keeping it reliable, keeping it consistent at a scale where inconsistency is what kills you.

Action. The network infrastructure ran across more than a thousand servers. How systems got deployed was standardised, so a server came up the same predictable way instead of each one being a little bit bespoke; the security was hardened rather than trusting that nobody would come looking; and the troubleshooting got handled when something did break. At that scale the standardisation is what saves you — a thousand snowflakes is unmanageable, a thousand of the same thing is just work.

Result. The estate ran with reliable deployment, solid security and problems that got dealt with promptly instead of festering. That’s the kind of infrastructure work that’s invisible when it’s going well, which is the point — it was the stable backbone everything else at the company ran on.

Automated SSL/TLS certificate creation for 100 Docker applications, ensuring secure connections across Ubuntu Linux hosts.

2026-09-13T01:45:50+02:00

Situation. A hundred Docker applications all needed SSL/TLS certificates, and certificates are the kind of thing that’s fine right up until they aren’t. Issuing and renewing a hundred of them by hand is slow, it’s dull, and it’s exactly the sort of manual job where one forgotten renewal takes an app down with an expiry error at the worst possible moment.

Task. The goal was certificate creation and renewal automated — every app with valid, trusted encryption, and nobody having to remember to do anything.

Action. An ACME‑based workflow handled the whole lifecycle for the hundred Docker applications — creating the certificates and renewing them before they lapsed — and deployed them automatically across the Ubuntu Linux hosts, which were running a mix of Apache and Nginx. The whole aim was to take the human out of it, because the human is the part that forgets.

Result. All hundred applications kept valid certificates and secure connections on their own. The manual certificate work just disappeared, and with it the whole category of outage where something breaks not because it failed but because a cert quietly expired and nobody noticed.

Streamlined CI/CD processes, saving 4,000 hours by introducing automation in software development pipelines.

2026-09-13T01:45:50+02:00

Situation. Getting software out the door relied on manual, inconsistent steps — someone remembering the sequence, doing it slightly differently each time — and it slowed releases down and ate engineering hours that should have gone into building things.

Task. The aim was to streamline the CI/CD process and get automation into the pipelines, so releases stopped being a manual ritual.

Action. Automated build, test and deployment pipelines went in, so the path from a change to it running in production was standardised instead of improvised. The repetitive manual steps — the ones that were slow and, worse, done differently depending on who was doing them — came out. Once the pipeline is doing it the same way every time, a whole class of “it worked on my machine” and half‑remembered deploy steps just stops happening.

Result. The automation gave back roughly 4,000 hours and made releases both faster and more reliable. The team could ship without bracing for it — the confidence came from the process being consistent, not from everyone being careful.

Streamlined data analysis and software development processes, saving 4,000 hours by introducing GitHub, GitLab, Bash, and Python CI/CD practices.

2026-09-13T01:45:50+02:00

Situation. Both the data‑analysis work and the software development were being held back by the same thing: manual processes. Work moved from development to delivery in a slow, inconsistent way, and the analysis side had its own pile of repetitive steps someone was doing by hand every time.

Task. The aim was to streamline both by bringing modern automation and CI/CD practices to workflows that hadn’t had them.

Action. CI/CD practices went in, built on GitHub and GitLab, with Bash and Python doing the automation work underneath. The repetitive steps across both the data‑analysis and the development workflows got automated, and how work moved from development through to delivery got standardised so it was the same every time rather than reinvented per project. Bringing the analysis side into the same disciplined pipeline as the dev work was a big part of it — it had been treated as a separate, more manual world.

Result. The streamlined processes saved roughly 4,000 hours and sped up both the data analysis and the software development, and just as usefully made what got shipped more consistent — fewer surprises from work that had been done a slightly different way each time.

Automated data processing tasks using Shell scripting, PL/pgSQL, Python, and Transact‑SQL, increasing productivity and efficiency.

2026-09-13T01:45:50+02:00

Situation. There was a steady load of recurring data‑processing work being done by hand. Manual data work has two problems at once: it eats time, and it’s inconsistent — do the same task by hand enough times and it’ll get done slightly differently, and some of those differences are errors.

Task. The aim was to automate these tasks, both to get the time back and to make them reliable.

Action. The data‑processing work got automated across the databases and systems it touched, using whatever fit the job — Shell scripting for the glue, PL/pgSQL and Transact‑SQL down in the databases, Python where it needed more than SQL could give. Manual steps got replaced with jobs that ran the same way every time, which is the whole point: a script doesn’t get bored, doesn’t skip a step, and doesn’t do it differently on a Friday afternoon.

Result. Productivity and efficiency both went up, the manual effort came off people’s plates, and the data processing became consistent and dependable instead of a source of small, recurring errors.

Configured and deployed 1,000 Wi‑Fi routers, improving network accessibility and performance for clients.

2026-09-13T01:45:50+02:00

Situation. Clients needed wireless that just worked, and that came down to configuring and rolling out a large number of Wi‑Fi routers — and doing it the same careful way every time, because a router set up sloppily is either insecure or slow, and usually you find out which one later.

Task. Configuring and deploying those routers to give clients better network access and performance was the job.

Action. A thousand Wi‑Fi routers got configured and deployed. The trick at that number is standardising the setup — a consistent, secure, sensible configuration — rather than tuning each one from scratch on the day, because a thousand hand‑crafted routers is a thousand different things to support later. So they were set up for security and performance the same way each time, and rolled out reliably across client sites.

Result. The thousand routers gave clients dependable wireless — better access, better performance — and did it consistently, because the setup was standard rather than improvised. A router nobody has to think about again is the goal; most of these got there.

Administered 100 Bare Bone servers, physical networks, and IP telephony systems, ensuring robust infrastructure for company growth.

2026-09-13T01:45:50+02:00

Situation. Underneath everything the company did was the physical stuff — servers, the actual networks, the IP‑telephony — and it all had to just run. That layer is invisible when it works and extremely visible the moment it doesn’t, and the company’s growth was resting on it staying dependable.

Task. Administering that infrastructure and keeping it stable as the company grew was the job.

Action. A hundred bare‑bone servers, along with the physical networks and the IP‑telephony systems, were looked after — the setup, the maintenance, the troubleshooting when something went wrong. Bare‑bone servers mean dealing with the hardware directly, so there’s a hands‑on, physical side to it: the cabling, the boxes, the phone system that everyone notices the second a call drops. The job was to keep all of it boring, in the good sense.

Result. The servers, the networks and the telephony ran reliably, and that dependable physical foundation is what let the company keep growing without the ground shifting underneath it.

Administered 40 websites on Ubuntu Linux hosting servers with Apache and Nginx, ensuring high availability and performance.

2026-09-13T01:45:50+02:00

Situation. There was a portfolio of live websites that needed to stay up and stay fast — and hosting is another of those jobs that’s invisible until a site goes down, at which point it’s the only thing anyone cares about.

Task. Administering those sites and keeping them highly available and quick was the job.

Action. Forty websites ran on Ubuntu Linux hosting servers, on a mix of Apache and Nginx — the configuration, the performance tuning, the ongoing maintenance to keep them reliable under real traffic. Real traffic is the operative bit: a site that’s fine when nobody’s using it and falls over when they are hasn’t been administered, it’s just been left alone. So the work was keeping them healthy under actual load.

Result. All forty ran with high availability and good performance, which gave clients hosting they didn’t have to think about. Stable and dependable under real use is the entire point of hosting, and that’s what these delivered.

Architected, developed, implemented, supported infrastructure, data processing, and the map application for 2 years non‑stop without any weekends, holidays, or vacations, 10–14 hours a day.

2026-09-13T01:45:50+02:00

Situation. An early‑stage green‑energy startup depended on a single platform to track, monitor, and optimize renewable energy assets, yet had neither a dedicated infrastructure team nor an established engineering organization to build and operate it. The entire technical foundation — cloud infrastructure, data‑processing pipelines, and the customer‑facing GIS map application — had to be created and kept running continuously, in a market where any downtime or data gap directly eroded customer trust and revenue.

Task. The task was to single‑handedly architect, build, and operate the whole system end to end, spanning platform and data engineering, DevOps, and site reliability. Beyond writing the software, this meant owning production: provisioning and hardening infrastructure, designing the data‑processing layer that fed the map, and guaranteeing the application stayed available around the clock for a growing customer base — all within the constraints and relentless pace of a fast‑moving startup.

Action. For two years the infrastructure, data pipelines, and map application were designed, implemented, and supported without interruption — no weekends, holidays, or vacations, often ten to fourteen hours a day. A pragmatic, modular architecture was chosen to keep a one‑person operation maintainable, with automated provisioning, monitoring, and alerting so issues could be detected and resolved quickly. Data processing was continuously tuned for reliability and performance, releases were shipped incrementally, and every layer — from servers to the user‑facing map — was personally maintained and improved in response to real customer usage.

Result. The platform stayed continuously available and evolved from a fragile early prototype into the dependable backbone of the product, sustaining the company through its critical growth phase on the strength of a single engineer’s ownership. This hands‑on stewardship kept infrastructure, data, and the map application reliable enough to support upsells, data licensing, and new‑customer acquisition, and demonstrated a rare degree of commitment, breadth, and end‑to‑end accountability across the full stack.

Optimized budget costs 10 times with zero loss in productivity for the Saudi Arabia company by reimagining the overall infrastructure, eliminating unnecessary services, and relocating from the AWS cloud.

2026-09-13T01:45:50+02:00

Situation. A company operating out of Saudi Arabia was carrying badly bloated infrastructure costs. Their AWS setup had been over‑provisioned and had collected services they no longer used, so the cloud bill had drifted completely out of proportion to what the business actually needed. It’s a common story — nobody sets out to overspend, it just accretes when no one’s watching the meter.

Task. The engagement was to cut the costs substantially without losing any productivity, which meant rethinking the infrastructure properly rather than trimming round the edges — edge‑trimming rarely moves a bill that’s structurally too big.

Action. So the work went end to end. First came an audit of what was actually being used — which is where the duplicated and unnecessary services show themselves — and those got cut. Then whatever was left was right‑sized to match real demand instead of the worst‑case guesses the original setup had been built on. And the big move was relocating the workloads off AWS entirely, onto a more cost‑effective hosting arrangement — done carefully, in stages, so the running business never felt the migration happening underneath it.

Result. The budget costs fell roughly ten‑fold, with zero loss in productivity — the same capability at a fraction of what they’d been paying. It freed up a real amount of money that had quietly been leaking into an oversized cloud bill month after month, which for the business was money straight back to the bottom line.

Designed an organization context‑switching system with client localStorage and server‑side cookie mirroring, letting users act as managed organizations while enforcing least‑privilege authorization.

2026-09-13T01:45:50+02:00

Situation. On NextMariner a maritime professional can manage organizations — companies, academies — that don’t have their own logins. The person is the account; the organization is something they act on behalf of. So a user needs to move through the whole app as any organization they manage, switching between them freely, and that convenience can’t turn into a hole in the authorization.

Task. Context‑switching had to be quick and unobtrusive for the user while making sure the active context could never, on its own, hand someone access they weren’t entitled to.

Action. An OrganizationContext handles it, with storage on both sides. On the client, localStorage is the source of truth for which organization you’re currently acting as, so switching is instant — no round‑trip. A server‑side cookie mirrors it so that server‑rendered pages resolve the same context during SSR; there’s a getServerViewMode on the server that reads it. The important part is that none of that is trusted for access decisions. Authorization gets re‑checked on the server on every request. The frontend context is there for the experience — showing you the right thing — and the server is the only authority on what you’re allowed to do.

Result. A user can act as any organization they manage without friction, and the interface stays in sync on both client and server. But because permissions are verified server‑side every time, none of that convenience weakens the security. Someone tampering with what’s in localStorage changes what their own UI shows them and nothing more — the server still says no.

Generated the API contract outward from the database — OpenAPI, a 44,076‑line typed TypeScript client, 61 mock handlers and the limits the UI enforces — with a guard at every hop that fails on drift.

2026-09-13T01:45:50+02:00

Situation. Frontend and backend move at their own pace, and their assumptions about a request payload can drift apart without anyone noticing. The way you usually find out is a 422 in the browser — after the mismatch has already shipped, which is the most expensive moment to learn about it. Writing the two sides by hand from the same document does not fix it; it just moves the drift to whoever forgot to reread the document.

Task. The two sides had to be generated from one source rather than agreed between two, with every step of the generation checked instead of trusted.

Action. The chain starts at the database and runs outward. The schema and its functions define the shapes; the Go types define the API; Huma emits the OpenAPI description from those; a typed TypeScript client — 44,076 lines of it — is generated from that description; 61 mock handlers are generated alongside it so the frontend’s own tests run against the real contract rather than a hand‑written fixture; and the limits the UI enforces on a form come from the same place instead of being retyped into a validator. Every hop has a guard. A contract check in CI compares what the frontend sends against what the API expects and fails the build on divergence, with a schema probe underneath it that checks the real shapes rather than a description of them. It deliberately covers where drift likes to hide: optional body fields, where “missing” and “null” get confused, and query‑parameter enums, where the two sides can quietly disagree on the allowed values.

Result. A field cannot change on one side only — it fails at the first hop that notices, in a build, minutes after the change. That took out a recurring and genuinely annoying class of bug, the kind invisible in code review that only shows up at runtime. The cost is a generation step in the middle of everything: regenerating is a chore, and the chain is only as trustworthy as its least‑guarded link, which is why each hop got one.

Built email as a platform capability — three providers with failover, delivery webhooks, send and delivery logging, templating and campaigns — behind a startup check that will not boot without one.

2026-09-13T01:45:50+02:00

Situation. Email carries a lot of weight on NextMariner — verification, notifications, digests, campaigns, the things a user actually waits for. And a mail provider is exactly the kind of dependency that fails quietly: the config looks fine, the app boots, and you only find out something’s broken when a real person never gets the message they were promised. That’s the worst way to learn about it. One provider makes it worse, because the failure is total and someone else’s to fix.

Task. Email had to be treated as a capability the platform owns rather than a client library it calls — able to survive a provider outage, able to say what happened to a given message, and loud at startup in the environments where silence is dangerous.

Action. Three providers sit behind one interface — SendGrid as primary, with SMTP2GO and Azure Communication Services behind it — and failover between them is automatic rather than a configuration change made under pressure. Delivery is not assumed: inbound webhooks report what each provider did with a message, and both sides are recorded, in a send log and a delivery event table, so “did this person get their verification mail” is a query rather than a guess. Templating keeps the message bodies out of the code, and a separate broadcast schema — 5 tables and 24 functions — carries campaigns to segments of users, which is a different problem from transactional mail and was built as one. In front of all of it, a startup check sends a real message through the stack, behind a flag: in development it logs a warning and carries on, because nobody wants their laptop refusing to start over an expired sandbox key, and in staging and production a failure is fatal and the process exits rather than deploy a build that cannot send mail. The send path itself goes through an SSRF‑protected client with a 30‑second timeout, and the async delivery path has retries and backoff so a momentary blip doesn’t drop a message.

Result. A whole category of silent failure moved from “a user notices days later” to “the deploy stops”, and a provider having a bad afternoon became a degraded path rather than an outage. The cost is three integrations to keep working instead of one, and delivery logs that grow and have to be pruned — both accepted, because email is the channel the platform cannot route around.

Provisioned Azure infrastructure as code with Bicep — Container Apps, PostgreSQL Flexible Server, Front Door/WAF and networking — across the development, staging and production environments.

2026-09-13T01:45:50+02:00

Situation. NextMariner lives on Azure, and Azure done by hand — clicking through the portal, tweaking a setting here and there — is a trap. It drifts, nobody remembers why something is the way it is, and rebuilding it after a bad day is slow and nerve‑wracking. With more than one environment to keep in step, that only gets worse.

Task. Put the whole thing in code, so an environment is something you can read, review and recreate rather than a pile of manual state.

Action. The estate is defined in Bicep. Each environment — testing, staging, product — comes out of the same templates: the api and www running as Azure Container Apps on a managed environment, a PostgreSQL Flexible Server, Redis for caching, Front Door with a WAF policy out in front, and the networking underneath it (VNet, NSG, private DNS), with Log Analytics wired in for diagnostics. Images are pulled from the project’s Azure Container Registry. Because it’s all parameterised, standing up a fresh environment or changing an existing one is a pull request, not a support ticket to yourself.

Result. The environments became reproducible and reviewable. Drift stopped being a mystery, because the source of truth is the code, and bringing infrastructure up or back is a matter of applying the templates rather than remembering what got clicked last time. It’s the difference between infrastructure you own and infrastructure that owns you.

Built GitHub Actions CI/CD pipelines with a distroless production frontend image and multi‑environment promotion.

2026-09-13T01:45:50+02:00

Situation. Shipping shouldn’t depend on someone remembering the steps, and what ends up running in production shouldn’t be a fat general‑purpose container carrying a shell and a package manager it’ll never use — that’s just attack surface sitting there for no reason.

Task. Make the path from commit to running‑in‑Azure automatic, and keep the production images as small and locked‑down as each workload allows.

Action. The pipeline is GitHub Actions. Separate workflows handle the code‑quality gate, the tests, and the per‑environment deploys, with CodeQL, dependency review and an SBOM step alongside them, so nothing reaches an environment without passing the checks first. The images are multi‑stage builds, and the base for each part was picked on its merits rather than as one blanket choice: the frontend ships on a distroless image (gcr.io/distroless/cc‑debian13 — no shell, no package manager), the Go API on a slim Alpine, and the database image on postgres‑slim. Promotion moves a build through the environments along a defined route rather than by hand.

Result. Releases stopped being a careful manual ritual and became a routine, boring event, which is exactly what you want from releases. The production frontend runs on about as little as you can give it, the checks catch problems before they land, and “deploy” is something the pipeline does rather than something anyone sweats through.

Authored 578 go‑task automation targets spanning native, Docker and HTTPS dev modes, linting, testing, database and deployment.

2026-09-13T01:45:50+02:00

Situation. NextMariner is a polyglot monorepo — Go, TypeScript, SQL, Python, shell — and every one of those brings its own way to build, test, lint and run. Left alone, that means everyone carrying a mental cheat‑sheet of tool‑specific commands, and newcomers spending their first day just working out how to make things go.

Task. Give the whole project one front door: a single, consistent way to run anything, whatever language it happens to be written in.

Action. That’s built out with go‑task — a Taskfile layer that has grown to 578 named targets, 50 in the root file and 528 in namespaced files beneath it. There are the development modes (native, Docker, an HTTPS variant for testing PWA and mobile), the code‑quality side (lint, format, test, fix across all the languages), database management, and the environment‑specific build and deploy tasks. There’s even a low‑memory mode for machines that can’t spare the RAM to build the frontend the usual way. The point was never to have a lot of tasks; it was that you never have to know the underlying command.

Result. Anyone can run task –list and see the whole toolchain laid out, and run any part of it the same way regardless of what’s under the hood. Onboarding got shorter, and the small, stupid mistakes — wrong flag, wrong directory, half‑remembered command — mostly went away. The number is also a warning: 578 targets is past what anyone can hold in their head, so the naming and the namespacing are what keep it usable rather than the count being something to be proud of.

Owned end‑to‑end deployments of the platform to Azure, managing releases across development, staging and production environments.

2026-09-13T01:45:50+02:00

Situation. The platform had to reach users across several Azure environments, and deployment is the seam where the infrastructure, the build pipeline and the application all meet. It’s also where a small mistake stops being a bug and becomes an outage, so it’s the part you least want to be doing by hand and half from memory.

Task. Deployment was owned end to end, so that a change moved out to each environment the same predictable way every time.

Action. Releases move along a fixed route — development, then staging, then production — rather than anyone pushing straight to a live environment. The CI/CD pipeline builds the images and ships them, and the Bicep templates keep the target infrastructure identical from one environment to the next, so a build isn’t promoted into a subtly different place each time. Configuration that differs per environment is kept separate from the secrets, which means the same built artefact can be promoted through the environments and just picks up the right settings where it lands, instead of being rebuilt for each one.

Result. Changes reach each environment predictably, along a defined path, with no ad‑hoc manual deploys in the mix. Releasing turned into a controlled, repeatable step instead of a held‑breath moment, and that’s a big part of what kept the live platform stable while it was still changing quickly underneath.

Configured database backup retention as infrastructure‑as‑code, then audited the recovery position and documented the restore procedure — naming the remaining gaps rather than leaving them to be found during an incident.

2026-09-13T01:45:50+02:00

Situation. A product that lives on its data can’t afford to lose any, and “there are backups somewhere” is a hope, not a recovery plan. The only backup worth having is one you know restores, into an environment you know you can rebuild. NextMariner had neither half written down.

Task. Get the recoverable position defined instead of assumed — the data, the environment around it, and an honest account of how far that actually reaches today.

Action. Backup retention is configured in the Bicep beside the database it protects, so point‑in‑time restore is a property of the template rather than a setting someone once clicked in a portal. The environment around it is defined as infrastructure‑as‑code too, which is the quiet half people forget: restoring a database into an environment you’d have to rebuild by hand from memory isn’t really recovery. Then the position was audited and written up — the restore procedure, the drill that would measure it, and the gaps still open: geo‑redundancy is switched off, and the recovery‑time objective is proposed rather than measured, because no drill has been run yet.

Result. Recovery stopped being a vague reassurance and became a documented position with its gaps named. That reads less impressively than “disaster recovery: done”, and it is worth considerably more — whoever touches it next knows what is covered, what isn’t, and exactly which drill closes the difference. A named gap is one you can close; an unnamed one is discovered during an incident.

Hardened the application with nonce‑based CSP, HSTS, SameSite cookies, least‑privilege database roles and server‑side entitlement re‑checks.

2026-09-13T01:45:50+02:00

Situation. NextMariner holds professional and organizational data, the kind people expect to be handled properly, so a single line of defence was never going to be enough. The working assumption has to be that the client is hostile — that anything the browser enforces can be switched off by whoever’s holding the browser — and the security has to hold up anyway.

Task. The platform had to be hardened at every layer — frontend, API, database — so that security was enforced by the server independently of whatever the interface happened to allow.

Action. On the frontend, the Next.js proxy middleware — proxy.ts — sets a Content‑Security‑Policy with a per‑request nonce and strict‑dynamic, plus HSTS and SameSite cookies, so the browser is locked down about what it’ll run and send. At the API, there’s rate limiting, CORS, request‑size limits, input validation before anything touches the database, and logging of the security‑relevant events. In the database, the API logs in as a least‑privilege role that can only EXECUTE the app functions, the functions run SECURITY DEFINER, and everything is parameterised. And the entitlements — tier, role, organization, ship scoping — are re‑checked on the server on every request, with the frontend gates treated as UX only. The gates decide what you see; the server decides what you can do.

Result. Security doesn’t depend on the UI behaving. The protections are layered so that getting past one doesn’t get you past the rest, and the whole thing is built on the assumption that the client can’t be trusted — which is the right assumption for data people are handing over in confidence.

Set a zero‑warnings quality bar across six languages — Go, TypeScript, SQL, Python, Shell and Markdown — enforced by pre‑commit hooks.

2026-09-13T01:45:50+02:00

Situation. Warnings that pile up are quietly corrosive. Every diagnostic you ignore lowers the bar a little, and once the build spits out forty of them nobody reads any of them, and a real problem sits in that list in plain sight because “warnings” have become background noise. In a polyglot codebase there are that many more sources of noise to let it happen.

Task. The repo needed one uncompromising quality bar across every language, so things got fixed instead of accumulating.

Action. A zero‑warnings policy went in, with the tooling as the thing that enforces it, because a policy that relies on everyone’s vigilance loses to the first busy week. Every linter diagnostic is an error — there’s no “warn” tier to hide in — and it’s the same across the whole stack: Go with golangci‑lint, TypeScript with ESLint, SQL with SQLFluff, Python with Ruff, shell with ShellCheck, Markdown with markdownlint. Inline suppressions are banned, so you can’t paper over a diagnostic; you have to actually fix the thing. Pre‑commit and pre‑push hooks run the linters and the tests, so a commit that would introduce a problem doesn’t get made in the first place. There are even function- and file‑length limits, to keep modules from sprawling past the point of being readable.

Result. Problems get fixed at the source instead of deferred into a backlog nobody clears, and the codebase stays clean by default rather than by periodic heroics. The standard is identical whatever language you’re in, and it’s the tooling holding it — not anyone’s willpower — which is why it actually holds.

Provided round‑the‑clock 24/7 infrastructure support for an IPTV/OTT streaming platform, administering ~1,000 servers plus client‑owned systems for global customers in China, the US and Germany.

2026-09-13T01:45:50+02:00

Situation. This was an IPTV/OTT streaming platform with customers spread across China, the US and Germany, which meant there was no quiet hour to do maintenance in — someone, somewhere, was always watching. Downtime on a platform like that isn’t an abstract metric; it’s someone’s television just stopping, and they don’t care why.

Task. Keeping the platform’s infrastructure available around the clock was the job — genuinely around the clock, not “business hours plus an on‑call rota nobody answers.”

Action. 24/7 support ran for roughly a thousand servers, plus a good number of client‑owned systems on top of that — administering them, monitoring them, keeping them secured and configured across the whole streaming estate. Because customers sat in three very different time zones, “after hours” didn’t really exist; a problem at 3am local was primetime for someone else, so it got treated as primetime. A lot of the work was noticing something drifting before it became an outage, because on a live streaming platform you don’t get to fix things quietly after the fact.

Result. The platform stayed continuously available for a global audience, with problems caught and dealt with at whatever hour they turned up, before they reached a viewer’s screen. On a 24/7 service that’s the whole job — success looks like nothing happening, which is exactly what the viewers wanted.

Ensured uninterrupted delivery of IPTV streaming signals between suppliers and clients, monitoring and maintaining the streaming network and IP telephony around the clock.

2026-09-13T01:45:50+02:00

Situation. IPTV lives or dies on the signal getting through. The streams flow from suppliers, through the platform, to the clients, and any break anywhere in that chain is a black screen for someone. The IP telephony sat alongside it, with the same requirement: it just had to work.

Task. Guaranteeing that the signal delivery and the telephony stayed uninterrupted was the job.

Action. The streaming network and the IP telephony were monitored, troubleshot and maintained around the clock. The point of watching it constantly is that streaming problems announce themselves as degradation before they become an outright drop — a stream that starts stuttering, a link that’s getting flaky — and if you’re paying attention you can catch it at the stutter instead of the black screen. So a lot of it was staying ahead of the signal rather than reacting to complaints about it.

Result. The streams and the calls stayed reliable across the platform, with problems detected and fixed before they turned into service anyone noticed dropping. Keeping a signal flowing between suppliers and end clients without a visible gap is quiet, constant work, and quiet is what it should be.

Planned and implemented new infrastructure functionality for internal and external systems, building solutions durable enough to still run years later with minimal change.

2026-09-13T01:45:50+02:00

Situation. As the organisation grew, its internal and external systems kept needing new capabilities bolted on. The easy way to do that is whatever’s quickest today; the trouble with the easy way is you’re back fixing it in six months.

Task. Planning and building infrastructure functionality that would actually last was the task — not just work now, but keep working.

Action. New infrastructure functionality was planned and implemented across the internal and external systems, designed to be durable — the kind of thing you build once, properly, so it keeps running for years with minimal touching rather than needing constant attention. That’s a deliberate choice each time: spend a bit more thought up front so you’re not signing yourself up to babysit it forever.

Result. The systems stayed functional and effective long after they were built, running for years with barely any change. That longevity is the real measure of infrastructure work — anyone can make something that works today; making something that’s still quietly working years later is the harder and more useful thing.

As one of the first hires, designed and built the entire core infrastructure and supporting processes from scratch for a green‑energy SaaS startup, laying the foundation for rapid growth.

2026-09-13T01:45:50+02:00

Situation. This was a green‑energy SaaS startup with a promising idea and essentially no technical foundation under it yet. Coming in as one of the first hires meant the stage where there’s nothing to maintain because nothing exists — building the ground everyone else will stand on.

Task. Building the core infrastructure and the processes around it, from scratch, was the job.

Action. The entire core infrastructure and its supporting processes were designed and built — the servers, the networks, the data flow, the security, the operations side. Doing that at a startup means making decisions that are hard to unmake later, so the goal wasn’t just “get something running,” it was to lay a foundation that could take the weight of rapid growth without needing to be ripped out the moment the company got bigger. Early infrastructure choices either become the thing that lets you scale or the thing you spend a year undoing; the aim was firmly the first kind.

Result. The startup came away with a solid technical foundation, and it’s what let the business grow quickly afterward. Being the person who builds that base from nothing is a particular kind of responsibility — get it right and nobody notices, get it wrong and everyone does — and this one held up.

Automated team collaboration, password management, task and time management, and built a semi‑automatic project‑showcase system, raising team productivity.

2026-09-13T01:45:50+02:00

Situation. A lot of the team’s operational work — coordinating, managing passwords, tracking tasks and time — was being done by hand, and manual coordination is a quiet tax: it’s never the thing you notice, but it steadily eats hours that could go somewhere better.

Task. Automating that repetitive operational work was the goal.

Action. The pieces that lent themselves to it got automated — team collaboration, password management, task management, time management — with a semi‑automatic project‑showcase system on top. The idea across all of it was to take the routine coordination off people’s plates so it ran itself, and let them spend the recovered attention on work that actually needed a human. The showcase system was the same instinct applied to something more visible: make presenting the work mostly automatic rather than a manual chore each time.

Result. Operational efficiency went up and the team got more productive, because the routine coordination that used to need constant human attention now largely ran on its own. The time that was leaking into busywork went back into the actual work.

Integrated a company‑wide password‑management system, strengthening security and streamlining access control.

2026-09-13T01:45:50+02:00

Situation. Credentials were being handled inconsistently — different people storing and sharing them in different, ad‑hoc ways — and that inconsistency is itself the security risk. It’s rarely a dramatic breach; it’s a password in a chat message, a shared login nobody rotates, the slow accumulation of small exposures.

Task. Centralising the credentials and making them secure was the task.

Action. A company‑wide password‑management system went in, so there was one consistent, secure way credentials got stored and shared instead of everyone’s personal habit. The value of company‑wide is exactly that it’s not optional per person — a password manager only half the team uses barely helps, because the risk lives in the half that didn’t. So the point was to make the secure way the default way, everywhere.

Result. Security improved and access control got simpler and more consistent across the organisation. Once the credentials all live in one managed place, a whole set of small, boring exposures just stop being possible — which is most of what real‑world security actually is.

Built a layered automated test suite — 981 Go tests, 543 frontend and browser specs, 494 SQL behavioural tests — with mutation testing, property‑based tests and an accessibility gate.

2026-09-13T01:45:50+02:00

Situation. A platform that keeps its business logic in the database has a testing problem most projects do not. The logic is not in the language the test framework is good at — it is in SQL, behind function boundaries, and SQL is exactly the sort of code that goes untested because testing it is awkward. Add a Go API and a Next.js frontend on top of that, and “the important parts are covered” quietly turns into “the parts that were easy to cover are covered.”

Task. Each layer needed its behaviour checked where that behaviour actually lives, rather than everything being checked from the outside through a browser.

Action. Four layers got four kinds of test. The Go API carries 981 test functions across 310 files. The frontend carries 543 Vitest and Playwright specs, including 48 end‑to‑end files and 30 browser specs. The database carries 494 behavioural test files — 116,876 lines of SQL asserting through 6,654 raised exceptions — so a stored function is tested in the database instead of through three layers of application above it. Above those sit the tests that test the tests: Stryker mutation testing deliberately breaks a line and fails when nothing notices, and fast‑check generates inputs nobody thought to write down. An axe‑core gate asserts zero WCAG 2.0 and 2.1 A and AA violations, which makes accessibility a build failure rather than an audit finding months later. Coverage thresholds only ever move upward. The whole suite runs as 10 jobs in a 612‑line workflow.

Result. Changing something structural stopped being frightening, which is the only thing that keeps a codebase this size from calcifying. The honest cost is time — the suite is slow, it taxes every change, and at this size it needs maintenance of its own. What it buys is the ability to keep moving quickly, and that is worth more than the minutes it takes.

Built the repository's guard engine — 268 registered commit checks, 277 lint rules and 15 custom ESLint rules — plus 146 tests of the guards themselves, so the build holds the standard, not review.

2026-09-13T01:45:50+02:00

Situation. Standards written down in a contributing guide are suggestions. Everyone agrees with them, and then it is Friday, the change is small, and the guide loses. The zero‑warnings bar was only ever going to hold if something other than goodwill was holding it — and the linters that ship with each language stop well short of the project‑specific rules that actually matter, the ones about how this codebase in particular is meant to work.

Task. The rules the project cared about had to be executable, so that breaking one failed a commit rather than waiting for a reviewer with the time and the memory to catch it.

Action. What grew out of that is a guard engine. There are 274 check scripts, 268 of them registered into the commit hooks, alongside 277 JavaScript linters and 96 shell and 17 Python validators covering the things off‑the‑shelf tooling has no opinion about — that a migration can be rolled back, that a translation key exists in both locales, that a registered route appears in the OpenAPI description, that nobody quietly added an inline suppression. Fifteen custom ESLint rules hold the house patterns in TypeScript. One rule is worth naming on its own: any commit prefixed fix: has to carry a test that fails without it, so a bug that is fixed stays fixed. And because a broken guard is worse than no guard at all — it passes everything and nobody finds out — the guards have 146 tests of their own.

Result. Review time moved off mechanics and onto design, because the mechanical objections were already made by a machine before the branch was pushed. The trade‑off is real and worth stating: committing is slow, and a badly written guard is genuinely infuriating to work around. Those 146 tests exist because that happened.

Built the payments and entitlements layer — Stripe alongside Apple and Google in‑app purchase — gating the directory, search and export through an 11‑table access model checked on the server.

2026-09-13T01:45:50+02:00

Situation. Money is the part of a platform nobody gets to be casual about. A subscription has to survive a card expiring, a refund, a plan change, a webhook that arrives twice and a webhook that arrives out of order. The moment payment can happen on three storefronts — a card on the web, Apple in one app store, Google in the other — there are three different accounts of what somebody bought, and the product still needs one answer to one question: what is this person allowed to do right now?

Task. Taking money and granting permission had to be two systems rather than one, so that adding a storefront would not mean rewriting every gate in the product.

Action. Stripe handles cards and subscriptions through 18 Go files, and Apple and Google in‑app purchases arrive through their own receipt verification. All three converge on a payments schema of 9 tables and 34 functions — and then stop there. What the product actually asks is a separate access schema, 11 tables and 34 functions, which answers “may this account do this?” without knowing or caring which storefront paid for it. That answer gates the company directory, search and data export, and it is re‑checked on the server on every request, because a hidden button is a courtesy and not a control.

Result. Adding a storefront now touches the payments side and leaves the gates alone, and a support question about someone’s access has one table to look at rather than three. The cost is two schemas where a smaller product would want one, plus an entitlement lookup on requests that would otherwise have been free.

Moved slow work off the request path onto a River job queue — 15 worker modules, 8 scheduled tasks and 20 pg_cron jobs — so a request returns while the work behind it carries on.

2026-09-13T01:45:50+02:00

Situation. Some work has no business happening while a user waits. Sending mail, rebuilding a search index, generating a document, recalculating standings — do any of it inside the request and the user watches a spinner for something they never asked to see. Do it in a goroutine instead and it disappears the moment the process restarts, which it will, mid‑deploy, with no record that it was ever supposed to happen.

Task. Background work needed somewhere durable to live: a queue that survives a restart, retries a failure, and can be looked at when something has not happened.

Action. River was the choice, largely because it keeps its queue in PostgreSQL — the database is already the system of record, so a job and the rows it touches commit or roll back together, and there is no second piece of infrastructure to run and reason about. There are 15 worker modules behind it and 8 scheduled tasks. Underneath, 20 pg_cron jobs handle the maintenance the database is better placed to do itself: pruning partitions, rotating salts, refreshing aggregates. Anything slow enough to notice was moved off the request path and onto one of the two.

Result. Requests return quickly and the slow work still finishes, with retries and a visible history when it does not. Keeping the queue in Postgres rather than a dedicated broker is a deliberate limit: it will not scale forever, and at some volume it becomes the wrong answer. For a platform whose bottleneck is the database anyway, one less moving part was worth more than headroom that was not going to be used.

Built first‑party error monitoring and OpenTelemetry tracing rather than buying them — payload sanitising, spike and regression detection, symbolication and a synthetic heartbeat — behind 11 operator views.

2026-09-13T01:45:50+02:00

Situation. The difference between a platform that is up and a platform that is working is whether anyone would know. An error a user hits at eleven at night, on a page nobody tests, is invisible unless something goes and collects it. The usual answer is to buy a hosted error tracker, which is a good answer — and it also means the platform’s own errors, stack traces and user context leave for a third party.

Task. Errors and traces had to be collected, grouped and made actionable, without the platform’s internals leaving the platform.

Action. Two pieces got built. OpenTelemetry handles tracing over OTLP, so a slow request can be followed across the frontend, the API and the database rather than guessed at. Alongside it sits an in‑house error pipeline — an errmon service and an ingest service — that sanitises payloads before storage, groups errors into recurring problems rather than a flat list, detects spikes and regressions with a cooldown so one bad deploy does not page anyone forty times, symbolicates minified frontend stack traces back into readable code, and runs a synthetic heartbeat to prove that the pipeline itself is alive. It all lands in the database as error events, error groups, an inbox, API latency and stack samples, and it surfaces through 11 operator views, including one for service level objectives.

Result. Errors turn into a queue that someone can work through, and a regression announces itself instead of being discovered by a user. Building rather than buying cost real time and means this is one more thing to maintain — a bought tracker would have been running the same afternoon. What it bought was that nothing sensitive leaves, and that the alerting rules fit this platform rather than a generic one.

Kept the schema honest across 1,022 migrations with a CI gate that builds the database both ways — a fresh install, and an install plus every migration — and fails when the two disagree.

2026-09-13T01:45:50+02:00

Situation. A schema is described twice in most projects: once by the initial setup that builds it from scratch, and once by the accumulated migrations that grew it. Both are supposed to produce the same database. Nothing checks that they do, so they drift — and the drift is invisible until a fresh environment behaves differently from production, usually at the worst moment.

Task. The two descriptions had to be provably identical, automatically, rather than periodically believed to be.

Action. The database is versioned as 1,022 migrations, numbered from 036 through 1102, and the ordering discipline around them is unglamorous and non‑negotiable. What makes it hold is a CI job that builds the database twice on every change: once from the fresh initial schema, once from the initial schema plus every migration replayed in order — and then compares the two. Not just the structure, which is the easy half, but the seeded data too, because a migration that backfills a lookup table wrongly is exactly as damaging as one that forgets a column, and only one of those shows up in a schema diff. Any disagreement fails the build with the difference printed.

Result. A fresh environment and a long‑lived one are the same database, and that is checked rather than assumed. The cost lands on whoever writes a migration: it has to work replayed and it has to work from cold, which is more thought than a quick ALTER usually gets. That is the point — the alternative is finding out during a restore, when the answer matters and there is no time to work it out.

Built fail‑closed abuse controls — 22 Redis‑backed rate limiters, Cloudflare Turnstile, request idempotency and an origin lock — so the platform sheds bots and floods instead of trusting its callers.

2026-09-13T01:45:50+02:00

Situation. A public directory of companies and professionals is a target the day it goes live. Scrapers want the data, spam accounts want the reach, and an endpoint that costs the platform real money to serve — search, export, anything that touches an external API — is worth abusing simply because it is free to call. None of that is malice aimed at this platform in particular; it is background weather on the open internet.

Task. The expensive and abusable paths needed limits that hold under pressure, including the pressure of the limiter’s own dependency being unavailable.

Action. There are 22 rate limiters, each constructed for the path it protects rather than one global cap, because a login attempt, a search and a bulk export are abusive at wildly different rates. State lives in Redis so a limit is shared across instances instead of being per‑process and trivially escaped. The important decision is what happens when Redis is not there: the limiters fail closed. Traffic is refused rather than waved through, which is the less convenient answer and the only defensible one. Around them sit Cloudflare Turnstile on the paths worth challenging, 351 lines of idempotency middleware so a retried write does not become two, an origin lock that refuses requests not arriving through the front door, and a bespoke challenge on the directory itself.

Result. Abuse gets expensive for the abuser and cheap for the platform, and an outage in the limiter’s own store degrades into refusal rather than into an open door. Failing closed does mean a Redis problem becomes a user‑visible problem — accepted deliberately, because the alternative is a Redis problem becoming a billing one.

Built company ownership claims end to end — a user claims a company, an administrator adjudicates, and an approval rewrites the authorization graph that decides who is allowed to edit what.

2026-09-13T01:45:50+02:00

Situation. A directory seeded from public sources has a structural problem: the companies in it did not put themselves there. Sooner or later somebody from one of them turns up wanting to correct their own entry — and there is no relationship between that person and that record, only an assertion that one exists. Grant it too readily and a competitor edits your page. Grant it too slowly and the directory stays wrong.

Task. There had to be a route from “this is my company” to genuine authority over the record, with a human decision in the middle and a trail behind it.

Action. Claims were built end to end, across 108 commits and both applications. A user submits a claim with evidence; it enters a queue in the back‑office; an administrator reviews it and approves or rejects it with a reason that goes back to the claimant. The interesting part is what approval does — it is not a flag on a row. Approval rewrites the authorization graph, so the account gains a real relationship to the organization, which is the same relationship every permission check in the platform already consults. The gate is the adjudication, not the code path, and no feature had to learn about claiming in order to respect it.

Result. Companies can take over and correct their own entries without anyone editing the database by hand, and every grant of authority has a named approver and a reason attached. Human review is the bottleneck by design; an automated check on a domain name would be faster and would be wrong in exactly the cases that matter most.

Established a continuous security programme — code scanning, DAST, dependency and vulnerability checks, SBOM generation, secret scanning and SHA‑pinned actions — alongside 21 written security audits.

2026-09-13T01:45:50+02:00

Situation. The security work in the application — the content security policy, the least‑privilege database roles, the entitlement re‑checks — protects the platform at runtime. None of it says anything about the thing being shipped: whether a dependency picked up a known vulnerability last Tuesday, whether a credential was committed and reverted, or whether a third‑party action pinned to a tag has quietly become a different piece of code.

Task. Supply chain and code security had to be continuous and automated, so the state of it was a build result rather than an opinion.

Action. Static analysis runs through CodeQL, dynamic testing against a live instance through ZAP, and Go dependencies are checked with govulncheck. A software bill of materials is generated on every build with Anchore and Syft, so what shipped is known rather than reconstructed later. Gitleaks scans history for credentials. Every third‑party GitHub Action is pinned to a commit hash rather than a tag, which is the unglamorous control that stops a tag being moved under you. Dependabot watches 6 ecosystems. Alongside the automation sit 21 written security audits, including a threat model and an assessment against the OWASP testing guide — because scanners find the classes of problem someone has already described, and a threat model is where the ones specific to this platform get named.

Result. A vulnerability disclosed upstream shows up as a failing build rather than as news. The volume of findings is the real cost — a scanner that reports everything trains people to ignore it, and keeping the signal usable takes ongoing triage rather than a one‑time setup.

Built the company's own infrastructure as 19 Ansible playbooks and 34 roles across 12,065 lines of YAML, converging a live host to a declared state with every play idempotent.

2026-09-13T01:45:50+02:00

Situation. Engineer ApS runs its own estate — a web presence, a git forge, a database, backups, mail transport and DNS — and there was nobody to hand the operations to. A one‑person company has the same failure modes as a large one and none of the redundancy, which makes the usual answer, a person who remembers how the host was set up, the least available option there is.

Task. The whole estate had to be described in a repository rather than in a head, and described in a form that converges a real host rather than documenting one.

Action. What grew from that is 19 playbooks and 34 roles across 12,065 lines of YAML. The shape matters more than the size. Composition is data rather than flags: a host sits in a tier group whose variables declare which roles it runs, so provisioning with no arguments converges every host to its declared state. Idempotence is a contract rather than an aspiration — a converged host reports zero changes, and a play that cannot say that is not finished. Four command namespaces keep the promises apart: a gate that touches no host, reports that read one and never change it, provisioning that changes a host to match the repository, and verification that changes a host on purpose and returns a verdict.

Result. The estate is reproducible from the repository, and the parts of it that were only ever true because somebody remembered them are now assertions that fail a run. The cost is real: every change is slower to make than editing a file on the server would be, and a converge that half‑applies is worse than one that refuses, which is why a preflight play was later added in front of it. That trade was made deliberately and it has held.

Ran the whole company on one 512 MB single‑core host — a git forge, a web server serving seven domains, Tor, two alternate‑protocol servers, backups and intrusion banning — by treating 464 MB of usable memory as the binding architectural constraint.

2026-09-13T01:45:50+02:00

Situation. The company’s production host is a single‑core cloud instance with 512 MB of memory and 10 GB of disk, of which about 464 MB is usable. Everything the business runs in public sits on it: the web server terminating TLS for seven domains, the git forge, a Tor onion service, a Gemini server, a Gopher server, encrypted backups and intrusion banning. The usual reaction to that list is to buy a larger box.

Task. The constraint had to be treated as an architectural input rather than a problem to spend money on, because the honest question was not whether a bigger machine would work but whether the design needed one.

Action. Memory became the argument that settled decisions. There is no monitoring agent, no metrics pipeline and no dashboard — reporting is a pull, seven commands that read the host and render Markdown, changing nothing and running only when asked. The supervisor is systemd rather than a second process manager layered on top of it, and the container plane is Quadlet units under the same supervisor rather than a daemon with its own. Web‑panel platforms were ruled out at the design stage for the same reason. When the question came up of whether the host could carry an onion service, the answer came from a day of measured samples rather than an opinion: available memory never fell below about 310 MB of 464, swap sat at 2.6 per cent and the processor was 99.7 per cent idle.

Result. The whole company runs on a machine that costs less per month than a lunch, and the design is better for the discipline rather than merely cheaper. What it cost is headroom for anything careless — mail is deliberately not on this box at all — it is written as an install scaffold awaiting a host of its own, because a mail server needs headroom this machine has already spent, and that is written down rather than discovered later.

Found and closed three SSH brute‑force protections that had never worked: a ban jail watching port 22 while the daemon listened on 1986, a rate limit shadowed by a broader rule above it, and a ban action whose binary never resolved, so no ban had ever applied.

2026-09-13T01:45:50+02:00

Situation. A hardening review of the production host in August 2026 asked a question that usually gets a confident answer: are the SSH brute‑force protections working. All three were configured, all three appeared in every report anyone looked at, and all three had been inert since the day the host was built.

Task. The controls had to be checked against what the kernel actually does with a packet, rather than against the configuration files that describe what should happen to one.

Action. Reading the configuration would have confirmed the wrong answer three times, so the review read the running system instead. The intrusion‑banning jail was watching port 22 while the daemon had been moved to 1986 during the initial hardening — every ban it wrote named a port nothing listened on. The firewall’s rate limit was worse in a subtler way: the rule existed, and it sat below a broader rule that matched first. Firewall user rules are evaluated top down and the first match wins, so a broad allow above a rate limit makes the limit dead code that still prints in every status listing. The third was the quietest of them: the ban action shells out to a packet‑filter binary that the package system only recommends rather than requires, so on a host without it the jail starts, counts and decides, and then fails at the one moment it tries to ban. All three fixes were small. What came out of it was not the fix but two rules that now govern the repository: a security control gets an assertion rather than a comment, and a firewall is verified by rule position rather than by rule presence.

Result. Three protections that had never once fired now do, and the class of defect they belong to — a control whose failure mode is that it keeps reporting healthy — is the class the platform’s checks are now built to catch. All three had been inert from bootstrap. All three printed healthy in every place anyone looked, which is the whole reason they lasted.

Hardened SSH to 24 asserted directives with three‑stage validation — the candidate file, the assembled config, then the daemon's own read‑back — after the read‑back caught the running server silently overriding two of the twenty‑four.

2026-09-13T01:45:50+02:00

Situation. SSH is the only interactive way into the company’s host, and its configuration is written by an automation role that had been running clean for months. An access report in September 2026 read the daemon’s own resolved settings and found two of them disagreeing with what the role had been writing on every single converge.

Task. The hardening had to become something the daemon confirms rather than something the repository asserts, because the gap between the two had already been open for months without anyone noticing.

Action. The cause was configuration ordering. The operating system ships its own defaults uncommented, above where a drop‑in file lands, and for the two settings in question the first occurrence wins. The fix was a filename prefix that sorts ahead of the vendor’s, which is a one‑character class of change and exactly the sort that stays broken because nobody thinks to look. What was built around it matters more: three stages of validation on every converge. The candidate file is syntax‑checked before it is installed, so an invalid config never reaches the host. The assembled configuration is checked after installation. Then the daemon’s own resolved output is read back and 24 directives are asserted against it, so a setting that is written but overridden fails the run. Two directives were deliberately left out, both because the daemon no longer implements them and writing them would only look thorough.

Result. The SSH posture is now the daemon’s answer rather than the repository’s claim, and the difference is not theoretical — it was already two settings wide when the check was written. A converge that cannot see whether a control took effect has not verified it, and grepping for a directive only proves it was written.

Proved the intrusion‑banning path end to end on every hardening run by banning a reserved test address, reading the resulting kernel rule and unbanning in a guaranteed cleanup block, so a jail that stops working fails a run instead of reporting healthy.

2026-09-13T01:45:50+02:00

Situation. The intrusion‑banning service had already been caught banning a port the SSH daemon did not listen on. Fixing the port closed that instance. It did nothing about the reason the fault survived so long, which is that a ban path has no visible failure: the service runs, the jail is listed as active, and nothing anywhere says whether a ban ever reaches the kernel.

Task. The ban path had to be exercised on every converge, against the live rule set, rather than inferred from the service being up.

Action. The converge now bans an address from the block the standards reserve for documentation, reads back the resulting rule in the kernel’s packet filter, and unbans it in a cleanup block that runs whether the check passed or failed. The reserved range is the load‑bearing choice — the test address belongs to nobody, so a stray ban that outlives a failed run cannot lock out a real network. The kernel read‑back is the other half: the service’s own status output would report success for a ban that produced no rule, which is the exact failure being tested for. Alongside it, the ban backend is pinned rather than left to the service’s detection, and the log backend is set to detect because the operating system ships no traditional authentication log and the wrong choice fails silently by watching a file that never appears.

Result. A banning path that stops working now fails a converge instead of continuing to report healthy, which is the only property that mattered. It costs a few seconds on every run and it writes and withdraws a firewall rule against production every time, which is a real intrusion into a live system and was accepted on the grounds that the alternative had already been demonstrated to be worse.

Verified firewall rules by position rather than presence, reading the numbered rule list and the live packet‑filter chain, because a rule that exists is not a rule any packet reaches.

2026-09-13T01:45:50+02:00

Situation. A firewall status listing shows a set of rules. A packet filter evaluates an ordered list and stops at the first match. Those are different things, and the difference is invisible in every tool that prints a summary — which is how the host ran for a year with a rate limit that no packet ever reached, sitting below a broader rule that matched first.

Task. Firewall verification had to read position rather than membership, because presence had already been demonstrated to prove nothing.

Action. The checks now read the numbered rule listing and the live filter chain, and assert against order. Exactly one rule exists per port and always with a protocol, because a rule without one silently widens the surface. The SSH port carries a rate limit and never an allow beside it, since an allow above a limit is precisely the shadowing that was found. The declared public surface is a short list of five ports, asserted as a whole rather than checked individually, so a port that appears without being declared fails rather than being noticed. The security report renders the rules in match order with a warning at the top of the section explaining how to read it, because the next person to open that file will otherwise read a set.

Result. The firewall posture is now checked the way a packet experiences it. The most useful outcome was not the check but what it revealed about the reporting that preceded it: every tool involved had been printing the rate limit for a year, correctly, and none of them had been asked the only question that mattered.

Built encrypted off‑host backups on restic with retention pruning, an integrity check and a monthly automated restore drill, then audited the recovery position and wrote down the gaps rather than leaving them to be found during an incident.

2026-09-13T01:45:50+02:00

Situation. Everything the company holds — the git repositories, the database, the site — sat on one cloud instance whose provider snapshot was the entire recovery position. A provider snapshot is a fine thing to have and a poor thing to rely on: it is in the same account as the machine it protects, it is not encrypted by anyone here, and nobody had ever restored from one.

Task. Backups had to be encrypted, off‑host, pruned on a retention policy, and — the part that is usually skipped — actually restored from, on a schedule, without a person remembering to do it.

Action. The backup runs on a systemd timer: a database dump where one exists, then an encrypted deduplicated snapshot to storage at a different provider over SFTP, then a retention prune, then an integrity check. A dead‑man’s switch is pinged on success only, which is the distinction that makes it an alarm rather than a log — a run that fails says nothing, and saying nothing is what raises the alert. Separately, a restore drill runs monthly: it pulls a known file out of the repository and compares it, so the thing being checked is a restore rather than a backup. The passphrase is written to a file the units read rather than passed through the environment, because the supervisor processes escapes in environment values and a passphrase containing a backslash would have been silently different from the one that created the repository. Afterwards the recovery position was audited and written up, and the remaining gaps were named in the document rather than left to be discovered during an incident.

Result. The company can lose the host and get its data back, and that sentence rests on a restore that ran last month rather than on a backup that ran last night. The audit’s most valuable output was the list of things still not covered, which is the part a green backup report is structurally unable to tell anyone.

Built dead‑man's‑switch monitoring that pings only while memory and disk are healthy, so a degraded host raises an alert by going silent — and caught six variable names saying "free" where the check correctly measured "available", an order of magnitude apart on a 464 MB box.

2026-09-13T01:45:50+02:00

Situation. A 464 MB host running a forge, a database and a web server has two realistic ways to die: it runs out of memory, or it runs out of disk. Neither announces itself. Both are entirely predictable a few hours in advance if anything is looking, and nothing was.

Task. The host needed an alarm that works when the host does not — which rules out anything that has to send a message at the moment of failure.

Action. The answer is a dead‑man’s switch on a timer. Every fifteen minutes with jitter, a small script measures available memory and free disk and pings an external service only if both are above their thresholds. Silence is the alert. A machine that has run out of memory, lost its network or stopped booting produces exactly the same signal as one that is unhealthy, which is the correct behaviour and the reason this shape was chosen over an agent that reports a status. The measurement is available memory, not free memory, and that distinction turned into the most instructive part of the work: the script had been reading the right column all along while six variable names around it said “free”. On a healthy Linux box free memory is near zero because the kernel uses idle memory for cache, so a reader checking those names against a ten per cent threshold would see twelve megabytes free on a 464 MB machine, conclude the check was broken, and fix it with a one‑character change that converts a working alarm into one that breaches permanently and gets muted inside a week.

Result. The host now has an alarm whose failure mode is to fire, and the naming that would have destroyed it has been corrected. The switch’s real limitation is stated in its own documentation: it proves the box is healthy, not that the site is serving, and those are different questions.

Made check mode tell the truth across the whole platform after finding six probes deciding on a value the host never gave, because Ansible's command module reports success under --check while skipping the command entirely.

2026-09-13T01:45:50+02:00

Situation. A dry run against production is supposed to be the safe way to find out what a change will do. In September 2026 a dry run failed on an assertion that was simply wrong about the host, and advised the operator to set a flag that would have relaxed a security sandbox. The dry run had not read the host. It had read a value the host never gave and drawn a conclusion from it.

Task. Every probe whose result feeds a decision had to be made honest under check mode, and the ones that could not be had to say so out loud instead of staying quiet.

Action. The cause is a property of the tool that is documented and easy to forget: the command module does not execute under check mode, and what it registers is not an empty result — it is a success with empty output. Any conditional reading that register therefore decides on a value that was never measured, and it decides in whichever direction its own logic happens to point. A sweep of every registered probe found six of them, each lying differently: one reported that the supervisor accepts every unit directive without having asked the supervisor, another reported nothing to do on a host with the firewall switched off. Each was given one of two shapes. A probe that reads pre‑existing state the run has not touched is marked to run even under check mode. A probe that cannot run is skipped, and a message names what was specifically not verified — because silence in a run log reads exactly like a pass.

Result. A dry run now either measures something or says it did not, and neither of those is the third option it used to have. The general rule went into the development guide in the same change: check mode must not lie, and a probe that cannot see is required to announce its blindness rather than infer a verdict from it.

Added a preflight play that runs the same code as the converge against operator‑local secrets in about a second, after a half‑applied production run died on its ninth task with the swap settings already written to the live host.

2026-09-13T01:45:50+02:00

Situation. Operational secrets had just been split so that each host carries its own backup repository, passphrase and alert switch. The repository code was correct. The operator’s encrypted vault still held the previous single‑host value, and nothing could see the mismatch, because the vault is deliberately local to the operator’s machine and invisible to the repository’s own quality gate.

Task. That class of failure needed somewhere to fail cheaply, because it had just failed expensively.

Action. The first converge after the change connected to the live host, ran eight tasks and refused on the ninth — correctly, but from inside a run that had already written swap settings to production. A half‑applied converge is a worse answer than a refusal, so a preflight play was written: it reaches no host at all, runs locally, gathers no facts, and executes the same two task files the real converge uses against the operator’s vault. It answers in about a second, and provisioning runs it first. The load‑bearing decision is that it is the same code rather than a second implementation of the same rule, because a check that restates a rule in another language eventually disagrees with it and disagrees silently. Writing it surfaced a trap it nearly created: facts set during a play outlive that play, so chaining the preflight and the converge into one invocation would have handed the real role a passphrase the preflight had already resolved, and passed the check on a vault that was still wrong. That was reproduced before it was guarded, in both task files.

Result. A secret that does not resolve now costs a two‑second refusal on a laptop instead of a partly‑applied change to a live server. The preflight is deliberately not tagged to always run and is deliberately not serialised, and both of those are recorded as decisions rather than defaults.

Reconciled a 20‑record DNS zone declaratively against the Cloudflare API with separate audit and BIND‑export entry points, and turned the CDN proxy back off on privacy grounds after building it.

2026-09-13T01:45:50+02:00

Situation. The company’s DNS carried about twenty records across a web presence, a git subdomain, two personal redirects, mail routing through a hosted mailbox provider, a transactional sending domain and the authentication records for both. All of it lived in a provider’s web console, which means the current state was whatever the last person to click had left behind.

Task. The zone had to become a declaration in the repository, with a way to compare that declaration against what the provider is actually serving.

Action. The zone is written as data — live records, application records, transport‑policy records and a separate list of records pending removal, which is the honest way to record a deletion that has not happened yet. One play reconciles it against the provider’s API. Two more entry points sit behind explicit opt‑in tags: an audit that reports the difference without changing anything, and an export that writes the zone out in the standard zone‑file format so it can be read by something that is not this repository. The most interesting decision was a reversal. Proxying the two web faces through the provider’s content network was built, worked, and was then turned back off — because it costs the one sentence the company’s privacy page is about. With a proxy in front, another company handles every visitor’s address and every URL they request, before we do, under their policy rather than ours. For a business whose distinguishing claim is that nobody is watching, that is a worse trade than the attacks it defends against.

Result. The zone is versioned, comparable and exportable, and the one decision that went against the obvious default is written down with its reasoning so that nobody re‑litigates it by accident. The audit path is the part that gets used most, because knowing the difference is more often what is wanted than closing it.

Cut systemd sandbox exposure across every unit this platform installs — a dead‑man's‑switch service from 9.6 UNSAFE to 1.5, an internet‑facing git forge from 8.3 EXPOSED to 1.5 — and added a converge‑time parser check after finding a misspelled directive silently ignored in three unit templates.

2026-09-13T01:45:50+02:00

Situation. Thirteen unit files were installed by this repository and run on the production host with the privileges the supervisor gives by default, which is most of them. A hardening score put the dead‑man’s‑switch service at 9.6 and called it unsafe; the internet‑facing git forge scored 8.3 and exposed. Both numbers were accurate and neither had prompted anything, because a score with no threshold attached is a number people learn to skip.

Task. Every unit needed a sandbox proportional to what it actually does, and the sandboxes needed a check, because the failure mode of a wrong sandbox is that the unit starts and then breaks somewhere the parser cannot see.

Action. Privilege restrictions went into every unit template — a privilege ceiling fixed at the unit’s own, a system‑call filter, restricted address families, and memory that is writable or executable but never both. The measured results: the dead‑man’s switch went from 9.6 to 1.5, the forge from 8.3 to 1.5, and both backup units from 9.6 to 2.3 and 2.5. Writing them turned up something better than the scores. One directive had been misspelled in three unit templates for as long as those roles had existed — a plausible‑looking name that does not exist, which the supervisor answers by logging that it does not recognise the key and starting the unit anyway. Grepping for a directive proves it was written; only the parser proves it took effect. So a verification role now runs every unit through the supervisor’s own verifier during the converge, and is included by every role that installs a unit. What that still cannot see is stated plainly in the same documents: every directive capable of breaking these units breaks them at runtime, not at parse time, and a score cannot tell whether a script still reaches its last line.

Result. Every unit runs with the privileges it needs and not the ones it does not, the numbers are measured rather than asserted, and the class of defect where a typo becomes a silently absent control now fails a run. The limits of the measurement are recorded next to it, which is the part that keeps the next reader from over‑trusting a green band.

Built seven read‑only reporting roles that render a live host to Markdown — facts, access, git, metrics, traffic, security and provider inventory — under a rule that no number is printed the run did not measure.

2026-09-13T01:45:50+02:00

Situation. The host had no dashboard and was not getting one, because a metrics stack does not fit on 464 MB and would not have been worth its cost if it did. That left an honest gap: there was no way to answer questions like who has access, how much has this repository grown, what does the traffic look like, or is the hardening actually enforced.

Task. Those questions needed answers that could be produced on demand, cost the host nothing while nobody was asking, and could never change the thing they were describing.

Action. Seven read‑only roles, each rendering a live host into Markdown in the repository. A facts snapshot covering operating system, hardware, storage, network, services, packages, listening sockets and processor utilisation computed from two samples of the kernel’s own counter. An access report covering accounts, sudo scope, key fingerprints, forge users and database roles. A git report per repository. A metrics report over the day’s collected samples. A traffic report built from the web server’s privacy‑masked log. A provider inventory across four provider surfaces — DNS, two cloud APIs and a dedicated‑server API. And a security report that answers whether hardening is enforced rather than configured, printing the firewall rules in match order and the live ban rule set. The governing rule across all seven is that no number is printed that the run did not measure — a report that fills a gap with a plausible figure is worse than one that leaves it blank, because only the plausible one gets believed.

Result. The estate can be described from the repository on demand, and the reports have found real things: the two dead security controls, an undeclared listening port, and the observation that the traffic archive is only as old as the last time somebody remembered to harvest it. That last one is written up as a limitation with a specific cost recorded — sixteen days of one site’s history that rotated away between harvests and do not exist anywhere.

Deployed the company's own git forge on Soft Serve, private by default with no web panel and its SSH port bound to loopback behind a jump host, and made the landing page in front of it a build artefact of the main site rather than a hand‑kept copy.

2026-09-13T01:45:50+02:00

Situation. The company’s source code lived on a third‑party hosting service, which is a reasonable place for it and a poor fit for a business whose argument to clients is that it does not hand their data to intermediaries. Running a forge instead means running a forge: authentication, access control, storage, backups, and a public face for it.

Task. A canonical git server had to be stood up on the existing host, with the smallest possible attack surface and no web administration panel, and a landing page in front of it that does not rot.

Action. The forge is a single binary supervised by the operating system, installed from its vendor’s package repository, deliberately chosen for being SSH‑first with no administrative web interface — the less interface, the less to defend. It is private by default: anonymous access is refused, keyless access is refused, and every declared repository is marked private rather than relying on obscurity. Its SSH listener binds only to the loopback interface on port 23231 and is reached from outside through a jump host, so the firewall’s declared surface does not grow. A protocol multiplexer that would put HTTPS and git SSH on one public port by inspecting the first bytes of a connection is written and ready behind a master switch, and that switch is off: multi‑user git over the public port is not needed yet, and a listener nobody uses is surface. The landing page in front of it was the more interesting problem. It had been a hand‑maintained copy of the main site’s design in its own repository, and every single difference ever found between the two turned out to be an accident rather than a decision: a type scale rendering the wordmark about nine per cent too large, a font declaration that collapsed two weights to one on any machine with the family installed, ornaments hidden below a certain width so they were absent on every phone, a weight used with no font shipped for it, and no main landmark or top‑level heading on any page. Five for five, and not one visible in a screenshot. The copy was deleted; the landing page is now built by the main site’s own templates and delivered as an artefact.

Result. The company hosts its own code on its own hardware, and the page in front of it inherits every check the main site passes rather than drifting away from it in ways only a measurement can see. Divergence is still available and now has to be written down.

Deployed a container plane on Podman and Quadlet under systemd rather than Docker, because Docker publishes container ports above the host firewall's own rules — and gave deploys an unprivileged user with one fixed command instead of root.

2026-09-13T01:45:50+02:00

Situation. The platform needed somewhere to run application containers. The obvious choice was the industry default, and the obvious choice was wrong for this host in a specific way: it rewrites the kernel’s packet‑filter rules and publishes container ports above the firewall rules the hardening role installs, so a container quietly becomes reachable from the internet regardless of what the firewall was told.

Task. A container plane had to be chosen that does not bypass the firewall, does not add a second supervisor beside the one already trusted, and does not require a person to be root to ship a release.

Action. The plane is a daemonless container engine driving units generated by the operating system’s own supervisor. There is no second process manager: containers are services, they start the way services start, and they are described in the same declarative form as everything else. The containers use host networking with no published ports at all, which removes the firewall‑bypass question rather than mitigating it. Units run at the system level rather than rootless, and that is recorded as a trade — it keeps the automation simple and normal, at the cost of the stricter isolation rootless would give, and the note says which way to move if container escape ever matters more than automation simplicity. Deployment was separated from provisioning: root sets up the deployment path once, and thereafter an unprivileged user redeploys by running one fixed script through a scoped rule that permits that command and no arguments. Build the image, restart the service. No root shell, no arbitrary commands.

Result. Containers run under the supervisor that was already trusted, behind the firewall that was already declared, and a routine release needs no privileged access. The choice against the default is written down with its reason, which matters more than the choice — the next person will be told to use the default by every article they read.

Automated the provisioning of a second server on a second cloud provider, creating the firewall before the machine so it is born behind one, with both providers' firewalls written directly against their REST APIs to avoid a third‑party collection.

2026-09-13T01:45:50+02:00

Situation. A second machine was needed for an application tier, at a different provider from the first, and the first machine’s own history was the argument for how to do it: it had been created by hand, and its firewall had been added afterwards, which leaves a window where a fresh host with a default password policy is reachable from the internet.

Task. The machine had to be created from the repository, and it had to be born behind its firewall rather than acquiring one shortly afterwards.

Action. Ordering carried the design. The firewall play runs before the server play, so the rule set exists before there is anything for it to protect; on the other provider, tagging runs before the cloud firewall because a firewall that targets tags needs the tags to exist first. Both providers’ firewalls are written directly against their REST APIs through a generic HTTP task rather than through a vendor collection, which removes a dependency and its version drift at the cost of writing the request shapes by hand. The role that creates the machine adopts an existing one rather than duplicating it if it is already there, so re‑running is safe. It is also documented, in its own file, as the one role in the repository that spends money — the instance type, the region, the specification and the monthly cost in euros are all written down, because a play that bills someone should say so where they will read it. That play is also the one that cannot be rehearsed the usual way: the generic HTTP task declares no check‑mode support, so a dry run skips every task in it. The rehearsal is done against the firewall play instead, which is free and reversible.

Result. A second host can be created, or re‑adopted, from the repository, behind a firewall that already exists, at a cost the file states. The one thing dry‑run cannot cover is named in the same file rather than left as a surprise.

Wrote a scope rule into the repository after a restructure carried another company's inventory, firewall allowances and prose into it — and kept the quarantined residue under the secret scanner rather than excluding it.

2026-09-13T01:45:50+02:00

Situation. Work done for a different company rode along through a repository restructure and stayed. What came with it was not abstract: a scratch log holding live plaintext credentials, which sat in the tree past every guard for most of the repository’s history; a stale inventory naming that company’s host; and a firewall task file opening ports to sixteen of its client networks. None of it ran. That is exactly why it survived — nothing that runs nowhere is ever reviewed again.

Task. The boundary had to become a rule with a test attached, rather than an intention, and the residue had to be dealt with in a way that did not simply hide it.

Action. The rule is now the opening section of the repository’s instruction set: this repository manages one company’s infrastructure and only that, and another party’s host, inventory, firewall allowance, DNS zone or credential does not belong in it — not even disabled, commented out, or parked in a file no playbook imports. The practical test is written next to it: would this company still be responsible for this if the relationship ended. The residue was quarantined into a clearly‑named directory rather than deleted, so the history stays legible, and the quarantine is deliberately partial — the two structural linters skip it, and the secret scanner, the vault guard and the emoji check deliberately still read it, because those are the three that would catch the thing that got in. The leak itself produced a linter rule: a custom pattern that flags a credential passed as a command‑line flag, which is the shape the leaked one had and which the standard rule set did not match.

Result. The scope boundary is a written rule with a stated test, the residue is visible rather than buried, and the specific shape of credential that got through now fails a commit. The lesson recorded alongside it is the general one: the dangerous artefact is not the one that runs, it is the one that does not, because that is the one nobody reads again.

Split the four per‑host operational secrets after establishing that two hosts sharing one dead‑man's switch alert less than two switches, not more, and that a shared backup passphrase makes two hosts one repository.

2026-09-13T01:45:50+02:00

Situation. The platform was built for one host and was about to have two. Several operational secrets had been written as single values on that assumption: one backup repository, one passphrase, one alert switch. Extending them to a second host by sharing them is the path of least resistance and it is wrong in two separate ways, both of which fail quietly.

Task. Each secret had to become a map keyed by host, and the reason had to be written down, because the shared version looks correct and costs nothing until the day it matters.

Action. Two arguments settled it, and both are recorded next to the configuration. A dead‑man’s switch shared by two hosts alerts less than two switches, not more: either host still pinging keeps the check green while the other one is dead, so adding a host to a shared switch actively reduces the coverage of the one that was already there. And a backup repository string is only a location — the passphrase is the whole of the encryption, so two hosts sharing a passphrase are not two repositories with a common secret, they are one repository with two directories in it. Each value became a map keyed by inventory hostname, resolved per host by a shared task file that both the converge and the preflight run. The existing host was then named in all four maps with its repository and both switches left empty, which is the truthful state and keeps the two open initiatives honest about owing the operator half of the work rather than the code.

Result. Secrets are per host by construction, and the two failure modes that would have followed from sharing them are written down where the next person edits the file. The empty entries are the part worth keeping: they say the wiring exists and the value does not, which is a different and more useful statement than a missing key.

Built a commit gate of 22 one‑line linters plus five that earn a paragraph, with no warning tier and no inline suppressions permitted, covering HTML, CSS, JavaScript, Python, YAML, Markdown, shell, links, spelling, secrets and typography.

2026-09-13T01:45:50+02:00

Situation. A contributing guide is a set of suggestions. Everyone agrees with it, and then it is Friday, the change is small, and the guide loses. On a one‑person project that is worse rather than better, because there is no reviewer at all — the only thing between a bad change and production is the person who wrote it, at the moment they are least inclined to argue with themselves.

Task. The standards had to be executable, so that breaking one fails a commit rather than waiting to be noticed.

Action. What grew is a gate of 22 linters with no warning tier: every diagnostic is an error, and the site generator itself runs with warnings promoted to failures so even a deprecation stops the build. The obvious ones are there — HTML validation, CSS, JavaScript, Python, YAML, Markdown, shell, spelling, secrets, dead links. The interesting ones are the project‑specific checks that no off‑the‑shelf tool has an opinion about: that both colour themes paint every layered surface with the same number of layers, that no photograph is displayed wider than half its source pixels, that the deployed tree contains no private path or hostname, that the prose obeys the tone rules in all three languages, that a page has a Markdown twin and a valid alternate‑protocol representation. Inline suppressions are prohibited outright — no ignore comment, no disable directive, no bypassing the hook, and no renaming a file to dodge a matcher. The rule that keeps that honest is that a pre‑existing failure is not an excuse: a check that surfaces a defect nobody introduced gets fixed in the same pass.

Result. Mechanical objections are made by a machine before a commit exists, and the gate is the only reviewer this project has. The cost is stated rather than hidden: committing is slow, and a badly written guard is genuinely infuriating to work around, which is why the guards themselves were later brought under a formatter and a linter of their own.

Cut the site's browser‑driven quality gate from 1,636 seconds to 615 by scheduling its checks longest‑first through a worker pool bounded to four lanes, after measuring that alphabetical order cost 320 seconds against 224.

2026-09-13T01:45:50+02:00

Situation. Eleven of the site’s checks drive a headless browser: layout at every window shape the design draws, type scale, translated‑text expansion, contrast, accessibility rules, forced colours, mascot sizing, motion, print across six paper combinations, console errors and visual regression. Run one at a time they took 1,636 seconds, a little over twenty‑seven minutes. A gate that takes twenty‑seven minutes is a gate that gets skipped, and a skipped check is indistinguishable from a passing one.

Task. The wall clock had to come down far enough that running them was the default rather than a decision, without weakening any of them.

Action. The work was measurement first. Each check was timed individually on a twelve‑core machine: layout at 405 seconds, type at 301, translation at 282, contrast at 189, accessibility at 178, and so on down to seventeen. Two findings shaped the answer. Running all of them at once was slower than running four at a time — 265 seconds against 224 — because each check is itself a browser doing parallel work, and oversubscribing the machine costs more than the concurrency wins. And ordering by longest processing time first beat alphabetical order by nearly a third, 224 seconds against 320, which is the classic scheduling result and shows up here because the checks vary by a factor of twenty in cost. So the runner is a bounded worker pool, sized from the core count with a floor of two and a ceiling of four, fed longest‑first. Alongside it the checks were widened rather than narrowed: they now share one viewport table of twenty‑two window shapes, derived from every media query the stylesheet actually contains, which took the layout check alone from 95 seconds to 405.

Result. The gate went from 1,636 seconds to 615, while the checks got broader rather than thinner — the serial cost went up and the wall clock came down. The measurement is the part worth keeping: two reasonable‑sounding choices, running everything at once and running things in the order they were written, were each measurably worse than the alternative.

Mirrored the entire site as 777 Gemini documents and 777 Gopher documents off the same deployed tree, at zero bytes of change to the HTML.

2026-09-13T01:45:50+02:00

Situation. The site is static, has no tracking and no runtime backend, and its argument is that a document does not need a megabyte of JavaScript to be read. That argument is easy to make and hard to demonstrate. Two small internet protocols demonstrate it directly, because neither of them can carry a script at all.

Task. The whole site had to be published over Gemini and Gopher from the same content, without a second content tree and without changing the HTML.

Action. Both are output formats of the same build rather than a separate pipeline. The site generator was given custom media types and output formats, and every page, section and taxonomy term gained two more representations alongside its HTML and its Markdown twin. The result is 777 Gemini documents and 777 Gopher menus produced from the same source, deployed to the same tree, and served by two small daemons on the same host. Two properties made it worth doing rather than a curiosity. Both formats are marked as non‑alternative representations, so nothing about the HTML changed — not one byte, and that was measured rather than assumed. And the Gopher format has no way to put a link inside a sentence, since a menu line is tab‑separated fields, so the prose is hard‑wrapped at sixty‑eight columns during the build; that constraint sharpened the writing in a way the HTML never demanded. A Tor onion mirror sits alongside them, and it is the only public face the platform could add without opening a firewall port, because the daemon dials out and nothing dials in.

Result. The site is readable over four protocols from one build, at 777 documents each and zero HTML change, and the alternate representations cost about twelve per cent of the HTML’s own weight on disk. Only one of the four is measured for traffic, and that is stated in the platform’s own statistics document rather than being quietly ignored.

Fixed a sitemap where 172 of 176 URLs shared one modification timestamp, by taking the date from git history after establishing that the export rewrites every file on every run.

2026-09-13T01:45:50+02:00

Situation. The site’s sitemap told search engines that 172 of its 176 pages had last changed at the same instant. That is not a subtle inaccuracy — a modification date is a promise to a crawler that a page’s content changed, and a site making that promise 172 times simultaneously is either telling the truth about a full rewrite or telling nobody anything useful.

Task. The dates had to describe the content rather than the file, without inventing precision the repository does not have.

Action. The cause was that the date came from the file’s modification time and the content export rewrites every file on every run, so a single sync stamped the whole corpus. The fix moves the source to the version history, with a fallback chain that tries an explicit field first, then the commit history, then the file. The trap in that fix is worth recording: the literal field name has to appear in the list or the generator never reads it, so a configuration that looks like it prefers an authored date but omits the name silently ignores every authored date. Two other date decisions came out of the same work. The database now carries when each record was written and last revised, kept deliberately separate from when the work happened — those are years apart and conflating them would date a page written this year to a decade ago. And rows in that file are optional and nothing is derived, because the repository’s own history begins after the content did: a missing date leaves the field empty rather than recording a migration, on the principle that a precise wrong number is worse than an absent one, since only the wrong one gets believed.

Result. A re‑sync after a month of content edits now touches eight files of 186 instead of all of them, and the sitemap says something true. The general rule went into the metadata document alongside it, because the same trap applies to every field that has a fallback chain.

Brought 10,242 lines of quality‑gate JavaScript under a formatter and a linter after establishing it was the largest body of code in the repository and the only one nothing read, fixing 13 findings and suppressing none.

2026-09-13T01:45:50+02:00

Situation. The site’s quality gate is thirty‑two scripts totalling 10,242 lines of JavaScript. It was, by a wide margin, the largest body of code in the repository, and it was the only body of code nothing read — no formatter, no linter, no type checking. The programs enforcing every rule in the project were the only programs subject to none of them.

Task. The checkers had to be held to the standard they exist to enforce, and the formatter had to be fitted to them rather than the other way around.

Action. The formatter came first, and the indentation width was the interesting decision. The repository’s default is four spaces; at four spaces the formatter would have rewritten 2,173 lines of the largest checker. An override to two spaces for those files reduced that to 38 lines of genuine drift. The principle recorded with it is that the formatter is fitted to the code, not the code to the formatter — reformatting two thousand lines to satisfy a preference destroys the ability to read the history of the file. Then the linter, which produced thirteen real findings, all fixed and none suppressed. The Python checkers got the same treatment through a type checker configured at a middle strictness with thirty individual strict‑tier rules enabled on top, chosen by measurement: at that setting the tree is silent, and of the thirty candidates twenty‑nine were already silent and one fired — and that one was fixed rather than exempted. The type checker found two real defects the linter had passed clean, both about a value’s shape rather than its syntax.

Result. The largest and most load‑bearing code in the repository is now formatted, linted and type‑checked, with thirteen findings fixed and zero suppressions. The reason it matters more than the line count suggests is stated in the plan: a defect in the site’s stylesheet shows up as a page that looks wrong, and a defect in a checker shows up as a check that passes when it should not.

Held the generator to 981 test cases at a 92% branch‑coverage floor with warnings treated as failures, and asserted idempotence by running the whole build pipeline twice from an empty file and requiring the second pass to change nothing.

2026-09-13T01:45:50+02:00

Situation. A generator that assembles a database from scratch has a particular kind of bug: it works the first time and corrupts the second. Steps that insert without checking, steps that depend on the order of a previous step’s output, steps that are safe alone and not together. None of that shows up in a test that starts from nothing and runs once.

Task. The build had to be proved repeatable rather than merely working, and the test suite had to be large enough and strict enough that a regression could not pass through it quietly.

Action. The idempotence test is the blunt one and the most useful: build the entire database from an empty file, snapshot it, run the whole nineteen‑step pipeline again over the result, and require the second pass to change nothing. Row counts, contents and identifiers all have to match. Around it sit 383 test functions – 981 cases once the parameterised ones expand – across forty‑five files and 7,944 lines, covering the pipeline, the renderers, the content loaders, the targeting logic and the checkers. Branch coverage carries a floor of ninety‑two percent enforced in the build rather than reported in a summary, and the suite currently measures about 95 percent, so the floor has headroom without being decorative. Warnings are configured as failures, which is the setting that matters most in practice — a deprecation notice that prints for two years is a deprecation notice nobody reads, and the run that turns it into a red test is the run that gets it fixed.

Result. The pipeline can be re‑run against a live database without fear, which is what makes incremental content work possible at all. The floor is a floor, not a target, and it is worth saying that ninety‑two percent branch coverage still leaves branches nothing has ever taken — the number bounds the risk, it does not remove it, and two of the defects found later in the project were in code the coverage report showed as covered.

Selected every rule the Python linter has as an error, working through 1,815 findings to reach zero, with each of the few exemptions carrying a written reason and two of them backed by a checker instead of a comment.

2026-09-13T01:45:50+02:00

Situation. Most projects pick a comfortable subset of their linter’s rules, and the subset is chosen by whichever rules were quiet on the day it was configured. That makes the configuration a record of the code’s existing habits rather than a standard the code is held to, and every rule left off is a class of defect nobody will ever be told about.

Task. The default had to be inverted — every rule the tool implements enabled as an error — and the resulting backlog worked to zero rather than negotiated down by turning rules back off.

Action. Selecting the complete rule set produced 1,815 findings on first run. They were worked through by category rather than by file, because the categories tell you something: unused arguments and shadowed builtins are noise, but the security category, the mutable‑default category and the exception‑handling category each pointed at real behaviour. Genuine incompatibilities exist — a formatter and a linter can disagree about the same line, and a few rules contradict the project’s own deliberate choices — and each of the small number of exemptions carries a written reason at the point of exemption saying what the rule wanted and why this code does otherwise. Two of them go further and are backed by a check rather than a comment, so the exemption cannot quietly widen: the rule is off, and a test asserts the specific property the rule would have enforced. Type checking runs in strict mode alongside it, which is a separate and harder standard, and it is the one that caught defects the linter could not see because they are about what a value is rather than how it is written.

Result. The linter runs at full strength with zero findings, and every deviation is documented at the line where it is taken. The cost is real and worth naming: the strictest setting produces findings that are genuinely not worth acting on, and someone has to make that judgement 1,815 times rather than once.

Wrote tests for the checkers themselves after establishing that a checker fed only clean input will one day report clean because it read nothing — planting a misspelling to confirm the spell‑check finds it, and taking an id range from the database rather than from a number in the test.

2026-09-13T01:45:50+02:00

Situation. The project runs a set of custom checkers over its own content — a spell check, an identifier‑range check, a voice check, a figure‑consistency check. Each of them had run green for months. A checker that has only ever seen clean input and only ever reported clean is indistinguishable from a checker that reads nothing at all, and there was no test in the suite that could tell the two apart.

Task. The checkers had to be made to prove they can fail, and the fixtures they check against had to stop being hand‑maintained copies of the thing they describe.

Action. The pattern applied throughout is to plant the defect the checker exists to find and require the checker to find it. The spell check is fed a deliberately misspelled word and the test fails if the run comes back clean. The voice check is fed prose in the first person and must reject it. The buzzword check is fed a banned word. Each of these is a small test and each one closed a real blind spot, because two of the checkers turned out to be reading a narrower set of files than their documentation claimed and had been silently skipping content. The second change is about where a test gets its expectations: the identifier‑range check previously compared against a number written in the test file, which meant every content addition required editing a test, and an editor who updated the number without looking had disabled the check. It now derives the range from the database, so the check describes the data rather than a stale memory of it.

Result. Every checker in the project now has a test that proves it fails on bad input, and the range assertions read from the source of truth. The uncomfortable part is what this exposed: a green check had been meaningless in at least two places for an unknown length of time, and there is no way to find out retroactively what passed through.

Found the commit hooks and the quality gate running different checks while a document promised they were the same, by comparing the two lists in a test — the five that only ever ran by hand were the ones reading the CV prose.

2026-09-13T01:45:50+02:00

Situation. The project had a commit hook that runs checks before a commit is accepted, and a full quality gate run on demand. A document stated that the hook runs the gate, so nothing could reach history without passing everything. Both lists were maintained by hand, in two different files, and nothing compared them.

Task. The claim had to be turned into an assertion, which meant enumerating both sets programmatically and failing when they diverge.

Action. The test reads the hook configuration and the gate’s task definitions, resolves each to the set of checks it actually invokes, and compares. They did not match. Five checks existed only in the gate and never ran on commit, and the five were not random — they were the ones that read the CV prose itself: the voice check that keeps reviews impersonal, the buzzword check, the figure‑consistency check across languages, the notation check, and the spell check over content. In other words, every check protecting code ran automatically and every check protecting the writing ran only when someone remembered. Given that the writing is the entire product, the exposure was inverted from where anyone would have guessed. The fix was to bring the five into the hook, which required making two of them fast enough to survive a pre‑commit budget, and then to keep the comparison test in place so the two lists cannot drift apart again. The document that had been describing an intention now describes something enforced.

Result. The hook and the gate provably run the same checks, and the prose checks now run on every commit. The trade‑off is the commit hook’s runtime, which grew and will keep growing as content grows, and there is a point at which a slow hook gets bypassed — so this fix has a shelf life measured in how long the checks stay fast.

Rehearsed the forge‑side CI hook and found two defects unreachable by reading the file: a fallback that put an unresolvable argument on the hook's input, and git's own environment variable following the gate into the checkout and turning 19 tests red.

2026-09-13T01:45:50+02:00

Situation. The self‑hosted git forge accepts pushes and can run a hook on the server side to reject work that does not pass the gate. A server‑side hook is the one piece of automation that cannot be tested by running it locally: it executes in a bare repository, with no working tree, under an environment the forge sets, reading the pushed references from its standard input. Reading the script and concluding it is correct is a guess.

Task. The hook had to be rehearsed against a real bare repository and a real push before being trusted, rather than deployed and discovered.

Action. The rehearsal creates a bare repository, installs the hook, clones it, commits, pushes, and asserts on both the accept and the reject path. It found two defects, neither of which was visible in the file. The first was a fallback in the argument handling: when the hook could not determine a reference it substituted a placeholder that was not a resolvable object, and because the value arrives on the hook’s standard input rather than as a parameter, the failure surfaced as an unrelated error much further down. The second is the one worth remembering. Git exports an environment variable naming the repository directory when it invokes a hook, and that variable is inherited by everything the hook runs. The gate checks out the pushed revision and runs the test suite inside it, and the test suite inherited the pointer to the bare repository — so nineteen tests that touch git resolved against the wrong repository and went red, on code that was correct. The environment has to be cleared at the boundary, and that boundary is invisible unless the thing is actually run.

Result. The hook works on both paths and both defects were found before any real push met them. The limit is that a rehearsal is still a simulation: it proves the hook survives one push shape, and the forge in production sees pushes this rehearsal does not construct, including forced updates and branch deletions.

Stopped an application filling memory at 41 MB a second — a recorded 111 GB of compressed pages on a 36 GB machine — by bounding every event stream, subscribing by event type and putting a rate budget on logging, taking 610,996 log lines down to 1,411.

2026-09-13T01:45:50+02:00

Situation. The application filled memory until the operating system killed it. The recorded incident reached 111 GB of compressed pages on a 36 GB machine, climbing at roughly 41 MB a second, and the log file for a single short session held 610,996 lines. A machine in that state is not slow, it is unusable — the kill arrives after the swap has already made everything else on the desktop stop responding.

Task. The growth had to be found rather than guessed at, and every unbounded path had to be given a limit, because one bounded queue next to three unbounded ones is not a fix.

Action. There were three multiplying causes and the multiplication is why it was so fast. The event stream from the core was consumed without any bound, so events arrived faster than the interface could apply them and the backlog was retained rather than dropped. Every subscriber received every event and filtered afterwards, so the cost of one event was multiplied by the number of listeners, and each listener’s filtering allocated. And logging was unbudgeted, so each event produced log lines — which is the compounding term, because the volume of logging was proportional to the volume of the thing going wrong. The fix addressed all three: bounded buffers with an explicit policy for what happens when they fill, subscription by event type so a listener is only woken for events it wants, and a rate budget on logging that collapses repeats rather than writing each one. A regression test drives a high event rate and asserts the memory ceiling holds, so the bound is a property of the build rather than a comment.

Result. Memory stays flat under sustained load and the same session that produced 610,996 log lines produces 1,411. The honest note is that the rate budget on logging discards information: when something goes wrong quickly now, the record of it is deliberately incomplete, and that is a trade made knowingly against the alternative of a machine that stops.

Wrote a parser that reads the real 7,308‑line C header and verifies every call site, every enum constant and that every pointer‑owning class is final, after a hand‑written placeholder header let calls to three removed functions compile, link and crash.

2026-09-13T01:45:50+02:00

Situation. Early in the project the C interface was represented by a hand‑written header describing the functions the application expected. That header compiled, the application linked, and calls to three functions that no longer existed in the core reached the point of being called and crashed. The compiler and the linker had both been satisfied by a description of the library rather than the library, and the gap only appeared at runtime.

Task. The real header — 7,308 lines and 268 declarations — had to become the authority, and every use of it in the Swift code had to be verified against it automatically rather than by review.

Action. The check is a parser that reads the actual upstream header and builds the set of functions, enum constants and types it declares, then reads the Swift source and resolves every call site and every constant reference against that set. A call to a function the header does not declare fails the build. A reference to an enum constant that has been renamed fails the build. The current count is 132 of the 268 declarations referenced, and knowing which 136 are unused is itself useful, because it says exactly how much of the core the client has not reached. The parser also enforces a rule the compiler cannot: every Swift class that owns a pointer into the core must be final. A non‑final pointer‑owning class can be subclassed, and a subclass that overrides deinitialisation or adds its own lifetime changes when the pointer is freed — a use‑after‑free with no unsafe keyword anywhere near it. The rule is checked by name across the whole tree.

Result. The class of defect that produced the original crash cannot recur, because a stale reference is now a build failure rather than a runtime one. The limitation is that the parser understands the header’s declarations and not its semantics: it proves a function exists with a matching name, and a function whose meaning or ownership rule changed upstream while keeping its signature passes without comment.

Took the test suite from six tests over a sixty‑second limit to 135 passing in 8.9 seconds by profiling the main thread and removing the two calls it sat inside for 3,989 samples out of 4,017.

2026-09-13T01:45:50+02:00

Situation. The test suite had a sixty‑second time limit and six tests were over it. A test suite that takes over a minute stops being run before every change, and a suite that is not run before every change is a report on the past. The instinct in this situation is to raise the limit, and raising the limit is how a suite gets to ten minutes.

Task. The time had to be found rather than budgeted for, which meant profiling the suite instead of reasoning about which tests looked expensive.

Action. The profile was taken on the main thread while the suite ran, and the result contradicted the guess. The main thread sat inside two calls for 3,989 samples out of 4,017 – so the slow tests were not the ones doing the most work, and almost every test passed through the same two places. The first was a fixed wait used to let asynchronous work settle before asserting — a sleep, effectively, paid by every test that touched the event path whether or not the work had already finished. It was replaced by waiting on the actual condition with a timeout, so a test that is ready in five milliseconds takes five milliseconds and only a genuinely stuck test pays the full wait. The second was per‑test setup that rebuilt an expensive fixture each time, where the fixture was read‑only and could be built once for the suite. Neither was in a test anyone would have nominated as slow; both were in the shared path, which is why the whole suite was uniformly slow rather than a few tests being outliers.

Result. The suite went from six tests over sixty seconds – 74 seconds of wall clock – to 135 tests passing in 8.9, which puts it inside the window where it runs on every save. The caveat is that the shared read‑only fixture is now a coupling point — a test that mutates it will produce a failure in a different test, and the suite’s speed depends on a discipline the compiler does not enforce.

Reached the half of the messaging core the application had never used — backup transfer, disappearing messages, message editing and resending, verified invitations, proxies and encryption policy — driving every test against the real library with no mocks.

2026-09-13T01:45:50+02:00

Situation. The client used roughly half of what the messaging core offers. The unused half was not obscure — backup transfer between devices, disappearing messages, message editing and resending, verified invitation links, proxy configuration and the encryption policy for a chat. Each is a feature a user would expect and each was an untested region of the C interface, which is the more dangerous fact, because an untested region of a C interface is where the ownership mistakes live.

Task. The unreached capability had to be driven and covered, and the tests had to run against the real library rather than a stand‑in.

Action. The rule adopted was no mocks for the core. A mock of a C interface encodes the developer’s belief about what the library does, and every defect worth finding here is a place where that belief is wrong — so a passing mock‑based test is evidence about the mock. Instead the tests create real accounts in temporary directories, drive the real library, and assert on what it actually returns, with each suite cleaning up its own state. That is what made the coverage meaningful: exercising backup transfer meant handling a real transfer’s state machine and its failure paths, and exercising verified invitations meant constructing the real link format and having the library parse it. Coverage floors were set per layer rather than as one number, at eighty‑four percent for the core wrapper, seventy‑five for utilities and sixty‑five for models, on the reasoning that the layer touching raw pointers should be held highest and a model that is mostly stored properties should not be padded with tests to hit an average.

Result. The previously unreached half of the core is driven by tests against the real library, and the wrapper carries the highest floor in the project. The cost is speed and determinism: real‑library tests are slower than mocks and they can fail for environmental reasons, which is the price of them being able to fail for real ones.

Audited 644 Rust crates for licence compatibility on every build, and proved the check fires by rewriting one crate's licence and by moving the pinned core revision without regenerating.

2026-09-13T01:45:50+02:00

Situation. Linking a Rust core into a shipped application means shipping everything that core depends on. The dependency graph is 644 crates. Each carries a licence, some carry more than one, and a single copyleft crate arriving three levels down through a routine version bump changes what the application as a whole may be distributed under — silently, in a lock file nobody reads line by line.

Task. The licence position had to be verified on every build rather than reviewed once, with an explicit allow list, so that a change in the graph is a build failure and not a discovery made later by someone else.

Action. The audit resolves the full transitive graph and checks each crate’s licence expression against a list of terms the project accepts, evaluating the boolean expressions properly — a crate offering a choice of two licences is acceptable if either is on the list, and a crate requiring both is only acceptable if both are. Anything unmatched fails the build rather than warning, and adding a term to the allow list is a deliberate edit with a reason. A check that has never failed is indistinguishable from one that cannot fail, so it was made to fail on purpose, twice. Once by rewriting a crate’s licence expression to something the list does not accept, which is the shape of a crate changing its terms upstream between versions – the case no human review catches, because the crate itself is not new. Once by moving the pinned core revision without regenerating the audit, which is the shape of a core bump quietly bringing a new dependency with it. Both provocations failed the build as intended, and the check was left in place rather than widened.

Result. The distribution position is checked on every build, and the check is known to work because it was made to fail rather than because it has never spoken. Before it existed, the bundle, the licence file and the project’s own README all made a claim about 644 crates that nothing had verified. The limit is precise and should not be overstated: the check reads the declared licence metadata, and metadata can be wrong or incomplete. It proves nothing about a crate that misdeclares itself, and it is not legal advice.

Held four repositories to one history standard — conventional, emoji‑free, no attribution trailers, enforced by a commit‑message hook — alongside 46 instruction documents that govern how the work is done.

2026-09-13T01:45:50+02:00

Situation. Five repositories built over eighteen months by one person is the situation where process is easiest to skip, because there is nobody to coordinate with and the cost of an unreadable history is paid entirely by a future self who has not complained yet. It is also the situation where an inconsistent history is most likely, since each repository can drift into its own habits with nothing pulling them together.

Task. One standard for history and one standard for instructions had to apply across every repository the work is authored in, and it had to be enforced mechanically rather than remembered.

Action. The commit standard is a conventional prefix naming the kind of change and a scope, a subject under a fixed length, no emojis, and no attribution trailers of any sort — the last of these because a trailer crediting a tool is not a fact about the change, and history is for facts about changes. A commit‑message hook rejects anything that does not conform, in every repository that carries work, so the standard is a property of the repository rather than of whoever is committing. The distribution on the largest repository shows what the work actually was: 156 feature commits, 133 fixes, 128 documentation, 81 chores, 26 refactors, 20 style, 5 performance and 3 test. Documentation is close enough to fixes to be worth noticing, and that is a consequence of the second half of this — 46 instruction documents across the four, each covering one domain, each written as rules rather than description, all reachable from a single entry document per repository so there is one place to start. A documentation linter enforces per‑file size budgets and index membership, so the set stays navigable rather than growing into an archive.

Result. Five repositories share one history format and one instruction structure, both enforced by hooks rather than by discipline. What this does not do is make the history good: the format is checked and the content is not, so a conforming subject line that describes nothing passes exactly as well as one that explains the change.

Opening a dApp inside Solana mobile wallets

2026-09-13T01:45:50+02:00

A React dApp built on @solana/wallet-adapter-react connects desktop wallets without trouble, but on a phone the same flow falls apart: the wallet has to open the dApp inside its own in-app browser, and the deep links that should make that happen quietly do not. Backpack lands on a “download the app” page; Solflare opens the app but never the site; every variant seems to fail. We took the problem apart, and it turned out to be four separate problems wearing one symptom.

The four problems

The Backpack link was malformed. The only documented format is https://backpack.app/ul/v1/browse/<url>?ref=<ref> — a universal link with the target URL in the path and a required ref. A custom-scheme guess like backpack://ul/v1/browse?url=... matches no route the app registers, so the user ends on the wallet’s install page.
Solflare needs its universal link too: https://solflare.com/ul/v1/browse/<url>?ref=<ref>, not the bare solflare:// scheme. A bare scheme can launch the app without routing it — which is exactly “the app opens, but the site tab has to be opened by hand”.
Both parameters must be encoded. url is the full absolute dApp address and ref is the requesting origin, each passed through encodeURIComponent. An unencoded ? or & in the target corrupts the parse, and the wallet opens on its home screen instead of the browser tab.
The trigger matters as much as the link. Universal links only switch apps on a navigation the operating system trusts — and they deliberately do nothing when pasted into the address bar, which is also how a perfectly correct link “fails” during testing.

The documented formats

Phantom: https://phantom.app/ul/browse/<url>?ref=<ref> — no /v1 in this one.
Solflare: https://solflare.com/ul/v1/browse/<url>?ref=<ref>
Backpack: https://backpack.app/ul/v1/browse/<url>?ref=<ref>

One pattern serves all three:

const WALLET_BROWSE = {
  phantom: (url, ref) =>
    `https://phantom.app/ul/browse/${url}?ref=${ref}`,
  solflare: (url, ref) =>
    `https://solflare.com/ul/v1/browse/${url}?ref=${ref}`,
  backpack: (url, ref) =>
    `https://backpack.app/ul/v1/browse/${url}?ref=${ref}`,
};

function walletBrowseLink(
  walletName,
  targetUrl = window.location.href,
) {
  const build = WALLET_BROWSE[walletName.toLowerCase()];
  if (!build) return null;
  return build(
    encodeURIComponent(targetUrl),
    encodeURIComponent(window.location.origin),
  );
}

Triggering the link so iOS and Android accept it

Render a real anchor, precomputed. A plain <a href={walletBrowseLink('phantom')}> is the most reliable trigger on both platforms.
If it must be programmatic, assign window.location.href synchronously inside the tap handler — no await, no fetch, no setTimeout first. After asynchronous work the gesture context is gone, and iOS falls back to the wallet’s website. Never window.open.
Never test by pasting into the address bar. Universal links deliberately do not fire there; test with a tapped link or a QR code scanned by the camera.
Mind the messenger webviews. Opened inside Telegram’s or Instagram’s in-app browser, universal links are frequently swallowed and the wallet’s plain website loads instead. User-agent detection is heuristic at best, so also give users a visible escape hatch: “open in Safari or Chrome, then connect”.

The bigger fix on Android

Hand-rolled deep links are the iOS story. On Android, Solana Mobile’s Mobile Wallet Adapter lets a dApp running in the mobile browser connect straight to the installed wallet app, with no in-app-browser detour at all. Recent versions of @solana/wallet-adapter-react register the mobile adapter automatically, so upgrading the wallet-adapter packages can fix Android by itself. The target architecture: Mobile Wallet Adapter on Android, browse universal links on iOS, where Apple allows no equivalent.

Verifying on a device

Real device, wallet installed, link opened from the system browser — not from a messenger.
Tap a rendered link or scan a QR code; never paste into the address bar.
Confirm the wallet opens and the dApp loads in its in-app browser tab — the second half is the part that fails.
Repeat without the wallet installed: the universal link should degrade to the wallet’s website. That page appearing while the app is installed means the link or the trigger is still wrong.
Then test the messenger path, and add the “open in browser” hint if it fails there.