Orchestration and containers

The backend of Cropman went through 3 platforms before it settled: Heroku first, then Azure Web Apps, then Google App Engine. On Heroku a free machine goes to sleep, and the next batch waits for it to wake up. Google App Engine is where the running cost fell, because a service that drops to its minimum while idle and scales up only when a batch comes in costs very little.

Most of the machine time went into editing a client's batch, so the edits ran inside a pool of worker threads (that's also where the image work moved from jimp to sharp). The pool had 4 kinds of worker, each with its own queue: download, edit, upload, and building the preview sheet. Each kind always kept at least 1 worker and could never go past 100. The pool grew with the batch, and an extra worker ended itself once its queue was empty. The machines scaled the same way, 1 small machine at rest and 2 at full load, with the second one starting when processor use went above 60 percent. So at rest we paid for 1 small machine, and around 100 paying subscribers covered the running cost. Btw, I once ran 10 workers in parallel and the database didn't even notice.

The hard move at Amerix was the CRM, and it came with one condition: it must not stop. First the hosting went behind Cloudflare, so traffic that used to go straight to the old host now came through Cloudflare, which made the origin replaceable later. Then I built the new CRM separately in Kubernetes from a database snapshot, and put everything that could go into a container into one. The old CRM stayed live while the two databases ran in sync, and for that whole period I made sure fresh data kept moving across from old to new. The last step was switching the route inside Cloudflare to the Kubernetes instance, and that switch was the entire cutover. All the data had moved before it, so nobody had to stop working. The CRM moved with zero downtime, and it's faster on the new platform than it was on the old host.

The CRM went onto the cluster as a single Helm release. One definition describes every part it needs, sized for the load of that business, with each part isolated from anything it has no need to reach. On install it even prints its own operator notes: which default passwords to replace, and a reminder to set up backups of the stored data. That one Kubernetes cluster holds the whole estate: the CRM, the databases, the object store, the translation server (so there's no subscription to pay for it), the new site with its content backend, and the old WordPress site, which kept selling tours until the new one took over. Hosting went from about $250 a month to under $100, with more resources than before.

At The Parker Avery Group my first months, in 2022, went into foundations: the delivery pipeline for the platform, the Kubernetes cluster under it, and the federated GraphQL gateway the application queries through. The pipeline builds and deploys every change the same way each time. None of it is anything a user ever sees.

A schedule can be a written promise of the product. On the faculty site a scheduled task makes the vote records anonymous 90 days after a campaign ends, and that 90 is a constant in the code. The faculty's privacy policy sets 12 months as the outer bound, so the code stays well inside it.