Amal Zacharia All Chapters

Chapter 3 of 4

The Year Everything Accelerated (2020)

In 2020 I stopped experimenting and started building for real.

The first thing I did was port the esports tournament cogs from Python 2 to Red bot v3. Small but important: first time I took existing code, understood it deeply enough to migrate it, and made it better rather than just running it.

Then a chain of events started that changed everything.

A partner from the esports days knew someone building a new Discord community and looking for a developer. He asked me. I declined: I wasn’t confident enough yet. A month later my partner came back and told me that guy had hit 6,000 members in a month.

I reached out myself.

He asked what I could do. I told him I could edit HTML. That was it. He gave me two template edits and a Discord OAuth2 login: show the username and profile picture, nothing more. I built an Express server with EJS, MongoDB for session management and Passport for the OAuth flow. Read the docs. Googled what I didn’t know. Got it working.

Then he gave me the project that changed everything.

He needed a Discord bot: essentially a control panel to manage and monitor an external service, with commands to start, stop and check status. Restricted to a private server. I built it in a day in JavaScript with MongoDB.

A week later we needed to scale. The single VPS approach: one bot, one database, one web server per deployment: wasn’t going to hold. I learned React without knowing CSS and built a landing page alongside the scaling work. Things were moving faster than the infrastructure could keep up.

I asked a friend managing a public Discord bot across 200,000+ servers for advice. He asked if I wanted to try Kubernetes. I didn’t know what Kubernetes was. I asked if it was like a database.

I tried DigitalOcean’s managed Kubernetes. Too expensive. Too confusing. My friend pointed me to a vendor running shared managed RKE clusters with proper isolation. They helped me write Dockerfiles and set up a GitHub Actions pipeline: build the TypeScript codebase, push the image to GitHub Container Registry, deploy manually via kubectl or Rancher UI depending on the situation.

The architecture we landed on was deliberately simple. A MongoDB replica set: two data nodes and one arbiter: managed by the vendor. A single replica set serving multiple clients, each using a separate database name for isolation. nginx ingress handling routing based on host headers, with Cloudflare in front and a self-signed certificate between Cloudflare and the nginx origin. Bots running as individual deployments.

The Rancher dashboard handled most configuration through UI. Raw YAML only when Rancher couldn’t expose the specific field we needed.

During the migration I rewrote the entire JavaScript codebase in TypeScript from scratch. Not a gradual migration: a full rewrite. The primary motivation was type safety. At the scale we were operating, JavaScript’s looseness was becoming a liability.

The transition wasn’t without mistakes. During the move I accidentally exposed MongoDB to the public internet with no authentication. I didn’t know automated scanners existed: tools constantly probing for exactly this misconfiguration. Fortunately I had a backup. The data contained encrypted secrets so the exposure risk was limited. I revoked everything, rotated credentials, locked it down.

It was the kind of mistake you make exactly once. Every MongoDB instance since has been cluster-internal only.

I set up daily mongodump backups packaged in Docker, running as a Kubernetes CronJob, uploading to Wasabi cloud storage via AWS CLI. Three month retention then automatic removal. Automated before I knew the word DevOps.

By end of 2020 I had also built giveaway bots for two additional clients in TypeScript and a timer website in React. Four to five production deployments running simultaneously on the same cluster.

Running production Kubernetes workloads. Managing Cloudflare, nginx ingress, MongoDB replica sets, CI/CD pipelines.

None of it came from a course. All of it came from a problem that needed solving.

Available
Senior platform / SRE roles. Remote.
contact@crza.dev
Share
X / Twitter
DISCUSSION