Five Software Best Practices I'm Not Following

It's 5PM on a Friday. The kids have soccer practice in an hour, the fridge is empty, and the groceries aren't going to buy themselves. Laundry is piling up, and there's only so many calls you can get from HR about how pants are required working attire. So you pack your laptop and get ready to head out and begin your weekend, briefly changing your life's flavor of insanity.

You put your hand on the door to freedom just as your manager comes running for you. Everything is going to shit. Dashboards are full of red icons, web servers are throwing errors like they're training for the MLB, and after being up for 48 hours straight, you see the lead SRE walking to their car with a resignation in the form of two middle fingers.

Despite no developer wanting to live their life like this, many have at least once in their career. And in making sure to keep shitshows in their past, they end up developing a collective wisdom from their battle scars. I've developed some wisdom over almost a decade of professional development experience, and I have collected far more from my colleagues along the way.

But now for the first time in my life I am working on a personal project that is throwing half this wisdom out the window. It turns out a passion project for fulfillment suits a different purpose than an enterprise project designed to turn a profit. This creates different incentives for each codebase, and the hobby incentives invalidate many of the development practices that keep the enterprise incentives sustainable.

So let's talk about rules, and let's talk about breaking them.

1. Make changes on branches instead of main

It's hard getting multiple people to work on the same codebase. While Sam is updating the code that draws to the screen, Jenny is updating the data models that Sam's code reads to put graphics on the monitor. Both developers end up having to modify the same files simultaneously, and there needs to be a way for these changes to not overwrite each other. The obvious solution for the two developers is trial by combat, where the winner earns the right to add their changes, and the loser has bigger issues to worry about.

Unfortunately we live in a civilized society, so we have to settle for version control.

With traditional version control, there is a code repository with a main version of the code. Developers take this code, and make their changes in ephemeral versions (usually called branches) based off this main version. These branches get merged together, a process that can involve human review and messy cleanup, but is still better than two developers overwriting each other's work on the same set of files. Out of this tradition comes a hard rule: never make changes to the main codebase.

However, this maxim comes from the assumption that there are multiple people working on the same code. Right now, I'm the only maintainer for Web Drones. That means there's nobody else to make changes concurrently with me, which means there's nobody for me to make conflicting changes with, and therefore nobody to challenge in trial by combat. Developing on a branch off the main version of the project and merging it back in ends up being the exact same as directly writing my changes to main.

There have been a few times where I wrote a particularly big change off a branch, but I consider this a process smell. I know myself, and I know if I start multiple things at the same time, none of them are going to get done.

2. Work with what you know, and add new tech slowly

Everyone knows that guy. You know, the one guy that jokes about about using Arch, except he's only pretending to joke after experiencing the wraith of disdain from people that know what sunlight looks like. Let's call him Greg.

As annoying as Greg may be, he's also insanely smart. Because he's willing to work with the latest and greatest, he knows an insane amount of stuff. And in a field where the only constant is change, Greg ends up knowing some of the best tech for whatever problem is at hand, and can use it to whip up some of the most elegant solutions to whatever is thrown his way.

And many have dealt with Greg applying the latest and greatest to a project, and wanted to smack their head on a table until the pain in their skull rang over their internal screaming. Greg introduced a nightmare of defects and completely destroyed anyone else's ability to make changes to the codebase. What happened?

There's two issues at play. One is that software only works after it's been tested, and the best way to test something is to have a wide range of end users using a product and running into its edge cases. This isn't to say QA is unnecessary, but it's a matter of numbers that there should always be more end users than there are testers, and the longer a codebase has been around, the more end users it has to run into more edge cases. As a consequence, new software has less time for people to run into bugs (which devs then fix). This means if you're running the latest and greatest, you are going to be doing a lot more volunteer QA work than shipping your code.

The other issue is that humans are creatures of habit, and work both quickly and predictably with what they already know. There's a massive cost and risk when changing a programmer's workflow, and that cost has to be mitigated when trying to keep up to date on new tech and trends.

In practice, the easiest way to strike a balance between stability and obsolescence is to add new technology slowly to a project, which maintains a level of consistency to the work at hand.

So what happens when something fundamental to the infrastructure of your project, like say, your programming language starts accumulating warts, tech debt, and whatever is left of your sanity? How do you add something as fundamental as a programming language in pieces without generating some ffi/serialized ipc nightmare? Maybe you're learning something super new with little to no battle testing. How long is it going to take to hunt down a weird v0.0.1 compiler bug from a polymorphic macro monad machination that was a lot more fun to think up than to implement? There's no good answer to these questions, and you have to accept you're going to fail frequently while learning and developing with the latest and greatest.

Fortunately, there's places to learn large and complicated pieces of tech where there's no risk with failure. And that's why I decided to learn on a server that is hosting a game. In a similar way Greg will never have a paying customer to sue him when his Arch build crumples under the weight of its own updates, if someone runs into a problem with Web Drones, they're not going to lose more than a few hours of access to a game.

I've been channeling my inner Greg and diving into new technology I know nothing about, enjoying its benefits and suffering through its consequences. Having spent my entire working software career writing Python I've developed increasing frustration from its almost thirty five years of accumulated technical debt. So I switched over to Go. No hello world tutorials, no twenty line projects to learn how a goroutine works, just a quick search on how how to write an OpenAPI spec, a second search on how to run a code generator on that spec, and a world of rabbit holes to follow. If I had a deadline trying to work like this, I would have burnt out, quit, and spent a month feeling sorry for myself in some delusion that I'd be better off working on a farm. But with no risk and no stress, this was easily one of the most rewarding ways I've learned something as large as a programming language. In hindsight, I probably should have taken the opportunity to go even nuttier and switch over to something like Elixr, Clojure, or maybe even Haskell. There's always next time.

3. Strive for an automated test suite with 70% code coverage

Have you ever asked a developer how they know their code works? If you get lucky, the dev will tell you their code has been rigorously proven correct with formal verification software like TLA. However, most will tell you they tested it. That being said, tests are repetitive, tedious, and boring. This makes devs more likely to skimp on how thoroughly they're run, and with human operation comes room for human error. For these reasons, most devs write application code, and then write additional code whose sole purpose is to automatically test the application. It's similar to how mechanical engineers build a second bridge on top of the main one to see how much weight it can handle. Or something like that. I'm not a mechanical engineer.

Anyway, it's important to gauge how thoroughly this test code is actually testing the application code. If you only design software with a happy path of execution in mind, the minute it leaves your hands it's going to fall apart, as new users will subject it to eldritch and arcane horrors surpassing the greatest Lovecraftian imaginations in existence. A common way to gauge thoroughness is to use a coverage tool that counts every line the tests exercise. This is usually measured as a ratio of exercised lines over total codebase lines and listed as a percentage. Although it may seem appealing to write tests that exercise every line in a codebase (100% coverage), in practice this leads to complicated code, and the 100% coverage isn't worth the added complexity. There's no hard number as to how much of a codebase needs to be covered by tests, but in my experience 70% is a reasonable and accepted number.

In Web Drones, I am only testing two functions, giving it about 10% test coverage. Why?

Writing tests takes time. That's valuable time I'm not spending on promoting the project, writing articles like this, and most importantly, adapting the game to what I learn as people play it. Time testing code that I end up throwing out is time I've wasted.

Keeping this in mind, why is there a small 10% I decided to test? I ended up writing two complicated functions with multiple edge cases. Additionally, the functionality they implement is also based on time passing. It was a lot easier to make these functions accept any time objects I gave them and write unit tests with any time delta I want than it would have been to try and manually test those deltas. Unit testing these two pieces of code ended up saving me more time than what it would have taken to test it manually.

This isn't to say that I am writing Web Drones without any software quality in mind. As alluded to above, I am using the strict server generation in oapi-codegen, which uses Go's type system to guarantee my code will match the exact inputs and outputs I've written in my OpenAPI specification. I also have a serverless function in the cloud that runs a heartbeat on my server every minute, and a small monitoring suite in the form of a Grafana dashboard.

4. Use CI/CD

Spitting on a dumpster fire isn't going to get you anywhere, but what if you instead spit on ten thousand candles one by one? Slowly but surely you'd be able to spit a dumpster fire's worth of flame away from the candles. This is the general idea of CI/CD. You continuously integrate small changes into your codebase, running code manipulations like linting, compilation steps, automated tests, and then submit them for human review. After these steps the changes get deployed in a similar form of continuity. When done well problems end up being small, contained, and easy to revert instead of unmaintainable disasters.

I have the CI in Web Drones, but right now there's no CD. It's a manual process, and for a good reason that has to do with how my infrastructure is hosted.

There are many ways to run CI/CD, but the quickest way to get a pipeline up and running is to use something cloud hosted like Github Actions. It's perfect for a small project like Web Drones. However, my infra budget consists of the two Raspberry Pis I had sitting in my closet collecting dust. I can run continuous integration with Github Actions, but if I wanted to continuously deploy to the physical servers in my apartment, I would have to open up a way for the internet to modify code on my hardware. Securing that involves far more time and risk than manually pulling changes from Github's container registry.

5. Use a test utility to measure your code coverage

So about that test talk above. I'm going to be honest, I never bothered to actually set up a coverage utility. Instead, I used the time-tested technique of spitballing that 10% number. What's the worst that can happen, the non-existent CEO of Web Drones fires me for not giving him a report to appease his non-existent shareholders? This is my project, and if I want to look at a line going up I can write a counter and plug it into Grafana.

-

In a world surrounded by pressure to publish often and as quickly as possible, it's crucially important for developers to prioritize writing quality software. However, we have to keep in mind that quality is a subjective term, and exists in the context of why a piece of software is being written. My goal in writing Web Drones is to explore the joy of creativity in a way that can inspire others to join me, which is a far different goal than the typical one of trying to turn a profit.

If you enjoyed this article, feel free to explore more of my writing and check out Web Drones for a fun way to burn a weekend!