In engineering orgs, it’s a famous trope to preach against complexity. We even have acronyms like K.I.S.S. (Keep It Simple Stupid).

Saying “Things need to be less complex” in a coding dungeon is always met with roars of agreement from the crowd. However, I feel this philosophy lacks nuance so I will take the bold contrarian stance and say:

Complexity in engineerings orgs is a good thing, and we should want more of it!

Let me explain.

Flashback

Do you remember? Back in the days, we used to have complex TV remotes with a bunch of buttons. In the past two decades, they have been dramatically simplified.

Progress of the TV remote since 1999

If you look at the remotes system from the early 2000s and were to model where the complexity lived, you would draw a diagram somewhat like this:

It was designed such that the remote carried all of the complexity and the receiver inside your TV was dumb and simple; it just responded to the signals it received.

In a typical Apple TV remote there is 97% less buttons but we can still control our TVs. How is that possible? Apple didn’t erase the complexity, they shifted it… Where to? Well. Here’s how the diagram might look:

They shifted it onto the software stack of the Apple TV box and the machine learning models in the cloud that power the voice recognition. We as users don’t see it but the engineers at Apple do.

The method of controlling our TV changed dramatically but not the base amount of complexity in the system.

This is because complexity must live somewhere.

My thesis is complexity is akin to energy. Therefore, extending the law of conservation of energy, complexity is a quantity that can neither be created nor destroyed but can be moved or converted.[1]

Example: Bad Music Streaming App

What is the purpose of a good music streaming app? It’s two-folds:

1. I can stream music I already like
2. I can find music I’m going to like

Now, can you imagine if the Spotify app was entirely designed around its basic CRUD primitives? The backend endpoints would be:

POST /artists/search
POST /artists/like/{artistId}

POST /songs/search
POST /songs/like/{songId}

POST /albums/search

You would have a dedicated screen for each of these distinct resources. There is no such thing as recommendations. No no. Instead, you’d need to search for individual songs and artists you want to add to your library…

Oh, and did I mention you would have a separate search page for each Data Object?

The horror! Yet, this is the default state of what a would-be music app is.

The backend complexity of such a system is relatively low (similar to old TV receiver) at the expense of a higher one in the user’s hands.

The way Spotify works in reality: when you onboard, you provide a few artists you like, and Spotify’s systems builds a profile of your taste based on similarity distance across songs as well as pattern matching with their user base such that if I say “I like Drake” their algorithm will say “I have a 98% chance of also liking J.Cole”.

This dramatically cuts down the amount of cognitive effort I need to expand to listen to the music I already like.

In relation to discovery, when a new song picks up momentum with members of my “music taste tribe” Spotify raises the likelihood of me running into this song via either the Homepage, Playlists or Augmented Shuffle next time I come around —all thanks to very complicated machine learning computation.

It’s rather effective. In fact, this the main way I discover new songs nowadays.

If we take a step back from our familiarity and consider it, there is an insane amount of complexity behind the scenes for what appears to be two simple tasks in the Spotify app:

1. Help me listen to music I like
2. Help me find new music I will like

The Spotify team made the correct decision to swallow up all the complexity away from me. The engineering team suffers from having to wrap their head around very difficult problems in these complex inter-connected systems so that I, as a user, don’t have to.

I just open the app and it just works.

Their noble sacrifice is rewarded through both my appreciation and my repeated purchase.

Boyle’s Law

If we image complexity in your applications like water in a container (wink to LC#11) a few interesting properties emerge.

First: it is a fixed amount of liquid that will want to settle evenly from end to end in search of equilibrium. However, you don’t want that! Rather, you want less on the left side.

Second: this requires applying energy and pressure to squeeze the water/complexity out of one end (the consumer-facing) and move it to the other (the company’s).

As I was reaching for an fitting analogy to illustrate the tension here, Boyle’s Law felt right plus bonus point for staying on the theme of physics. Boyle’s Law states that for a fixed amount of gas at a constant temperature, pressure and volume are inversely proportional.

The core intuition is if you halve the volume, the pressure doubles. Cut it to a quarter, and pressure 4x. The smaller the confined space, the harder the gas pushes back and the higher the pressure.

In our case, the relationship is not linear. It takes an ever-increasing amount of effort to squeeze more and more complexity away from the users and into our systems. Going from 50 to 80% swallowed complexity is orders of magnitude easier than going from 80 to 90%.

Conclusion

In conclusion, you should favor swallowing complexity away from the consumer of what you produce. This act requires acting against entropy and injecting directed energy.

There are times when the reasonable balance is to have a 30/70 (user/company) split due to various constraints. For instance, it takes time and money which you may not have in sufficient quantity to push for more.

There are other times when the floor is 5/95 to even stand a chance in a highly competitive market or 1/999 to meet your contractual SLA requirements for your API’s uptime.

Regardless of the unique situation at end, the complexity has to be live somewhere, and ideally a greater share of it lives with you, the maker.

This means the engineering orgs in the most successful companies will see a growing amount of complexity as they seek to serve customers better. This is a good thing! Great engineering orgs should be rabid dogs for any bit of complexity they can swallow up at all times.

Lastly, this framing can be applied in the reverse direction: it can be used to judge whether an increase in complexity is justified. For instance, if a proposal would increase complexity in a system but it’s unclear where else it is being “sucked out off” then it’s a strong signal this is may not be worthwhile endeavor.

[1]: This excludes ridiculous extremes of writing 100 if conditions instead of 1 for-loop. We’re considering the peak of the bell curve of cases here, not the long tails… At either end, anything is possible.