This post is based on a talk I gave twice: at Software Crafters Barcelona 2021 and at Commit Conf 2023. Both were in Spanish. The examples differ a bit between the two, so this post combines them.
A legacy system with dice
All the characters an artist’s clients ever played, in one picture (shared on Reddit). After enough years, characters pile up.
I have been playing role-playing games for more than 25 years, more or less with the same group of friends since we were 13. Some of our characters are more than ten years old. During the pandemic we were all desperate to see each other, so I offered to run a game again, and I picked up the one I had been running years before, back in 2002.
A new edition had come out since then. The rules had changed and the system had changed, so we tried to bring our old characters over, and we couldn’t. It was a legacy system in every sense. On top of that, the new rulebook was a maze: the rules for energy shields were in one chapter, the rules for armor in another, and the armor rules added a detail about energy shields. Between how hard it was to migrate our old characters and how the new system was written, I said: forget it, new character sheets, new adventure.
We have been playing for almost two years, and we still find rules in some paragraph in another part of the book that change how something works. It didn’t have to be that complicated.
There is a parallel with how we build software. Between the complexity that is already there and the complexity we go looking for, things get out of hand. That is what this post is about: complexity seen from a high level of abstraction, with some examples where more than one of you will think “I’ve been there”. I certainly have.
Complexity is everywhere
Pick the agenda of any software conference and read the talk titles: working with legacy code, sustainable code, modernizing a monolith, vertical slicing, understanding your systems with OpenTelemetry. Most of them deal with complexity, usually from the angle of “you already have it, now what?”. I want to take a step back and look at what makes us end up with it.
The dictionary definitions don’t help much: something made of diverse, interrelated elements, intricate and hard to understand. In our context I’d summarize it as many elements that end up raising the difficulty of whatever we are trying to do.
Inherent versus accidental
The distinction everyone uses is inherent versus accidental complexity (or essential versus accidental). Inherent complexity belongs to the problem. If you need to send a rocket to the Moon, the laws of physics are going to get in your way, and you can’t remove them. Sometimes you can reframe the problem, but most of the time the inherent complexity is simply there. Accidental complexity belongs to the solution: we analyzed the problem, chose a path to solve it, and that path added complexity of its own.
The usual advice is “accept the inherent one and avoid the accidental one”, and that’s where the conversation ends. It reminds me of “high cohesion, low coupling”: fine, and now what do I do on Monday?
I find it more useful to change the reference point from problem and solution to delivering value:
- Inherent complexity is the minimum we have to take on to deliver value.
- Accidental complexity is what we pick up along the way while delivering that value, based on the decisions we make.
Once you look at it this way, the label matters less. Very often the complexity that hurts comes from a problem created by a solution, which in turn was solving a problem created by another solution, or by a poor analysis. It’s a matryoshka. I don’t care where it came from. If it’s slowing us down and getting in the way of delivering value, we have to deal with it.
What we know about it
Whatever we call it, complexity:
- Feeds on our decisions, both the ones we make and the ones we don’t. Every decision is a chance to add complexity, reduce it, or leave it as it is. Spoiler: it usually goes up.
- Tends to grow. Most commits add code rather than delete it, as satisfying as deleting is. A system only stays still if nobody works on it.
- Is inevitable. As soon as you can’t remove the problem, there will be some minimum complexity in whatever solves it.
- Is impossible to eradicate. Don’t set out to avoid it completely, because you won’t. Focus on what is stopping you from delivering value. If part of your system is complex but well organized and isn’t hurting you, accept it.
- Is easier to spot from a distance. Like any toxic relationship. When you’re deep in the mud you don’t see it; someone new to the team, with no familiarity with the problem, spots it right away.
Someone in a hallway conversation once told me that none of this applied to them, because they changed jobs every two years. Sure, and wherever you go next, you’ll be dealing with the complexity the previous team generated.
The greenfield dream
A new project. Green meadows. This time we’ll make the right decisions: a proper architecture, everything super scalable, no repeating the mistakes of the last project. We arrive with our bags full of patterns and tools.
A month later the meadow is on fire, and we’re in the kind of project that made us change jobs last time. Only now it’s our fault.
A new project, and a little later.
What happened? Sometimes we believe that something like a reference architecture is the answer to our engineering problems. Search for Spring’s microservices architecture and you’ll get a diagram with an API gateway, a service registry, a config server, distributed tracing, and so on. I’m not picking on Java: I come from .NET, and any cloud diagram from any ecosystem would do. Conferences are full of talks that hand us these formulas. The formulas aren’t wrong. The problem is assuming that everything has to be built that way.
Most decisions, least knowledge
The real issue is when we make our decisions. Our knowledge of the problem grows over time. But we make the biggest solution decisions, the ones that will be most expensive to change, at the very beginning, when we know almost nothing about the problem or its context.
| At the start | Later on | |
|---|---|---|
| Knowledge of the problem | Minimal | Growing |
| Decisions made | Most of them, and the biggest | Fewer, and constrained by the early ones |
Who hasn’t decided at the start of a project which infrastructure it will run on, which languages, which frameworks, which tooling for operations, “and let’s add Kafka here”? That’s a mortgage. We’re trying to read the future and adding complexity for things we think we will need. Kubernetes, and since we have Kubernetes, Prometheus, Grafana and Istio as well. For what? A website. “But the forecasts say…” Sure.
It gets worse. At the beginning almost every problem looks like CRUD, because the business is still emerging. So we pick architectures, designs and ways of working that make CRUD easy: an ORM, some services, microservices of course, even if the team is three people.
Not everything is ours, but then it is
We decide on infrastructure, languages, frameworks, tooling, deployments, persistence, messaging, service discovery. And a lot of it isn’t even ours: external tools, libraries, third-party products and services. Remember the xkcd comic about the whole of modern digital infrastructure resting on one project maintained by a random person in Nebraska.
Each of those decisions brings more decisions. Choose PostgreSQL and you’ve chosen a lot of things; choose MongoDB instead and you’ve chosen others; then add Elasticsearch next to it. All of that constrains you. It’s like creating a wizard in Dungeons & Dragons: there are things you simply can’t do anymore, and in a rigid rule system they’re out of reach.
Then there’s the irony. The same person who pulls in Kafka, an ORM, or a framework “because it’s the standard” calls it overengineering when you want to encapsulate something or write a small wrapper that speaks the language of the business. You just added Spring Boot. Spring Boot adds no complexity at all, as we all know. And before anyone makes the usual node_modules joke: go look at your NuGet package cache. We have plenty of our own.
It’s not just about code
We associate complexity with code: design, design patterns, architecture patterns. But it also lives in infrastructure, support, deployments, configuration, hiring, and the structure of the company itself. And in the code too, of course.
Consul and three months of waiting
At a previous company we worked for the automotive sector, which now likes to call itself “mobility”. Every product but one was deployed on premises, in our clients’ facilities, and we had point-to-point lines with factories to order vehicles. It pays the bills, but it isn’t exactly the cutting edge of infrastructure.
At one point the development teams needed centralized configuration, with service discovery coming later. From the code’s point of view the decision was easy: don’t reinvent the wheel, Consul exists and works, and it’s open source. Then reality arrived. Agreeing with each client’s systems team on how to install Consul in their infrastructure took three months. The tool we picked to move fast and save ourselves complexity left us three months without delivering on that front.
When we looked at what we actually needed, it was quite specific: centralized configuration now, and service discovery maybe later. We ended up building an ordinary API service for it. We could deploy services like that in our sleep, because that part was fully automated, and it covered our needs perfectly.
The same thing happens with the cloud. In some regions bandwidth isn’t a given, so moving something to the cloud can make it unusable for a dealership there, or force you to complicate the deployment infrastructure a lot. And support: will first-level support be able to diagnose this, will we even have access to the infrastructure, what does the SLA say if it goes down?
Hiring and team structure
Building your product in Erlang, or any language outside the usual suspects, may make complete sense because of the problem you’re solving. It also affects how easy it will be to hire, depending on how you need to grow. We tend to forget that.
We complicate hiring processes themselves, too. I’m involved in hiring, and when I look at the comparisons of selection processes some companies publish, I wonder whether we really need Google-sized processes.
And the organization. Not long ago we had two teams that, judging by how they worked and how they depended on each other, should clearly have been one. Twice the ceremonies, someone acting as a bridge between them. Why? Because at some point it made sense. Merge them, and stop making it so complicated.
This is why sociotechnical architecture interests me so much. A colleague says I’m getting pompous when I use the term. I didn’t invent it; read about it, it’s worth it. It helps a lot with exactly this kind of complexity.
Context matters
Nick Tune has a post on turning domain discovery into product and organizational improvements with a chart I like a lot: business differentiation on one axis, model complexity on the other. Domains that are complex and differentiating are your core. If something brings no business differentiation but has a huge amount of complexity, don’t bother owning it.
Core, supporting and generic domains, from Nick Tune’s post.
The example I always use is corporate email. Running your own Exchange server used to be completely normal. It eats your systems team’s time and keeps them from working on what brings you value. Does it make any difference to your customers whether the email they receive came from your own server or from Google Workspace or Microsoft 365? None. The same goes for chat tools, unless you happen to work on the team building one.
Wardley maps
Wardley mapping makes a similar point at a strategic level. On the vertical axis you have the value chain, from what the user sees down to what they never see. On the horizontal axis, how evolved each component is: genesis, custom built, product, and commodity. Simon Wardley’s classic example puts power at the bottom right.
Simon Wardley’s example map, from Finding a path.
When you open a tap, you expect water. You don’t expect it to come out dirty or with no pressure. When I came to give this talk I didn’t wonder whether there would be electricity or internet; I took it for granted. Nobody builds a power plant to supply their office. Power is a commodity, and computing is heading the same way: we used to buy dedicated servers or plug a machine into the office’s fixed IP address, and now we don’t even think about data centers. The trend is to push everything the user doesn’t care about as far right as possible. Does the user care whether your emails go out through your own mail server or a cloud service? They care that the emails arrive.
But context changes the map. If your project is somewhere where water and power aren’t guaranteed, you can’t take them for granted, and you’ll move left, towards products or something built to measure.
The part of Wardley’s work I find most useful against complexity is the table of characteristics for each stage. In genesis, failure is tolerated and even expected, understanding is poor, and efficiency means reducing the cost of change, because you’re experimenting. As a component becomes a commodity, failure becomes a surprise, and you care about operational efficiency.
A lot of accidental complexity comes from a mismatch between our decisions and the reality of our project. How many times have we chosen something because it performs better, scales better, or will supposedly be easier to maintain in the future? Is performance in microseconds important right now? If a page takes a minute to load, obviously yes, this isn’t the nineties. But we have to ask. Agreeing as a team on questions as simple as these changes the design:
- If this fails now and then, is it a disaster? Will someone stop paying us?
- Is changing this feature next week a problem?
- Is it a problem to change the user interface every other week?
If your users don’t even pay yet because there’s no validated business model, and they understand that the product is changing, you can afford to rebuild the interface or have the occasional failure. You’ll make very different decisions, and you won’t spend time on a super-optimized query that saves three milliseconds.
Our software is used by car dealerships, body shops, repair workshops and vehicle importers. A workshop has 20 people, not 2,000. Selling cinema tickets means dealing with concurrency and massive load; selling a few thousand cars, with luck, doesn’t.
The map also helps you plan the trend. Maybe to validate an idea a pay-per-use SaaS product is enough, because you don’t want to buy the complexity of building it from scratch yet. But you know that as the model grows, the product will fall short, and that piece will have to move left. Knowing that ahead of time helps you find the balance between all those forces, and add the minimum complexity that lets you keep delivering value.
Example: a stream without Kafka
A stream of messages is, put simply, a continuous, persisted flow of messages that we can go through and query. The important difference from a queue in a pub/sub model is that a stream stays there: you can revisit it, replay it, go back in time and check which message you were on. That opens up some very interesting reactive programming patterns.
Mention streams of messages and many people will think of event sourcing, which brings in much more complexity than a stream actually needs. And today a stream is associated, in 99% of cases, with Kafka or its cousins, like Apache Pulsar or the equivalent cloud services.
We needed a stream. We had been using ordinary message queues, and we needed to travel back in time and have a consolidated, persisted stream to integrate with other products. Then we read the best practices for running Kafka: three nodes at least, five recommended, 32 GB of RAM is acceptable but don’t go lower, and the more cores the better, with 24 being a good number.
We ran the whole Peruvian market of a couple of car brands, all sales and imports, on three or four machines: a database server, a few web servers, some services. Was I going to tell the client I’d double or triple their infrastructure just to have a stream? And Kafka needs to be administered, monitored and secured, and it’s not easy to run. Looking for literature on how to have streams without that kind of setup is hard, too, and we were on premises and on Windows, which narrows it further.
So we evaluated a lot of products, knowing the cloud wasn’t an option, and looked at our volumes. Kafka shines with volume, and we needed the pattern, not the volume: around one or two million events a day, which is peanuts for a database server.
We built an API that persists messages in a table in SQL Server, which we already used. Our oldest product is almost 20 years old, and SQL Server was what the company used. The API lets services consume messages in order, as a stream, and a couple of internal libraries keep track of which message each consumer is on. SQL Server was already in production, already backed up, already administered and monitored. And because the stream lived in the same database server, we could play with transactions without distributed transactions. A lot of the complexity you buy with messaging, such as not having guaranteed order or consumers receiving the same message several times, simply went away. The literature says you have to handle all of that. In our case we didn’t need to.
It has been running for well over a year and works very well. It’s not glamorous. I couldn’t give a talk about that solution and show off; we basically put messages in a table. But in that context it enabled something we didn’t have otherwise: delivering value.
We’re not alone. The people at Prequel wrote about why they ditched RabbitMQ and replaced it with a Postgres queue.
Some people hear this and say we’re reinventing the wheel. In this scenario, the complexity is adding a new piece of infrastructure that someone has to administer and monitor, and that doubles the infrastructure budget. When there’s not enough time to explain the context, people want to take away your developer license. I never studied for a degree, so I never had one to lose.
Example: CQRS is smaller than you think
If I say CQRS, many of you picture something like this: commands come into an API with a write model, events synchronize it with a separate read model, each with its own database, maybe event sourcing on top. That’s the CQRS of conference talks, blog posts and videos.
In February 2010, Greg Young, who coined the term, wrote a post called “CQRS, Task Based UIs, Event Sourcing agh!” (the original site is gone, so the link goes to the Internet Archive), saying people were confusing CQRS with other things. If he named the creature, I think he gets a say in what it is. His example starts with a service that mixes reads and writes:
CustomerService
void MakeCustomerPreferred(CustomerId)
Customer GetCustomer(CustomerId)
CustomerSet GetCustomersWithName(Name)
CustomerSet GetPreferredCustomers()
void ChangeCustomerLocale(CustomerId, NewLocale)
void CreateCustomer(Customer)
void EditCustomerDetails(CustomerDetails)And CQRS is this:
CustomerWriteService
void MakeCustomerPreferred(CustomerId)
void ChangeCustomerLocale(CustomerId, NewLocale)
void CreateCustomer(Customer)
void EditCustomerDetails(CustomerDetails)
CustomerReadService
Customer GetCustomer(CustomerId)
CustomerSet GetCustomersWithName(Name)
CustomerSet GetPreferredCustomers()That’s it. Separate the read model from the write model. Separate persistence isn’t required. Events aren’t required. Event sourcing isn’t required either. From there the problem may take you much further, and it often should, but the core of CQRS is just this separation. I personally prefer separate command and query handlers over services like these, but that’s a design discussion for another day.
I do a lot of interviews, and you can’t imagine how many people only understand CQRS as the big version. Arguing with that is like arguing with Alan Kay about what object orientation is.
Why does it matter? Because with legacy systems we sometimes get stuck on “we should be doing proper CQRS here”. At a previous company our legacy was 15 to 17 years old. We wanted to start separating read and write models in some areas, and the reaction was: wait, for a read model I’d need events first, and then…
No. If it helps you deliver value and keep the design maintainable, create a view in the database, and let that view be the persistence of your read model. The read model itself is in the code; don’t forget that. First, you lose the mental block of “I’m doing it wrong”. You’re not; there’s no right or wrong, and if it helps you, you’re doing it perfectly. Second, you’re free to take the next steps: enrich the domain, introduce events, and one day that view can become a table fed by those events, as you decouple the system. It takes months, depending on how big your system is, but you keep moving forward along the way.
Example: the outbox pattern in a legacy system
In systems that don’t use events yet, people are often afraid to start. Especially business people: “Can I rely on events? I can’t lose events. What if I save an order and then publishing the event fails?”
The answers are usually: wrap it in a retry. Fine, but what if the process dies in the middle of the retry? The order is saved and the event is gone. Then someone suggests the other order: publish the event first, then save. Now if saving fails and the transaction rolls back, you have an event that lies. Either you don’t have the event, or you have one that lies. Next suggestion: event sourcing, so you only ever write events. With 17 years of legacy, that’s not happening tomorrow.
The outbox pattern is very simple. If you control the design and the transactions of your system, separate the decision to publish a message from the publication itself:
- Start a transaction.
- Insert the order (or the customer, or the invoice) into its table.
- Insert the message you want to publish into an outbox table, in the same database. Please don’t put it in a different one; that means distributed transactions, and a huge amount of complexity.
- Commit.
A background process reads the pending messages from the outbox table and publishes them. If publishing fails, it retries, and if it keeps failing, it’ll pick them up again on its next read. Messages are eventually published, and with messaging that’s what you have anyway. This needs transactional persistence; without it you’d have to look for other mechanisms.
With almost no effort, you’ve introduced events with guarantees into a system that wasn’t using them, and where people were afraid to.
It gives you something else in a legacy system. Ours had thousands of stored procedures, some of them dozens of pages long. Is anyone else feeling seen? Since the stored procedure can write to the outbox table, you can publish an event from that monster you haven’t decided to tackle yet. It’s not ideal, but it lets you keep delivering value while you evolve towards what you want to have.
Bonus: the fake connection class
Thanks to Fermín (@mintxelas) for this one. Our legacy is almost 20 years old. It’s a beast, but it pays the bills and the salaries, so you love it anyway.
Our legacy, more or less.
The product was a classic client-server application: a desktop app that connected directly to SQL Server. We’re moving to the cloud at our own pace, and connecting a desktop app straight to a SQL Server over the internet isn’t an option. We needed a web server in between. But there were around 7,000 stored procedures holding all the business logic, and building an API for each of them wasn’t going to happen any time soon.
Data access wasn’t even encapsulated. The app used the ADO.NET connection classes directly, the .NET equivalent of raw JDBC. Fermín’s idea: create a class with exactly the same contract as the ADO.NET classes we were using, and change the namespace in the using statements so the app uses ours instead. Behind the scenes, our class takes the stored procedure the app wants to call, calls an API, the API runs the stored procedure, and the result comes back.
It brings latency and bandwidth issues, which we had to deal with, but a huge problem, migrating 7,000 stored procedures to something that works in a web environment, was solved. It’s still running today. I’m sure Fermín was scared when he proposed it, and that he took a shower afterwards. But once you have the complexity, you have it, and you have to fight the beast you’ve got.
What can we do?
- Pay attention to the requirements. The real ones. Don’t invent them: “the client might want to be notified when…”. Half the time the client knows what they want, but not what they need. Neither do we.
- Consider the error cases. We tend to design the happy path only. Then errors show up, you realize you never considered that the system you integrate with could fail, and you end up with a mess. Have you checked how often a payment or a call to another system fails? A process designed without error cases can be synchronous; once you think about the errors, it may need to be asynchronous. People say asynchronous is more complex. What’s complex is converting a process to asynchronous after you’ve gone a long way with it synchronous.
- Beware of fashions. “I want to work with Kubernetes.” You should want to work on solving problems.
- Isolate complexity. One of the advantages of microservices, among their many disadvantages, is that they isolate things. But this isn’t only about microservices: every time you create the right abstraction, expressed in the language of the business, you isolate the complexity behind it, and you can later replace the implementation. All the complexity behind a power plant doesn’t matter to you because it’s isolated.
- Care about the (ubiquitous) language. Interesting things happen when we stop describing our problems in technical terms (“I have a lock on such and such”) and start asking which business problem we’re solving. So many times the problem we’re trying to solve was created by the solution we chose for a previous problem, cascading through several levels. When someone asks me for help without much context, I always ask the same question first: what business problem are we solving here?
- Master the art of kicking the can down the road. Eduardo Ferro (@eferro) has a great talk with that title, in Spanish, about the cost of undoing decisions and postponing them. If “kicking the can down the road” doesn’t go down well with your boss, call it defer commitment, a Lean principle highly recommended in high-performing organizations. It’s the same thing. It’s the way out of the trap from the beginning of this post: making huge decisions with a tiny amount of information.
Run away from accidental complexity, please. The business has enough interesting, attractive challenges waiting for us; we don’t need to put our focus on technical ones, however fun they are.