This post is based on my talk “¡Larga vida al legacy!” (in Spanish), which I gave at Commit Conf 2018 in Madrid. The recording is on YouTube.
Starting a new project is always fun. You get to make all the early decisions: language, technologies, architecture, database, how to automate deployments and tests. It’s your chance to leave your mark on the software and grow it with everything you’ve learned from past successes and failures.
Then there’s the other kind of software, the one we call legacy. The one that smells stale, that you only touch when you have no choice, and that you touch with fear. Its insides feel foreign to you, even if you wrote most of its lines.
I think we should see having legacy software as a stroke of luck. The unlucky thing is building software that never reaches the market, or that disappears a few months after launch. The systems that survive usually pay the bills, give us a user base, and give us the oxygen (money) to try again with that new project where we’ll finally do everything right.
What is legacy code?
There are plenty of definitions. Wikipedia talks about code tied to an operating system or technology that’s no longer supported. Another popular one says legacy code is code you’ve inherited from someone else. Sure, but sometimes that someone else is you from a couple of years ago.
Michael Feathers, in his book on the subject, says that legacy code is code without tests. In the context of his book it makes a lot of sense: if you have no tests, you’ll never be sure you can change the code safely unless you fall back on manual testing, and that doesn’t scale. But I’ve also seen code with tests that we considered legacy anyway, because the tests themselves were a mess. We wrote them while we were learning to test, and we got the whole approach wrong.
So here’s my own definition: legacy code is code I don’t want to touch, not even with a laser pointer. It doesn’t matter what state it’s in. The tests may be terrible or great, it may be hard to deploy or even to compile. What makes it legacy is that working with it hurts.
That gives the title two readings. Long live legacy, if by legacy we mean software that comes from far back, that we’ve inherited, that has aged like a good wine. Death to legacy, if we mean the code nobody wants to touch with a stick.
We already know the recipes
We’ve spent many years, many conferences, many talks and many books on how to build software well. The recipes are out there; we just have to apply them, which isn’t always easy.
Take Domain-Driven Design. It gives us strategic and tactical patterns that tell us how to approach the business side and the design and architecture of our software, so it doesn’t turn into that crusty legacy nobody wants to touch, but into mature, George Clooney-style software instead. We’ve all seen talks with diagrams full of load balancers, replicated servers, read models, caches and Kubernetes. And then there’s hexagonal architecture, clean architecture, onion architecture… call it whatever you like, because in the end it all comes down to the same thing.
There are no silver bullets, though. The message that comes with those diagrams (or that I hope comes with them) is that we don’t always need all that complexity. Almost all software is simple when it starts, unless you’re writing software for a rocket. So the decisions we make in those early days, to keep things simple and avoid adding complexity to the architecture, can come back later and kick us in the backside. You start with a plain CRUD, because it’s just a few forms, and by the time you realize you need something else, you’re stuck.
Or the software has been around for many years and the team isn’t ready. We can talk at conferences about read models, eventual consistency and domain events, but taking all of that into a real team is hard. If I bring it in tomorrow, we’ll spend months just learning, as a team, how to do it properly.
All these recipes work well when you start from scratch and can care for every layer from day one. But what do I do with a product that’s fourteen years old?
Throw it away and start over?
Since we know how to do things right when we start from scratch, why not throw it away and begin again?
It depends, of course. But recent history, even if it’s anecdotal evidence, shows that rewriting software from scratch is rarely a good idea once it reaches a certain size. Netscape is the famous corpse. While you rewrite, the old system, the one paying the bills, keeps evolving. Can you keep up with its pace, build the new features without creating new bugs, and integrate with the old system along the way? It usually doesn’t end well.
Rewriting is the option that crosses everybody’s mind, because we all love working on greenfield projects. It’s rarely the best one.
Legacy code, or legacy business?
We talk about legacy code, but we should talk about legacy software. “Legacy code” suggests that all the problems are in the code. Yet that code reflects business requirements exactly as someone thought of them at the time. If we got the design wrong or made bad decisions along the way, why assume the business side didn’t make the same mistakes? Why assume we only need to rewrite the design while every process stays exactly the same?
It sounds obvious, but it happens a lot. We focus on the code, and all we think about is “this call to a stored procedure, I’ll turn it into a call to a repository or an API, so the database is out of the business logic”. The better question is whether you needed that call at all. Is that really how booking a room should work? Those are the questions we have to ask ourselves.
Sometimes the answer is redeeming fire: burn it. It’s rare, but don’t ignore it; it’s a valid answer, and it shouldn’t hurt. We get attached to our code, and throwing it away feels like failing it. Well, we fail all the time. Maybe what has to go isn’t the code but the whole project, because it no longer makes sense as it is, and it needs a new focus. If that’s your case, it’s the option that saves you the most work.
It isn’t usually that simple, though, because the money keeps coming in. Unfortunately. Or fortunately, I’m not sure.
Fill your toolbox, then pick the right tools
Before we embark on adapting, transforming or evolving a legacy product, we need a toolbox full of tools. Books on working with legacy code, evolutionary architectures, refactoring and domain-driven design are a good start, but there are plenty more.
What we can’t do is show up knowing nothing about Kubernetes, Kafka or microservices and announce that Kubernetes with Kafka and microservices is exactly what will solve all our problems. We need to know how each tool fits our context. We need to know about CQRS, event sourcing, the actor model, DDD and every refactoring technique we can find. Not because we’re going to use them all, but because we have to pick the right ones, and we can’t pick a tool we don’t know exists.
If we look back, our past selves made those mistakes precisely because they didn’t know the better solution. Nobody messes things up on purpose thinking “I’m going to make my own life miserable”. We all have gaps, and we keep learning. So it makes little sense to take on a titanic mission with a mega-system unless you know quite a bit more than when it all began. I’d say a lot more, because you need the whole context.
Why do you want to change it?
“Legacy” covers a huge range of situations. Maybe the application is in maintenance mode, new features go into another product, and all you want is to make fixing bugs less painful. Maybe there’s a performance problem tied to a new customer. Before changing anything, think about why. The usual reasons are:
- Adding a feature.
- Fixing a bug.
- Optimizing resource usage.
- Improving the design.
The last one, on its own, is useless. If you don’t want to fix bugs, add features or improve performance in a system, improving its design for its own sake gets you nothing. Unless you’re about to sell the code and it has to pass someone’s quality gate, but those are dark and dirty motives we should never pursue.
Then find where the pain comes from, because it may not be where you think. Your code may be fine and your problem may be deployments. Your code may be fine but have no tests, and you want to evolve it. Those call for very different approaches. If you have decent code and tests but deployments hold you back, maybe you need to focus on automation rather than code (although if nothing is automated, you’ll probably end up touching the code anyway). The pain may be in the architecture, the tooling, the monitoring, the deployments or the tests.
And measure it. What hurts me is what I suffer every day in the corner of the product where I work. In a big product, my pain may be nothing next to a teammate’s, or the biggest problem may be somewhere else entirely. You can have several pains, and each one needs a different line of attack. I wrote more about this in Measure more, suffer less.
Your two most powerful tools
Before getting to recipes, there are two tools that do most of the work.
Keep business logic apart from everything else
Hexagonal architecture, clean architecture, or whatever name comes next: they all boil down to the same idea. Don’t mix business logic with the delivery mechanism, with infrastructure, with databases or with communications. Keep them apart. Each part gets a cool name and its own color in the diagram, which is fine, because it helps us understand each other.
Robert C. Martin’s clean architecture diagram, as I used it in the talk.
It sounds trivial, but it’s hugely important, and we ignore it all the time. Often because of tools, like overly invasive ORMs, we end up touching infrastructure from inside the business logic, or making an HTTP call from an entity. That’s the root of all evil in software development today. If you manage to keep those parts separate, half the work is already done.
It matters most when the business grows and what is one application today becomes two tomorrow. If that piece of business logic lives inside one application and I need it from the other, I’m in trouble. I have to move code into an API and similar operations, and that hurts a lot.
The anti-corruption layer
Eric Evans came up with this name when he described DDD, and it’s a great name. From the domain’s point of view, an anti-corruption layer comes down to Gandalf on the bridge: you shall not pass. It’s a layer we put into our architecture that stops the domain from leaking into the infrastructure, and the infrastructure from leaking into the domain.
“No puedes pasar” is the Spanish dub of “You shall not pass!”
“Come on, Modesto, you’ve just described a facade and given it another name.” Not quite. One of the most important ideas in DDD is the ubiquitous language: inside the domain, we speak the domain’s language. When we end up with many applications, we usually end up with several domains, subdomains and bounded contexts. An anti-corruption layer isn’t just a chance to add a facade, a proxy, a gateway or whatever pattern we like. It’s our chance to speak our own language, the domain’s, from that point inward, and let the outside world speak whatever it needs to.
That’s the key to evolving software that’s a complete mess into something that, I won’t say is a pleasure to touch, but at least doesn’t make you cringe.
An anti-corruption layer:
- Makes the requests to whatever the legacy system exposes. That may be an API, or it may not; in our case it was a stored procedure.
- Translates concepts between the new code and the legacy system. With an old application, the domain itself may have changed since then, or we picked the names badly, so this is where we fix the language.
- May talk to several legacy systems at once.
If you want to think of it as a facade on steroids, fine. But anti-corruption layer is a much better name.
Three strategies for growing around legacy
Eric Evans covers these strategies in far more depth in “Getting Started with DDD When Surrounded by Legacy Systems”. There’s a three-hour workshop version and a shorter talk; I recommend the short one. There are really four strategies, but I’ll fold one of them into the last.
All of them start from the same idea: a bubble context. You create a small bubble, work inside it as you would on a brand-new project, and let the rest of the world be. We applied this to our own product; the steps below are what we did.
1. The anti-corruption layer as a repository
You have your legacy product, and it doesn’t matter what’s inside. It may be an API connected straight to the database, or a desktop application calling stored procedures that hold the business logic. You want to build new things, and you don’t want to put them in there.
Hexagonal architecture says that the repository contracts belong to your domain, and the implementations live outside it, in the infrastructure. So don’t touch the legacy system for now. Build your new part separately, whether it’s a few new screens or a whole new application, and give it a proper architecture: a web service, a rich domain, the best you know how to do. Didn’t we say that if we started from scratch we’d know how to do it right? Then do it.
In our case it was a module for car dealerships. When you walk into a dealership and want an offer for a new car, the salesperson prepares one. We wanted to build that on a tablet, on top of business logic that already existed. So we built the tablet application with a clean architecture, a rich domain and even domain events, although only in memory, not published to any queue. When a use case needed to fetch the list of cars or save an offer, we designed the repository interface the way we wanted the business to be modeled. The implementation behind it called the legacy system, and that was that.
It’s a silly little trick, but it keeps everything new clean, and that’s a pleasure to work with. Fine, behind it you’re calling a stored procedure with eight billion lines of code that would never pass a linter. That’s the trade-off. We couldn’t take on a more radical change from the start. This let us build the whole application, take it to production in a short time, start delivering value, take measurements and get feedback from users, all on top of a cleaner base.
2. The anti-corruption layer that synchronizes
Things are going well, but you’re still reading from the old database. And then it turns out that some of the data you need isn’t in that database, but in another system. Or you need to aggregate data from both sources, and pulling it all in and joining it yourself makes latency high, performance poor, and the experience bad.
The next step is an anti-corruption layer that synchronizes. The legacy system is still the one in charge, but a mechanism copies the data into the new system, and the new system starts using its own database. That database can act as a read model, like the ones in those diagrams full of colorful boxes. It may even have its own write model. In another application where we followed more or less the same steps, we suddenly needed our own users for authentication, with their own persistence; the legacy system’s users didn’t matter to us.
It’s another small step, and it lets you own the data model. That’s critical: not owning your data model should be temporary, a sacrifice you accept because you have to ship the product.
How do you synchronize? There are plenty of ways: events, or old-school database triggers on the legacy database. It depends on your context and what your system lets you do. In one of our systems, a trigger felt really ugly, and the first thing we said was that we’d have to replace it. But it was the fastest option, and the first one that let us validate our hypotheses. Trade-offs, with all the nasty things triggers come with.
This connects with the toolbox. If your team is full of experts used to distributed architectures, you can skip this step. Our team was on a path of continuous improvement, but eventual consistency and distributed architectures were still far away for them, so we had to go one step at a time.
The good part: around the legacy system, new products appeared that could deliver value independently, and the business kept growing without the legacy system holding us back. To be clear, the legacy system was still a total mess. We hadn’t eased that pain yet. But we were working on the parts we wanted to evolve, where we wanted new features, while the legacy system stayed in maintenance mode. The pain of maintaining it didn’t justify more work.
3. The anti-corruption layer as an event stream
The third step looks similar in the drawing, but it’s a much bigger leap. Now the new system’s database stops being the secondary one: for the things that belong to the new system, it becomes the single source of truth. That doesn’t mean all the data lives there. That database is a subset of the main system.
Here you have to modify the old system. It has to be able to both publish events and listen to events from your other systems. This is where we enter the more complex architectures we talked about earlier, with bidirectional communication, event buses and all the rest.
I said there are really four strategies. The one before this step skips the event bus: the new system exposes services with its piece of the business, and the legacy system calls them. The problem is coupling. Every time I take a bite out of the legacy system and move it to another product, the legacy system has to know where I moved that bite and which API to call. If I make the jump to events, it no longer matters where the bite ended up. The legacy system listens to the event, wherever it comes from.
When the numbers don’t add up
This comes with a trillion problems along the way. To give you an idea, we’re dealing with a system with almost 8,000 stored procedures. It’s a festival.
We won’t always be able to use the anti-corruption layer to rethink our domain, redefine its concepts and redefine its processes. If I divide 8,000 by the number of people on the team and the time we have, the numbers don’t add up. At this same conference there was a talk about a similar migration that took around 50 people and eight months. We can’t work at that scale. So in our case the change has been, and will be, almost mechanical: this stored procedure takes these parameters and returns this response, and I’ll turn it into something I can take out of the database, somewhere I control better, that lets me scale, search, and get faster feedback.
Every project needs its own tricks. In ours, before touching a single line of code, we plugged Prometheus into our SQL Server to measure which stored procedures were called most and which took longest. We combined that with how many lines each one had (assuming the bigger ones were more complex), whether they were read-only, and how many places in the code called them. When you want to get rid of direct calls to 8,000 stored procedures, you can’t go one by one. Well, you could, and then die.
Before that, we had even bigger problems: we couldn’t even compile the whole application, because it had 800 separate modules with circular references between them. In fourteen years we had plenty of chances to do things wrong, and we made good use of them. The first thing we did was get those modules building automatically, solving the circular references. Now our goal is to build almost everything in under half an hour.
If your pain is small, maybe one step is enough. If it’s big, it’s very unlikely you’ll solve it in one step. It will take several, and you may have to get creative.
And the tests?
How do you make sure the behavior stays the same when you move things from the legacy system to the other side? There’s a simple algorithm:
- Decide what you want to change.
- Think about how you’re going to test it.
- Write the tests.
- Change the code.
- With the tests green, consider refactoring.
With legacy code you’ll write lots of tests, but not test-first in the TDD sense. You write them before touching the production code, so that when you do touch it, you know you’re not breaking anything. Sometimes, as part of the refactor, you’ll change those tests again, because they pushed you toward an approach that wasn’t right.
If your legacy system is large, think hard about tooling, because you may have to build your own. For the read side of those stored procedures, we’re generating the controllers of a read API, and their tests, with our own code generation, so we can migrate them one to one. We lose the chance to change the domain, but we don’t have the time; that stays for the future. What matters is having the tests first. Without them, you don’t know whether the system works.
And without being naive: sometimes the code is so hard to test that you have to change something first, opening a door for dependency injection or something similar. When you do, touch as little code as possible before the tests are in place. Sometimes there’s simply no other way.
Long live legacy
None of this gets done in a day. But these three strategies (or four, with one in disguise) give you a lot of room to start working on a legacy product, at least at the level of architecture and high-level design. The same idea works at the level of the code, through the boy scout rule: small steps, leaving things a little better each time.
You still need to master the toolbox. You’ll have to choose between an event bus and a persisted event stream like Kafka, and decide whether it’s too early for separate services or whether you keep everything together for now, even if it’s behind APIs. There are plenty of decisions to make, and the software that pays the bills deserves that we make them well.