This post is based on my talk “Aproximación al testing en proyectos de desarrollo” (in Spanish), given in February 2020.
I’ve been building software for more than twenty years, and these days I’m the CTO of Grupo Domingo Alonso. I’m one of those CTOs who still writes code. Not much of it reaches production, but I don’t stray far from the code or from the practices. Still, my point of view has shifted over the years, and that shift is what this post is about.
First, a warning. However long I’ve been doing this, the most I can offer is an appeal to authority. There’s no scientific evidence about how to develop software. Nothing proves that doing things one way will go better or worse than another. At best I can promote what follows from anecdotal evidence to expert advice, assuming I’m an expert.
The closest thing we have to science is the DORA State of DevOps report, whose team Google has since bought. It surveys more than 30,000 professionals around the world about the practices they use and how their companies deliver software, and it tries to move from correlation towards causation. The funny thing is that when you read it, once again everything depends on context. Not on any one practice. Maybe you don’t do TDD, but you write tests afterwards and combine them with a few other practices, and you get the same result.
I usually say that software is Galician, because the answer to almost every question is “it depends”. So drop the dogma. “You don’t do TDD, so you’re doing everything wrong” isn’t true. What you need to do is understand the context of the problem, so you can work out what the context of the solution should be.
Fast, good, cheap: pick your unicorn
Fast, good and cheap at the same time doesn’t exist. It’s a unicorn. When we build software we have to decide where we want to stand, and sometimes dropping good to stay fast and cheap is the right answer.
I once talked to someone whose team tested everything by hand. Everything. It sounded crazy until they explained: they built mobile apps for marketing campaigns, the kind where you buy a chocolate drink, install an app and get a gift. Each app lived on the user’s phone for about a week and never evolved. Automated testing made no sense for them, and neither did TDD or any practice like it.
That’s a valid decision, but only because they knew the alternatives. If you don’t know how to do something, how on earth are you going to decide whether to do it one way or another? Dismissing a practice out of ignorance isn’t valid, and it won’t be valid for your clients either.
Iterative and incremental means automated testing
Waterfall worked, more or less, back when the user wasn’t at the center. Your technical limits in RPG/400 or COBOL were so strong that you had to bend the problem until your one possible solution roughly fit it. Things were done in three ways and that was it. That’s why some big department stores still run green-screen terminals.
We’ve turned that around over the last few decades. Now we put the user at the center, think about their problem and try to solve it as well as we can, because the range of possible solutions is huge. Any snake-oil salesman will tell you it’s infinite. It isn’t, but it’s huge. With that many options we can’t afford anything other than working iteratively and incrementally: the risk would be too big.
Iterative and incremental means small vertical slices that reach production. Not horizontal layers, and not processes you haven’t finished even in their smallest version. In the first iteration a small piece of functionality goes to production and users start using it. In the next one it grows a bit. And so on.
The result is that building the software isn’t what costs the most. Development costs something, obviously, but it’s tiny compared with the cost of evolving it. If you use traditional development techniques inside an iterative and incremental model, you’ll reach increments where you have to throw away everything you did before to respond to the new demand.
This is where testing comes in. Everyone tests, even if only by hand. (If you push code without even trying it by hand, I don’t want to work with you.) The question is what happens as you go:
- In the first iteration you build feature 1 and test it before releasing.
- In the second you build feature 2. Before releasing, you need to know that feature 2 works and that you haven’t broken feature 1.
- Keep going, and on every release the number of features you should test isn’t the ones you built in this iteration. It’s every feature the system has.
Manual testing doesn’t scale with that. It only scales in the pockets of the companies that sell manual testing, not in your clients’ pockets, and it doesn’t scale in time. So if you work iteratively and incrementally, skipping automated testing makes no sense. And I’m not talking about TDD yet.
Testing is about confidence
You’ve heard the joke: never deploy on a Friday. It’s a good joke. In many contexts I wouldn’t do it either, because the risk is too high. But it’s a matter of risk assessment. If the chance of a failure is low, and a failure wouldn’t cost much, the risk is low and you can deploy without worrying. If people could die because you deployed on a Friday, please don’t.
Testing is a matter of confidence, and confidence comes from feedback loops. Extreme Programming puts a lot of weight on them. Its classic picture draws practices in concentric circles, from the fastest loops to the slowest. What matters most is having very quick feedback about what you’re doing right now and its impact.
A manual test pass that takes two hours won’t run on every change. But your automated tests, unit or not (more on that later), should be close at hand. Don’t wait for continuous integration to run them: run them locally. Even better, have them running constantly, telling you whether what you’re doing works. When we push testing to the end of the iteration, right before building a package, the feedback isn’t as bad as finding the bug in production, but it’s close. And it breaks the iteration.
And it’s about money
If the cost of a product built iteratively and incrementally isn’t the feature you’re building now but what it will cost to evolve it, then we’re talking about money. That goes well beyond the technical side.
That’s where TDD starts to play a role, depending on the kind of development you do. In product development especially, TDD helps a lot to keep complexity down. But look at it from an economic perspective first.
A map with many roads
You’ve surely seen the testing pyramid. Unit tests at the bottom, cheap and fast to run; service, UI, integration or end-to-end tests above. Call them whatever you like. The point is that fast tests are cheap and slow tests are expensive. They’re expensive because they need infrastructure to run, and because they tend to be more fragile, so they slow you down twice: they take longer to run, and you break them more often.
People love to argue about whether a test is unit or integration. Some purist will tell you, “That test touches the system clock, so it’s an integration test.” Look, it touches the clock, but the clock isn’t part of the business logic, and the test is fast. I don’t care.
There are many taxonomies, and it helps to know which one you’re using:
- By scope: unit, integration, system.
- By visibility: black box, white box.
- By intent: acceptance, functional, non-functional, and on and on.
- By execution: manual or automated.
Google used to split tests into small, medium and large. Small ones run on the developer’s machine, medium ones maybe on every push to that giant monorepo, and large ones once a day because there’s no time for more. I don’t know if they still do it that way, but the criterion is interesting: they classify by cost.
Which classification you use doesn’t matter. What matters is writing the right tests. Maybe your system doesn’t need any UI tests, and that’s fine. Unless some certification forces you to tick a box (and I don’t much care about those), you don’t need to tick boxes saying which kinds of tests you have. You need to think about what your context needs and how best to cover it. And for that you need to know the options. You don’t need to master them all, but you need to know as many as you can.
Schools and approaches
TDD, ATDD, BDD. Outside-in, inside-out. “I’m London school.” “I’m Chicago school.” We’re not rappers. We don’t need these battles.
The two approaches do lead to different designs, though, so it’s worth knowing how they differ:
- Outside-in starts from the outside of whatever you’re building. If it’s an API, you start from its entry points, from how a developer would use it, and work inwards: aggregates, services, then repositories. Since the inner pieces don’t exist yet, you can’t check that something was created; you check that the API called the repository’s
Create. You test interactions. - Inside-out goes the other way. You start with the repositories and work upwards, and you usually test state: in a simple CRUD, after creating something you check it exists.
That’s the London versus Chicago distinction: verifying interactions and behavior versus verifying state.
Why start outside? Because the decisions you make first leak into the steps that follow. If you start designing from the repository, something from it will probably leak up into your API, something that isn’t part of the problem. You carry accidental complexity from the solution into the problem.
Outside-in is harder, though. Since you test interactions, you’re fixing very early how the parts of your system talk to each other, and that makes tests rigid. It takes a lot of design experience. Its biggest advantage, for me, is how quickly you can validate hypotheses. Teams with enough flight hours to work outside-in can hand the frontend team a working API very early, maybe with an in-memory repository behind it. Yes, you could agree on the API in a meeting, share the Swagger file and build a mock. But an in-memory implementation is cheap, and it already works; it isn’t a mock.
Techniques by intent
Beyond TDD there’s A/B testing, performance testing, acceptance testing, smoke testing, and many more. The intent of a test largely decides which technique fits. That doesn’t mean you have to use them all. A few that people usually ask about:
Mutation testing is the test of your tests. A tool parses your production code and changes it: it flips the sign of a multiplication in your calculator, or makes it not multiply at all. If your tests are still green after that, they’re not testing well. Since the mutations are automatic, you can get plenty of false positives, like values out of range. I’ve never used it professionally. I’d love to face a problem that needs it, but it’s expensive and hard to justify economically in the kind of applications I build. If lives depended on it, I’d use every technique I could.
Smoke testing is better explained with the anecdote than the definition. On hardware production lines there used to be a quality check that went like this: plug it in. If it smokes, it doesn’t work. If it doesn’t, it meets the bare minimum. A smoke test suite is the smallest set of tests that tells you whether you’ve broken the system. Technically they may be end-to-end tests, but the name states a clear intent: the minimum that tells you if it works at all.
Architectural testing, which we’re starting to introduce in our teams, is close to consensus testing. You already do consensus testing, maybe without the name. A linter checks the team’s agreements: spaces instead of tabs, no lines longer than 80 or 120 characters, variables named however you decided. Architectural tests do the same for your architecture. If you’ve chosen hexagonal, clean or onion architecture, you’ve decided your system has certain parts and that infrastructure details mustn’t leak into the business. An architectural test makes sure nobody runs a SQL query straight from a controller or an entity.
They help a lot with distractions, like that day when you went out the night before. And they help bring new people into a team. Are we going to keep less experienced people out because they don’t know all the rules we’ve come up with? No. We check the rules automatically. In Java or .NET, where the runtime supports reflection, they’re easy to write. You can make them as sophisticated as you want, but the simple version already pays off: a class in this namespace must never reference classes in these other namespaces. ArchUnit does this in Java, and there are ports for .NET.
Which brings us to the obvious reaction: what a mess, Modesto. Exactly. We need to know the whole range of options so we can pick one or another. If all you know is a hammer, everything looks like a nail. And sometimes TDD isn’t the most cost-effective option, because the production code is already written. Maybe you’re walking into a system that has been in production for fifteen years.
That’s our case. We build new products too, but we also have a system that, technically, could be the example in a legacy code book. Do we abandon it? It pays my salary. Never test it, then? But we want to evolve it. So you need to know the techniques that let you improve it bit by bit, and that brings me to the boy scout rule.
Leave the code a little cleaner
Leave the campground cleaner than you found it. I’ve never been a scout, so I’m going by the movies, but I live in the countryside and I apply it there too. Don’t throw things in the countryside, please. There are bins.
In code it works the same way, with one nuance. If you’re just passing through some code, ignore it. If it’s in production, works, has no bugs and you’re not going to evolve it, it doesn’t need your time. But if you’re there because there’s a bug, or because you need to change it, leave it a little better than it was. If you never come back, nothing happens: it’s code that works and causes no trouble. If you come back in a few months, leave it a bit cleaner again. If you’re in it every day, maybe it deserves more love. At the very least rename that function: instead of XX1 and XX2, say what it does. The basics from Clean Code.
Choose where your tests start and end
The next question is always: fine, but what on earth do I test? Picture a typical server, simplified: a controller, application services, maybe domain services, the model, and repositories. Remember that in DDD the repository contract belongs to the domain, but its implementation doesn’t. Here’s a simple heuristic: pick the segment you want to test. If, for a given case, it makes sense to check that everything from here to there does what it should, that’s your test.
Some people in .NET test from the controller class: they new it up, pass something in and check what comes back. The problem is that serialization and security aren’t configured in the controller. They’re configured in the pipeline around it. You can break an entire API just by changing how dates are serialized, and if you never test through the pipeline, you won’t notice. That doesn’t mean every test should go through HTTP. Write one test that hits a controller with everything on, so it exposes configuration problems, and start the rest further in.
We stopped testing from the controller altogether. Today most of our controllers are a couple of lines: receive the request, build a command (one of those typical command classes) and let the domain do its thing. Testing them didn’t give us anything, so we test from the commands instead. Is it better? Worse? It’s what works for us. Anecdotal evidence again.
Sometimes a controller does have its own logic: a validation that belongs to the application layer rather than the business, or a transformation, like taking a full date and keeping only the day. Then we test that piece, and only that piece, in the controller. The rest is covered from the command inwards. The feedback loop stays short, the test stays a unit test, and anyone reading it sees what it’s about: the only thing that mattered in that controller was that validation. This is the hard part. You need some experience with testing to slice like this, but it helps us a lot.
Tests should speak the business
FizzBuzz is the simplest kata used to teach the TDD flow, and it has no business content at all. Given a number from 1 to 100, return it as a string, except multiples of 3, which become Fizz, multiples of 5, which become Buzz, and multiples of both, which become FizzBuzz. I also use it to show how many chances we waste, even in something this simple, to bring the code closer to the business.
That matters a lot to me. At AIDA, when someone asks me to review code, newcomers send me straight to the production code. I tell them: no, take me to the tests. I want to see what someone facing a problem in this code will find when they open it. I want to see if your code speaks the language of the business, or if it talks about technical things only you understand. I want to be able to discuss a business rule with a product owner through a test.
A very common way to test FizzBuzz is a set of examples, inputs and expected outputs. Some people would even put them in a text file:
[Theory]
[InlineData(1, "1")]
[InlineData(3, "Fizz")]
[InlineData(5, "Buzz")]
[InlineData(15, "FizzBuzz")]
public void calculate(int number, string expected)
{
FizzBuzz.Calculate(number).Should().Be(expected);
}It’s clever. Not a single repeated line. Now run it and read the output. Can you infer the business rules from it? Here, yes, because they’re trivial. Imagine a business that’s a bit more complex. Business rules shouldn’t have to be inferred. They should be read.
public class FizzBuzzShould
{
[Fact]
public void return_the_same_number_by_default() =>
FizzBuzz.Calculate(1).Should().Be("1");
[Fact]
public void return_fizz_when_divisible_by_3() =>
FizzBuzz.Calculate(3).Should().Be("Fizz");
[Fact]
public void return_buzz_when_divisible_by_5() =>
FizzBuzz.Calculate(5).Should().Be("Buzz");
[Fact]
public void return_fizzbuzz_when_divisible_by_3_and_5() =>
FizzBuzz.Calculate(15).Should().Be("FizzBuzz");
}Anyone who isn’t allergic to curly braces can read this: “FizzBuzz should return Fizz when divisible by 3.” Our product owners have some coding experience, but they can read it, and anyone new to the team understands the rules straight away.
The production code should ooze business too. Why write a bare if (number % 3 == 0) when you can ask whether it’s fizz? In .NET, an extension method on int gives you the semantics of divisible by:
public static string Calculate(int number)
{
if (IsFizzBuzz(number)) return "FizzBuzz";
if (IsFizz(number)) return "Fizz";
if (IsBuzz(number)) return "Buzz";
return number.ToString();
}
private static bool IsFizz(int number) => number.IsDivisibleBy(3);It’s getting a bit fancy, I know. But divisible by matters to this business, so it belongs in the code. If you want to see the opposite, look up Enterprise FizzBuzz: the kata solved the way we wrote Java at the end of the nineties, with abstract factories and factory implementations everywhere. A complete parody.
Legacy code: trust what runs
The Gilded Rose kata is set in an inn with a shop, World of Warcraft style. Items have a number of days to be sold and a quality, and quality drops as time passes until it reaches zero. Except that some items behave differently. Aged Brie gets better with age instead of worse. A Backstage pass is worthless once the concert has passed. Sulfuras is legendary and never has to be sold. The kata exists in about eight thousand programming languages.
It hands you a pile of nested ifs with no tests. “If it isn’t Aged Brie, and it isn’t a backstage pass, do this; but if it also isn’t Sulfuras, do that; and further down, ask again whether it’s Sulfuras.” Garbage. Recently I found something like this in a bank’s code, with nested try/catch four levels deep instead of ifs. I’d always thought the kata was exaggerating.
The task: add a new kind of item. Since there are no tests, first build a test suite, then leave the code in a state where you can change it decently. Not one more if. Your next teammate could be a psychopath, and it might be your name that shows up when they run git blame.
In my TDD courses we do this kata on the second day, and I always make a bet: I’ll buy a round for whoever, once their tests are in place, refactors the code into something decent without ever turning a test red. Usually 11 or 12 tests are enough to give real confidence. I’ve run that course a dozen times or more, and nobody has won. It’s a bit of a trap: they’re still learning TDD, so they don’t have the tools from this post yet.
Which tests?
The usual strategy is to take the requirements document and write one test per requirement. Everyone’s happy until the test for “Sulfuras, being legendary, never decreases in quality” fails: the item ages anyway. People get stuck and tell me the kata is broken. Go to the code. There’s no item called Sulfuras. It’s called Sulfuras, Hand of Ragnaros. You’ve been reading a functional document that was archived when the code was written, and you’re stuck on something that silly.
First lesson: what’s in production is the code. The code is what runs. Can it have bugs? Absolutely. But with code like this, imagine how far the specification and the code have drifted apart. And if the document can’t be trusted, a test suite built from it can’t give you much confidence either.
Someone at the talk suggested counting the paths through the code (its cyclomatic complexity) and writing a test for each one. That works. Another option is the golden master. We’re great at grand names, if nothing else: it comes from the vinyl era, when a master record was made and taken to the plant to press all the copies.
The code is in production, running. Great. Log the input and output of every real call: Sulfuras came in with this age and this quality and left like this; a backstage pass came in like this and left like that. Here we can describe the whole thing as “something with these properties goes in, something with those comes out”, which makes it easy. How much data do you need to be confident? It depends on your business: a week, a month, six months. Maybe you already have last year’s logs. Turn them into a file of examples, and that’s your acceptance test suite.
Notice the irony. In FizzBuzz I showed a table of inputs and outputs as the bad example. In this context, the same thing in a text file is the good one: it gives you enormous confidence that the rules you’re preserving are the real ones, the ones running in production, and the feedback when you break something is immediate. You’ll still want tests that speak the business and state its rules, but the golden master is what lets you move.
Refactoring with the IDE and code coverage
With a suite you trust, people start refactoring and suffer. They split ifs, extract methods, and everything is tangled. But you don’t need to be a hero. Refactoring techniques are worth learning just like testing techniques, and the IDE already knows a lot of them.
Start with the boy scout rule. Let the IDE convert the for over items[i] into a foreach over item, then extract the body of the loop into a method called Age. The code is already cleaner than before. Enough? No.
One rule of the kata is that you can’t change the Item class, because it’s used in production; open/closed principle. So add your own concept on top of it, an item that can degrade, and let the IDE generate most of it (I’m lazy, and I’m a CTO, so I don’t write the code):
public void UpdateQuality()
{
foreach (var item in items)
{
DegradableItemFactory.Create(item).Age();
}
}
public interface IDegradableItem
{
void Age();
}
public static class DegradableItemFactory
{
public static IDegradableItem Create(Item item) =>
new RegularItem(item);
}At first, RegularItem.Age() is the old method, pasted as is, with every if. Run the tests: still green. “Fine, Modesto, but you still have that mess of ifs.” Wait.
Copy RegularItem into a new class, AgedBrie, without changing a line, and make the factory return it when the name matches:
public static IDegradableItem Create(Item item) =>
item.Name switch
{
"Aged Brie" => new AgedBrie(item),
_ => new RegularItem(item),
};The tests are still green, of course. Now you have a lot of duplicated garbage. But you also have a tool most people forget: code coverage. Run the tests with coverage and open AgedBrie. The IDE shows you which lines run and which don’t. Delete everything that doesn’t run, and also the checks that are now always true, like asking whether the name is Aged Brie. What’s left is a couple of lines. Do the same for each kind of item.
Coverage tells you something else, too. Run it on the original code before touching anything and there’s already a branch that never runs. That means one of two things: your tests are weak, or that code never runs. Here, it never runs.
A monkey could do this refactor. Don’t pay it in peanuts, though; we want it to grow. People go crazy trying to refactor this kata, and something as simple as code coverage solves it.
TDD isn’t a testing technique
It isn’t always about TDD. There are many testing techniques, and TDD isn’t even one of them: it’s a design technique. Use it in your projects, of course. Above all, though, know the whole spectrum, so you can see what options you have, and talk with your client in business and economic terms. You want this to grow? Then you need this. You don’t care whether it grows? Then go the other way.
If you want to think about software development in economic terms, beyond the technical side, look for the work of Dr. Luis Artola on software economics. He was writing a book on it at the time, and I recommend it.