A few days ago I gave a talk at work about events. I opened it with a joke about Sheldon Cooper’s Fun with Flags: this was going to be Fun with Events.
The material came from a workshop I have run a few times about working with messages in distributed systems. There was no time for the workshop, so I went through the slides quickly and let the conversation go wherever it wanted. My goal was not to explain everything but to surface a few unknown unknowns: to show how many questions you end up asking yourself as soon as you start designing with events.
So, I thought it might be interesting to write up some of the content from that workshop in a series of posts and have a bit of Fun with Events.
This series is the long version of this talk. It includes the parts I skipped because the audience already knew them, a few topics that did not fit in the talk at all, and some of the best ideas that came from the people in the room, credited to them.
The classic way
Picture an online shop. It will follow us through the whole series. A shopper browses the website, puts a few products in the cart and checks out. Behind the website there are a few services, each owned by its own team: Cart keeps the shopper’s cart and handles the checkout, Orders turns a checked-out cart into an order, Billing takes the payment through an external payment provider, and Notifications sends emails through an external mail service.
The classic way to wire them is a chain of calls. When the shopper checks out, Cart calls Orders to place the order. Orders calls Billing to take the payment, and then Notifications to send the confirmation email. Each one waits for the answer of the next, and the shopper waits for all of them.
Look at what the shopper got: an error, and a charge on their card. Will they try again and pay twice? And all of it because of an email that nobody needed before the order was placed.
Every arrow in that chain is a synchronous call, and every call is a dependency. If any of those pieces breaks, the whole flow stops. If any of them saturates, the whole flow slows down with it. And because everything is wired to everything else, evolving the system gets rigid: changing one piece means coordinating with all the others.
Compare that with working on something small that you own end to end. You move fast. Much of the appeal of events is getting some of that speed back.
Three different reasons
“Let’s use events” can mean very different things, and it pays to be honest about which one you mean.
Lower coupling. Sometimes I want events only to reduce coupling, and performance does not matter at all. Cart announces that a cart was checked out. Orders reacts and places an order. Billing reacts to the order and charges the card. Notifications reacts to the payment and sends the email. Each of them publishes what happened to a topic, a channel that anyone interested can subscribe to, and none of them calls the next one. Adding a new reaction does not require touching the service that started the story.
There is a second gain hidden in that diagram. Because nobody waits for the next piece, a failure stays where it happens: if the mail service goes down, shoppers can still buy, and the emails go out when it comes back. That is called temporal decoupling. Payments are a harder case. Should the shop accept an order before the card is charged? That is not a technical question, and it gets a post of its own.
Scalability. Other times what I want is to scale. Think of Black Friday. When many instances of Cart publish into the same channel and each consumer can grow on its own, every piece scales with its own load instead of the load of the whole chain.
Reliability. And sometimes it is not about scaling at all. Someone is hammering me with requests and I just want to survive it. My budget is fixed; I do not want more machines. In the shop, think of the warehouse: an old system that takes ten orders per second, and nobody is going to rewrite it for Black Friday. I want to know that the orders arriving at a thousand per second will be processed at ten per second, eventually. A queue between Orders and the warehouse absorbs the peak.
A queue only absorbs peaks. If a thousand orders per second keep arriving and the warehouse can only take ten, the queue grows forever: the warehouse has to keep up on average, or Orders has to slow down.
These reasons lead to different designs. A decision that makes sense for absorbing load can be pointless for decoupling, and the other way around.
And many others
Over time, many more uses have emerged. Systems that work closer to real time. Audit trails, once you start persisting what happened. Metrics. The UI world has adopted events extensively, too.
Two more deserve a mention. An event log can become the single source of truth, the record from which every other view is derived. And messaging can replace synchronous calls even when you need an answer, with request/reply patterns over a broker.
The benefits usually quoted are low coupling, integration, scalability and something I like to call free audit: if you keep the events, you keep the history.
There is always coupling
Here is the first warning: events do not remove coupling, and nobody should sell you that idea.
When I call an API, I am coupled to that API: to its endpoint and to the schemas of its requests and responses. When I publish an event, I am no longer coupled to the service on the other side, but I am coupled to a broker and to a set of schemas. What we try to do is reduce that coupling as much as possible, and choose a kind of coupling we can live with.
It helps to name the kinds. Events remove temporal coupling, because nobody waits for anybody, and location coupling, because nobody needs to know where the others are. They do not remove the coupling to the schema, or to the meaning of the story.
That shift is real and valuable. It is also a trade, not a gift.
Pick the scenario before the technology
Because the reasons differ, the first question is which scenario you are in:
- Horizontal scalability: absorbing load, or scaling pieces independently (Black Friday checkouts).
- External integration: telling the world that something happened, so that others, inside or outside the company, can react to it (telling the carrier that an order is ready to ship).
- Workflow and process management: a complex process whose steps you want decoupled from each other, with its own coordination strategies (from checkout to delivery). Choreography and orchestration get a post of their own later in the series.
- State transfer: keeping other systems informed about changes in yours (a new price that the website and the search must show).
The reasons say what you want to gain, and the scenarios say where you are going to use events. Horizontal scalability serves scale and reliability; external integration, workflows and state transfer are mostly about coupling.
Using an event to scale horizontally is not the same as using it to tell the world a story so that the world can react. The first is a delivery mechanism. The second is a contract with an audience you may not even know. Most of this series is about the second case.
Why not?
Because it makes your life much harder.
That was my honest answer in the talk, and it is the reason this series exists. Sometimes the answer cannot be events, however much I like them, because they introduce a level of complexity that someone has to manage.
A synchronous call fails right in front of you. A message can be lost, delivered twice or delivered out of order. Data becomes eventually consistent instead of immediately consistent. Contracts outlive the code that created them. Understanding what happened means following a trail across several services. None of these problems is unsolvable, and every one of them is a question you did not have to ask before.
A synchronous call is usually the better choice when:
- you need the answer to carry on, like checking stock before the shopper can check out;
- the user expects to see the result immediately and consistently;
- one team owns the whole flow and deploys it as a single piece;
- it is a simple CRUD, and nobody else cares when it changes.
Messages, not only events
One last framing before we start. I prefer to talk about messages in distributed systems rather than events. A queue is a delivery mechanism, and the same infrastructure that carries events also carries commands. Knowing which of the two you are sending changes almost every design decision that follows, so that is where the next post begins.
Questions before you start
- Which of the three reasons is yours: coupling, scale or reliability? And which scenario are you in?
- What will you be coupled to instead, and can you live with it?
- Who will pay for the complexity: operations, monitoring, failure handling, versioning?
- Can the business live with this data being eventually consistent?
- Is this the most critical process you have? If it is, it is probably not the place to start learning.