Queues, streams and the consumers that read them

This post is part of Working with events, a series about messages in distributed systems.

So far the series has been about why you would use events at all and what they cost, and then about the stories themselves: commands and events, and who owns each one. This post is about the pipes those stories travel through, and about the consumers waiting at the other end.

There are two parts. The first is about where messages live: a queue delivers a message and forgets it, a stream keeps it, and that difference decides who keeps track of what has been read, how long the data lives and what you can build on top of it. The second is about how messages are read: by one consumer, by competing consumers or by partitions with a key, and what each choice does to order, concurrency and scale.

One thread runs through both: the consumption patterns. The broker you choose decides which patterns you can have, and the pattern you choose decides which properties you keep and which ones you give up.

Queues deliver, streams remember

From a distance, queues and streams look the same. A publisher puts messages in and consumers take them out. The difference appears after a message has been consumed: a queue forgets it, and a stream remembers it.

That sounds like an implementation detail. It is not. It decides which consumption patterns you get, who keeps track of what has been read, how long your data lives and how much trouble GDPR is going to give you.

Two different things

	Queues	Streams
What they are	Message transport	Data storage
What they guarantee	Durable	Replayable
Tracking	Inbox with tracking	Only tracking
Message lifetime	Easy	Complex
Personal data	Easy with GDPR	GDPR hell

The tracking row deserves a word. A queue works like an inbox: the broker keeps the pending messages for each consumer and tracks the state of every single one. A stream only tracks positions: the messages stay where they are, and each consumer remembers how far it has read.

And a word on vocabulary. Every broker ends up using whatever terms it likes: exchanges, topics, queues, subscriptions, channels. Someone can always get technical and point out that a queue is not, strictly speaking, what you think. I am going to simplify and call it a queue. As Matías Nabarro summarized it during the talk, the concept is simply the place where messages end up when a publisher sends them. In the previous post, commands went into a queue that only their consumer reads, and events into a topic that anyone can subscribe to. That is still the picture here, until we get to streams.

Queues: transport

As the previous posts said, a queue is a transport mechanism and nothing more, like HTTP.

What usually happens with queues is that the information disappears. If you process messages with RabbitMQ, for example, once every consumer of a message has processed it, the message is gone. Nobody expects a message that arrives in RabbitMQ today to still be there next month.

While a consumer works on a message, the broker marks it as in flight, so that no other consumer takes it. Salvador Madrid raised this during the talk: if a queue already hands each message to a single consumer, what does a stream add? The answer is that a stream can reproduce the consumption patterns of a queue and add one of its own: going back in time. What a classic stream gives up is what works message by message: individual acknowledgement and redelivery, dead-lettering by the broker, and having more consumers than partitions, the pieces a stream is split into so that it can be read in parallel. Kafka’s share groups are starting to bring some of that back. With a queue you can have:

• one consumer receiving every message, like the warehouse behind its queue in the first post;
• several consumers competing for the messages, each message going to only one of them, like the instances of Orders that scale out on Black Friday to place the checked-out carts;
• several consumers receiving every message, because each uses them for a different purpose, like Notifications and Loyalty, both listening to OrderPaid. Brokers usually build this one with a topic that copies each message into one queue per subscriber.

Every broker complicates this in its own way: you publish to an exchange, the exchange routes to two queues, and each consumer reads from one of them. It does not matter much. What you need to understand are the consumption patterns, and how the broker you actually use implements them.

Streams: storage

A stream, which is what you mostly get with Kafka, works the other way around. The broker persists the messages over time. You can configure retention periods, but reading does not remove anything. Here a topic is not a channel that copies each message into one queue per subscriber, as in the previous posts: it is the stream itself, the log that every consumer reads at its own pace. It is also common to have a single topic where a service tells every story about one thing, such as an orders topic where Orders publishes all the events about its orders, instead of one topic per event as in the previous posts. That keeps all the events about orders in one place, which matters in the second part of this post. A consumer tells Kafka that it wants to read from a topic, and Kafka keeps track: you belong to this consumer group, and this is where you were. The next time you connect, you continue from there.

Eduardo Ferro put it in very concrete terms during the talk. Picture one consumer that is an application, and another one that computes statistics. In the shop, that could be Billing and Analytics, both reading the orders topic. Each has its own identifier and its own offset, and each knows where it is. Tomorrow, without telling anybody, you create a third one, a process that builds consolidated datasets, say, and it reads the same events from the beginning. One is at 33, another one is at 0, another one somewhere else.

For me, the key is that the offset moves. With a traditional queue, a new consumer only hears about the messages published after it arrived. The past does not exist for it. With a stream you can go back in time, and you can even do it on purpose: you discover that some calculation was wrong, fix it and reprocess the history, or you run a simulation over last month’s events.

The most a queue will do is keep your messages for a while if you declared yourself as a consumer and then went offline. That period is usually configurable. It is a safety net, not a history.

It is the difference between live TV and on-demand TV.

Who keeps track

Several instances of the same consumer, started to scale it, belong to the same consumer group and split the work between them. Matías asked about this exact point, because the way Kafka tracks offsets per group is not obvious at first.

But be careful: that is Kafka. Other brokers tell you to keep track yourself. At a previous company we did exactly that. And sometimes, even when the broker tracks it for you, you want your own tracking anyway, for transactional reasons: you want to record that you processed an event in the same database transaction in which you changed your data. That is called the inbox pattern, and it comes back later in the series, in the post about delivery guarantees.

The price of remembering

The first disadvantage is obvious: a stream accumulates an enormous amount of data. The main advantage is just as obvious: a new consumer of the story can appear at any moment.

When we take that step, we stop merely delivering messages and start recording our history, for anyone who arrives later and wants to listen. That brings three new concerns.

Message lifetime. It is not the same for a message to live with us forever, for a year or for less than a month. Somebody has to decide, and the decision has consequences.

Personal data. Legally alone, GDPR forces you to care. People have the right to have their data deleted, and an append-only log is designed precisely so that nothing in it ever changes.

There are known ways out, none of them free. You can keep personal data out of the events and carry references instead. You can set retention periods and use compaction to drop old values. Or you can encrypt each person’s data with a key of their own and delete the key when they ask to be forgotten, what Mathias Verraes calls crypto-shredding.

Versioning. If anyone can consume a whole year of events, you have to be respectful with the schemas, or have mechanisms that let old events still be read properly. Evolving events comes back later in the series, in the post about the event contract.

Brokers converge

Everyone copies everyone, as Eduardo reminded us. RabbitMQ has added streams, and Kafka has added queues. What you need to know are the technological pros and cons, but above all the consumption and production patterns: fan-out, competing consumers, replay. Eduardo has written about the difference in Queues vs Distributed logs.

The line keeps moving, and some capabilities come from the libraries rather than from the broker itself. Even so, the infrastructure you choose shapes what you can do. At a previous company we built our own broker instead of using Kafka or anything similar. It was a matter of cost: we deployed on-premise, we needed the pieces of a distributed system to exchange information without being coupled to each other, and performance was not a problem. So we could afford to persist messages in SQL Server, which was already there, for free, in every installation. It was the right decision for that context, and it shaped every pattern we could use afterwards.

The bus that wanted to be smart

About twenty years ago, a fever in the world of distributed systems gave birth to the enterprise service bus. I like to ilustrate it with a picture of Cthulhu, and I would stay away from it.

The problem is that the bus becomes too intelligent and ends up holding business rules. Eduardo named the principle it breaks: smart endpoints and dumb pipes. The endpoints should be the smart ones and the pipes should be dumb, and an enterprise service bus is exactly the opposite. As he added, people start with a little logic in there, and you end up with business logic that nobody tracks and no team owns. On top of that, it couples you so much to the bus that the vendor lock-in is brutal.

Consuming events: order, concurrency and scale

The first part introduced the ways of consuming messages: one consumer, competing consumers and several independent subscribers. This part looks at what each of them does to order, concurrency and scale, and the choice depends mostly on three questions:

• How much load does a single consumer have to keep up with?
• Do priority and order matter?
• How much concurrency can your processing tolerate?

Every consumption pattern answers those questions differently, and each one gives you some properties and takes others away. I like to look at four of them: whether you are safe from concurrency problems, whether the order is respected, whether it scales, and whether you get to choose the order.

	One consumer	Competing consumers	Partitioned by key
Safe from concurrency problems	Yes	No	Yes, per key
Order respected	Yes	No	Yes, per key
Scales	No	Yes	Yes, across keys
You choose what the order means	No	No	Yes, through the key

One consumer

With a single consumer, all you get is decoupling. The order is the order of arrival and, since you are the only consumer, it is respected. There are no concurrency problems either, because nothing happens in parallel. But it does not scale, and you cannot choose the order in which you consume things.

Competing consumers

With competing consumers, each message goes to one of the consumers: if three messages arrive, maybe one consumer takes one and the other takes two. It scales, but you can no longer count on the order, and two messages about the same thing may be processed at the same time.

Whether that matters depends on the business. If Analytics only counts the orders placed each day, the order does not matter at all. But if Shipping reads OrderPlaced and OrderCancelled from the same queue, and one instance handles the cancellation of an order while another one is still handling the order itself, the cancellation finds nothing to cancel and the order gets shipped anyway. Very often, the order clearly does matter.

Time to think

So here is the challenge: how do you get all four at once? It is worth thinking about it for a moment before reading on.

The talk gave the hint. Things get more interesting because we can play with filters: within the same topic, one consumer can take the events of customer A and another one the events of customer B. They no longer compete for the same messages, but they still share the work of one topic, and the order of each customer’s events is respected.

That is the idea behind partitioning by key. Kafka does it with partitions: every message with the same key lands in the same partition, and each partition is consumed by a single member of the consumer group at a time. Other brokers offer the same idea under other names, such as sessions or message groups. You get order, and no concurrency, within each key, and you scale across keys. The price is that the number of partitions caps your parallelism, and a single very busy key cannot be spread over several consumers.

Two more things to know. When a consumer joins or leaves the group, the partitions are reassigned, and messages that were being processed can arrive again at their new owner; the post about delivery guarantees, later in the series, is about living with that. And changing the number of partitions changes which partition each key goes to, so the order per key only holds while that number stays the same.

Noisy neighbours and premium customers

Salvador Madrid pointed out a side effect: scalability, in quotes, also becomes fairer. With everything in a single pipe, one customer can suffocate all the others, the classic noisy neighbour. When the work is split by customer, a flood from one customer only slows down the customers that share its partition. Sessions or message groups, where the broker offers them, get closer to isolating each customer.

It also lets you do something else: offer premium services. The premium customer gets a partition of its own, or a bigger consumer, while regular customers share the rest. If one consumer is still not enough for it, its events need a finer key, such as the order, so that they can spread over several partitions, and the order then holds per order instead of per customer. It is used, and it works.

Some teams do it differently: one topic for premium customers and another one for everybody else, with more consumers on the first. The problem appears the day a regular customer becomes premium. All their messages are already stored in the other topic, and you are stuck. Notice how a design decision ends up limiting what the business can do. That is why it is usually much better to have few topics and play with filtering than to have lots of topics. Filtering has some cost, but it helps a lot.

Filters, priorities and other tools

Where the filter lives depends on the broker. Some, like Azure Service Bus, Amazon SNS or RabbitMQ, filter in the subscription, so a consumer only receives what it asked for. Kafka does not filter on the broker: every consumer reads the whole partition and discards what it does not need, or a stream processor writes the filtered events into a new topic. Around Kafka there is also Kafka Connect, a framework of connectors that move data in and out of it; change data capture tools such as Debezium run on it, and they come back with the outbox, in the post about delivery guarantees.

There are other usual tools, and every one of them is a trade-off:

• Priority queues: urgent messages are consumed before the rest, usually through separate queues with more consumers.
• Throttling: limiting how fast you consume, to protect a downstream system or an API with quotas.
• Sharding: splitting the work by a key, which is what partitioning does.
• Filtering: letting each consumer take only what it cares about.

If you work with Kafka, filtering in the consumer works quite well, as long as consumers that only care about a small part of a busy topic do not multiply.

The same ideas, different names

Since every broker uses its own vocabulary, here is a rough map from the ideas in this series to the names that four common brokers give them.

Idea	Kafka	RabbitMQ	Azure Service Bus	Amazon SQS and SNS
Where you publish	Topic	Exchange	Topic or queue	SNS topic or SQS queue
Where a consumer reads	Partitions of a topic	Queue	Subscription or queue	SQS queue
Independent consumers	One consumer group each	One queue each, bound to the exchange	One subscription each	One SQS queue each, subscribed to the topic
Competing consumers	Members of a group, up to one per partition; share groups lift the limit	Several consumers on one queue	Several receivers on one subscription or queue	Several consumers polling one queue
Order	Per partition	Per queue, with a single active consumer	Per session	Per message group, in FIFO queues
Confirming a message	Committing the offset	Ack or nack	Complete or abandon	Deleting it before the visibility timeout
Dead letters	Built by the consumer or its framework	Dead letter exchange	Built-in dead letter subqueue	Redrive policy to a dead letter queue
Replay	Seek to an offset, within retention	Streams	Not built in	Not for queues; archive and replay for SNS FIFO topics

The map is rough on purpose. Each broker has details that matter, and the documentation of the one you use always wins. Azure and AWS also offer streams as separate services: Event Hubs and Kinesis.

Questions to ask

• Does anyone need the history, or only what happens from now on?
• How long should a message live, and who decided it?
• Is there personal data in it, and how will you delete it?
• Who keeps track of what has been consumed: the broker or you?
• Is any business logic hiding in your pipes?
• Which consumption patterns do you need: one consumer, competing consumers, several independent ones?
• Does the order matter for this process, and in which scope: global, per customer, per entity?
• What happens if two messages about the same entity are processed at the same time?
• Which key would you partition by, and how uneven is the traffic across keys?
• Could one customer’s volume hurt everybody else?
• Would a business change, like a customer upgrading their plan, force you to move messages between topics?
• Where does your filtering happen: in the subscription, in the consumer or somewhere else?

Next in the series: designing an event, from its purpose and the way it will be consumed to the content it carries.