Publish & subscribe messaging on Dapr Docs

Publish and subscribe overview

Mon, 01 Jan 0001 00:00:00 +0000

Publish and subscribe (pub/sub) enables microservices to communicate with each other using messages for event-driven architectures.

The producer, or publisher, writes messages to an input channel and sends them to a topic, unaware which application will receive them.
The consumer, or subscriber, subscribes to the topic and receives messages from an output channel, unaware which service produced these messages.

An intermediary message broker copies each message from a publisher’s input channel to an output channel for all subscribers interested in that message. This pattern is especially useful when you need to decouple microservices from one another.

How to: Publish a message and subscribe to a topic

Mon, 01 Jan 0001 00:00:00 +0000

Now that you’ve learned what the Dapr pub/sub building block provides, learn how it can work in your service. The below code example loosely describes an application that processes orders with two services, each with Dapr sidecars:

A checkout service using Dapr to subscribe to the topic in the message queue.
An order processing service using Dapr to publish a message to RabbitMQ.

Dapr automatically wraps the user payload in a CloudEvents v1.0 compliant envelope, using Content-Type header value for datacontenttype attribute. Learn more about messages with CloudEvents.

Publishing & subscribing messages with Cloudevents

Mon, 01 Jan 0001 00:00:00 +0000

To enable message routing and provide additional context with each message, Dapr uses the CloudEvents 1.0 specification as its message format. Any message sent by an application to a topic using Dapr is automatically wrapped in a CloudEvents envelope, using the Content-Type header value for datacontenttype attribute.

Dapr uses CloudEvents to provide additional context to the event payload, enabling features like:

Tracing
Content-type for proper deserialization of event data
Verification of sender application

You can choose any of three methods for publish a CloudEvent via pub/sub:

Publishing & subscribing messages without CloudEvents

Mon, 01 Jan 0001 00:00:00 +0000

When adding Dapr to your application, some services may still need to communicate via pub/sub messages not encapsulated in CloudEvents, due to either compatibility reasons or some apps not using Dapr. These are referred to as “raw” pub/sub messages. Dapr enables apps to publish and subscribe to raw events not wrapped in a CloudEvent for compatibility and to send data that is not JSON serializable.

Publishing raw messages

Dapr apps are able to publish raw events to pub/sub topics without CloudEvent encapsulation, for compatibility with non-Dapr apps.

How-To: Route messages to different event handlers

Mon, 01 Jan 0001 00:00:00 +0000

Pub/sub routing is an implementation of content-based routing, a messaging pattern that utilizes a DSL instead of imperative application code. With pub/sub routing, you use expressions to route CloudEvents (based on their contents) to different URIs/paths and event handlers in your application. If no route matches, then an optional default route is used. This proves useful as your applications expand to support multiple event versions or special cases.

While routing can be implemented with code, keeping routing rules external from the application can improve portability.

Declarative, streaming, and programmatic subscription types

Mon, 01 Jan 0001 00:00:00 +0000

Pub/sub API subscription types

Dapr applications can subscribe to published topics via three subscription types that support the same features: declarative, streaming and programmatic.

Subscription type	Description
Declarative	Subscription is defined in an external file. The declarative approach removes the Dapr dependency from your code and allows for existing applications to subscribe to topics, without having to change code.
Streaming	Subscription is defined in the application code. Streaming subscriptions are dynamic, meaning they allow for adding or removing subscriptions at runtime. They do not require a subscription endpoint in your application (that is required by both programmatic and declarative subscriptions), making them easy to configure in code. Streaming subscriptions also do not require an app to be configured with the sidecar to receive messages.
Programmatic	Subscription is defined in the application code. The programmatic approach implements the static subscription and requires an endpoint in your code.

The examples below demonstrate pub/sub messaging between a checkout app and an orderprocessing app via the orders topic. The examples demonstrate the same Dapr pub/sub component used first declaratively, then programmatically.

Dead Letter Topics

Mon, 01 Jan 0001 00:00:00 +0000

Introduction

There are times when applications might not be able to handle messages for a variety of reasons. For example, there could be transient issues retrieving data needed to process a message or the app business logic fails returning an error. Dead letter topics are used to forward messages that cannot be delivered to a subscribing app. This eases the pressure on app by freeing them from dealing with these failed messages, allowing developers to write code that reads from the dead letter topic and either fixes the message and resends this, or abandons it completely.

How to: Set up pub/sub namespace consumer groups

Mon, 01 Jan 0001 00:00:00 +0000

You’ve set up Dapr’s pub/sub API building block, and your applications are publishing and subscribing to topics smoothly, using a centralized message broker. What if you’d like to perform simple A/B testing, blue/green deployments, or even canary deployments for your applications? Even with using Dapr, this can prove difficult.

Dapr solves multi-tenancy at-scale with its pub/sub namespace consumer groups construct.

Without namespace consumer groups

Let’s say you have a Kubernetes cluster, with two applications (App1 and App2) deployed to the same namespace (namespace-a). App2 publishes to a topic called order, while App1 subscribes to the topic called order. This will create two consumer groups, named after your applications (App1 and App2).

How to: Horizontally scale subscribers with StatefulSets

Mon, 01 Jan 0001 00:00:00 +0000

Unlike Deployments, where Pods are ephemeral, StatefulSets allows deployment of stateful applications on Kubernetes by keeping a sticky identity for each Pod.

Below is an example of a StatefulSet with Dapr:

apiVersion: apps/v1
kind: StatefulSet
metadata:
 name: python-subscriber
spec:
 selector:
 matchLabels:
 app: python-subscriber  # has to match .spec.template.metadata.labels
 serviceName: "python-subscriber"
 replicas: 3
 template:
 metadata:
 labels:
 app: python-subscriber # has to match .spec.selector.matchLabels
 annotations:
 dapr.io/enabled: "true"
 dapr.io/app-id: "python-subscriber"
 dapr.io/app-port: "5001"
 spec:
 containers:
 - name: python-subscriber
 image: ghcr.io/dapr/samples/pubsub-python-subscriber:latest
 ports:
 - containerPort: 5001
 imagePullPolicy: Always

When subscribing to a pub/sub topic via Dapr, the application can define the consumerID, which determines the subscriber’s position in the queue or topic. With the StatefulSets sticky identity of Pods, you can have a unique consumerID per Pod, allowing each horizontal scale of the subscriber application. Dapr keeps track of the name of each Pod, which can be used when declaring components using the {podName} marker.

Scope Pub/sub topic access

Mon, 01 Jan 0001 00:00:00 +0000

Introduction

Namespaces or component scopes can be used to limit component access to particular applications. These application scopes added to a component limit only the applications with specific IDs to be able to use the component.

In addition to this general component scope, the following can be limited for pub/sub components:

Which topics can be used (published or subscribed)
Which applications are allowed to publish to specific topics
Which applications are allowed to subscribe to specific topics

This is called pub/sub topic scoping.

Message Time-to-Live (TTL)

Mon, 01 Jan 0001 00:00:00 +0000

Introduction

Dapr enables per-message time-to-live (TTL). This means that applications can set time-to-live per message, and subscribers do not receive those messages after expiration.

All Dapr pub/sub components are compatible with message TTL, as Dapr handles the TTL logic within the runtime. Simply set the ttlInSeconds metadata when publishing a message.

In some components, such as Kafka, time-to-live can be configured in the topic via retention.ms as per documentation. With message TTL in Dapr, applications using Kafka can now set time-to-live per message in addition to per topic.

Publish and subscribe to bulk messages

Mon, 01 Jan 0001 00:00:00 +0000

With the bulk publish and subscribe APIs, you can publish and subscribe to multiple messages in a single request. When writing applications that need to send or receive a large number of messages, using bulk operations allows achieving high throughput by reducing the overall number of requests between the Dapr sidecar, the application, and the underlying pub/sub broker.

Publishing messages in bulk

Restrictions when publishing messages in bulk

The bulk publish API allows you to publish multiple messages to a topic in a single request. It is non-transactional, i.e., from a single bulk request, some messages can succeed and some can fail. If any of the messages fail to publish, the bulk publish operation returns a list of failed messages.