Cloud Pub/Sub – Google Cloud Platform (GCP)
Pub Sub is a fully managed real-time messaging service that can send and recieve messages between independent applications.
There are several key concepts to pub/sub:
- Message: the data that moves through the service.
- Topic: a named entity that represents a feed of messages.
- Subscription: a named entity that represents an interest in receiving messages on a particular topic.
- Publisher (also called a producer): creates messages and sends (publishes) them to the messaging service on a specified topic.
- Subscriber (also called a consumer): receives messages on a specified subscription
Benefits of Cloud pub/sub
- Scales globally
- Low latency
- Durability – replicated storage messages
- End-to-end reliability via application ACKs
- Security – encrpytion in motion and at rest
- Maintenance free
How Cloud Pub/Sub works
Cloud Pub/Sub uses two levels of indirection between the publisher (producer) and the subscriber (consumer). This decouples the sender’s transmission of the message from the receiver’s receipt of the message.
- A publisher wishes to publish a message. A message is simply data in transit through the system.
- The message consists of a payload and optional attributes that describe the payload.
- The message is published to a specific topic. A topic is a feed of messages.
- The topic stores the message, ensuring availability and reliability.
- The message is transmitted to one or more subscriptions. A subscription is an entity that represents interest in receiving messages.
- The subscription determines which subscribers are registered to receive the message, and queues up the message to be sent.
- Subscribers can either receive the message through pull or push.The subscribers either receive messages by Pub/Sub pushing them to the subscriber’s endpoint, or by pulling them from the service.Pull subscribers use HTTPS requests to google APIs. Push subscribers use Webhook endpoints that can accept POST requests over HTTPS
- The message arrives at the subscriber, where it is consumed.
- After the message is consumed, the subscriber sends an acknowledgement (ACK) to the subscription.
- The subscription registers each delivery. When all of the deliveries are complete, it removes the message from the queue.
Publishers and Subscribers
- Two publishers publish messages on a single topic. Publisher 1 sends message A, Publisher 2 sends message B.
- The topic aggregates their messages. The messages are stored before they are delivered.
- The messages are sent to specific subscriptions. In this example there are two subscriptions, 1 and 2.
- Each subscription forwards the information to the Subscriber — and only the information requested.
- Subscriber 1 gets message “A”, subscriber 2 get message “B”.
- Subscriber 3 gets both messages “A” and “B”.
- For each message received, the subscriber sends an acknowledgement (ACK).
- The subscription responds to the ACK by removing the message from the queue.
How Cloud Pub/Sub integrates
When to use Pub/Sub?
- Balancing workloads in network clusters. For example, a large queue of tasks can be efficiently distributed among multiple workers, such as Google Compute Engine instances.
- Implementing asynchronous workflows. For example, an order processing application can place an order on a topic, from which it can be processed by one or more workers.
- Distributing event notifications. For example, a service that accepts user signups can send notifications whenever a new user registers, and downstream services can subscribe to receive notifications of the event.
- Refreshing distributed caches. For example, an application can publish invalidation events to update the IDs of objects that have changed.
- Logging to multiple systems. For example, a Google Compute Engine instance can write logs to the monitoring system, to a database for later querying, and so on.
- Data streaming from various processes or devices. For example, a residential sensor can stream data to backend servers hosted in the cloud.
- Reliability improvement. For example, a single-zone Compute Engine service can operate in additional zones by subscribing to a common topic, to recover from failures in a zone or region.