What is GCP Pub-Sub

Google Cloud Pub/Sub is a fully-managed real-time messaging service that allows you to send and receive messages between independent applications. It provides a flexible and reliable way to exchange data between distributed systems without the need for complex infrastructure.

Pub/Sub is a publish/subscribe messaging system, which means that messages are sent to a topic and then delivered to any number of subscribers that have subscribed to that topic. This decoupling of publishers and subscribers allows for greater scalability and reliability, as subscribers can consume messages at their own pace without affecting publishers or other subscribers.

Google Cloud Pub/Sub is often used for streaming data processing, event-driven computing, and microservices-based architectures. It integrates with other Google Cloud services such as Cloud Functions, Cloud Dataflow, and Cloud Storage, as well as with external systems via various client libraries and APIs.

Pub Sub is a fully managed real-time messaging service that can send and recieve messages between independent applications.

Pub/Sub Key Concepts

There are several key concepts to pub/sub:

There are several key concepts in Google Cloud Pub/Sub that are important to understand:

1. Topic: A named resource to which publishers send messages. A topic can have multiple subscribers.
2. Subscription: A named resource representing the stream of messages from a single, specific topic to be delivered to the subscribing application.
3. Message: A payload of bytes that can be delivered to subscribers of a topic. Messages can contain any arbitrary data, including binary data.
4. Publisher: An application or service that sends messages to a topic.
5. Subscriber: An application or service that receives messages from a subscription.
6. Push Subscription: A subscription that delivers messages to an endpoint specified by the subscriber, typically an HTTP endpoint.
7. Pull Subscription: A subscription where the subscriber polls the subscription to retrieve messages at its own pace.
8. Acknowledgment: A mechanism by which a subscriber confirms that it has successfully received and processed a message.
9. Dead Letter Topic: A topic where messages that cannot be delivered to any subscribers are sent

Benefits of Cloud pub/sub

• Scales globally
• Low latency
• Durability – replicated storage messages
• Reliability
• End-to-end reliability via application ACKs
• Security – encrpytion in motion and at rest
• Maintenance free

How Cloud Pub/Sub works

Cloud Pub/Sub uses two levels of indirection between the publisher (producer) and the subscriber (consumer). This decouples the sender’s transmission of the message from the receiver’s receipt of the message.

1. A publisher wishes to publish a message. A message is simply data in transit through the system.
2. The message consists of a payload and optional attributes that describe the payload.
3. The message is published to a specific topic. A topic is a feed of messages.
4. The topic stores the message, ensuring availability and reliability.
5. The message is transmitted to one or more subscriptions. A subscription is an entity that represents interest in receiving messages.
6. The subscription determines which subscribers are registered to receive the message, and queues up the message to be sent.
7. Subscribers can either receive the message through pull or push.The subscribers either receive messages by Pub/Sub pushing them to the subscriber’s endpoint, or by pulling them from the service.Pull subscribers use HTTPS requests to google APIs. Push subscribers use Webhook endpoints that can accept POST requests over HTTPS
8. The message arrives at the subscriber, where it is consumed.
9. After the message is consumed, the subscriber sends an acknowledgement (ACK) to the subscription.
10. The subscription registers each delivery. When all of the deliveries are complete, it removes the message from the queue.

Publishers and Subscribers

1. Two publishers publish messages on a single topic. Publisher 1 sends message A, Publisher 2 sends message B.
2. The topic aggregates their messages. The messages are stored before they are delivered.
3. The messages are sent to specific subscriptions. In this example there are two subscriptions, 1 and 2.
4. Each subscription forwards the information to the Subscriber — and only the information requested.
5. Subscriber 1 gets message “A”, subscriber 2 get message “B”.
6. Subscriber 3 gets both messages “A” and “B”.
7. For each message received, the subscriber sends an acknowledgement (ACK).
8. The subscription responds to the ACK by removing the message from the queue.

How Cloud Pub/Sub integrates

When to use Pub/Sub?

• Balancing workloads in network clusters. For example, a large queue of tasks can be efficiently distributed among multiple workers, such as Google Compute Engine instances.
• Implementing asynchronous workflows. For example, an order processing application can place an order on a topic, from which it can be processed by one or more workers.
• Distributing event notifications. For example, a service that accepts user signups can send notifications whenever a new user registers, and downstream services can subscribe to receive notifications of the event.
• Refreshing distributed caches. For example, an application can publish invalidation events to update the IDs of objects that have changed.
• Logging to multiple systems. For example, a Google Compute Engine instance can write logs to the monitoring system, to a database for later querying, and so on.
• Data streaming from various processes or devices. For example, a residential sensor can stream data to backend servers hosted in the cloud.
• Reliability improvement. For example, a single-zone Compute Engine service can operate in additional zones by subscribing to a common topic, to recover from failures in a zone or region.