Tutorial: Integrating Knative Eventing with Google Cloud Pub/Sub

Having experimented with Knative Serving in the last part of the series, let’s see Knative Eventing in action.

This tutorial will show you how to consume messages sent to a Google Cloud Pub/Sub topic within a Kubernetes Pod.

Setting up the Environment

If you have provisioned a GKE cluster with Istio, you can skip this step. Otherwise, go ahead and run the below commands to launch the cluster.

Change the cluster name and GCP zone to your choice.

The above commands will enable appropriate APIs.

Notice that the Istio addon is enabled.

The above step will add the current user to the cluster-admin role.

You should now have a three-node GKE cluster with Istio preinstalled.

Since we also need a Cloud Pub/Sub topic for this tutorial, let’s go ahead and create it.

Installing Knative on GKE

Knative comes as a set of Custom Resource Definitions (CRDs). We will first deploy the CRDs followed by the rest of the objects.

After a few minutes, Knative Eventing will be ready. Wait until you see that all deployments in the knative-eventing namespace are ready.

Installing GCP Specific CRDs in Knative

Google has built a set of CRDs that connect GCP resources such as Cloud Storage and Pub/Sub with Knative Eventing. These CRDs deliver messages when events are raised by the services within GCP. Subscribers running in Kubernetes receive these messages in a format based on the Cloud Event specification. The Knative Eventing component acts as the conduit between external GCP publishers and internal subscribers running in Kubernetes.

In order to connect Knative Eventing to GCP, we need to create a service account bound to an IAM Role with sufficient permissions to retrieve the messages. This step results in a JSON file that has the key to access Cloud Pub/Sub programmatically.

Run the below commands to create a service account and download the JSON key file associated with it.

Finally, let’s deploy the Knative-specific GCP resources in the Kubernetes cluster.

This step results in the creation of a namespace called cloud-run-events with all the CRDs.

At this point, we have a Kubernetes cluster with all the appropriate resources (CRDs) to consume events raised by GCP services.

Subscribing to a Google Cloud Pub/Sub Topic from Knative

We will now create a PullSubscription resource that listens for new messages published to a specific Cloud Pub/Sub topic. This is a Knative GCP CRD installed in the previous step.

The above YAML file has all the necessary ingredients to pull the messages from GCP Cloud Pub/Sub topic, and forwarding the messages to a service named event-display. The secret will enable the resource to programmatically talk to the GCP API.

Now, let’s create a Kubernetes deployment that receives the messages from the Pub/Sub topic via the PullSubscriber. We will expose this as a service to enable PullSubscriber to invoke an endpoint with the Cloud Events payload.

If you are interested in looking at the source code of the subscriber written in Go, it’s available here.

It’s time to deploy both the YAML files to the Kubernetes cluster.

Let’s watch the logs to see the messages for the subscriber.

Launch another terminal window and publish messages to the Google Cloud Pub/Sub topic. Don’t forget to set appropriate environment variables.

You should see the message in the logs of the subscriber.

By combining Knative Eventing and Knative Serving, we can develop and deploy event-driven serverless applications in Kubernetes.

Feature image via Pixabay.