Tutorial: Integrating Knative Eventing with Google Cloud Pub/Sub

Having experimented with Knative Serving in the last part of the series, let’s see Knative Eventing in action.
This tutorial will show you how to consume messages sent to a Google Cloud Pub/Sub topic within a Kubernetes Pod.
Setting up the Environment
If you have provisioned a GKE cluster with Istio, you can skip this step. Otherwise, go ahead and run the below commands to launch the cluster.
1 2 |
export CLUSTER_NAME=mi2-knative export CLUSTER_ZONE=asia-south1-a |
Change the cluster name and GCP zone to your choice.
1 2 3 4 5 |
gcloud services enable \ cloudapis.googleapis.com \ container.googleapis.com \ pubsub.googleapis.com \ containerregistry.googleapis.com |
The above commands will enable appropriate APIs.
1 2 3 4 5 6 7 8 |
gcloud beta container clusters create $CLUSTER_NAME \ --addons=HorizontalPodAutoscaling,HttpLoadBalancing,Istio \ --machine-type=n1-standard-4 \ --cluster-version=latest --zone=$CLUSTER_ZONE \ --enable-stackdriver-kubernetes --enable-ip-alias \ --enable-autoscaling --min-nodes=1 --max-nodes=10 \ --enable-autorepair \ --scopes cloud-platform |
Notice that the Istio addon is enabled.
1 2 3 |
kubectl create clusterrolebinding cluster-admin-binding \ --clusterrole=cluster-admin \ --user=$(gcloud config get-value core/account) |
The above step will add the current user to the cluster-admin role.
You should now have a three-node GKE cluster with Istio preinstalled.
Since we also need a Cloud Pub/Sub topic for this tutorial, let’s go ahead and create it.
1 2 |
export TOPIC_NAME=testing gcloud pubsub topics create $TOPIC_NAME |
Installing Knative on GKE
Knative comes as a set of Custom Resource Definitions (CRDs). We will first deploy the CRDs followed by the rest of the objects.
1 2 3 4 |
kubectl apply --selector knative.dev/crd-install=true \ --filename https://github.com/knative/serving/releases/download/v0.9.0/serving.yaml \ --filename https://github.com/knative/eventing/releases/download/v0.9.0/release.yaml \ --filename https://github.com/knative/serving/releases/download/v0.9.0/monitoring.yaml |
1 2 3 |
kubectl apply --filename https://github.com/knative/serving/releases/download/v0.9.0/serving.yaml \ --filename https://github.com/knative/eventing/releases/download/v0.9.0/release.yaml \ --filename https://github.com/knative/serving/releases/download/v0.9.0/monitoring.yaml |
After a few minutes, Knative Eventing will be ready. Wait until you see that all deployments in the knative-eventing namespace are ready.
Installing GCP Specific CRDs in Knative
Google has built a set of CRDs that connect GCP resources such as Cloud Storage and Pub/Sub with Knative Eventing. These CRDs deliver messages when events are raised by the services within GCP. Subscribers running in Kubernetes receive these messages in a format based on the Cloud Event specification. The Knative Eventing component acts as the conduit between external GCP publishers and internal subscribers running in Kubernetes.
In order to connect Knative Eventing to GCP, we need to create a service account bound to an IAM Role with sufficient permissions to retrieve the messages. This step results in a JSON file that has the key to access Cloud Pub/Sub programmatically.
Run the below commands to create a service account and download the JSON key file associated with it.
1 2 3 |
gcloud projects add-iam-policy-binding $PROJECT_ID \ --member=serviceAccount:knative-source@$PROJECT_ID.iam.gserviceaccount.com \ --role roles/pubsub.editor |
1 2 |
gcloud iam service-accounts keys create knative-source.json \ --iam-account=knative-source@$PROJECT_ID.iam.gserviceaccount.com |
1 |
Let’s create a Kubernetes secret from the downloaded JSON file. |
1 2 |
kubectl -n default create secret generic google-cloud-key \ --from-file=key.json=knative-source.json |
Finally, let’s deploy the Knative-specific GCP resources in the Kubernetes cluster.
1 |
kubectl apply -f https://github.com/google/knative-gcp/releases/download/v0.9.0/cloud-run-events.yaml |
This step results in the creation of a namespace called cloud-run-events with all the CRDs.
At this point, we have a Kubernetes cluster with all the appropriate resources (CRDs) to consume events raised by GCP services.
Subscribing to a Google Cloud Pub/Sub Topic from Knative
We will now create a PullSubscription resource that listens for new messages published to a specific Cloud Pub/Sub topic. This is a Knative GCP CRD installed in the previous step.
1 2 3 4 5 6 7 8 9 10 11 12 13 14 |
apiVersion: pubsub.cloud.run/v1alpha1 kind: PullSubscription metadata: name: testing-source spec: topic: testing sink: apiVersion: v1 kind: Service name: event-display project: janakiramm-sandbox secret: name: google-cloud-key key: key.json |
The above YAML file has all the necessary ingredients to pull the messages from GCP Cloud Pub/Sub topic, and forwarding the messages to a service named event-display. The secret will enable the resource to programmatically talk to the GCP API.
Now, let’s create a Kubernetes deployment that receives the messages from the Pub/Sub topic via the PullSubscriber. We will expose this as a service to enable PullSubscriber to invoke an endpoint with the Cloud Events payload.
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 |
apiVersion: apps/v1 kind: Deployment metadata: name: event-display spec: selector: matchLabels: app: event-display template: metadata: labels: app: event-display spec: containers: - name: user-container image: gcr.io/knative-releases/github.com/knative/eventing-contrib/cmd/event_display ports: - containerPort: 8080 --- apiVersion: v1 kind: Service metadata: name: event-display spec: selector: app: event-display ports: - protocol: TCP port: 80 targetPort: 8080 |
If you are interested in looking at the source code of the subscriber written in Go, it’s available here.
It’s time to deploy both the YAML files to the Kubernetes cluster.
1 2 |
kubectl apply -f pullsubscription.yaml kubectl apply -f subscriber.yaml |
Let’s watch the logs to see the messages for the subscriber.
1 |
kubectl logs -f --selector app=event-display -c user-container |
Launch another terminal window and publish messages to the Google Cloud Pub/Sub topic. Don’t forget to set appropriate environment variables.
1 2 |
export TOPIC_NAME=testing gcloud pubsub topics publish $TOPIC_NAME --message='{"msg": "Hello Knative"}' |
You should see the message in the logs of the subscriber.
By combining Knative Eventing and Knative Serving, we can develop and deploy event-driven serverless applications in Kubernetes.
Feature image via Pixabay.