How to Integrate RESTful APIs into CI/CD Pipelines
To fully automate the building, testing and deployment of modern applications, RESTful API calls are essential for information transfer between services. In this beginner-level tutorial, we explore the real-world usage of these APIs.
Modern applications rely heavily on HTTP requests to external services to GET, POST, PUT and DELETE data. This information transfer is done with RESTful or REST APIs to reduce the bandwidth associated with traditional APIs. REST APIs are essential for distributed computing and cloud-based technologies. One example is publishing packages to an AWS S3 bucket using the Amazon S3 REST API. Another is using cURL to download a package from the web to be installed.
With these examples, it becomes clear that one would need to integrate these API calls into their CI/CD pipeline for fully automated building and deployment. In this tutorial, we will build an application that will use both of these APIs. It is a simple Scala application that we will build and deploy to an S3 bucket. The goal of this post is to show real-world examples of REST API usage. Extensive knowledge of Scala is not required.
Building the App
First, let’s create a directory and cd into it by typing this command in your terminal:
|
1 |
mkdir scala-demo && cd $_ |
This is where we will add the files for our application. Now, we need to add a HelloWorld.scala file in a /src/main/ folder. Add the file by typing these two commands:
|
1 2 |
mkdir -p src/main/ && cd $_ touch HelloWorld.scala |
Open your favorite editor and save these lines to HelloWorld.scala:
|
1 2 3 4 5 |
object HelloWorld { def main(args: Array[String]): Unit = { println("Hello, World! I'm the Sample Scala App!") } } |
We need a few more files before we have a working application. Let’s define a few files for using sbt. The first is build.sbt and it needs to be in the projects root. Type these lines into your terminal:
|
1 |
cd ../.. && touch build.sbt |
Return to your editor and save these lines to the file:
|
1 2 3 4 5 6 7 8 9 10 |
name := "scala-demo" version := "0.1" scalaVersion := "2.12.5" lazy val root = (project in file(".")).enablePlugins(JavaAppPackaging) //rename zip file created from dist command packageName in Universal := "scala-demo" |
Next, let’s add a couple of files to a project folder that will define variables for sbt. For the first, type this into your terminal:
|
1 2 |
mkdir project && cd $_ touch build.properties |
Return to your editor and save this line to the file:
|
1 |
sbt.version = 1.1.4 |
The last file will be added with this line:
|
1 |
touch plugins.sbt |
Return to your editor and save this line to the file:
|
1 |
addSbtPlugin("com.typesafe.sbt" % "sbt-native-packager" % "1.3.4") |
Pushing to GitHub
To commit our code to Git, we need to return to our project’s root and type the following lines in our terminal:
|
1 2 3 |
$ git init $ git add . $ git commit -m “Initial commit” |
Open GitHub in your browser and create a new repository by clicking New in the top left corner of the browser. If you don’t have a GitHub account, you can create one here.
Add the name of the repo and click Create repository.
The next page provides instructions for getting started. We are going to use the third method. Type the lines in your terminal. In my case, the lines are as follows:
|
1 2 |
$ git remote add origin https://github.com/ronpowelljr/scala-demo.git $ git push -u origin master |
Using REST APIs in Your Pipeline
You may have used REST APIs for building an app locally, for manual scripting and/or for debugging. Tools like Wget and cURL are common tools in an engineer’s belt, so let’s explore the benefit you get from using these tools in your CI/CD pipeline. Next, we’re going to use CircleCI to build our application package and deploy it to an AWS S3 bucket. If you don’t have a CircleCI account, you can sign up for one here.
Log in to your account and select ADD PROJECTS from the left nav bar. Then click Set Up Project to the right of our new project.
If we click Start building now, our project will not run. We don’t have a configuration file yet.
Return to your terminal and in your project’s root type these lines:
|
1 2 |
$ mkdir .circleci && cd $_ $ touch config.yml |
Return to your editor and save these lines to the file:
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 |
yaml version: 2 jobs: build: working_directory: ~/scala-demo docker: - image: openjdk:8 environment: - SBT_VERSION: 1.0.4 steps: - run: echo 'export ARTIFACT_BUILD=$CIRCLE_PROJECT_REPONAME-$CIRCLE_BUILD_NUM.zip' >> $BASH_ENV - run: name: Get sbt binary command: | apt update && apt install -y curl curl -L -o sbt-$SBT_VERSION.deb https://dl.bintray.com/sbt/debian/sbt-$SBT_VERSION.deb dpkg -i sbt-$SBT_VERSION.deb rm sbt-$SBT_VERSION.deb apt-get update apt-get install -y sbt python-pip git pip install awscli apt-get clean && apt-get autoclean - checkout - restore_cache: key: sbt-cache - run: name: Compile scala-demo dist package command: cat /dev/null | sbt clean update dist - store_artifacts: path: target/universal/scala-demo.zip destination: scala-demo - save_cache: key: sbt-cache paths: - "~/.ivy2/cache" - "~/.sbt" - "~/.m2" - deploy: command: | mv target/universal/scala-demo.zip $CIRCLE_ARTIFACTS/$ARTIFACT_BUILD aws s3 cp $CIRCLE_ARTIFACTS/$ARTIFACT_BUILD s3://ronpowelljr-scala-demo/builds/scala-demo/ --metadata {\"git_sha1\":\"$CIRCLE_SHA1\"} |
Note: you will need to replace my S3 bucket location with yours.
The Get sbt binary step of the build job has a common usage of cURL: grabbing a package from the web to be installed. The deploy step shows the S3 API call to upload the artifact. Let’s commit this code to our repo so that we can start building with CircleCI. Return to your terminal and in your project’s root type these lines:
|
1 2 3 |
$ git add . $ git commit -m “added circleci” $ git push origin |
Go back to CircleCI and click Start building. The first run will fail because we haven’t added our AWS credentials as environment variables. You can add your AWS creds following the instructions found here. If you don’t have an AWS account, sign up for one here. You will then need to add an S3 bucket for uploading your application package. Instructions for that can be found here. Create a default bucket with public access.
After you have them set, let’s add a README.md file to our project to kick off a new build. We’ll add a title, a description of the app and a status badge. Return to your terminal and in your project’s root type these lines:
|
1 |
touch README.md |
Return to your editor and save these lines to the file:
|
1 |
# Demo Scala app [](https://circleci.com/gh/ronpowelljr/scala-demo) |
This project is a Helloworld application I like to use for demos, blogs and examples. It’s a great starting point for Scala projects.
Note: you will need to replace the URL for my CircleCI project with yours.
After a couple of minutes, your build will pass and your package will be uploaded to your S3 bucket. Go to your S3 dashboard to make sure that the package arrived.
Early Exit
Now let’s explore a novel way to include cURL in your pipeline to communicate with the Bintray REST API. It is common for a build to fail because of an external service that is down. One great way to use REST APIs in your pipeline is to check that these services are operational before continuing the build. This provides you with an opportunity to craft a custom message for this particular method of failure and eliminates any need for a developer to spend time debugging the fail.
Let’s use cURL in another job before build that will check whether or not Bintray is available to provide you with the package to download. Return to your editor and replace the lines above the build job in the config file with these lines:
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 |
workflows: version: 2 check-and-build: jobs: - check - build: requires: - check version: 2.1 jobs: check: docker: - image: circleci/python:2.7.14 steps: - run: name: Get Bintray API status command: | BT_STATUS=$(curl -s https://status.bintray.com/api/v2/status.json | jq --raw-output '.status.description') echo “Bintray API Status: $BT_STATUS” if [ “$BT_STATUS” != “All Systems Operational” ]; then echo “Error [Bintray API Status: $BT_STATUS]” exit 1 fi build: ... |
We set up the Get Bintray API status step of the check job to run first and the build job is now dependent on it. It checks the status and then uses logic to fail and exit our job if the status is anything other than “All Systems Operational.” This can limit your costs as well as save time for your engineers because if Bintray is unavailable, the job doesn’t sit waiting for cURL to timeout and your engineers don’t need to spend much time investigating the fail.
Finally, commit the changes:
|
1 2 3 |
$ git add . $ git commit -m “check” $ git push origin |
Wrapping up
REST APIs will only grow in popularity and usage. The services you need most will have REST APIs for communicating with them over the web. Start thinking of the different ways to put them to use in your pipelines. There may be a few services that you want to check the status of and the above example can be expanded to do more checks. You may need to install more packages from the web for your application and you can see how that is done above, too. You now have a working pipeline for pushing an application package to an S3 bucket on every new push.
All of the code in this tutorial can be found in this repo.