What news from AWS re:Invent last week will have the most impact on you?
Amazon Q, an AI chatbot for explaining how AWS works.
Super-fast S3 Express storage.
New Graviton 4 processor instances.
Emily Freeman leaving AWS.
I don't use AWS, so none of this will affect me.

How to Integrate RESTful APIs into CI/CD Pipelines

Jun 19th, 2019 2:05pm by
Featued image for: How to Integrate RESTful APIs into CI/CD Pipelines
Feature image via Pixabay.

Ron Powell
Ron has a background in space physics, having worked as a Cassini team member analyzing plasmas trapped in Saturn's magnetosphere. He now works for CircleCI in San Francisco, Calif. as a developer advocate producing content that enables developers to build, test and deploy their projects faster. Previously, he was a developer evangelist at Samsung Electronics, America where he specialized in 360-degree video production and developing applications for virtual reality hardware, wearables and IoT.

To fully automate the building, testing and deployment of modern applications, RESTful API calls are essential for information transfer between services. In this beginner-level tutorial, we explore the real-world usage of these APIs.

Modern applications rely heavily on HTTP requests to external services to GET, POST, PUT and DELETE data. This information transfer is done with RESTful or REST APIs to reduce the bandwidth associated with traditional APIs. REST APIs are essential for distributed computing and cloud-based technologies. One example is publishing packages to an AWS S3 bucket using the Amazon S3 REST API. Another is using cURL to download a package from the web to be installed.

With these examples, it becomes clear that one would need to integrate these API calls into their CI/CD pipeline for fully automated building and deployment. In this tutorial, we will build an application that will use both of these APIs. It is a simple Scala application that we will build and deploy to an S3 bucket. The goal of this post is to show real-world examples of REST API usage. Extensive knowledge of Scala is not required.

Building the App

First, let’s create a directory and cd into it by typing this command in your terminal:

This is where we will add the files for our application. Now, we need to add a HelloWorld.scala file in a /src/main/ folder. Add the file by typing these two commands:

Open your favorite editor and save these lines to HelloWorld.scala:

We need a few more files before we have a working application. Let’s define a few files for using sbt. The first is build.sbt and it needs to be in the projects root. Type these lines into your terminal:

Return to your editor and save these lines to the file:

Next, let’s add a couple of files to a project folder that will define variables for sbt. For the first, type this into your terminal:

Return to your editor and save this line to the file:

The last file will be added with this line:

Return to your editor and save this line to the file:

Pushing to GitHub

To commit our code to Git, we need to return to our project’s root and type the following lines in our terminal:

Open GitHub in your browser and create a new repository by clicking New in the top left corner of the browser. If you don’t have a GitHub account, you can create one here.

Add the name of the repo and click Create repository.

The next page provides instructions for getting started. We are going to use the third method. Type the lines in your terminal. In my case, the lines are as follows:

Using REST APIs in Your Pipeline

You may have used REST APIs for building an app locally, for manual scripting and/or for debugging. Tools like Wget and cURL are common tools in an engineer’s belt, so let’s explore the benefit you get from using these tools in your CI/CD pipeline. Next, we’re going to use CircleCI to build our application package and deploy it to an AWS S3 bucket. If you don’t have a CircleCI account, you can sign up for one here.

Log in to your account and select ADD PROJECTS from the left nav bar. Then click Set Up Project to the right of our new project.

If we click Start building now, our project will not run. We don’t have a configuration file yet.

Return to your terminal and in your project’s root type these lines:

Return to your editor and save these lines to the file:

Note: you will need to replace my S3 bucket location with yours.

The Get sbt binary step of the build job has a common usage of cURL: grabbing a package from the web to be installed. The deploy step shows the S3 API call to upload the artifact. Let’s commit this code to our repo so that we can start building with CircleCI. Return to your terminal and in your project’s root type these lines:

Go back to CircleCI and click Start building. The first run will fail because we haven’t added our AWS credentials as environment variables. You can add your AWS creds following the instructions found here. If you don’t have an AWS account, sign up for one here. You will then need to add an S3 bucket for uploading your application package. Instructions for that can be found here. Create a default bucket with public access.

After you have them set, let’s add a file to our project to kick off a new build. We’ll add a title, a description of the app and a status badge. Return to your terminal and in your project’s root type these lines:

Return to your editor and save these lines to the file:

This project is a Helloworld application I like to use for demos, blogs and examples. It’s a great starting point for Scala projects.
Note: you will need to replace the URL for my CircleCI project with yours.

After a couple of minutes, your build will pass and your package will be uploaded to your S3 bucket. Go to your S3 dashboard to make sure that the package arrived.

Early Exit

Now let’s explore a novel way to include cURL in your pipeline to communicate with the Bintray REST API. It is common for a build to fail because of an external service that is down. One great way to use REST APIs in your pipeline is to check that these services are operational before continuing the build. This provides you with an opportunity to craft a custom message for this particular method of failure and eliminates any need for a developer to spend time debugging the fail.

Let’s use cURL in another job before build that will check whether or not Bintray is available to provide you with the package to download. Return to your editor and replace the lines above the build job in the config file with these lines:

We set up the Get Bintray API status step of the check job to run first and the build job is now dependent on it. It checks the status and then uses logic to fail and exit our job if the status is anything other than “All Systems Operational.” This can limit your costs as well as save time for your engineers because if Bintray is unavailable, the job doesn’t sit waiting for cURL to timeout and your engineers don’t need to spend much time investigating the fail.

Finally, commit the changes:

Wrapping up

REST APIs will only grow in popularity and usage. The services you need most will have REST APIs for communicating with them over the web. Start thinking of the different ways to put them to use in your pipelines. There may be a few services that you want to check the status of and the above example can be expanded to do more checks. You may need to install more packages from the web for your application and you can see how that is done above, too. You now have a working pipeline for pushing an application package to an S3 bucket on every new push.

All of the code in this tutorial can be found in this repo.

Group Created with Sketch.
TNS owner Insight Partners is an investor in: Saturn, Pragma, Unit.
THE NEW STACK UPDATE A newsletter digest of the week’s most important stories & analyses.