How to Push Code with Semaphore and ZEIT (and Forget about Server Management)

Semaphore sponsored this post.

Serverless is one of those concepts that in hindsight seems obvious and makes you wonder why no one came up with it earlier. At its core, it involves breaking up the application into smaller units of code, called lambdas, and distributing them in the cloud.
Internally, Now Lambdas work on top of AWS Lambdas. ZEIT Now also supports other cloud providers — but only for caching and routing. As we’ll see next, we won’t have to configure or set up any AWS service to run our application.
Lambdas, by their very nature, scale; this is because they are only run when needed, with resources allocated on-demand. Also, since there are no wasted resources, they tend to be cheaper to run than a full-fledged server or even a container.
Servers also rear their ugly heads during project development. Apps must be tested somewhere — too often a spare machine running some musty integration product. Once more, we find the same answer to the questions of upkeep and scaling: use a serverless continuous integration and delivery platform.
In this article, we will use two great platforms to test and deploy an application; ZEIT Now, a platform for running serverless functions, to build and host it and Semaphore to drive continuous integration and delivery pipelines.
Serverless can take many forms: we can write serverless functions for the different cloud providers, we could use an all-in-one solution platform, or we could pick among one of the many serverless frameworks that have appeared in recent years — most of them require some degree of re-training and have the risk of vendor lock-in.
ZEIT Now deployments, however, are seamless. ZEIT does not ask us to learn a new framework to be serverless. We can deploy our existing applications without modifications.
In this article, we will learn to use Semaphore to continuously test and conditionally deploy our application to ZEIT Now.
Meet the Application
During the course of this post, we will work with the Semaphore Demo app. Step by step, we’ll see how to get it online and how to make part of a continuous delivery process.
The project consists of a simple API Server that functions as a Hello World! program. APIs are the way applications communicate with each other over the web, so mastering them is crucial.
The server is written for Node.js and uses a few extra modules:
Express is a framework designed for building websites and APIs. Its popularity has made it an integral part of web development.
Unlike other frameworks, Express doesn’t force us to adopt any particular design pattern. Instead, it only provides the bare minimum functionality, which can be extended with middleware.
Helmet is middleware that provides enhanced security for Express. It can prevent exploits such as cross-site scripting (XSS) and click-jacking by adding special headers in the HTTP messages.
Testing is vital in any development project, not only to find bugs but also to validate that the code does exactly what it has been designed to do. Jest implements unit testing, a technique that runs small pieces of code and checks the results.
Setting up Your Dev Environment
Before we continue, you may need to install some tools on your machine. You should check if you have installed the Node.js and npm bundle:
1 2 |
$ node -v $ npm -v |
If you need to install them, instructions can be found at the Node.js website.
To work with the code, you will also need Git. Check if you already have it installed:
1 |
$ git --version |
Lastly, while not strictly necessary, it’s a good idea to get curl, the Swiss army knife of networking. It’s very handy to view the responses from servers.
1 |
$ curl --version |
Fork and Run the Demo
To get your own copy of the project, fork the repository on GitHub:
The next step is to build and test the app:
- Install the required packages:
1$ npm install - And run the test suite:
1$ npm test
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 |
> express-ziet-now-semaphore-ci@1.0.0 test semaphore-demo-zeit-now > NODE_ENV=test PORT=3000 jest --coverage --bail PASS test/integration/index.test.js :index ✓ [GET /] Should get server is running response (54ms) console.log test/index.js:12 Server running on port 3000. console.info test/_testUtils/ApiClient.js:20 ---REQUEST.BODY--- console.info test/_testUtils/ApiClient.js:21 {} console.info test/_testUtils/ApiClient.js:22 ---RESPONSE.BODY--- console.info test/_testUtils/ApiClient.js:23 { "text": "Hello from express server." } ---------------------|----------|----------|----------|----------|-------------------| File | % Stmts | % Branch | % Funcs | % Lines | Uncovered Line #s | ---------------------|----------|----------|----------|----------|-------------------| All files | 100 | 100 | 100 | 100 | | index.controller.js | 100 | 100 | 100 | 100 | | ---------------------|----------|----------|----------|----------|-------------------| Test Suites: 1 passed, 1 total Tests: 1 passed, 1 total Snapshots: 1 passed, 1 total Time: 1.591s Ran all test suites. |
Jest prints quite a lot of output. The first part is the result of the unit testing. We only have one test: check if the server’s response is valid JSON. The second part is the coverage report, which tells us whether any part of the code was not executed. No problems found? So far, so good.
The only thing left to do is to get the app started:
1 |
$ npm start |
You can view the response from your server either with curl or by opening your browser to http://127.0.0.1:3000:
1 2 3 |
$ curl -w "\n" http://127.0.0.1:3000 {"text":"Hello from express server."} |
You got it working. Way to go!
ZEIT Now
With ZEIT Now, we can make a global serverless deployment with just a few keystrokes (seriously, it only takes one command). The magic resides in their builders, which take your existing application code and transforms it into lambdafunctions. Do you have a lot of static files?
No problem. With minification and a smart built-in global CDN, they have you covered.
In ZEIT Now, there are no servers to manage and no containers to build. It integrates nicely with any workflow and plays well with CI/CD platforms. With instant, automatic deployments, ZEIT Now is a perfect fit for our microservice app.
Apps in ZEIT Now are immutable, meaning they cannot be changed once deployed. Each newly published version gets a unique deployment URL. As it turns out, this clever approach allows us to roll back to any previous version at any time if there are any problems.
ZEIT Now addresses follow this format:
1 2 |
● https://APP_NAME.USERNAME.now.sh: Public-facing URL that points to the latest version. ● https://APP_NAME-UUID.now.sh: Deployment URL. UUID is a unique, automatically generated string for each release. |
Deploy From Your Machine
Enough theory. Time to get this app online:
1. Create a ZEIT Now account.
2. Install the official now tool:
1 |
$ npm install now -g |
3. Connect your machine to ZEIT Now:
1 |
$ now login |
4. Follow the on-screen instructions.
The project already ships with two deployment files: production.json and staging.json. Production is our public facing site, while staging will act as a guinea pig and playground for us to test things out.
Take a look at staging.json:
1 2 3 4 5 6 7 |
{ "version": 2, "name": "semaphore-demo-staging", "builds": [ { "src": "**/*.js", "use": "@now/node" } ] } |
That is the minimum information that now needs to make a deployment. The config defines the name of the application and which files to include in the build.
Try deploying the staging site:
1 2 3 4 5 6 7 8 9 10 11 |
$ now --local-config staging.json > Deploying ~/semaphore-demo-zeit-now under myname > Using project semaphore-demo-staging > Synced 1 file (37B) [2s] > https://semaphore-demo-staging-hklhb6xbg.now.sh [v2] [3s] ┌ index.js Ready [29s] └── λ index.js (473.97KB) [iad1] ┌ controllers/index.controller.js Ready [26s] └── λ controllers/index.controller.js (33.96KB) [iad1] > Ready! Aliased to https://semaphore-demo-staging.myname.now.sh [in clipboard] [37s] |
now does all the heavy lifting:
- Uploads the code to ZEIT Now.
- Downloads packages, builds, and starts the app.
- Assigns deployment and public URLs.
Check out your new server. The ZEIT Now dashboard shows all deployments:
What’s All the Buzz about Continuous Integration
At this point in the post, you may be asking: I got the app online all right, aren’t we done yet? Not by a long shot, the best part is yet to come. Think about all the things we did to get to this point. The first time it is always interesting and fun, but doing it over, again and again, will get old fast. Wouldn’t it be great if we could automate away all of it? This is precisely the problem that continuous integration (CI) and continuous delivery (CD) solve. CI/CD takes care of all the testing, building and deploying. And they do it as a reproducible, battle-hardened process.
CI/CD is even more vital when working on a team. Lots of hands code faster but have a higher chance of introducing conflicts. The surest way to mitigate this is by building and testing the app as frequently as possible. Then, once everything is working, we can make high-quality releases earlier.
About Semaphore
Older continuous integration and delivery platforms, like the stand-alone version of Jenkins, encumber developers an infrastructure to manage. In Semaphore there is no back-end to maintain, no servers to install, or any Java versions to troubleshoot. We only define pipelines in a clean, easy-to-read format and do as many Git pushes as needed. Semaphore will silently provision everything to drive the pipeline, at any scale.
Before diving deep with Semaphore, we need to learn about a few key concepts. Semaphore pipelines are written in YAML, a text format that is easy for both humans and computers to read. When working with YAML, the important thing is to pay attention to indentation as, in the same vein of Python, spaces do matter.
For our project, we only need to know about a few properties, if curious, you may also check the full pipeline spec:
- Name
Pipelines have name which is shown on Semaphore’s dashboard. We also have to set the config version number. Right now the stable version is “v1.0.”
1 2 |
version: v1.0 name: Build and test Express.js app |
- Agent
Semaphore offers several machine types with different specs. The combination of machine and operating system is defined under agent property. Semaphore provides an Ubuntu 18.04 image that is just right for our needs:
agent:
1 2 3 |
machine: type: e1-standard-2 os_image: ubuntu1804 |
- Blocks
Blocks, tasks, and jobs define what to do at each step of the pipeline. Each block must have a single task, and each task defines one or more jobs. Jobs contain the list of commands to execute. Within a block, jobs run concurrently; each one runs in a fully isolated virtual machine. Once all jobs in a block are done, the next block begins.
This is how a block with two jobs looks like:
1 2 3 4 5 6 7 8 9 10 11 12 |
blocks: - name: Block name task: jobs: - name: My Job commands: - command 1 - command 2 - name: Another Job commands: - command 1 - command 2 |
Promotions chain pipelines together to build complex workflows. Since they can be triggered by user-defined conditions, they can be used to gracefully manage failures or to make a release when the pipeline runs successfully.
1 2 3 4 5 |
promotions: - name: Deploy pipeline_file: deploy.yml auto_promote_on: - result: passed |
Continuous Integration Pipeline
In this section, we’ll review in detail our CI pipeline. Here’s the full annotated config:
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 |
# Use the latest stable version of Semaphore 2.0 YML syntax: version: v1.0 # Name your pipeline. In case you connect multiple pipelines with promotions, # the name will help you differentiate between, for example, a CI build phase # and delivery phases. name: Build and test Express.js app # An agent defines the environment in which your code runs. # It is a combination of one of the available machine types and operating # system images. # See https://docs.semaphoreci.com/article/20-machine-types # and https://docs.semaphoreci.com/article/32-ubuntu-1804-image agent: machine: type: e1-standard-2 os_image: ubuntu1804 # Blocks are the heart of a pipeline and are executed sequentially. # Each block has a task that defines one or more jobs. Jobs define the # commands to execute. # See https://docs.semaphoreci.com/article/62-concepts blocks: - name: Install dependencies task: jobs: - name: npm install and cache commands: # Get the latest version of our source code from GitHub: - checkout # Use the version of Node.js specified in .nvmrc. # Semaphore provides nvm preinstalled. - nvm use - node --version - npm --version # Restore dependencies from cache. This command will not fail in # case of a cache miss. In case of a cache hit, npm install will # run very fast. # For more info on caching, see https://docs.semaphoreci.com/article/68-caching-dependencies - cache restore client-node-modules-$SEMAPHORE_GIT_BRANCH-$(checksum package-lock.json),client-node-modules-$SEMAPHORE_GIT_BRANCH,client-node-modules-master - npm install # Store the latest version of node modules in cache to reuse in # further blocks: - cache store client-node-modules-$SEMAPHORE_GIT_BRANCH-$(checksum package-lock.json) node_modules - name: Run tests task: jobs: - name: npm test commands: - checkout - nvm use - cache restore client-node-modules-$SEMAPHORE_GIT_BRANCH-$(checksum package-lock.json),client-node-modules-$SEMAPHORE_GIT_BRANCH,client-node-modules-master - npm test promotions: # Deployment to staging can be triggered manually: - name: Deploy to staging pipeline_file: deploy-staging.yml # Automatically deploy to production on successful builds on master branch: - name: Deploy to production pipeline_file: deploy-production.yml auto_promote_on: - result: passed branch: - master |
Install Dependencies
1 2 3 4 5 6 7 8 9 10 11 12 13 |
blocks: - name: Install dependencies task: jobs: - name: npm install and cache commands: - checkout - nvm use - node --version - npm --version - cache restore client-node-modules-$SEMAPHORE_GIT_BRANCH-$(checksum package-lock.json),client-node-modules-$SEMAPHORE_GIT_BRANCH,client-node-modules-master - npm install - cache store client-node-modules-$SEMAPHORE_GIT_BRANCH-$(checksum package-lock.json) node_modules |
The first block downloads and installs all the required packages:
- checkout clones the code from GitHub.
- nvm and npm are used to set Node.js version and install the packages.
- cache is used to share node modules between jobs.
npm Test
1 2 3 4 5 6 7 8 9 |
- name: Run tests task: jobs: - name: npm test commands: - checkout - nvm use - cache restore client-node-modules-$SEMAPHORE_GIT_BRANCH-$(checksum package-lock.json),client-node-modules-$SEMAPHORE_GIT_BRANCH,client-node-modules-master - npm test |
This is the unit test block. Since each job lives in an isolated environment, we need to get the code and packages. npm testruns the Jest tests.
Deployments
1 2 3 4 5 6 7 8 9 |
promotions: - name: Deploy to staging pipeline_file: deploy-staging.yml - name: Deploy to production pipeline_file: deploy-production.yml auto_promote_on: - result: passed branch: - master |
Two promotions branch out of the CI pipeline:
- Deploy to production: automatically started once all tests are green for the master branch.
- Deploy to staging: can be manually initiated from a Semaphore workflow on any branch.
Putting It All Together
Now that we have all the pieces of the puzzle in place, you will see for yourself the power and convenience of CI/CD.
A Shared Secret
In order to connect Semaphore and ZEIT Now, we need to get a token:
- Log in on your ZEIT Now account;
- Go to Settings;
- Go to the Tokens tab;
- Click on the Create button;
- Enter a name for the token, maybe something descriptive like: semaphore-zeit-now.
The token, being private information, does not belong in the repository. Semaphore has a secure mechanism to handle secrets:
- Create an account for Semaphore with your GitHub login;
- On the left navigation bar, under Configuration click on Secrets;
- Hit the Create New Secret button;
- Create the secret as shown below:
Continuous Delivery Pipeline
We have two, almost identical, continuous delivery pipelines. The only practical difference, apart from the name, is in the –local-config they use:
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 |
# ~.semaphore/deploy-staging.yml version: v1.0 name: Deploy to staging agent: machine: type: e1-standard-2 os_image: ubuntu1804 blocks: - name: Deploy to staging task: secrets: - name: now jobs: - name: Deploy to ZEIT Now commands: - checkout - nvm use - npm install now -g - now --token $ZEIT_TOKEN --local-config staging.json |
The deployment itself couldn’t be easier:
- the token is decrypted and imported as $ZEIT_TOKEN;
- npm installs the now;
- and now deploys takes care of the deployment.
Start the Pipeline
This is where all our hard work pays off:
- Go to your Semaphore account.
- Follow the link on the sidebar to create a new project.
- Semaphore will show your GitHub repositories, click on Add Repository.
- The pipeline will start as soon as any file is modified on your repository:
1 2 3 4 |
$ touch any_file $ git add any_file $ git commit -m "start semaphore pipeline" $ git push origin master |
Once all the blocks are done, you should have the production site online:
You Did It!
Nice work! Once you taste the power of serverless architecture and CI/CD, you will want to use it in all your projects. I hope that what you learned today helps you to build better and smarter.
Feature image by from Pixabay.