Implementing Service Discovery of Microservices with Consul

Consul from HashiCorp is one of the early implementations of service mesh technology. It comes with a full-featured control plane with service discovery, configuration, and segmentation functionality. The best thing about Consul is the support for a variety of environments including traditional applications, VMs, containers, and orchestration engines such as Nomad and Kubernetes.

Jun 26th, 2019 10:09am by Janakiram MSV

Featued image for: Implementing Service Discovery of Microservices with Consul

Feature Image by DarkWorkX from Pixabay.

At a high level, Consul provides the below functionality:

Distributed Architecture: Consul supports multiple data centers and for delivering a scalable, highly available, abstract, and resilient service mesh infrastructure. A Consul deployment may span multiple zones or even regions in a public cloud environment.
Service Discovery: Distributed applications can use Consul to dynamically discover service endpoints. Once a service is registered with Consul, it can be discovered using typical DNS or custom API.
Health Checking: Consul can track and report the health of registered services. Each microservice can provide an endpoint that Consul probes to check the health. This is useful in implementing patterns such as circuit-breaking.
Key/Value Store: Consul is one of the popular distributed, in-memory KV store. Applications and infrastructure can utilize to store hierarchical data that’s accessible to any participating service. A simple HTTP API exposes the read/write functionality to deal with the data.
Secure Communication: Like most of the service mesh implementations, Consul enables implicit communication security for services through mTLS. The traffic between the proxies/sidecars is automatically encrypted without the need to code or configure.

This tutorial explores the concepts of Consul through a simple scenario based on three microservices.

We have three microservices — order, invoice, and tax — that are a part of an online retail application. Order invokes the invoice service which in turn calls the tax service.

These microservices are written in Flask with a very simple functionality — they return the service name and the version.

from flask import Flask
import requests
import os

app = Flask(__name__)

@app.route('/')
def get_invoice():
url=os.environ.get('TAX_SVC_URL')
response = requests.get(url)
ver="1.0"
res='{"Service":"Invoice", "Version":' + ver + '}\n'
res=res + response.content.decode('utf-8')
return res

if __name__ == '__main__':
app.run(debug=True,host='0.0.0.0',port=5001)

from flask import Flask

import requests

import os

app = Flask(__name__)

@app.route('/')

def get_invoice():

url=os.environ.get('TAX_SVC_URL')

response = requests.get(url)

ver="1.0"

res='{"Service":"Invoice", "Version":' + ver + '}\n'

res=res + response.content.decode('utf-8')

return res

if __name__ == '__main__':

app.run(debug=True,host='0.0.0.0',port=5001)

from flask import Flask
import requests
import os

app = Flask(__name__)

@app.route('/')
def get_order():
url=os.environ.get('INV_SVC_URL')
response = requests.get(url)
ver="1.0"
res='{"Service":"Order", "Version":' + ver + '}\n'
res=res + response.content.decode('utf-8')
return res

if __name__ == '__main__':
app.run(debug=True,host='0.0.0.0',port=5000)

from flask import Flask

import requests

import os

app = Flask(__name__)

@app.route('/')

def get_order():

url=os.environ.get('INV_SVC_URL')

response = requests.get(url)

ver="1.0"

res='{"Service":"Order", "Version":' + ver + '}\n'

res=res + response.content.decode('utf-8')

return res

if __name__ == '__main__':

app.run(debug=True,host='0.0.0.0',port=5000)

The order, invoice, and tax microservices are listening on ports 5000,5001, and 5002 respectively.

Let’s launch the services and invoke them.

INV_SVC_URL=http://localhost:5001 python order.py &
TAX_SVC_URL=http://localhost:5002 python invoice.py &
python tax.py &

INV_SVC_URL=http://localhost:5001 python order.py &

TAX_SVC_URL=http://localhost:5002 python invoice.py &

python tax.py &

The URI of the upstream service is passed as an environment variable. Invoking order service listening on 5000 will print the below output:

We will now register these microservices with Consul for dynamic service discovery.

Installing and Running Consul

Download an appropriate flavor of the Consul binary from HashiCorp’s official website, and move it to a directory which is added to your $PATH environment variable.

Launch Consul with the below command line switches.

consul agent -dev -ui -datacenter zone1 -node host1

1	consul agent -dev -ui -datacenter zone1 -node host1

The -dev switch instructs Consul to run in a development environment. The -ui parameters launches makes the dashboard available. The -datacenter and -node provide identifiers for the datacenter
and the host.

Visiting http://localhost:8500/ui will show the Consul dashboard.

Since we are have not registered the services, the dashboard looks empty. Let’s populate it with our microservices.

Registering Microservices with Consul

Before we can dynamically discover the services, we need to register them with Consul. This is done by creating a Consul service definition file for each service with the name and the port.

The below JSON document saved in a file named order.json represents the order service definition.

{"service": {"name": "order", "tags": ["order"], "port": 5000}}

1	{"service": {"name": "order", "tags": ["order"], "port": 5000}}

Create the definitions for the other services and move the JSON files to a directory called consul.d

{"service": {"name": "invoice", "tags": ["invoice"], "port": 5001}}

1	{"service": {"name": "invoice", "tags": ["invoice"], "port": 5001}}

{"service": {"name": "tax", "tags": ["tax"], "port": 5002}}

1	{"service": {"name": "tax", "tags": ["tax"], "port": 5002}}

With the service definition files in place, we will relaunch Consul pointing it to consul.d directory to load the definitions. The config-dir switch tells Consul to include all the configuration files in the directory.

Hit CTRL+C to terminate the Consul agent and relaunch it with the below command.

consul agent -dev -ui -datacenter zone1 -node host1 -config-dir ./consul.d/

1	consul agent -dev -ui -datacenter zone1 -node host1 -config-dir ./consul.d/

The Consul web UI now shows the registered services.

Discovering the Services with DNS and HTTP API

Consul has its own DNS server listening on port 8600 that can respond to standard queries. Each registered service gets its own DNS entry which resolves to the service endpoint.

Each service assumes a FQDN in the format of service_name.service.consul. Consul will resolve the name to the exact service running in a specific data center and node.

The below command shows how Consul resolves the order microservice endpoint.

dig @127.0.0.1 -p 8600 order.service.consul SRV

1	dig @127.0.0.1 -p 8600 order.service.consul SRV

The SRV record shows the port (5000) and the location (host1.node.zone1.consul) where the service endpoint is hosted. Feel free to explore the DNS entries for invoice and tax services.

Applications can query the HTTP API for discovering the service.

curl http://localhost:8500/v1/catalog/service/order

1	curl http://localhost:8500/v1/catalog/service/order

We can integrate Consul DNS with the host DNS to resolve service FQDN. On macOS Mojave, updating /etc/resolver/consul with the below entry makes this possible.

nameserver 127.0.0.1
port 8600

1 2	nameserver 127.0.0.1 port 8600

You can now resolve the service names with FQDN.

In the next part of this tutorial, to be posted Friday, we will take a look at Consul Connect, the service mesh component of Consul. Stay tuned.

Janakiram MSV’s Webinar series, “Machine Intelligence and Modern Infrastructure (MI2)” offers informative and insightful sessions covering cutting-edge technologies. Sign up for the upcoming MI2 webinar at http://mi2.live.