The following post originally appeared on the Cloudsoft blog. Those attending the OpenStack Summit in Austin this month can learn more about how Cloudsoft AMP and IBM BlueBox can work together to scale workloads in an April 26 talk by Cloudsoft CEO Duncan Johnston-Watt and IBM Blue Box Chief Product Officer Hernan Alvarez.
In this tutorial, we will show you how to quickly and easily deploy a full Apache Mesos environment into IBM Blue Box using Cloudsoft AMP Pro, and exercise it by deploying a application stack to support the DayTrader benchmark application. We will also show how AMP Pro automates scaling of the environment under load.
What you will need to get started:
Getting Everything Running
This section will guide you through the process of deploying AMP Pro using Vagrant, adding a cloud location to its catalog (where you will be deploying the Apache Mesos Cluster), adding the demo application catalogs and subsequent deployment of a DayTrader Cluster to Apache Mesos.
The accompanying video illustrates the process and gives additional insight into the workflow.
Starting AMP Pro Using Vagrant
Important: AMP Pro requires Vagrant 1.8.0 or higher.
First, clone and cd into the demo repository:
git clone https://github.com/cloudsoft/amp-daytrader-mesos-demo.git cd amp-daytrader-mesos-demo
Next, copy public and private keys for your Cloud Provider to the my_keys directory, these keys will be copied to the /home/vagrant/.ssh/ directory on startup so as to be available to AMP Pro. For example, if you use an OpenStack key pair to connect to your IBM Blue Box tenant you should copy them as follows:
cp ~/.ssh/openstack.pem ~/.ssh/openstack.pub ./my_keys/
Then start the Vagrant box:
Note that the instructions below are lifted from the README.md in your cloned amp-daytrader-mesos-demo git repository.
Configure AMP Pro for Blue Box
- Make a copy of the sample-bluebox.bom in the locations folder of your cloned amp-daytrader-mesos-demo git repository. We will use this file to add IBM Blue Box as a target location for our applications
- Update the file, replacing the REPLACE_THIS tokens with appropriate values from your Blue Box OpenStack environment
- Connect to the AMP Pro web console at http://localhost:8081
- From the AMP Pro web console Catalog tab, add the updated sample-bluebox.bom to the catalog by pasting the contents into the blueprint composer and clicking Add to Catalog
- The location ibm-bluebox-mesos will now be available in AMP Pro and should be used as the target location for your Mesos cluster
Deploy the Apache Mesos Cluster
- From the AMP Pro web console Home tab click add application, select Apache Mesos and hit Next
- Supply values for the following fields, leaving the rest with defaults:
- Set Locations to your added location, for example, IBM Bluebox San Jose Clocker Meso
- Give the Apache Mesos Cluster a name, for example, Bluebox Apache Mesos Cluster (San Jose)
- Point mesos.slave.privateKey at the location of your locations private key, for example, ~/.ssh/openstack.pem
- Hit Deploy; your Apache Mesos cluster will be ready in a few minutes
When we move over to the Blue Box OpenStack Dashboard and refresh, you can see all nodes are now present.
Switching to the Applications tab and expanding the tree view you can find the Mesos URL on the Apache Mesos Cluster Summary page, and Marathon URL under Mesos Frameworks on the Marathon Framework Summary page.
You now have a fully functional Apache Mesos cluster, deployed and managed by Cloudsoft AMP Pro. As an example of using AMP Pro to deploy an application into Mesos, we’ll be deploying DayTrader, an Open Source benchmarking application.
In this section, we will start a DayTrader cluster backed by IBM WebSphere Liberty and MariaDB.
- From the AMP Pro web console Home tab click add application, select DayTrader App Cluster (WebSphere Liberty/MariaDb) and hit Next
- Supply values for the following fields, leaving the rest with defaults:
- Set Locations to my-mesos-cluster
- Give the DayTrader Cluster a name, for example, DayTrader WSL MariaDB
- Provide a WebSphere Liberty Archive URL for the webProfile7 release archive (wlp-webProfile7-8.5.5.x.zip), either hosting it yourself or via an IBM-provided source. At the time of writing a valid option is
- Hit Deploy; your DayTrader Cluster will be ready in a few minutes. While the deployment to Apache Mesos is proceeding, you can navigate to the Marathon Dashboard where you can observe AMP Pro adding application instances
- When the DayTrader Cluster finishes deploying (the green icon will stop flashing), you can connect to Daytrader via the mapped load balancer URI (main.uri) found on the application’s Summary tab
Now that the DayTrader application has been deployed, we will use Apache JMeter to exercise it, and see that Cloudsoft AMP Pro will manage autoscaling the application based on load.
Acquire the Test Suite
First, we need to acquire an appropriate JMeter test suite for use with Daytrader.
Follow the instructions on the IBM Blog post Measuring performance with the Daytrader 3 benchmark sample to download the DayTrader3Install.zip via the WebSphere Performance Page. This archive contains a sample Daytrader JMeter test suite file daytrader3.jmx.
curl -O http://www.mirrorservice.org/sites/ftp.apache.org/jmeter/binaries/apache-jmeter-2.13.tgz tar zxf apache-jmeter-2.13.tgz curl -O ftp://public.dhe.ibm.com/software/webservers/appserv/was/DayTrader3Install.zip tar zxf DayTrader3Install.zip
Populate the DayTrader Database
Before we can begin running JMeter against DayTrader, we will need to populate the DayTrader Database.
- Return to the DayTrader web page (or open it using the main.uri on the DayTrader’s Summary tab in AMP Pro)
- Navigate to the Configuration tab and click the link named (Re)-populate DayTrader Database
This will open a new, initially blank, page which will periodically update as sample stock and user data is loaded. This can take some time to complete, and updates only infrequently. You will know it is complete when the plain update text is followed by what looks like a normal configuration page.
Now that the DayTrader database has been populated with some test date, we are ready to begin using JMeter to generate load.
- Start the JMeter GUI substituting the HOST and PORT values for your deployed DayTrader cluster (from the main.uri on the DayTrader WSL MariaDB Summary page) in this command
./apache-jmeter-2.13/bin/jmeter.sh \ -t DayTrader3Install/DayTrader3-EE6/Apache_JMeter_script/daytrader3.jmx \ -JHOST=MAPPED BALANCER HOST \ -JPORT=MAPPED BALANCER PORT \ -JDURATION=600
- On opening JMeter you should be presented with the Thread Group page, you should alter the Number of Threads (users) setting as desired. For demonstration purposes, it is recommended to lower this to 10 which results in approximately 60 requests/second from an i7 MacBook Pro. The default autoscaling thresholds for both Wildfly and Websphere Liberty have been set to upper 40 and lower 20. JMeter will generate traffic that exceeds the upper bound, and Cloudsoft AMP will automatically manage to scale the application to meet demand
- Right click and enable Aggregate Report in the left-hand tree, and when ready to start traffic click the green Play menubar item. You should immediately start to see results fill the report pane. The reqs/sec rate should approximately match that show in the Dynamic Cluster Groups reqs.per_sec.per_node sensor
- Expanding the DynamicCluster of WebSphere Liberty Servers entry on the Applications tab you should observe AMP Pro provision new DayTrader nodes which are automatically added to the Nginx balancer group, examining the reqs.per_sec sensors for each node and for the Dynamic Cluster Group you can observe the traffic being distributed evenly
- To stop traffic click the red Stop menubar item. You can also click the sweeping brush icons (two brushes to the left of the binoculars), to clear the results window
AMP Pro will scale the DayTrader nodes back down in response to the decreased traffic, eventually to just the original single node.
Also included in your cloned amp-daytrader-mesos-demo repository are blueprints that allow you to deploy a Daytrader application on WebSphere backed by PostgreSQL, instead of MariaDB. There are also blueprints that use the WildFly 10 application server in place of IBM WebSphere, again backed by either PostgreSQL or MariaDB.
The readme in the repository has further options, advice and opportunities for exploration.
In this tutorial, we have done quite a bit. We have set up a Cloudsoft AMP Pro instance, deployed a fully functional Apache Mesos environment into IBM Blue Box, deployed a DayTrader application, backed by MariaDB and IBM WebSphere Liberty into Mesos, and finally we used JMeter to show how AMP Pro automatically scales our environment in response to load. We have shown that AMP Pro can deploy Apache Mesos easily, and blueprints can target either a deployed Mesos environment or a public or private cloud provider, in this case, Blue Box, directly. What’s next? Kubernetes of course!
Cloudsoft and IBM are sponsors of The New Stack.