API load testing and Continuous Integration

What is Continuous Integration?

Continuous Integration is a software engineering practice in which automation is used to tie the main parts of the software development lifecycle together. This movement started in the development space and advocates improving the process that starts with developers committing code to a repository and ends with that code actually being deployed in production.
As software testing has matured, this definition has involved in order to include testing practices as well.
A modern CI framework still begins with a developer checking code into a repository, which would then automatically kick off functional tests of the application that determine whether or not that code will be allowed to proceed to the next step. Once functional tests are passed, the code could then be automatically deployed into production, with reports emailed out to relevant stakeholders.
A natural extension of this would be to add in a load testing framework. After functional tests pass, the code could potentially be deployed to a staging or similar environment, where we can run our API load testing scripts to see if there have been any performance regression bugs introduced by the code. In this way, load testing could be an extra check before sending out code to production.
This sort of CI framework is preferable to the traditional process, which involves many manual checks and potential bottlenecks as the code is passed among different teams (development to functional testers to nonfunctional testers, with DevOps engineers helping each team along the way). A good CI framework may take a little bit more time to set up in order to come up with the correct path, but it dramatically reduces the amount of time required and encourages a development practice that is iterative, quick, and truly agile.
In this chapter we’ll discuss how to add load testing to your CI framework.


At the heart of your CI framework will be an automation server that is flexible enough to integrate with a variety of tools and applications that you use as part of your development process. There are a few options for this, but one of the most popular and user-friendly ones is Jenkins.
Jenkins is an open-source tool that will handle the interactions between disparate development and testing activities and serve as the traffic cop that will tie everything together. In the succeeding sections, we’ll go over the intricacies of installing Jenkins, running your load testing script through Jenkins, and automating test execution through Jenkins.

Installing Jenkins

Download Jenkins here. Choose the appropriate version of Jenkins and download it. Then execute the downloaded file.
If a browser page does not automatically open, open your favourite browser and navigate to localhost:8080. You should see something like this:
You’ll need to get an administrator password to unlock Jenkins. You can click on the link in the dialog box to get specific information for your operating system, but here are some instructions for macOS:
1. Navigate to /Users/Shared/Jenkins/Home
2. Right click on the folder secrets and click Get Info.
3. Scroll down to the bottom right corner of the dialog box. You may have to expand the last item in order to see it. Then click on the lock image.
4. Enter the password to your machine when prompted.
5. Click on the + and add your local user with the permissions “Read and write”.
6. Click on gear icon and click “Apply to enclosed items”.
7. Double click on the secrets folder and open the file named initialAdminPassword. It will contain your administrator password.
You should then be able to enter your administrator password into your browser window with Jenkins and click Continue.
When asked to install plugins, opt to install the recommended plugins and you’ll see a screen like this:
After those plugins are installed, you’ll be prompted to create an admin user with which you’ll use Jenkins.
Fill out the information and remember your admin credentials. On the next screen, leave the Jenkins URL as the default.
Click Save and Finish.
Click Start Using Jenkins.
When Jenkins starts, login with the username and password you set for your admin user. Click Back to the Dashboard. You should see something like this:
Congratulations! You’ve installed Jenkins along with the plugins you’ll need to run your load testing scripts through it.
For more instructions on installing Jenkins for specific operating systems, check out the Jenkins installation documentation.

Running a JMeter script in Jenkins

Jenkins has a good guide for running JMeter scripts, but we’ll go through the basics here anyway.
From the Jenkins dashboard, click on Create new jobs.
Enter an item name (“JMeter”), click Freestyle project, and then click OK.
Add a description for your project and then head to the Build section. Click on the “Add build step” dropdown menu and select Execute shell for macOS.
In the Command field, type in the command you would normally use to run JMeter in CLI mode.
Note that you’ll need to include the absolute path of JMeter, your script, and your results file such as :
/Users/nvanderhoeven/jmeter/apache-jmeter-5.0/bin/jmeter -n -t /Users/nvanderhoeven/jmeter/jmeter-scripts/challenge_jmeter.jmx -l /Users/nvanderhoeven/Downloads/results.csv
Click Save.
Click Build Now.
Click on the Build number in the “Build History” section that appears.
Then you can click on the Console Output link to see the real-time logs of the test:
You’ve just run your very first JMeter script through Jenkins!

Build triggers

Build Triggers are really where the magic happens. You’ve managed to run your script manually through Jenkins, but that doesn’t really add any more functionality than just running your test through your terminal. The beauty of Jenkins is in setting up build triggers that automatically execute your test once certain triggers are met.
To set these build triggers, go into your project from the Jenkins dashboard.
Click Configure.
This will take you to the same screen where you first set up your project. This time, scroll down to the section labelled “Build Triggers”.
There are a few options here to use as a trigger for your project.
Trigger builds remotely will allow you to create a special URL that you can use to execute the script from another computer. This is especially useful if you want to build this into a batch job on one of your servers. For example, you could add a command in your script to go to IPADDRESS:8080/job/JMeter/build?token=TOKEN_NAME or /buildWithParameters?token=TOKEN_NAME and your project will automatically be built.
Build after other projects are built will start this project after another Jenkins project has been built. This is handy if you need scripts to run in sequence.
Build periodically will run your test according to a schedule that you define.
GitHub hook trigger for GITScm polling will allow you to configure your project upon a commit to GitHub. This is handy if you want to trigger load tests after your developers commit new code in a certain repository.
Poll SCM is similar to the GitHub hook trigger, but allows you to kick off Jenkins project builds based on commits in other SCM software.

Schedule your load testing

The most popular build trigger is Build periodically, so we’ll go over how to set that up here.
First, go to your project in Jenkins and click on Configure. Scroll down to the Build Triggers section and click the checkbox next to Build periodically. Then you’ll be prompted to enter a schedule for your test.
The schedule will need to be defined in a specific format. You can click on the ? next to the text box in Jenkins, but in general, you’ll need to know that Jenkins expects five parameters separated by a whitespace or tab in this format:
minute hour dom month dow
minute is a value from 0-59 that represents minutes within the hour
hour is a value from 0-23 that represents hours within the day
dom is a value from 1-31 that represents the day of the month
month is a value from 1-12 that represents the month within the year
dow is a value from 0-7 that represents the day of the week, beginning and ending with Sunday
So something like 0 0 * * * will run:
minute: every 0th minute
hour: of the 0th hour
dom: every day of the month
month: every month
dow: every day of the week
On top of this, H is a symbol that you can use to represent “hash”. Using it means allowing Jenkins to determine a schedule within your parameters that will optimise the build so that you won’t have multiple projects getting built at the same time. Jenkins recommends that you use H whenever possible, unless your intention is simultaneous execution.
If instead of the example above, we used H H * * *, this means that the build will still be triggered once a day, but the exact hour and minute will vary depending on how many other projects there are with the same schedule. In this way, you can stagger your project builds.
Here are some examples of schedules you can use, along with what they mean:
H/15 * * * *
every fifteen minutes (perhaps at :07, :22, :37, :52)
H(0-29)/10 * * * *
every ten minutes in the first half of every hour (three times, perhaps at :04, :14, :24)
45 9-16/2 * * 1-5
once every two hours at 45 minutes past the hour starting at 9:45 AM and finishing at 3:45 PM every weekday.
H H(9-16)/2 * * 1-5
once in every two hours slot between 9 AM and 5 PM every weekday (perhaps at 10:38 AM, 12:38 PM, 2:38 PM, 4:38 PM)
H H 1,15 1-11 *
once a day on the 1st and 15th of every month except December
0 2 * * 1-5
every weekday at 2:00 AM
Once you’ve chosen and entered your schedule, click Save. Your script will be executed according to your schedule as long as Jenkins is running on your machine.

Cloud-based load testing with Jenkins

Up until this point, I’ve shown you how to schedule a load test running on your Jenkins server, which may well be your local machine. If so, you’ll be running your test on your local machine as well, which may or may not be ideal. What if you want to scale up your load test?
One option is to install Jenkins on each load generator you want to use, setting each one up using the instructions here.
Running on the cloud, though, is another story. Depending on the platform you’re using, you may be able to integrate it into Jenkins as well. I’ll go over how to achieve this using Flood.
Flood has an extensive API that allows you to start and repeat grids and floods. This makes it really easy to orchestrate through Jenkins. This means that you could conceivably add in not just a quick single-machine load test before deployment but also a full-blown load test with thousands of users across several geographical regions in the world— all executed without manual intervention.
To do this, you’ll need:
1. A Flood account.
2. Your API access token, which you can get by clicking on your profile pictures once logged into Flood and then clicking API Access. Click Reveal token and you’ll see something like this:
Copy the value of your access token. You’ll need it to allow Jenkins to access your Flood account and start floods and grids on your behalf.
3. A shell script to use in Jenkins. In the Configure settings of your project on Jenkins, add a build step with Execute shell and enter your calls to the Flood API in that text field. Here’s a sample one to get you started:
curl -u ${API_TOKEN}: -X POST \
-F "flood[tool]=jmeter" \
-F "flood[threads]=10" \
-F "flood[privacy]=public" \
-F "flood[name]=MyTest" \
-F "flood_files[][email protected]" \
-F "flood[grids][][infrastructure]=demand" \
-F "flood[grids][][instance_quantity]=1" \
-F "flood[grids][][region]=us-west-2" \
-F "flood[grids][][instance_type]=m5.xlarge" \
-F "flood[grids][][stop_after]=60"
You will need to replace ${API_TOKEN} with the API access token that you revealed in Step 2 above.
Two of my colleagues have already written about this in detail, so for more information, check out:
Jason Rizio’s help article on Using Jenkins to Run Continuous Load Test with Flood and
Tim Koopman’s blog post on Jenkins Load Testing with Flood.