Links

Choosing a tool

An in-depth look at each supported performance test tool to help you choose the right one for the job

Protocol-level and browser-level tools

At Flood, we support a number of open-source tools as well as our own simple Test Builder tool. We are performance testers ourselves, and we believe there is no "best" test tool-- just the best one for your particular situation. The tools available on the market are generally divided into two depending on how they generate load. ​ JMeter and Gatling are traditional, purpose-built load testing tools which generate load on the HTTP protocol level. This means that they don't involve real browsers and instead send requests directly to your application, which is generally more efficient. We call virtual users that are created using these tools Protocol-Level Users (PLUs).
Selenium and Element are newer tools that work differently: they generate load on the browser level, so they simulate real users interacting with your web application through a browser. We call virtual users that are created using these tools Browser-Level Users (BLUs).
Both protocol-level and browser-level tools are very useful for performance testing, but each one (like anything in life) has trade-offs. That trade-off is complexity vs concurrency.

Protocol-Level Users (PLUs)

A lot of today's web applications are significantly complex. Try visiting amazon.com with the Chrome Developer Tools add-in enabled on the network tab. You can easily view every request that the browser makes on the front page for images, dynamic content (XHR), javascript libraries, etc., and as of the time of writing, that page contains 349 individual requests.​
If you were using JMeter or Gatling to simulate simply going to the front page, you'd have to script every single request and worry about correlating session IDs and other dynamic values. There's more bad news than good news with this:
  • It would take even the most expert scripting wizard a significant amount of time to script a workflow on the Amazon web app using protocol-level tools.
  • If anything changes on the page, such as when a new build is released and functionality on the page changes (even just something like the names of elements on the page), guess what? You'd need to update your script!
The good news:
  • You will be able to simulate a lot more concurrent users scripting this front page on the protocol level than on the browser level.

Browser-Level Users (BLUs)

How to run a load test using real browsers
In Selenium or Element, for the same web app, you would focus on interacting with the page objects themselves and not worry about the individual requests. Scripting BLUs is a significantly easier scripting task than scripting PLUs because you don't need to know the underlying dynamic values that are changing; all you need to know is which buttons a user would click on or which fields a user would interact with.
The bad news is that with this method, concurrency suffers due to the amount of resources a single browser level user needs to have available. Since BLUs are starting browser instances, they take more CPU and memory on load generator nodes.
At the present time, a single demand Flood node can only support approximately 5 concurrent Selenium users or up to 40 Element users as opposed to approximately 1,000 JMeter or Gatling users per node: a significant difference. This translates into increased cost for test execution since you'd need to start more nodes with BLUs than PLUs to generate the same load.
Now, different technologies aside, let's take a look at each tool's advantages and disadvantages in terms of fit for purpose.

Flood Test Builder

The Test Builder built into the Flood user interface is a great tool for very simple HTTP-based load tests. Its aim is to get up and running very quickly for new users without needing the knowledge to script a test in JMeter, Gatling or Selenium.​

Pros

  • URLs to be tested can be added in a simple way
  • Ease of configuration of load test parameters such as number of users, AWS region, ramp-up time and test scenario duration
  • Smaller learning curve: a simple test can be executed in minutes
  • Ability to create framework for simple tests and then export it to JMeter for further advanced scripting

Cons

  • Not suited for advanced scenarios such as using parameterization or using dynamic data sets
  • No support for BLUs.

Apache JMeter

JMeter is arguably the most feature-rich and powerful open-source load testing tool available at the moment. You can start from scratch with it, but it also allows you to build upon basic tests created with our very own Test Builder with more advanced techniques such as:
  • Full parameterization of variables instead of having to hardcode values into each request
  • Capturing and verifying a wide variety of responses for manipulation & re-use
  • Building quite structurally complex scenarios
and many more features!

Pros

  • Open-source
  • Platform-independent (Java based)
  • Proven in Enterprise-level testing
  • Support for a wide variety of standard and 3rd party add-ins which enable you to cover almost all possible types of web performance testing scenarios
  • Ability to create very complex load tests (if needed)
  • Ability to use a Ruby DSL code rather than the default tree view. This is accomplished by using the Ruby-jmeter gem which also allows you to run tests in Flood.

Cons

  • No support for BLUs
  • Does not run client-side Javascript

Gatling

Gatling is a well-proven and powerful open-source load testing tool. Using the intuitive Scala scripting language, it helps even first-time users get up and running fairly quickly.
Like JMeter, it has powerful features:
  • Full parameterization of variables
  • Ability to capturing and verify a wide variety of responses for manipulation & re-use
  • Enables the development of complex scenarios

Pros

  • Open-source
  • Easy to use and descriptive Scala scripting language used to create tests
  • Proven in Enterprise-level testing
  • Ability to create very complex load tests (if needed)

Cons

  • No support for BLUs
  • Does not run client side Javascript
Trying to choose between Gatling and JMeter? Check out our blog post on situations where each is better.

Selenium

Selenium is a very powerful open-source testing tool mainly used for automated functional testing via interacting with browser-level objects. It can be very useful even for load testing as it allows users to re-use existing functional tests and run them with virtual concurrent users.
One of the main attractions in choosing Selenium over a protocol-level load testing tool such as JMeter or Gatling is the ability to create your scripts in user-based functional flows rather than looking at how to simulate users on a protocol level. PLUs are often much harder to script for complex applications.

Pros

  • Open-source
  • Supports a number of languages to develop scripts including Java, which is what Flood supports
  • Ability to create tests using functional flows instead of diving into the protocol and coding every single request
  • Ability for virtual users to execute client-side javascript
  • Functional tests can be reused as load tests

Cons

  • Concurrency is limited due to resource-intensiveness of using BLUs

​Flood Element

Element is a new tool that we developed as a more cost-effective way to run BLUs than Selenium. Unlike Selenium, which is built for functional testing, Element was built with performance testing in mind first. Element is open-source and free to use.

Pros

  • Open-source
  • Uses Javascript (Typescript) as a scripting language, which makes it easy to adapt Selenium scripts for Element
  • Ability to re-use load testing scripts for functional testing
  • Runs more users per node than Selenium

Cons

  • Limited support for load testing: currently only Flood can run Element load tests
  • Runs fewer users per node than protocol-level tools