Load Testing Alternatives to JMeter: Locust, Selenium, or Gatling?

Load testing an application means evaluating it, on the infrastructure that will host it, under realistic or even heavy use. It is probably one of the most important types of testing that any web service can have but is often overlooked since most developers are more concerned with testing the application’s functionality.   

On one of our projects, we prioritized load testing after major infrastructure and refactoring work for our client. The client provides a billing service that allows thousands of users to connect to different institutions to view and manage their billing accounts. 

Load Testing Requirements

One of the flows that we needed to test was a user logging into their account and downloading a bill (PDF). The application uses scripts to fetch information from our REST API endpoints and displays the information for the end-user to interact with. 

Given this functionality, it’s not enough to just load test static pages; the testing needs to: 

• simulate users logging on 
• use the application as a user would 
• find out how many of these users the system can handle at any given time.  

Our Previous Solution: Load Testing with JMeter 

We had previously tested with JMeter, one of the pioneers of load testing, but the team was not in love with what was in place. The feeling was that it was clunky, irritating to set up and get going, resource-intensive, and just not developer-friendly.  

JMeter Negatives

While an open-source industry standard for decades and richly featured, JMeter has a lot of negatives: 

• Barebones out of the box 
• Plugins are required for even a little bit of complexity 
• No support for DevOps due to age (except some support from the SaaS platform Blazemeter) 
• Not Developer friendly, with all test plan creation done through their slow and clunky GUI 

Our Options: Alternatives to JMeter for Load Testing

With our new overhaul to our client’s infrastructures and applications, we needed to create new test plans in order to properly simulate user activity on dynamic pages. Given its shortcomings, we used this opportunity to evaluate alternatives to replace JMeter. After some research, we established a shortlist: Selenium, Gatling, Locust.

Selenium

Selenium is what is known as a headless browser. A headless browser is a web browser that lacks a graphical user interface but provides an automated way to browse and test web pages. We considered a headless browser to ensure that the simulated user traffic for the dynamic pages was loading properly. This way we can interact with the various buttons and with the security tokens that are embedded in the HTML upon loading the page. Quick introduction to Selenium:

• Headless browser
• Open source
• Geared towards web apps
• Supports parallel and distributed testing  

Selenium Positives

• Built for testing web applications and checking UI elements
• Headless browser allows the developer to simulate a user interacting with their web browser and allows the developers to check the behaviour of UI elements
• Supports multiple web browsers, mobile browsers, and scripting languages
• Large and active community

Selenium Negatives

• Selenium is very resource-heavy as it launches a web browser per user 
• Not suitable for load testing as scaling up is very costly given the high resource consumption 
• Does not have DevOps compatibility  
• The GUI is not developer-friendly 
• Needs to be paired with JMeter, which is what we were trying to avoid in the first place 

Selenium’s shortcomings caused us to go back to the drawing board and assess whether or not we really needed a headless browser, and finally we opted not to. 

Gatling

Gatling is probably one of the most popular options on the load testing market right now, and for a good reason. Gatling is Scala based, which is a little weird, but not a deal-breaker. It’s actually easy to get comfortable, especially for those who have experience with Java.  

• Open source 
• Code-based testing framework 
• Geared towards DevOps 
• Runs as a Java application 

Gatling Positives

• Great connection handling engine
• Engine based on Akka and uses asynchronous computer. Reducing performance hit of having many users on a single thread
• Run from a command-line interface (CLI)
• No need for GUI to create and run tests
• Not a headless browser means it uses fewer resources and allows scaling the users to a significantly higher number

Gatling Negatives

• The CLI does not provide a lot of detailed information during runtime.
• For live monitoring via a graphical web interface, you need to integrate into another application called Taurus.
• While performance is better than JMeter, it’s still average compared to other options.

• As a Java-based application, we run into the same setup pains as with JMeter.

Locust

Right off the bat, we saw the potential for Locust as the Python language is much better suited to writing test plans. The CLI provides a nice and simple overview of the current live operations, reporting any errors it encounters along the way.

• Lightweight Python Library
• DevOps Friendly
• Runs on the CLI
• Developer Friendly
• Scalable User Testing

Locust Positives

• Distributed load testing can be scaled up to a large number of users. 
• Python is easier and faster to set up on new machines. 
• Creates a web interface that allows developers to pause/resume the testing.
• Graphs progress as minutes go by and breaks down the data by endpoint.
• Easy to use multi-core CPUs to ramp up testing for a large number of users.
• Not a headless browser.

Locust Negatives

• Barebones out of the box
• Due to the nature of Python, with each user you simulate, the slower the execution of the code would be
• No built-in support for assertions

The Best Solution for Load Testing – Locust 

At the end, we opted to go with Locust, as it best fitted our needs. Python was a very big selling point as it’s much easier to get started on a new machine. It also allows us to use Python’s very large library to custom tune our tests to fit our needs. One library that we were able to make the most use of is BeautifulSoup, a very popular and powerful HTML parser.  It is by far better than all the other HTML parsers that are built into other load testing solutions.  

The asynchronous approach and multi-core support of Locust was another great selling point. I was able to test a very high number of users on my local machine without needing to deploy it on an AWS server with more resources. Down the line, if we do need to simulate more users, we can take advantage of the Locust distributed testing support. This feature allows Locust to be deployed on multiple machines and interact with each other via messaging system.  

The web interface option is also really nice to have. I usually opt for CLI when I can, but for monitoring a large number of results, it’s very helpful to have a graph being updated in real-time.