Apache JMeter is one of the most popular open-source tools for performance and load testing. In this article, we shared a few Logicify best practices, tips and lifehacks to make testing with JMeter even more effective and indicative of a web app performance.
Performance Testing: Definition, Types, Purposes
Performance testing helps to determine responsiveness and stability of a software application under a simulated workload that is equal to or higher than expected. No other QA practice gives an idea about the number of concurrent users a website could handle without performance degradation.
Writing an effective test plan for performance testing is challenging; in most cases, reason for that being a lack of specific testing goal. Quality assurance engineer performing the testing should determine what he/she is doing this for:
- to check/prove that the system is capable of handling the expected load;
- to determine a system component that causes performance degradation, which could later be optimized;
- to compare capacity of two or more systems or environments.
Depending on the testing goal, different types of performance testing should be practised.
- Load testing: to analyze the system’s behavior under expected load. While an expected number of virtual users simultaneously “attack” the system, it is easy to record the maximum response time of its major components and discover potential bottlenecks.
- Stress testing: to determine the system’s throughput limits and overall behavior for a potential disproportional or extremely high load. This gives an idea about the maximum number of users the system could handle and how long this would take the system to recover after it fails.
- Peak testing: to subject the system to rapid intensive load peaks followed by regular load. Such load profile can be seen in the real world, consider Black Friday and consumer electronics e-commerce solution: it might receive 10x normal load on the day.
- Stability testing: to ensure the system handles the expected load during a long period of time. This type of testing helps to determine memory leaks and performance degradation, i.e. longer response time after the system has been loaded for a while.
- Volume testing, to see how the system reacts while the quantity of stored and used data is being gradually increased.
Ideally, performance requirements for a system should be outlined in the specification, performance test plan, or any other documentation on the project’s requirements engineering stage. These requirements are based on the company’s business plan and long-term goals. However, more often than not, on the specification stage, there is no defined maximum response time for a system under a certain server load.
JMeter as a Tool for Load Testing
- JMeter is an open-source application.
- It does not require complex installation. Download the installation package, install the tool and simply click *.bat file to run it.
- JMeter’s basic functionality can be extended by additional plug-ins.
- It can be integrated with BeanShell, Java-like scripting language, and Selenium, a framework for web applications test automation.
- The tool has a nice logging and visualization of testing results in graphs, diagrams, tables etc.
- It supports distributed testing - heavy load could be tested for a single, multiple, or a network of client machines controlled from a single node.
- JMeter supports basic protocols for the test subject: LDAP, SOAP, FTP, HTTP among others.
However, test results you receive with JMeter are (sometimes and to a given extend) a black-box. This concerns the cases when the bottleneck is not in the app or server under testing but local machines’ hardware limitations and/or network configuration. If you fail to create a certain load on server because of these factors, you could perform distributed testing. It is supported by JMeter out-of-the-box functionality and allows to simultaneously launch tests on multiple nodes joined with the server by the same network.
A test plan in JMeter contains the settings and steps to be executed. Every test plan begins with setting a Thread group, whose properties control the following:
- Number of Threads (users) - the number of virtual users that simultaneously execute the given test case
- Ramp-up Period - the period (in seconds) that tells JMeter how long it takes to start the full number of threads chosen. For example, with 30 threads and 180 seconds ramp-up period, a new thread would start every 6 seconds.
- Loop Count - the number of times each user would execute the test.
Each test thread will execute the test plan independently. Multiple threads simulate concurrent connections to the server. The server load is adjusted as you tune Thread group parameters.
Most frequently used controllers in JMeter are Samplers and Logical Controllers; they drive any testing process. All Controllers and Samplers are placed under a Thread group. Samplers tell JMeter to send requests to the server (via FTP, HTTP, LDAP etc) and wait for its response. They are processed in the order they appear in the tree. To modify the number of samplers or customize any other JMeter logic, you could use and combine different Logical Controllers.
JMeter thread executes Samplers without pausing, which does not resemble the behavior of real website users. To add a certain delay between samplers, you could use available Timers. Timers are added hierarchically before Samplers or Controllers to restrict them.
The outcome and information about any test case gathered by JMeter is accessed through Listeners, which allow to view test results - in HTML/XML files or visuals (graph, tables), and save them for further use.
To find an optimal combination of JMeter elements for every specific test case, it is usually required to run the test script multiple times.
10 requests vs 10 requests
The Thread group properties should be setup wisely and follow specific testing goals. Let’s consider two Thread groups examples:
Though in both cases, there would be 10 requests sent to the server (if the test plan contains just 1 request), the loads they’d create would differ. In Example 1, 10 users would start executing the test case almost simultaneously and send 10 individual requests. In Example 2, one user would execute the test case 10 times sequentially. So, the load on server created per Thread group 1 would be higher.
The ramp-up period should be set wisely. With 0-second ramp-up, JMeter would immediately create all virtual users, causing undesirably high load on server. On the contrary, with a long ramp-up period, some virtual users may already finish their job and terminate, while others have not yet started. Then the actual number of users in the system would be lower than expected. This is undesirable too.
Here are a few tips on how to choose the number of virtual users for an effective JMeter test plan.
How to Choose the Number of Threads for a Test Plan
The number of virtual users (threads) executing the test would depend on multiple factors, such as your goal while testing, parameters of your local PC and the network, and script complexity. Before you decide on the optimal number of virtual users for your JMeter test plan, consider the following tips.
1. Examine Server Logs
Server logs contain the history of web-requests with users’ IP-addresses, date and time. So, a graph showing the server load over a week would give you an idea about the average number of real users and their requests (hits) for low, normal, and high server loads. Target the peak number of requests as opposed to average one to be replicated in your JMeter test plan.
Though JMeter does not offer a native listener to identify the number of active virtual users for a thread group in every given point of time, in Logicify we use a third-party plug-in called Active Threads Over Time. It presents this number on simple yet laconic graphs.
Approaches to Server Loading
While load testing, you could build up various loads on the server by adjusting JMeter Thread group properties.
- Linear load - one of typical server loads while performance testing.
Linear load is good for majority of cases, yet test results would not be indicative if the server crashes during the load rise. In this case, it makes sense to examine the point where the system failed and start from there to determine its maximum load.
- Step load - allows to let virtual users out in chunks, for example, 25, then 25 more, and 25 more. This approach is more reliable and indicative of system performance under different loads, with varied number of virtual users executing the test in every given time interval.
Native JMeter functionality does not support such a complex load scenario, so at Logicify we use a third-party plug-in called Ultimate Thread Group to set Thread group parameters and examine the system behavior after every new chunk of virtual users is added. This plug-in is also good for stress testing and determining the so-called “system saturation point” - a load point where further increase in the number of users leads to longer response time or system instability.
- Peak load - short-time extremely intensive load on the server. It allows to examine the reaction of the system on such extreme load jumps and the way it recovers back to its initial performance indicator after the peak load is over.
2. Investigate Google Analytics for a Website
If the website under testing is available publicly, Google Analytics reports on Audience and Behavior would also be helpful in determining the optimal number of virtual users in Thread groups. When testing a company website, the total count of requests (hits) could be predicted based on the number of employees, their working hours and schedule, plus a certain number of request added on top of these - as a reserve.
3. Check System Requirements
Technical requirements for a system performance should be documented in the specification or performance test plan and discussed with the client in advance. They aid in finding the optimal number of virtual users for testing.
If you connect the server with the webpage and JMeter client via local network, with a more stable connection as compared to the Internet - this could be figuratively called “lab” testing - you minimize the number of factors affecting response time and focus more on the system itself to find potential bottlenecks. On the other hand, connecting to the server via the Internet is a more true-to-life scenario. Yet, it makes root cause analysis and issues isolation harder.
4. Anticipate Traffic Splashes
Find out if the website may see a rapid splash in visitors while you plan performance testing. For instance, when testing an e-commerce application, be ready for seasonal sales or Black Fridays, which attract more visitors. You should be sure that the website is ready for such hit-and-run raids and execute load testing with a large number of virtual users in advance.
Choosing the optimal number of virtual users in JMeter makes testing results true-to-life and more indicative.
How to Make your Test Plan Effective?
Here are a few more tips on making JMeter test plan even more efficient.
1. Make Test Plan Realistic
Test plan should replicate the actual journey of a user on the web-site. The sum of experiences in customer journey could be taken from Google Analytics Behavior Flow (if the website is available publicly) or drawn based on typical user behavior on a certain website type. For e-commerce websites, for instance, a typical customer journey would be landing on the Home page, searching for a product, reading through product description, adding it to cart, checking out.
1.1. Mind the Funnel
The funnels you use while testing should be realistic too. As a rule, the number of users who visit a Home or Landing page is higher than the number of users actually buying a product. To take this into account while testing with JMeter, you could use Throughput Controller.
It allows to define a ratio of users to have every possible experience on the website. For instance, you could let 100% of your virtual users visit the Home page, 90% of this amount would look for a product. 60% of these users would add a product into their cart, and only 35% would proceed to checkout to actually buy the product.
1.2. Add “Think Time”
When real users visit a website, they usually need the so-called “think time” to read the content through, enter search keyword, and analyze search results. These pauses between requests, called Timers in JMeter, should also be added into the test case to make it even more true-to-life. Do not just add 1 - 2 seconds delay between every new request; use an additional, random one in-between.
Here is an example of a Uniform Random Timer in JMeter, with a constant (200 milliseconds) and a random (0 - 750 milliseconds) delays.
2. Make Test Plan Flexible
In the majority of cases, JMeter test scripts are launched via Command Line - easily and quickly. However, for every change in Thread group parameters, you need to open JMeter script in GUI mode. This is time-consuming and not convenient. To avoid this, you could use variables and property functions for some Thread group parameters:
This way, property functions could be specified directly in the Command Line:
jmeter -t TestPlan.jmx -Jusers=10 -Jcount=50
This saves a great deal of time and efforts and makes test scripts far more flexible.
Moreover, while stability testing, when a system is being tested for a long period of time (5 - 10 hours), you could watch the testing progress and regulate server load in real time using Constant Throughput Timer. With this timer, server load is adjusted with pauses, so target throughput does not exceed the set parameter (in samples per minute):
You could make the target throughput a variable too. This would allow you to adjust it via Command Line in real time, as the test script is being executed.
3. Use Unique Data for Virtual Users
To make your virtual JMeter users behave more like real website users, make sure they use unique logins & passwords, search for different product(s) with various keywords, and add different contact details while checkout. Unique user data, such as logins and passwords, could be stored in CSV-files and imported into JMeter while test execution.
4. Add Demo Data to the Website
There is no sense and use in performance testing for a website with 0 content. So before you start executing the test plan in JMeter, make sure there is demo data on the website, such as catalogue, product pages with description, forum. Otherwise, the outcome of your testing would not be objective.
5. Analyze Previous Test Reports
Test reports are provided by JMeter listeners, which capture responses coming back from server while test execution and showcase them in tables, graphs, or log files. Some listeners include Summary Report, Aggregate Report, Aggregate Graph. Let’s have a closer look at Summary Report columns:
- Label - with the name/URL of a specific request
- Samples - with the number of virtual users per request
- Average - showing the average time between the moment when a request was sent and the moment it received a response
- Min - showing minimal response time
- Max - showing maximum response time
- Std. Dev. - showing standard deviation from the average value of sample response time
- Error % - with percentage of failed requests
- Throughput - with the number of requests processed by the server per a time unit (seconds, minutes, hours)
- Kb/sec - with the throughput measured in Kilobytes per second
- Avg. Bytes - average response measured in bytes
Though the report is informative, you should not judge about the test outcomes solely on these tables columns. Combine table values with graphs drawn by Listeners.
Visual representation of test reports is especially helpful in analysing results of stability testing, i.e. when the server was subjected to a high load for a few hours. Let’s say the average response time for a transaction “Product Search” was around 5 seconds, which is an expected value. However, during the last 20 minutes of test execution, server response time for the transaction increased to 20 seconds, which is way longer than the expected value. This points out to a potential bottleneck causing system degradation, but the average value in the Summary Report would be around 5,99 seconds. This way, the degradation would go unnoticed. It would be seen only on a graph drawn by the plug-in Response Times over Time, which allows to track every single transaction and locate the problematic ones to further address the bottleneck.
It is important to find the system component that causes degradation and improve it.
JMeter as a flexible and smart tool for performance testing of both static and dynamic web resources. To make the most use of it, keep the goal of testing at the head, compose a wise and realistic test plan with both native and third-party JMeter tools and adjust it in the process. This would help you discover system bottlenecks and potential stress points to further eliminate them.
- Automated Markup Testing with Galen Framework
- Grafana as Yet Another Tool for Technical Monitoring of Software Products We Build
- Graylog as a Tool for Technical Monitoring of Software Products We Build