With a plethora of options to choose from, consumers expect an application to have lightning-fast, glitch-free performance. Tech companies therefore need to build faster products that stand out in an intensely competitive market. By conducting performance testing—an automated process that measures the performance of applications and identifies bottlenecks—tech leads, developers, and managers can optimize the application’s performance and their underlying systems, ultimately improving user experience and satisfaction.
But performance testing, for all its benefits, isn’t common in tech organizations’ cultures: While the tests are relatively easy to run, tech organizations struggle to deploy this method in their large-scale distribution systems. These systems have complex, interconnected architecture made of multiple components—including third-party systems that may fail to accommodate high request rates—that make it challenging to identify where the problems lie and, thus, where to run performance testing. Issues with an application, including inefficient data flow (such as the retrieval of redundant data) and faulty code (which puts pressure on instance resources such as CPU or memory usage), also make integrating performance testing more complicated. Additionally, developers may not have the right tools to support a robust performance-testing process.
In this post, we discuss three types of performance testing—load, stress, and endurance testing—and the challenges of performance testing in a large-scale distributed system, illustrated by a use case from a revamped retail banking application.
Types of performance testing for the application stack
Tech organizations can run three main types of performance testing on the application stack to ensure that users have a seamless experience.
Load test
Load tests examine an application’s response time and throughput to determine if it meets its defined service-level objectives while operating under an estimated volume of user traffic for an estimated duration of time. Load tests deploy multiple concurrent simulated users who navigate different user journeys based on real-life use cases, such as a user making a transfer through a bank application. Load testing ramps up user traffic via continuous integration/continuous development (CI/CD) using only critical-path scenarios. Once the test reaches an estimated or peak volume of simulated users, it maintains the volume for the estimated duration of time and then ramps down user traffic.
Stress test
A stress test evaluates an application’s breaking point, its bottlenecks, and how it recovers from failure; it measures an application’s capability to scale and recover under extreme loads of users. A stress test deploys a steady stream of simulated users, incrementally raising user traffic until the test indicates a breakdown. Tech organizations can perform stress testing ad hoc before major events or promotions.
Endurance test
Endurance tests identify points of failure during application use by measuring the amount of resources (for example, servers, memory, or network infrastructure) the application uses. Endurance tests use a sustained workload (made up of simulated users or requests per second) that incrementally raises traffic volume, sustains its level for longer than a load test, and incrementally decreases traffic. Tech organizations can schedule endurance tests nightly or weekly via CI/CD.
Challenges with performance testing in large-scale distributed systems
Large-scale distributed systems are a network of interconnected computers working together to handle significant amounts of data and high volumes of traffic while preserving fault tolerance (Exhibit 1). Tech organizations can deploy an application stack on top of a distributed system, leveraging its abilities to enhance the system’s scalability, performance, and fault tolerance.
The complex, multilayered architecture of distributed systems complicates performance testing because each layer requires a different approach to testing. For example, APIs and back-end services require load, stress, and endurance testing. Collectively, the architecture has many interdependent components that require a variety of tools and infrastructure assets to conduct performance testing.
While running performance tests on the application stack, developers may encounter performance issues such as long start-up times, poor response times, and client-side errors in the distributed system.
Performance testing a redesigned retail banking application
A global retail bank’s revamped mobile-banking application that served more than two million customers was experiencing multiple breakdowns under high traffic. The mobile app is hosted via on-premises technologies and integrates with legacy systems in the backend (Exhibit 2).
Testers ran load and endurance testing using a k6 tool to locate issues in the revamped application architecture. Load testing required 6,000 concurrent users for one hour, while endurance testing required 3,000 concurrent users for 12 hours.
Performance-testing approach
Testers hosted the application stack and third-party systems on OpenShift, a separate performance-testing environment. In OpenShift, testers created a separate namespace with a larger pool of physical resources, allowing them to change the configuration of physical resources. Testers configured third-party systems to be production-like in terms of allocated physical resources and configuration.
Solving for common challenges
While running performance test cases, developers encountered five common challenges, determined their impact on the application stack, and deployed a solution:
- Extra load time on downstream systems. While using the application, simulated users waited longer for pages to load due to sluggish API performance. The solution implemented APIs to follow a lazy information-fetching approach and retrieve information only when absolutely needed.
- Bottlenecking in the connection pools between systems. When the system reached a specific number of concurrent users, response times spiked and the system timed out. The solution was to fine-tune the connection pools (such as database configurations) and keep-alive HTTPS connections whenever possible.
- Difficulty in identifying nonperformant code. It took more time and resources to debug performance issues and enhance application code. The solution was to implement tracing—that is, tracking a single user’s journey through the entire application stack, visualized with the appropriate tools (such as data visualization tools or app insights). This enabled testers to measure execution times on granular levels.
- Problems setting up environment resources and configuration to be production-like. The system was not able to scale to production load with insufficient physical resources or faulty software configurations. The solution was to inspect the physical resources of all systems in the stack and cross-check configuration with production.
- Issues preparing testing data to be production-like. There were errors and crashes during performance test journeys due to unfit users or faulty test results stemming from bias in the data (for example, users belonging to the gold or silver banking tier). The solution was to retrieve user data that followed well-defined criteria for test journeys and mimic the production statistics from various users.
Results
The application could achieve the required load (that is, 6,000 concurrent users) while keeping the API response times within the desired ranges. The application went live with no performance issues.
Performance testing is a critical aspect of software development that enables tech leads, managers, and development teams to identify bottlenecks, optimize resource usage, and enhance user experience. Highly mature companies that prioritize adopting a performance-testing culture can build high-performing applications that can handle real-life customer demands, giving those companies a competitive advantage in today’s fast-paced digital landscape.
Erdem Senol is a partner in McKinsey’s Abu Dhabi office, Mohamad Assaf is a senior software engineer in the Doha office, and Sallah Kokaina is a senior director of software engineering in the Casablanca office.