Taking the stress out of operational-risk stress testing

The past few years have seen the emergence of a new normal in the discipline of operational risk, especially in the financial-services sector. Financial institutions have experienced an increased number of significant incidents with major financial implications. These have ranged from cybersecurity breaches to rogue-trading events to problems in sales to large supervisory penalties and class-action lawsuits.

These events have led to heightened supervisory scrutiny of both measurement and management practices in operational risk. In the United States, supervisors have raised the bar for strong operational-risk-management practices and have mandated bank holding companies (BHCs) to perform comprehensive operational-risk stress testing as part of the overall comprehensive capital analysis and review (CCAR) process. Projections of losses arising from inadequate or failed internal processes, people, and systems or from external events must be reported by the BHC as operational-risk losses, a component of pre-provision net revenues.

This paper focuses on the measurement of operational risk, specifically for stress-testing purposes. With practices in operational-risk stress testing still evolving, banks are faced with a range of questions on methodological choices and the corresponding trade-offs. These questions primarily are centered on the challenge in correlating operational-risk losses with macroeconomic factors and business environment and external control factors; the handling of large historical losses in internal loss data sets; stressing historical, current, and future legal losses; and incorporating large plausible events that might occur during the nine-quarter forecast period for stress-testing purposes.

Hence, it is important for BHCs to establish a structured and calibrated approach to operational-risk stress testing. Establishing such an approach will help them avoid supervisory objections (matters requiring immediate attention and matters requiring attention) by suitably addressing rising regulatory expectations. It will also benefit the institution through the establishment of strong foundational risk and business practices, for example, loss-data capture and loss-reduction actions, scenario analysis and risks/controls assessments and corresponding risk-mitigation actions, and getting a dynamic understanding of the true risk profile, including sensitivities of losses and capital to key events and drivers.

Key challenges in operational-risk stress testing

BHCs have been facing a common set of challenges in operational-risk stress testing over the past two to three CCAR cycles. These challenges have occurred in the same areas where a majority of the supervisory objections have been focused:

1. Ensuring sufficiency and quality of data being used for modeling. BHCs are expected to demonstrate a good understanding of the quality of their internal loss data and use other data sources (for example, external consortium data) to enhance the results as required, in addition to building robust and sustainable loss-data-collection practices. Operational-risk loss-data quality has been a long-standing challenge for banks given the wide-ranging sources of these data (beyond the financial systems of the bank) and the dispersed set of stakeholders involved in the data-collection process. Other drivers include a perception that operational-risk loss-data collection and reporting is not mandatory and an aversion to reporting bad news.

2. Correlating operational-risk losses with macroeconomic factors. While it is well understood that operational-risk losses may not always be correlated with macroeconomic factors, BHCs are nevertheless expected to attempt to model operational-risk losses for stress scenarios to the extent that they are able to, and justify the results from a sound statistical standpoint as well as on the basis of business intuition.

3. Estimating legal losses under stress conditions. Legal losses form a large chunk of a BHC’s total operational-risk losses. Hence, it is important to be able to estimate the impact of legal losses—historical, pending, and future—under stressed conditions. The process for stressing legal losses is still evolving from both a methodological standpoint and a process standpoint (for example, deciding which stakeholders should be involved in the process given the privileged nature of the information).

4. Estimating the impact of the future unknowns using scenario analysis. While modeling of the stressed operational-risk losses using historical loss data provides some estimate of future losses, BHCs also need to have a robust scenario-analysis process and choose the appropriate number and types of scenarios in order to estimate the impact from large unknown events that might occur during the nine-quarter CCAR forecast period. While many banks now have a scenario-analysis process in place, their programs often need to be strengthened with regard to use of the right information sources, involvement of senior business leaders, and effective challenge and bias control in workshops.

5. Aggregating total stressed losses across the components and ensuring strong review and challenge of the results. Once the BHC has estimated the baseline losses and the different components of stressed losses, it needs to have a sound methodology to aggregate the results and adequately review and challenge them, using appropriate data and tools.

A structured and calibrated approach to address these challenges

BHCs have in the past used a range of approaches for operational-risk stress testing for CCAR. These include, among others, regression models, loss-distribution-approach (LDA) models, historical averages, and scenario analysis. However, our experience has shown that on its own, any one of these approaches is not sufficient to address the challenges described earlier. Our view is that BHCs need to have a hybrid approach that combines the power of these individual approaches to build up to the total stressed losses for operational risk in a stepwise manner. The exhibit illustrates the stepwise approach, which is described in greater detail in the remainder of this section. (The relative sizes of the four blocks that are shown are purely illustrative; the actual contributions of each block vary from one bank to another.)

Calculating baseline losses

To quantify baseline operational losses reliably, the BHC needs to consider the following elements:

1. The BHC must ensure robust quality of the available historical data. It must also have strong visibility into potential gaps and consider suitable steps to address these gaps, both in the near term for stress-testing purposes and in the longer term to improve the quality of loss data being collected. Such assessments usually include a thorough review of the loss-data-collection process, including the ownership of the first line of defense, reconciliation with other internal sources, and the governance and oversight of the end-to-end process. Comparing the profile of internal loss data with external (consortium or vended) data that are appropriately filtered to include comparable peers is also useful in assessing the overall profile of internal loss data.

2. Once the quality and sufficiency of the internal loss data has been established, the baseline losses should be calculated based on historical average realized losses, taking into account the expected outcome of current or pending operational-loss events, including legal-loss provisions.

3. The BHC should place a very high bar on justifying any potential exclusions of either large loss events or losses arising from discontinued businesses or products or from divestitures.

4. The BHC should also take into account the strategic plan and associated budgets and adjust the baseline to reflect changes in business strategy.

Modeling stressed losses based on historical loss data

While it is well understood that operational-risk losses may not always be highly correlated with macroeconomic factors, BHCs are expected to examine the relationships that might exist in their internal loss data sets.

A robust approach to examining these relationships is to estimate correlations between macroeconomic factors (for example, the ten-year US Treasury rate, unemployment rate, house-price index, and credit-card delinquency) and historical loss frequency and loss severity, respectively.

BHCs should not try to force the use of unstable or unobservable correlations. They should also be able to justify the correlations using strong business intuition and reasoning. If the BHC has limited loss history that limits its ability to model macroeconomic correlations using internal loss data, it can consider the use of suitably filtered external loss data—for example, data from the American Bankers Association or the Operational Riskdata eXchange Association—to compute the correlations. Also, defining units of measure (UOMs) that are more homogeneous than just the Basel event categories and modeling the losses around these UOMs may help in finding stronger statistical relationships between operational-risk losses and macroeconomic factors. For UOMs that do not show relationships with macroeconomic variables, the use of nonparametric modeling approaches can be considered. Statistical tests, for instance, the Kruskal–Wallis test or analysis of variance (ANOVA), can be employed to understand if the frequency and severity are different between stress and nonstress periods, and a stress-multiplier approach can be applied to the baseline to compute stressed losses. BHCs can also try to find correlations between losses and business environment and external control factors (for example, risk and control self-assessment scores or key-risk-indicator values) based on the assumption that these would be affected during the course of macroeconomic stress.

Despite all their efforts, BHCs might still fail to establish a clear relationship between macroeconomic variables and operational losses. This potential outcome is especially likely for efforts to stress severities and is driven by the very nature of operational-risk losses.

Quantifying stressed legal losses

Legal losses form a large part of overall operational-risk losses. Hence, there is considerable regulatory scrutiny of the capture and use of litigation-related information for stress-testing purposes.

Broadly speaking, there are three components of stressing legal losses, each of which should be considered separately:

1. Stressing historical legal losses. Legal losses contained in the historical internal loss data set are stressed as a part of the correlation analysis. The historical legal-loss data should be included in the overall internal loss data set being used to estimate correlations between macroeconomic factors and operational-risk losses.

2. Stressing current, pending, and threatened litigation. The recommended methodology follows these steps: use a robust process for estimating the impact of unfavorable, stressed outcomes on known current, pending, and threatened legal claims; apply sound judgment, taking into account the reasonably possible adverse outcomes based on the specific merits of the cases in question and of similar past cases; apply a suitable estimate of legal fees and expenses, supported by past data and an up-to-date fact base.

3. Stressing potential litigation-related losses. This involves estimating losses from potential litigation actions that are not known at this time and is captured through the scenario-analysis process. If the specific scenarios chosen for the workshops have a litigation component, estimating the severity of this component using the scenario-analysis process will provide visibility into potential future litigation-related losses.

Enhancing stressed-loss results using scenario analysis

Modeling stressed losses based on historical loss data sets has the inherent limitation of not being able to get at the unknown events that might not yet have materialized but are plausible in the future, based on the risk profile of the bank. In fact, such “tail” loss events, if they happen during the nine-quarter CCAR forecast period, are often likely to make up the bulk of the stressed losses. Hence, BHCs are typically expected to have a structured, transparent, well-supported, and repeatable scenario-analysis process that is subject to independent review and validation.

BHCs need to use multiple data and information sources along with strong business inputs to generate a list of potential scenarios that reflect the operational-risk profile of the institution. These inputs can include external sources of industry-standard scenarios. A set of key criteria that can be used to select specific scenarios for discussion in the workshops is described below:

• Plausible. Is the scenario relevant to the risk profile of the BHC? Is the scenario realistic?

• Forward looking. Does the scenario incorporate anticipated trends and emerging risks?

• High severity. Is there large direct financial impact associated with the anticipated loss event, not including opportunity costs?

• Low likelihood. What is the likelihood that the anticipated loss event might occur over a defined time frame, say, once in x years (where x might, for example, be determined by reference to a once-in-a-career type concept or by reference to the implied likelihoods of adverse and severely adverse outcomes as defined by the Fed scenarios)?

To quantify these scenarios in a workshop setting, the BHC needs to ensure the following:

1. strong business representation in the workshops, along with functional and subject-matter experts

2. well-researched and succinctly written preread material that the participants can use prior to the workshops, to get smart on the scenario in order to effectively engage in the discussion

3. strong facilitation by trained facilitators to ensure adequate challenge and bias control

4. bias-controlled ways of quantifying the scenarios, for example, the use of anonymous voting

The process and the outcomes of these scenarios should be documented in a well-structured scenario library to ensure transparency and facilitate strong independent review and validation.

Aggregation of the total operational-risk stressed losses and review and challenge of the results

Once each of the four components described in the exhibit have been quantified, the BHC needs to aggregate these components to calculate the total operational-risk losses for stressed conditions. The simplest approach is to add the severities across each of the four steps to produce the overall stressed-loss estimates. That said, there are certain considerations that the BHC should take into account while aggregating the stressed-loss numbers:

• If there are one or more large (tail) loss events in the historical internal loss data set that is being modeled, the regression models might lead to significant amplification of these losses. In such cases, the BHC needs to be careful while selecting and quantifying scenarios that might capture similar loss events in order to avoid substantial double counting.

• If the BHC believes that through its scenario-analysis program it needs to quantify a relatively broader set of scenarios in order to reflect the true loss profile of the institution, it will be faced with deciding how many of these scenarios to include in the stressed-loss-estimation process. This decision is particularly important since a relatively larger number of low-likelihood scenarios are unlikely to happen at the same time during the same nine-quarter forecast period and might lead to artificially high stressed-loss numbers for the BHC (effectively quantifying losses greater than the implied likelihoods of the Fed’s adverse and severely adverse scenarios).

Finally, once the overall process for estimating operational-risk stressed losses has been executed and documented, the results must be adequately reviewed and challenged by suitable governance forums and committees. In order to facilitate this review and challenge, a range of benchmarking data and tools can and should be presented to the review and challenge committees. These include benchmarks based on historical internal and external loss data, for example, average nine-quarter losses from the internal loss data set, the most recent nine-quarter losses, and the worst nine-quarter losses. If the BHC has a robust LDA model, it may want to compare the severities predicted by the LDA model (for a range of percentile cutoffs, for example, 85th, 90th, and 95th percentiles) with the stressed-loss results derived from the approach described in the previous sections.

---

In our view, it is important for financial institutions to invest early to build the foundational capabilities of strong operational-risk stress testing, which can then transition into a business-as-usual activity for the institution. This involves creating and executing on a plan to strengthen the quality of internal loss data being collected, robust capture of operational-risk events and near misses, rolling out a robust scenario-analysis program with strong business involvement, and ensuring strong ongoing involvement of key stakeholders (for example, legal and compliance) in the program.

These efforts will have direct business benefits in the following ways:

• getting a better understanding of the overall operational-risk profile of the bank, including sensitivities to key events and macro factors

• providing greater visibility into operational-risk losses and loss events, thereby driving efforts to reduce losses

• helping the institution get a handle on unknown risks and the safeguards and controls that may need to be established or strengthened

• driving operational-risk appetite and capital-allocation decisions based on the stress-test results

In addition to the benefits described above, this approach will also ensure that the BHC can avoid supervisory objections in an environment where the bar is constantly rising—and thereby take the stress out of its operational-risk stress-testing activities.