Model Risk Management

In April 2011, the OCC released Bulletin 2011-12, Supervisory Guidance on Model Risk Management, describing the elements of a sound program for effective management of risks arising when banks use quantitative models to make decisions. Superseding guidance issued in 2000 which focused primarily on model validation, this guidance significantly expands the elements required to meet the supervisory standards. Consistent with other risk management related supervisory guidance, all banking organizations are expected to implement a process – commensurate with the size, complexity, and risk profile of the institution - to identify, measure, monitor, and control model risk. The guidance articulates three categories of activities that should be included in your program: 1) Model Development, Implementation, and Use; 2) Model Validation; and 3) Governance, Policies, and Controls. Does your program measure up?

Model Development, Implementation, and Use

Model development is not something that most Community Banks undertake; more often, the institution acquires a model from a vendor. Nonetheless, the institution retains accountability for evaluating the model to verify that the model’s purpose, design, theory, and logic are appropriate for the institution’s intended use. The institution should, to the extent possible, obtain documentation from the vendor that sufficiently explains the: methodologies and processing components, including the mathematical operations; model merits and limitations; and integrity, relevance, and suitability of data or data proxies used.

During implementation and periodically thereafter, testing of the model should be performed and documented. The type of testing is directly correlated with the type of model; however, testing should confirm the validity and accuracy of the model’s output, re-validate the limitations and assumptions, and confirm the overall functionality and performance of the model. Testing should also include a verification of data flow between, and integration with, other related systems or models. One often overlooked testing process involves the evaluation of feedback and questions from line of business users relating to the reports provided and the model’s output. Finally, the testing should incorporate an evaluation of continued applicability of the model given current business strategy and a re-assessment of the costs and benefits associated with the model. With respect to vendor-supplied models, the vendor should provide the institution with its testing results confirming that the model performs as expected.

Model Validation

The purpose of model validation is to re-affirm that the model is performing as expected. The validation process and activities should: 1) verify that the model is aligned with design objectives and business use; 2) identify the potential impact and validity of limitations and assumptions; and, 3) generally assess the overall accuracy and soundness of the model. Validation should be performed, to the extent possible, by individuals independent of development and use of the model, but with appropriate knowledge, skills, and expertise. The validation framework should encompass three core elements: 1) evaluation of the conceptual soundness, including developmental evidence; 2) ongoing monitoring, including process verification and benchmarking; and 3) outcomes analysis, including back-testing. In addition to reviewing the results of vendor-completed validation, institutions relying on vendor-provided models are expected to perform their own periodic validation of the model. This validation should include a re-evaluation of the appropriateness of any bank-specific customization and vendor-input data and assumptions, an examination of the extent to which vendor-data is representative of the bank’s situation, and an analysis of model performance using the institution’s outcomes.

Governance, Policies, and Controls

Institutions are expected to develop and maintain strong governance, policies, and controls over model risk management. The board and senior management are expected to establish a framework for model risk management, based on a solid understanding of model risk in the aggregate, and addressing the development, implementation, use, and validation of models. Senior management is charged with responsibility for establishing adequate policies and procedures, assigning competent staff, providing oversight with respect to the development, implementation, use, validation, and corrective action associated with identified model deficiencies. Senior management is also accountable for apprising the board of the level of model risk and compliance with policy. The board ensures that risk levels are within tolerance and directs changes where required.

Internal audit, acting at the direction of the board, should assess the overall effectiveness of the model risk management framework. This includes an assessment and evaluation of: 1) the framework’s ability to address risk at the individual model and the aggregate model levels; 2) the related policies, procedures, and internal controls; and 3) the sufficiency of documentation to support the model risk management framework. Audit will also ensure that adequate validation is performed and appropriately documented, and the identified deficiencies are resolved.

Policies and procedures, consistent with the guidance, define responsibilities, include model and risk definitions, and provide for the development, implementation, use, testing, and validation of models. Policies should require maintenance of an inventory of models across the institution, incorporate standards for utilizing external resources, model accuracy, acceptable levels of discrepancies, and define procedures for resolving unacceptable discrepancies. The board will, at least annually, review and, when necessary, revise the policies and procedures to adjust for changes in market conditions, bank products and strategies, bank exposure and activities, and industry practices.

While many of the activities included in the guidance are common industry practice, regulators expect each institution to confirm that its practices conform to the principles contained therein. But it’s vital to remember, programs will vary from institution to institution; each institution’s program will be evaluated in the context of its own size, complexity, and risk profile.