May 31, 2011 Articles

Financial Model Mistakes Can Cost Millions of Dollars

Ongoing vigilance regarding financial models is critical to gauging whether the models are working as they should.

By Susan Mangiero

Aristotle wrote, "Quality is not an act; it is a habit." Applied to investment matters, the message is paramount. Ongoing vigilance regarding financial models is critical to gauging whether the models are working as they should. The failure to catch a financial model's mistakes can cost millions of dollars and create a litany of problems. Some financial services organizations and their investors have learned the hard way that computer-driven models are only as good as the people in charge of creating, reviewing, and modifying them as needed. Given the fact that the proper management of more than $30 trillion in global assets significantly depends on having good financial models in place, it is important to understand what can go awry and how to avoid material losses.

A recent example of losses resulting from a financial model error should serve as a wakeup call. Three AXA Rosenberg entities settled a $242 million enforcement action by the U.S. Securities and Exchange Commission (SEC) related to a computer programming error. According to SEC Director Carlo di Florio, the firm swept the error "under the rug" as opposed to dealing with it immediately by communicating with clients. Doing so cost the firm $217 million to be paid as recompense to investors who were harmed, $25 million in penalties, and the costs of "hiring an independent consultant with expertise in quantitative investment techniques who will review disclosures and enhance the role of compliance personnel." Carlo di Florio, Speech by SEC Staff, Remarks at the IA Watch Annual IA Compliance Best Practices Seminar, Office of Compliance Inspections and Examinations, SEC, March 21, 2011. Bruce Karpati, co-chief of the Asset Management Unit in the SEC's Division of Enforcement, cautioned "quant" managers to be "fully forthcoming about the risks of their model-driven strategies, especially when errors occur and the models don't work as predicted." SEC, Press Release No. 2011-37, "SEC Charges AXA Rosenberg Entities for Concealing Error in Quantitative Investment Model: Firms Agree to Pay More Than $240 Million to Settle SEC Charges."

The Importance of Financial Models
Nearly every financial transaction that takes place between capital market participants involves a model of some type. Examples include a private equity fund manager who forecasts projected cash flows of a portfolio company or an endowment seeking to optimize its mix of stocks, bonds, and alternatives such as hedge funds or real estate. Valuation models are a mainstay in determining the right price for a stock, bond, or derivative instrument like an executive stock option. Countless asset managers rely on computer algorithms to determine how much to hedge particular holdings. "Quants" deploy various formulas to guide how they trade. Main Street investors are undeniably impacted by the pervasiveness of computational finance in Wall Street dealings. A failure to build correct models and review them over time for continued appropriateness can, and has, resulted in large economic losses for numerous individual and institutional investors.

Models are a means to an end. Wall Street does not hire programmers simply to create elegant mathematical code. The transfer of securities (whether via an exchange or through a private negotiation) with a tally of more than $30 trillion and the associated bottom line of broker-dealers, advisers, portfolio managers, and investment banks are tied to algorithms that dictate, predict, and allocate. Millions of individuals move money every day as a result.

Conceptually, a model is meant to be a numerical reflection of financial conditions in a perfect world that is characterized by rational buyers and sellers, no unprecedented directional jumps in price, zero transaction costs, and a free exchange of information. As the last few years have revealed, however, markets can get messy, and people do not always behave in a disciplined fashion.

Care must be taken to construct a model and to test it. Underlying assumptions must be revisited on an ongoing basis, preferably by an independent expert who will not receive a raise or bonus tied to flawed results from a bad model. Someone has to kick the proverbial tires to make sure that answers make sense and to minimize the adverse consequences associated with mistakes in a formula, bad assumptions, incorrect use, wild results that bear no resemblance to expected outcomes, difficulty in predicting outputs, and/or undue complexity that makes it hard for others to understand and replicate outputs. Absent fraud or sloppiness, precise model results may be expensive to produce and therefore unrealistic in practice. As a consequence, a "court or other user may find a model acceptable if relaxing some of the assumptions does not dramatically affect the outcome." Susan Mangiero, "The Risks of Ignoring Model Risk" in Litigation Services Handbook: The Role of the Financial Expert (Roman L. Weil et al, eds., John Wiley & Sons, 3d ed. 2005).

Given the importance of models, regulators are watching. During the previously mentioned March 21, 2011, presentation, di Florio listed valuation, asset verification, and portfolio management as exam focus areas, all of which are impacted by the viability of financial models.

Potential Damages Resulting from Financial Model Errors
When things go wrong in an asset manager's financial model, investors can incur outright losses along with a foregone opportunity to have allocated their money elsewhere had they known about model problems in a timely fashion. Economic losses likewise occur for an asset manager when current clients pull millions of dollars or prospective investors go away, limiting future fees that can be earned as a percentage of assets under management.

Damages also occur when assets or liabilities are incorrectly priced because inputs to a valuation model are erroneously calculated. Garbage In, Garbage Out (GIGO) spells trouble. For example, what an investor ought to pay for a complex instrument such as a mortgage-backed security relies on inputs such as projected interest rates and the extent to which homeowners will repay their debt and at what level. If interest rates have just fallen considerably and a large number of borrowers refinance as a result, another plunge in interest rates is unlikely to encourage yet another large spate of prepayments. With respect to projecting interest-rate paths that will be used to discount security cash flows, the shape of the yield curve, macroeconomic considerations, and volatility are just a few of the factors that comprise the "nested" model. Said another way, if interest rate projections and/or prepayment assumptions are not effectively modeled first, the results from applying the mortgage backed security valuation model will be influenced accordingly.

Using unsuitable models also causes damages. For example, the use of a simple Black-Scholes option pricing model is often criticized as a less than perfect choice for valuing executive stock options because few stock options trade in an open market, their exercise is restricted, and issuance by closely held companies complicates stock price determination. Susan Mangiero, "Model Risk and Valuation," Valuation Strategies (March/April 2003).

When a model is used for risk-management purposes such as determining the size of a hedge, mistakes may result in too much or too little protection for the investor or issuer of debt (Hedging techniques can be a useful tool for both investment and corporate-finance purposes.). Neither outcome is ideal. In addition, hedging may end up costing more than it should if a risk-averse investor buys or sells a derivative instrument that has been incorrectly valued.

For long-term investors such as defined-benefit plans or university endowments, models are frequently used to determine how to properly allocate large sums of money. Should model problems occur, institutions may find themselves unduly overexposed or underexposed to certain industries, issuers, and/or money managers. This in turn opens a veritable Pandora's box in the form of potential fiduciary breach, imprudence, violating donors' mandates, accelerated funding for plan sponsors, and the cost of potentially having to rebalance.

Mitigating Model Risk
In light of the importance of financial models and the potential damages resulting from model errors, expect more attention to be paid to the use of models and the risk that accompanies poor practices.

A prescription for model risk mitigation includes, but is certainly not limited to, the following action steps:

  • Hire knowledgeable programmers with capital market experience;

  • Create and follow a set of policies and procedures that govern how and who will validate financial models over time and what will trigger revisions in a model(s);

  • Avoid conflicts of interest that would reward managers for ignoring problems and would potentially preclude an independent and objective assessment of problems and related corrective action(s);

  • Test assumptions for validity in stable markets as well as extreme circumstances;

  • Stress a model using a sufficient number of economic scenarios to gauge its predictive power and whether results can be relied upon in both good or bad times;

  • Educate personnel about how a particular model is supposed to work;

  • Establish a response strategy should a problem occur and investors need to be informed before things get out of hand;

  • Scrap models that are overly complex and expensive to replicate;

  • Don't be afraid to ask questions about inputs, data quality, results, and concerns; and

  • Invite informed outsiders to offer an independent and regular critique on a confidential basis.

Conclusion
In the words of famed philosopher George Santayana, "Those who do not remember the past are condemned to repeat it." Certainly, the recent market meltdown offers valuable lessons. For example, as Elliot Noma, managing director with Garrett Asset Management, LLC, rightly points out, "Those traders who assumed adequate diversification of home loans backing mortgage securities were exposed to model risk, just as some of their peers incurred losses for underestimating counterparty default risk." Famed writer Ray Bradbury was not far off the mark when he said, "Living at risk is jumping off the cliff and building your wings on the way down." Executives who ignore model risk do so at the peril of their investors and may want to think about how to explain their actions in subsequent legal proceedings.

Keywords: litigation, expert witnesses, financial model, model risk, valuation

Susan Mangiero, Ph.D., is a CFA charterholder and certified financial risk manager.


Copyright © 2011, American Bar Association. All rights reserved. This information or any portion thereof may not be copied or disseminated in any form or by any means or downloaded or stored in an electronic database or retrieval system without the express written consent of the American Bar Association. The views expressed in this article are those of the author(s) and do not necessarily reflect the positions or policies of the American Bar Association, the Section of Litigation, this committee, or the employer(s) of the author(s).