NY Fed: The Failure to Forecast the Great Recession

The Failure to Forecast the Great Recession
Simon Potter
November 25, 2011

>

Experience shows that what happens is always the thing against which one has not made provision in advance.
— John Maynard Keynes¹

Our best plan is to plan for constant change and the potential for instability, and to recognize that the threats will constantly be changing in ways we cannot predict or fully understand.
— Timothy Geithner²

>

The economics profession has been appropriately criticized for its failure to forecast the large fall in U.S. house prices and the subsequent propagation first into an unprecedented financial crisis and then into the Great Recession. In this post, I examine the performance of the forecasts produced by the economic research staff of the Federal Reserve Bank of New York (New York Fed) over the period 2007-10 and consider some of the reasons why we, like most private sector forecasters, failed to predict the Great Recession. This spreadsheet contains staff forecasts, the outcomes, and a standard measure of private sector forecasts—the Blue Chip consensus. In addition, staff material prepared for bi-annual meetings of the New York Fed Economic Advisory Panel provide some further insights into the evolution of the staff outlook.

The staff forecasts of real activity (unemployment and real GDP growth) for 2008-09 had unusually large forecast errors relative to the forecasts’ historical performance, while the forecasts for inflation were in line with past performance. Moreover, although the risks to the staff outlook were to the downside throughout this period, it wasn’t until fall 2008 that a recession as deep as the Great Recession was given more than 15 percent weight in the staff assessment.

How Bad Were the Forecasts for Real Activity?

Economic forecasters never expect to predict precisely. One way of measuring the accuracy of their forecasts is against previous forecast errors. When judged by forecast error performance metrics from the macroeconomic quiescent period that many economists have labeled the Great Moderation, the New York Fed research staff forecasts, as well as most private sector forecasts for real activity before the Great Recession, look unusually far off the mark.

One source for such metrics is a paper by Reifschneider and Tulip (2007). They analyzed the forecast error performance of a range of public and private forecasters over 1986 to 2006 (that is, roughly the period that most economists associate with the Great Moderation in the United States).

On the basis of their analysis, one could have expected that an October 2007 forecast of real GDP growth for 2008 would be within 1.3 percentage points of the actual outcome 70 percent of the time. The New York Fed staff forecast at that time was for growth of 2.6 percent in 2008. Based on the forecast of 2.6 percent and the size of forecast errors over the Great Moderation period, one would have expected that 70 percent of the time, actual growth would be within the 1.3 to 3.9 percent range. The current estimate of actual growth in 2008 is -3.3 percent, indicating that our forecast was off by 5.9 percentage points.

Using a similar approach to Reifschneider and Tulip but including forecast errors for 2007, one would have expected that 70 percent of the time the unemployment rate in the fourth quarter of 2009 should have been within 0.7 percentage point of a forecast made in April 2008. The actual forecast error was 4.4 percentage points, equivalent to an unexpected increase of over 6 million in the number of unemployed workers. Under the erroneous assumption that the 70 percent projection error band was based on a normal distribution, this would have been a 6 standard deviation error, a very unlikely occurrence indeed.

Did We Calibrate the Risks to the Forecast Appropriately?

Of course, there is much more to forecasting than the point forecasts reported in the spreadsheet. In particular, it is crucial to assess the uncertainty and risks around any point forecast.

Throughout this period, the uncertainty and downside risks assessed around our point forecast were substantial relative to economic fluctuations in the Great Moderation. One way of gauging the appropriate calibration of downside risk is to measure the depth of the implied recession if the risks were realized.

The chart below does this by considering the probability distribution of the four consecutive quarters with the lowest GDP growth in a recession. It presents results based on the staff outlook in April 2008 and November 2008. The depth of the mild recessions shown in the chart was typical of the type of recessions expected during the Great Moderation. The actual depth of the 2007-09 recession as gauged by this metric is currently estimated to be 5 percent. As the chart shows, it was only by November 2008 that the probability of the actual outcome was above 15 percent.

Upon seeing this type of calculation, Robert Barro, a Harvard professor and member of the New York Fed Economic Advisory Panel, noted that the decline in real stock market values in the United States was similar to that observed in countries experiencing depressions. Taking this relationship into account in the calibration of the downside risks produced about a 50/50 chance of the currently observed depth of the Great Recession (see the March 2009 Wall Street Journal op-ed article by Barro for his assessment of the probability of a depression).

Recent Forecast Performance

In contrast, the New York Fed staff forecasts for 2010 made in 2009 and early 2010 are quite accurate (under the assumption of no major revisions to the estimates of GDP growth in 2010). This accuracy, however, has not been sustained through 2011. As widely discussed by a number of Federal Reserve officials, the level of real activity in 2011 has been disappointing relative to expectations. This shortfall is evident in the chart below, which compares forecasts for GDP growth in 2011 and 2012 produced in April and October 2011. However, this chart also depicts the uncertainty and risks around the staff forecast as of April 2011. Given the uncertainty around the April forecast, the subsequent changes to the outlook are not very surprising. On the other hand, near-term downside risks to this forecast were low compared to other forecasts produced in the last four years, so the direction of the change was more surprising.

Why Did We Fail to Forecast the Great Recession?

The quotations from Keynes and Geithner at the start of this post capture the importance of constantly striving to ensure that policy is robust to unexpected events. As explained in much of the recent work of the 2011 Nobel Prize–winning economist Tom Sargent, the unexpected events for which policymakers need to make provision have the characteristic of being the most likely unlikely bad event. The collapse in housing prices and its propagation to the economy certainly fit this description.

A leading example of how effective a robust approach to policymaking can be is the 2009 Supervisory Capital Assessment Program. In this program, large U.S. banks were evaluated against a capital standard under the assumption of a longer and deeper recession than contemplated in the prevailing consensus estimate. The idea was that if banks had sufficient capital to continue performing their intermediation function under this more adverse scenario, the scenario was less likely to occur.

Indeed, in the period leading up to the financial crisis, analysts who were suspicious of the stability of the Great Moderation, such as Nouriel Roubini, offered assessments that proved to be significantly more accurate than the point forecasts of New York Fed research staff or most professional forecasters in gauging the potential for unlikely bad outcomes. On a more positive note, if one compares the downside risk in the New York Fed research staff outlook with that of the Survey of Professional Forecasters (see the chart below from April 2008), there is some evidence that we had a more sober assessment of the risk of a severe downturn than did private sector forecasters.

Looking through our briefing materials and other sources such as New York Fed staff reports reveals that the Bank’s economic research staff, like most other economists, were behind the curve as the financial crisis developed, even though many of our economists made important contributions to the understanding of the crisis. Three main failures in our real-time forecasting stand out:

1. Misunderstanding of the housing boom. Staff analysis of the increase in house prices did not find convincing evidence of overvaluation (see, for example, McCarthy and Peach [2004] and Himmelberg, Mayer, and Sinai [2005]). Thus, we downplayed the risk of a substantial fall in house prices. A robust approach would have put the bar much lower than convincing evidence.
2. A lack of analysis of the rapid growth of new forms of mortgage finance. Here the reliance on the assumption of efficient markets appears to have dulled our awareness of many of the risks building in financial markets in 2005-07. However, a March 2008 New York Fed staff report by Ashcraft and Schuermann provided a detailed analysis of how incentives were misaligned throughout the securitization process of subprime mortgages—meaning that the market was not functioning efficiently.
3. Insufficient weight given to the powerful adverse feedback loops between the financial system and the real economy. Despite a good understanding of the risk of a financial crisis from mid-2007 onward, we were unable to fully connect the dots to real activity until 2008. Eventually, by building on the insights of Adrian and Shin (2008), we gained a better grasp of the power of these feedback loops.

However, the biggest failure was the complacency resulting from the apparent ease of maintaining financial and economic stability during the Great Moderation. Perhaps most important, as noted by some analysts as early as the 1990s, these adverse consequences of the Great Moderation were most likely to arise from the actions, judgments, and decisions of financial market participants:

Longer stretches of economic growth imply greater leverage and complacency and thus, greater financial problems when recessions do occur.
–William Dudley and Edward McKelvey³

___________________________
¹Letter to Jacob Viner, June 9, 1943, Collected Writings of John Maynard Keynes, ed. Donald Moggridge, vol. 25. London: Macmillan, 1980.

²Letter from the Chair, Financial Stability Oversight Council Annual Report.

³The Brave New Business Cycle: No Recession in Sight, Goldman Sachs Economic Research Group, January 1997.