July 02, 2018

The accuracy of benefits transfer for environmental valuation

Mark Berkman, Martha Rogers, and Gina Waterfield

Environmental valuations are an important component of many regulatory and legal proceedings. The U.S. Environmental Protection Agency, for example, is required to evaluate all benefits and costs for any regulatory change that would impact the U.S. economy by $100 million or more. Outside the regulatory realm, both the National Oceanic and Atmospheric Administration and the Department of the Interior include lost environmental values resulting from natural resource injuries, such as lost recreational value, in natural resource damage assessments.

Environmental valuation methods

Common environmental valuation methods include primary valuation methods like travel cost models, property value analysis, and survey-based methods; and secondary valuation methods, otherwise known as benefits transfer (43 C.F.R. Subtitle A § 11.83(c)). Primary valuation methods involve the collection and analysis of primary data from the site of interest. In contrast, benefits transfer repurposes valuation estimates from pre-existing studies and applies them to the context in question. The accuracy of benefits transfer depends on both the quality of the original study and on the extent to which extrapolation from the original study context is appropriate.

Although benefits transfer is often regarded as a “second best” alternative to primary valuation, it is still frequently used in environmental litigation and policy advocacy due to its relative simplicity, ease of use, and low cost. Moreover, despite recommendations from the academic literature, single study value transfers, where value estimates are borrowed from one existing study, remain one of the most commonly utilized benefits transfer approaches. This article explores the potential for substantial inaccuracies in single study and average value benefits transfer, where estimates from multiple studies are averaged together, and provides a demonstration of the potential validity gains from careful selection of original studies.

A simple test of the statistical validity of benefits transfer estimates

In statistics, all estimates are associated with some degree of uncertainty or error, due to the noisiness of real-world data. The most standard means of accounting for this noise is to report the “95% confidence interval,” or the estimated range that would include the true value 95% of the time if the data were collected many times over.

To demonstrate the potential validity, or lack thereof, of benefit transfer estimates, we start with a sample of studies that estimate the value of a day of outdoor recreation for one person, taken from Oregon State University’s Recreational Use Value Database. We drop each study one at a time and estimate the 95% confidence interval of the average from the remaining studies (the “transferred” estimates), to simulate an average value benefits transfer approach. For each dropped estimate (the “true value”), we note whether it is contained within the 95% confidence interval generated by the transferred estimates. If the average value transfer method is statistically valid, then the true value should fall within the 95% confidence interval 95% of the time.

Statistical validity depends on the comparability of studies selected

We first ran the experiment on the entire sample of 3,192 recreational use value estimates and then on more selective subsamples of these estimates: (i) the 957 estimates that relate only to freshwater fishing and (ii) the 957 freshwater fishing estimates limited to specific geographic regions of the United States. Figure 1 summarizes the results.

For the full sample, only 8% of true values were contained within the 95% confidence intervals of the transferred estimates—far less than the 95% that would be expected for statistical validity. When we limit the analysis to the freshwater fishing estimates, however, 16% of the true values are contained within the 95% confidence intervals of the transferred estimates. The statistical validity of the transferred estimates improves further when we limit the freshwater fishing estimates by geographic region. For example, 67% of the 54 true values from the West South Central Region of the United States are contained within the 95% confidence intervals of the transferred estimates.

Table: City of Atlanta kWh electricity consumption by sector

Recommendations to reduce the scope for error in benefits transfer

Defensible value estimates are critical to environmental litigation wins and to successful arguments for policy change. Given the impressive scope for error demonstrated by our simple test, benefits transfer is especially vulnerable to criticism and skepticism. A high degree of care and judgement must therefore be exercised in the implementation of benefits transfer. Based on the recommendations of the academic literature, and as demonstrated by the validity test above, we make two primary recommendations.

First, average value transfers are preferred to single study value transfers. As an example of the risk of inaccuracy associated with the single study approach, note that, within our sample of study estimates, one study estimated the value of a day of freshwater fishing in Texas at $1, while another study on freshwater fishing in Texas published within five years estimated that value at $288. Unless a compelling case can be made as to why a particular study is most directly comparable to the site of interest, reliance on a single prior estimate lends itself to cherry-picking results.

Second, transferred estimates are more likely to be statistically valid when they are based on a set of studies that are carefully chosen to be comparable to the context of interest along multiple dimensions. For our full sample of 3,192 estimates, only 8% of true values were included in the 95% confidence intervals of the transferred estimates. The percent of true values included in the 95% confidence intervals increases after applying a single criteria filter and increases further after two levels of criteria filters are applied—by over 50 percentage points. As demonstrated, these recommendations will result in more valid applications of benefits transfer.

Mark Berkman, Martha Rogers, and Gina Waterfield

Mark Berkman is a principal of The Brattle Group, an economic consulting firm, where he specializes in environmental matters. Martha Rogers is a natural capital economist at the Nature Conservancy. Gina Waterfield is an associate at The Brattle Group