Value of Information Methods to Prioritize Research

Value of Information (VOI) is an outgrowth of advances in Bayesian decision theory and welfare economics that seeks to quantify prospectively the benefits and costs of research and development (R&D) activities under uncertainty. VOI allows for the identification of sources of treatment uncertainty and provides a method to calculate the incremental value of pursuing research to inform clinical practice. This article provides details of the principles behind its estimation, and the different decisions it can inform. Some practical applications of VOI in research prioritization are considered.

Before exploring current applications of the method in detail, it is important for readers to appreciate why the development of economic methods to prospectively assess the value of medical R&D is important. Economists have long noted that economic growth in advanced economies is driven by the creation of innovative ways of producing goods and services, a process that triggers productive investments and allows its benefits to spill over from one country to others. Typically, these economies rely on the profit motives of private enterprises and capital markets to fund innovative efforts. Yet in some economic sectors, such as biomedicine, there is substantial public investment in R&D. According to economic theory, public R&D efforts should act only to complement private investments, where the expected social value is large, but expected profit is small. Economic work since Smith has suggested relief from premature morbidity and mortality offers significant individual and social value. Recent empirical work suggests that the spillovers from public investment in medical R&D to the private sectors’ production of novel medical therapies and diagnostics are significant. Schumpeter (1942), Nelson (1959), and Arrow (1962), among others, are the first to articulate the view that R&D and innovative activities are difficult to finance in a freely competitive marketplace. The general argument goes as follows: the primary output of resources devoted to invention is the knowledge of how to make new goods and services, and this knowledge is nonrival so that use by one firm does not preclude its use by another. Moreover, if that knowledge cannot be kept secret, the returns to investment in knowledge cannot be captured by a firm undertaking the research investment, so that firms will be reluctant to invest, leading to under provision of R&D in the economy.

As a consequence, public policies that promote and underwrite innovation are centerpieces of national economic strategy. In the US, National Institute of Health (NIH) funding leads both the public and private sector as the single largest source of support for medical R&D. Within the NIH’s purview are direct investments in intramural research, including basic science, preclinical and clinical medical studies, and extramural grants supporting university research efforts. Other advanced economies support medical R&D efforts along similar mechanisms.

And yet, over the past several decades, the unprecedented increase in healthcare expenditures throughout the world, and especially in the US, has prompted increasing concerns that current levels of healthcare expenditures are excessive. It has been suggested that medical care has been provided not only beyond where its marginal benefit exceeds its costs, but often into ranges where there is little or no benefit. The importance of determining when specific medical technologies are worthwhile has intensified with the growing recognition that increases in medical spending have been largely driven by the development and diffusion of new medical technologies. As a consequence, controlling healthcare costs will ultimately require controlling the development and diffusion of medical technology. In accomplishing this goal, reimbursement systems that provide both developers and users of new technology with the appropriate incentives to control costs and produce quality healthcare are essential. Similarly, it is critical to have tools and policies to prioritize investment in public medical R&D efforts.

Retrospective analysis of previous investments may provide useful information to inform assessments of future research endeavors. Recent empirical economic research suggests that improvements in health have been a major component in the overall gain in economic welfare during the plast century for the US, developed, and developing countries. Murphy and Topel (2006) have used cost-benefit analysis to estimate the overall value of medical research and the value of R&D for specific medical conditions. They have found that gains in US longevity due to advances in medical research since 1970 have had an aggregate value of $3.2T, a figure roughly equal to half of gross domestic product. Furthermore, programs aimed to expand public support for specific types of medical innovation, such as the 1970 declared ‘War on Cancer’ appear to have produced substantial gains in morbidity and mortality. For example, pediatric cancer patients in the US and abroad, and adult patients suffering from breast and prostate cancer, some forms of leukemias, have experienced substantial gains in life expectancy over the past 15–20 years and many believe that such efforts have a great potential to produce more success. Philipson and Jena (2005) have used cost-benefit analysis to estimate the net value of antiretroviral therapies for the treatment of human immunodeficiency virus infection/acquired immunodeficiency syndrome. They have found that patients gained substantial benefits (measured in survival) from the introduction of antiretroviral therapy. In addition, US patients diagnosed with cancer between 1983 and 1999 experienced greater survival gains than their European counterparts; even after considering higher US costs. These findings do not appear to have been driven solely by the earlier application of diagnostic methods.

Whether research funds underwriting these efforts are being allocated to the ‘correct’ opportunities, and whether the fruits of these investments are valuable at the margin, applied to specific patients, are important and complementary questions. For example, novel approaches to cancer have been a significant focus of public and private sector investment in the past 30 years. These investments are bearing much fruit – in 2012 alone, the Food and Drug Administration has approved 19 anticancer drugs and over 900 anticancer drugs are in various phases of preapproval testing, more than the number for heart disease, stroke, and mental illness combined. Yet, anticancer drugs also rank first in terms of total drug spending by therapeutic area: $23 billion in 2011, up from $18 billion in 2007. Global spending on anticancer therapies is projected to amount to $75–80 billion by 2015, more than any other therapeutic class of pharmaceutical products. Conti et al. (2013) have found that commonly used, novel chemotherapies are more often used onlabel than offlabel in contemporary practice. Total national spending on these chemotherapies has amounted to $12 billion (B; $7.3B onlabel, $2B offlabel and supported by additional clinical evidence and expert judgment, and $2.5B offlabel and unsupported by clinical evidence and expert judgment).

Both Congress and the National Institutes of Health have faced increasing pressures from disease specific interest groups in recent years to justify their decisions regarding medical resource allocation, and questions such as these have been sufficient concern to Congress that they have played a role in recent discussions regarding increased funding for research, and have led Congress to request the advice of the Institute of Medicine (IOM) on whether priorities for the allocation of funds at the NIH have been appropriate. Indeed, although the resulting IOM report did not conclude that medical expenditures to date have been allocated inappropriately, it did conclude that NIH should pay greater attention to the burden of illness in assessing research priorities. Building on this, others have suggested that the NIH could better identify the most promising projects if it capitalizes on the formal approaches to assess the burden of illness and opportunities for research to lessen this burden.

VOI is the most well-developed method based in economic theory and may have potential as a practical tool to assist decision-makers in the process of identifying the value of specific medical technologies and the most promising avenue for future research. In the remaining sections, it is argued that VOI may have the potential to provide important insights into the value of medical research if applied in the right settings with methodological rigor and a thoughtful understanding of its underlying assumptions, strengths, and limitations.

A Review Of Value Of Information Analyses Applied To Clinical And Policy Questions

VOI has been increasingly used by researchers to inform stakeholders whether additional research would be worthwhile, and to demonstrate the benefits and feasibility of using such analytic methods to inform policy decisions within the timelines demanded by existing procedures. Illustrations of the potential value of the methods include applications to issues of resource allocation in neurology, oncology, and ophthalmology, clinical areas with high burden of disease and/or cost of care. For example, in neurology, VOI decision analytic methods have been utilized to determine the feasibility of magnetic resonance imaging (MRI) as a cost-effective approach to treating multiple sclerosis. The analysis was twofold. It was first determined that the cost of immediate MRI exceeded the cost of the expected value of perfect information. Next, advanced MRI technology had to be shown preferable to the fallback strategy of waiting, given a reasonable estimate of accuracy in MRI. A similar study was completed in orthopedics, utilizing VOI analysis to assess technological advancements in MRI technology to estimate the decision uncertainty that remained after a randomized control trial was completed.

VOI has also been proposed to identify and prioritize medical R&D in a number of clinical areas, where public investment in the later stages of novel therapeutic development has been significant. This interest requires VOI analyses to be calculated for some investment decisions from the public’s perspective and with available data and a timely manner consistent with NIH’s decision-making process. The idea is to incorporate economic decision analytic tools into trial consideration alongside scientific and trial design criteria to help ensure that public resources are spent efficiently and equitably. For example, in oncology, VOI methods have been analyzed to determine phase III clinical trial research prioritization, feasibility, and areas for greater investment into personalized therapies. Basu and Meltzer’s (2007) analysis suggests that identifying cost-effective treatments at the individual level could be greater than 100 times the annual value of identifying the cost-effectiveness treatment on average for the population. In ophthalmology, a VOI analysis was completed to inform governmental health spending and technological priorities. The results of the analysis suggests that the expected value of perfect information (EVPI) could be implemented in a timely fashion to inform the type of research prioritization decisions faced by any healthcare system.

The current reporting standards in the VOI literature is for mean estimates of all stochastic and deterministic model parameters to be described. The uncertainty of the intervention should also be assessed based on the distribution of the incremental costs and incremental quality-adjusted life-years (QALYs) in the cost-effectiveness plane. Additionally, the assumptions underlying the approach should be enumerated. Similar to traditional cost-effectiveness analysis (CEA), typically, main analyses are undertaken with standard assumptions: the discount rate for benefits and costs accrued in the future is 3.5%, and the research findings have a 10 year life span. Sensitivity analysis should be performed to test these assumptions over a range of possible values drawn from the literature. For example, CEAs performed for private insurers tend to use an alternative discount rate that allows for both the timing of costs and revenues and the risk associated with the trial. One commonly used metric is to estimate the adjusted discounted rate based on the capital asset pricing model. The inclusion and exclusion of benefit and costs outcomes and the sources of this information in all analyses should be reported.

A critical practical challenge in the application of VOI methods is that the method has most often been performed by constructing decision analytic models, which is very time consuming and, therefore, costly. More recently, VOI methods have been developed that use data from existing but often underpowered clinical trials to develop estimates of the value of more definitive trials, or an understanding of the conceptual basis of VOI to bound VOI estimates with even more limited information. Application of practical methods for VOI such as these will continue to be important in developing and validating VOI as a tool to provide timely guidance for decision-making with regard to medical R&D investments.

The promising areas of concern for future methodological advancement in VOI include the following:

1. Individualized care: VOI methods need to be expanded to understand how costs may be better internalized as to capitalize on the value of individualized care, utilizing an expected value of individualized care (EVIC) measure. EVIC is the expected cost of ignorance of patient-level preference heterogeneity and represents the potential value of research that helps to elicit individualized information on heterogeneous parameters, which can be used to make individualized decisions. The heterogeneity parameters of interest are random; hence, rather than larger samples, individualized elicitation will reveal the true values of these parameters. This measure is rather different than the EVPI, in which the parameters of interest have a fixed value in the population. Individualized care offers enormous cost savings, as the value of such may far exceed the value of improved decision-making at the group level; however, such benefits will vary immensely with insurance. EVIC can provide a guide as to when the high value of individualized care may make population-level decision-making especially at risk of providing poor guidance for coverage decisions.
2. Product lifecycle concerns: Despite advancements, uncertainty remains sufficiently high in some potential clinical application areas, hampering VOI calculations, and yet decision-making and research prioritization are required. Analytic methods must evolve further to address significant uncertainty in the potential costs and benefits of novel therapies over the lifecycle of the product. It is believed that questions regarding how risk and uncertainty should be assessed in policy decisions deserves more analytic consideration, because preferences concerning these dimensions are critical to decision-making. Meltzer et al. suggest that it may also be useful to distinguish between uncertainty in insured and uninsured costs in assessing the implications of uncertainty in costs in cost-effectiveness analyses and further characterizations of optimal decision-making when insurance is not complete. Further questions of uncertainty include assessments of the changing value of research due in part to technological or demographic changes.
3. Technological change: Standard CEA and VOI calculations assume a general and uniform rate of technological diffusion across technologies and diseases in clinical practice. However, recent work by Conti, Bernstein and Meltzer (2012) suggests that diffusion patterns of novel molecular-based therapeutics may not follow standard diffusion paths implicit in the standard assumptions of CEA, and may differ substantially from that of new therapies in other clinical areas. Progress on understanding the rate of technological advance across different clinical settings, as well as the product-level, provider-level, and patient-level determinants of this rate, are important inputs for next generation CEA analysis and VOI calculations. A priori, replacing standard assumptions with an empirical based model of technological diffusion alters the numerator and denominator of such estimates. How sensitive CEA and VOI calculations are to actual rates of practice that change across a variety of acute, emergent, and chronic disease settings are important subjects for future work.
4. Public versus private investments in research: Finally, future applications of VOI should explore the validity and applicability of the method to help guide decision-making in clinical areas where the public is the main funder of R&D, and also the major source of funding treatment purchases. In the US and other countries, funding for medical R&D, insurance coverage, access to new diagnostic methods and treatment modalities are not shared under the public government budget; rather, the presence of private insurance and private funding for medical R&D challenges the adoption of the social perspective in the widespread use of VOI to guide investment decision-making. VOI can be explored as a tool to guide decision-making in the US, where public monies are a main source of medical R&D, and the main source of insurance coverage and access, once new treatments are developed. The developing world are typically funded by government sources, sometimes in collaboration with experts in public health at the World Health Organization and the Gates Foundation. High profile and sustained gifts from the Gates Foundation in recent years have played an important role in vaccine development successes and in seeding the pipeline for more development in the near future. Recent economic work identifying financing barriers for underwriting R&D in this area have produced novel insights and new approaches to public policy incentives to either ‘pull’ R&D efforts from the private sector through the credible reward of research activities or to ‘push’ R&D through direct and indirect underwriting of the perceived costs of R&D and the delivery of vaccines to relevant populations. In this context, the use of VOI methods could be seen as an alternative push mechanism, one that public agencies and public-private partnerships use as a tool to invest funds wisely in the development of new vaccines.

Key Empirical Challenges

A number of empirical challenges are encountered in the practical implementation of VOI, for which practitioners should be aware of, when implementing these methods. First, the most fundamental ambiguity is how to best measure the benefits of a medical intervention. Although disease specific measures such as the number of cancer cases detected or cured may be useful in certain circumstances their effects on mortality (as measured by life years saved) have the advantage of comparability across diseases, they do not capture the important effects of medical care on quality of life. In some empirical applications, analysts assume for analytical simplicity that quality of life is not a concern so that outcomes may be measured in life-years. This assumption likely provides a lower bound on the total benefits of treatment for some diseases. Yet, there are many clinical examples where including quality of life measurements into a fuller assessment of mortality and morbidity gains could decrease overall benefits of an alternative therapy; for example, if the side effect profile of a treatment that provides mortality gains is quite severe. Recognition of this has led to the development of the concept of QALYs. Using this approach, each year of life is weighted by a factor between 0 and 1, intended to reflect the quality of life in that year, where 0 is equivalent to death and 1 to perfect health. These quality of life weights are most commonly derived by psychometric techniques based on responses to hypothetical choices. Two common approaches to assessment can be fairly readily connected to neoclassical economics; these describe either choices between life with a given illness and a gamble involving life in perfect health and death, with some probability or choices between longer life with illness and a shorter life in full health.

There are also clinical situations where the benefits of alternative therapies that potentially affect morbidity and/or quality of life are not available. For example, quality of life may not be as outcomes in a phase II or phase III trial of novel therapeutic modalities for the treatment of some cancers. In these cases, an extensive literature review of other trials may produce some supportive data. Judgment is required regarding the likely effect of excluding these outcomes on the magnitude and direction of bias introduced into the VOI calculation. Additionally, even when more complete information regarding the impact of treatment on morbidity or quality of life outcomes are available, index QALY weights for these effects may not be available in the published literature for all illnesses and treatment modalities. This hampers the analyst’s ability to capture the full range of potential effects of alternative treatments, and also limits the ability of the analyst to compare the full potential benefits of research into one area for research prioritization across alternative uses of supporting funds. When such values are available, it is important to perform a sensitivity analysis over a range of plausible values.

Challenges may also be encountered in the analysis due to the availability of data on the full health costs of alternative treatments. In many settings, treatment costs for standard of care and alternative therapies may not be available from clinical trial data collection efforts, prior studies or estimated using observational data. In addition, innovative therapies that alter the bundle of treatments provided to patients, including the use of diagnostic tests, the length and use of inpatient admissions and physician input, may substantially alter the costs of standard treatment protocols for some illnesses. Validation exercises may need to be performed using actual per person resource use collected on observation data. When the availability of short-term and long-term costs of treatment are lacking, the analyst may choose to ignore costs or perform sensitivity analyses over a range of plausible values. It is important to keep in mind that the dropping of costs from the VOI calculation altogether may be required for analytic convenience, but it limits the comparability of the analysis for research prioritization efforts.

Conclusions

This article provides a review of the rationales behind and the recent practical applications of economic methods to assess priorities in medical research and development. VOI is the most well developed set of tools based in economic theory and advanced cost-effectiveness analysis that could be used by analysts to construct measures of the potential gains from investing in further research. Although these methods have been recently applied to a number of challenging scenarios, the work required to move from what is theoretically possible to the practical application of these principles, to produce valid and reliable estimates of the value of research, involves a series of methodological and empirical challenges. Methodological challenges include the measurement of benefits and costs. Additional issues specific to VOI include developing meaningful priors concerning the parameters of decision models. This may often require extensive review of existing data, primary data collection or even, sometimes, analyses based on a variety of arbitrary priors. It may be difficult to determine priors for the likelihood that the research project will find a meaningful result. Whether it is possible to adequately address these challenges will be resolved through efforts to address these ideas empirically in a number of promising areas.

To apply these approaches to prospectively inform medical research and development decision-making, there are a number of additional and important analytic considerations. These include whether and when typical assumptions of uniform medical technology diffusion rates and discount rates for benefits and costs accrued in the future are justified. Future work in this area needs to empirically grapple with the possibility that the research may be less valuable over time, as other technological or demographic changes can arise that alter the management frequency or natural history of disease and the unpredictability of how the results of research might be useful in areas outside the initial areas of inquiry. These issues imply that the sort of formal analysis suggested here may be more likely used for evaluating clinical research rather than basic preclinical work. Such difficulties suggest that the practical development of VOI for identifying and prioritizing future research is important as one additional tool in the current and evolving armentarium of public research and development decision-making alongside scientific and biostatistical criteria.

Despite these concerns, the importance of making more informed decisions regarding the allocation of resources to medical interventions and medical R&D suggests that work in this area should be an important priority in health economics. It is important to keep in mind – even with evidence that some treatments may have little value at the margin, and with limited evidence of the connection between research and gains in health – health is a domain that people value very highly and at which great strides have been made in recent decades. There is ample reason to believe that such gains may continue in the future. Progress on methods, such as VOI, and the applications for work on the value of medical research and development as a complement to existing methodologies for prospectively evaluating the potential benefits of future investments, have a critical role in ensuring the sustainability of medical spending and gains in mortality and morbidity that has been conferred by medical science over time.

References:

1. Basu, A. and Meltzer, D. (2007). Value of information on preference heterogeneity & individualized care. Medical Decision Making 27(2), 27–112.
2. Conti, R. M., Bernstein, A. and Meltzer, D. O. (2012). How do initial signals of quality influence the diffusion of new medical products? The case of new cancer treatments. In Grossman, M., Lindgren, B., Kaestner, R. and Bolin, K. (eds.) Advances in Health Economics and Health Services Research, vol. 23, pp. 123–148. UK: Emerald Group Publishing Limited.
3. Conti, R. M., Bernstein, A. C., Villaflor, V. M., et al. (2013). Prevalence of off-label use and spending in 2010 among patent-protected chemotherapies in a population-based cohort of medical oncologists. Journal of Clinical Oncology 31(9), 9–1134.
4. Murphy, K. M. and Topel, R. H. (2006). The value of health and longevity. Journal of Political Economy 14(No. 5), 871–904.
5. Philipson, T. and Jena, A. B. (2005). Who benefits from medical technologies? Estimates of consumer and producer surpluses for HIV/AIDS Drugs. Article 3, Forum for Health Economics and Policy, vol. 0(1), pp. 1005–1005. Berkeley, CA: BE Press.
6. Arrow, K. (1963). Uncertainty and the welfare economics of medical care. American Economic Review 53(5), 941–973.
7. Bojke, C. K., Sculpher, M. and Palmer, S. (2008). Identifying research priorities: The value of information associated with repeat screening for age-related macular degeneration. Medical Decision Making 28(1), 33–43.
8. Claxton, K. P. and Posnett, J. (1996). An economic approach to clinical trial design and research priority setting. Health Economics 5, 513–534.
9. Conti, R., Veenstra, D. L., Armstrong, K., Lesko, L. J. and Grosse, S. D. (2010). Personalized medicine and genomics: Challenges and opportunities in assessing effectiveness, cost-effectiveness, and future research priorities. Medical Decision Making 30(3), 40–328.
10. Cutler, D. M., McClellan, M. and Newhouse, J. P. (1998). What has increased medical-care spending bought? The American Economic Review 88(2), 132–136.
11. Groot, K. B., Nikken, J., Oei, E., et al. (2008). Value of information analysis used to determine the necessity of additional research: MR imaging in acute knee trauma as an example. Radiology 246(2), 420–425.
12. IOM (Institute of Medicine) (1998). Scientific opportunities and public needs: Improving priority setting and public input at the National Institutes of Health. Washington DC: National Academy Press.
13. Meltzer, D., Hoomans, T., Chung, J. W. and Basu, A. (2011). Minimal modeling approaches to value of information analysis for health research. Medical Decision Making 31(6), E1–E22.
14. Meltzer, D. O. (2001). Addressing uncertainty in medical cost-effectiveness analysis implications of expected utility maximization for methods to perform sensitivity analysis and the use of cost-effectiveness analysis to set priorities for medical research. Journal of Health Economics 20(1), 109–129.
15. National Institutes of Health (2005) Report of the Clinical Trials Working Group of the National Cancer Advisory Board: Restructuring the National Cancer Clinical Trials Enterprise. Washington, DC: National Cancer Institute. Available at: https://deainfo.nci.nih.gov/advisory/ncab/workgroup/archive/CTWG/FinalReport2005.pdf.
16. Philipson, T., Eber, M., Lakdawalla, D. N., et al. (2012). An analysis of whether higher health care spending in the United States versus Europe is ’worth it’ in the case of cancer. Health Affairs 31(4), 75–667.