This assessment method is reminiscent of the traditional approach to assessing new surgical techniques that, after more rigorous testing, have sometimes proven worthless or worse. For example, radical mastectomies were used extensively for decades before randomized trials showed that a much less extensive and disfiguring surgery followed by radiation was an equally effective treatment.
Of course, conducting randomized trials can pose greater challenges to evaluating a new surgery than evaluating a new drug. It can be more difficult to standardize a surgical technique enough to test it on a broad population, and it can be far more difficult to blind the patient for whom he is receiving treatment.
However, such feasibility problems do not apply to new payment methods, which are well-defined and standardized interventions and where it is not desirable to blind medical providers for the payment rules.
However, as in medicine, not all public interventions can or should have randomized ratings. Unique government projects – like the Big Dig in Boston or the Superconducting Super Collider in Texas – have no natural comparison group, either randomly or otherwise. In times of crisis or when political disagreements are more about ideology than impact, the assessment itself can be ill-advised.
However, if – as is so often the case – there is the possibility of a prospective assessment and the law prescribes it, the experience of the innovation center underscores the value and feasibility of randomized socio-political studies. Often times, they can be conducted at the same speed and cost as any prospective study and produce more compelling results. A random assignment, where the government uses a lottery to choose who can receive the program, may also be the fairest way to assign intervention on a limited basis.
Randomized testing may not yet be the standard for government assessment, but such things take time. For example, the Food and Drug Administration was given authority in 1962 to obtain “substantial evidence” of the safety and effectiveness of a new drug. However, it took more than five years for the agency to accept randomized trials as an appropriate standard.
Now that the Biden government correctly reiterates that all federal agencies must make “evidence-based decisions” based on the highest scientific standards, really tough social policy testing may become as natural as it is with new vaccines. This would help ensure that government services are provided as effectively and efficiently as possible.
Amy Finkelstein is John and Jennie S. MacDonald, professor of economics at MIT. She is co-director of J-PAL North America, a research center at MIT that conducts randomized assessments.