- Rashmi Venkatesan*
At a time when the world is facing its biggest existential crisis with the dangers of climate change becoming imminent, inequality progressively worsening and capitalism due for an overhaul, the 2019 Nobel Prize for Economic Sciences last month, was awarded to Abhijit Banerjee, Esther Duflo and Michael Kremmer for their “experimental approach to alleviating global poverty” that breaks down a complex problem into “smaller, more manageable questions”. The Nobel bestowed Randomised Controlled Trials (RCT), the experimental method that they evolved, with an imprimatur of legitimacy. RCTs, that claim “to reduce poverty by ensuring that policy is informed by scientific evidence” were recognised by the Nobel Committee for having "dramatically improved our ability to fight poverty in practice". While the idea of controlled trials has been widely used in clinical research, especially in the case of drug trials, the genius of the idea was to apply this clinic method, conducted in sanitised laboratory conditions to the messiness of the real world. And this is where much of the criticism of the method lies.
While the trio’s personal and professional feats are undoubtedly exemplary, their methodology has always attracted criticism from other notable economists like Ravallion (2018), Kabeer (2019), Reddy (2013), Dreze (2018) and Deaton & Cartwright (2018). They question the manner in which RCTs are designed, implemented and interpreted; their scalability and appropriateness as a policy intervention tool; and the ethics of conducting randomised trials on people. All methods have their limitations and the intention behind pointing some of these out with respect to RCTs is not to delegitimise them. Rather, it is to be alive to the politics of this methodology and to challenge their dominance as the ‘gold standard’ of evidence in development economics based on their purportedly scientific, unbiased approach.
Let us leave the econometrical analysis of RCTs aside, and ask a much simpler, albeit rhetorical question - if randomised control trials lead to ‘scientific’, ‘evidence-based’ policy intervention, why aren’t these commonly conducted on the rich too? Wouldn’t that make for better development policies overall which go beyond just poverty alleviation?
One way to answer this question is to see who carries out these RCTs and why. The stated motivation is to make policy intervention more ‘effective’. But effective for whom? That is a question that is guided by who is interested in the trial. Given that many of the RCTs are concerned with public policy interventions, governments and other developmental agencies funding and implementing these policies are their biggest stakeholders. What they are generally looking for, through these trials, is to see how the funds expended can be best utilised for maximum impact. Therefore, one finds that across RCTs, cost is one of the most important parameters of testing a policy. While better utilization of funds is undoubtedly desirable, circumscribing policy ‘effectiveness’ within economic ‘efficiency’ is a political choice that determines different results for different stakeholders.
Let us take the example of evaluating the Balsakhi programme, a remedial education intervention programme rolled out in several primary schools across Vadodara and Mumbai and supported by ICICI Bank, World Bank and an NGO called Pratham, through an RCT. It found that employing a Balsakhi, typically a young woman from the local community who is “paid a fraction of the cost of civil-service teachers”, improved the test scores of weak students. Therefore, it concluded that that program was effective as it “was very inexpensive, since the main cost of the program was the tutors’ relatively small salaries.”[1] Now, there are two ways of looking at this. One is to say that this policy is efficient because for a small cost, the benefits are much larger. Or, one could argue that it is unfair because although Balsakhis perform such a crucial part in improving learning outcomes and help supplement the shortcomings of school teaching, their remuneration is not commensurate to their efforts. The Balsakhi is analogous to hiring unpaid or badly paid interns, ad-hoc faculty in universities, contract workers, etc. – all of which are cost effective and productive systems, but have serious negative implications for labour.
However, many would argue, isn’t combining economic efficiency and effectiveness an ideal policy intervention? Yes, in principle. But the important question to ask is why these arguments of efficiency are overwhelmingly levelled only in cases of money spent on poverty alleviation. One reason is that spending targeted towards the poor is always highlighted as welfare, and seen as a drain on the public exchequer. However, states ‘spend’ far more funds on the rich through policies such as tax cuts, concessional loans, cheap land-acquisition, subsidies, cutting cost of regulations etc. For instance, although the recent tax cuts to boost the competitiveness of Indian companies amounts to a foregone revenue of approximately Rs. 1.45 lakh crores[2] and the cost to Delhi Government in allowing women riders to ride Delhi buses for free is only 350 crores[3], it is the latter that is attacked viciously for handing out ‘freebies’. Such examples are countless.
Not only is spending on the non-poor invisibilised, such policy interventions are not subjected to the same level of scrutiny as those on the poor. So, if trials are required to test whether free transportation will indeed help greater participation of women, why is the assumption that tax cuts will make Indian firms more competitive not subject to the same level of proof? Even if there might be no in-principle objection to testing this claim through an RCT, one is hard pressed to see such trials conducted. The fact that we do not see the need to validate certain policies through unbiased, informed ‘scientific evidence’, is because they are considered economic truths, whereas others are often discredited for being ideologically motivated or promoting ‘vote-bank politics’.
There is also a serious ethical issue involved in RCTs, which many of its critics point out. Imagine the outcry if a set of entrepreneurs, middle-income households or billionaires were subjected to RCTs, where one group received a ‘treatment’, the other group acted as a control, and the differences were observed by researchers. The fact that RCTs are indiscriminately used to ‘study’ behaviours of only the poor not only objectifies them, but also dehumanises them. And it does so in more insidious ways than in just conducting the trial itself.
Let us go back to the Balsakhi example. The premise of the evaluation was that education is more than enrolment in school and that it should reflect in the learning outcomes of children. The background was that according to a study, 44 percent of Indian children between 7 to 12 years could not read a basic paragraph, half of them could not do simple subtraction and less than 20% of children enrolled in Grade 3 in Vadodara could correctly answer Grade 1 math questions.[4] There is no gainsaying that improving learning outcomes is a critical priority. However, what kind of ‘learning’ is tested and privileged as ‘education’ holds deep pedagogical biases. Urban children educated in elite schools might not know important or ‘basic’ facts that children from rural agricultural communities readily do, such as knowledge of local geography, histories, social hierarchies, languages and dialects, uses of common plants and trees, etc. If tested for these basic knowledge or skills, many of the urban children, who are more likely to influence government policy later, might also have limited, socially relevant learning outcomes to show for their school education. However, such trials on urban elite schools will never be funded or conducted or considered necessary. To be abundantly clear, the argument made here is not that rural children do not need to study mathematics and science, or that these are not important. Every child has an inalienable right to and should access at least elementary school education. Instead, the argument is that education should not be construed narrowly so as to devalue other forms of knowledge.
The limitations of using RCTs as an unbiased tool to inform policy should become apparent here. They function within the framework that improving education, where education is understood only as school textbook education, alleviates poverty. And even within this premise, RCTs are only effective in answering narrow questions that can be tested within their experimental parameters. However, they neglect, not by accident but by deliberate design, deeper questions relating poverty and education. Like, how can primary education be made more pedagogically inclusive so that it becomes a tool for social and economic empowerment rather than being primarily geared for the labour market? Why and what makes different children perform differently? Or what role do caste, migration, familial factors, language etc., play in determining learning outcomes? In tackling poverty through this prism of narrow programmatic questions, the RCTs end up functioning within and reinforcing the prevailing conceptualisations and assumptions of poverty, progress and economic growth.
‘Randomistas’ (as RCT proponents are often called) work within the paradigm that there are two ways of doing economics. One, is a scientific, evidence-based way of policy intervention that asks the ‘small questions’ and is concerned with ‘what works’, and the other, an unscientific alternative that is based on politics and ideology which is always lost in the ‘big questions’. Such a binary is an illusion. None of the critics of RCTs are against evidence-based policy or fail to appreciate the usefulness of this methodology in answering some critical, albeit narrow questions. However, in choosing to look at poverty only in a programmatic way and choosing to leave the ‘big’ ones of redistribution, inequality and wealth accumulation unchallenged, the randomistas too are making a political choice. Whatever their claim to scientific truth maybe, their claim to ideological neutrality is untrue.
*Rashmi Venkatesan is an Assistant Professor at the National Law School of India University, Bengaluru. Views expressed are personal.
[1] Abhijit Banerjee and others, ‘Balsakshi Remedial Tutoring in Vadodra and Mumbai, India’ <https://www.povertyactionlab.org/evaluation/balsakhi-remedial-tutoring-vadodara-and-mumbai-india> (accessed 17 November 2019).
[2] Deepshikha Sikarwar, ‘Revenue concerns for the government make immediate tax cuts tough’ The Economic Times (New Delhi, 31 October 2019) <https://economictimes.indiatimes.com/news/economy/policy/revenue-concerns-for-the-government-make-immediate-tax-cuts-tough/articleshow/71829093.cms> (accessed 17 November 2019).
[3] ‘’Brother’ Arvind Kejriwal’s free bus ride gift to Delhi women on Bhai Dooj’ Hindustan Times (New Delhi, 29 October 2019) <https://www.hindustantimes.com/delhi-news/brother-arvind-kejriwal-s-free-bus-ride-gift-to-delhi-women-on-bhai-dooj/story-q4D9HCZOLIIAdnx9clKXUL.html> (accessed 17 November 2019).
[4] Banerjee (n 1).
References
Deaton, Angus & Cartwright, Nancy (2018) Understanding and misunderstanding randomized controlled trials, Social Science & Medicine 210 pp. 2-21.
Dreze, Jean (2018) Evidence, policy, and politics, Ideas for India available at https://www.ideasforindia.in/topics/miscellany/evidence-policy-and-politics.html (accessed on 17th November, 2019).
Naila Kabeer (2019) Randomized Control Trials and Qualitative Evaluations of a Multifaceted Programme for Women in Extreme Poverty: Empirical Findings and Methodological Reflections, Journal of Human Development and Capabilities 20(2), pp. 197-217.
Ravallion, Martin (2018) Should the Randomistas (Continue to) Rule? Centre for Global Development, available at https://www.cgdev.org/publication/should-randomistas-continue-rule (accessed on 17th November, 2019).
Reddy, Sanjay G. (2013) "Randomise This! On Poor Economics," Review of Agrarian Studies 2(2), available athttp://www.ras.org.in/randomise_this_on_poor_economics (accessed on 17th November, 2019).