You are here
What can we learn from RCT with partial compliance? The case of the effect of doing an internship on the employment of the mentally ill
This paper shows the evaluation findings of a social intervention aimed at increasing gainful employment among severely mentally ill patients, by offering them to enroll in a structured job-search experience. The experiment involved four provinces in Lombardy, a total of 29 local mental health centers and 311 mentally ill patients, most with a diagnosis of schizophrenia. The programme as well as its random assignment evaluation were financed by the Cariplo Foundation, and it was named Lavoro&Psiche. The recruitment of patients started in late 2009 and continued throughout the whole 2010. Upon referral to the project by their mental health center, patients went through a face-to-face baseline interview, their informed consent was obtained and were then randomized into experimental and control groups. For a period of about two and half years, the control group had access to the job placement services normally available to all patients, while the experimental group members received help in their job search by well trained job coaches, with a limited workload of 12-13 patients per coach. Employment related outcomes were obtained from the archive containing information on all job contracts every private sector employer has a legal obligation to communicate to the local PES office. Using the COB data we were able to construct monthly individual work histories from 2008 through the end of 2013.
More than one interpretation of results of the demonstration is possible. If the impact is obtained as the intent-to-treat effect of receiving the whole set of services of the coaches, the results are indeed disappointing: during 2013 (the first post-demonstration year) 25% of control group patients had some paid job, versus 30% of the experimental, a 5-point difference not significant at conventional levels. This standard approach however ignores all the various form of partial compliance that plague this like many other experiments. We believe that hiding all non compliance under the ITT blanket might make us miss vital insights.
The frequent use of internships, for example, reveals that the coach were offering the internships as the main form of support, but these were also offered to the controls by program staff, most likely to “compensate” those randomized out. Internships were rarely used among these patients before 2008. At the closing of the demonstration, over 71% of the experimental had been in an internship, versus 43% of the controls. Yet, the coach influence is still evident in the 28 points differential. This can be rationalized as response heterogeneity, as this effect can come only from the compliers, that is, those patients who went through the internship when they were assigned to a coach, but they would not have done so if assigned to the control group. Considering such case it is plausible to consider the effect of the intervention only on the compliers, commonly known as ´Local Average Treatment Effect ‘(LATE). The LATE can be obtained as a Wald estimator, which in our case yields a much larger effect than the Intent-to-Treat – that is 18.5% = (0,30-0,25)/(0,71-0,43). As often the case, standard errors for IV estimates are so large that we fail to reject the null hypothesis of zero employment impact for the compliers. We also estimated, separately for those who did an internship and those who did not, a linear probability model. The results imply that there is no selection bias in the decision to doing an internship, and therefore the use of IV might not be warranted after all. The policy implication is that having done an internship doubles the probability of having a paying job a year later. We believe that the latter finding deserves some attention from policy-makers.
Alberto Martini is terminal associate professor of Statistics and Public Policy Evaluation at the Università del Piemonte Orientale. He is scientific director of ASVAPP, a non-profit center whose mission is to disseminate the use of rigorous evaluation methods. After obtaining a Law Degree in 1980 and a Ph.D. in Economics from the University of Wisconsin-Madison in 1988, he joined Mathematica Policy Research in Washington, to work on the evaluation of social policies by means of experimental and non-experimental methods. In 1993 he joined the Urban Institute to work on microsimulation models to estimate the impact of changes in welfare programs.
Between 1993 and 1999 he was a consultant of the World Bank for the design of income and expenditure surveys in transitions countries. He designed and oversaw the implementation of the Income and Expenditure Surveys of Belarus and Ukraine.
Since moving back to Italy in 1998 in pursuit of Mission Impossible, he has conducted evaluations in a variety of fields, including enterprise support, employment policies, and education. He co-authored with Marco Sisti “Valutare il successo delle politiche pubbliche” e in 2011 with Ugo Trivellato “Sono soldi ben spesi? Perché e come valutare l’efficacia delle politiche pubbliche”.
Ironically, in 2001-02 he was President of the Italian Evaluation Association. In his spare time, he likes to invent cute acronyms.
Vicolo dalla Piccola 12, Trento