1.
Spurious Correlations and how to fix it
Dr. Manish Gupte, PhD
1) I did psychiatry study to see effect of employment on mental health improvment. In that simple
correlation table shows when patients take up employment their disability reduces. But on closer
analysis, we found, only ppl who did not have bad initial disability got job. And severe patients did
not get job. So only ok patients got job and ok patients only got better. This role of employment is
not actually there. It is suprious correlation.
So we made subsets in data based on initial disability level. So we compare initial severe patients
with job and without job. Both groups were initial severe patients. then it become clear, job has no
role in disablity improment and in short term it causes stress and increases disability. same way
group of initially ok patients with and without job. This is in this type of job, maybe creatve job
helps with health - some research, art, etc not simple tailoring job. This is short term study, maybe
longer term studies show other results
2) Then I analyzed Boston area crime with gun ban. A politician was saying gun ban reduced crime,
but when he put ban he increased police level also. So we had to removed effect of police force on
crime level. For this again we found how change in police level in same county effects change in
crime level. And then found more police means less crime. Simple correlation show no correlation
as Govt puts more police where there is more crime. By using difference to calculate correlation we
got proper negative sign. Then removing effect of police on crime correctly, we found gun ban
actually increased crime.
3) And in Pune we were thinking of using heat map like visualization to place police and ambulance
in hot spots. But again police is already near crime hot spots so crime there is under control. And
ambulance is already near accident prone and old ppl localities. So again difference and then
removing the effect of Govt measures and then putting back police/ambulace for optimization.
With simple correlation it give wrong result and it can cause lapse