Chakraborty Reproduction
View the GitHub repository here
In our attempt replicate the original Chakraborty (2021) Covid-19 Disability study, we’ve gleaned several valuable lessons about the scientific research process. The replication study yielded results that closely paralleled the original research but also introduced deviations that warrant attention and analysis.
The deviations that emerged during our replication were not necessarily errors but rather variations that arose due to differences in computational environments. Our utilization of Spatial Epi contrasted with the original study’s SaTScan approach, contributing to some of these disparities. These deviations, while not inherently detrimental, underscore the importance of meticulous research design scrutiny, internal validity assessment, and robustness testing of key parameters. They serve as a reminder that even slight variations can impact research outcomes. My biggest takeaway from this exercise was that two software packages that at first glance both appear to be implementing a spatial scan statistic based on the Poisson Model (SatScan and Spatial Epi) can actually operate differently (ex. how they calculate distance, grid vs. circle approach, etc) such that they yield different results, the difference of which is statistically significant.
We reiterate Chakraborty’s carefully limited interpretation emphasizing county-level relationships and the need for additional data and research, because the estimated relationship at the county level may not hold at the individual level. Interpretation of the results were based on aggregate statistics at the county level and on statistical methods which produce approximate estimates, not inferences. Further work is needed to reliably infer relationships between minority PwDs and COVID-19 morbidity and mortality, especially at finer geographic scales. Further replication and reanalysis work would be beneficial on this topic to further test the robustness of Chakraborty’s findings. Future versions could more thoroughly vary the structural criteria of how counties are clustered (ie. using variables beyond just state and relative risk statistics) to better account for inter-county and intra-state correlations in COVID-19 cases. Ideally, a package gets added to R in the near future future that provides GINI-based secondary cluster detection to classify counties by their maximum cluster-based relative risk score, which would allow us to more directly test results and compare to the SatScan software used in the original report.
Our replication study has illuminated the significance of replication itself. It not only reaffirms the original findings but also demonstrates the value of conducting replication studies to ensure the reliability and validity of research conclusions. Moreover, it serves as a blueprint for improving research design, enhancing reproducibility, and adapting findings for educational purposes. Even just picking through the R code of the report was challenging. There was ambiguity in commenting, and some instances of no commenting at all. Another big lesson that I learned was the importance of thorough commenting such that individuals with less technical experience can easily run your code and understand your methodological steps.
In terms of improving the study’s research design, our replication study has highlighted the potential to refine various aspects, such as the selection of secondary clusters in spatial analysis. We’ve also uncovered opportunities to improve reproducibility by addressing the differences arising from computational environments. Additionally, our findings can contribute to enhancing the study’s reproducibility for teaching purposes, allowing for more effective dissemination of knowledge to a broader audience.
Lastly, our replication study opens the door to further research, replication, and reanalysis. The deviations we observed and the variations in results between the original study and our replication provide fertile ground for meaningful follow-up studies. These studies can delve deeper into the nuanced aspects of the research, providing a comprehensive understanding and further testing the theories established by the original study.
In conclusion, our replication study has reinforced the critical role of replication in scientific inquiry. It has also highlighted opportunities for refining research design, improving reproducibility, and expanding the educational applications of research findings. Moreover, it has paved the way for future investigations that can build upon the foundations laid by the original study, offering the potential for deeper insights and a more comprehensive understanding of the subject matter.
Original Chakraborty (2021) Study: Chakraborty, J. 2021. Social inequities in the distribution of COVID-19: An intra-categorical analysis of people with disabilities in the U.S. Disability and Health Journal 14 (1):101007. DOI: <10.1016/j.dhjo.2020.101007]>.