Malcomb Reproduction

Reproduction of Malcomb et al. 2014.

I sought to determine whether the original Malcomb study could be reproduced using an open-source approach with open source software and easily-accessible data sources.

You can access my reproduction report HERE

You can access the repository for my reproduction HERE

I made several relevant contributions to the original study. First, I took the study’s existing raster vulnerability map and replicated it but in vector form, showing the mean vulnerability for each Traditional Authority (similar to a county) in Malawi. Additionally, I joined the names of Traditional Authorities to the polygons layer, allowing policymakers to associate which TAs are most at risk. Since the map is interactive, policymakers and stakeholders can simply click on TAs with high vulnerabilities, see the name of that TA, and the exact value of its mean vulnerability score. The original raster map did show spatial patterns of vulnerability, but it was not very useful for policymakers and stakeholders looking to identify exactly which TAs are most vulnerable and require targeted aid. Additionally, I loaded maps from the original study into the report so that readers can visually compare the differences in maps between the original study and the maps in the reproduction report. Lastly, because attempts at reproducing some of the original figures did not yield the exact same results as the original study, I created difference maps and histograms of those difference values. The difference values tended to be normally distributed (phew!), which implies that the sources of variation likely had to do with computational environments or something minor, and that we weren’t missing anything major that created lopsided results.

Specifically, this reproduction aims to replicate the original multi-criteria analysis of vulnerability to climate change in Malawi, using figures 4 and 5 as primary outputs. The reproduction of Figure 4, representing the map of adaptive capacity by traditional authority in 2010, revealed consistent overestimation issues, particularly in the northern coastal areas. The Spearman’s rank correlation indicated a moderately strong positive but imperfect correlation between the original and reproduced values. Significant discrepancies in adaptive capacity scores, especially in the southernmost part of the country, raised concerns about the reproducibility and internal validity of the study.

The reproduction of Figure 5, representing the map of vulnerability in Malawi, highlighted a consistent undercounting issue, with vulnerability values about 20% lower than the original. The Spearman’s rank correlation for vulnerability was notably weaker than that for adaptive capacity, indicating more substantial discrepancies. The discussion emphasized potential sources of error, including georeferencing problems, unaccounted-for lake/park chunks, normalization errors, differences in computational environments, or human error.

Planned deviations from the original study, represented in the difference maps between reproduced and original figures 4 and 5, demonstrated moderate discrepancies. The southern part of Malawi exhibited significant differences in both adaptive capacity and vulnerability, with potential sources of uncertainty ranging from incorrect sign attribution to subjective data exclusions and georeferencing problems.

Unplanned deviations from the protocol were necessary due to the lack of access to the original DHS data, leading to the use of pre-aggregated, publicly accessible adaptive capacity data. Subjective weighting in the original study and ambiguity in variable transformations further contributed to uncertainties in the reproduction. The reproduction study, despite challenges, provided valuable insights into the complexities of replicating climate vulnerability research, underscoring the importance of transparency, standardized practices, and enhanced data accessibility for more credible vulnerability assessments. It goes to show how important it is that any and all subjective/ambiguous decisions are clearly communicated, that data and code is made as accessible as possible, and that reproduction studies are just as important as original studies for validating their accuracy from another perspective and computational environment

References: Malcomb, D. W., E. A. Weaver, and A. R. Krakowka. 2014. Vulnerability modeling for sub-Saharan Africa: An operationalized approach in Malawi. Applied Geography 48:17–30. DOI:10.1016/j.apgeog.2014.01.004{.uri}