Through droughts, a global pandemic, and a military coup, Queen’s PhD Student Ardyn Nordstrom has been studying the effects of education interventions on girls’ education in rural Zimbabwe since 2017. In Summer 2022, she finished her PhD studies and begin her new position as an Assistant Professor in Program and Policy Evaluation at Carleton University’s School of Public Policy and Administration.
Article by Brock Mutic Queen’s Economics Department with Chris Cotton
Ardyn Nordstrom, a recent PhD graduate from the Queen’s Economic Department (QED) and a John Deutsch Institute (JDI) fellow, had no idea her research on the effects of educational interventions in rural Zimbabwe would take her to the places it did. In fact, Nordstrom had not expected to be studying educational interventions in rural Zimbabwe at all.
Nordstrom’s undergraduate path, like many others, was formative and ended in a place it did not begin. Nordstrom studied commerce as an undergraduate at Carleton University, a world away from rural Zimbabwe. She became interested in development economics after taking several courses in the subject. The spark had caught fire. Going on several service trips to Guatemala and Mexico and working with people on the ground influenced her future path even more. Nordstrom’s travels inspired her to pursue a career in development and development economics.
Nordstrom went on to earn a master’s degree in economics and data science at Carleton before working at Employment and Social Development Canada. While working as a data scientist there, she became “excited about using data to study social programs” and “fell in love with data for social research.” Nordstrom then came to Queen’s for a PhD, where she spent several years researching how economists could use “innovative tools and methods to understand social programs,” as well as to inform program evaluation and design in the real world. So, when the opportunity arose to join a team led by Queen’s professors Christopher Cotton and Bahman Kashi through Limestone Analytics to evaluate a major girls’ education intervention in Zimbabwe, she jumped at it.
When Nordstrom joined the research team in 2017, it had just been asked by the UK Department for International Development (UKAid/DFID) and its implementing partner World Vision to serve as the official external evaluator for a major “Girls’ Education Challenge” (GEC) project just kicking off in rural Zimbabwe. The GEC was a challenge fund to support innovative efforts to improve learning opportunities and outcomes of girls around the world. Nordstrom was involved with the Zimbabwe project from the beginning, from designing the evaluation methodology to conducting the qualitative and quantitative analysis. She traveled to Zimbabwe several times to help with training the data collections team and overseeing the data collection and served as the evaluation manager during the later years of the five-year project.
This research opportunity provided the foundation on which Nordstrom wrote a dissertation, which ultimately extended well beyond studying the GEC project she helped manage. She also studied earlier rounds of interventions targeting improvements in education outcomes and empowerment and looked found opportunities to apply new research methods involving satellite imaging, machine learning, and text analysis to develop better indicators of outcomes of interest.
One set of research projects undertaken by Nordstrom and Cotton has used data from early waves of the Girls Education Challenge, implemented between 2013 and 2016, to study the impact of information campaigns on girls’ education outcomes. In Zimbabwe, girls in primary school tend to outperform boys on learning assessments. But, once they reach secondary school, girls drop out at higher rates. By providing information to girls and their families about the returns to education, and by teaching caregivers about the barriers girls faced and strategies for overcoming them, the GEC “project hoped to improve support of girls’ education among parents and teachers and the agency of girls themselves, leading to increased enrolment and school performance” [1]. In a research paper released in 2020 [1], Nordstrom and colleagues found that the “information campaign resulted in a significant improvement in mathematics performance and school enrolment within a relatively short time frame, and that improvements in mathematics persisted through the life of the project” [1], although “no similar improvement in literacy that can be attributed to the information campaign” [1].
While Nordstrom and colleagues were processing data collected during their surveys for the evaluation, they noticed local residents kept mentioning the deep effect a recent drought had had on them. The drought in question occurred between 2015 and 2016, and was one of the worst in the country’s history, affecting large swaths of Zimbabwe. However, the pattern of effect across the country was essentially random; it did not correlate with geographic area or regions previously affected by droughts. In the areas that were hit, the drought was a tragic economic shock, leading to food insecurity across many communities.
This random effect created a kind of natural experiment. Nordstrom and Cotton combined rainfall estimation techniques using remote sensing and satellite imagery with the extensive GEC education data in order to study the impact of droughts on the economic and educational outcomes of girls and their families [2]. The research finds that the drought had mixed impacts. The drought increased the probability that students advance in school, likely due to lower opportunity costs to education. However, the drought also led to a significant decline in performance on mathematics assessments and leadership attitudes, likely due to stress or other factors associated with a drought that more than offset increases in attendance. These mixed results highlighted to the researchers the importance of using multiple educational indicators to assess the impacts of large shocks like droughts to holistically understand its effects, as using only a single indicator may provide a distorted positive or negative view of the impacts of shock. This research provided another way in which Nordstrom was able to apply her skills in data analysis to understand the social and economic impact of real-world phenomenon, helping marginalized communities affected by them to be more effectively supported.
The second set of research projects undertaken by Nordstrom and colleagues has been based on the official evaluation of the GEC’s IGATE-T program that the team ran from 2017 to 2021. This time, the interventions focused on four areas: teacher professional development, community-based education, leadership clubs, and community champions networks. The data collection stretched across periods of economic and political turmoil, and the COVID-19 pandemic. The team from the QED and Limestone Analytics again had the opportunity to conduct an evaluation of the intervention’s effectiveness.
The first step of the process, undertaken by Nordstrom and others, was to design the evaluation to be used. To evaluate the effectiveness of the interventions, the team needed to collect baseline data on key indicators from the communities in question before the interventions took place. The team also needed some of its members to travel to Zimbabwe to oversee the start of data collection, an opportunity that Nordstrom accepted with alacrity. Getting the opportunity to not only conduct field work for her PhD, but to contribute to such a large global development project with the potential for helping thousands of marginalized children – all while deploying her unique skills in data science and innovative program evaluation research – was an “incredibly exciting opportunity”, she says.
In October 2017, Nordstrom headed for Zimbabwe for what was the first of several trips to oversee enumerator training and data collection. The team oversaw the collection of baseline data with the help of local data collection experts assembled with support of World Vision and partner organizations. Their initial trip to the country was to train these local data collectors on proper protocol for conducting the baseline data collection. Nordstrom arrived several weeks after other members of the team and assisted in providing support to the local enumerators, ensuring they were comfortable with the instruments designed, understood necessary protocols like random selection of subjects, and followed best practices for ethics and getting informed consent. Nordstrom initially spent one week in Harare, and two more in rural villages supporting baseline data collection. It was a remarkable experience; although Nordstrom admits there were a “lot of long days in the field”, “actually being in the schools to see what the learning environment was like and hear the stories directly from students and teachers” helped her to better understand the context she was trying to learn from. She noted that “everyone we worked with was exceptional. It was amazing to see how much the community wanted to rally and help these students”.
After two weeks, the baseline data collection was well underway, and Nordstrom found herself swapping beautiful sub-Saharan Africa back for the limestone building on the Southwest corner of University and Union. Back in Kingston, the team continued to support the baseline data collection remotely through WhatsApp groups, where Nordstrom and others would answer questions the local data collectors had. “The data collection team could run into situations our protocols did not explicitly account for, and we would answer questions about it”, she said. This state of affairs meant Nordstrom was living on Africa time for several weeks after she returned. “It was all good” she remarked about the schedule, “I’m a morning person anyways”.
IGATE-T’s interventions began not long after the baseline data collection ended. The team had to return to the field in January 2019 to complete a midline data collection. However, these plans were soon turned upside down. Shortly after the team left Zimbabwe in 2017, a military coup occurred in the country, resulting in political instability. Then, in 2019, Zimbabwe was suffering from a currency crisis, with the Zimbabwean dollar inflating 97.85% by May and 175% by June of that year [3]. This crisis led to political protests and instability in cities across the country, including Harare and Bulawayo.
Almost immediately after Nordstrom arrived in the country to oversee training and midline data collection, protests erupted near the training facility. The team was advised to return to their hotel immediately, where Nordstrom and her colleague stayed in lockdown for almost a week as protests and “stayaways” led to uncertainty across the country. At the beginning, the internet and phone lines were shut down, and they were unable to contact their families and colleagues back in Kingston to ensure them they were safe.
After several days in lockdown with their colleagues, the decision was made to return to Canada, without having the chance to collect any data since these protests and stayaways led to most schools being closed across the country. Upon their return, they stayed connected with local partners and waited for the situation to stabilize. In May 2019, four months later, they were able to return to the country for a period of two weeks to finish training data collectors for collection of midline data. Once again, Nordstrom had the chance to work on the ground in rural areas of the country, this time in the rural area of Chivi. The team’s dedication was compensated with statistical luck; they were able to reconnect with most of the girls they had contacted in the baseline analysis, with an attrition rate of only 20% during midline data collection—relatively low for a study of this nature.
Back in Kingston, the team spent six months working through the data for the midline analysis, and produced a report for UKaid on the impact of the interventions. During this time, Nordstrom also worked on a project to use machine-learning and text mining methods aimed at predicting characteristics that can inform project and evaluation design, such as tools that can perform text mining to inform program evaluation, and outcome harvesting tools, which allow researchers to explain and understand empirical results that may not fit into any theoretical model currently developed.
During this time, she also travelled to Kenya and Uganda to undertake an empirical analysis of the relationship between trade and food security with Dr. Huw Lloyd-Ellis for USAID Kenya/East Africa. This has since evolved into two academic papers. [4]
Then the COVID-19 pandemic hit. Like many things in early 2020, the ongoing educational interventions were turned upside. Some international development resources that were conducting educational interventions were shifted to providing emergency support to local communities during the pandemic, and all interventions shifted to meet the new reality of the situation, instituting remote-learning where possible and providing students with community-based education while schools were closed. Though delayed, the IGATE-T educational interventions were ultimately completed by December 2020, and the team began preparing for endline data collection. Endline data was used alongside base and midline data, to ultimately evaluate the interventions’ impact. Due to COVID-19, the endline data collection was postponed, and began in April 2021. It followed enumerator training that month, supported online by the team, which started after COVID-related lockdowns in Zimbabwe lifted in March. Data collection for the evaluation ended in late May 2021, and the team has started working through the final data after years of planning, research, and several trips to the country. The official endline evaluation for the project wrapped up in Fall 2021, after which Nordstrom has focused on converting her extensive experience into academic papers and a dissertation, while continuing to advise NGOs and educators in the field.
Nordstrom’s PhD dissertation grew out of her five years of fieldwork, data collection, and analysis of the GEC projects in Zimbabwe. It used new, state-of-the-art analysis techniques, from remote sensing and satellite imagery to machine learning to text mining and sentiment analysis, to study different barriers to girls’ education in rural communities, the impact of droughts and other educational disruptions, and the effectiveness of interventions.
Ardyn Nordstrom’s research in Zimbabwe has allowed her to work on the questions she is passionate about as a researcher, while contributing to a global development project aimed at helping tens of thousands of disadvantaged youths. Through a global pandemic and a military coup, the work evaluating the effectiveness of IGATE-T’s educational interventions in Zimbabwe conducted by the team from Queen’s and Limestone, including Nordstrom, will help inform economic development program design and evaluation going forward, and ultimately allow researchers to better understand and aid marginalized communities. Her ultimate goals, she says, is “to figure out how we can use innovative tools and methods to understand social programs. There is such a huge opportunity to start leveraging methods like text mining and machine learning to uncover unexpected patterns that can improve program design and program evaluation. This is an area I’m really passionate about, and want to keep helping organizations use these methods for social sector programs”.
References
[1] Community information campaigns improve girls’ education: Evidence from an RCT in Zimbabwe (2020) – Working Paper with Christopher S. Cotton, Jordan Nanowski, and Eric Richert. https://papers.ssrn.com/sol3/papers.cfm?abstract_id=3547786
[2] Impact of a severe drought on education: More schooling but less learning (2021) – Working Paper with Christopher S. Cotton (https://papers.ssrn.com/sol3/papers.cfm?abstract_id=3601834)
[3] https://www.aljazeera.com/economy/2019/07/15/could-new-figures-forecast-hyperinflation-for-zimbabwe/
[4] Trade, poverty and food security: A survey of recent research and its implications for East Africa (2021) – working paper with Huw Lloyd-Ellis https://www.econ.queensu.ca/research/working-papers/1460
Economics and Policy 