The COVID-19 pandemic posed unforeseen challenges in conducting evaluations and providing evaluative services. IES however quickly adapted and invested resources into developing innovative methods and related guidance, moving to digital responses, ensuring finalization of evaluations and meeting of accountability needs of stakeholders and Member States. See below IES guidance as regards evaluation during a pandemic or other crisis.
The COVID-19 Guidance Note for Managers and Evaluators contains a decision matrix that shows managers on how to adjust to a pandemic or other crisis in ongoing and planned evaluations. Moreover, there is guidance for evaluators on how to tailor evaluation methodologies, ensuring that no one is left behind. See some essential details below from the Guidance Note.
CONSIDERATIONS WHEN PLANNING AN EVALUATION
a) ensure a do no harm approach; b) adhere to UNEG Ethical Guidelines for Evaluation; c) avoid overburdening and/or off-loading work onto field offices; d) safeguard the quality standards of evaluative work including representation of all stakeholder groups, leaving no one behind; e) only be undertaken when there is a clear plan for utility and the needs for the evaluation results outweigh the risks and costs associated.
For more information, see the Evaluation Planning Decision Tree (page 3) in the Guidance Note
CONSIDERATIONS FOR ONGOING EVALUATIONS
a) communicate on the information needs and any changes from the original ToR based on COVID-19; b) share any relevant data on COVID-19 responses and or impact to the programme; c) remain open to innovative data collection tools and methods; d) actively engage with IES and the evaluation team; e) ensure an evaluation design and final report that meet the evidence needs; f) ensure adequate representation, leaving no one behind.
For more information, see the Evaluation Management Decision Tree (page 5) in the Guidance Note
POTENTIAL RISKS FOR EVALUATIONS
a) bias against under-represented groups; b) lower quantity and quality of data; c) inability to conduct on-site-data collection; d) limited access to field sites; e) limited availability of national evaluators; f) limited availability of field office staff and counterparts; g) health-related risks; h) hesitance to novel methods.
For more information, see Table 1 on Risks and Mitigation Measures (page 7) in the Guidance Note
REMOTE DATA COLLECTION OPTIONS
a) on-line interviews per e.g. Skype, Microsoft Teams etc. or per phone; b) surveys via mobile phone, email, online tools; c) on-line discussion platforms; d) video calls; e) remote observation via on-line feed or video footage; f) web scrapping; g) web search data analysis; h) crowd sourcing.
For more information, see Table 2 on Evaluation Data Collection Options, including limitations and requirements (page 9) in the Guidance Note
As a response to COVID-19, IES adapted its approaches and identified solutions to supplement data collection and analysis to increase the utility of evaluation results. IES also invested heavily in enhancing remote data collection methods and in new technology to ensure continued robust methodology in all UNODC evaluations. See IES Response to COVID-19 Crisis (Respuesta de IES COVID-19).
In June 2020, IES also held webinars for managers and project staff involved in evaluations on "Planning and undertaking evaluations in UNODC during the COVID-19 crisis". Read more in IES Web Story: Evaluation in the Time of COVID-19.
OECD/DAC and IEO/UNDP (2020) Guidance Note: Good practices during COVID-19
UNICEF Evaluation Office (2020) Technical Note: Response of the UNICEF Evaluation Function to the COVID-19 Crisis
WFP Office of Evaluation (2020) Technical Note for {Planning and Conducting Evaluations During COVID-19
UNFPA Evaluation Office (2020) Adapting evaluations to the COVID-19 pandemic
UNDP Independent Evaluation Office (2020) Evaluation planning and implementation during COVID-19
ILO Evaluation Office (2020) Implications of COVID-19 on evaluations in the ILO
FAO Office of Evaluations (2020) Risk analysis and guidance for the management and conduct of evaluations during international and national level COVID-190 crisis and restrictions
