The USIP Learning Agenda: An Evidence Review

What Constitutes Effective Use of Evidence to Inform Peacebuilding Project Design?

Ruth Rhoads Allen

President and Chief Collaboration Officer, CDA Collaborative Learning Projects, Cambridge, MA

Pushpa Iyer

Director, Center for Conflict Studies, Alexandria, VA

Lillie Ris

Associate, Planning and Evaluation, CDA Collaborative Learning Projects, Cambridge, MA

MARCH 2023

Introduction

BACKGROUND AND RATIONALE

In light of the growing interest in and demand for evidence-based peacebuilding projects from donors, international agencies, conflict-affected countries, and peacebuilding organizations, the United States Institute of Peace (USIP) sought to understand how evidence from conflict analysis and assessments informs project design and how subsequent adaptations to project design are made based on evidence gathered during project implementation. This evidence review is intended to help inform the development of a new USIP Programmatic Conflict Analysis Tool (PCAT), part of the Institute’s wider focus on enhancing conflict analysis capacity and programming effectiveness.

USIP engaged CDA Collaborative Learning Projects (CDA) to partner in conducting the evidence review because of CDA’s technical expertise in evaluating the effectiveness of peacebuilding projects and its trusted role leading processes that span sectors and organizations to produce practical learning. Research was conducted in September through December 2021 and employed a mixed methods approach that consisted of (1) a metasynthesis of information acquired from published and unpublished sources, (2) primary data collection through six interviews with two USIP project teams, and (3) three roundtable discussions with a total of thirty-two external expert-practitioners.¹

The research initiative was based on the premise that the effectiveness of peacebuilding projects stems from a web of factors, including project design and whatever informs the aims and approaches of that design, both concepts and data. Over the past decade in particular, the growing demand for evidence-based practices has revealed the strengths and weaknesses of existing approaches to assessing the performance of peacebuilding projects,² and thus has revealed opportunities to design projects that are truly evidence-based, along with the limitations of such methods. The relatively new demand for evidence-based practices in peacebuilding comes from both top-level (states, policymakers, major donors) and grassroots or bottom-level (local groups, communities, implementing staff of global institutions) organizations. Many organizations and their leaders seeking peacebuilding impact are keen to review their peacebuilding project cycle, from design to implementation and evaluation, and to incorporate evidence-based practices at every stage.

For USIP, the interest in evidence-based practices is tied in particular to conflict analysis as a tool for gathering evidence for project design and implementation in conflict-affected contexts.³ As a diagnostic tool to understand conflicts, conflict analysis takes many forms and is amenable to diverse analytical approaches.⁴ As such, USIP focuses on the role of evidence, generated through extant conflict analysis approaches more broadly, in the design and adaptation of programs. This research initiative sought to build on the foundation of recent surveys and definitional work about what constitutes evidence in the peacebuilding field⁵ and to sharpen understanding, based on the literature review and original research with practitioners, of how practitioners make effective use of evidence to inform the design and implementation of peacebuilding projects.

In 2021, the world marked a thirty-year high in violent conflict.⁶ How USIP and the wider peacebuilding field translate evidence into effective programming is thus an urgent matter. Ultimately, USIP aims to advance internal and sector-wide efforts to make project design evidence-based so that it can better contribute to preventing and ending conflicts and promoting stability.

RESEARCH QUESTIONS

The primary research question explored for this evidence review was: What constitutes effective use of evidence to inform peacebuilding project design?

The CDA team found that this overarching question would benefit from further nuancing in its specifics. For example, Snilstveit et al. (2012) hold that evidence-informed decisions require systematic study but should not be narrowly focused on the question of what works. Instead, the research should address a range of questions that go beyond a narrow measurement of effectiveness to ask why something works or does not work.⁷ With this in mind, the CDA team developed the following subquestions:

A. What constitutes evidence in peacebuilding project design, implementation, and evaluation?

B. How do projects utilize existing research and evidence for project design and adaptations to the design?

C. What are the barriers to using evidence from conflict analysis and assessment reports or from monitoring and evaluation in project design and adaptation?

D. How can existing evidence be integrated with local knowledge and the results of monitoring and evaluation to contribute to evidence-based project design and adaptation?

Two additional subquestions that warrant further exploration, as mentioned in the Recommendations section, are:

A. How do projects generate or capture local knowledge and local understandings? How is this knowledge integrated into project design and adaptations?

B. How do projects collect rigorous monitoring and evaluation data to contribute to project design and adaptation?

The methodology uncovered a wealth of practical knowledge among peacebuilders working in individual contexts and globally to advance the pursuit of peacebuilding effectiveness.

BOX 1. Key Definitions: Conflict Analysis and Evidence

USIP defines conflict analysis as the systematic study of conflict in general and of individual or group conflicts. Conflict analysis, therefore, can be broadly understood as an approach to using evidence collected about a conflict to inform project design. This evidence review uses “conflict analysis” as the inclusive term and “assessment” or “mapping” where specifically relevant, though the terms are often used interchangeably in the literature and by participants engaged in this study’s primary research efforts.

For this study, evidence is both formal and informal assessments (including evaluative), analyses, and experiences that explicitly inform project/program design. This definition guided all aspects of the mixed methods approach of this evidence review. The Findings section further discusses distinctions, including evidence-based and evidence-informed practices, as well as local and global evidence.

SUMMARY OF MAJOR FINDINGS AND IMPLICATIONS

An initial assessment of the published literature directly addressing the primary research question and subquestions confirmed our expectation that evidence-based design for peacebuilding is an understudied area of the peacebuilding field.⁸ That deficit notwithstanding, hundreds of peacebuilding projects are designed every year.⁹ Moreover, the people and teams seeking funding sources, designing peacekeeping projects, and conducting research on project impact understand the role of evidence in their work. For this reason, the CDA team decided to use a mixed methods approach to explore these questions. A metasynthesis of published and unpublished sources formed one prong of the research; primary data collection through interviews and roundtable discussions with USIP and external expert-practitioners formed the second and third prongs.

The CDA team found that in the absence of standard practices, individual organizations designing projects with peacebuilding objectives develop and follow their own practices for integrating the evidence gathered into the design of the projects and the subsequent adaptations during implementation. As such, organizational experience varies widely, with many organizations reporting challenges in adopting evidence-based practices. The reported challenges extend from difficulty gathering evidence and incorporating it into program design to struggles in implementing and evaluating the programs and programmatic learning. Moreover, while we found instances of the effective use of evidence in project design, the approaches were often isolated and not robustly practiced across projects even within a common context, let alone across the organization; nor were the teams carrying out evidence-based practices systemically supported in their work. As a result, it was very difficult to sustain such approaches within a context or at an organizational level. In addition, the study team found that in many organizations the mechanisms to learn from approaches that had proven effective were nascent, aspirational, or fully absent.

The uneven or ad hoc nature of evidence-based practices, the disconnect between intent and practice, and the lack of a systems approach in many peacebuilding organizations all signal that a more focused and sustained effort is needed to institutionalize the approach to evidence-based practices. But the challenge is larger: individual organizations’ efforts are more likely to achieve meaningful impact on peace and conflict dynamics if the organizations work within an ecosystem comprising other organizations, donors, policymakers, and academic researchers who are equally committed to making effective use of evidence in project design and implementation. This systems perspective informs the report’s recommendations for USIP and the wider peacebuilding sector.

Effective evidence for project design in conflict-affected contexts, the CDA team concluded, is the local and situationally specific evidence that is generated, interpreted, put into practice, and reinterpreted to become both more relevant and generalizable. This framing shifts the discussion from evidence-based practices to evidence-informed practices. Both terms are used in this report.

Key Findings of the Study

Regarding the origins and evolution of evidence-based practice for project design and adaptation:

Evidence-based practices were adopted in the social sciences only about three decades ago, with the peacebuilding field following suit only in the past decade or so.

The push for adopting evidence-based practices in the peacebuilding field is both top down, coming from donors, states, policies, and bottom up, coming from local organizations, communities, and implementing staff.

Evidence-based practices can be politicized when those in power at the top demand this approach be implemented without empowering practitioners to do so, or else make unilateral decisions as to what constitutes evidence.

Regarding what constitutes evidence in peacebuilding project design and what constitutes effective evidence:

The peacebuilding field has not developed a consensus understanding around what constitutes evidence.

Whether evidence of any kind is able to be generalized and transferred across contexts is circumstance specific; thus there is not a consensus in this respect either.

Looking for more relevant evidence (which requires interpretation of evidence) makes peacebuilding practices more evidence informed.

Regarding how evidence is currently used in peacebuilding project design and what the challenges are:

Context matters. Evidence must be gathered by including as many local perspectives as possible to ensure the relevance of the evidence and its application to design.

Capacity matters. Leveraging evidence-based practices requires appropriate organizational capacity.

Perspective matters. There is a need for negotiation on what constitutes evidence and how evidence should be used because of potential differences in how members of a local community and those working within programming boundaries might view or use evidence.

Process matters. More effective than individual, formalized conflict assessments for project design are iterative engagements with local partners and stakeholders.

Study Design

A three-pronged mixed methods design was employed for this evidence review. The CDA team developed the design keeping in mind the purpose of the evidence review (described in the Introduction) and cognizant of the limitations of each method when used in isolation. We integrated the following three methods for the evidence review:

Metasynthesis: We reviewed published and unpublished materials on the use of evidence (particularly evidence drawn from conflict analysis) in project design and adaptation. The metasynthesis methodology is described in appendix 1.

USIP case study: We conducted key informant interviews with two in-country USIP teams and reviewed relevant documents to learn how they used evidence in project design. The full case study, including methodology, is presented in appendix 2.

Expert-practitioner roundtables: We facilitated small-group discussions to hear challenges to and successes in using evidence from conflict analysis, monitoring and evaluating data, and local sources for project design and adaptation. The roundtable methodology is described in appendix 3.

During the initial stages of the study, the CDA team recognized that a metasynthesis would not result in a holistic understanding of the theory and practice of evidence-based project design by peacebuilding organizations. Reports of individual organizations’ experiences in the peacebuilding field and some theoretical articles on the application of evidence-based approaches in other fields, including nursing, social work, and education, constituted the bulk of our research results as we sought to construct a metasynthesis. Concurrent with CDA performing the metasynthesis, USIP’s Learning, Evaluation, and Research team created an Annotated Guide to Resources for Conflict Analysis to map approaches developed both internally by USIP and externally by sources outside USIP.¹⁰ The compilation includes scholarly conflict analyses and models of conflict assessment used by humanitarian and development practitioners. Both the literature reviewed for the metasynthesis and USIP’s Annotated Guide found that the current documentation on conflict analysis and the use of evidence for peacebuilding project design does not detail the internal processes that organizations undergo to adopt evidence-based practices. The metasynthesis thus led to discussion of the challenges to adopting evidence-based practices but to far less practical discussion of project design or of how evidence might be shared among peacebuilding organizations, donors, and researchers. The CDA team sought to address this lacuna through other methods, namely, direct informant interviews and roundtable discussions. The gaps in knowledge revealed through the metasynthesis highlight the importance of using a mixed methods approach.

The primary data collection effort included two sets of activities designed to complement the metasynthesis and enable exploration of the secondary research questions (questions A–D listed on p. 3). This format allowed a more focused consideration of how evidence is currently used to inform project design. The interactions of the CDA team with the roundtable participants, who had firsthand knowledge of how evidence-based practices worked within their organizations and the peacebuilding field, together with the empirical data gathered from key informant interviews with USIP teams, afforded the CDA team a practical understanding of the gap between current practices and the goal of rigorous evidence-based project design. It also yielded actionable insights for USIP’s development of the envisioned PCAT, explored in the Recommendations section.¹¹

METASYNTHESIS

The evidence review began with a metasynthesis of approximately 125 articles, which the study team narrowed down to a set of 43 core resources on how evidence is interpreted and applied in project design and subsequent adaptations (table 1). The metasynthesis began with a systematic series of keyword searches performed using various search engines, such as PRIO, CIAO, JSTOR, Academic Search Elite, Sage Premier Journals, and Google Scholar. Organizations’ websites were also viewed for studies and reports. Finally, many roundtable participants sent the study team reports that were not easily accessible on their websites. A detailed discussion of the data sources, data selection and eligibility criteria, sampling targets, and limitations is provided in appendix 1.

CASE STUDY INVOLVING USIP PROGRAMMATIC TEAMS

The case study method allowed the CDA study team to ground the findings of the metasynthesis and roundtable discussions in the experiences of USIP teams. USIP selected the two in-country teams based on a broader internal conflict analysis mapping exercise indicating that the teams’ experiences with integrating evidence into project design might allow useful comparisons across two key dimensions, time and resources. The CDA team conducted key informant interviews with four team members on team 1 and two team members on team 2, all of whom had been actively involved in conducting conflict analysis or project design processes in recent years. To complement the interview data, the study team reviewed internal documents such as conflict analyses, assessment reports, and project proposals produced by each team. Reviewing these documents yielded a good perspective on the context and dynamics in which the USIP teams were operating and how they used different types of evidence in project design.

Table 1. Metasynthesis Sample Sources

Type

Number of resources

Academic (conceptual and empirical)

16

US government (primarily USAID) and United Nations

5

NGOs (primarily INGOs), development banks, and private foundations

15

Blogs, websites, and conference reports

7

Total

43

The CDA team synthesized interview data from each USIP team using thematic analysis to identify patterns related to the types of evidence used most and least effectively in project design, and the related limitations and implications. Drawing on the interview data and document review findings, we developed a picture of how each project gathered and used evidence in designing projects. We compared the two teams’ experience gathering, interpreting, and applying evidence and examined differences in their approaches, along with factors that contributed to each team’s capability to use evidence. The full case study is provided in appendix 2.

EXPERT AND PRACTITIONER ROUNDTABLES

The participatory roundtable discussion sessions afforded the CDA team the opportunity to listen to and learn from academic researchers, donor representatives, and practitioners from international and local peacebuilding organizations outside USIP (table 2). The use of force field analysis techniques helped participants identify enablers of and barriers to the effective use of evidence in peacebuilding project design and to generate ideas about ways to address key barriers. Force field analysis is a tool for exploring root causes and taking action, often applied by peacebuilders to understand factors effecting the balance of conflict dynamics and their impacts. The roundtable methodology is discussed in more detail in appendix 3.

The roundtable discussions produced a rich dataset of both influential factors and potentially practical considerations for USIP to consider in developing tools and frameworks for future use. These tools and frameworks should be enabling as practitioners undertake more strategic, potentially systematic shifts in ways of working.

Table 2. Roundtable Participants

Roundtable

Participants

Women

Men

Total

Roundtable 1

Experts from USAID, foundations, and academia

6

6

12

Roundtable 2

Technical and project design experts working locally or with national NGOs

7

1

8

Roundtable 3

International NGO technical and project design experts working globally or regionally

7

5

12

Total

20

12

32

ITERATIVE ANALYSIS

The three-pronged approach to compiling data for the evidence review produced different kinds of evidence, as intended, and facilitated iterative learning about the topic. For example, because the initial metasynthesis yielded no unpublished literature that met our criteria for relevance, we asked invited roundtable participants directly for such materials, and made a second request while following up on the roundtable discussions. The planning process for the roundtables also helped the study team identify additional key online search terms and establish definitions (see the Introduction). The key informant interviews with USIP teams informed adaptation of the force field analysis approach used in the roundtables. While the CDA team was in the process of completing the draft case study, USIP introduced the preliminary results of other research initiatives of the organization’s Learning, Evaluation, and Research team, leading to conversations that began the interpretation phase and the recognition of patterns across the datasets. An interim fieldwork report summarized themes from both the case study and the roundtable discussions. The internal presentation to USIP of the initial findings of the metasynthesis, which drew on techniques for visually organizing information, facilitated further pattern identification and prioritization.

LIMITATIONS

The design of the evidence review evolved over the course of the data collection, with minimal differences from the planned approach. The metasynthesis methodology was intentionally leveraged to systematically select 43 key published articles or reports from three general sources, broadly categorized as (1) theoretical or conceptual in nature, (2) issued by the US government (primarily USAID), and (3) issued by NGOs (primarily INGOs). The CDA team had initially hoped to find articles and case studies that would distinguish between the use of evidence in the initial project design stage and later, when adaptations to the initial design were made. However, it proved very difficult to make this distinction. The materials are best classified as falling under evidence-based practices, which include project design, implementation, adaptation, and monitoring and evaluation. The study team faced a similar problem with the case study method, and hence the findings from this method similarly do not emphasize the distinction between initial project design and subsequent adaptation.

For the metasynthesis, the study team was unable to access unpublished sources of data that met our criteria for significance. We believe unpublished sources would have given greater insight into an organization’s internal dilemmas and debates when using evidence-based practices. Requesting specific documents from key practitioners, particularly after-action reviews or guidance notes prepared by project design teams on structuring conflict analysis for design purposes, could be a beneficial next step as USIP advances development of the PCAT.

For the USIP key informant interviews, one of the two country teams expected to participate was replaced with a different country and team. Ultimately, the teams selected still offered the envisioned comparison. The small number of teams we were able to engage in this research makes the findings from this method difficult to generalize to the experiences of all USIP teams, much less the field at large. However, since there was significant alignment with many of the themes that emerged from the roundtable discussions, and no findings contradictory of the metasynthesis emerged, we feel this method offers an important nuance relevant to USIP’s planned PCAT.

For the roundtable discussions, the study team chose to include a larger number of expert-practitioners than initially anticipated owing to the high degree of interest among the invited participants. Expanding the group size yielded greater diversity both in the personal and professional identity of the participants and in the types of experience proffered in using evidence in project design across a broader set of contexts. As planned, all participants assembled for the orientation to the roundtable format, but the large number of participants necessitated small-group facilitation to achieve robust conversation among peers. For all roundtables, a closing report from each of the small groups allowed main themes to be shared. As such, the small-group discussion format enriched this method, even if it limited exchanges among the three larger groups.

The metasynthesis and the USIP case study, but not the roundtable discussions, directly engaged the secondary question, “What constitutes evidence in peacebuilding project design, implementation, and evaluation?” This is consistent with the original research design. As such, the corresponding section of this report includes only complementary anecdotes from the roundtables, not a full comparative analysis.

Three notable limitations arose during the evidence review, which the CDA team identified and found strategies to address.

1. Language of research. We did not include non-English sources of data. We are aware that this lacuna resulted in data that do not fully represent the thoughts of and methods used by non-English-speaking communities. Failure to include such communities may also have led to our using data sources that simply reproduced ideas shaped by colonial power relations.¹² We acknowledge this limitation. One way we sought to overcome it was by ensuring a broad geographic distribution of US government and NGO and empirical materials reviewed.

2. Self-reporting. The case study analysis was based on information that was self-reported by members of each USIP team. It is possible that team members were motivated to paint their efforts in as favorable a light as possible or otherwise to color the information they shared. We sought to mitigate this possibility by operationalizing a robust interview protocol, including the use of questions that elicited recall of the tangible processes and steps undertaken.

3. Positive deviance. The case study method focused on two USIP teams that have used evidence to inform project design. This meant that no comparisons could be made with teams that might not use evidence to inform project design. This was a method choice: we decided to go for depth, given the time constraints and limited scope of our study.

A summary of the methodological limitations and mitigation strategies is found in table 3 (below).

Table 3. Summary of Methodological Limitations and Mitigation Strategies

Limitation

Mitigation strategy

Literature reports rarely distinguished between project design and subsequent adaptations.

The CDA team sought to understand evidence-based practices through a project cycle.

Unpublished data from INGOs were difficult to access because of internal confidentiality and unconsolidated formats.

Materials were requested directly from roundtable participants.

Changes in availability and choice of USIP case study teams and in number of roundtable participant numbers.

Research strategies were flexibly adapted to realities.

Only English-language sources were considered, potentially precluding insights from non-English-speaking areas; all roundtables and interviews were conducted in English.

Articles and reports on empirical research conducted in diverse parts of the globe were included.

Choice of USIP teams for case studies created the potential for favorable reporting of evidence-based practices used.

A robust interview protocol was used.

Choice of case studies did not allow comparisons to be drawn between groups that used evidence-based practices and those that did not.

The team opted for depth, given the scope of the evidence review.

The Findings section of this report presents an integrated analysis of the findings obtained by applying the three methods. The Recommendations section highlights the cross-cutting areas relevant to USIP’s subsequent development of tools and frameworks, along with broader considerations of the effective use of evidence to inform project design.

Findings

The major findings of the study are presented first to demonstrate how the effectiveness of the evidence is itself a factor influencing design. The discussion then moves on to findings about effective use, including why barriers to and enablers of use exist, and names four challenges.

ORIGIN AND EVOLUTION OF EVIDENCE-BASED PRACTICES FOR PROJECT DESIGN AND ADAPTATION

Highlights

Evidence-based practices were adopted in the social sciences only about three decades ago, with the peacebuilding field following suit only in the last decade or so.

The push for adopting evidence-based practices in the peacebuilding field is both top down, coming from donors, states, and policymakers, and bottom-up, coming from local organizations, communities, and implementing staff.

Evidence-based practices can be politicized when those who have the power at the top of an organization demand that this approach be used without empowering practitioners designated to implement such practices, or are making decisions as to what constitutes evidence.

The origins of evidence-based practice lies in the medical field, and particularly in the nursing field (box 2). The social sciences followed suit in the 1990s, with the related fields of development, peacebuilding, and policymaking subsequently adopting both the language and the logic behind the use of evidence in project design and practice. Since then there has been an increasing demand for use of evidence-based practices in international development and the peacebuilding field.¹³

Various authors and institutions, including Van Dyke and Naoom, USAID, Eyben and Roche, Whitty and Dercon, and Brown, highlight the following as some of the reasons why peacebuilding has ascribed to evidence-based practices:¹⁴

A desire to achieve better results and outcomes from interventions;

A requirement for evidence-based project design and adaptations from donors;

A solid push for including data from the communities designated for receipt of the interventions;

A recognition that project designers should not craft projects based solely on favorite methods, politics, or funding sources;

A realization that change is happening fast, while learning is taking place more slowly, underscoring the importance of learning from evidence; and

A growing awareness that evidence-based practices reduce the risk of causing harm through usual practices and reduce the bias that practitioners may bring to a project.

BOX 2. The Roots of Evidence-Based Practice

McMenamin et al. trace the roots of evidence-based practices to nursing, starting with the approach of Florence Nightingale in the 1850s.a Nightingale recorded data on deplorable hospital conditions and the role of hygiene in nursing care. They note that this was one of the first examples of evidence—actual data—being used to understand why ten times as many soldiers died from disease as from wounds. Since then, the medical field has pioneered the use of evidence instead of intuition to guide decision-making and practices in health care. Borrowing from some seminal writings in the medical field, other disciplines such as nursing, social work, child development, psychology, and education began documenting the benefits of using evidence-based practices.

a. McMenamin et al., “The Evolution of Evidence-Based Practice.”

A theme that emerged from all three research methods was the lack of consensus on whether evidence-based practices must be followed, whether they are effective, and how to overcome the many challenges to adopting them (see, for example, the discussions by Whitty and Dercon and by Eyben and Roche).¹⁵

Research investigating why the peacebuilding field adopted evidence-based practices invariably ends up emphasizing government or donor accountability and community-driven demand, reflecting both a top-down and a bottom-up push for using evidence-based practices. Mack highlights the importance of using evidence in peacebuilding initiatives because absent evidence, there is the risk of governments (and by extension international donor agencies and their NGO partners) disguising project failures or claiming progress where little exists.¹⁶ Further, donor accountability to invest in evidence-based practices by strengthening the capacities of local governments and institutions needs to be incentivized for peacebuilding policies to be effective.¹⁷ Finally, acquiring robust evidence can aid in exposing government deficiencies such as corruption and make governments responsible for building peace.¹⁸

It is important to recall here Ghate’s caution that donors often refuse to accept that sometimes evidence-based practices are not as effective as “low-cost, home-grown, practitioner-developed programs” and instead defer to credentialed external experts or turn to approaches that were deemed successful in other contexts.¹⁹ However, the “home-grown,” practitioner-developed approach may well be evidence-based without being formally labeled as such. This point echoes the central reflection of the USIP project teams that contributed to the case study and of the peacebuilding practitioners, both local and international, who participated in the roundtables. As one participant noted, “There’s a lot we know that we don’t write down as ‘evidence’ because it’s just our experience. It’s the same for our partners. That’s why they have to be at the design table.” As the following sections show, both an upward and a downward push and an internal and external drive to gather and use evidence is critical for the success of peacebuilding projects.

Even as the field recognizes the need for and the importance of using evidence, many peacebuilding actors report they often feel pressured to adopt specific evidence-based practices when designing, implementing, and evaluating their projects regardless of whether or not they think those practices will lead to effective results. This pressure was noted by local peacebuilding practitioners in the roundtable discussions as coming from donors and international agency partners, who want to use evidence they know and trust, even if it does not feel relevant to the context or circumstances. One participant described a donor meeting that was also attended by a newly arrived embassy staff member, who recommended an approach the staff member felt had been effective in a previous posting. The practitioner asked, “What am I supposed to do in that situation with a donor?” Local peacebuilders noted a different dynamic, with pressure coming from their own government, such as ministry officials requesting they use data or approaches created by a government in the global north with which the ministry was trying to raise its own profile.

Mack similarly asserts that the pressure to engage in evidence-based practices comes mostly from donors, international agencies, and governments with power over funding decisions. He explains that it is now commonly acknowledged that the decline in high-intensity armed conflicts owes to successful peacebuilding, peacekeeping, peacemaking, and humanitarian assistance projects.²⁰ However, a lack of precise data on both the conflict context and peacebuilding efforts has made it difficult to understand this phenomenon. With the governments of fragile states now leading dialogues with their counterparts in the global north on the new Peacebuilding and Statebuilding Goals (PSGs), there is a growing demand that they use indicators to track progress toward meeting each goal, acquiring quantitative data where possible.²¹ Indicators yield an evidence database. Governments in the global north are forcing their southern counterparts to use evidence-based practices without increased donor aid, which further entrenches the north-south power dynamic.²² But when organizations adopt evidence-based practices because of mandates from outside the organization, the entire process inevitably becomes politicized because there is a push to show either evidence or outcomes that are expected by the donors or international agencies. Eyben et al. hone in on politicization by asking such questions as “Who decides what data is needed, how it should be collected—and why and how it is used as evidence? What drives their choices?”²³ As they point out, a politics of what counts as good or credible evidence is deeply embedded in these questions.

Additionally, the power dynamics in peacebuilding organizations result in the valuation of some processes and protocols within the organization. Eyben et al. opine that how an organization determines what counts as evidence or as effective evidence is never made explicit.²⁴ The reason often adduced in the literature as to why organizations do not make their values explicit is so that they can control what constitutes effective evidence to suit their own agendas. Roundtable participants from INGOs raised related points, describing the lack of clarity around what counts as evidence as a known operational limitation resulting from the reality of having multiple donors with different expectations concerning evidence, and from the type of evidence that can be gathered in any given context. Roundtable participants from both government and private donor organizations offered that they have a role in this dynamic, and are significantly dependent on academic and practitioner research to define what evidence is possible and important. The dynamic points to a vicious circle.

Another point raised by the roundtable participants who represented implementing organizations was the approach that external actors such as donors and INGOs take to valuing some local groups. It is not difficult to imagine that elites within a community might have more of a voice and greater access to external actors. However, the roundtable participants brought up another, newer dynamic: the recent global effort to amplify the voices of previously marginalized groups has had adverse effects. Specifically, marginalized groups contend with each other to present themselves to donors and INGOs as most victimized. This situation complicates evidence gathering because the politics of conflict define the types of evidence considered for project design and adaptation. Adam adds another dimension to the politics of gathering evidence. He explains that conflicts themselves are multidimensional, making it difficult to evaluate any one project or approach to peacebuilding separately for its effectiveness. This makes it hard to identify evidence that is useful or effective.²⁵ One could argue that when organizations, donors, or governments emphasize one project or approach as most effective, evidence takes on a political tinge because data in support of counternarratives or alternative theories of change often are not collected, and it is challenging to evaluate which combinations of projects or approaches contribute to success.

If gathering evidence and applying it to project design is a political process, that complicates reaching a consensus on what constitutes evidence in peacebuilding project design or what constitutes effective evidence. Additional complicating factors include the co-occurrence of political drivers (top-down) and a bottom-up quest for impact, the mandate to “do no harm,” and the desire to professionalize the use of evidence-based practices in peacebuilding. In the following section, the study team’s metasynthesis of relevant literature looks at how the peacebuilding field has grappled with these topics and relationships, and its findings from primary data collection offer insights from practice.

WHAT CONSTITUTES EVIDENCE IN PEACEBUILDING PROJECT DESIGN? WHAT CONSTITUTES EFFECTIVE EVIDENCE?

Highlights

The peacebuilding field is not united around one definition of what constitutes evidence in peacebuilding.

Whether evidence of any kind lends itself to being generalized and transferred across contexts is highly context- and circumstance-specific, which works against the development of a consensus.

Evaluating the relevance of evidence (which requires interpreting the evidence) makes peacebuilding practices more evidence informed.

What constitutes evidence remains a fraught question for peacebuilding professionals. And, while not the primary research question, it was important to examine for context as the study team dived deeper into how evidence is used effectively in design decisions. We saw this definitional challenge heavily emphasized across the literature scrutinized in the metasynthesis and further reinforced by the primary data sources, the USIP interviewees and the roundtable participants.²⁶ By examining how evidence is used, the CDA team was able to validate and develop in-depth insights to support the findings noted in the Seyle et al. report regarding what constitutes evidence (discussed further below).²⁷

The following discussion refers repeatedly to global and local evidence. For this report, global evidence is taken to be evidence introduced from outside the local context and largely by external actors. Local evidence is understood as evidence that is generated in the context in which the peacebuilding efforts are undertaken, often by people directly affected by conflict. No assumption is made that local evidence is monolithic, and, as noted in the previous section, the decision on what constitutes evidence is often a very political one. The references to local evidence therefore are meant only to distinguish it from what INGOs and donors might consider—even prioritize—as evidence. Global evidence itself often takes the form of a collection of local evidence, acquired in a variety of contexts, that is then generalized and interpreted by external actors. As such, there are inherent power dynamics in play that must be examined when categorizing types of evidence, and even more so when selecting evidence for design purposes or for programming implementation, adaptation, and evaluation.

In the key informant interviews, the USIP teams identified eight primary forms of evidence that served as inputs to devising a definition of evidence. This categorization is largely consistent with the survey findings reported in Some Credible Evidence in response to the question, “What does the term evidence-based mean to you in the context of peacebuilding work?” (table 4). Of note, the One Earth Future-Alliance for Peacebuilding study that was the basis of the report explored a broader set of research questions than the CDA team did: How is evidence-based practice implemented? What kinds of evidence should be considered valid? Who gets to decide? Our questions address the nature of evidence as it applies to the design of peacebuilding projects in conflict-affected contexts. This examination makes a significant difference related to the importance of local evidence and conflict analysis versus global evidence and traditional or academic methods.

For example, the two types of evidence most commonly referenced by USIP interviewees were (1) input from partners and (2) consultations with communities, local and national stakeholders, or those affected by conflict. These points align with the categories in the 2021 report by Seyle et al., Some Credible Evidence, indicating that credible evidence is evidence that is “supported by field experience” and “inclusive of beneficiaries/local perspective.”²⁸ However, the emphasis placed on these types of evidence by the USIP teams was much greater than suggested by the findings supplied in Some Credible Evidence, wherein these types of evidence were highlighted by only two percent and six percent of survey respondents, respectively (see table 4). Furthermore, while Some Credible Evidence identifies several types of methods or processes that peacebuilders find credible, conflict assessments or analyses do not make that list as a separate method. Many roundtable participants also referenced the report as helping define categories of evidence, as well as what is widely understood to not be evidence.

This notable difference between the USIP interview findings and Some Credible Evidence in the degree to which local knowledge counts crops up again in the larger body of published literature and stood out as a notable discrepancy between the perspectives shared by academic and donor roundtable participants and the perspectives of local peacebuilding participants. Deciding on what constitutes evidence was noted as “a power dynamic in itself” by local peacebuilders in the roundtable discussions and was raised by participants from INGOs as well. Throughout the USIP case study, local knowledge was both upheld as evidence and noted for its effectiveness when applied to peacebuilding project design. Two excerpts from the case study are illustrative:

Input from local partners is valued highly because of partners’ access to stakeholders and communities affected by conflict and because of their working knowledge of USIP’s mandate and approaches, which is key to their ability to translate their knowledge and experiences into information that USIP can act on. Interviewees described an ongoing flow of information between partners and USIP country staff, with country staff integrating partner input into project design and implementation decisions. Often, such information was not documented formally because it was transmitted in an organic and ongoing manner.

Because of the direct relevance and applicability of local knowledge, relative lack of evidence generated by formal monitoring and evaluation processes, and the specialized nature of recent formal conflict analyses and other existing evidence, USIP teams weigh local knowledge heavily in the design process. A key strength of this approach is that the consultative manner through which this information is generated reinforces USIP’s goals by building confidence among key actors and ensuring that programming is tailored to local priorities and conditions. The current realities of project design mean that teams are less able to draw on rigorous evidence of what works in their own and other contexts.

Table 4. Defining Evidence: USIP Teams compared with Some Credible Evidence report (2021)

Components of USIP Definition

IN

PCT

1. Input from partners and other relationships

2. Consultations with communities, national stakeholders, and those affected by conflict

3. Document reviews (including literature, other project documentation, media coverage)

4. Conflict assessments and analyses

5. Staff knowledge and expertise

6. Input from other implementers

7. Technical assessments

8. Project monitoring information and activity reports

Founded on a Collection of Other Work

93

27%

Supported by data

3

3%

Supported by evaluation

16

17%

Supported by facts

15

16%

Supported by field experience

2

2%

Supported by past programming

20

22%

Supported by personal experience

4

4%

Supported by previous research

33

35%

Method or Process Employed

62

18%

Tested method or approach

40

48%

Use of mixed methods (quant and qual)

25

40%

RCT/IE methods

5

8%

Case studies

2

3%

Rigor or Quality of Evidence/Method

49

14%

Rigorous

13

27%

Scientific approach

9

18%

Backed by Theory of Change

7

14%

Replication

5

10%

Meeting internal or external validity

4

8%

Peer-reviewed/public information

4

8%

Indication of causality

3

6%

Independence

2

4%

Triangulation

2

4%

Indication of Programmatic Success

38

11%

Proof of achieving desired outcomes/impact

26

68%

Using programming/approaches that have been proven to work/succeed

12

32%

Inclusive of Beneficiaries/Local Perspective

21

6%

Using Data for Decision-making

17

5%

Data-driven Design

14

4%

Context-specific/driven

13

4%

Other

10

3%

Presence of Indicators/Measures

9

3%

Reflects Learning

7

2%

Imposed Framework/Critique

4

1%

Employing Research Ethics

2

1%

Note: Adapted with permission from One Earth Future.

Related to evidence generated from conflict assessments specifically, there was also strong alignment between the roundtable participants and USIP interviewees that both formal and informal assessments, analyses, and experiences that explicitly inform design all constitute evidence.

While there was some overlap between individual organizations on what they considered evidence in both the metasynthesis and the roundtables, and to an extent with the USIP interviews on the issue of local knowledge, little was said about organizations negotiating among themselves to reach a common understanding of what constitutes evidence. This might be because negotiating to decide on a common understanding of evidence can be a political act, as suggested by Eyben et al. and Mercy Corps.²⁹ However, it could also be that even organizations with numerous sources of evidence might have neither the resources nor the skills to gather information from all those sources. Or, as discussed by practitioners participating in the roundtables, design teams may lack the analytical capacity to determine whether the evidence is complementary or contradictory. Consequently, it seems likely that each organization selects and highlights the one or two sources of evidence that best suit its needs.

The lack of congruence among organizations as to what constitutes evidence is driven home by the findings reported in Some Credible Evidence, which were especially referenced by the roundtable participants.³⁰ The authors of the report concluded there is little agreement among peacebuilding organizations as to what constitutes evidence. In part, this situation owes to organizational structure: the report notes that many peacebuilding organizations operate in silos, and therefore their methods and intervention contexts lead them to develop idiosyncratic perspectives on what constitutes evidence. The survey echoes the opinion of many observers that peacebuilding organizations should negotiate with donors and with other organizations in deciding what constitutes evidence in the context of each project at a given time and place. Even a context-specific agreement on what constitutes evidence is subject to sudden change: as some observers have pointed out, the constantly evolving dynamics and crises in any conflict context make it difficult if not impossible to gather “indisputable” facts. Moreover, the data need interpretation, which means bias continues to be a factor.

Our primary research findings suggest that peacebuilding organizations should examine whether generalizability or transferability of evidence for the purposes of project design is even possible if evidence is so dependent on context, especially a dynamic conflict context. A rapidly changing context notwithstanding, generalizability could allow organizations to converge on a common understanding of what might be considered evidence, and further, on what might be considered effective evidence. An evidence-based practice might then be understood as effective if it led to “success” in one context and practitioners were able to replicate the success in another context by using or adapting the evidence gathered from the first context. Mack refers to this as the tension between context and commonality in peacebuilding.³¹ In the discussion and execution of PSGs, he writes, two levels of indicators emerged, country-level and common ones. Country-level indicators reflect the economic and political contexts of the particular countries, while common indicators are general to all countries categorized as fragile states. There is a natural resistance to common indicators among practitioners in fragile states. These indicators are often viewed as externally imposed by donors and local practitioners believe such measurements do not adequately reflect the unique circumstances of fragile states.

The USIP team interviews did not directly address the generalizability or transferability of data for project design, but the issue came up in all three roundtables. Local peacebuilders offered the strongest perspective, noting that cultural harm can be done, with negative results for programming, when evidence derived from other countries or contexts is applied without adaptation. Data collected is often biased and generalized from one context to another which can confuse information and analysis or negatively affect local communities. “The evidence is seen as totally foreign. People do not feel it represents them or their culture.” The inappropriate application of data developed in other contexts sets up a dynamic of cultural insensitivity at best and can also perpetuate dynamics fueling conflict.

Generalizability and transferability were points of discussion as well among academic and donor roundtable participants. Many said that peacebuilding organizations rely on their expertise and knowledge to navigate local problems, particularly in rapidly changing contexts where it is difficult to access and confirm information in a timely way. There was much discussion as to whether local knowledge is better than generalized knowledge. Referring to this as the “generalizability puzzle,” Bates and Glennerster discuss the tensions some organizations experience by incorrectly framing effective evidence as a choice between a more robust generalizable knowledge and a less robust local knowledge. This wrong framing, they say, makes it even more difficult for a common understanding to emerge regarding what constitutes evidence and what constitutes effective evidence.³²

One of the strategies used by the Abdul Latif Jameel Poverty Action Lab at the Massachusetts Institute of Technology, or J-Pal, to ensure policy is informed by rigorous evidence does not require analysts to conduct new randomized evaluations each time. J-Pal believe that evidence generated from other contexts can be used to inform local decision-making. J-Pal uses its own generalizability framework to test for evidence suitability for a local context, saving both time and money.³³ Arguing for J-Pal’s approach and its success in achieving generalizability, Bates and Glennerster opine that global evidence (more generalized, often in the form of quantitative research) should be used as a basis for policy.³⁴ The evidence should be verified locally (as J-Pal does) before it is integrated into design and implementation. Bates and Glennerster point out that testing every piece of global evidence in every local context would be unduly time-consuming, and therefore, if the evidence has met with success in most contexts, it should be adopted in all local contexts.

Eyben and Roche, however, find that J-Pal’s approach treats poverty as a technical problem and not as a human or political problem. They suggest that framing poverty reduction as an evidence-based scientific problem deflects attention from the “centrality of power, politics, and ideology in shaping society.”³⁵ By contrast, researchers who characterize a problem and therefore the evidence emerging in that context as political shift the emphasis to recognizing the problem as a social one and not a technical one.³⁶ Countering this view, Bates and Glennerster believe that human behavior is more generalizable than a single-context approach or discrete evidence would suggest. If policymakers focus on evidence that helps change human behavior, they believe, most social problems would benefit from evidence (or knowledge) that was more global and more generalizable.³⁷ During a roundtable session, a local peacebuilder captured the breadth of these arguments and how much human and political context matters by noting, “Evidence should always see the paradoxes that exist in reality.”

The generalizability and transferability discussion around evidence-based practices finds a place in the liberal and critical peacebuilding debates (overwhelming reliance on democratic institutions vs. reliance on local and indigenous experiences and institutions) as well. Critiques of liberal peacebuilding point out the inherent tension of project designers adopting noncontext-specific peacebuilding measures, perhaps out of necessity, while at the same time advocating for more locally owned and locally executed peacebuilding efforts.³⁸ Lilja and Höglund believe that the growing trend toward centering local peacebuilding efforts on the policy agenda requires more interaction between local and international (external) actors.³⁹ It implies building trust so that external actors can become supporters of local peacebuilding efforts. Advocating for coleadership between external and local actors in peacebuilding efforts, from evidence gathering to designing projects, Lilja and Höglund say that building trust between actors takes a long time, and they urge the donor community to recognize this in their funding approaches.⁴⁰

The generalizability discussion thus is very closely tied to the effective evidence conversation but with no particular resolution. However, the term “relevant evidence” as used by Simons and colleagues and Bates and Glennerster suggests a possible way to reframe the question of credible or effective evidence.⁴¹ Effective evidence for project design in conflict-affected contexts, therefore, consists of the local and situationally specific evidence that is generated, interpreted, put into practice, and reinterpreted to be both more relevant and generalizable. This framing shifts the discussion from evidence-based practice to evidence-informed practice, which is a primary finding of Anderson and Olson and an argument for systems analysis as an aid to producing “new insights” and analytical perspectives.⁴²

This slight shift in terminology brings the discussion full circle. The methods used in this evidence review underscore a major theme, namely, that evidence, whatever its source, must be interpreted and used by those designing and implementing peacebuilding projects. The process of interpreting and using evidence itself creates new evidence, which then is reinterpreted for the design and adaptive implementation of projects. Thus, evidence gathering, interpretation, and application represent a cyclical and potentially unending process. This suggests that peacebuilding organizations must continually incorporate learning and fold in new evidence. This theme was explored in various ways by all three roundtables and resulted in recommendations for major groups of stakeholders:

For donors: “Flexibility is required as a factor to build up the evidence and be open to evidence saying that what you have been doing may not be effective for a particular goal.”

For INGO practitioners: “Assessment and evidence aren’t be-all/end-all. They are a starting point in project design. During implementation is where ground truthing and honing [of the] approach happen, especially [if you] consider that contexts are dynamic and changing.”

For local peacebuilders: “Yes, we can learn from evidence, but there’s a need to be innovative when we use that evidence.”

While discussions of what constitutes effective use of evidence have undoubtedly advanced the field, debates over evidence-based design often end up being debates over what should be considered evidence rather than over how evidence should be used in project design.⁴³ We conclude that the emphasis on reaching a common definition or even a negotiated definition for the peacebuilding field is misplaced and more attention paid to nuancing and deepening the discussion and ensure that these debates are relevant both conceptually and practically. Beyond the question of what constitutes evidence, primary data collection underscores the need for more robust exploration of the relevance of evidence as it is used to inform design and adaptation. Further, policymakers are encouraged to ask how evidence was selected for relevance and what evidence was left out. Knowledge of the selection criteria should help expose any political bias or agenda in evidence gathering and use while ensuring that organizations, staff, and donors give full attention to evidence that can help them make informed decisions about project design and adaptation. Roundtable participants likened this practice to the concept of the “easy to reach” versus the “hard to reach” effected people, part of the Do No Harm Framework.⁴⁴

Building on the foundation of recent surveys and definitional work on what constitutes evidence in the peacebuilding field,⁴⁵ the next section focuses on how practitioners can make effective use of evidence to inform project design.

HOW DO PRACTITIONERS CURRENTLY USE EVIDENCE IN PEACEBUILDING PROJECT DESIGN?

This section summarizes the main ways in which the roundtable participants and USIP teams described using evidence in peacebuilding project design and the major barriers to be overcome. This report then takes up the shift from evidence-based practice to evidence-informed practice and the resulting structural considerations.

Highlights

Context matters: Relevance to people directly affected by programming requires that evidence be gathered intentionally, with an understanding of the range of stakeholders and seeking all relevant perspectives to inform accountable and effective design.

Capacity matters: Implementing or adapting evidence-based practices requires that an organization have appropriate capacity to do so.

Perspective matters: Because members of a local community are likely to view evidence and its use differently from other actors involved in programming (governments, NGOs, businesses, civil society actors), there is a need for negotiation on what is evidence and how it should be used.

Process matters: In project design, repeated engagement with local partners and stakeholders is more effective than conflict assessments by external actors.

The practitioners overwhelmingly said “the how” matters in making effective use of evidence in project design and that it takes agile systems thinking on many levels to do so. As one roundtable participant said, “Evidence has to look at all the context, not just one issue within it. Design has to look at all the evidence, not just one perspective.” What systems thinking might look like in practice is less well defined. For the USIP case study, interviewees from the two project teams reported relying heavily on local knowledge and an “organic” process to access it, but with little detail on the analytical process (see table 5).

BOX 3. USIP Case Study, Key Finding 1: Priority of Ongoing Input from Partners

For team 1, ongoing and informal input from partners was the most frequently mentioned source of evidence to inform programming. “Partners” ranged from community-based activists to individuals who represented national institutions participating in USIP projects. Input from local partners is valued highly because of partners’ access to stakeholders and communities affected by conflict and because of their working knowledge of USIP’s mandate and approaches, which is key to their ability to translate their knowledge and experiences into information that USIP can act on. Interviewees described an ongoing flow of information between partners and USIP country staff, with country staff integrating partner input into project design and implementation decisions. Often, such information was not documented formally because it was transmitted in an organic and ad hoc manner.

Interviewees on team 2 also identified input from partners and field staff as an important source of information to inform programming. As one interviewee put it, “We have tried to create a trusted network of partners and staff. They are the best asset that we have.” The team described a process of ongoing consultation to get partners’ and local staff’s views on evolving ground conditions and ideas for interventions.

For the roundtables, we asked: What about the way evidence is used now (1) contributes to effective project design and adaptation, and (2) is having a negative impact on effective project design and adaptation? We were struck that, despite the broad parameters of the inquiry, participants in the three roundtables largely did not focus on individual technical strengths or gaps related to specific programmatic outcomes or intervention types (such as those categorized by Sonnenfeld et al. in Building Peaceful Societies). Rather, participants focused on characteristics of context and systems, and specifically the capacities, conditions, and mentalities that enable or hinder the effective use of relevant evidence in peacebuilding project design. There were two exceptions: roundtable participants did point to specific technical aspects of gathering and using evidence as potentially having a dramatic positive impact on effective project design when done well and serious negative impacts when not.

Navigating the influence of militarization: Roundtable participants representing the perspective of local peacebuilders observed that in violent contexts experiencing repeated cycles of violence, project designs often become “policies for achieving something,” and that, “in conflict areas, the first answer is militarization. This makes people [both implementing staff and intended participants] very afraid; they feel there are no possibilities to do anything.” Prioritizing inclusive evidence generation and analysis processes that center affected populations in design development, such as participatory action research methods, was offered as a way to counteract mistrust and disengagement in highly charged contexts.

Women and gender dynamics: Local and INGO peacebuilders and participants from the donor and academic arenas all underscored the importance of power dynamics—especially between men and women, but also among different ethnic groups—in affecting reactions to evidence use and project design among local communities. “We think this [gender analysis] is [regarded as] optional. It cannot be optional. We need to do gender analysis of children that are girls differently from analysis of others. We presume that people in a territory will agree on issues—we assume agreement on project implementation between ages, ethnicities, men and women. Many activities are designed without dialogues. We know this evidence will lead to harm.”⁴⁶

USIP team members interviewed for the case study likewise focused on the practices important to make use of effective evidence in designing peacebuilding initiatives. This method of evidence review gives specific insight into conflict analysis approaches tailored to a specific organization and context, with resulting implications for how the evidence so generated should be used. In the instance of a USIP team that conducted a regional conflict analysis, the evidence was intended to inform the development of new projects, yet it was mostly described as an effort to make the case for larger-scale programming and partnerships with international organizations or US government agencies. The other USIP team employed evidence to host a roundtable discussion to inform other organizations about USIP’s work, bolstering USIP’s reputation as active in that area and able to conduct conflict analyses. Both teams assigned greater weight to local knowledge and repeatedly engaging with local partners and stakeholders than to conflict assessment evidence in the actual design of USIP programming.

For both research methods, the practitioner roundtables and the USIP team interviews, the CDA team observed a related set of experiences and factors contributing to the effective use of evidence in the design of peacebuilding projects. These are listed in table 5, along with barriers to the effective use of evidence in project design that were identified by both USIP and roundtable participants.

We see a strong relationship between the barriers to effective use of conflict analysis and assessment and other forms of evidence and the factors working in favor of effectiveness, such as the noted enabling factor of “emphasis on evidence in design and as a starting point for adaptation” being an implied response to rapidly changing context dynamics as a stand-out barrier. In the course of the roundtable discussions, such barriers and enabling factors were repeatedly brought up as “two sides of the same coin.” Participants also discussed what influenced a given issue to become more enabling of effective use of evidence in design versus what influenced it to become more of a barrier. Three themes emerged from these discussions.

Table 5. Practitioner Experience of and Barriers to Effective Use of Evidence in Project Design

Factors identified by USIP teams as aiding in effective use of evidence in project design (experiential)

Barriers to effective use of evidence in project design identified by USIP teams and roundtable participants

Factors viewed by roundtable participants as contributing to effective use of evidence in project design

Ongoing input and team deliberation to understand community needs and conflict factors and determine how to address them

Developing a shared background understanding of the context and problem

Rationale for proposed solutions bolstered by existing evidence

Appetite for evidence as to what works

Teams’ and local partners’ knowledge and understanding foundational to decision-making

Reliance on informal tools and systems

Short design process time frames

Rapidly changing context

Varying reliability and accessibility of data, or a digital divide preventing timely transmission of data

Limited applicability of analytical outputs

Approach to designing programming focused on shaping the project more than on understanding the problem

Limited utility of formal monitoring and evaluation data

Difficulty finding and applying tools for conducting conflict analyses

Undue influence of external target audiences in conflict analyses (power relations dominant)

Geopolitical imperatives*

Lack of a common language to understand the conflict*

Insensitivity to the context or culture*

Cultural bias and lack of transferability of evidence*

Prohibitive cultural expectations around the role of women*

Misaligned incentives*

Intrinsic conceptual value of peacebuilding understood

Emphasis on evidence in design and as starting point for adaptation

Rigorous analysis, presence of expert capacity on design teams

Flexible funding mechanisms

Strong knowledge of management practices

A focus on context and systems

Local knowledge made central to design decisions

Informal means of generating and sharing evidence sought

Relevance of perspectives developed in other contexts and through other approaches understood

Use of existing evidence in design

Use of cross-sector partnering for evidence generation and design of integrated programming

Positive mentality of adaptive management

Availability of local women’s perspectives on evidence generation, analysis, and use

*Barrier discussed only in the roundtables.

First, multiple reinforcing structural issues negatively affect the way conflict analysis evidence is used in project design. Roundtable and case study participants said that such structural issues were “beyond the awareness or influence of those doing either the evidence generation or programming design.” Responsibility for identifying and mitigating these issues is often attributed to organizational leaders, who in turn point to the structural limitations and negative impacts of funding mechanisms and the wider policy environment.

Second, certain cultures and mentalities enable the conditions for evidence-informed practice and effective design, while others get in the way or actively undermine the pursuit of same. Though the culture espoused by the leadership is important, related factors were identified by roundtable participants and USIP team interviewees as more widely relevant across levels and functions and thus part of wider organizational culture and practice.

Third, a set of operational factors was identified as relevant to design efforts making effective use of evidence. Access to tools, a team’s capacity, the availability of resources to navigate changing conflict dynamics, and the like were raised as technical factors that should be largely manageable with the know-how available within an organization, if the culture and an empowering organizational structure are in place.

These three themes are adumbrated in the rest of this section on the challenges to and opportunities associated with evidence-informed practices. In the Recommendations section they are explored more deeply as interrelated and interdependent conditions of a systems approach necessary to make more effective use of evidence in peacebuilding project design.

THE SHIFT TO EVIDENCE-INFORMED PRACTICE

Evidence-based practices have received considerable attention in the peacebuilding literature, which appears increasingly insistent on adopting such practices in project design, adaptive implementation, and evaluation. However, the tactical application of evidence-based practices or evidenced-informed practices has received far less attention. Setting aside for now the question of whether organizations choose or are incentivized to adopt evidence-informed practices by their funders, the discussion that follows focuses on how organizations are using evidence and the challenges they face in doing so.

Challenge 1: Creating an organizational mindset responsive to the use of evidence-informed practices. Can evidence-informed practices be adopted by organizations whose top leadership lacks a commitment to evidence use, learning, and adaptive management?⁴⁷ Though organizations may tout the need for or the benefits of evidence-informed practices, they are inclined to pay less attention to identifying factors that might facilitate the adoption of such practices. In this respect, J-Pal’s approach of working with state agencies seems very important. In what it describes as creating a culture of evidence, it believes that support from the leadership for evidence-informed practices is crucial in getting an organizational commitment to using evidence.⁴⁸ The lack of leadership support came out strongly in the direct research methods as a barrier to the effective use of evidence. INGO roundtable participants in particular expressed strong criticism of organizational leadership focused predominantly on donor policies and the management of reputational risks and opportunities, to the detriment of a commitment to evidence-informed practices.

In the event top leadership does firmly support the use of evidence-informed practices, a trickle-down effect may result in a broader organizational commitment. A caveat is that it is not enough for leadership alone to support such practices. If evidence-informed practices are a low priority for anyone in the organization or for donors, there is a good chance that evidence gathering and use might suffer. Carter et al. suggest that some kind of reward mechanism might help to motivate staff.⁴⁹ Rewards such as promotions, access to funds or training, and greater ownership over design choices through the use of evidence are ways in which organizations might strengthen a commitment to evidence-informed practices among their employees.

Similarly, the United Nations Institute for Disarmament Research (UNIDR) discusses the need to empower employees by strengthening their capacities to use evidence-influenced practices while requiring the use of evidence in project planning and design.⁵⁰ Local peacebuilders placed significant emphasis on “having capacities to do a good conflict analysis and take the next step of looking at how that translates into design and implementation.” They raised the need for “foundational training and working more with those individuals to plan, lead, do conflict analysis [with the goal of] having more people in our organizations who can do this, rather than rely on a limited number of specialists.”

Challenge 2: Institutionalizing the adoption of evidence-informed practices. Ababou and Alzate reiterate the lessons offered by J-Pal and UNIDR when they propose that evidence-informed practices be institutionalized in an organization.⁵¹ By institutionalization they mean organizations must be intentional about how such practices are introduced and adopted. All of our methods showed that many organizations introduce the use of evidence-informed practices because of pressure from donor institutions and other international organizations. This creates an urgency that can come across as an imposition to adopting evidence-informed practices organization-wide. This in turn puts undue pressure on employees and may even demoralize them as their current practices, often based on an informal or experiential evidence base, are rendered ineffective. Instead, as J-Pal’s experiences suggest, the use of evidence-informed practices must be introduced incrementally within an organization. This means that organizations seeking to adopt such practices should (1) make clear that practitioners do not need to gather all the evidence at the very outset; (2) support data gathering occurring simultaneously with project development and implementation, mainly because changing contexts will generate novel and more relevant evidence; (3) accept that if evidence points to a complete overhaul of a project, it might not be possible in terms of time, resources, and donor-established parameters; and (4) evaluate the evidence used for bias and error as virtually no evidence is credible enough to outweigh or eliminate all previous knowledge or practice. As one INGO participant said, “Analysis needs to be embedded into the peace practice itself. We sometimes describe it as a peacebuilding intervention if done well.” Likewise, the USIP teams discussed change as a constant in highly fragile contexts and thus the inevitability of new evidence appearing nearly continuously.

The incremental application of evidence not only is more feasible for practitioners, it is also necessary to avoid making sudden changes to project design and implementation, which could have a negative impact on the recipients of peacebuilding interventions.⁵² The incremental approach is endorsed by the UNIDR, which recommends that any new evidence gathered should be used to modify rather than replace existing practices.⁵³ To this point, the roundtable participants presented a more nuanced understanding of how existing practices could be modified and reshaped. Referring to the Do No Harm Framework, which encourages a dynamic approach to project design and redesign or adaptation to accommodate new information, they discussed the evaluation of existing practices as an ongoing process that entails both adding new activities and halting or modifying other activities if the evidence indicates they are negatively affecting the peace and conflict dynamics.

The process of institutionalizing practices mandates an internal review to assess whether adopting evidence-informed practices is about becoming more of a learning organization or a more accountable organization. For example, while many organizations describe learning as a key goal of using evidence-informed practices, donor organizations often emphasize using evidence as a way of staying accountable.⁵⁴

Challenge 3: Education and training in evidence-informed practices. Evidence-informed practices are time-consuming. Owing to the nature of peacebuilding work, organizations often find it difficult to invest the resources, time, and space to build the necessary tools, skills, and processes needed to engage evidence-informed practices. Institutionalizing evidence-informed practices and creating a culture supportive of their use requires an investment in training and educating staff. Data gathering might be assigned to a separate team that is not the project team. Coordination between teams is important, but evidence gathering should not fall to project teams as they are often overly burdened and stretched for time.⁵⁵ USIP interviewees did note that the process of gathering evidence and analyzing the context with partners contributes to programmatic effectiveness because it strengthens the relationship with those partners, so there are situations in which evidence gathering can be the work of the project team and reinforces its other work.

Further, there needs to be recognition that some forms of data gathering are technical processes requiring individuals with unique skill sets. This means that organizations need to spend time and resources to prepare their employees. Peacebuilding organizations often assume their employees will simply adapt to the new practices without formal training or technical assistance.⁵⁶ Hansen adds that gathering and interpreting evidence takes time and resources away from “actual implementation of the project.” This seems to be especially true if evidence-informed practices are imposed on an organization-wide basis. Trained and knowledgeable employees who use evidence-informed practices produce more high-quality research, which will help bolster the validity and relevance of the evidence. Further, if such research is based on practice, it will avoid the risks of gathering evidence that has less relevance in practice.⁵⁷

Challenge 4: Creating and building a network of organizations engaged in evidence-informed practices. Evidence gathering requires a lot of investment. Peacebuilding organizations that operate in silos are disadvantaged as they lose the opportunity to learn from shared evidence. Moreover, as noted earlier in this report, evidence can be generalizable from one context to another once the threshold of validated transferability has been reached, and therefore maintaining a database of relevant and reliable evidence might reduce the need to gather evidence for every project in every location. However, using generalized evidence does increase the risk of missing some context-specific and culturally sensitive information.⁵⁸

Creating networks and sharing knowledge and information helps organizations overcome other challenges, such as the following:

Finding the right evidence at the right time. This is probably one of the most important challenges for project staff. If the right evidence is not available at the right time, there is a strong possibility of failure or harm resulting from implementation. When evidence is shared, there is greater possibility of accessing the right information at the right time.⁵⁹

Selective perception or knowledge. Even people trained and skilled in evidence-gathering methods can have biases. Biases and previous knowledge may become barriers to gathering new data or information. Having a network of organizations and people from different contexts and experiences will help practitioners overcome these biases.⁶⁰

Gaps in evidence. Not all evidence required for a project may be available, leading to gaps that might end up being filled by intuition and experience.

Requiring a long-term presence and involvement. To gather data, one needs to be present and involved for a long period of time to gain the trust and confidence of local communities. Organizations often are unable to commit for a longer period, primarily because of budget constraints. A network may alleviate these challenges.⁶¹

Catholic Relief Services’ Connector program is a strong example of individuals across the divide being brought together by providing incentives to share knowledge and information. Using the example of the Philippines, Catholic Relief Services reports on the increased social relations and cooperation between formerly divided groups. The Connector program builds on the conflict analysis done to understand the conflict context and shares the analysis while also allowing adaptations to it by including local voices. The design and execution of the Connector program to address, for example, land issues in the Philippines results in new shared knowledge, which then informs the next iteration of conflict analysis and adaptations.⁶²

Evidence-informed practices, while more established in other fields, are still evolving in the peacebuilding field. Hansen reviews the use of evidence-informed practices in social work to assess how valuable they might be in conflict management. He suggests that the use of evidence-informed practices may lead to a hierarchy in which information and knowledge are privileged over values, objectivity is privileged over subjective meanings, and technical skills are privileged over relationship building. With such considerations in mind, he further opines that the field of conflict management should emphasize practice-based research. Using research that comes from local sources and working together with the community can help humanize evidence, he says, and make it more relevant and robust.⁶³

Taken as a whole, the findings point toward the structural, cultural, and operational preconditions that enhance the chances of increasing the effectiveness of evidence-informed project design—in other words, a systems approach to how evidence is used would be desirable. The CDA team’s translation of these findings into recommendations was heavily informed by the valuable insights of the USIP teams interviewed for the case study and the experts and practitioners who participated in the roundtable sessions. The recommendations that follow are further validated by the metasynthesis findings—both by what exists in the literature and by the notable gaps.

Recommendations

This section provides recommendations for organizations interested in strengthening their use of relevant and effective evidence in peacebuilding project design. The main recommendations are specifically relevant to USIP for the immediate purpose of developing the PCAT and related guidance for the field teams. The recommendations are also relevant to other organizations that seek to adopt evidence-informed practices when designing, implementing, and learning from peacebuilding projects. The major recommendations are followed by a list of additional recommendations drawn from the comments of participants in the direct research methods of this evidence review. On a broader scale, the overarching theme of incorporating a systems perspective is important for donors, policymakers, and academic researchers, who directly contribute to making evidence relevant and useful for project design.

SUMMARY OF RECOMMENDATIONS

The following recommendations are directed toward organizations seeking to make effective use of evidence in peacebuilding project design and include specific information intended for USIP’s immediate purpose of developing the PCAT:

Steps toward Using Evidence-Informed Practices in Peacebuilding Project Design

1. Conduct an organizational audit and set standards for evidence use in project design.

2. Reframe conflict analysis from a tool to an approach.

3. Invest in the structural, cultural, and operational conditions of an enabling system to support effective evidence-informed practices in peacebuilding programming.

4. Consider four areas for further research: more fully integrating local knowledge; intentionally connecting evidence generation with monitoring and evaluation efforts; scrutinizing the literature on conflict, violence, democracy, and governance; and becoming familiar with donors’ use of evidence.

The evidence review uncovered many factors that could contribute to the effective use of relevant evidence in peacebuilding project design. Any one of those factors alone, such as the use of an established conflict analysis tool, can contribute to effectiveness. Yet study participants focused more on the need for individual tools and practices to be supported by a wider system of structural, cultural, and operational conditions to fully enable the design of (and, by extension, the implementation and adaptation of and learning from) evidence-informed practices in peacebuilding. Table 6 summarizes the conditions and identifies the organizational “champions” who need to act.

Inherent in a systems approach of any kind is a recognition of the influence and interdependency of factors. The findings resulting from operationalizing the three research methods employed in this evidence review underscore that structural conditions are foundational. Without a clear articulation of the value and strategic focus of evidence-informed practices in peacebuilding project design, organizations struggle to implement and sustain investments in cultural and operational conditions. In keeping with the emphasis on values in a systems approach, two main recommendations for USIP are provided below, followed by a number of more specific factors and related recommendations for each of the three types of condition—structural, cultural, and operational—requisite to building an enabling system.

Recommendation 1. Conduct an organizational audit and set standards for evidence use in project design. As USIP embarks on the development of the PCAT, an organizational audit to determine what components and factors are in place and where further investment might make a significant difference to the effectiveness of the PCAT is warranted. The USIP case study provides a strong start to this process, which would benefit from the addition of perspectives from more teams with divers experience in using evidence in project design, from other USIP teams, and from USIP’s external partners with perspectives on the questions below. The resulting picture could (1) inform general strategic decision-making and resource investment for USIP and (2) define the minimum standard preconditions for a USIP country or regional office, or other team engaged in conflict analysis and project design. With such a standard in place, USIP teams would have an early indication of the answer to the question, “How can I take this evidence and use it to help me make more informed decisions about my project?” Teams would be better able to evaluate whether embarking on conflict assessment or another form of analysis had a strong likelihood of influencing effective project design or whether doing so would be a poor investment of resources or possibly risk doing harm. Further, adhering to such a standard could facilitate more consistent budgeting and professional development investment in existing USIP teams and ease the rapid startup of new country offices.

Table 6. Conditions of an Enabling System to Support Effective Evidence-Informed Practices in Peacebuilding Programming

Condition

Description relevant to enabling effective EIPs in peacebuilding project design

Organizational champions

Structural

Changeable factors that require organizational investment and capacity prioritized by a clear articulation of value and strategic focus by leadership; for example, creating an agency-level strategy for being evidence-based or evidence-informed.

Leadership, to set the vision and spur resource investment

Cultural

Changeable factors that demonstrate a widely shared disposition related to mission and purpose as reflected in a culture of learning. Such factors often address power relations within and among partners, for example, specific investment in the capacity of people to be equipped for evidence generation, analysis, and application, as well as accountability of and support from leaders and managers of colleagues responsible for creating and using evidence.

Leadership, to model a growth-oriented mindset⁶⁴

Operational

Practical systems, tools, and other resources that exist or can be created to facilitate, implement, and learn from use of evidence. This may require a shift in how existing ways of working or technologies are employed; for example, tools may be needed or repurposed to translate conflict analysis evidence into the design of monitoring and evaluation of phases of new project design, or guidance may be needed in the use of monitoring and evaluation data derived from existing projects as evidence for new project design.

Directors and managers, to prioritize investment, accept accountability for use, and create means for innovating based on learning from doing

When designing an organizational audit to advance the effective use of evidence in peacebuilding project design, any organization could adapt the following questions.⁶⁵ If piloting this approach or if there is a time-sensitive need by a specific country office or other team, the most relevant questions could be tailored as a “quick audit” and still yield actionable information.

Questions useful in designing an organizational audit:

What strong structural, cultural, and operational elements exist within our organization or team that provide a foundation for our project design goals and needs?

What structural areas can be strengthened, and which ones need more investment to make clear about our institutional (or team) commitment to evidence?

Where is there existing political will within the organization (or team) to address structural or cultural areas in need of attention for the effective use of evidence in project design? Where is the political will insufficient, and how can it be strengthened to influence change?

What cultural practices or champions exist that could influence good practices for using evidence in project design to become the organizational (or team) norm?

What operational tools and systems exist that could provide a foundation for amplifying existing practices or developing new ones to facilitate the effective use of evidence in design?

What change would have the greatest impact on more effectively using evidence in peacebuilding project design? How can investment be leveraged to achieve this end?

What relevant partnerships has the organization (or team) already established? Are there strategic partnerships related to using evidence in peacebuilding project design that could be prioritized to help address operational gaps or create complementary opportunities?

These questions form a systems-thinking perspective that could form the basis of an exercise in applying USIP’s expanding capacity for systems analysis. Notable current efforts in this direction include a focused evidence review (which was conducted concurrently with the present evidence review) and a new USIP training course, “Systems Approaches to the Management of Reform and Peacebuilding Projects in Conflict-Affected Environments.”⁶⁶

Recommendation 2. Reframe conflict analysis as an approach rather than a tool. Based on the findings of the present evidence review, particularly those that emerged from the primary data collection methods used in the USIP case study and the expert-practitioner roundtables, we recommend that USIP consider revisioning the development of a practical conflict analysis tool to support a practical conflict analysis approach. Like other organizational approaches, a programmatic conflict analysis approach could consist of multiple elements designed to reinforce each other and to be used modularly as needed. Potential elements, informed by this evidence review, could include the following:

Build on USIP’s Annotated Guide to Resources for Conflict Analysis to develop tools to analyze how types of evidence inform principled choices in design decision-making, including (1) tools that broaden the spectrum of locally and globally derived evidence and aid in effectively combining evidence from different sources, and (2) tools to tease out substantive political information to help explore power dynamics in specific contexts and between donors and implementers.

Provision of robust guidance in the use of the suite of USIP tools, and tailored recommendations for the use of complementary tools not developed by USIP.

Orientation to use of available tools and to choosing the right tool or mix of tools for a given need and context.

Training in the use of tools, including participatory action research (PAR) or political economy analysis (PEA) methods (or both) as relevant to working with local partners and stakeholders.

Training in analyzing evidence generated by the tools and through other sources, including PAR or PEA methods, to center local actors in meaning making.

Training in the effective use of evidence for USIP peacebuilding programming, including means for evidence-generating teams to work collaboratively with project design teams.

Awareness sessions or training for leaders and managers about their role and the organizational policies and structures available to equip teams.

Facility for ongoing or as-needed technical support to country teams and others planning for conflict assessment, strengthening systems and capacities for analysis, and implementing and learning from evidence-informed practices in project design.

Practical matters for USIP to consider in developing a programmatic conflict analysis tool and/or approach include the following:

Plan for a pilot phase. Seek feedback about the conditions in which parts of the tool or approach are more relevant and feasible than others and incorporate the feedback into the final version. Consider aligning some of the pilot locations with priority countries of the Global Fragility Act.⁶⁷

When using the analysis tool and/or approach in the pilot phase, ensure that it is tested in various global contexts and incorporates the perspectives of local community, international, and donor groups. Include evaluation of whether the goal of broadening the spectrum of locally and globally derived evidence is actually possible and is informing principled decision-making.

Establish a community of practice for pilot users of the analysis tool and/or approach and for a wider community of practitioners. Participants in the roundtables appreciated the engagement with stakeholders the evidence review was able to convene and noted excitement about continuing to engage with the group and others on topics of evidence generation and use. Specifically, some participants identified the format as a way to move from “one-off” conversations to a way to more consistently share lessons, brainstorm, and prioritize sector wide actions.

Identify opportunities to incorporate USIP’s Gender Inclusive Framework and Theory to facilitate the integration of gender analysis into project design.

Connect with other conflict assessment and analysis frameworks more broadly and coordinate with related framework development efforts underway, such as USAID’s forthcoming update to its Conflict Assessment Framework (CAF 3.0).

We make this recommendation to USIP independently of whether the Institute chooses to conduct an organizational audit or set minimum standards, as discussed above.

Recommendation 3. Invest in structural, cultural, and operational conditions of an enabling system to support the use of effective evidence-informed practices in peacebuilding programming. Participants in the roundtables and the USIP team interviewees shared several priority points and insights that have practical implications for putting in place the structural, cultural, and operational conditions of an enabling system to support the use of effective evidence-informed practices in peacebuilding programming (see table 6). With reference again to a systems perspective, appropriate structural conditions enable teams to act on cultural and operational factors. Similarly, effective operational and cultural factors can reinforce the organization’s vision and structural commitments.

STRUCTURAL FACTORS

Invest in sustained context expertise and building local trust. Without a vision that prioritizes a clear and deep understanding of context, organizations risk negative impacts from how they use evidence. With in-country presence “practice iterative collection and use of information obtained through local interactions.” Otherwise, “invest to build relationships with partners who have good access in areas of concern” and “prioritize gaining their trust by understanding and investing in what is important to them.” (USIP case study interviewee)

Be systems thinkers about global dynamics. “Conflict and peacebuilding efforts do not necessarily happen exclusively locally” (INGO roundtable participant). Make systems thinking a valued leadership trait, and build the analysis skills of staff to understand how regional economic dynamics or a global pandemic can exacerbate tensions or unite people around a common cause, so that staff can use evidence to design projects that factor in both opportunities and contingencies.

Establish an organizational vision for how and why evidence is used. Commit to adopting a Do No Harm lens on an organization-wide basis and use the Do No Harm Framework to understand the implications of using (or not using) evidence, including how ignoring available evidence might increase the risk of unintended harm.

Be realistic about the limits of evidence. Create space among leaders and staff for honesty about the changes required to achieve the level and scope of analysis and develop a common understanding of the limitations of transferability beyond an immediate context.

Cultivate a mentality for broad interpretation of innovation. There is vast scope for innovations in collecting and using evidence/data, particularly to facilitate analysis with local actors. Proactive curiosity and an understanding of innovation should include but not be limited to use of new technologies. Empower staff with the latitude to make choices and try new things, including encouraging “learning from failure.” (INGO roundtable participant)

CULTURAL FACTORS

Build a culture of ongoing learning. “Evidence should always see the paradoxes that exist in reality” (local peacebuilder roundtable participant). To keep pace with rapidly changing contexts and the complex human and political dimensions of conflict-affected environments, seek evidence from diverse sources that colleagues can connect with easily and far more regularly than quarterly monitoring and evaluation activities or annual conflict assessments. Prioritize creating “a trusted network of partners and staff. They are the best asset that we have.” (USIP case study interviewee)

When considering the legitimacy and interpretation of evidence, develop a common understanding of the role of power in the kind of information acquired. Power relations shape what kind of evidence is pursued, who generates it, and how and by whom it is analyzed. Onboarding of any staff helping generate or use evidence should include conversations about how legitimacy is conveyed and access granted, and how evidence-informed design reflects these dynamics. Leaders of analysis and design teams should make this a lens for approach and analysis, and support efforts to build it into organizational systems, including professional development investment in analysis capacity so that interpretation of evidence into project design can be inclusive.

Align incentive structures and objectives. Develop incentives related to advancement and investment to motivate most project designers and implementers to conduct conflict analyses and other forms of evidence generation with the “goal of genuinely learning from and changing based on the analysis.” (INGO roundtable participant) Transforming this dynamic could increase a sense of safety, to encourage staff to be open to evidence that says what is being done is not working or doing harm, and to explore the potential for creative, impactful design.

Embrace team-wide accountability for use of evidence. Use project cycle phases to motivate investment in and use of conflict assessments by a wide group of actors (leadership, programming, operations, partners). Build evidence-use check-ins into team meetings and “incorporate adaptive management into MEL [monitoring, evaluation, learning] structures.” (INGO roundtable participant)

OPERATIONAL FACTORS

Create formal and informal channels for capturing local knowledge. Identify how partners and other effective and accessible sources of local knowledge most readily broadcast evidence and critical analysis, including through such venues as periodicals, radio broadcasts, and social media. Designate staff to monitor. Develop direct channels, such as private messaging, and build trust through consistent and trustworthy use of information. Apply best practices of “community feedback and accountability mechanisms most relevant to the context.” (INGO roundtable participant) Pay special attention to how global knowledge might be interacted with local knowledge and practices. Develop a process for maintaining open lines of communication between global and local stakeholders.

Create evidence-sharing systems for use by staff and partners. Jointly develop a common system for sharing and disseminating evidence to address timely access to the right evidence, problems resulting from lack of evidence, and the chronic tendency to try to generate original evidence on already well-studied topics or in known contexts. The potential to do harm is significant if actors replicate the mistakes of others or design programming based on incomplete data or miss windows of opportunity to act with confidence. Effective use depends on simplicity and trustworthiness, so monitor regularity of use and adapt if needed.

Carefully consider evidence through triangulation and verification. Confirming conflict assessment and analysis data is a “huge challenge,” according to both USIP interviewees and INGO roundtable participants. Organizations that have been working in a context for some time can provide guidance on the extent of and access to relevant data. Identify and build reciprocal relationships with other organizations in the affected context and with project implementers, researchers, government actors, and journalists. Carefully considered evidence helps guide us towards fewer interventions that backfire. Implementing adaptive management for MEL approaches supports ongoing triangulation and decision-making.

Ensure data security. Plan carefully for how to protect conflict analysis and other sensitive data before beginning data collection. The extent to which data security is ensured will have an impact on the quality of evidence gathered and on community trust and organizational reputation. “With the right professional development, staff and partners in any function can contribute” to data security or have specific responsibilities. Be transparent with all partners and evidence sources about how data will be used and kept secure: “We need to close the loop whenever we do conflict analysis.” (Local peacebuilder roundtable participant)

Recommendation 4. Four areas for further research. Four areas for further research merit specific attention. As we note in the Introduction to this evidence review, two of the original subquestions identified in the Inception Protocol could not be sufficiently investigated within the scope of our research. The process of conducting the evidence review underscored the importance of these themes for USIP and beyond; thus, the first two areas for further research are:

How do projects generate or capture local knowledge and local understandings? How is this knowledge integrated into project design and adaptations?
While we discuss the role of local knowledge in the USIP case study and draw on important factors (for and against) mentioned during the roundtables and identified in the metasynthesis, there is a need to shine a brighter light on these questions so that practitioners and policymakers can become more knowledgeable about different types of locally and globally derived evidence and make principled choices of evidence in decision-making. For example, participatory action research is rarely used in peacebuilding project design; could it be used more frequently? What could political economy analysis contribute to helping peacebuilders capture, triangulate, and evaluate sources of local knowledge?

How do projects collect rigorous monitoring and evaluation data to contribute to project design and adaptations?
The peacebuilding field has developed a major focus on monitoring and evaluation through such projects as the Peacebuilding Evaluation Consortium, such products as the 3ie evidence maps, and various platforms, including the Eirene Peacebuilding Database of monitoring indicators and DM&E for Peace as a community of practice. A specific research effort could be carried out to (1) learn in more depth from the practical experiences of USIP teams about how their monitoring and evaluation data have informed project adaptations or new project design and (2) inform USIP about the most effective ways to connect project or country teams with the increasingly available sectorwide resources.

A third area of further research could focus on the political content of different conflict analysis approaches and of peacebuilding evidence more generally. The conceptualization, gathering, and application of evidence constitute a political process that is rife with issues of power at every step, from where evidence is generated and by whom to who controls interpretation and analysis to donor-implementer dynamics. Political economy analysis, as a structured approach to examining power dynamics and economic and social forces, could provide a central framing for such further research and facilitate comparing this evidence review with the literature on conflict, violence, democracy, and governance—all areas that, like peacebuilding, present challenges to documenting results and making evidence-based decisions. The findings would be practical for designing complex programming that more rigorously responds and adapts to dynamic conflict-affected contexts and other systems approaches to using evidence-informed practices, by USIP and other practitioners; an example is the current effort to approach implementation of the GFA in a way that is more aligned with the lessons and practices of thinking and working politically. For donors such as USAID that are attempting to adopt a thinking and working politically mindset and the collaborating, learning, and adapting practices developed by USAID to improve development practices, such research could further contribute to improving the effectiveness and sustainability of such efforts. This intentional alignment might also highlight further policy and targeted donor investment to advance evidence-informed practices across interdependent sectors of programming.

Finally, a fourth area of research focus could entail a deeper investigation of the barriers to and opportunities attending donor use of evidence in creating funding opportunities for peacebuilding projects, with an emphasis on political will. For USIP’s purposes and for the wider peacebuilding sector, a significant benefit of incorporating original research in this evidence review was the chance to hear donors and expert-practitioners alike express interest in developing stronger partnerships to facilitate funding decisions that draw on the peacebuilding evidence base. Just as for the three previous recommendations of areas warranting further research, improving donor knowledge would answer a need in the democracy and governance sector, where donors generally lack the political will to implement even well-supported findings. Making it easier for donors to use evidence could benefit donors’ decision-making on several levels: in setting funding priorities and establishing mechanisms; in supporting policy efforts to build support for peacebuilding; in engaging with others on the effective use of evidence in peacebuilding; and in enhancing efforts among donors in the peacebuilding and governance arenas, such as through USAID, Sida, or the OECD’s Development Assistance Committee.

Appendices

APPENDIX 1. METASYNTHESIS METHODOLOGY

The information in this appendix follows the outline of the methodology in the Inception Protocol and reflects how the metasynthesis was actually planned and conducted.

CDA conducted a metasynthesis of published and unpublished materials and studies to answer the research questions. The metasynthesis design and analysis adopted are detailed below.

DATA SOURCES

The CDA team chose to focus the metasynthesis on identifying themes and patterns and their interrelationships to each other. Reinterpretation was done where possible and necessary, but with the goal of reorganizing and recategorizing according to identified themes. Further, we highlighted and analyzed the evolution of concepts and ideas by key authors over the years. At the same time, we examined key themes from the perspective of different study groups or authors.

Because of the limited availability of studies on the use of evidence in peacebuilding project design, we included the following materials and studies:

Conceptual and theoretical foundational materials and studies that include empirical methods highlighting the need, importance, latest trends in using evidence for project design and adaptation. These include:

Articles on peacebuilding and related subject matter accessed through various electronic databases (PRIO, CIAO, JSTOR, Academic Search Elite, Sage Premier Journals, Google Scholar).

Resources (reports, studies, conference proceedings, how-to guidance materials) available through leading peacebuilding organizations (for example, Alliance for Peacebuilding, Search for Common Ground, Mercy Corps, DM&E for Peacebuilding, Peace Direct, Catholic Relief Services, USIP, CDA), networks, sector initiatives, government organizations (USAID), development banks, and the private sector. Manuals developed to guide practice and project design were reviewed to support conceptual data and informed our roundtable conversations.

Web-based search using such keywords and terms as “evidence-based practice,” plus “peacebuilding/development,” “project adaptation and peacebuilding,” “adaptive peacebuilding,” “conflict transformation/resolution/management,” “building peace,” and the like. The search terms came from the key questions formulated for this evidence review.

Minutes or other records of project staff meetings, planning retreats, design workshops, and the like, documenting the thought processes of teams in position to use evidence in project design and adaptation. These records were solicited by directly contacting key expert-practitioners invited to the roundtable discussions.

Project proposals selected by USAID and categorized as exemplary or mediocre and as strong or weak in their use of evidence in project design and adaptation.

Table A1.1. Metasynthesis Sample Sources

Type

Number of resources

Academic (conceptual and empirical)

16

US government (primarily USAID) and United Nations

5

NGOs (primarily INGOs), development banks, and private foundations

15

Blogs, websites, and conference reports

7

Total

43

Table A1.1 provides an overview of the number of materials and studies we reviewed from each of the different types of sources.

DATA SELECTION AND ELIGIBILITY CRITERIA

In our preliminary search, all materials and studies related to evidence-based peacebuilding design and adaptive peacebuilding were gathered.

Because materials and studies on this subject are scarce, we started broadly and used our initial searches to narrow down search terms and identify articles that emphasized evidence-based design and adaptation.

A quick review of the abstract helped us select the materials that required a more in-depth review.

Studies that were empirical were preferred over ones that were purely conceptual.

We searched for a broad period of the last ten years, although we did not limit ourselves to a time period for relevant studies.

When we identified an article or study that we thought was highly relevant to the present evidence review, we studied the bibliography to identify additional sources.

We kept the focus on materials and studies that referred to the use and relevance of evidence in project design and implementation.

Unpublished materials and studies were subject to similar selection criteria.

All three CDA team members were involved in the selection process, although Pushpa Iyer took the lead in conducting initial searches and narrowing down the materials and studies that needed to be reviewed jointly. A consultative process was adopted to identify the materials and studies that informed our metasynthesis. The consultative process helped us make decisions that did not fall into neat “include” or “do-not-include” buckets.

LIMITATIONS

We did not target non-English-language materials or studies. We understand that this exclusion in all probability resulted in the omission of data acquired by non-English-speaking communities. One may also accuse us of reproducing the ideas promulgated by colonizers.⁶⁸ We acknowledge this limitation. To overcome this limitation, we present a broad geographic distribution of US government, NGO, and empirical materials reviewed. We also hope that this initial review will lead to more in-depth study of how evidence-based practices are used globally.

DATA SYNTHESIS

Our goal in conducting the metasynthesis was to integrate and interpret the data we collected by identifying patterns. Themes drawn from the data and findings are presented, together with recommendations. These patterns (themes) guide and provide a structure to our findings from the case studies and roundtable discussions.

Final list of keywords searched for in published and unpublished literature:

Peacebuilding + evidence

Development + evidence

Building peace + evidence

Evidence + project design/development + peacebuilding (international/global development, building peace)

Evidence + project planning + peacebuilding (international/global development, building peace)

Evidence + project adaptation + peacebuilding (international/global development, building peace)

Evidence + adaptive peacebuilding/management (international/global development, building peace)

Evidence + peacebuilding (international/global development, building peace) + challenges/barriers/failures/successes

Evidence + adaptive peacebuilding/management (international/global development, building peace) + challenges/barriers/failures/successes

Evidence + peacebuilding (international/global development, building peace) + indicators

Evidence + adaptive peacebuilding/management (international/global development, building peace) + indicators

As alternative terms to peacebuilding only, cross-referenced with the above: conflict transformation/resolution/management, building peace

APPENDIX 2. SUMMARY OF CASE STUDY REPORT ON TWO USIP COUNTRY OFFICES’ USE OF EVIDENCE IN PROJECT DESIGN

Following is an excerpt from the full case study report prepared by CDA for USIP as part of this evidence review. It has been lightly edited for style.

PURPOSE AND OVERVIEW

The purpose of the case study was to generate insights into the use of evidence in project design processes in two USIP country offices as part of the broader Conflict Analysis Evidence Review. The case study was informed by interviews with members of the two country offices and a review of selected project proposals and conflict analyses.

The case study affords an opportunity for CDA and USIP’s Learning, Evaluation, and Research team to ground the findings of the metasynthesis and roundtable discussions in the experiences of USIP teams on the ground. Additionally, by exploring what, when, and how evidence is used in two USIP country offices, as well as the challenges and barriers faced in using such evidence, the case study highlights and contextualizes some of the major issues attending the use of evidence in project design in the broader peacebuilding field.

This study is not an assessment of capacity or systems within the two country teams and is not an evaluation of their ongoing efforts. The findings are intended to inform considerations for the broader evidence review.

Both country programs are managed by small teams operating in rapidly changing contexts. Country 1 is undergoing a political transition, with ongoing tension and sporadic outbreaks of violence. Country 2 is experiencing active armed conflict in several regions. (The names of the countries have been redacted from this report.) These countries were selected on the basis of the reported differences in their experiences and of analysis outputs derived through an internal USIP conflict analysis mapping exercise.

METHODOLOGY

Lines of Inquiry

CDA adapted a subset of the evidence review research questions to form the lines of inquiry for the case study. The questions were tailored to reflect practices at USIP rather than within the broader peacebuilding field.

1. What constitutes evidence in peacebuilding program design, implementation, and evaluation at USIP?

2. How do USIP programs utilize existing research and evidence for project design and adaptations to design?

a. How do USIP programs generate or capture local knowledge and local understandings? How is this knowledge integrated into project design and adaptations?

b. How do USIP programs collect rigorous monitoring and evaluation data to contribute to project design and adaptations?

c. How is existing evidence integrated with local knowledge and the results of monitoring and evaluation to contribute to evidence-based design and adaptation at USIP?

3. What are the barriers to using evidence in project design at USIP?⁶⁹

Data Sources and Analytical Methods

CDA conducted key informant interviews with four team members on team 1 and two team members on team 2, all of whom were actively involved in conducting conflict analysis and/or project design processes in recent years. USIP selected the country offices based on a broader internal conflict analysis mapping exercise indicating that the two teams’ experiences with integrating evidence into project design differed across two key dimensions: though they shared intentions to translate the conflict analysis into a programmatic effort, how they accomplished the task differed in the time and other resources (including partnerships) available to conduct the analysis and the ability of the team to articulate how the evidence gathered became part of the programmatic design. In line with the agreed-on key informant recruitment strategy, CDA asked respondents whether there were other individuals who should be interviewed, but others identified from team 2 were unavailable for interview. To complement the interview data, CDA reviewed conflict analysis and assessment reports and project proposals from the country offices.

CDA synthesized interview data from each country office using thematic analysis to identify patterns. Drawing on the interview data and document review findings, CDA developed a picture of how each team gathers and uses evidence. CDA compared the two teams and examined differences, along with the factors that contributed to each team’s capability to use evidence.

Limitations and Methodological Considerations

The case study analysis was based on information that was self-reported by each country office’s team members. It is possible that team members were motivated to paint their efforts as favorably as possible or otherwise color the information they shared. CDA sought to mitigate this potential bias by using a robust interview protocol, including the use of questions that elicited recall of the tangible processes and steps undertaken. CDA interpreted information on tangible processes and steps taken, along with more subjective reflections and perceptions shared by interviewees. A second limitation was the small sample size of just two country teams. A third limitation was the focus on two teams that have used evidence to inform project design and the resulting lack of input from teams that have not.

SUMMARY OF TOP-LEVEL FINDINGS

In both country offices, teams rely on local knowledge to design effective programming while facing limitations with respect to access, capacity, and time available to collect and apply more robust evidence.

This case study found that for the two USIP teams under study, the following observations apply:

Local knowledge is weighted heavily. Because of the direct relevance and applicability of local knowledge, the relative lack of evidence generated by formal monitoring and evaluation processes, and the specialized nature of recent formal conflict analyses and other existing evidence, USIP teams weigh local knowledge heavily in the design process. A key strength of this approach is that the consultative manner in which this information is generated reinforces USIP’s goals by building confidence among key actors and ensuring that programming is tailored to local priorities and conditions. The current realities of project design mean that teams are less able to draw on preexisting rigorous evidence of what works in their own and other contexts.

Eight types of information constitute the primary forms of evidence used. They are: input from partners and other relationships; consultations with communities, national stakeholders, or those affected by conflict; document reviews (including literature, other project documentation, and media coverage); conflict assessments and analyses; staff knowledge and expertise; input from other implementers; technical assessments; and project monitoring information and activity reports. Not all projects are informed by all eight forms of evidence.

Local knowledge and understandings are integrated effectively into formal and informal design processes. The teams access local knowledge and local understandings implicitly through hiring majority national staff and prioritizing input from local partners and stakeholders. Local knowledge is integrated into project design and adaptation in a conflict-sensitive manner that is aware of potential biases of information sources.

In most cases, project teams translate evidence into project designs through iterative processes of deliberation. USIP teams utilize research and evidence for design and adaptation primarily through ongoing processes of partner input and team discussions to understand the drivers of conflict or problems contributing to instability (often at the local level) and develop solutions to address them, in concert with developing an understanding donor interest (for interagency agreements and other externally funded programming).

Externally generated evidence is used to reinforce project proposals. Existing evidence (for example, externally generated evidence found in analytical reports) is drawn on to bolster the rationale for proposed solutions. Team members with backgrounds in peacebuilding and related fields draw on good practices they have encountered in prior experiences and in academic settings when designing projects.

Time constraints, rapidly changing dynamics, challenges with accessibility, and reliability of data are key barriers to the use of evidence. Barriers to using evidence from conflict assessments or monitoring and evaluation in project design include short windows of opportunity in most design processes, challenges with the reliability and accessibility of data, the lack of programmatic recommendations in USIP analytical outputs, the teams’ approaches designing programming that emphasizes the development of project ideas over obtaining a deep collective understanding of the problem, limited utility of formal monitoring and evaluation data, and the specificity of conflict analyses that limits broader applicability to programming in other regions or on other themes. USIP teams are actively seeking to improve their ability to collect rigorous monitoring and evaluation data to contribute to project design and adaptation. Both country offices currently rely on informally gathered information in the absence of useful formal monitoring and evaluation processes.

APPENDIX 3. EXPERT-PRACTITIONER ROUNDTABLE METHODOLOGY

The information in this appendix follows the outline of the methodology in the Inception Protocol and reflects how the roundtables were planned and conducted.

PURPOSE

The purpose of conducting roundtables comprising experts and practitioners was to elicit diverse perspectives and further corroboration from multiple data sources.

CDA and USIP jointly finalized the goals and process for three roundtable discussions with practitioners from outside USIP. As part of this process:

CDA proposed and finalized with USIP the agenda and criteria for participants in each of the roundtables.

CDA and USIP jointly identified and prioritized participants, set dates, and confirmed roles in preparation for and during the roundtables.

USIP hosted the roundtables, including managing the virtual platform and participant invitations, with CDA supplying support for messaging.

All roundtables were held November 9 or 10, 2021. Each roundtable lasted ninety-minutes. While the Inception Protocol targeted a minimum of five participants and a maximum of ten participants each, actual participation was higher, with a total of thirty-two participants (twenty women and twelve men) across three sessions.

ROUNDTABLE COMPOSITION

Invited participants of the three roundtables allowed for rich discussion among peers or near peers.

Roundtable 1: USAID, foundation, and academic experts

Roundtable 2: INGO technical and project design experts

Roundtable 3: technical and project design experts with experience at the local/ national NGO level, as well as sector-wide perspective

ROUNDTABLE METHOD

Earlier work of the evidence review informed how the roundtable sessions were designed. As proposed in the Inception Protocol, sessions were structured using a force field analysis exercise.⁷⁰ Force field analysis is a systems analysis tool that that seeks to elicit a comprehensive picture of barriers and enablers related to a common challenge, in this case the use of conflict assessment evidence in peacebuilding project design.

Along with other organizations in the peacebuilding field, CDA often makes use of force field analysis as the first step in designing a conflict system map or other maps that help make sense of the structure of, and behaviors and relationships emanating from, complex, highly dynamic contexts. CDA has used force field analysis to understand institution-specific barriers to systemic change and to generate solutions as to how these barriers can be addressed; it is currently using the approach on a sectorwide initiative with humanitarian actors about effective risk management practice. The goal is to facilitate reflection on the systemic and interconnected factors that either facilitate or block a desired change and to encourage a constructive process of outlining how to overcome or leverage those factors to support that change.

For this evidence review, the force field analysis approach efficiently drew out the experiences of key stakeholders for more effective use of evidence in project design and identified existing best practices or potential solutions to aid targeting future USIP tools and frameworks for more strategic, systematic shifts in ways of working.

Step 1. Factor analysis is a structured process to help participants reflect on their experience and expertise to identify the systemic and institutional factors enable and hinder a desired change. Learning highlights a broad range of perceptions, challenges, and priorities related to the use of conflict assessment evidence in project design. The process captures factors and barriers in the words of people directly involved in the use of evidence and generates ownership of the overall findings of the evidence review, contributing to future development of a community of practice making use of USIP’s toolkit initiative. The factor analysis framework for this phase is given in Table A3.1.

Step 2. Options generation is typically understood as the “troubleshooting” phase. Participants identify specific options and solutions, and the details necessary to carry out these solutions, for the barriers (“Factors Against”) identified in the factor analysis. Option generation asks participants to identify the main factors about the way evidence is used now that either prevent or negatively affect the quality of project design and adaptation.

In the context of conflict assessment evidence, we recognize that there are unlikely to be clear or definitive “solutions.” Rather, there are strategies, approaches, or innovations that may help mitigate or address key barriers, for which new tools may be helpful. The CDA team used this step to elicit both tried-and-tested approaches that participants had experienced themselves or had seen others try and their ideas for what might work to address any barriers. This was essentially a structured crowdsourcing exercise intended to generate a large pool of ideas. The evidence review team did not assess the effectiveness of the findings but rather incorporated them into the evidence review report to USIP to inform future tool development. The options generation framework for the roundtables is given Table A3.2.

Table A3.1. Factor Analysis Framework

Factor For

Factor Against

Key People

What about the way conflict assessment evidence is used now helps you in project design and adaptation?

Strong conflict assessment evidence used in project design and adaptation

By contrast, what about the way conflict assessment evidence is used now has a negative impact on effective project design and adaptation?

These are the individuals or organizations that are necessary to conflict assessment evidence use in design and adaptation.

Factors can include tangibles (e.g., policies, structures, processes) or intangibles (behaviors or attitudes) at your individual organization or across the sector.

Table A3.2. Options Generation Framework

Causes

Factor against

Effects

What has worked to address or mitigate this factor?

What might work to address or mitigate this factor?

What resources or support could help this strategy?

Priority 1 from above discussion

Priority 2, etc.

To maximize the time of roundtables, CDA prepared and provided to participants pre-reading material about the topic for discussion and process of FFAs. CDA also prepared a facilitator’s guide, slide deck, and workbook (initial template in steps 1 and 2 above) for use during the roundtables. CDA colleagues led the facilitation of all roundtables, with support from USIP colleagues for capturing notes in real-time.

ANALYSIS AND FOLLOW-UP

CDA and USIP staff met briefly after each roundtable to share initial reflections to inform a synthesis. CDA drafted a synthesis along with the USIP case study as an interim step leading to the evidence review report that also incorporates the metasynthesis of published and unpublished literature.

Roundtable participants were invited to take a short survey to provide feedback on the session they attended; they were also invited to contribute published and unpublished material for consideration in the metasynthesis. Many participants did contribute sources. Feedback was incorporated into the overall evidence review.

APPENDIX 4. BIBLIOGRAPHY

The bibliography includes the 44 sources that informed the metasynthesis. Figure A4.1 reflects the themes.

Figure A4.1. Themes of Sources Reviewed

SOURCES

Ababou, Abil, and David Alzate. 2021. “Developing an Evidence-Based Mindset: Fostering a Culture of Evidence-Based Policymaking through Research, Training, and Policy Engagements.” In Changing Mindsets to Realize the 2030 Agenda for Sustainable Development: How to promote new mindsets and behaviors in public institutions to implement the Sustainable Development Goals, under the direction of the UN Department of Economic and Social Affairs, Division for Public Institutions and Digital Government. New York: United Nations.

Adam, Jeroen. 2014. “The Boundaries of Evidence in Conflict Management and Peacebuilding.” The Asia Foundation, December 11, 2013.

Anderson, Mary B., and Lara Olson. 2003. Confronting War: Critical Lessons for Peace Practitioners. Cambridge, MA: CDA Collaborative Learning Projects.

Bates, Mary Ann, and Rachel Glennerster. 2017. “The Generalizability Puzzle.” Stanford Social Innovation Review 15 (3): 50–54.

Biglan, Anthony, and Terje Ogden. 2008. “The Evolution of Evidence-Based Practices.” European Journal of Behavior Analysis 9 (1): 81–95.

Blum, Andrew, and Melanie Kawano-Chiu. 2012. Proof of Concept: Learning from Nine Examples of Peacebuilding Evaluation. A Report on the 2011 Peacebuilding Evaluation Evidence Summit. Washington, DC: United States Institute of Peace and Alliance for Peacebuilding.

Brown, Annette N. 2019. “Some Good News in Evidence for Peacebuilding.” R&E Search for Evidence (blog), fhi360, June 25.

Carter, Samantha, Iqbal Dhaliwal, Julu Katticaran, Claudia Macias, and Claire Walsh. 2018. Creating a Culture of Evidence Use: Lessons from J-PAL’s Government Partnerships in Latin America. Cambridge, MA: Abdul Latif Jameel Poverty Action Lab, MIT.

CDA Collaborative Learning Projects. 2016. Designing Strategic Initiatives to Impact Conflict Systems: Systems Approaches to Peacebuilding. Cambridge, MA, December.

Church, Cheyanne, and Mark M. Rogers. 2006. Designing for Results: Integrating Monitoring and Evaluation in Conflict Transformation Programs. Washington, DC: Search for Common Ground.

Democracy International and USAID. 2019. Evaluation of the Conflict Assessment Framework 2.0. Washington, DC: United States Agency for International Development.

Eyben, Rosalind, Irene Guijt, Chris Roche, Cathy Shutt, and Brendan Whitty. 2013. The Politics of Evidence Conference Report. Brighton, UK: Institute of Development Studies.

Eyben, Rosalind, and Chris Roche. 2013. “The Political Implications of Evidence-Based Approaches.” From Poverty to Power (blog), Oxfam, January 22.

Ghate, Deborah. 2018. “Developing Theories of Change for Social Programmes: Co-producing Evidence-Supported Quality Improvement.” Palgrave Communications 4 (1): 90.

Hall, Ronald E. 2008. “From Generalist Approach to Evidence-Based Practice: The Evolution of Social Work Technology in the 21st Century.” Critical Social Work 9 (1).

Hansen, Toran. 2017. “Evidence-Based Conflict Management Practice.” Journal of Conflict Management 5 (1): 43–62.

IDinsight. “Evidence Toolkit: Supporting the Development Sector in Using Evidence.” www.evidencetoolkit.org.

Kamatsiko, Valarie Vat, Shamsia Ramadhan, Nell Bolton, and Robert Groelsema. 2021. Connector Projects Approach: Sustained Interactions for Strengthened Relationships and Collective Wellbeing. Baltimore, MD: Catholic Relief Services.

Kurtz, Jon, and Greg Scarborough. 2012. From Conflict to Coping: Evidence from Southern Ethiopia on the Contributions of Peacebuilding to Drought Resilience among Pastoralist Groups. Washington, DC: Mercy Corps.

Lilja, Jannie, and Kristine Höglund. 2018. “The Role of the External in Local Peacebuilding: Enabling Action—Managing Risk.” Global Governance: A Review of Multilateralism and International Organizations 24 (3): 411–30. https://doi.org/10.1163/19426720-02403007.

Mac Ginty, Roger, and Oliver P. Richmond. 2013. “The Local Turn in Peacebuilding: A Critical Agenda for Peace.” Third World Quarterly 34 (5): 763–83.

Mack, Andrew. 2014. “Measuring Peacebuilding Performance: Why We Need a ‘Data Revolution.’ ” In Legitimacy and Peace Processes: From Coercion to Consent, under the direction of Alexander Ramsbotham and Achim Wennmann. London, UK: Conciliation Resources.

McMenamin, Amy, Carolyn Sun, Patricia Prufeta, and Rosanne Raso. 2019. “The Evolution of Evidence-Based Practice.” Nursing Management 50 (9): 14–19.

Mercy Corps. 2015. Youth & Consequences: Unemployment, Injustice and Violence. Washington, DC: Mercy Corps.

Mia, Irene. 2021. The Armed Conflict Survey 2021. London: IISS, September 21.

Miller, Derek B., and Lisa Rudnick. 2014. A Prototype for Evidence-Based Programme Design for Reintegration. Geneva: United Nations Institute for Disarmament Research.

Owen, Mark, and Anna King. 2019. “Enhancing the Efficacy of Religious Peacebuilding Practice: An Exploratory Evidence-Based Framework for Assessing Dominant Risks in Religious Peacebuilding.” Religions 10 (641): 1–19.

Peace Direct, Adeso (African Development Solutions), Alliance for Peacebuilding, and WCAPS (Women of Color Advancing Peace, Security, and Conflict Transformation). 2021. Time to Decolonise Aid. London: Peace Direct, May 10.

Queen, Emily Forsyth, Jessica Baumgardner-Zuzik, Elizabeth Hume, and Melanie Greenberg. 2018. Snapshot of Adaptive Management in Peacebuilding Programs. Washington, DC: Alliance for Peacebuilding.

Seyle, Conor, Sarah Heyborne, Jessica Baumgardner-Zuzik, and Shaziya DeYoung. 2021. Some Credible Evidence: Perceptions about the Evidence Base in the Peacebuilding Field. Washington, DC: One Earth Future and Alliance for Peacebuilding.

Simons, Helen, Saville Kushner, Keith Jones, and David James. 2003. “From Evidence-Based Practice to Practice-Based Evidence: The Idea of Situated Generalisation.” Research Papers in Education 18 (4): 347–64.

Snilstveit, Birte, Sandy Oliver, and Martina Vojtkova. 2012. “Narrative Approaches to Systemic Review and Synthesis of Evidence for International Development Policy and Practice.” Journal of Development Effectiveness 4 (3): 409–29.

Sonnenfeld, Ada, Hannah Chirgwin, Miriam Berretta, Kyla Longman, Marion Kramer, and Birte Snilstveit. 2020. Building Peaceful Societies: An Evidence Gap Map. New Delhi: International Initiative for Impact Evaluation.

Strand, Virginia, Marciana Popescu, Robert Abramovitz, and Sean Richards. 2016. “Building Agency Capacity for Trauma-Informed Evidence-Based Practice and Field Instruction.” Journal of Evidence-Informed Social Work 13 (2): 179–97.

Tuten, Michelle, Darnell Morris-Compton, Tina Abrefa-Gyan, Jeongha Hwang, and Donna Harrington. 2016. “Predictors of the Use of Evidence-Based Interventions among National Association of Social Work (NASW) Members.” Journal of Evidence-Informed Social Work 13 (3): 253–62.

USAID (US Agency for International Development) and Deloitte Consulting. 2012. Conflict Assessment Framework: Application Guide. Washington, DC: USAID, Office of Conflict Management and Mitigation.

________. 2016. Strengthening Evidence-Based Development: Five Years of Better Evaluation Practice at USAID 2011–2016. Washington, DC: USAID, March, https://www.usaid.gov/sites/default/files/documents/Strengthening_Evidence-Based_Development_Cover_Update_-_Five_Years_of_Better_Evaluation_Practice_at_USAID-compressed.pdf.

________. 2021. USAID Capacity Assessment for Evidence Management and Use. Washington, DC: USAID, Bureau for Policy, Planning and Learning.

US Department of State. 2018. CSO Monitoring Guidance: Data to Inform. Bureau of Conflict and Stabilization. Washington, DC.

Van Dyke, Melissa K., and Sandra F. Naoom. 2016. “The Critical Role of State Agencies in the Age of Evidence-Based Approaches: The Challenge of New Expectations.” Journal of Evidence-Informed Social Work 13 (1): 45–58.

Wallace, Marshall. 2015. From Principle to Practice: A User’s Guide to Do No Harm. Cambridge, MA: CDA Collaborative Learning Projects.

Washburn, Micki, Danielle E. Parrish, Holly K. Oxhandler, Brianna Garrison, and Anny K. Ma. 2021. “Licensed Master of Social Workers’ Engagement in the Process of Evidence-Based Practice: Barriers and Facilitators.” Journal of Evidence-Based Social Work 18 (6): 619–35.

Whitty, Chris, and Stefan Dercon. 2013. “The Evidence Debate Continues: Chris Whitty and Stefan Dercon Respond from DFID.” From Poverty to Power (blog), Oxfam, January 23.

Woodrow, Peter. 2015. Conflict Analysis Framework: Field Guidelines and Procedures. The Hague, Netherlands: Global Partnership for the Prevention of Armed Conflict GPPAC.

Notes

CDA Collaborative Learning Projects is an action research and advisory organization dedicated to improving the effectiveness and accountability of peacebuilding, development, and humanitarian efforts wherever communities experience conflict. More information at www.cdacollaborative.org. Please send questions and comments to Ruth Rhoads Allen, ruth.rhoads.allen@cdacollaborative.org

1. Additional USIP colleagues observed during the three roundtables.

2. See, for example, the work of the Peacebuilding Evaluation Consortium and DM&E for Peace platform at https://www.cdacollaborative.org/cdaproject/peacebuilding-evaluation-consortium/ and https://www.dmeforpeace.org/.

3. The authors are grateful to David Connolly and Marcia Mundt for their inputs throughout the design and drafting of this evidence review, and to many staff at USIP and DevLab@Duke for extensive feedback on earlier drafts of the report.

4. CDA, Designing Strategic Initiatives to Impact Conflict Systems; USIP, Annotated Guide to Resources for Conflict Analysis, internal research product, December 2021.

5. Seyle et al., Some Credible Evidence.

6. Mia, The Armed Conflict Survey 2021.

7. Snilstveit et al., “Narrative Approaches to Systemic Review and Synthesis of Evidence for International Development Policy and Practice.”

8. CDA research showed that there are fewer conceptual and experience-based articles emanating from organizations defined as peacebuilding institutions and more written by and about organizations engaged in development and related sectors. Our statement is also supported by the fact that fewer peacebuilding organizations that we interviewed had clear methods of showing how they were using evidence in their project design and implementation.

9. Sonnenfeld et al., Building Peaceful Societies.

10. Completed in December 2021 for USIP internal use. Please contact USIP’s Learning, Evaluation, and Research team for a copy of case study report.

11. The case studies targeted all secondary questions, whereas the roundtable discussions were less able to address secondary questions B and E.

12. Peace Direct et al., Time to Decolonise Aid.

13. Snilstveit et al., “Narrative Approaches to Systemic Review and Synthesis of Evidence for International Development Policy and Practice”; Ghate, Developing Theories of Change for Social Programmes.”

14. Van Dyke and Naoom, “The Critical Role of State Agencies in the Age of Evidence-Based Approaches”; Whitty and Dercon, “The Evidence Debate Continues”; Eyben and Roche, “The Political Implications of Evidence-Based Approaches”; Brown, “Some Good News in Evidence for Peacebuilding”; USAID, Strengthening Evidence-Based Development; USAID and Deloitte Consulting, USAID Capacity Assessment for Evidence Management and Use.

15. Whitty and Dercon, “The Evidence Debate Continues”; Eyben and Roche, “The Political Implications of Evidence-Based Approaches.”

16. Mack, “Measuring Peacebuilding Performance.”

17. Miller and Rudnick, A Prototype for Evidence-Based Programme Design for Reintegration.

18. Mack, “Measuring Peacebuilding Performance.”

19. Ghate, “Developing Theories of Change for Social Programmes.”

20. Mack, “Measuring Peacebuilding Performance.”

21. The International Dialogue on Peacebuilding and Statebuilding (IDPS) was launched in 2008. Within the IDPS, the New Deal for Engagement in Fragile States was endorsed by forty-one countries and multilateral organizations in November 2011. Central to the New Deal was the commitment to pursue five Peacebuilding and Statebuilding Goals (PSGs). These are (1) enhancing the legitimacy of political processes, (2) improving security, (3) increasing citizens’ access to justice, (4) promoting good economic governance, and (5) managing revenue and building the capacity to deliver services. See the website at https://www.pbsbdialogue.org/media/filer_public/07/69/07692de0-3557-494e-918e-18df00e9ef73/the_new_deal.pdf.

22. Mack, “Measuring Peacebuilding Performance.”

23. Eyben et al., The Politics of Evidence Conference Report.

24. Eyben et al., The Politics of Evidence Conference Report.

25. Adam, “The Boundaries of Evidence in Conflict Management and Peacebuilding.”

26. Adam, “The Boundaries of Evidence in Conflict Management and Peacebuilding”; Blum and Kawano-Chiu, Proof of Concept; Church and Rogers, Designing for Results.

27. Seyle et al., Some Credible Evidence.

28. Seyle et al., Some Credible Evidence.

29. Eyben et al., The Politics of Evidence Conference Report; Mercy Corps, Youth & Consequences.

30. Seyle et al., Some Credible Evidence.

31. Mack, “Measuring Peacebuilding Performance.”

32. Bates and Glennerster, “The Generalizability Puzzle.”

33. J-Pal’s generalizability framework has four steps, based on answers to four questions: (1) What is the disaggregated theory behind the program? (2) Do the local conditions hold for that theory to apply? (3) How strong is the evidence for the required general behavioral change? (4) What is the evidence that the implementation process can be carried out well? Bates and Glennerster, “The Generalizability Puzzle.”

34. Bates and Glennerster, “The Generalizability Puzzle.”

35. Eyben and Roche, “The Political Implications of Evidence-Based Approaches.”

36. Eyben and Roche, “The Political Implications of Evidence-Based Approaches.”

37. Bates and Glennerster, “The Generalizability Puzzle.”

38. Ginty and Richmond, “The Local Turn in Peacebuilding.”

39. Lilja and Höglund, “The Role of the External in Local Peacebuilding.”

40. Lilja and Höglund, “The Role of the External in Local Peacebuilding.”

41. Simons et al., “From Evidence-Based Practice to Practice-Based Evidence”; Bates and Glennerster, “The Generalizability Puzzle.”

42. CDA Collaborative Learning Projects, Designing Strategic Initiatives to Impact Conflict Systems; Anderson and Olsen, “Confronting War,” p. 89.

43. Owen and King, “Enhancing the Efficacy of Religious Peacebuilding Practice”; Hansen, “Evidence-Based Conflict Management Practice.”

44. Wallace, From Principle to Practice.

45. Seyle et al., Some Credible Evidence.

46. This perspective correlates with the significant finding by Seyle et al. in Some Credible Evidence about peacebuilding programming involving women as the only design type having “some credible evidence” showing impact.

47. Queen et al., Snapshot of Adaptive Management in Peacebuilding Programs.

48. Carter et al., Creating a Culture of Evidence Use.

49. Carter et al., Creating a Culture of Evidence Use.

50. Miller and Rudnick, A Prototype for Evidence-Based Programme Design for Reintegration.

51. Ababou and Alzate, “Developing an Evidence-Based Mindset.”

52. Carter et al., Creating a Culture of Evidence Use.

53. Miller and Rudnick, A Prototype for Evidence-Based Programme Design for Reintegration; Hansen, “Evidence-Based Conflict Management Practice.”

54. Hansen, “Evidence-Based Conflict Management Practice.”

55. Van Dyke and Naoom, “The Critical Role of State Agencies in the Age of Evidence-Based Approaches”; Hansen, “Evidence-Based Conflict Management Practice.”

56. Van Dyke and Naoom, “The Critical Role of State Agencies in the Age of Evidence-Based Approaches.”

57. Hansen, “Evidence-Based Conflict Management Practice.”

58. Bates and Glennerster, “The Generalizability Puzzle.”

59. Van Dyke and Naoom, “The Critical Role of State Agencies in the Age of Evidence-Based Approaches.”

60. Carter et al., Creating a Culture of Evidence Use.

61. Anderson and Olson, Confronting War; CDA Collaborative Learning Projects, Designing Strategic Initiatives to Impact Conflict Systems; Seyle et al., Some Credible Evidence.

62. Kamatsiko et al., Connector Projects Approach.

63. Hansen, “Evidence-Based Conflict Management Practice.”

64. Relevant framing is drawn from both within the wider peacebuilding and development sector, such as the Movement for Community Led Development, and related fields, such as organizational and social psychology, including the work of Carol Dweck, Ph.D.

65. This could have elements of the generalizability framework that J-Pal developed for its own organization. See Bates and Glennerster, “The Generalizability Puzzle.”

66. The training course was piloted in February 2022 with USIP and USAID participants to inform the development of a Global Campus online course.

67. See USIP analysis by Susanna Campbell and Corinne Graff, “Implementing the Global Fragility Act: What Comes Next?,” April 7, 2022, https://www.usip.org/publications/2022/04/implementing-global-fragility-act-what-comes-next.

68. Peace Direct et al., Time to Decolonise Aid.

69. The evidence review research question is: “What are the barriers to using evidence from conflict analysis and assessment reports or from monitoring and evaluation in project designs at USIP?” The question in the “Recommendations” section has been expanded beyond assessments and monitoring and evaluation to capture the barriers encountered when applying any type of evidence.

70. See, for example, CDA Collaborative Learning Projects, Designing Strategic Initiatives to Impact Conflict Systems.