Imagine this scenario. 

The evaluation team has just completed an exceptionally challenging assignment. After weeks of planning, preparation, data collection, and analysis – they have finalized their two-hundred page report, complete with tables, figures, and appendices. Their in-depth analyses have discovered long-lost insights into the social value of the project and formulated evidence-based recommendations for the organization to move forwards. Satisfied with their work, the team sends the final product to their partner. The consulting team is paid, a few key stakeholders skim the executive summary, and the report is filed away – to be forgotten. 

 The waste of time and resources described in the example above is a too-common occurrence in the field of monitoring and evaluation. Even the most thorough methodology and innovative findings are meaningless if the report is ignored by the intended users and participants. In fact, this lack of follow-through on consulting reports can lead to a decline in the perceived value of future consulting engagements and stakeholders’ participation, as stakeholders become skeptical of the benefits of such projects. At the same time, the organization has wasted resources, missed opportunities for growth and improvement, ignored potential programmatic challenges, and lost an opportunity to increase stakeholder satisfaction. From both an organizational and a consulting perspective, this scenario is lose-lose. 

 So, whose responsibility is utilization, the consultant or the contracting organization? 

 The responsibility for utilization is shared between the consultant and the organization. First, responsibility lies with the organization to create a culture that values evaluation findings, strategically uses reports to learn and adapt, and creates systems to ensure that reports are utilized as intended. Developing this kind of culture starts in the strategic planning process and the design of programs which clarify how the findings from reports will be used and specific mechanisms to follow up on these reports. Therefore, even if the organization experiences challenges that could disrupt organizational culture like staff turnover, the commitment to organizational learning and follow-up is built into the system. An internal commitment to utilization must be established before and during evaluation projects, including clear identification of intended users in the terms of reference and coordination with the consultant during the project. 

When an organization is well-equipped to use the findings of a consulting assignment, it becomes the sole responsibility of the consultant to produce a deliverable that will be useful. The findings must be presented in a clear way that is curated to the needs of the end users, whether it is the project manager,  donors and project’s participants. The information from the report is only valuable insofar as it meets the needs of the organization, reflects insights from stakeholders,   and is comprehensible to these users while suggesting clear actionable strategies for a follow up. An example that recently went viral on the internet is the “insane” design concept behind the old Pepsi logo, which includes “evidence” for the design concept rooted in the earth’s layers and magnetic fields, the design of the Mona Lisa, and the movement of light through space and time.  Although this is not in the field of monitoring and evaluation, it is a fantastic example of a consultant that was more committed to their own ideas and methodology than the needs of the end users. From a value perspective, the time and resources invested into creating diagrams of the earth’s magnetic fields and relating it to the Pepsi logo add little value to the project managers and marketing department.

How does Octopus Consulting and Insight (OCI) do utilization-focused evaluation?

OCI is committed to creating reports that can be utilized by organizations to learn, innovate, and grow. We seek to not only report evidence-based findings, but also to ensure that the entire project aligns with the needs of the end-users, especially local stakeholders. Sometimes, a two-hundred page report with multiple appendices describing a complex phenomenon or scenario will be necessary, such as an in-depth research project. On the other hand, most organizations are seeking a report that will clearly and concisely describe key findings from the evaluation process, maximizing user-friendly graphics and visualizations that can be presented to staff, partners,   donors and local project participants. The evidence must be concrete and clear with actionable recommendations, but short in length.

There are two essential components of utilization-focused evaluation, which are (1) clear identification of the final users and (2) intentional alignment with end-user needs throughout the entire process. Clear identification of the end users should be possible by reading the terms of reference, but can sometimes require additional clarification during initial meetings and a quick discussion with key stakeholders in the field (see image below of a women’s focus group discussion in DRC). The intended users of a report are more likely to use the findings if they have a sense of ownership of the process and are actively involved, which starts during the inception process (meetings and inception report). Likewise, their expectations, and the expectations of the participants of the project/program, must be seriously taken into account. The intended users must be the primary focus during the development of evaluation questions, methodology design including analysis methods, and the final report. 

FGD with Women in DRC

Moreover, Octopus Consulting and Insight is familiar with program evaluation standards of utility, feasibility, propriety, and accuracy (defined and established by The Organization for Economic Cooperation and Development (OECD)), striving to meet and exceed these standards. Utility standards guide OCI to create reports that will meet and exceed the needs of the contracting organization – whether for internal or external use. We prioritize discussions from the beginning on report utilization, seeking roles that will truly contribute to organizational learning and program development. In fact, good reporting can be foundational for developing a program knowledge sharing system, which will enable the program to become a learning opportunity for all. From the example in the intro, it is probable that the main users of this report were managers with limited time on their hands. If the consultants had learned this in their initial alignment process, they would have understood the need for a much more concise report. The information needs of various stakeholders are different, such as donors, local project’s stakeholders,  executive managers, field staff, etc. and must be a priority in the evaluation process.

Additional Resources

Patton, MQ. (2021) Utilization Focused Evaluation. (Fifth Edition) Sage Publications.