Evaluate Together

STEPS
  1. Determine who wants to learn from the evaluation
  2. Form a representative evaluation team with community members and other interested parties
  3. Determine what participants want to learn from the evaluation
  4. Develop an evaluation plan and evaluation instruments
  5. Conduct the participatory evaluation
  6. Analyze the results with the evaluation team members
  7. Provide feedback to the community
  8. Document and share lessons learned and recommendations for the future
  9. Prepare to reorganize

STEP 5: Conduct the participatory evaluation.

When the team has developed the plan, methods, and instruments to collect information and has practiced using them and has worked out the logistical details (transportation, meals, accommodations, meeting places, equipment needs, communication with community participants and interviewees to set schedules), it is time for the team to conduct the evaluation.

The team leader should determine how she or he can best support the other members, identifying which subteams may need more help with certain tasks or communities, whether all teams have the materials they require, and assisting team members as needed. The team leader should ensure that the information is organized as it comes in, so that field notes, surveys, and other evaluation documents, pictures, audio tapes, and videotapes don’t get lost. If you have access to a computer and to someone who knows how to set up a database, the team can enter the data into the computer to facilitate analysis. If your team has developed analysis tables as mentioned in the planning phase, the person entering data should use these tables to organize the information. If you don’t have access to a computer, you can record data manually using tally sheets.

BOLIVIA: Evaluating Warmi

The final evaluation of the Warmi Project included both qualitative and quantitative methods and was participatory in design, analysis, and writing. A wide range of persons played key roles in implementing the evaluation and interpreting the results, including representatives from the Ministry of Health, CARE, The Population Council, San Gabriel Foundation, Save the Children's Home Office, Save the Children/Bolivia staff and nine women representatives from communities participating in the Warmi Project. Emphasis on participation at all levels allowed the team to capture the explicit and implicit achievements of the program.

Using a variety of methods, the evaluation tried to consolidate the quantitative and qualitative results of the project.

Quantitative methodology: A retrospective case-control study (cases were maternal, perinatal and neonatal deaths; controls were babies born in the same communities the same year that did not die) that had been done at baseline was repeated. The results from the final evaluation were compared with those of the original study in order to identify trends in the project indicators. The manual information system was reviewed, and the last project year's results were compared with project goals and objectives, as well as with the results of the case-control study.

Qualitative methodology: Individual interviews were held with pregnant women, lay midwives, MOH staff at local facilities, SC/Bolivia staff, husbands, women of reproductive age and representatives from the PVO and NGO health network, PROCOSI, San Gabriel Foundation, CIEC and AYUFAM (local NGOs that had participated in the project). Group discussions were held with nine women's groups from the three project zones, midwives, and local authorities from each zone.

The evaluation team developed interview guides for all interviews. SC/Bolivia and MOH staff took a short written exam to determine their level of knowledge regarding several important indicators. The group interviews also utilized several participatory techniques, include a "pile sort" of project intervention cards in small groups, judgments of characters' actions (good, bad, don't know) in stories told by the interviewers, and drawings or written statements of the groups' vision of the future.

If at all possible, the team should meet every day after fieldwork has been completed. These meetings help team members to identify problems, challenges, share and consolidate learning, and make adjustments in the plan and information collection instruments and methods if necessary. The meetings can also help to build a more cohesive team, leading to more trust and rapport in the group, helping team members to more openly communicate their thoughts and feelings about what they are learning.

More often than not, something will not go exactly as planned and the team will need to be flexible. This is to be expected. The better your team has identified its priorities and has planned alternative approaches to conduct the evaluation should problems arise, the more easily team members will be able to cope with unanticipated changes.