Evaluate Together

STEPS
  1. Determine who wants to learn from the evaluation
  2. Form a representative evaluation team with community members and other interested parties
  3. Determine what participants want to learn from the evaluation
  4. Develop an evaluation plan and evaluation instruments
  5. Conduct the participatory evaluation
  6. Analyze the results with the evaluation team members
  7. Provide feedback to the community
  8. Document and share lessons learned and recommendations for the future
  9. Prepare to reorganize

STEP 4: Develop an evaluation plan and evaluation instruments.

Once the team and core group have identified the questions they want to ask and the indicators they will be looking at, it’s time to put this information together into an evaluation plan. At a minimum, an evaluation plan should answer seven key questions, as presented in the Evaluation Plan worksheet on page 198. The seven questions are:

1. What were the project’s objectives and expected outcomes?

Here you will make a list of the overall goals and objectives of the community mobilization project and a second list of the expected outcomes.

2. What questions do we need to ask?

This is a list of questions you will need to ask to determine what the outcomes of the project were, whether or not the project met its goal and objectives, and why or why not. Sample questions include:
What happened?
What elements of the project worked?
What elements did not work?
What were some conspicuous successes?
What were some conspicuous failures?
What still remains to be done?
What is the community’s vision for the future?
How has the community’s capacity grown during the project period?
Which results obtained during the project period are likely to be sustained or improved upon?

3. What information do we need to answer these questions?

This will be a list of the indicators you will have developed (or will now need to develop) with the help of the information in the latter half of Step 3.

4. How will we collect this information?

BOLIVIA & PERU: Ultimate Causes

The Warmi project team believed that low women's status was a large contributor to maternal mortality and morbidity. To address this issue, the project aimed to increase women's status by increasing their participation in women's groups and other community organizations. The Warmi project evaluation then documented the extent to which women actually did participate and whether their status was increased as a result of their participation. This was done using a combination of quantitative and qualitative methods with the women themselves, men, community leaders and health service providers. Clearly, this information is context specific and can only be understood in relative terms (what it was like before the project and what it is like now). In the case of the Warmi project, the team was not able to compare participating communities to non-participating communities as most communities in the area participated and finding comparable non-participating "match" communities was difficult.

In the Puentes Project in Peru, shared responsibility was a key underlying theme. The team analyzed the transcripts from videos and meetings throughout the project to identify changes in community and service provider dialogue and action related to this and other key underlying themes.

This part of your plan should include a list of the instruments and methods you will be using to collect the information you’ve now decided you need. Some suggested methods here would include:

Repeating the baseline survey
Stories (peak moments, peak achievements)
Drawings (e.g., of project history)
Skits to present important milestones or events
In-depth interviews with project participants and observers
Group discussions
Picture card pile sorts (sort interventions that worked/did not work)
Rankings (rank initiatives/interventions from worked best to worked least)
Review of the project proposal, reports, and documents

One of the most common ways of evaluating a project (listed first above) is to repeat the original baseline assessment to determine the extent to which key indicators may have changed. If you are repeating a baseline survey or other baseline assessment to compare pre- and post-project implementation status, you must use the same instruments and techniques that were used at baseline. You will need to determine whether you would like to repeat the exercise exactly as it was carried out before or whether you would like to add new questions. For example, the team may want to add questions when members have learned through their experience with project implementation that there are additional factors that may influence results, factors they may want to understand better or want to use as information to build upon in the future. The team might delete questions if they are no longer relevant to the program, such as a planned intervention that was never implemented.

5. Who will collect the information?

You will have to decide who is going to collect which information in what places. If you have already decided how the various kinds of information are going to be collected (in 4. above), using which instruments and methods, then you may want to match people up with the methods/instruments with which they are most familiar and comfortable.

6. What resources/materials will be needed?

Make a list of who will need what in order to collect each kind of information from the various sources where it will be collected.

7. When will this information be collected?

Create a timeline for the various participants.

With regard to assessing community capacity (one of the questions under number 2 above), you may want to use (or modify) Save the Children’s community management and participation scale (phase two). Your team can ask participants to document or describe in some way (see number 4 above for suggestions) ahow the community worked in the past and how it is different now. The question can also be dealt with by referring to the community’s vision of the future and discussing with participants how they plan to get there, what steps they will take, what specific experience have they had in the past that will contribute to them being able to move forward, what skills and abilities they plan to strengthen and how?

Answering these questions in a systematice and thoughtful manner should give you a basic evaluation plan that will serve as a good starting point for your efforts. As you develop this participatory evaluation plan, here are three important points to keep in mind:

  1. Every team member doesn’t need to do everything.

    As with any team, it is important to maximize members’ skills and program resources. If you have many team members, it can be counterproductive for all members to attend all evaluation activities. In most cases where group discussion is involved, for example, one or two facilitators is sufficient. For example, one person can facilitate the discussion while another observes and records information. If team members are interested in building their skills in conducting a variety of evaluation methods and techniques, they can divide up the work keeping this objective in mind.

  2. Select methods, tools and techniques after evaluation questions and content are agreed upon.

    Sometimes, in their zeal to try out new participatory techniques and methods, team members look for ways to include them in field work without taking the time to determine whether these particular tools or techniques would be effective to answer a particular question. Using these techniques can lead to interesting discussions, but may not ultimately help to answer priority questions that evaluation team members and the groups, which they represent, have set out to answer. (See next section on selecting methods and refer to the Participatory Facilitator’s Guide for further assistance.)

  3. Keep it simple; don’t make things more complicated than they are or need to be.

    It is easy to find yourself buried in pages of proposed indicators, potential methods and tools, and a variety of possible evaluation designs, so much so that some teams may forget what it was they initially had set out to learn. It helps to go back to your “burning questions” and develop a brief one- or two- page plan to which everyone can refer. Try to limit the indicators to a manageable number. Make sure that everyone on the team can explain what the evaluation is about, how it will be carried out, and what they hope to learn from it. If team members cannot describe what they will be doing, the plan is not clear enough and the team leader will need to work with team members to simplify and clarify concepts and processes. To facilitate data organization, the team should develop some draft analysis tables to keep track of information as it comes in so that it will be easier to analyze later on. For examples, see the matrices presented in phase 3, Explore the Health Issue and Set Priorities.