Evaluate Together

STEP 2: Form a representative evaluation team.

As we have seen in step 1, many individuals and organizations may be interested in learning from an evaluation. It is obviously not practical for everyone to participate in every step of the evaluation; neither do most people feel the need or desire to do so. Creating a representative, effective, and appropriate evaluation team requires careful consideration and negotiation with all parties concerned. In some cases, donors specify who should participate in an evaluation, and their perspective should be taken into account as you put together the team.

Experienced observers debate the pros and cons of internal versus external evaluators. The table below (adapted from Partners in Evaluation, Feuerstein, 1986) summarizes some of the advantages and disadvantages. Because this is a participatory evaluation, internal participation is, by definition, important. However, there is also much to be gained from external perspectives and skills. If you can afford the necessary time and resources, we strongly recommend a mix of internal and external evaluation team members.

Two additional advantages to internal evaluators are that: (1) internal evaluators are well placed to use the information from the evaluation to improve future programs; and (2) they may be more committed and interested (than external evaluators) in learning about why the program was or was not effective.²

When putting together an evaluation team, be sure to consider group dynamics, power relations, technical and interpersonal skills, credibility, diversity of strengths, weaknesses, and perspectives. Practical concerns such as potential team members’ availability and logistics (e.g., transport, lodging, food, safety) also need to be considered. The larger and more diverse the team becomes, the more interesting and fruitful the process can be. It can also become more complex, and you will need to make sure that you have a team leader who is respected by team members and who has strong facilitation, management, and technical skills .

If you decide that each group to be represented on the evaluation team should select its own representative, it is helpful to discuss with these groups what will be required of their representative so they can make an informed selection. It is helpful to know such things as:

• How much time each participant will need to be available.

• Whether the participants will be expected to travel, stay overnight in other communities, need to walk long distances, etc.

• Whether the participant needs to be able to read and write. To enable the participation of people who may not know how to read and write, it is helpful to clarify that this is not necessary.

• A general description of what the participant will be doing.

Teams sometimes inadvertently limit the participation of people they most want or need to include when they establish criteria that others later decide excludes that group. If you would like to ensure that women participate in the evaluation, for example, and then stipulate that participants will need to travel and be away from home for several days at a time, community leaders may decide that this makes it inappropriate for women to participate. They will likely choose men to represent the community unless you explicitly state that women need to be adequately represented. In this case, the focus shifts to which women can participate and how the family and community can support their participation. Alternatively, if communities or the women themselves do not accept some of the terms of participation, you may need to negotiate new terms (e.g., change schedules).

The process that each participating group uses to select representatives will directly influence who is chosen. Some teams prefer to let each group determine how they will select their representative. In many cases, the result is that a community leader chooses someone he or she believes would be best, whether or not this person really represents the broader group. Given the opportunity to choose, community members may support someone else’s participation, someone they believe would better represent their interests and concerns on the team. The selection process is almost always fraught with difficult decisions as to how much to structure the selection versus leaving groups to set their own course. Try to anticipate what some of the difficulties may be and determine what role your team will play. In some cases, you may be able to accommodate existing selection processes (community leader-selected) with other processes (community votes or comes to consensus) by expanding the team to include more than one representative from the group. This type of compromise may help encourage greater acceptance of the evaluation results and recommendations later on.

Once you have established an evaluation team, it is a good idea to agree upon ground rules or operating norms with team members so that everyone understands how the team will work together. Here are some suggestions:

• We will decide priorities based on a set of criteria established by all members of the evaluation team.
• We will be on time for meetings.
• All members will agree on who will facilitate each meeting and who will take notes before the meeting begins.
• All team members will be given the opportunity to participate in discussions but will not be forced to do so.
• When we don't understand something or need help, we will ask for help- all questions are valid.