Step 6: Selecting the Right Independent Grant Evaluator

What You Need to Know
Evaluation is important to give grant stakeholders and funding sources an idea of the effectiveness of their programs. The demand for accountability is rising, with more funding programs demanding to know how programs are working compared to what was outlined in the grant proposal. Funding sources desire to put their limited monies into programs run by the most competent agencies to yield the greatest new findings and have the most significant impact.

Through their research, evaluators can have a positive, much-needed influence by showing the impact of a grant program on the people it is intended to affect, the benefits of certain techniques or methods, and areas for improvement. Competent evaluators provide ongoing feedback to program staff so they can improve and stay on track.

But how do you find an evaluator who meets your program’s needs and provides unbiased feedback? How can you avoid disasters like the one mentioned above? And how do you build a partnership between the program staff, funding source, and evaluators?

The first task is finding candidates whose education, experience, and outlook make them fit to evaluate your program. As the demand for accountability and demonstrable results increases, more and more people are entering the evaluation field. But just as there are good (and not so good) plumbers, so too are there good (and not so good) evaluators. There is no evaluator ‘license,’ no test to pass, and no recognized credentials. While it is helpful, having a Ph.D. does not qualify a person to conduct high quality evaluations.

In a field where there is no certification to prove the quality of evaluators’ services, how do you find the right evaluator for your program? First, you must determine the needs of your program and your funder. Then, ask yourself: Why am I seeking an evaluator? What are my expectations, and what are the requirements of my grant funder? Answers to these and other questions are contained in a Request for Proposals (RFP) to evaluate a specific grant program. If you have not developed a specific grant- or program-driven RFP, you can issue a Request for Qualifications (RFQ) asking interested parties to submit résumés, price estimates for evaluations, and other supporting materials demonstrating their capacity to evaluate programs in a given area. Then, when a third-party evaluation is required in the future for any given program, you will have an assortment of evaluators from which to select the most qualified. For example, you may select three evaluation firms or consultants to provide services to your organization: Vendor 1, who works with school districts and education; Vendor 2, whose expertise lies in criminal justice; and Vendor 3, who specializes in programs concerning alcohol and drug abuse. If your organization received a GEARUP grant to help high school students continue their education in college, then you would select Vendor 1 as your evaluator. If you received a crime reduction grant for a community, then you would employ Vendor 2. Regardless of whether you issue an RFP or an RFQ, knowing what you are seeking and why will go a long way toward ensuring the best fit between your needs and the evaluator’s abilities.

Through Uniform Administrative Requirements, federal funding sources have tightened their regulations, requiring agencies that receive federal grant funds to issue bids for contractors. Federal agencies like to see several organizations bid on evaluations and frown upon awarding contracts to single companies without competition. Thus, it is prudent for organizations to bid out evaluation services when the need is low instead of waiting until an award is made and scrambling to find a competent evaluator then.

Robert M. Johnson of the Council on Foundations suggests you “develop relationships with sources of evaluators—preferably before you need them—and keep scouting around for new and better ones.” By maintaining contact with other grantees and evaluation firms in your area, you can ensure that you interview only evaluators whose goals and standards match yours. Another way to find quality evaluators is to ask around with other people who have received the same or similar grants that you have. They will know from experience if an evaluator they have worked with produces quality work. In fact, as the US Department of Education notes in a paper for Smaller Learning Communities Grant project directors and program staff, “other SLC grantees can be an invaluable resource in finding a good evaluator.”

You want to develop a fair bid package that will be open to and promote competition. Use them as a starting point, altering them to fit your organization’s needs and to ensure that you have an open, competitive search and bid process. Selecting evaluators with proven quality and experience should be your foremost objective.

Outside evaluators come in several forms, but the US Department of Education says that there are generally four categories: university faculty, private for-profit evaluation firms, non-profit organizations, and independent contractors. You can also search the website of the American Evaluation Association (AEA), www.eval.org, by state to find evaluators in your area.

Finding experienced, quality evaluators, especially in small- to moderately-sized communities, is not an easy task. It usually requires extensive advertising, and the process can be time-consuming. That is why it is important to seek out qualified evaluators before you need them.

An organization could spend several thousand dollars advertising for an evaluator through traditional print media like newspapers. Outside of referrals from trusted colleagues and searches through the American Evaluation Association and CareerBuilder.com are often more effective ways to seek evaluators. Be as detailed as possible in describing the skills, experience, and education you are seeking. Universities and colleges are some of the greatest sources of evaluators. The schools of nursing, education, criminal justice, health, social work, and psychology often have professors who are involved in writing or evaluating grants. Try contacting the associate dean of each school about faculty members who are involved in grants.

Selecting an Evaluator
Once you have your applicant pool together, it’s time to select a qualified evaluator who you feel will work effectively with you and your staff. There are six questions to ask when choosing an evaluator:

  1. What are the qualifications and skills of the evaluator? At a minimum, the evaluator should have a Masters degree from an accredited university in the area under investigation or in a related field (a PhD is preferred). They must also have the skill set necessary to perform the work. Depending on your needs, the skills will range from the relatively simple (one-on-one interviews) to the complex (statistical analysis using multivariate techniques). Evidence that the evaluator has the necessary skill set can be found in examples of previous work.
  2. Does the evaluator have the demonstrated capacity to do the work? Ideally, the evaluator should have experience and success in evaluating similar grant programs. The only way to know this with any certainty is to ask for examples of previous work. Look over reports, published articles, briefs, bulletins, PowerPoint presentations, etc. to get some idea of their quality and capacity. The evaluator should be able to write for a variety of audiences. Good evaluators tailor their reports to their audience—there is no one-size-fits-all report. The writing should also be clear, the tables and graphs unambiguous, and the format of the report pleasing to the eye. Recommendations and commendations for timely and informative program enhancement should also be included.
  3. What do other people say? Ask for referrals, but be cautious about references since many individuals will only list friends and colleagues who will give them glowing recommendations. Interpersonal skills are critical to being a good evaluator, and this is impossible to assess from finished reports. References should report that the evaluator is reliable, responsive, easy to talk to, and a good listener. Communication is a key to successful evaluation. The evaluator should be able to present information in a clear, concise, easy-to-understand, and jargon-free manner.
  4. What is their approach to evaluation? There are many ways to conduct evaluations. Ask evaluators to explain their model or philosophy of evaluation. Some evaluators only use quantitative methods, while others use exclusively qualitative methods. Most use a mixed-methods approach that uses both qualitative and quantitative techniques. Regardless of their preferred method, in the spirit of collaboration the evaluator should actively encourage stakeholder participation in the evaluation. Will they share drafts of instruments and reports? Do they enlist key stakeholders to discuss design, methods, and data collection procedures, or do they dictate the course of the evaluation without seeking input? In other words, is their approach to do evaluation to you or with you?
  5. Do they have adequate resources? Evaluators not only need to have the know-how and experience, but they need to have the resources as well. Having a cadre of well trained staff, the necessary equipment, and infrastructure indicates that the evaluator is prepared to take on the evaluation. You can also determine more about an evaluator’s professionalism by factors like the appearance of their website, the quality of their letterhead, their promptness in returning calls, and the quality of their written communications.
  6. Are they familiar with the standards and guidelines of the field? Evaluators should be familiar with the American Evaluation Association’s Guiding Principles for Evaluators (www.eval.org/Publications/GuidingPrinciples.asp) and the Program Evaluation Standards of the Joint Committee on Standards for Educational Evaluations (www.wmich.edu/evalctr/jc/). If they are unfamiliar with these standards and guidelines, cross them off your list immediately.

You will also want to consider other aspects of the evaluator’s personality. The Kellogg Foundation Handbook says that, based on the foundation’s experience, “the most important overall characteristics to look for in an evaluator are the ability to remain flexible and to problem solve.” Johnson recommends working with evaluators “who are both disciplined in their approach to the task and convivial and tactful in their approach to people.” Your evaluator needs not only the skill set to conduct an effective evaluation, but to interact positively with others as well. Johnson also recommends that you find an evaluator “with an identifiable personal interest in conducting the evaluation you want.” Evaluators are stakeholders as well and have goals and perspectives of their own, according to the Kellogg Foundation. When selecting an evaluator, make sure their interests and areas of expertise lie in the area of your project.

Interviewing an Evaluator
When interviewing candidates to provide evaluation services for your program, you should preferably form a committee of individuals who are experienced in program and grants fields. This committee should include at least one person who can get a feel for the person versus the professional. You should interview several evaluators to give you a wide range of skill levels and personalities from which to choose. Take your time to ensure that you select an evaluator whose interests and abilities match the demands of your program.

Send candidates for your evaluation position a sample grant RFP and a copy of the grant proposal in advance. This does not have to be one that you have submitted or have been awarded. Consider asking applicants the following:

  1. Tell me about your experience in evaluating grant programs and the types of projects you have evaluated.
  2. What is your philosophy or the evaluation model you use in regards to evaluating a grant program?
  3. Why do you evaluate grant programs?
  4. What has been your greatest success story during your evaluation career?
  5. What has been your greatest struggle in evaluating grants?
  6. Can you describe a situation where you and the program staff were at odds on how to proceed?
  7. Describe how you work with the grant program staff and constituents.
  8. We sent you an RFP and a grant proposal. Please describe how you would evaluate the grant program. Describe the process and steps you would implement to ensure a high-quality grant evaluation.
  9. What other staff and resources will you utilize in conducting your evaluation?
  10. How often would you meet with our staff to provide input and suggestions on how the program is functioning?
  11. How could our staff help you to succeed in conducting a good evaluation?
  12. What other work or job responsibilities do you have in motion that might interfere with you working with us? Will you have the time to provide a quality evaluation for our agency?
  13. When you develop a report, how will our staff be able to review and provide input prior to submitting the report to the funding source? What is your philosophy toward our staff making suggestions about your report provided that it does not interfere with the findings?
  14. How will you communicate with us?
  15. How will you bill for your services? Can you provide us with a list of measurable outcomes or deliverables that are to be accomplished and the times of year they will be accomplished before payment is made?
  16. We would like to see several copies of your evaluation reports and other documents such as survey instruments that would help us make a decision. Can you provide us with some examples?
  17. Can you provide us with a list of grants you have evaluated and the project directors’ contact information?

Cost of Services
There is no standard amount that is charged for evaluation services. Some government staffs and funding sources do not fully understand how involved evaluation can be and have arbitrarily capped allowances for evaluation at extremely low percentages of the overall grant funds. For those who are allowed to choose how much money is spent on evaluation in their programs, there are a wide range of prices—and services—out there.

The cost of an evaluation is largely determined by what you want to know and how confident you want to be in the findings. With this in mind, evaluation fees can range from three to 20% (or more) of the grant program’s total cost. Generally speaking, when evaluating for program effects there are three tiers: for a descriptive evaluation using a pre-posttest design, 8-12% of the program budget is reasonable; add on a control or comparison group and the cost can rise to 13-17%; and using a randomized experimental design with longitudinal follow-up can cost 18% or more.

Ensure that you allot enough money to cover a thorough and quality evaluation that gives you all the information you need to know about your program. As Johnson notes, “Mediocre evaluations lead to mediocre programs in the future. A modest program that shows strong signs of success, fresh ideas and potential influence, for instance, may warrant an evaluation that costs much more than the program itself.”

Because there are rarely concrete guidelines for how much money should be used on evaluation, the fees recommended by evaluation experts vary widely. The US Department of Health and Human Services Administration for Children & Families estimates that 15 to 20 percent of the total funds for programs are usually set aside when the funder specifies an amount to cover the evaluation. The University of Tasmania’s Project Evaluation Toolkit says that a “common 'rule of thumb' is to allocate around 10% of the total project budget to evaluation costs.” In practice, the variation in evaluation funding is also large. Some funding sources cap the available funds for evaluation at three to five percent. Others allow higher amounts for more in-depth evaluations. For example, the Community Tool Box at the University of Kansas cites a Massachusetts adult literacy program grant mandating that 10% of the award be set aside for evaluation in the first year of the program. David Nee and Maria Mojica, senior staff members at the W.C. Graustein memorial fund, expected their foundation’s evaluation of the Connecticut Center for School Change to cost nearly 10% of the center’s yearly operating budget.

Again, it is important to remember that the more money set aside for evaluation, the greater the depth of knowledge that can be had about a program and its effects on its targets. Even the greatest evaluators in the world can’t provide an extremely thorough evaluation without being given the funds to perform the necessary research. Cost should certainly be a factor in choosing an evaluator, but spending a lot of money will not necessarily guarantee good results either. Your goal should be to find an experienced, qualified, personable evaluator at a reasonable cost.

Working with Your Evaluator
Once you have chosen your evaluator, you need to meet and discuss roles and expectations in detail. Then, create a written contract stating what services the evaluator will provide. The Juvenile Justice Evaluation Center says that the contract should cover “the questions to be addressed by the evaluation, the tasks that need to be performed to address those questions, who will be responsible for these tasks, and when they will be accomplished.” Family Connection of Georgia also recommends that grantees should discuss desired deliverables and available budget, who will own and manage databases resulting from data collection efforts and where the databases will be located, and prepare a contract reflecting these factors, deadlines, and a fee payment schedule.

Collaboration is key, which is why it is important that your staff and the evaluator work together to answer the questions the evaluation seeks to answer. The grantee and evaluator should meet frequently over the phone or in person to discuss the progress of the evaluation. This will give the evaluator the opportunity to understand the program more thoroughly and for the grantee to gain a new, outside perspective on their program.

Using an outside evaluator has measurable benefits for any grant program. By learning what works (and what does not), grantees can improve the efficiency of their programs and thus positively affect the lives of more people. Thorough evaluations can help grantees pinpoint problem areas in their programs and come up with ideas on how to fix them—or affirm that a program is having a positive effect on the community, which could help secure more funding in the future.

The right evaluator can be a huge asset to your program, and the wrong one can be your worst nightmare. Be careful and take your time to find the evaluator who can help guide your program to success.