Publication | 2009
INSTITUTIONAL DESIGN AND UTILIZATION OF EVALUATION: A CONTRIBUTION TO A THEORY OF EVALUATION INFLUENCE BASED ON SWISS EXPERIENCE
Growing interest in the institutionalization of evaluation in the public administration raises the question as to which institutional arrangement offers optimal conditions for the utilization of evaluations. 'Institutional arrangement' denotes the formal organization of processes and competencies, together with procedural rules, which are applicable independently of individual evaluation projects. It reflects the evaluation practice of an institution and defines the distance between evaluators and evaluees. The article outlines the results of a broad-based study of all 300 or so evaluations which the Swiss Federal Administration completed in the years from 1999 until 2002. On this basis, it derives a theory of the influence of institutional factors on the utilization of evaluations.
The authors conducted a qualitative analysis of a random sample of 12
external evaluation reports from the year 2002. A list of 23 evaluation
standards was devised, based on the DAC Minimum Sufficient Evaluation
Standards (DAC standards) and the widely used SEVAL standards of the
Swiss Evaluation Society. The 23 evaluation standards were divided into
the four categories also used by the SEVAL standards: utility, referring
to readable, accessible and timely evaluations with a good and useful
summary; feasibility, ensuring that an evaluation be executed in a
realistic, well-thought out manner; propriety, referring to ethical
aspects; and accuracy, ascertaining that proper methods and procedures
are used. Each of the 23 evaluation standards was translated into a set
of questions and applied to the evaluation reports systematically. The
authors reviewed relevant documents (evaluation reports, terms of
reference, agreements, budgets, financial statements) and interviewed
the SDC desk managers who commissioned and accompanied the evaluations
as well as the evaluators. For all 12 evaluations the list of criteria
was worked through, the findings filled in fact sheets and then compared
synthetically for each evaluation standard.