Evaluation
Reviewed and Updated: January 2008
This page has been kept for reference.
It will no longer be updated by ICVET
This century is characterised by demand for ‘return on investment’ in the vocational education and training sector. RTOs are being pressured to operate as profitable businesses. As a result, they are now looking at the evaluation of training not just in terms of whether the participants were satisfied with the experience, but also with the question of how well the training met the needs of the sponsoring organization and/or industry.
Kirkpatrick’s training evaluation model was originally developed in 1959 but is still one of the simplest and best around. The four levels of Kirkpatrick's evaluation model essentially measure:
reaction of student - what they thought and felt about the training
learning - the resulting increase in knowledge or capability
behaviour - extent of behaviour and capability improvement and implementation/application
results - the effects on the business or environment resulting from the trainee's performance
Whether you choose to use the Kirkpatrick model or another, there is general agreement that evaluations should be:
- analytical – based on recognised research techniques
- systematic – carefully planned and using chosen techniques consistently
- objective – where the evaluator is as neutral as possible and avoids bias, values and/or prejudice
- valid – internally valid because the causal link between the intervention and the observed effects is certain; and externally valid because the conclusions about the intervention can be generalised and applied to other people, settings and times
- reliable – able to have findings that are reproducible by a different evaluator with access to same (or similar) context and using the same or similar methods of data analysis
- issue-oriented – address important issues relating to the program, including its relevance, efficiency and effectiveness
- user-driven – the design and implementation of the evaluation should provide useful information to decision-makers.
Websites
Australasian Evaluation Society
This website contains contacts, resources and upcoming events.
Practical Assessment, Research and Evaluation
Practical Assessment, Research and Evaluation (PARE) is an on-line journal with the purpose of providing education professionals access to refereed articles that can have a positive impact on assessment, research, evaluation, and teaching practice.
Resources for Methods in Evaluation and Social Research
This page lists FREE resources for methods in evaluation and social research. The focus is on 'how-to' do evaluation research and the methods used: surveys, focus groups, sampling, interviews, and other methods. Most of these links are to resources that can be read over the web.
Wikipedia - Evaluation
Wikipedia overview of Evaluation and links to related entries and websites.
Publications
BRAGG, DD 1995, Working together to evaluate training. Performance & Instruction 34:10, 195-199.
Although a little dated, Bragg’s article still offers some good
ideas that are worth considering.
ESERYELl, Deniz 2002, Approaches to evaluating training Educational Technology & Society 5 (2).
There is an ongoing debate in the field of evaluation about which approach is best to facilitate the processes involved. This article reviews current approaches to evaluation of training both in theory and in practice. Particular attention is paid to the complexities associated with evaluation practice and whether these are addressed in the theory. Furthermore, possible means of expediting the performance of evaluations and expanding the range and precision of data collection using automated systems are discussed. Recommendations for further research are also discussed.
FETTERMAN, D M, Kaftarian, S J & Wandersman, A 1996, An Empowerment Evaluation, Sage Publications, California.
This method of evaluation is particularly useful in reducing the resistance traditionally associated with evaluation.
GUBA, E G & Lincoln, Y S 1989, Fourth Generation Evaluation, Sage Publications, California.
Fourth Generation Evaluation provides an alternative approach to evaluation in that it emphasises the importance of the human, political, social and cultural context of those taking part in the evaluation.
KAUFMAN, R, Keller, J & Watkins, R 1995. What works and what doesn't: Evaluation beyond Kirkpatrick. Performance & Instruction 35:2, 205-209.
Rather than criticise Kirkpatrick’s model, this article extends his work and offers some real solutions to real problems.
KIRKPATRICK, D L 1998, The four levels: An overview. Ch. 3 in Evaluating Training Programs: The Four Levels, 2nd edn. Berrett-Koehler, San Francisco.
This is an introduction to Kirkpatrick’s model by the man himself. In it, he provides rationale for the approach in terms of resolving some of the age-old issues associated with evaluation.
KIRKPATRICK, D L 1998, Evaluating Training Programs, Berrett-Koehleer, San Francisco.
Kirkpatrick’s four level model of evaluation is probably the most widely used approach. In this book, he elaborates on his original work with case studies on its application.
NEWBY, Tony 1992, Training Evaluation Handbook. San DiegoCA Pfeiffer.
A Training Evaluation Handbook focusing evaluation on the trainer. It is useful in that it provides ready to use checklists and inventories.
See Also
Evaluating capability development in diverse and adaptive environments | eZine February 2007