ICVET Promoting Emerging Practice, TAFE NSW International Centre for VET Teaching and Learning

Home

• Workplace Learning
• Recognition of Prior Learning
• Partnerships
• Workforce Development
• Capability Development
• Life Based Learning

eZine Back Issues A-Z ResourcesAbout Us

eZine | RSS Help

• A-B
• C-D
• E-F
• G-I
• J-N
• O
• P-S
• T-Z

or select another topic at the bottom of the page

VET Research Links

Professional Development Evaluation – Models and tools

Reviewed: January 2008

This page has been kept for reference. 
It will no longer be updated by ICVET.

• Program stages and evaluation questions
• Some models and tools of professional development evaluation
  • Donald Kirkpatrick and Thomas Gusky
  • Anona Armstrong
  • Zita Unger and Anthea Rutter
  • Return on Investment
  • Program Logic
• The Most Significant Change technique
• Appreciative Enquiry
• Websites
• Publications
• References

This page follows on from Professional Development Evaluation which provides definitions, concepts and purposes of PD evaluation.

Program stages and evaluation questions

In general, programs and projects are developed through three identifiable stages. Most PD evaluation models include these stages in some way. At each stage specific evaluation questions can be asked before proceeding to the next stage. These stages and questions are:

Design

• Does the program/project meet the needs of the stakeholders and the customers?
• Do we know who the stakeholders are, their relationship to the program, their needs and concerns?
• Is it feasible and viable?
• Have other solutions been considered?

Implementation

• Have clear targets been set and are they being achieved?
• Is there an information system and does it provide regular reporting on inputs and outputs?
• Are the program/project objectives still relevant?
• Are objectives likely to be met? If not, what can be done?

Impact

• Was the program/project completed and were participants satisfied?
• Did the program/project meet the objectives?
• What were the medium and longer term impacts?
• How worthwhile and valuable was this? Was it value for money? Was there a reasonable return on investment?

Some models and tools of professional development evaluation

Some models and tools have been devised specifically for evaluating PD, others are general evaluation approaches that could usefully be used for PD.

Donald Kirkpatrick and Thomas Gusky

Donald Kirkpatrick set out his four-level approach to the evaluation of training in a series of articles in the journal then known as the American Society of Training Directors between November 1959 and February of 1960.

The world of evaluation is vast and rich, and is becoming more vast and richer each year. The evaluation universe, like the physical one, is still expanding.

Patton (1996: p142)

The four-level evaluation framework has survived 30 years of turbulence in training and development evaluation, and remains the basis of many frameworks in training evaluation. One might even say that it has prospered. The four elements of the framework are defined below using Kirkpatrick's original definitions. Thomas Gusky, in his book Evaluating Professional Development (2000) has further expanded this model with sample questionnaires and proformas. His focus is on PD in educational organisations whereas Kirkpatrick looks at organisations in general. Both assume that the PD is primarily training event focussed.

The first three elements centre on the trainees - their reactions, their learning, and changes in their behaviour. The fourth element shifts to a concern with organisational benefits or business results. There is an assumption that the results of training and development programs are more than simply equipping people to do their jobs. The logic is that there will be an effect on business results and that this effect can be evaluated. As Nickols (1999) states:

Anona Armstrong

Associate Professor Anona Armstrong, (School of Management, Faculty of Business and Law, Victoria University), has specialised in the evaluation of public sector performance, in particular organisational development and change. She has written extensively about the strategic evaluation of training and development. Her model, based on Kirkpatrick, is summarised below.

Stages in Training Program Design, Delivery and Evaluation

Stage in Program Design	Evaluation Issues	Evaluation Procedures	Evaluation Product
Need for training exists	Who has the need? What is the need? What are solutions? What are program objectives?	Needs assessment	Establish program objectives
Design of the program	What is best design - alternative? How should it be delivered?	Evaluation of alternatives Quality assurance	Produce program design Produce implementation plan
Training is delivered	How was the training delivered, (organised, content, presentation, resources, etc)?	Process evaluation	Improvement in program delivery
Results and impacts	Reaction Acquisition Behavioural intention Work behaviour Change in others Organisational changes Impacts	Performance evaluation	Demonstrate the worth of the program

Evaluation Schema

Evaluation Level	Data Collection Procedure	Criteria and Analysis
Reaction	‘Smile’ sheets	75% of students satisfied 93% average positive ratings
Learning Acquisition	Tests Observation checklists Work sample analysis	Before-after tests 80% mean test scores 4.0 improvement pre- to post-test
Behavioural Intention	Action plans	Plans formulated
Work Behaviour	Observation checklists Work sample analysis Superior - peer - subordinate reports	Achievement of training objectives 80% reported improvement Comparison of trainees and non-trainees Cohort analysis of trainees entering the program at different times Trainees’ performance at different locations Longitudinal analysis of trainees and non-participants’ performance, career path, etc
Changes in Others	Superior - peer - subordinate reports - interviews - surveys	80% reported changes
Organisational change	Organisational reviews - interviews - surveys - focus groups	80% reported changes
Performance	Monitoring – information systems: Outputs - workload Productivity - efficiency Outcomes - effectiveness Cost of outcomes Effectiveness surveys	Benchmarks - standards - targets Comparisons Quality analysis Utility analysis Customer satisfaction 80% Telephone response in 3 minutes

Zita Unger and Anthea Rutter

Drs Unger and Rutter, principals of Strategic Training Evaluation Pty Ltd, have extensive experience in the evaluation of training in private and public sectors.

They have developed an evaluation process that aligns training needs to business needs. The Strategic Training Evaluation Model (STEM) is based on determining the stakeholder information needs at each stage in the development of a training program, their different expectations and their perceptions of return-on-investment at each stage. Once this is done, the STEM process is used to help align training needs to business needs and to evaluate the contribution of training and development to achieving business objectives (ie return-on-investment). The table below outlines the process.

Strategic Training Evaluation Model (STEM)

Step	Key Question	Key Outcome
Environment Scan	Where are you at? Where do you want to be?	Identify performance gaps
Identify Needs	How do you know the answer is training?	Justify training needs
Training Strategy	Are training objectives aligned with business needs?	Develop training strategy
Return-On-Investment (ROI) Indicators	What results would convince you that training was effective?	Identify ROI Gain advocacy
Return-On-Investment (ROI) Report	How do you know that training was effective?	Measure performance: Reaction? Learning? Behaviour on-the-job? Impact?
Review	Have training needs been met?	Review process
Stakeholders Audience	WHO needs to know WHAT, WHEN?	Identify information needs

Each step in the Strategic Training Evaluation process asks a key question, which helps generate an important evaluation outcome. The questions that guide the evaluation process are based on an alignment of:

Return-on-Investment (ROI)

The notion of return-on-investment is important to help understand the diverse range of interests of stakeholders and to help determine what is at stake for them.

Richard Lincoln 1996

Lincoln suggests that each stakeholder has something quite specific in mind that defines success or failure – to them – of a program, often quite apart from issues about the effectiveness of a program to achieve stated objectives. This assumes that stakeholders make an investment of some kind in the training event. They may commit personnel, money, expertise, time, prestige, status or credibility ie, a resource or resources of value. The kind of outcomes therefore, that justify such investment in training are often not measurable in monetary terms. Strategic evaluation seeks to elicit their understandings.

In recognition of the different kinds of investment that stakeholders make, we distinguish between ROI (direct) to calculate the cost-benefit of training in dollar terms, and ROI (indirect) to take account of other benefits to the organisation, such as satisfaction rates. Return-on-investment, therefore, is understood broadly to take account of the different ways of assessing the value of training and measures of effectiveness. (Rutter & Unger:1997)

Stakeholders' information needs and ROI in the Training & Development Cycle

Stage in development of a T & D program	Stakeholders	Stakeholders' information needs
Training needs analysis	Clients Line managers Supervisors Teams HRM managers Training coordinators	Job/performance analysis Key result areas Strategic training plans
Development stage: Selection of suitable programs Methods of delivery Customisation of existing programs	Clients Line managers Subject matter experts Media specialists Instructional designers Facilitators HRM managers Training coordinators	Course content Materials: appropriateness difficulty interest challenge clarity Feedback on course delivery Preferred modes of delivery
Delivery stage: Training event Learning	Clients Line managers Facilitators Support staff Participants	Feedback on performance Participant satisfaction rates
Results	Clients Line managers Supervisors Administrators HRM managers Participants	Evidence of skill acquisition/upgrade Feedback on performance Participant satisfaction rates Delivery on time/budget Cost-benefit analysis Impact analysis

Program Logic

Program Logic is a tool for understanding a program's theory of action. It is useful, not because it is specific to the evaluation of learning and development, but rather for helping managers to tease out the specific evaluation issues that may need to be addressed when reviewing a training and development program or project.

Program logic is particularly useful in relation to current and emerging styles of professional development that are structured around work-based learning. It is recommended for its usefulness in developing an understanding of the critical elements in the workings of a program, and as a means of helping learning and development specialists to track the complex strands of their strategies.

The Program Logic model was initially developed in the mid 1980's by the Program Evaluation Unit in the NSW Public Service Board. The model described here is the work of Sue Funnell, who is acknowledged worldwide for having successfully used this approach to evaluate professional development and educative programs that incorporate a range of delivery methodologies and strategies.

Program logic starts with program analysis. This is the process of identifying and making explicit the logic of a program. A Program Logic Matrix is the framework used to create an explicit outline of the causal linkages among the components of the program. This is the first step towards being able to evaluate the program.

The diagram (link below) represents the key components of a typical program logic matrix.

The first column is always the Outcomes Hierarchy. It is the starting point for thinking about what the program is hoping to achieve. The hierarchy starts at the lowest level of activity associated with the program, ie outputs and then traces through various levels of results, ie intermediate outcomes, to the ultimate intention of the program, ie the ultimate outcome. The main objective of the hierarchy is to show the influences between outputs and outcomes. The outcomes hierarchy helps to tease out what needs to be in place first before the next step can be taken – the cause-effect and/or contingency relationships.

A program logic matrix generally takes from a few days to a week to develop depending on the complexity of the program. Most of the information that goes into the matrix can be found in the program documentation, plus one or two workshops with the program specialist/s and their staff, as well as some consultation with key stakeholders.

Program Logic Diagram (49KB) Explanation (single page diagrams for each column with a summary page at the end).

Program Logic Template (38KB) A blank template to use for planning.

The Most Significant Change technique (MSC)

The most significant change (MSC) technique is a different form of monitoring and evaluation where everyone judges and evaluates. This is called participatory evaluation because many project stakeholders are involved in identifying, analysing and learning from the different sorts of change that occurred. It can be successfully applied to PD to identify the critical success factors and program impact.

Essentially, the process involves the collection of significant change (SC) stories emanating from the field level, and the systematic selection of the most significant of these stories by panels of designated stakeholders or staff.

The designated staff and stakeholders are initially involved by searching for project impact. Once changes have been captured, various people sit down together, read the stories aloud and have regular and often in-depth discussions about the value of these reported changes. When the technique is implemented successfully, whole teams of people begin to focus their attention on program impact.

Appreciative Enquiry

Appreciative Inquiry (AI) is a system of asking questions that exposes the strengths of a system. In this way it could be applied to developing an understanding of the way in which a PD strategy relates to successful organisational change in the knowledge era.

In the recently published Reframing Evaluation Through Appreciative Inquiry Hallie Preskill and Tessie Catsambas draw on examples using this approach to frame, design, and conduct evaluations in various sectors worldwide. They also suggest how to apply AI to various aspects of evaluation, providing guidance on how to build evaluation capacity using AI.

 

Home | Top
copyright - disclaimer | privacy