Professional Development Evaluation: A-Z Resources

Professional Development Evaluation

Reviewed: January 2008

This page has been kept for reference.
It will no longer be updated by ICVET.

How does evaluation relate to professional development activities in organisations?

What is evaluation?
Definitions of evaluation
Characteristics of evaluation
Purposes of evaluation
PD in the knowledge economy
Building an evaluative culture
Websites
References

Evaluating Professional Development – Models and tools follows on from this page, looking at stages of professional development and linking to the thinking, models and tools of prominent evaluation experts.

What is evaluation?

First, some basic concepts. Many of us are familiar with the concepts of formative and summative points in the life of a training activity evaluation. However current directions in evaluation methodologies stress a broader understanding of the place of evaluation.

The world of evaluation is vast and rich, and is becoming more vast and richer each year. The evaluation universe, like the physical one, is still expanding.

Patton (1996: p142)

Definitions of evaluation

There are a number of different definitions of the term ‘evaluation’. These are representative of some agreed understandings. (Nagarajan & Vanheukelen:1997)

Evaluation – an in-depth study which takes place at a discrete point in time, in which recognised research procedures are used in a systematic and analytically defensible fashion to form a judgement on the value of an intervention.

Formative evaluation – an evaluation concerned with examining ways of improving or enhancing the implementation and management of interventions. Formative evaluations tend to be conducted for the benefit of those managing the intervention with the intention of improving their work.

Summative evaluation – is concerned with determining the essential effectiveness of programs. Summative evaluations tend to be conducted for the benefit of external actors (groups who are not directly involved in the management of a program), for reasons of accountability or to assist in the allocation of budgetary resources.

Characteristics of evaluation

While definitions may vary there is general agreement that evaluations should have these characteristics or elements:

analytical – based on recognised research techniques
systematic – carefully planned and using chosen techniques consistently
objective– where the evaluator is as neutral as possible and avoids bias, values and/or prejudice
valid– internally valid because the causal link between the intervention and the observed effects is certain; and externally valid because the conclusions about the intervention can be generalised and applied to other people, settings and times
reliable – able to have findings that are reproducible by a different evaluator with access to same (or similar) context and using the same or similar methods of data analysis
issue-oriented – address important issues relating to the program, including its relevance, efficiency and effectiveness
user-driven – the design and implementation of the evaluation should provide useful information to decision- makers.

Purposes of evaluation

In general, evaluation seeks to establish the value or worth of an activity.

Evaluation can be done in an ad-hoc manner with intermittent data collection with no comparative data or it can be planned so as to provide feedback on performance to different stakeholders. It is most successful when it is integrated with strategic planning, involves people at all levels and provides feedback on individual and organisation performance.

Armstrong (December 1996)

Evaluation often serves other purposes including, but not limited to, the following:

Generating general knowledge about and principles of program effectiveness
Developing programs and organisations
Focusing management efforts
Creating learning organisations
Empowering participants
Directly supporting and enhancing program interventions (by fully integrating evaluation into the intervention)
Stimulating critical reflection on the path to more enlightened practice

PD evaluation in the knowledge era

Professional development (PD) has changed from the classic training situation where the action involves an intensive short course model with participants, a trainer and classroom instruction, to a range of strategies such as coaching on the job, mentoring, outsourcing to external providers, multimedia delivery, flexible computer-based learning and formal courses of study.

PD researchers and evaluators have recognised this change and its implications.

Strategic training evaluators, Zita Unger and Anthea Rutter (1997) wrote,

'Not only has the training event itself changed, but the key players involved have also expanded. The work of people responsible for managing training often remains invisible to the rest of the organisation.

One possible reason is that information about the training function does not reach the right people, or does not reach them in a form that is useful or of interest to them. The effect of which is that active support (advocacy) for the training function, or particular training programs, is not forthcoming, and that training is seen as a cost to the organisation, rather than as a real investment in people. People in training are most often attuned to the learning outcomes – such as ‘did the learners achieve the training objectives?’ – and tend to over-report this kind of information. Clearly, the evaluation process needs to support a way of addressing the information-needs of stakeholders.'

It is clear that the role of the learning and development specialist in organisations has become complex and multi-faceted, incorporating the management of the function as well as providing a service to the organisation. The service role is often defined as an internal consultant and may include:

The design and conduct of courses and customisation of programs to meet clients’ needs (ie training delivery and its relationship to work organisation)

Facilitating innovation and knowledge management through individual and team learning (ie learning process management)

Facilitating change and development for part or whole of organisation (ie organisational change management).

Building an evaluative culture

Most PD evaluation is done by staff members from within the PD function or another unit (eg Planning and Evaluation). This is called internal evaluation (compared to external evaluation where an independent evaluator is engaged to conduct the research).

The capacity of an organisation to conduct evaluation that contributes to organisational development underpins process-based quality management systems. Quality management system AS/NZS ISO 9000:2000 is the standard that relates to quality systems in organisations. In particular:

2.9 Continual improvement
f) measuring, verifying, analysing and evaluating results of the implementation to determine that the objectives have been met

Evaluative culture is a concept that helps us to describe the way an organisation develops its capacity to manage meaningful internal analysis.

The Canadian evaluation expert, Arnold Love, has identified the stages that organisations go through in developing their capability to undertake internal evaluations. He says, ‘achieving a particular stage depends largely on developing adequate capability at the previous stages.’

It is possible to assess the stage of development for your organisation and determine how best to build capacity towards the next stage.

Key factors which contribute to developing effective internal evaluation capability are:

top management support for evaluation
positive leadership about evaluation
an organisational culture that supports continual learning and critical program review for decision-making
publicising the results of evaluations and using evaluations for program improvement

Arnold Love’s Stages of Evolution of Internal Evaluation Capability

Stage 1 – Ad hoc Evaluation
Managers see internal evaluation as useful, but use it on an ad-hoc basis to provide information to managers or project teams.

Stage 2 – Systematic Evaluation
The organisation supports identifying and collecting evaluation information across its programs. Evaluations of a descriptive and monitoring nature take place. At this stage the organisation uses systematic internal evaluation to help managers understand the processes that impact on the organisation’s activities. Evaluation techniques, therefore, concentrate on operations. Methods focus on assessing client needs, client satisfaction, service utilisation, program logic, performance measurement, information systems, continuous quality improvement, and self-assessment against established standards.

Stage 3 – Goal Evaluation
Once the organisation has the capacity to collect descriptive data (Stage 2), the next step is collecting comparative information about the planned versus the actual attainment of goals. This step requires an organisational structure that supports establishing goals, negotiating realistic goals among stakeholder groups, setting priorities, and designing data collection and reporting systems.

Stage 4 – Effectiveness Evaluation
To reach this stage, the organisation must be able to define effectiveness criteria and methods for measuring whether the criteria were attained. Techniques used may be quantitative, qualitative, or both. Evaluation designs range from case studies, to group designs, to time-series analysis.

Stage 5 – Efficiency Evaluation
At this stage the organisation evolves accounting, financial, and information systems using a common metric across programs (for example, cost per change in level of functioning). Effectiveness information (Stage 4) is used together with efficiency information to enable the organisation to attain high levels of efficiency, without substantially reducing the effectiveness and quality of the organisation’s programs.

Stage 6 – Strategic Benefit Evaluation
At this stage, managers assess the social costs and benefits of their services. To do this, they must integrate information from the external environment with accurate information about the organisation’s internal functioning. Internal evaluation becomes a strategic tool for flexibly selecting strategies that steer the organisation as it rides the tidal wave of change, by choosing the programs and activities that have the highest ratio of benefits to costs.

Websites

The Australian Organisation for Quality Inc (AOQ)

AOQ is Australia's peak professional body for quality practitioners and anyone interested in continual improvement in business leading to excellence in products and services. Each year they conduct an awards process that recognises business excellence. One of the eight areas for assessment, Factual Approach to Decision Making (PDF) relates to evaluation capability.

References

ARMSTRONG, A 1996, The value of evaluating training and staff development, Evaluation News & Comment, Vol 5, No 2

ARNOLD, Love 1991, Internal evaluation: Building organisations from within, Sage, California

NAGARAJAN, Nigel & Vanheukelen, Marc 1997, Evaluating EU Expenditure Programmes: A Guide, Ex post and Intermediate evaluation, European Commission, pp 92, 94, 102

ICVET Promoting Emerging Practice, TAFE NSW International Centre for VET Teaching and Learning