Evaluating capability development in diverse and adaptive environments
THINK PIECE REVIEW | Robby Weatherley, TAFE NSW ICVET
This paper comprises extracts from the research report Life Based Learning: A strength based approach for capability development in vocational and technical education. The researchers are Maret Staron, Marie Jasinski and Robby Weatherley.
‘The more complex the organisation, the more difficult becomes meaningful evaluation; the more pressure for change, the less time for reflection. Yet it is precisely in these circumstances that mature reflection has become crucial to progress, for change is unlikely to be simple to achieve, and the way forward unlikely to be obvious to everyone concerned.’ (Rose and Haynes 1999).
Finding an evaluative process that is appropriate for capability development in the Knowledge Era is likely to be a difficult task. Is it possible to evaluate dynamic, complex, unpredictable, multifaceted, emergent processes where there is a shift from highly structured and linear professional learning and development to embedding enablers that support confident, capable, connected, curious and committed learners? If evaluation, in general, seeks to establish the value or worth of an activity or program, is it possible to evaluate capability development using traditional approaches?
Traditional approaches to evaluation
Many of the traditional approaches to evaluating professional development:
- are deficit-based rather than strength based
- are trainer directed rather than self-directed
- are event-focused, with clear boundaries between learning and doing
- require evidence of return on investment
Many publications, including Misko (2001), are devoted to return on investment (ROI) models in relation to training programs and activities. However an ROI model alone may not fit the premise of working and learning in the Knowledge Era. Andrews (2005) cites Tobin (1998) and Conner (2002), who articulate issues with ROI measures of learning. They suggest that as a traditional financial measure based on historic data ROI in education has been used primarily for self-justification rather than improvement. Factors limiting the contribution of ROI assessments include the interaction between training and other variables and the difficulty defining and measuring intangibles. Tobin notes that by the time an educator is asked for an ROI study to demonstrate the value added it is too late — the decision has already been made to continue, downgrade or eliminate, and the ROI is merely a justification for the foregone conclusion.
Andrews (2005) suggests that while Kirkpatrick’s four-level model of training evaluation remains highly influential and popular, there is an obvious delineation between learning and working inherent in the four evaluation levels. These are:
Reaction-level evaluation — measures trainees’ reaction to the training and development input.
Learning-level evaluation — measures whether the specified learning has occurred.
Behaviour-level evaluation — measures on the job performance.
Results-level evaluation — focuses on the outcomes of training and development, for example on productivity and profitability.
Larri (2001) describes a range of evaluative processes for professional development in addition to Kirkpatrick’s Four Level Evaluation. These include Stages in Training Program Design, Delivery and Evaluation (Armstrong), Strategic Training Evaluation Model (Unger and Rutter) and Program Logic. These processes (and others) are still relevant in determining results for various areas of professional development. However, we still need to identify evaluative processes that take into account capability development which impacts on multiple stakeholders that also work well in an environment characterised by uncertainty, with high levels of innovation, creativity and knowledge sharing.
Rich learning environments are holistic and inclusive, adaptive and shaping, and diverse and complex. Outcomes are distinctively different from efforts best measured by increments in individual productivity. Changes are desired at individual, team and organisational level to create sustainability and resilience.
Where organisational goals are multiple, contradictory, dynamic and political, evaluation difficulties are exacerbated. Andrews (2005) cites a number of authors including Michalski and Cousins (2002), Rose and Haynes (1999), Brown and Duguid (1991), and Brown and Reid (2002), who highlight issues around attributing causality, the difficulties of evaluation in complex organisations where knowledge workers learn in many ways, and evaluating for multiple stakeholder views in capability development programs/models.
Authors such as Brown and Reed (cited in Andrews) note the complexity of comprehensive change strategies and the inappropriateness of evaluation measures centred on individual change. These authors argue that when capability development aims to achieve organisational as well as individual outcomes, we must recognise the unfolding nature of developmental stages. Secondly, evaluation must heed the criticality of organisation permissions and supportive social structures in addition to individuals’ skills attainment. These researchers argue that a focus on capacity building connotes simultaneous development on multiple levels. There is a clear conceptual linkage between these ideas and the learning ecology metaphor.
Further insights are provided by Bassi (cited in Andrews 2006), who also argues that accounting and reporting systems designed to measure Industrial Era wealth are inappropriate for Knowledge Era activities. Specifically, Bassi suggests that because human capital is the only asset that cannot be owned by an organisation, management faces a paradox. How can efficiency measures be mandated when the contributions of Knowledge Era workers are essentially discretionary? The challenge is to focus on efficiency and productivity (Industrial Era concepts) whilst simultaneously engaging the passion, creativity, loyalty and best efforts of the people on whom an organisation relies.
In reality the knowledge workers themselves control both inputs and outputs. Andrews cites Drucker, who echoes Bassi’s observations and concludes that in the Knowledge Era people must be led and managed differently, with businesses conceptualising themselves in an entirely new way; employees are assets to be valued, rather than costs.
A fresh approach to evaluation and improvement
A fresh and distinctively different approach to capability development demands a fresh approach to evaluation. The following two approaches, Appreciative Inquiry (AI) and Most Significant Change (MSC), take into account simultaneous development on multiple levels. They recognise multiple stakeholders (organisation, team or unit, and the individual), all of whom will have different expectations of evaluation and use evaluation measures in different ways.
It is important to remember the words of Patton (cited by Andrews 2005), that ‘The world of evaluation is vast and rich, and is becoming more vast and richer each year. The evaluation universe, like the physical one, is still expanding’.
As in the Strength Based Strategies section (see Part 6.3 of the Life Based Learning Project report) it is important that the evaluation and monitoring processes are embedded within the theoretical understanding of life based learning.
Appreciative Inquiry (AI)
AI has been used to embed self-evaluation by discovering an organisation’s own best practice and allowing all practitioners to assess their own performance against that benchmark (Elliott in Andrews 2005). Preskill and Coghlan (2004) recommend that using an AI approach to evaluation is particularly useful to accelerate change, build communities and overcome scepticism. In contrast with ROI measures, evaluation based on AI is unfolding, engages capability development participants, and focuses on improving and accelerating outcomes.
An AI approach to evaluation comprises four phases, as shown in Figure 14.
Figure 14: The 4D Model of AI Evaluation (Preskill & Coghlan 2004)
The phases and illustrative focus questions, adapted from Preskill and Coghlan (2004) are summarised as follows:
Discovery - identifying the successes and strengths:
- What has been my/our most powerful development experience during the last period?
- What were the nurturing conditions/enablers that produced these outstanding outcomes?
- Where and what are the wellsprings of PD in this environment?
- If I/we had three wishes for my/our PD going forward, what would they be?
Dreaming - visualising the opportunities:
- What can I/we do to replicate the outstanding experiences described?
- How can I/we better use the rich variety of enablers/resources to support the outcomes I/we wish for?
Designing - developing the architecture:
- What will I/we do to create the rich development experiences sought?
- What enablers will support me/us?
Delivery - implementing and experimenting.
The appropriate measure for rich, complex and emergent capability development processes should itself be rich, complex and emergent. AI-based evaluation fulfils this criterion and is both congruent with and applicable to working and learning in the Knowledge Era.
Most Significant Change (MSC
In a comprehensive guide to its use, The Most Signficiant Change (MSC) Technique (Davies and Dart 2005), is explained as a form of participatory monitoring and evaluation. It is participatory because many project stakeholders are involved in deciding the sorts of change to be recorded and in analysing the data. It is a form of monitoring because it occurs throughout the program cycle and provides information to help people manage the program. It contributes to evaluation because it provides data on impact and outcomes that can be used to help assess the performance of the program as a whole.
Davies and Dart describe the process as involving the collection of significant change (SC) stories emanating from the field level, and the systematic selection of the most significant of these stories by panels of designated stakeholders or staff. The designated staff and stakeholders are initially involved by ‘searching’ for project impact. Once changes have been captured, various people sit down together, read the stories aloud and have regular and often in-depth discussions about the value of these reported changes. When the technique is implemented successfully, whole teams of people begin to focus their attention on program impact.
This technique was initially developed for organisational learning in non-government aid organisations by Rick Davies (1996); since then it has been used with other theoretical perspectives, including AI. MSC is a strength based approach that looks at what works and determines how to do more of what works.
MSC is an emerging technique and can be adapted for different situations. Its 10 steps are:
- How to start and raise interest
- Defining the domains of change
- Defining the reporting period
- Collecting SC stories
- Selecting the most significant of the stories
- Feeding back the results of the selection process
- Verification of stories
- Quantification
- Secondary analysis and meta-monitoring
- Revising the system
Why use the MSC technique?
Davies and Dart list a wide range of reasons why organisations have found the MSC technique useful. These include:
- It is a good means of identifying unexpected changes.
- It is a good way to clearly identify the values that prevail in an organisation and to have a practical discussion about which of those values are the most important. This happens when people think through and discuss which of the significant changes is the most significant. This can happen at all levels of the organisation.
- It is a participatory form of monitoring that requires no special professional skills. Compared to other monitoring approaches, it is easy to communicate across cultures. There is no need to explain what an indicator is. Everyone can tell stories about events they think were important.
- It encourages analysis as well as data collection because people have to explain why they believe one change is more important than another.
- It can build staff capacity in analysing data and conceptualising impact.
- It can deliver a rich picture of what is happening, rather than an overly simplified picture where organisational, social and economic developments are reduced to a single number.
- It can be used to monitor and evaluate bottom-up initiatives that do not have predefined outcomes against which to evaluate (Davies & Dart 2005, p12).
MSC is more suited to monitoring that focuses on learning rather than just accountability.
The above approaches provide a guide to the evaluation for capability development. They are suggested strategies only and need to be applied in the context of the life based learning model.
References
ANDREWS, K 2005, Evaluating Professional Development in the Knowledge Era, TAFE NSW ICVET (International Centre for VET Teaching and Learning), Sydney
COOPERRIDER, D L & Whitney, D 2002, Appreciative Inquiry: The Handbook, Lakeshore Publishers, Euclid Ohio
DAVIES, R & Dart, J 2005, The ‘most significant change’ (MSC) technique: a guide to its use.
GUBA, E G (Ed) 1990, The Paradigm Dialogue, Sage Publications, Newbury Park
KIRKPATRICK, D L 1998, Evaluating training programs: the four levels, 2nd edition, Berett-Koehler Publishers, Inc, San Francisco
LARRI, L 2001, Evaluation Manual, Professional Development Network, NSW Department of Education and Training, Sydney
MORSE, J M 1994,‘Designing Funded Qualitative Research’ in Denzin, N K & Lincoln, Y S (Eds) The Handbook of Qualitative Research, Chapter 13, Sage Publications, London
NCVER ( National Centre for Vocational Education and Research) 2001, Getting to grips with returns on investment in training
PRESKILL, H & Coghlan A T (Eds) 2004, Appreciative Inquiry in Evaluation: New Directions for Evaluation, Jossey-Bass
ROSE, J & Haynes, M 1999, A soft systems approach to evaluation for complex interventions in the public sector, Journal of Applied Management Studies 8(2)
Also see
Life based learning: a strength based approach to capability development in vocational and technical education
Final report on the national research project Designing Professional Development for the Knowledge Era conducted by TAFE NSW ICVET with Department of Education, Science and Training (DEST)
Executive Summary
Full Report
STARON, Maret, Jasinski, Marie & Weatherley, R 2006, A Business Approach to Capability Development: Considerations and suggestions for applying Life Based learning in the Workplace, TAFE NSW ICVET with Department of Education, Science and Training (DEST)
This is a companion document to the research report above that will assist individuals, teams or organisations to apply the life based learning model
Life based learning for capability development in the Knowledge Era | A-Z Resources
Evaluation | A-Z Resources
Professional Development Evaluation | A-Z Resources
Professional Development Evaluation – Models and tools | A-Z Resources
Appreciative Inquiry | A-Z Resources