List of Materials:
eot-dlm-01 Part 1 - Introduction to Eot
eot-dlm-02 Part 2 - Models for Eot
eot-dlm-guide Course Guide
eot-dlm-intro Introduction to Distance Learning
eot-dlm-list List of Materials
Introduction:
Organisations in the public sector and funding agencies, such as the DoPT, use considerable resources to enable people to learn. There are a variety of reasons and motives for this, and it is likely to increase as new employees, technologies or working practices are introduced.
For progressive organisations, learning is considered an investment where, typically, 2-4% of a payroll is invested in training and development. Success for this investment is usually achieved by having a clearly defined policy, with an effective training function for its implementation. As with most forms of investment, the intention is to obtain benefit, both for the organisation and for its employees. Unfortunately, the management of some organisations see training simply as a cost. They fail to recognise the time and money being wasted by people engaged in ineffective learning. Management are usually well aware of the consequence’s poor performance, but do not always accept the link between this and having a well-managed, effective training function.
You might be fortunate to work for an organisation that sees benefit in investing in training and development. Alternatively, and less fortunately, you could be working for an organisation that needs convincing that training is a worthwhile investment. In either case, evaluation of training (EoT) will be an essential part of your work: it may also to be your responsibility.
There are different incentives to evaluating training, according to who is doing it and for what purpose. Trainers may face a conflict of interest in that evaluation of training may be a good way of demonstrating success. However, when the results of evaluation are less than satisfactory, it can provoke defensive criticism of the accuracy of the information being used, its relevance or to the adequacy of resources. Generally, trainers may have little incentive to evaluate training unless they also benefit from the process. EoT should therefore take into account incentives for:
• Funding agencies, who provide financial support and access to an increasing variety of resources. Here, the incentives for evaluation are to monitor implementation of government policy and to obtain information confirming effective use of available resources.
• Training Institutions use a major proportion of available funding to provide professional services to client organisations within the public sector. The existence and continuing development of a training institute, and therefore the incentive for evaluation, depends on being able to prove that resources are being used to the satisfaction of their clients.
• Client Organisations, whose performance is a primary concern of government, and the focal point for evaluation. However, unless there is a valid reason for Heads of Department’s and stakeholders to improve efficiency, quality or services to customers and beneficiaries, then there’s no reason to expect them to do so.
• People. An organisation is ‘an organised body of people with a particular purpose, such as a business or government department’. Therefore, training within a client organisation depends on people learning - and being able to evaluate both its process and outcomes.
The Government of India, along with other funding agencies, spend a great deal of money on training. It is mostly done through an extensive network of institutional training institutes, which serve national, state and departmental training needs. With the current emphasis on ‘value for money’, questions to be answered are therefore about whether such spending can be regarded as a good investment. If such questions are important - requiring an answer, then the issue is being able to show how investment in training benefits Indian public services, and the people for whom these services are provided.
When referring to ‘training’ we are not concerning ourselves solely with courses being run at training institutes: that is a simplistic and wholly inadequate view of the concept to which investment can be directed. With the introduction of distance learning and the policy of ‘training for all’, there is an immense potential to widen the concept and scope of training to realise far greater benefits from this investment. However, good investment practice requires trainers to be able to account for money spent and other resources used. By using EoT, you can justify investment and promote the use of training to improve performance. If you can’t show evidence of the benefits of training, why should the government continue to support it?
Definition of Training
A planned process to modify attitude, knowledge or skill behaviour through learning experience to achieve effective performance in an activity or range of activities. Its purpose, in the work situation, is to develop the abilities of the individual and to satisfy the current and future manpower needs of the organisation.
A great deal of money can be spent on training, but this doesn’t necessarily mean that it has been invested in the ‘planned process’, referred to in the above definition. For example, spending money building a training centre results in having a building, called a training centre: more money can be spent developing hostel facilities, or providing transport. Is this truly an investment in a planned process to improve performance?
Definition of Investment
The thing worth buying because it may be profitable or useful in the future.
Although this is how the Oxford English Dictionary defines investment, perhaps we should link the two definitions given so that we have a clear concept of an investment in training. It’s also worth remembering that our investors are not trainers, and they may have other equally appealing opportunities to spend their money. Another factor, especially at times of economic recession, is that investors are likely to seek evidence to justify their investment. Instead of being impressed by a tour of a training centre and seeing courses being run, they may adapt our definition of training to ask what might be difficult questions to answer. For example, based on the definition of training, above:
• What planned process has been used to ensure that the right people are attending your courses?
• How do your courses modify behaviour, attitude, knowledge, or skill - and how is this measured?
• How do you assess a learning experience, is it during the process of learning or as an outcome after attending a course?
• What evidence is available to show that people who attended your course can achieve effective performance, or at least show a significant improvement?
• How do your courses help to satisfy the current and future manpower needs - of a public service organisation?
Notice the reference to ‘you’ and ‘your’ - because it’s you who spent the investor’s money. You planned the process, helped people to learn and achieve effective performance. If you, or your director, or colleagues can’t answer these questions, can you claim the courses are a good investment? Might they be a waste of money? When trainers are competent and highly motivated professionally, it is in their interests to be able to answer such questions. These are trainers who investors are keen to support, because they are confident that money they invest will result in worthwhile benefits. However, investors are not just from funding agencies, and money is not the only form of investment. For example:
• People, who are willing to be trained, are prepared to invest their time and effort in relevant learning experiences. That is, they will do so - if they regard their investment to be of benefit to them, personally. If not, why should they bother wasting their time and personal inconvenience to attend a course that is of no value to them?
• Heads of departments, who are willing to contribute time, resources and support for a planned process of training to help improve performance. They will do so, if they believe their investment in training results in measurable organisational benefits. If not, what is the justification for sending key members of staff to attend courses that are of no benefit to the department, nor to the people being sent.
Rather than using the term ‘investment’ - implying a thing worth buying - let’s use ‘evaluation’, to give more emphasis to valuing what is being bought.
Definition of Evaluation of Training
The Oxford English Dictionary defines to ‘evaluate’ as: form an idea of the amount, number, or value of; assess. A more detailed definition used for the Evaluation of Training (EoT) is:
The assessment of the total value of a training system, training course, or programme, in social as well as financial terms. Evaluation differs from validation in that it attempts to assess the overall value of the course or programme, and not just the achievement of its laid-down objectives. The term is also used in the general judgmental sense of the continuous monitoring of a programme or of the training function as a whole.
Therefore, to evaluate training we have to form ideas to quantify and value a training process intended to achieve effective performance. However, who is ‘we’? Forming ideas can be done by anybody associated with training - for example: trainees, trainers, directors, managers, HoD’s, funding agencies and beneficiaries. In addition, considering the number of people to whom we have referred, there are likely to be many different ideas about the value of training. There will be some that are favourable and others that are critical - without necessarily being based on factual evidence. As a trainer, you are likely to encounter clients, stakeholders and funding agencies with a variety of ideas and opinions about the value of training - ranging from:
Enthusiasm - believes that training is the key to a successful career.
Supportive - keen to use training as a means to improve performance.
Critical - has had some poor training experiences.
Negative - sees training as a holiday, or a waste of time.
Indifference - does not attempt to value training and takes no positive initiatives.
By evaluating training ourselves, and encouraging others to do so, we can help people to value it as a worthwhile activity and one worth supporting. Rational opinions about training should be based on factual evidence about the process and outcomes of helping people and organisations to improve performance. If we, as training professionals, do not provide this evidence then we should not be surprised that people hold a variety of opinions. Also, bear in mind that some of these people will be senior officials, or funding agencies, with significant influence over investment policy.
Benefits of Evaluation
A survey reported that 80% of 450 organisations studied carry out only superficial evaluation of training. Often it is limited to ‘post-course questionnaires inviting trainees to comment on the course tutor, overhead slides and lunch’. As a result, most of these organisations do not know whether their investment in training is justified. However, according to the survey, the situation is changing due to a drive for improved business efficiency by senior management. This is resulting in moving towards more systematic evaluation of training.
The report lists the following benefits to be gained from evaluation:
• A better link between training and business strategy
• Improvement in training methods
• Better guidance on future training needs
• More credibility for training function
• Better able to prioritise training activities
• Improvement in trainers
• Better service to customers
• More senior management commitment to training
• Better control of training costs
• Improved staff morale
• Improved external credibility for organisation
As a trainer, you might become involved in the evaluation of your organisation’s investment in training. This will require you to have the systems to answer three basic questions:
1. Is the quality of training available your client organisation(s) satisfactory?
2. Does the provision of training result in improved performance?
3. Can your client organisation’s investment in training be justified?
The sequence of these questions is important, because your answer to the first question inevitably influences the answer to the next one. The questions are also closed requiring you, at least initially, to answer ‘Yes’ or ‘No’. If you were to answer yes, then the next question could be ‘How can you prove it’; if your answer is no, then the next question would be ‘Where are the problems - and what are you doing about them?
Key terms
To answer the three questions posed above, you will need to have a system developed around the following definitions:Internal Validation
‘A series of tests and assessments designed to ascertain whether a training programme has achieved the behavioural objectives specified’.
External Validation
‘A series of tests and assessments designed to ascertain whether the behavioural objectives of an internally valid training programme were realistically based on an accurate initial identification of training needs in relation to the criteria of effectiveness adopted by the organisation’.
Evaluation of Training
‘The assessment of the total value of a training system, training course or programme in social as well as financial terms. Evaluation differs from validation in that it attempts to assess the overall cost benefit of the course or programme, and not just the achievement of its laid-down objectives. The term is also used in the general judgmental sense of the continuous monitoring of a programme or of the training function as a whole’.
We illustrate the relationship between these three terms in Figure 1, below. It is essential to note that systems you develop for evaluating training need to incorporate procedures about the validity of training. This establishes the basis evaluating both training activities, and the training function as a whole.
Internal Validation:Have the specified objectives been achieved? External Validation: Has training solved the problem or improved job performance?Evaluation:Are the costs and benefits justified?
Training Performance
Fig.1
As an experienced trainer, you can assess whether people achieved the objectives specified - in other words, you will arrange to carry out internal validation. The results of this process will be to give you information about the quality of training being provided. This will enable you to plan further training, or make improvements to the training being provided.
Reasons for doing internal validation include it:
• Provides trainers with feedback about the training they provide
• Checks whether trainees have achieved the specified objectives
• Enables the quality of training to be monitored
• Indicates where the effectiveness of training can be improved
• Indicates where training can be more efficiently delivered
• Provides the basis for certification
Now consider the people you have trained, where they have achieved the objectives specified - based on the identification of their training needs. They will now return to their jobs and a process of external validation will now be used to ascertain whether they are now able to perform to a satisfactory standard. This process is likely to be done by the people concerned and their management.
Reasons for doing external validation include:
• Involving departmental management and stakeholders
• Focussing on actual performance
• Relating to identified training needs
• Establishing the basis for the transfer of learning from a course to job performance
• Indicating validity of specified training objectives
• Giving
• Essential data for cost benefit analysis
• Proving the benefits being obtained from training
Having completed the training and established its validity, both internally and externally, the organisation may now decide to evaluate their investment. This might be done by senior management, funding agencies, clients and stakeholders, or by customers or beneficiaries. Each will use what they consider appropriate criteria and it could be done with or without your involvement or knowledge. While doing this evaluation, it is likely that the results will also be used to evaluate the training function, as a whole.
Reasons for doing an evaluation of training include it:
• Recognises the importance of satisfying the needs of clients
• Provides justification for the investment in training
• Establishes the benefits of having a training function
• Encourages management to make further investment in training
• Enables training to be valued from different perspectives
• Includes cost benefit analysis
• Provides professional discipline for training management
• Encourages careful scrutiny of training proposals
Context for Evaluation
One extreme for evaluation of training is that it’s a random affair, using personal value judgements and without factual information. The other extreme is for every facet of training is measured, and the facts used to evaluate investment. Neither of these extremes offers a tenable basis for EoT.
1. Doing no evaluation may suggest that training is an act of faith that shouldn’t be judged in materialistic terms. Politicians and senior officials would not wish to say ‘no’ when asked if they believed in training the workforce.
2. Why waste time doing something nobody has asked to be done? Most trainers are happier helping people to learn and develop, rather than attempting to validate what they and their trainees accomplish.
3. Training institutions may be asked to justify themselves by their ‘output’ - the number of people trained, instead of the ‘outcome’ - what has been accomplished about improved performance.
4. EoT may discover things we don’t really want to know - lift the wrong stone and a scorpion might bite you. Some trainers, and training institutions are classified as ‘passive providers’, because they are content to offer a recurring menu of courses. They may seek feedback on their performance, but not on its performance-related outcome.
5. People may feel offended when accused of knowing the cost of everything and the value of nothing. Therefore, attempting to strike a sensible balance between seeking the costs of training and establishing value or benefit is likely to prove difficult.
6. Obtaining information about the costs and benefits of training is the start of a process of improving both the effectiveness of training and the efficiency with which it can be delivered. Therefore, EoT leads to decision-making about investment, strategy, quality, utility and professional development.
7. Perhaps a justifiable reason for not doing EoT is the absence of TNA. Without details of performance problems and training needs, it is unlikely that you’ll be able to link training to organisational performance.
Although EoT can be an uncomfortable experience, it is an essential feature of the systematic approach to training. It enables trainers to justify their existence and the professional services they offer. It is also psychologically rewarding to know that you are helping people to learn, organisations to prosper, and beneficiaries to benefit. Questions for you to deal with concern:
• What system and procedures should be used?
• Which techniques and tools are suitable?
• Who are the stakeholders?
• What should be done with the information obtained?
Inevitably, you will experience difficulties when evaluating training. Some difficulties can be attributed to common factors, which can be dealt with by management action and further development of institutionalised systems. Other factors are perhaps unique to your role and your organisation. Some of these factors can be regarded a ‘self-imposed’ - difficulties you can tackle personally - or they are ‘externally-imposed’, requiring action by others. Here, we will consider a selection of common factors that create difficulties for trainers when evaluating training.
Timescale
If evaluation of the benefits of training is to take place at the level of job performance or above, it cannot be done immediately after the training has taken place. Sufficient time needs to be allowed for the changes to take effect. There is a conflict between, on the one hand, the need to allow time for the effects of training to become apparent, and on the other, for evaluation to take place as soon as possible after training to minimize the effects of contaminating factors on the results.
Contamination
Contamination refers to the effect of factors other than training on the outcomes achieved. From the training evaluator’s point of view, these other factors contaminate the results of evaluation. We must be careful not to assume that any improvement occurring after training is the result of training or, alternatively, that a failure to improve after training is conclusive evidence that the training was wrongly designed or delivered. We need to be aware of other factors, which may have a positive or negative effect on the results of training. The results of this evaluation could be contaminated by:
• Changes in business objectives.
• Changes in technologies and systems.
• Leadership style of managers.
• Changes in work methods or culture.
• Conflict between implementation of training and non-training initiatives.
Other potential contaminating factors can be: work group pressures, the demands of clients and the self-motivation of the individual trainees. Although contaminating factors create problems and difficulties in evaluation, they may be important in producing desirable results for the organisation. It is often essential that management, when faced with a problem, does not select one solution - such as training. Instead, several solutions, or interventions may be implemented simultaneously. For example, to improve the effectiveness of supervision, the following approaches might be chosen:
• Provide training for supervisors
• Review selection criteria and procedures
• Increase the number of meetings between supervisors and their managers
• Review the content and priority areas of supervisory jobs
• Increase delegation to the supervisors.
In this example, we see training in its proper context as an integral part of management in the organisation, not as a separate activity on its own. Nevertheless, we should take account of contamination, although in many situations we need to learn to live with it rather than try to eliminate it.
Management Support
If you are to persuade senior officers and stakeholders to give you their full support, you will need to work closely with them. This is because setting up procedures for validation and evaluation requires more than their consent for you to take action. You will rely on them to dedicate time and effort to:
• Provide you with the resources needed to undertake an EoT strategy.
• Provide you with criteria to judge the corporate effectiveness of training.
• Agree with you the levels at which evaluation will take place.
• Arrange for the performance data, which you will need as the basis for the evaluation.
• Discuss and agree action to be taken in response to the findings of evaluation.
• Arrange the provision of regular information and constructive views when their trainees’ post-training results are being followed up.
• Integrate evaluation measures with other personnel systems within the organisation, such as appraisals, performance-based awards, management by objectives and so on.
An important part of working with management is the ability to explain your proposed course of action in a positive manner, responding creatively to any objections. In explaining how you would like to proceed in evaluating training, you will need to think of possible objections and of ways in which these can be countered.
1. Here you should point out the importance of a timescale. The effects of training do not become apparent immediately after it is completed. If we are to evaluate changes resulting from training at the levels of job performance or the operation of a department, we cannot do so immediately after training. We need to allow sufficient time for the changes to take effect.
2. You should explain to management about the nature of competence and skills. It is vitally important to evaluate competence in using skills required in the job, as distinct from simply having knowledge. When these skills are productive rather than reproductive, a large element of subjective judgement of the trainee’s competence is involved.
3. The issue here is contamination. In measuring the effects of training, we must be careful to measure things, which can properly be attributed to training and not to other factors. Many other possible influences could affect job performance, positively or negatively.
Performance Vs Resources
Figure 2 shows two axes illustrating two factors critical to the evaluation of training. As mentioned in the definition, effective performance - shown on the vertical axis - is the expected outcome of the training process: we are using the horizontal axis to show investment of appropriate resources. Let’s consider these factors separately, also in the context of training in the public sector. PerformanceResources
Fig. 2
Performance Parameters
Training Needs Analysis
The government is seeking to improve performance across organisations in the public sector - from higher agricultural yields, to increased traffic on the railways, to improved primary health services, better tea quality. These are just a few general examples; no doubt, you could provide more that are specific to your organisation. Training Needs Analysis provides the link between organisational performance and training - TNA consultancies are based on three key factors that affects performance - ‘environmental factors’; ‘motivational factors’; and ‘behavioural factors’.
As trainers, we can only help organisations in the third of these factors - helping people to develop knowledge and skills relevant to their needs. However, there’s no doubt that the other two factors are important when seeking to improve performance. You may therefore have a situation where, having provided effective training, no improvement in performance is achieved - due to flaws in the working environment, or poor motivation by the people concerned. The risk is that such failures are attributed to poor training. This is especially so if you have no evidence to show that the people trained acquired relevant knowledge and skills, but were unable to use them because of other factors. Although we are concerned here with EoT, it will be more effective if done in collaboration with TNA.
Standards
Although we have used the term ‘performance’, it has little meaning unless it is closely associated with standards. These provide criteria to judge either the process or the product of a person’s performance, or that of an organisation as a whole - without them it is difficult to achieve, or to measure effective performance. A typical performance standard that can be used for training purposes provides the following information:
Performance Objective What a person is doing in clear, unambiguous terms that feature one activity-related verb.
Performance Criteria Significant features of a person’s performance that determines their competence.
It is important that the performance standards agreed are realistic and achievable and, having agreed them, assessment measures can be agreed with a client and stakeholders. These specify the tools or techniques used to measure a person’s performance. TNA will often highlight the lack of clear standards where failure to have them may be a major cause of poor performance. For effective evaluation it is also essential to establish performance criteria to provide the basis for suitable measures of assessment
Resource Parameters
Referring to Figure 2, the other axis is labelled ‘resources’ to indicate financial and other resource investment implications needed to implement training. As the government moves towards a policy of ‘value for money’, far greater emphasis will be given to justifying this. An emerging feature of evaluation is likely to be for departments and funding agencies to closely examine resource implications required to support training proposals, and the outcomes expected from institutional course provision. In addition, as distance and open learning is introduced, funding allocation could be diverted from training institutes towards departmental training functions. Consider some resource implications that are likely to be taken into account:
Costs
These could be institutional costs required to run a course, or a breakdown of the costs of running a training institute. Evaluation is likely to examine costs per course participant - with the implications that such costs should be justified and perhaps reduced. With the increasing use of generic training packages, it will be possible to introduce standardised costs for national calendar courses - with these costs linked to EoT.
Time
Absence from the workplace to attend training courses costs money and possibly reduces operational effectiveness while staff are away. Can this be justified? Time for training and development has the potential to provide real benefits, but only if it is invested wisely - taking into account both the needs of the individual and those of the employing organisation.
Facilities
The main commitments are the provision of training institutions. At present these absorb most funding support, although this may change following the introduction of distance and open learning, also with the government’s policy of providing ‘training for all’. An evaluation of institutional facilities is likely to focus on indirect costs of running them, and the utilisation made of them. Most public sector training institutions feature large modern buildings on valuable sites. Considerable infrastructure resources are needed to provide training facilities - administration, amortization, hostels, transport, maintenance, staff quarters, etc. These are mostly fixed costs that have to be balanced against their professional use.
As with hotel bedrooms, unless training rooms have a high level of occupancy it becomes difficult to justify their continuing availability - or indeed the existence of an institution. The comparison with hotels is of relevance because if bedroom occupancy levels fall much below 60%, management is likely to be changed, leading to closure if utilisation cannot be improved. We have chosen hotels as an illustration because many hotels now offer training facilities on a commercial basis. So, as part of evaluation of facilities, consideration might be given to using commercial facilities, as an alternative venue for certain courses.
Faculty
If a training institution has only one member of faculty, then its capability is limited, irrespective of other resources that are available. However, if a training institution has several tutors, then it has greater potential to run courses and offer other professional services. A legitimate and important area for evaluation therefore concerns the utilisation of available training personnel. This could focus on the hours a tutor spends running courses - ‘contact time’, or their deployment to provide professional services for client organisations. These services may include carrying out TNA consultancies, developing training materials - or evaluating training.
Figure 3, below, builds on our consideration of the basic axis, showing a desired standard of performance and a lower level of performance as a consequence of no training. EoT should examine both issues.
PerformanceResources
Fig.3
Desired Standard of Performance
An apposite point to make is that if you don’t know where you are going how do you know you’ve got there. Consider an example faced by trainers involved in Information Technology. A training institution has used its resources to build and equip an ‘IT Centre’, with modern computers and competent trainers. They are asked to run a series of 5-day ‘IT Appreciation’ courses. These courses are listed in the national calendar and people from a variety of public sector organisations are nominated to attend. Although such courses may meet a genuine need, a wider concern, for EoT, would be evidence of specific training needs, and a design brief that includes agreed standards of performance.
Another factor could be disparity between what is ‘desired’ by departmental management and what training institutions can achieve. A 5-day course gives only a limited opportunity to learn about IT so, although management and prospective trainees may desire many things, an institute should offer only what can be delivered. The design of training requires careful consideration of constraints that limit what can be achieved - in this instance, a lack of time and the likelihood of differing training needs. A legitimate aim for EoT would be to establish where course participants are ‘going’, and what evidence is available to show they’ve got there.
Failure to limit expectations to what can be realistically achieved may lead to a large number of dissatisfied people. Former trainees might complain that the course didn’t give them sufficient time or opportunity to learn things of interest to them; client organisations complain of continuing work-related difficulties. These people’s views about the course and training generally will not have been enhanced by their experience, nor are they likely to support further training interventions.
Standards of performance, both related to the outcomes of training and to actual work, are likely to become a major concern in the development of EoT. By implication, the IT illustration exposes potential flaws in a range of training activities, such as: TNA, design, course provision, on-job-training management of training and the responsibilities of funding agencies.
No Training
Doing nothing could be a decision of management - perhaps reflecting negative opinions about training identified in the introduction. However, although no formal training may have been done, it doesn’t necessarily mean that people are not learning. Occasionally, well-run departments of government have adopted the principles of the ‘learning organisation’. This will especially occur when using or developing advanced technologies and associated systems. People can only learn from internal expert resources - which are not likely to be found at a training institution. These are issues worth including in EoT, because informal, effective, training is being done.
Another reason for doing nothing is that resources do not permit training. Despite government policy of ‘training for all’, some organisations are unable to provide training for subordinate staff. Depending on the opinions of management, this may lead to:
• Low standards of performance of individuals, working groups, or an organisation as a whole. This could be caused by management accepting the situation and reducing standards, or because non-training factors block attempts to introduce change.
• Low levels of performance, where individuals lack knowledge or skills to improve their performance. Although management may wish to improve standards of performance, they lack the support of a formal training function and access to resources.
A variation on ‘no training’ could be ‘training for some’ - implying that, for example, training is mostly limited to gazetted officers. Other factors, such as location, may limit what can be achieved - although with the introduction of distance learning this is no longer necessarily an obstacle.
What may become apparent is that a response of ‘no training’, when asked about training provision, is likely to lead to more searching questions. Doing no training may at times be a justifiable option, but this justification will need to be explained - perhaps with some difficulty, if levels of performance are low. A crucial aspect of EoT is likely to emerge when consideration is given to:
• Doing no training - where no attempt is made to provide formal on-job-training - nor seeking help from training institutions.
• Doing ineffective training - whether by failing to organise training on-the-job, or using incompetent people as trainers, or by poor nominations for institutional training.
Perhaps because of resource limitations, or a large number of subordinate staff, or wide geographical spread, doing no training is justified. Also, if organisations do not have a formal training function, they may find it difficult to provide evidence of any training they are doing. Doing ineffective training costs money and uses other resources - but does so without benefit to anybody.
Effective Training
Figure 4, below, illustrates two triangles, where one represents effective training and the other training that is ineffective - with both using the same resources. The two triangles could represent two courses intended to meet a similar training need.
PerformanceResources
No Training
INEFFECTIVE TRAINING
Desired Standard of Performance
EFFECTIVE TRAINING‘A’
Fig. 4
It’s evident that the triangle reaching point ‘A’ is the one offering greater benefits - bearing in mind that both courses require similar resources. However, in respect to effective training, we can make some general observations about the illustration:
• Unless TNA has been done, we are unlikely to have a clearly defined desired standard of performance. Without this information, and not knowing where is a course going, how can we claim it’s effective?
• How were the courses assessed? Should we assume that the same measures were used for both of them? Perhaps, if EoT is used, we can have specified criteria and measuring instruments that will be common to both courses.
• The line drawn to represent ‘no training’ is too simplistic, as it assumes all course participants are starting with the same entry behaviour. Another approach to assessing the course could be around ‘learning gain’, where the emphasis is on helping individual participants to improve - without attempting to impose an arbitrary standard.
Nevertheless, if we look at the illustration for a funding agency’s point of view, both courses cost the same to run - one was successful in achieving agreed standards and the other wasn’t. If EoT has been done, hopefully based on TNA, then the funding agency and training management have factual evidence to decide about further courses. However, if neither TNA nor EoT has been done how can such decisions be made? Let’s consider other issues, using our IT training course as an illustration:
• As we’ve discussed earlier, unless a clearly defined standard of performance has been established it is difficult to establish an outcome. Our IT trainers are unlikely to know the performance problems of each trainee and their needs, in respect to IT. Having been asked to run a 5-day course, they will do their best to do so - but they will be guessing at the desired outcomes.
• Perhaps criteria for nominations for the course were vague, thus attracting people with a wide range of abilities. If so, it’s unrealistic to expect trainers to create effective training for all of them.
• Off-the-job training is rarely a complete solution to a training need. Most off-job training has to be supplemented by continuing training on the job. Evaluating the course without including transfer to the work situation will result in a distorted picture that is unfair to all concerned.
• IT training at or near the work place, using computer-based training systems is now readily available. This provides people with an opportunity to learn at a suitable time and pace. So, who decided to locate all of the IT training resources at a training institution? Also, who decided that the course should be of 5 days duration?
It would be easy to continue throwing bricks at an imaginary IT course, especially in the absence of TNA. Nevertheless, it’s true that most of the problems mentioned above could be avoided by having analysed both training and non-training needs. Perhaps an essential feature of EoT is that it is based on thorough TNA, as well as learning processes that develop skills and knowledge.
The definition for training refers to a ‘planned process.... to achieve effective performance’, which implies that if you want to achieve effectiveness you have to plan for it - something lacking with our IT course. EoT is a feature of this process and we can use it to report on both the process of training and its outcomes. Using a famous golfer’s comment that the more he practised the luckier he got; perhaps it’s worth pointing out that the more we focus attention on all aspects of the Systematic Approach to Training, the more effective we are likely to get. However, let’s consider different perspectives of ‘effective training’:
Training Institutes
An effective institute might be one that runs a wide range of training courses, using available resources and marketing its services to a wide range of client organisations. Perhaps also it attracts funding from a variety of sources, attracting applications for tutorial appointments from keen, able officers who see institutional tenure as good for their career prospects. Trainees also like to attend courses at such institutes, which are usually fully booked. However, when carrying out EoT - when the focus of attention is on ‘process’ and ‘effective performance ... to satisfy ... the needs of the organisation’ - we may view the situation somewhat differently. For example:
• EoT could be less concerned with the output of the institute - in respect to the number of courses being run, or people trained - and more interested in the outcome achieved. If there’s no evidence to suggest improved performance, can the claim for institutional effectiveness be justified? In our IT course illustration, although people attend the 5-day course, it may not help them achieve a significant improvement in their performance. When there is compatibility between a high output and proven performance-related outcome, then an institute can truly claim to be effective.
• Stakeholder opinions within a client organisation will influence training effectiveness. For many public sector organisations training is seen as nominating people to attend courses being run at training institutions. The availability of TNA consultancy services should help to change this perspective - yet, in relation to effectiveness, there remains the problem of responsibility for improving job performance. Although institutes may run good courses for helping people to learn, ultimately effectiveness lies with client organisations. They have the responsibility to help trainees use their newly acquired knowledge and skills to achieve desired standards of performance. EoT can explore the effectiveness of links between the output of a training institution and the outcome of a training process.
Client Organisations
We are using this term to cover departments of government, public sector undertakings, or working groups employing people from the public services. Each of these organisations is expected to achieve a desired standard of performance, although for many this will be a challenging aspiration rather than reality. For EoT, our concern is to assess and value the effectiveness of training done either within the organisation, or on its behalf by training institutions. Consider some issues this will involve:
• The SAT process places emphasis on work place activities - TNA, coaching, mentoring, assessment, for example. Many organisations do not have a recognised training function, nor the services of a training manager, which results in two options. The first is a decision to ignore training for improving performance, and the second is to rely on a training institution to provide professional services. Of course, a third option would be to establish a training function, but that is outside the remit of EoT.
• As mentioned earlier, off-job training courses should be linked to a process of transfer of learning, so that trainees can apply and further develop their skills under normal working conditions. Unless client organisations prepare for this, and have necessary resources, then achieving a desired level of performance will be difficult. In these situations it would be unfair to lay responsibility on either the training institution or the client organisation.
• TNA consultancies identify both training and non-training needs. This presents three scenarios: in the first an organisation acts on TNA recommendations, tackling both training and non-training needs; in the second they only act on training recommendations; the third is that TNA isn’t done and neither is any attempt made to link training to performance. EoT has the potential to ask stakeholders in such organisations to make decisions about training - not attributing blame, but raising issues that need to be addressed.
Funding Agencies
Both government and non-government agencies provide financial support for training. EoT can be used to review their effectiveness in respect to achieving a desired standard of performance, or for them to assess how their funds have been used. In both situations EoT can review the process, outputs and outcomes resulting from funding support. Criteria for assessing institutional or organisational performance have to be agreed, as an initial stage of EoT and then used when appropriate. Taking our IT course as an example, can a training institute be ‘blamed’ for running the course when they were offered funds to purchase computers and to run 5-day courses? Equally, haven’t funding agencies the right, and obligation, to assess the effectiveness of training that results from their funding support?
EFFICIENT TRAINING
You note that Figure 5, below, shows two triangles. These illustrate the relationship between achieving a desired standard of performance against resources used. Both triangles show that the desired standard of performance has been achieved.
PerformanceResources
Desired Standard of Performance
No Training
EFFICIENT TRAINING‘A’‘B’ INEFFICIENT TRAINING
Fig. 5
The difference between them is that ’A’ shows performance being achieved using fewer resources than ‘B’. Consider some issues this raises - bearing in mind that both ‘A’ and ‘B’ are effective:
• To what extent are training resources ‘price sensitive’? Is a rigorous examination proposed to ensure ‘value for money’? In the IT course, to which we’ve referred earlier, what process was used to establish valid justification for resource expenditure? Did the training institute concerned already have computer-training resources and, if they had, what was the utilisation rate? These questions could be included in an EoT aimed at improving efficiency - because if resources are spent with ‘B’, rather than ‘A’ less funding support is available elsewhere.
• With the government’s policy commitment of providing ‘training for all’ greater attention is likely to be needed to provide effective training for lower cadre workers. This can only be achieved if the training is provided efficiently. It’s pointless training twenty people effectively, if there are a thousand with similar training needs. Equally, there’s no benefit in training a thousand, if it’s ineffective.
• The introduction of distance learning has the potential to improve efficiency. However, it is not enough to have a package that can provide effective training. An efficient system is also needed for its implementation. EoT should therefore be concerned with both effectiveness of the training and efficiency.
At present, it’s perhaps unrealistic to expect training to be both effective and efficient - although it’s a desirable aspiration. However, with the introduction of TNA consultancies and the development of EoT, we can develop our functional capabilities to realise the aspiration.
Distance Learning Part 2 - Models for EoT
Introduction
EoT is such a broad, poorly charted arena in which to work that we need all the guidance we can get. Unfortunately, there’s only a limited selection of literature available to help plan EoT, and much of it is repetition of earlier studies. During the forthcoming workshop we’ll present you with an overall ‘EoT Matrix’, based on well-established EoT models. This will enable you to plan evaluation activities.
For now, the purpose is to introduce you to these models so that you’ll understand how they contribute to the matrix. The models enable us to evaluate the purposes for evaluation, and the levels at which this can be done. In addition, you’ll also need to devise suitable systems for you and your professional colleagues to use when carrying out EoT.
Analogy
You may recall in Part 1 we used an imaginary course on Information Technology to illustrate issues affecting EoT. A training institution used its resources to build and equip an ‘IT Centre’, with modern computers and competent trainers. They are asked to run a series of 5-day ‘IT Appreciation’ courses. These courses are listed in the national calendar and people from a variety of public sector organisations are nominated to attend. We will refer to this situation throughout Part 2 of the EoT distance learning.
EASTERBY-SMITH MODEL
LEARNING IMPROVING PROVING MONITORING
Figure 1
This model is adapted from one proposed by Mark Easterby-Smith. We have modified the original to suit the structure of training in the Indian public service. It is from this model, shown in Figure 1 above, that we suggest EoT systems can be given a focus or purpose. The model consists of four interrelated purposes. These purposes can be applied to all stakeholders involved with training and development.
Four Purposes of Evaluation
Learning Processes - where the quality of learning experiences is at the heart of all training and development activities. Here, evaluation is concerned with the processes used to provide satisfactory experiences for individuals and groups, both on and off the job - and to ensure they are objectively measured.
Proving Learning and Development - demonstrates that something has happened because of training and development activities. This may be linked to judgements about the value of the activity - whether the right thing was done, whether it was well done, and whether it was worth the cost.
Improving Learning and Development emphasises procedures to ensure that either current, or future training and development courses or programmes or activities become better than they are at present.
Monitoring Learning and Development - an essential feature of EoT is the exchange of information to ensure that organisational training functions and training institution’s meet agreed targets; provide a satisfactory professional service; and make efficient use of available resources and facilities.
All four purposes can be regarded of equal importance, although your status, role and responsibilities will determine which of them will be a priority. An EoT function is likely to be unique to a particular organisation, using systems and procedures to satisfy internal and external clients, stakeholders and funding agencies. Easterby-Smith points out that although all four areas are of overlapping importance, you may focus EoT efforts in one or two of them.
LEARNING IMPROVING PROVING MONITORING LEARNING
Figure 2
A definition for learning is: ‘The process whereby individuals acquire knowledge, skills and attitudes, through experience, reflection, study, or instruction’. Having read this definition, you could probably recall it later – or perhaps you’ll quickly forget it. However, if we told you that you would be tested tomorrow on your ability to recall the definition would that improve matters?
There are two factors for you to note in respect to the definition. The first is that the focus of attention is on the individual, and the second factor is that there is the process for doing this. The process relates more to proving that the process is effective, and seeking ways to improving it. In this area of the EoT model, we are mainly concerned with evaluating how we help an individual to learn. Do we coerce individuals by threatening them with a test tomorrow? On the other hand, do we say ‘here’s a definition, learn it if you want’?
Learning is often seen as the heart of evaluation, but our brief is to develop systems and procedures for the evaluation of training. So, perhaps we should address different questions. For example:
1. How can we create optimum conditions for people to learn – both in a training institution and at the work place? How can we link this concern – helping people to learn, with the desire to improve?
2. Can we help people to learn without necessarily expecting them to be trained? What happens if people, through the absence of training, learn the wrong things?
3. Should we seek to adopt the principles of the ‘learning organisation’? If this is considered something worthwhile, what steps need to be taken – and how will we know whether those steps have been successfully achieved?
4. What about the concept of ‘andragogy’? Should this be included in EoT, after all we are involved with adult learning. The concept suggests that, as trainers, we create conditions for people to learn, which may require us to allow them to learn from each other. Case studies, role-plays and discussions – linked to peer/self assessment and process/product checklists could be used to create such conditions, where people can learn and be given feedback.
5. Should we help people to develop ‘learning skills’, so they are competent to learn for themselves? Mention of competence draws attention to the ‘outcome’ of learning and its assessment – should this be part of EoT?
6. When considering institutional training, and the over-reliance on lectures – notorious for poor learning – should we not make evaluation of methods of learning a feature of EoT?
7. With the introduction of distance learning, should we evaluate the quality of learning this material offers? Should EoT provision be included in distance learning packages? If not, how are packages to be improved?
8. How are we to assess people who have to cope with changes to their jobs – probably requiring them to ‘unlearn’ knowledge, skills and attitudes, acquired over years of service?
9. How are we going to evaluate an individual’s learning, when they are facing resistance from colleagues and superiors? Issues of culture and a failure to include provision for the transfer of learning may require a wider purpose for evaluation.
10. Taking into account the government’s intention of providing ‘training for all’, should evaluation draw attention to the use of performance or job aids? Should people be trained to use these aids to performance, rather than waste resources helping them to acquire knowledge and skills and possibly attempting to change their attitudes?
The assessment of knowledge through objective test items and skills through product/process checklists are accepted internal validation measure, even mandatory in safety sensitive situations. Unfortunately, as jobs become less reproductive, it is increasing difficult to measure, or even make a judgement about a person’s performance. Productive tasks can be performed successfully in different ways, so how can we impose our assessment in situations where there may be alternative approaches – that are equally valid?
Recently more learner-centred approaches to assessing learning have been introduced, such as: peer/self assessment, learning logs and learning contracts. These encourage self-development and facilitate transfer of learning. However, such techniques are suitable only for well-educated people, who possess learning skills. The policy of ‘training for all’ implies learning for all – leading to concerns about how we can evaluate learning for lower cadre workers. You are unlikely to find an answer in standard text books!
Analogy
If you’ve taken the course in Direct Trainer Skills you may recall reference to creating a ‘learning event’ – an activity that a trainee finds an interesting and rewarding experience. Such learning events must be an integral part of the IT Appreciation course – trainees are helped to overcome learning difficulties, encouraged to practise IT skills, so that they gain competence and confidence. The problem with IT training is that people are likely to have a wide range of entry behaviours – from the anxious novice to people with perhaps some but insufficient understanding of the subject.
It will be important to recognise that to evaluate learning we need to associate it with individuals’ learning needs. For EoT, in respect to the IT Appreciation course, this may require giving feedback on learning activities and accepting feedback on the suitability, pace and relevance of learning events. Frequently, learning is focussed not simply on the needs of individuals, but also collective learning experiences involving others. An example from the course could be learning how to use emails – possibly involving unlearning existing systems and developing a new approach to communication. In this example, learning is extended from the needs of an individual to all those exchanging information by email.
Proving
According to Easterby-Smith, the first major evaluation study was carried out during the Second World War, when more than one million people in the USA received supervisory training. When asked by a government committee for evidence that the expenditure was worthwhile, the training providers gave information to indicate a 25% increased productivity in two-thirds of the factories involved. Although this example is now dated, it serves to illustrate the need to be able to prove that effective training was being done, and had measurable benefit. It’s interesting that this supervisory programme was part of a major initiative to improve the quality of training in the US and Europe. Many concepts and practices then developed are still in use today.
LEARNING IMPROVING MONITORING PROVING
Figure 2
Although a considerable amount of training is being done within the Indian public services, some of it is being done as an ‘act of faith’ - rather than a conviction, supported by evidence of its value. An implication of the government’s policy of ‘value for money’ is to show proof of something accomplished. However, to enable you to show proof, you’ll need:
• A reference that clarifies what needs to be changed, and the criteria used to assess the resultant change. The recent introduction of TNA should provide more information about actual training needs, with details of the desired standards of performance. Obtaining this information, and including it a formal design brief, becomes an essential component to the provision and evaluation of effective training.
• Measuring tools that are appropriate to what is being measured and acceptable to the people using them. The development of an EoT toolkit provides a selection of techniques and guidelines, which can be adapted to suit organisational or institutional requirements.
This is perhaps the most difficult area for evaluation, as much depends on what you wish to prove and who will be involved. You are likely to encounter difficulties. For example:
1. Much will depend on the quality of information you have about client organisations, in particular whether TNA has been done - and to which you have access.
2. If no standards of performance have been agreed, nor levels of performance acceptable to management, then you don’t have a reference, upon which proof can be measured.
3. Proving training in the confines of a training institute - where you have some authority and control - may prove relatively easy, in contrast to an on-job situation.
4. The choice of measuring techniques could create problems. You may wish to use one’s that are likely to be effective, but unacceptable to client organisations or difficult to administer.
5. You may be unable be unable to gain access to key stakeholders, whose judgement could be a crucial factor.
6. Proving training effectiveness is different from proving its efficiency. Proving both will enable you to report an overall value of a training programme.
7. Non-training implications could impede training activities. In such situations, where external validation is likely to report a lack of success, you may need to ensure thorough internal validation measures are used.
8. You may encounter people with great sources of power for whom EoT may conflict with their interests. In such circumstances, you may find it impossible to obtain verifiable proof - depending on your status or access to alternative sources of power.
9. Although you are confident that the training you provide is of high quality and effective, you need to have sufficient evidence to confirm this belief. To maintain or attract resources for training, it’s advisable to focus attention and supporting evidence on the outcome of training. Make sure that clients, key stakeholders and funding agencies are aware of success stories - what people and organisations can do because of their training.
10. A weakness shared by many trainers is that by instinct we want to help people to learn and develop. This gives us job satisfaction. However, the instinct of somebody running a business would be to use evidence of success to promote further business. Perhaps it’s an instinct worth using to market our professional services. Having proved the success of the training, what are you going to do with the proof?
Notwithstanding these difficulties, there are considerable benefits to be obtained from EoT. At a professional you’ll find out whether your services as a trainer are well regarded by trainees and stakeholders; at a business level you’re establishing the viability of the training function. The advice on offer is limited and not necessarily suited to EoT in the Indian public service. Perhaps in a few years, when EoT has become an established feature of training, case histories and evolving systems and procedures will limit such difficulties, as listed above. Possibly the only sound advice we can offer is to emphasise the importance of TNA consultancies. Easterby-Smith and other authors make the point of engaging with stakeholders, using their values and criteria as a basis for evaluation - an inherent feature of TNA.
Analogy
Although our IT Appreciation course is imaginary, no doubt a real one would have objectives, stating what a trainee can do on completion of the course. However, how will we, or the trainee, or the sponsoring organisation, know that these objectives have been achieved? What proof will be available to provide evidence? Bearing in mind the significant resources used to train a person, surely it’s reasonable to expect some confirmation that he or she has successfully completed the course and achieved the stated objectives. Simply stating that a person has been ‘successful’, begs the question about who and how this assessment was made. What criteria were used?
It’s a fair comment to point out that ‘appreciation’ courses rarely offer specific, measurable outcomes. Nevertheless, unless trainers and training institutions can prove that the course they are offering leads to some meaningful result, why should organisations nominate staff? Also, from the point of view of funding agencies, what is the justification for running the course - or, indeed, providing funds to equip an IT training room. If there’s no proof that something is worth doing, why do it?
No doubt, with a genuine need for IT training, there’s a strong case for running IT courses. However, to justify the considerable costs incurred, funding agencies and client organisations should expect to be given proof that their investment leads to acceptable benefits. Using this ‘proving’ purpose for EoT perhaps we can see a situation where, for example, funding agencies only approve training proposals which include details of systems and procedures used to prove success. From an institution’s point of view, perhaps the best advice would be not to offer to do something you cannot prove to accomplish.
Improving
LEARNING PROVING MONITORING IMPROVING
Figure 4
Often, when we purchase consumer products, we see reference to the manufacturer’s commitment to introduce changes, as part of their policy of product development. ISO **** is a standard concerned with assuring customers that the vender’s products and services are backed by an effective system of quality control. ‘Customer Care’ is a major concern of the more progressive organisations in both the public and private sectors. During the writing of this text a representative of the Market Research Society knocked on the door and asked if he could ask questions about supermarkets. He issued a leaflet explaining the context and regulations for his market research, including the following paragraph:
‘Market research is your opportunity to give your opinion on things that may affect you and your family. Manufacturers, retailers, service companies, political parties and the Government can only succeed if they please you, the customer, so they need to find out what you need and what you want. Your opinion can influence a wide range of products from pension plans to washing powder and also have a bearing on issues that affect the quality of your life.
It is the job of the market researchers to ask questions - to find out what you, and people like you, think’
Some organisations make great efforts to seek feedback from customers about the quality of their products or services - the illustration, above shows how some do this. Hopefully, this information is used to make improvements to what they do. As trainers, we can do the same.
Authors on the subject of EoT regard improving training as an easier option than attempting to prove it. The problem is that without first proving that a training course, for example, has already a proven record of effectiveness, how can we attempt to improve it? Consider some ways in which a proven training course could be improved:
1. Ensuring that it is based on a thorough TNA, which may itself require improving if the course is run frequently. Changes to technologies, systems and regulations may lead to frequent revisions to course content.
2. Carry out rigorous internal evaluation. Establish procedures to link feedback from trainees, during their training to ensure that it satisfies their needs and helps them to achieve the stated objectives.
3. Innovate, or introduce new activities to improve the training process and the results obtained. For example, the government’s recent project to develop distance learning may lead to changes to course provision.
4. Comparing output from a course - the number of people trained - to the demand for it. If there’s a significant mismatch, then course provision can be reviewed - should there be more or less?
5. Efficiency is another area for potential improvement. This can be done, for example, by reducing the length of a course, or by using different methods for its delivery.
6. One of the weakest features of course provision is in transferring learning from a course to its application at the work place. Improvement initiatives can be based on analysing the reasons for problems of transfer - but who should be responsible for this?
7. There is a risk of concentrating on improving a course, rather than also including the systems being used. For example, no matter how well you improve a course, unless suitable people are nominated to attend it no real improvement is likely to be realised.
8. Although most trainers in the Indian public service have attended the Direct Trainer Skills course and the Design of Training course, perhaps greater efforts can be made to help them develop. For example, by encouraging newly appointed trainers to develop instructional skills, and for more experienced trainers to devise better courses, both they and their trainees will be helped to improve. Perhaps one initiative would be to introduce the concept of Continuing Professional Development (CPD), where learning is linked formally to assessment - both for the individual and for their employing organisation.
9. The counterpoint to 8. is, however, that unless initiatives are made to encourage improvements nothing is likely to improve.
10. Leading on from 8. and 9. is the need to establish communication, commitment and joint responsibility with all concerned with an improvement initiative.
The underlying philosophy could be: ‘Start with a good course and constantly seek ways to improve it’. Although Easterby-Smith comments on the difficulties associated with ‘improving’, he notes that in a more positive sense there has been a considerable amount of work done recently which views evaluation as an aid to decision-making. Perhaps this could be a major feature of the EoT systems and procedures used for improving.
Analogy
If you’ve no evidence, or means of measuring existing performance then how can you claim improvement? From the points made above, it’s evident that the basis for this ‘improving’ purpose of EoT can only be accomplished when it follows TNA. Without information about the present use of IT, details of what is done and the people involved, then there’s no reference base from which to measure any improvement.
Running an IT Appreciation course can be seen as a superficial, cosmetic approach to meeting a real performance problem. Also, even if the course is a proven success, it may not lead to improved performance - and, if it did, can this be attributed to attending the course?
For the IT Appreciation course to be regarded as successful, it must be associated with improving the knowledge, skills, attitudes and performance of trainees. The stronger we can make the link between the course and identifiable improvements, then the easier it will be to justify the existence of the training institute and this course. Failure to include this purpose within EoT is likely to lead to indifference from potential client organisations and a marginalisation of training services.
Essentially, improving is concerned with ‘input’ and ‘output’. EoT directed at improving input examines the quality of what we do - design, development, delivery and assessment of training. When evaluation is directed at improving output, the focus of attention is our efficient use of available resources - our productivity.
Monitoring
Figure 5
Research studies indicate that effectiveness of training and development is significantly increased if the monitoring of people involved in delivery and facilitation is undertaken. The development of a Training Management Information System (TMIS) will provide a framework for procedures to be developed, suited to the import and export of information. The central role played by the DoPT, as a funding agency, places it as a major feature of this area of EoT.
Perhaps an anecdote can be used to illustrate the importance of monitoring. Several years ago, during a training of trainer’s consultancy for Indian Railways, a chance, an unscheduled visit to a small training centre in Hyderabad produced some interesting findings. The centre ran courses on bridge inspections for Southern Railway and the trainer responsible for the course had produced a training package, developed specifically for the course. There was evidence of research, planning and the development of good quality.
training material. During a discussion with an experienced railway official, also a visitor, the trainer - on his own initiative, had evidently spent a considerable amount of time and effort developing the package, which provided detailed guidelines for the range of bridges likely to be inspected. It was a pleasure meeting this trainer and looking round a small, somewhat impoverished training centre - but one demonstrating good training practice. The only surprise was to find that nobody at central office was aware of the trainer’s existence, nor of the package he’d developed. Yet there are bridges to be inspected all over the country - perhaps this package could have been used nationally. Unfortunately, nobody - except the trainer and his trainees were aware of this. Possibly, if there had been a system for monitoring courses this trainer could have been spotted and encourage to continue his development.
Another strand to this anecdote was during a scheduled visit, also in Hyderabad, to a technical training centre. During a discussion with an instructor, he had evidently been running the same course for over ten years - without, it seemed, any attempt to validate it. Here’s an example of a ‘passive provider’ - although it would be unfair to blame the particular instructor we met. Possibly, if there had been a monitoring system, changes would have been made in the course to reflect technological or systems developments. A final example about this anecdote is a visit to Golden Rock and then to Bangalore - where the same course was being run at respective training centres, although for different railways. At one, there was evidence of thorough preparation, with excellent visual aids and handouts - all in what appeared to be a well organised training centre. Then we went to the other centre, where none to these indicators were in evidence. Why not? Surely one centre could have collaborated with the other - apparently not. Again, it’s unfair to blame the people concerned, but if the concept of ‘networking’ was built in to an effective monitoring system, good practice and access to resources could be shared - creating a ‘win/win’ situation.
Monitoring has the potential to make a vital contribution to training in the Indian public service, and one that is not simply concerned with evaluating training. An effective monitoring system, using an IT-based Training Management Information System (TMIS) would enable:
1. Costs monitored and compared.
2. Common interest networks to be established.
3. Resource support services evolved to facilitate access to resources.
4. Centres of excellence to be identified and encouraged to share their expertise
5. Training packages to be developed and shared.
6. Projects for continuing professional development.
7. Better communication with funding agencies.
Analogy
The quality and effectiveness of the IT Appreciation course will be greatly enhanced if trainers and the training institute are aware of the criteria being used to monitor course provision. Procedures devised to meet other purposes of EoT should be taken into account
The EoT function should include tasks that will be carried out by funding agencies, departmental personnel and institutional directors. Some of these tasks will be administrative, dealing with EoT systems; other tasks require the involvement of senior officers - usually leading to decision-making. Using the IT Appreciation course as an example, this may require decisions on:
• The views of trainees about the quality of their learning during the course.
• Is there sufficient evidence proving that the course worth running?
• Is there evidence to show that performance is improving?
• If not, why?
• If so, how many courses should be run with the existing facilities?
• Should additional facilities be set up at other institutions?
• Are there alternative strategies to consider?
Decisions based on answers to these questions can only be made when there is sufficient information available - obtained from within the EoT function.
HAMBLIN - KIRKPATRICK MODEL
Using Mark Easterby-Smith’s model, modified to suit training in the Indian public sector - we can clarify the intended purpose for EoT. So, although the first issue to resolve might be to ask ‘What is the purpose of your proposed EoT?’ Having clarified this, it may lead to further questions - for example:
• Who will be involved in the EoT?
• How many people will be involved?
• What is their likely reaction to the evaluation?
• Whose authorisation will be required?
• What is the timescale?
• What resource support will be needed - i.e. time and funding?
• How will the information you obtain be used?
• What are the anticipated decisions likely to result from the proposed EoT?
• What are the likely political implications to consider?
There are several models, other than Easterby-Smith, that can help to address the questions listed above. Two are of particular relevance to EoT:
Hamblin, A.C., (1974), Evaluation and Control of Training, McGraw-Hill.
Kirkpatrick, D.L., (1994), Evaluating Training Programmes, Berett-Koehler.
Using the concepts suggested by these authors we can develop an extra dimension to our overall concept of EoT by defining various levels of evaluation. We will describe each level by using our IT Appreciation course to illustrate typical applications.
Description of Model
The basis of the model, shown in Figure 6, is the reason for evaluation - training. Perhaps all too frequently, training has been done without any serious, structured, framework with which it can be evaluated. Hamblin makes an important point when stating ‘... we can learn to modify our behaviour as a result of all kinds of experience; but if we are being trained, this implies that we are being put through an experience designed to make us learn’.
TRAINING OBJECTIVES
Level 3Behavioural
Level 4Organisational
Level 1Reaction
Level 2Learning
EFFECTS
Figure 6
If we are to evaluate training, other than by subjective opinions, we must collect appropriate information about the changes caused by training. To do this systematically we must think about the effects that a certain training activity will have and the sequence with which these effects will occur. Consider this as a ‘cause and effect’ process. A chain linking four levels of training.
As Figure 6 shows, a training activity elicits reactions from people being trained. The concern for EoT is to have procedures available to enable these reactions to be recorded and information fed back to whoever is providing the training. At this point, in relation to the model we have a situation where:
• Training has been provided and resources used.
• Trainees are likely to have reacted to the learning opportunities designed for them.
• Procedures have been used to record their reactions.
• Level 1 feedback can modify objectives and further develop training provision.
• Training repeated, as required.
• Evaluation procedures continue to be used at levels 2, 3 and 4.
This chain may break at any of its links. A trainee may react correctly during a course but fail to learn; he or she may learn but fails to apply this on-the-job. Even if trainees change their job behaviour, it may not result in improved organisational performance. Without EoT, there is unlikely to be a process, and these links never identified or investigated for breaks. The task you face, when carrying out EoT is to:
• Develop an effective process for evaluating training that clearly links the four levels.
• Develop procedures suited to the collection of appropriate information.
• Ensure that procedures are used to monitor training activities.
• Identify breaks in the links.
• Give feedback to those responsible for remedial action.
• Take action to improve quality of training, or its effectiveness, or efficiency.
Levels of Evaluation
Level 1 Reactions
Obtaining feedback from trainees about the quality of training they have received initiates EoT. Their opinions about the content, pace, methodology, tutorial support, learning materials and the facilities available are essential components in monitoring and improving the quality of training. The basis for obtaining this information is usually done by using ‘Immediate Reaction Questionnaires’ (IRQ).
Remember the possibility of the ‘halo-horns’ effect, where trainees react to the messenger (trainer) rather than the message (what they are required to learn). Reactions that are ‘good’ or ‘bad’ may only be relevant at level 1, when trainees face interesting learning opportunities, or difficult challenges. It’s possible for trainee reactions to be anticipated if TNA has been done, and learning needs or performance problems identified.
It’s also worth noting that if only level 1 EoT is done, then the quality and value of information being monitored or fed back for decision-making is poor and misleading. For example, due to critical information obtained from IRQ’s could result in a course being cancelled, when EoT at a higher level shows evidence of ultimate benefit to both trainee and employing organisation.
Level 2 Learning
The purpose of training is to organise learning on the behalf of trainees so that they achieve specified objectives. The outcome of this process is to assess a trainee’s acquisition of knowledge, skills and attitudes relevant to their needs. This can be done using formal tests or with a less intrusive form of assessment. Examples of assessment techniques include:
Objective tests
Simulations
Performance tests
Assignments and projects
Checklists
Interviews
Observation
Peer/self-assessment
Level 3 Job Behaviour
The crucial factor here is the extent to which training needs have been analysed. This will identify the behaviours people need to develop in order for them to do their job to a satisfactory standard. The information is vital to determining the ‘input’ needed for effective training - specified in terms of knowledge, skills and attitudes. This information is also as the basis for assessing the ‘outcomes’ of the learning process. The following is a list of information needed to effectively evaluate job behaviour:
Current job profile/description
Specific tasks or competences
Standards of performance
Performance targets
Categories of knowledge - principles, concepts, facts, procedures, etc.
Categories of skills - manual, interpersonal, keyboard, problem solving Indicators of attitudes
Level 4 Departmental/Organisational
Ultimately, the final evaluation of training will be done by the organisation paying for it. Irrespective of the opinion of trainees to the quality of the training, and benefits they obtain from it, senior management will evaluate the investment by seeking answers to the following questions:
Cost benefit - is the cost of training justified in relation to the benefits it provides?
Does the organisation still require this training?
Can the organisation risk or accept the consequences of poor training?
Is certain training required by legislation - and is it effective?
Has training reduced wastage, administrative delays, customer complaints, etc...?
Has training improved productivity, sales, morale, quality, etc...?
If the answer to these questions is ‘No’, the consequences could be for senior management to ask ‘Why waste time, money and resources doing something that appears to have no value to the organisation’? However, if the initial levels of evaluation have been done, you will have substantial evidence to prove that training has helped employees and the organisation to improve performance.
A point referred to by Hamblin is that all organisations have four primary objectives, which are in order of primacy:
Survival of the organisation.
Creation of surplus - services, products, profit.
Welfare of interested parties.
Social/political welfare.
Considering these factors at level 4 may involve senior management and funding agencies scrutinising with great care the information available, especially from level 3.
Analogy
You may recall our use of an imaginary ‘IT Appreciation ‘ course to illustrate various aspects of EoT. We’ll continue to use this to describe how the model, based on Hamblin - Kirkpatrick, can be applied. Be assured that the course, as described, is a figment of our imagination and doesn’t refer to any particular course or institution.
Level 1 - Reaction
People react differently to situations, based on experiences, both good and bad. Also, reactions may reflect their status, motivation, culture, beliefs, etc. Often, as trainers, we don’t know how people will react to an opportunity to learn. Perhaps, when you were reading the above paragraph introducing the analogy, you reacted to the statement. You may regard the notion of an ‘IT Appreciation’ course as nonsense - what do you mean by ‘appreciation’? Alternatively, you may have designed or run such a course and feel offended by the prospect of criticism. Mention of ‘course or institution’ could lead you to relate the analogy to your own work situation. This may encourage you to pursue the opportunity to learn - or it might create a barrier to your learning and, consequently, switch off your motivation to learn.
Consider the people taking the IT Appreciation course and their potential reaction to it:
• Some people were on the course because they were told to attend and saw no relationship between the course and their work.
• Other people were having computer-based systems installed in their office and were keen to learn how to use MS Word and e-mails.
• Other people already had IT systems, but had never been trained to use them
• In most of the courses some people had keyboard skills, whereas others had no idea how to type or input data.
Note that we have used the term people, rather than our usual practice of referring to them as trainees. Does it matter? Well, many of the imaginary people we refer to above may regard themselves as government officials, doing important jobs, with many years experience and respected members of staff. As far as they are concerned, trainees are young probationers, who need to learn the basics. In the Design of Training course we deal with this concept, which is called ‘andragogy’ - adult learning. It’s important here, when we are considering reactions to training. Because the concept recommends creating shared learning experiences - where trainers work with trainees (or learners, or participants) to satisfy their needs, in relation to the course’s stated objectives.
However, this situation can only be accomplished if there are effective procedures for obtaining trainees’ reactions and being able to make an effective response. The model in Fig.1 refers to this as reactions to objectives - which can be interpreted in several ways:
• Course trainers, having obtained initial reactions from trainees, negotiate modifications to stated course objectives. This might be done for individual, small groups or for all trainees. By seeking reactions, trainers are demonstrating a willingness to adopt a learner-centred approach - much favoured by trainees.
• Another approach to reaction-level feedback is to use the information to revise course provision. Circumstances, such as lack of equipment, trainers or time may limit what can be done during a particular course but information obtained could be used to make changes to subsequent courses.
• When a course is being monitored there should be evidence of reaction-level EoT - then further evidence to show how this information has been used to improve the quality of course provision.
Level 2 - Learning
We once had a participant on a DTS-type course who introduced himself by stating the reason he was there was because his boss had told him to attend. Later, it emerged that the new boss had seen ‘our’ participant running a course and told him that he should not run any further courses until he had attended a course to learn how to do it properly. Understandably, the participant’s reaction to attending the course, and activities during it, were not favourable. So, why mention it? Well, some months later, our former participant contacted us about sending other trainers on the course. Evidently, learning had taken place, although this was an extended process done after the course - presumably with the support of his boss.
At level 2, in respect to the IT Appreciation course, we are looking for confirmation that people have acquired knowledge and skills, also evidence that they are adopting an appropriate attitude. A strong case in favour of such courses is that they can start a process of learning and development, without necessarily offering long-term support. This is where we can stress the distinction between a training course and a training programme. A course, usually done off-the-job at a training institute, offers people opportunities to achieve specific learning objectives. A training programme extends this to help people use their learning to improve job performance - often requiring not only a course but also continued practice on-the-job. Therefore, for level 2 EoT, for the IT Appreciation course, we need to consider:
• How to assess whether trainees have acquired the knowledge and skills stated in course objectives. Assessment can be done by means of a formal test or by means of personal formative feedback. As we are considering an appreciation course, any attempt to impose a formal test would be inappropriate, but in another situation essential. People working as air traffic controllers are using IT - no doubt, we’ll feel safer knowing that such people have been rigorously tested.
• Often assessment can be ‘embedded’ in the learning process. For the IT Appreciation course, this can be provided within tutorial software and is usually a feature of open learning systems.
• Perhaps, for the IT Appreciation course, learning about IT has only just started. Trainees need to be involved in a training programme, where their learning can continue and be focussed on job performance. This is where the concept of modular training can be used to sustain and assess learning and development. EoT at level 2 should be seeking to establish, use and monitor systems that reassure client organisations and funding agencies of the benefits of their investment. At least trainees will have requisite knowledge and skills - whether this leads to improved job performance is for level 3.
• The model, at level 2, shows feedback to learning objectives. Therefore, when monitoring EoT at this level we should include evidence of how we assess the effects of learning, and how this information is being fed back to course objectives, its content and learning process.
Level 3 - Job Behaviour
During the forthcoming workshop we’ll consider some general issues you are likely encounter. One issue is the transfer of learning - included as a concept paper in the Design of Training course. For level 3 EoT, it’s a major concern. Using our IT Appreciation course as an illustration, we can show evidence of successful learning outcomes, based on level 2 assessment. We can also provide evidence that despite occasionally negative reactions at level 1, most course participants acquire an appreciation of IT and started to develop skills they’ll require for their job. In other words, although the course is a success, the problem is evaluating its effects on the job. Taking our IT course to illustrate what is probably a frequent situation:
• Trainees do not practise their skills under working conditions, which results in a steady deterioration in performance.
• Management is not supportive, failing to recognise the needs to create conditions where learning can be channelled into appropriate behaviour.
• Although trainees have learned the correct way to perform IT tasks, when they return to their job they see other people doing them differently. Here, at EoT level 2, there’s evidence of successful learning - which due to other factors does not lead to effective behaviour.
• Trainees are moved on to other duties or responsibilities where appreciation of IT is no longer required.
When investment circumstances are tight - where funding agencies are looking for ways to reduce resources, the lack of level 3 evidence could lead to a decision to withdraw support. What is the justification for continuing to support a course that doesn’t improve the performance of people who have attended it? Although evidence is available to prove that learning as taken place - level 2, there’s no information about actual improvement - at level 3.
A factor that could significantly change the situation is the introduction of the TNA Consultancy skills course. Increasingly, there will be trainers and other government officers competent to carry out TNA consultancies with client organisations. If IT is a performance problem in a particular client organisation, then it can be analysed and recommendations made to management. This should produce performance-related criteria, which can be used for EoT, and to provide specific design briefs. This should enable feedback to be obtained from level 3 assessment and the information fed to course and learning objectives.
Level 4 Departmental/Organisational
Perhaps for our imaginary course there’s unlikely to be level 4 evaluation. Any attempt to do so, based on subjective opinion, has no validity in respect to departmental/organisational performance. In addition, any attempt to do so without having done Level 3 could lead to flawed conclusions.
Following the introduction of TNA consultancies, greater emphasis can be given to level 3 EoT. Once this is established, EoT at level 4 can be used to enable strategic decisions to be made. For our analogy, this could be to decide whether the IT Appreciation course should continue. Perhaps alternative approaches might prove more effective, or lead to more efficient means of satisfying government policy of providing ‘training for all’. This is a subject we’ll return to during the workshop.
Course Guide
Introduction
Investment in training can only be justified if it leads to improved performance. This applies to public and private organisations, departments of government, NGO’s and, especially to individual employees and beneficiaries. For training to make an effective contribution to improved performance, and to help people to develop their potential, it should be based on all four stages of the Systematic Approach to Training (SAT). The fourth and final stage of this requires that training is evaluated. This enables the management of the time, money and resources invested in helping people and organisations to improve performance and develop their workforce.
Evaluation should concern all those involved in training and development - funding agencies, client organisations, heads of departments; also, employees at all levels and beneficiaries. All will have views about the quality, effectiveness and utility of training carried out on their behalf. Are their views known? How are such views, with substantive data, collected and used to check quality, effectiveness and efficiency of training services? Only when this is done can decisions be made about initiating improvements.
The Evaluation of Training (EoT) course establishes a framework within which decisions can be made. Should a particular course continue to be run? How can it be improved? What response has been made to TNA consultancy reports? How effective is distance learning in meeting government’s intention to provide ‘training for all’? These are typical questions that should be asked when carrying out EoT, with different people expressing and exchanging a variety of views.
Designed for
The Evaluation of Training course is intended for institutional and departmental trainers; stakeholders in client organisations; and officials representing funding agencies. The course is especially suited to those who have a direct responsibility for the development, administration or management of an organisation’s EoT function.
Course Structure
The development of roles, responsibilities, systems, procedures and the use of appropriate techniques are essential features of EoT activities. This requires a general understanding of the systematic approach to training, and how EoT is linked to Training Needs Analysis, the Design of Training and its Implementation. The course has three distinct features, each helping participants to develop their skills and understanding of EoT:
1. Distance Learning. This provides a general background to issues which will be faced when carrying out EoT activities.
2. Workshop. Run over a period of five days, where participants can develop an understanding of their role and responsibilities, also EoT systems and procedures requiring development. During the workshop EoT techniques, exercises and a case study will be used to provide learning opportunities, and a basis for self-assessment.
3. EoT Project. On completion of the workshop, each participant will undertake a personal EoT project with a suitable client organisation. This will be carried out over a period of three weeks and a report submitted for assessment.
Course Style
The EoT Course is intended to help participants develop their competency to carry out EoT tasks, commensurate to their role, responsibilities and status in their respective organisations. This is not a ‘taught’ course. Rather, it is a series of practical learning activities - usually simulated by using exercises and case studies. Participants are therefore expected to take active responsibility both for their own learning and contributing to group activities.
Distance Learning
The purpose of the Distance Learning is to introduce the concept of EoT, in relation to the Systematic Approach to Training. The material also outlines issues they will face when carrying out EoT assignments. The expectation is for participants to have studied the material, completed assigned tasks and being prepared to contribute to workshop activities.
Workshop As the workshop is based on an intensive series of team activities, it requires full-time commitment, including some evening study. We therefore assume course participants will be either residential or within easy commuting distance.
Project On completion of the workshop participants will be required to complete an agreed EoT project for submission and assessment by course tutors. Participants who satisfactorily complete their project will be awarded a certificate of competence, recognised by the Department of Personnel and Training.
Objectives
At the end of distance learning participants are expected to:
1. Define the Evaluation of Training (EoT).
2. Describe potential benefits to be obtained from EoT.
3. Define internal external validation and their contribution to EoT.
4. Describe performance and resource parameters influencing EoT.
5. List factors influencing the achievement of a desired standard of performance.
6. Contrast ‘effective’ and ‘efficient’ training, in relation to EoT.
At the end of the workshop participants will be able to:
7. Describe current approaches to EoT.
8. Describe concepts of EoT suggested by Easterby-Smith, Hamblin and Kirkpatrick.
9. Apply EoT models to a typical training function.
10. Resolve issues influencing EoT for a particular training programme.
11. Apply the EoT matrix to a particular training programme.
12. Select EoT to achieve specific purposes.
13. Carry out a role analysis for an organisation’s EoT function.
14. Review their own training and development function.
15. Analyse their own role in EoT.
16. Agree an EoT project proposal for completion following the workshop.
At the end of their EoT project participants are expected to:
17. Complete an EoT project in a selected area of training activity.
18. Prepare a report to management on their findings and recommendations.
19. Submit a written report of their project for assessment.
20. Be certificated by DoPT to carry out EoT projects.
Timetable for the Workshop
The Workshop is run over a period of five working days, starting at 10.00 on Monday and concluding at noon on Friday.
Participants, working as members of teams, should expect evening activities. The following is an indication of how we allocate the time, although may be adjusted to suit a particular course.
--- Time Table ---
Project
Distance learning, followed by the workshop gives participants an opportunity to develop skills and understanding of the process and techniques used for carrying out EoT. The project is intended to continue this development and to confirm their competence. They are required to carry out a practical EoT project of benefit to their institution or organisation. The choice of a suitable project is left to participants and their management to decide, although it should be concerned with a genuine area of training where an evaluation is desirable.
The course tutor will require the following details of the proposed project:
• Name of participant.
• Client organisation where the EoT will be carried out.
• Issues likely to be encountered
• A brief description of how the completion of the project will be of benefit to both the client organisation and the associated training institution.
Please note that the final date for accepting the project will be agreed with the tutor. Participants who complete their project successfully will be ‘EoT Certificated’ by the Department of Personnel and Training.
Assessment of the project will be based on the Project Assessment Form. This gives the criteria to be used for assessment, although we will give emphasis to the imaginative application of these features, as we do not intend them to impose a rigid set of rules. The criterion score for an acceptable project is 70%.
If a project is not up to the required standard, we will refer it. Tutors will give participants advice about how they can improve their project with an invitation to resubmit it. Please note that the course provision allows for one referral, only.
Welcome to the Evaluation of Training (EoT) Course
We believe that the EoT Course, for which you are to be a participant, offers an interesting and challenging experience. EoT deals with a vital, but often ignored aspect of training - for example:
• How can we show that training is a worthwhile investment?
• Why is the evaluation of training important?
• How can the quality of training be improved?
• Who should contribute to EoT?
The EoT course will give you interesting opportunities to seek answers to these questions, and many other issues that should concern both trainers and non-trainers. Our main concern is the use of training to help organisations in the Indian public services improve performance. During the course you’ll experience ‘learning by doing’ - getting involved with a variety of practical, challenging activities. These are likely to occur during the forthcoming workshop and during your project, which forms the final part of the EoT Course.
Briefing for Studying the Distance Learning
It is important to note that only a limited amount of time is available during the workshop - time that we believe is best spent on active learning experiences, rather than using passive, lecture-based, methods. Therefore, as part of your preparation for the EoT workshop, we would like you to complete the Distance Learning materials provided - documents eot-dlm-1 and eot-dlm-2. This material provides underpinning knowledge for use during the workshop. It is therefore expected that before attending the workshop you will have studied the material, and completed the following learning tasks.
Distance Learning Tasks
To use the underpinning knowledge in your local context we would like you to address the following questions. You may need to ask others in your place of work what their views are to get rounded answers.
1. What evaluation is currently carried out for training programmes/courses?
2. What are the purposes for carrying out evaluation activities?
3. Who is responsible for deciding what measures to use, using them and analysing results?
4. What is done with the information collected?
5. What values influence the approach to evaluation adopted in your organisation?
6. Who can benefit from investing more in EoT?
You will be invited to share your views about these questions on the first day of the workshop. In addition, on the first day of the workshop there will be a test of your recall of the distance learning materials for formative assessment purposes. There will be no pass or fail but the information will be used to provide feedback.
Eot Toolkit
eot-tools-03 Guidance on Learning Logs
eot-tools-05 Responsibility Mapping
eot-tools-06 Benchmarking
eot-tools-07 Immediate Reaction Questionnaire
eot-tools-08 Behaviour Analysis
eot-tools-08a Behaviour Analysis
eot-tools-09 Behaviour Analysis - Exercise
eot-tools-10 Stakeholder Analysis
eot-tools-12 SWOT Analysis
eot-tools-13 EoT Report
eot-tools-14 Course Review Questionnaire
eot-tools-15 Improvement Teams
eot-tools-16 Construction of Checklists and Rating Scales
eot-tools-17 Assessing Management Competence
eot-tools-23 Constructing Questionnaires
eot-tools-24 Types of Objective Test Items
eot-tools-25 Steps Involved in Constructing Objective Test Items
eot-tools-26 Steps Involved in Constructing Questionnaires
eot-tools-27 Steps Involved in Constructing Skill Process Checklists
eot-tools-29 Venue Checklist
eot-tools-30 The Test Test
eot-tools-31 Objective Test Items Checklist
Guidance on Learning Logs
PURPOSE
For adults learning is a voluntary activity that can take place continuously as part of life experiences as well as at formal learning events. Learning can be easier when it builds on the learners existing experience and when the learning opportunity is seen as relevant to the immediate work needs or personal development needs and aspirations.
Keeping a learning log encourages the learner to build in reflection time, so that new learning can be recognised, weighed up and integrated into practice.
Successful learning is only achieved when the learner is able to apply new learning to everyday activities.
Key questions in reflecting on learning opportunities are: “What happened?” “Why did it happen?” and “How can I use this learning in the future?”. Learning logs can help learners to answer these questions.
SUGGESTED USE
During the three parts of the EoT course - distance learning, workshop and project, you will encounter frequent opportunities to learn. Keeping a learning log will help you to recall your thoughts, experiences, reactions and ideas - which may lead to further development. You can also present this as evidence of your learning experiences and include your learning log, along with your project report, for assessment.
Suggestions for possible inclusion in the log:
1. Key points of interest from your learning experiences
2. Conclusions reached
3. Actions taken or planned that put the learning into action
4. Further learning / development needs
5. Personal achievements in terms of knowledge and skill gain
6. Personal reflections on the experience
7. Actions planned to apply, consolidate and integrate learning
8. Potential opportunities for further professional development, in respect to EoT
Learning Log
LEARNING LOG
INTRODUCTION
When planning an EoT, it=s important to establish who is involved B either providing you with information, or taking decisions based on your findings. Identifying these people, with reference to their
respective roles and authority, will enable you to answer the following questions:
• Who can provide information and in what way this can be done?
• Who will take decisions?
• Who will influence successful implementation of your recommendations?
As Responsibility Mapping is a tool that can be used for TNA, so some useful analysis may already have been done. If not, you can use this tool to focus attention solely on EoT.
PURPOSE OF THE TOOL
A common problem that occurs frequently when evaluating training is establishing roles, responsibilities and relationships of the people involved. This is not necessarily simply people designated as >trainees=, but also a variety of stakeholders from client organisations, funding agencies and training institutions. You can use this tool to build up a map of people involved - referred to on the map as >Actors=. The work they do in relation to the training intervention is shown on the map as >Actions=. The map can be built up through discussions with the people concerned to show their involvement; also their relationships and the influence they may have on the training process.
The successful use of Responsibility Mapping depends on working with a representative selection of trainees to identify significant stakeholders. The purpose is to obtain information from these people about a particular training intervention, which could include:
• Data and views about performance problems and training needs, especially in relation to criteria that is likely to be used for external validation.
• Information about people, designated as >trainees=, with details of their number and range of entry behaviours.
• Expectations of those concerned with the performance problems, especially regarding the success of the training intervention.
• Non-training factors - motivational and environmental - that may influence what can be accomplished by means of training.
• Resources available to support training, such as facilities, access to coaching and mentoring, and further remedial training that might be required.
• Information about other training interventions or organisational changes that may influence
what can be accomplished.
Although the points listed above are generic to EoT, access to these sources of information will be greatly facilitated by developing a responsibility map. Institutionalising the use of this tool, for both TNA and EoT should enable you to gain access and a professional relationship with key people.
Responsibility Mapping
DESCRIPTION
Although done better following a TNA consultancy - when this and other TNA tools have been used to analyse and confirm specific training needs - you can nevertheless use responsibility mapping as a discrete tool for EoT. However, lacking information obtained from the use of other TNA tools, you=ll find some limitations, which we=ve taken into account in this EoT version.
During training, trainers and trainees are likely to be conscious of problems or difficulties associated with people and their relationships - especially confusion over responsibilities. These are not necessarily explicit, or even clearly defined - perhaps based on comments made by individuals, or issues raised during training activities. Typically, this Responsibility Mapping tool helps you to identify many of the actors involved and their relationships with others. The actions carried out by these people can also be listed.
You can use the matrix, shown on page 5, to show relationships between actors and their actions in relation to a training intervention. These relationships can be examined in more detail by using the following codes:
R Responsibility - where an actor has responsibility for an action, or responsibility for the consequences of the results of carrying it out. Note that having responsibility does not necessarily mean that an actor has also the authority. In such instances, training needs can occur when an actor has to seek authority, or decides to >delegate upwards= actions that he or she should be capable of taking.
A Approval - within a working unit there is likely to be a hierarchy, which may require an actor to seek permission or approval before taking action. The actor designated with approval has authority and the right to veto the proposed action. The need to seek approval from a senior actor, or somebody acting as a >gatekeeper=, can create barriers to learning.
S Support - resources, such as people, facilities, funds, equipment or further training available to support actions of other actors. These actors, who are depending on resource support, may experience performance problems because of delays, or a failure to supply essential support resources.
I Inform - coordinating actions requires effective communication. Actors need to be informed of actions being taken. Often information has to be communicated across functional or departmental boundaries, a likely source of barriers to effective and efficient learning.
HOW TO USE IT
A responsibility map can be evolved as training is being developed, or during its implementation and as a feature of external validation. For a new training intervention, and without benefit of TNA, you are unlikely to be aware of stakeholders, nor have had much contact with representative trainees. The map is therefore likely to evolve as you contact the people involved and gain an insight into their roles, responsibilities and relationships with others. The map can provide an informal basis for planning external validation, helping you to:
Responsibility Mapping (eot tools -05)
• Plan and assess assignments and projects.
• Involve stakeholders with EoT.
• Evaluate links between off-job training and further on-job development.
• Contrast EoT evidence from internal validation to that obtained externally.
• Identify actors or actions that block success.
• Review the quality of training provision.
Generally, as the map evolves, you can identify issues and the people involved. This could be during the
training, perhaps where on-job-training or open learning is being used or following attending a training course. Typically, this might be concerned with:
- Delays in carrying out activities concerning the transfer of learning involving of people across teams, functions or departments, where actions or commitment is essential for successful performance.
- Failure to delegate authority to (trainee) actors, who may have responsibility to take action but lack authority or confidence.
- Either too much or too little information is being exchanged. Both can result in confusion and delay - perhaps due to (trainee) actors being swamped by nonessential information, or because they cannot take action due to a lack of information essential to their performance.
Using your map of responsibilities and relationships, you can evaluate some learning purposes, spot where training can be improved and establish the basis for monitoring performance.
Depending on the people being trained, you can encourage trainees to develop their own version of this tool, so that they can make decisions about their development. This could be used also as part of EoT.
By interviewing several actors yourself, or getting trainees to do this themselves, valuable information can be obtained for most cells within the EoT Matrix. Consider doing this by asking questions based on codes used to define responsibility mapping. For example:
Responsibility Mapping (eot tools - 05)
R
Who is responsible (for this action)?
Does this person have sufficient authority to ensure a successful outcome?
What happens if the responsible or person is absent?
A
Whose approval is needed before action is taken?
How is approval sought?
What information does this person need to approve action?
What are the consequences (financial, timescale, etc.) in delaying approval?
S
What support is needed to enable action to be taken?
Who (which actors) are required to support this action?
How are resources allocated or authorised?
What happens (examples) when only limited support is available?
How can resource support be improved?
How can commitment be gained?
I
Who (which actors) need to be informed about action taken?
How are they informed, and in what medium?
What are the consequences of a failure to inform?
How can the information system be improved?
Responsibility Mapping (eot tools - 05)
Focus of Attention: ______________________________________________
Key:
R = Responsibility (not necessarily authority)
A = Approval (right to veto)
S = Support (put resources towards)
I = Inform (to be consulted before action but with no right of veto)
PURPOSE OF THE TOOL
Benchmarking goes beyond just looking at other organisations. The choice of organisations to compare is vital, illustrated by the following definition:
Benchmarking is the continuous process of measuring products, services and practices against the toughest competition or those organisations recognised as industry leaders. Kearns, 1990
For EoT within the Indian public service perhaps the main purpose of benchmarking can be a continuous process of measuring courses, other professional services and training methodologies against those training institutions recognised as leaders.
Note that this definition refers to the continuous nature of the process. It serves to point out the importance of always being prepared to look outwards to inform practices within.
To be more specific there is at least four purposes/perspectives that benchmarking processes can be focussed on:
1. Establishing the operational and strategic value to an institution of benchmarking training and development activities.
2. Demonstrating that lessons can be learned from benchmarking that will contribute to the enhancement of training and development processes, practices and philosophies.
3. As a by-product to show how comparison with other institutions can serve to strengthen the negotiating hand of those responsible for its management.
4. Demonstrating that benchmarking can be a key contributor to institution and organisational learning and should be seen as a process in its own right.
DESCRIPTION
The benchmarking process can generally be categorised as either of the following:
Competitive Benchmarking. This is concerned with assessing key parts of an institution’s processes, systems and procedures with those of a similar institution. To improve effectiveness and efficiency it is often held that an institution needs to reduce costs, improve productivity, enhance quality, be more customer-focussed, provide good service and be entrepreneurial and innovative. The development of new products, introducing new technologies, better marketing strategies may also put them in the lead position. Published information and personal contacts may provide most of the data for competitive benchmarking.
Best Practice Benchmarking.
The focus here may not be on competitors but to review institutions seento be ‘best in class’ in whatever field their business. Usually it’s easier to get access to these institutions when they are not competitors and will probably be more open. Because best practice benchmarking is about recognising and learning from others, who are not competitors, visits are an acceptable and usual way to collect information and offers potential benefits to the organisation being visited in gaining another’s view.
Either of the above two approaches can be compared at different levels and three are commonly used:
• Strategic benchmarking involves comparison of different strategies to identify key elements in a successful overall strategy. In terms of institutional training and development, this could be how other institutions set about identifying, developing and improving training provision.
• Process benchmarking focuses on an evaluation of institutional systems, procedures and processes that cut across functional areas. Perhaps comparison of systems underpinning performance management and staff development fits into this category.
• Operational benchmarking which focuses on the relative cost position, increase in product quality or improved service provision across functional areas for the training and development function. One measure could be the cost of training per trainee; another could be utilisation of facilities.
HOW TO USE IT
Whilst there is no definitive approach to how the benchmarking process should be used, one is based on Deming’s advice of ‘plan, do, check and act’.
1. Identify the function and activity to be benchmarked.
2. Choose the organisation for benchmarking against.
3. Determine methods of data collection and analysis.
4. Familiarise organisation with findings and set performance goals.
5. Implement and measure.
In more detail:
1. Identify the function and activity to be benchmarked.
Having identified the function and why benchmarking is necessary, a choice has to be made either to undertake general benchmarking across the range or to be more selective. The following diagram illustrates some decisions to be made.
2. Choose institution to benchmark against.
We have already mentioned either a competitive benchmarking or a best practice approach. We can use four different types of organisations for benchmarking purposes.
1. Internal benchmarking - against other parts of your institution.
2. Benchmarking against directly competing institutions - eg ones offering similar courses.
3. Similar institutions - such as other ATI’s.
4. Institutions in totally different sectors.
Further methods of identifying benchmarking opportunities would be to gain information from:
• Research publications, articles, reports, and books.
• Quiz experts, experienced people and consultants.
• Contacts gained through funding agencies, professional bodies and databases.
• Personal contacts, stakeholders, trainees, customers, suppliers, etc.
3. Determine methods of data collection and an analysis.
There are obviously many ways to collect data. With competitive benchmarking this may be more covert and drawn from written data in the public domain, to seeking opinion from friends and colleagues. Whilst best practice benchmarking can be more open, site visits and meetings with institutional directors, for example, can be helpful.
4. Familiarise institution with findings and agree performance goals.
Establishing goals for improvement involves careful planning to incorporate new processes and practices. Some resistance may be inevitable if a challenge is being made to the established ways of doing things. Communication with all stakeholders is vital to ensure that any recommendations have sufficient commitment and time frames.
5. Implement and measure.
In competitive terms, benchmarking should be viewed as a means to improve performance to gain superiority. In best practice terms, it should be viewed as a means of doing things better.
Improvements should result in significant leaps as opposed to small incremental changes although the change process itself may be gradual and incremental. The implementation of improvement or change should involve periodic measurement and assessment of attempts to reach stated goals. Implementation should be the outcome of a process, the danger may be that in benchmarking you may identify a range of possible improvement areas / processes that seem effective.
Some final thoughts:
Some useful general points to bear in mind in benchmarking:
• Be realistic - benchmarking is not a solution to all problems.
• Stay focussed - massive problems cannot be solved with one change.
• Prepare carefully - both with people involved and institutions chosen.
• Beware legal problems, data protection and using sensitive information.
Before seeking agreement for implementation, ask yourself such questions as:
• Is this a good idea?
• Will it work in our institution?
• How will it work?
• Who will drive its implementation?
Immediate Reaction Questionnaire
NAME : ____________________ OR NUMBER : ______
Please tick () on EACH of the scales to indicate your opinion to the statement and the scale.
1.0 SUBJECT COVERAGE
1.1 How much of the subject in relation to your expectations was actually covered?
None of it All of it
1.2 Generally, how much of the training you experienced seems likely to be of practical value to you?
None of it All of it
1.3 How much of the concepts and practices of EoT was new to you?
None of it All of it
1.4 Generally, how much do you think you learned about EoT?
Very little A great deal
1.5 Was the level of treatment of EoT:
Too low Too high
1.6 Was the amount of time given to studying EoT:
Too much Too little
2.0 TOPICS
2.1 Distance learning materials: Not useful Very useful
2.2 EoT Models (Easterby-Smith and Kirkpatrick): Not useful Very useful
2.3 EoT Matrix: Not useful Very useful
2.4 EoT tools and techniques: Not useful Very useful
2.5 StipTrain case study: Not useful Very useful
2.6 EoT role analysis: Not useful Very useful
2.7 In-tray exercises: Not useful Very useful
2.8 EoT project briefing: Not useful Very useful
3.0 STANDARD OF TRAINING
3.1 Help and advice: Very Unsatisfactory Very Satisfactory
3.2 Practical work: Very Unsatisfactory Very Satisfactory
3.3 Reviews: Very Unsatisfactory Very Satisfactory
3.4 Training materials and visual aids: Very Unsatisfactory Very Satisfactory
4. ADDITIONAL COMMENTS:
DESCRIPTION
Behaviour Analysis is a method of collecting information objectively about the content of the behaviour shown by individuals operating in groups and about the content of the behaviour of the group as a whole.
One or more observers carry out the analysis. They observe the group in action and identify each contribution made by each member of the group, categorise that contribution and record it against the name of the contributor and the category (see Appendix A for an example). At the end of the group activity, totals are made on the Analysis Sheet for categories and contributions from each individual and
from the group as a whole.
The analysis is used to give feedback to individuals and/or to the group as a whole on their contributions and the nature of their contributions. Learners can use this feedback as the basis on which they plan to modify their behaviour in future group interactions. The modified behaviour in turn can be analysed and feedback given on the extent, nature and effects of the change.
USES OF BEHAVIOUR ANALYSIS
This method can be used to increase the awareness of the learners of the kind of behaviour they use in social interactions. This can help them to modify behaviour where this is appropriate, and generally, to develop skill in a wide range of social interactions at work, both in one-to- one situations and in larger groups. For example, it has been used in social skill training in the following areas:
• Working in teams and groups
• Participating in meetings
• Customer contact
• Telephone selling
• Selection and appraisal interviewing
• Negotiating
• Consensus seeking
ASSUMPTIONS
There are four major assumptions underlying this approach to developing social skills:
Behaviour matters
The individual's behaviour in social interaction influences the behaviour of the other person or persons
involved and the results achieved by the interaction.
No single style is correct for all situations
The object of Behaviour Analysis is not to influence learners to use certain types of behaviour labelled "good" or "bad". The emphasis is on identifying the kind of behaviour used so that the participants can then consider how appropriate it is to the situation and to what the participants are trying to achieve. The situation may demand different roles and different patterns of behaviour from different participants, e.g. the role of chairing a meeting usually required the person to exercise control over the other participants and therefore to use the kind of behaviour which would not be acceptable from any of the other participants.
Behaviour must be observed accurately
If change is to take place and social skills are to be developed, accurate, clear and precise information is
needed on the kind of behaviour participant’s use. Incomplete, ambiguous and subjective information will distort feedback to the learner and is likely to be rejected.
People can plan and control their own behaviour The ability to use behaviour appropriate to the situation and to the participant's objectives is a skill, which can be developed with practice and feedback.
DEFINING A BEHAVIOUR CATEGORY
The need to observe behaviour accurately implies that the behaviour categories used in the analysis should be clearly defined. Before including a behaviour category in the analysis, the trainer should ensure that it meets the following criteria. It must:
Describe a behaviour, which can change during training
Be meaningful to both the trainer and the learner
Be readily identifiable during the interaction
Be behavioural, i.e. it must categorise observable behaviour without the need for interpretation of what is going on "beneath the surface" about attitudes, motives, emotions, etc.
Permit a high level of reliability between observers (i.e. when more than one observer is present at an interaction in which that behaviour is used, all the observers will have identified it as having been used).
Define a relatively simple unit of behaviour distinct from the other categories used.
The categories commonly used in group observation are listed and defined in Appendix B. It is important that any categories used should be defined in this way, for the benefit of both the observers and the learners.
OBSERVERS
The observers have a demanding role to play in Behaviour Analysis. They need to have a thorough knowledge of the distinction between the various categories used. They also need to be sensitive to what is being said and how it is said in the interaction so that it can be recorded accurately. They must have sufficient practical experience of using Behaviour Analysis to be able to provide consistently reliable information.
It is important that observers are able to select the categories so that they can analyse particular situations.
In providing feedback, the observer's aim is to assist learners to modify behaviour to meet their objectives. It is NOT to try to impose the observer's own ideas of what they should do.
People carrying out the role of observers should be briefed on the context, purpose and objectives for the learning event they are observing. They may also require training themselves on both the concept of
behaviour analysis and the practical process of recording observations.
ADVANTAGES
• Behaviour Analysis concentrates on observable behaviour and is therefore more readily accepted by learners rather than methods that explore attitudes, motives and emotions.
• It provides immediate, detailed, objective and non- threatening feedback on social skills performance.
• It can be learnt quickly and applied immediately to real-life situations.
• It provides the learners with a means of modifying their behaviour and giving them an awareness of their existing behaviour.
• It is highly flexible; it can be adapted to develop specific sets of behavioural skills.
DISADVANTAGES
• It requires skilled observers who can provide consistently reliable information.
• An observer can only cope with a few behaviour categories.
• It tends to concentrate on verbal behaviour and pay less attention to nonverbal behaviour.
• It provides feedback on the number and type of contributions rather than on the quality of these contributions, or when these contributions were made.
• The selection of categories appropriate to a particular situation is difficult.
• It does not provide information on which contributions were made to which people.
Appendix A
----- Table ----
Appendix B
BEHAVIOUR ANALYSIS
Proposing-
A behaviour which puts forward a new suggestion, proposal or course of action.
"Let’s deal with that when we come to item 5".
"I suggest that we talk about the overall structure first and get down to the details later."
Giving Information
A behaviour that offers facts, opinions or clarifications to other people.
“We can buy these items for Rs50 each”.
“There should be no problem completing the job on time”.
Supporting
A behaviour that makes a conscious and direct declaration of agreement or support for another person or their ideas and opinions.
“Yes I agree with Radha”
“That is a good idea.”
“Radha is right. That’s what we should do.”
Building
A behaviour that is usually in the form of a proposal, which extends or develops a proposal made by another person.
“Yes that’s an excellent idea. And if we costed it in detail we could present it as an alternative.”
Seeking Information
A behaviour that seeks facts, opinions or clarification from another person.
“How much would that cost”?
“When would that happen”?
“Can you tell me the facts on which your view is based”?
Summarising
A behaviour, which summarises, or otherwise restates in a compact form, the content of previous discussions or events.
“So were saying that we ought to go ahead with this project that we must not spend more than two lakhs and we can?t start before May.”
Disagreeing
A behaviour which states a direct disagreement or which raises objections to another person’s concepts or opinions
“That idea wouldn’t work here.”
“No, you’re wrong. It’s not like that”.
“The figures show it”s not true”.
Bringing In
A behaviour, which invites views or opinions from a member of the group who is not actively participating in the discussion.
“Radha, administration is your job. What do you think“?
“Could I ask Radha what her feelings are, as the representative of the general office”?
Shutting Out
A behaviour that excludes, or attempts to exclude another person.
Any interruption.
Any attempt to “talk over” what someone else is saying
Defending or Attacking
A behaviour that attacks another person either directly, or by defensiveness. Defending/attacking behaviours usually involve value judgements and often conveying emotional overtones. Note that defending or attacking is usually about people not issues.
“That idea can’t work”.
“It”s not my fault. It”s not my job to update records”.
“Why can’t you do what you promised to do”.
Testing Understanding
A behaviour that seeks to establish whether an earlier contribution has been understood.
“So, you think we should upgrade our IT system now”?
“Is Radha suggesting that we ought to cost the project first, before ordering new equipment”?
ASSESSING INTERACTIVE SKILLS
Listed below are a number of statements, which were made by participants in a meeting of managers. In order to check on your interpretation of the behaviour analysis category definitions, please read each one and tick the category which you think fits the statement best.
1. "Let's start at page 7"
Building
Proposing
Giving Information
Testing Understanding.
2. "We're getting nowhere"
Giving Information
Defending/Attacking
Blocking/Difficulty Stating
Disagreeing.
3. "So, what you're saying is that we should get rid of him. Is that right?"
Seeking Information
Summarising
Testing Understanding
Open
4. "So now, it's MY fault is it? Well let me tell you this, it's about time you..."
Disagreeing
Proposing
Giving Information
Defending/Attacking
5. "I think Fred's idea would work if we included an additional form of access in the central area."
Building
Supporting
Proposing
Giving Information
6. "I'm afraid that I'm not very good at financial problems so I'm going to need advice."
Seeking Information
Open
Giving Information
Testing Understanding
7. "I suggest that we consider the last point first."
Building
Proposing
Giving Information
Testing Understanding
8. "We're going round in circles."
Giving Information
Defending/Attacking
Blocking/Difficulty Stating
Disagreeing
9. "Can I just check that I've got this right? You want the first three by Monday."
Testing Understanding
Summarising
Seeking Information
Open
10. "Well, it could be attractive - even more so if we combined it with Joe's point about selection"
Building
Supporting
Proposing
Giving Information
Reviewing a Training and Development Function
INTRODUCTION
People in management or senior technical, administrative or professional jobs frequently perform tasks that are regarded as ‘productive’. Other people, or the same person on other occasions, will perform a similar task differently and achieve an acceptable outcome. Assessing whether people performing such tasks are competent depends on having:
1. Clearly defined standards of performance, as in a performance objective.
2. Relevant criteria specifying evidence a person should provide to confirm their competence.
3. Methods for assessing the evidence.
Although here we are concerned with how to assess competence, it should be noted that this can only be done when performance standards and relevant criteria are available. Management competences may be assessed in a variety of ways, some of which are similar to methods used elsewhere in the assessment of practical and/or cognitive skills, and by employers when selecting or evaluating staff. The following is a list of methods suitable for assessing competence using both work based evidence and activities to generate evidence upon which to make a judgement.
Assessment Centre
Assessment centres are used to judge competence using simulation exercises and other off-the-job assessment techniques. They employ observer-assessors and are used for staff selection, promotion, or development. In one programme sampled in the survey, an assessment centre used for recruiting graduates was adapted for use in assessing competences. There were five simulation exercises designed to allow managers to demonstrate behaviours associated with five basic management competences. Each exercise was intended to demonstrate several behaviours and competences. There were two group
exercises, a presentation, an in-tray exercise, and an interview simulation. Several assessors observed the managers’ behaviours from which they inferred competence.
Assignment
An assignment consists of any problem-solving exercise set with clear guidelines and specified length. While assignments are difficult to define precisely, they are more structured and less open-ended than projects. Assignments may deal with real work problems or with hypothetical questions. In the programmes surveyed assignments were normally presented in written form.
Case Study
Case studies, consisting of a description of an event concerning a real-life or simulated situation, usually as a text, a video, a picture or a role-play exercise. People are assessed as to their competence in analysing the situation presented, drawing conclusions and making decisions or suggesting courses of action. This method of assessment may be carried out with individuals or small groups.
Reviewing a Training and Development Function
Group Exercises
These are designed to simulate situations where managers meet and work together. There can be various formats:
• Competitive where individuals are in competition with each other for something, say a limited budget, and therefore have to utilise their negotiating and influencing skills to get what they want;
• Co-operative where individuals have to work together as a team to achieve a group goal;
• Leadership where one person in the group is nominated to be leader for a particular task.
Interview
The personal interview is probably the oldest and best-known means of eliciting information directly from managers. An interview can be two-way, providing managers with the opportunity to question the interviewer. It is most widely used to generate evidence of process-based skills.
Interview Simulation
These exercises can be designed to simulate interview situations in which the manager typically finds him/herself (e.g. staff counselling, disciplinary interview, selection). Assessors can observe how competent the manager is at dealing with staff in these situations.
In-tray Simulation
In-tray simulations are designed to accurately simulate the contents of a manager’s in-tray. Managers work through the contents, make decisions and take appropriate action. An in-tray exercise is particularly useful in gaining evidence on planning, organising and prioritising.
Log-book/Diary/Personal Journal
A self-report technique in which managers generate evidence of their performance, progress, experiences, attitudes and personal feelings in an on-going situation.
A log-book can, if properly structured, provide a useful means of assessing the progress of a manager’s achievements or performance. To be effective, it should have a clear specification and give guidance on how essential information is to be recorded.
Portfolio
The portfolio owes its derivation to the artist who presents a selection of his/her work to the prospective employer to demonstrate his/her ability. For managers a portfolio might include a business plan, a marketing plan, an investigative report, an extended CV, a video of a presentation or an interview conducted, etc. The purpose is to present evidence of the manager’s performance.
Presentation Exercise
These are used to provide evidence of oral communication, interpersonal, and analytical skills. They provide the opportunity for managers to put forward their ideas in a face-to- face situation and to demonstrate their ability to persuade the audience to their point of view.
Project
Projects have the following characteristics:
• Are more comprehensive and open-ended than assignments;
• May be tackled by individuals or by a group;
• Usually involve a significant part of the work being carried out without close supervision;
• Involve less direction from a third party.
All projects involve the following stages:
1. Planning the project.
2. Devising the investigation strategies.
3. Conducting the investigation.
4. Analysing the results.
5. Reporting on the conclusions.
Role-play
In role-play, managers are presented with a situation, often a problem or incident, which they then have to respond to, by assuming a particular role. The enactment may be totally unrehearsed or they may be briefed in a particular role to be played. Such assessment is more open ended and person centred than simulation and can be used to assess a wide range of the behavioural and interpersonal skills required in the world of work.
Self-report Techniques
A self-report technique is any form of assessment in which managers generate the necessary evidence of their performance, experiences, attitudes and personal feelings in an ongoing situation.
Simulation
A simulation is a structured practical exercise with specific objectives involving the organisation and completion of a particular task, which is product - or resource - based that seeks to simulate real-life conditions. To be effective, simulations must succeed in recreating the atmosphere, conditions, and pressures of the real situation.
Written Examinations
Written examinations are a traditional form of time-constrained, summative assessment. They may be closed or open book, seen or unseen, examinations.
Written Exercise
These are used to provide evidence of analytical ability and written communication skills, e.g. the ability to impart information and ideas accurately and precisely when communicating in writing; the ability to construct a clear, logical and persuasive argument; the ability to present issues and problems in a broad perspective.
Stakeholder Analysis
PURPOSE OF THE TOOL
It’s possible to view success or failure of training from a variety of perspectives. Stakeholders involved with a particular training programme are likely to have a range of views. When planning and carrying out EoT it is worth considering whether these views are positive, negative or simply indifference. What one stakeholder views as a successful outcome may not be true of others – and what effect will these views have on decision-making? The process of exploring the views of stakeholders can have a significant impact on evaluation. People who have a positive view of EoT are likely to become proactive allies, whereas the views of others may limit what can be achieved through EoT and what methods can be used. Stakeholder analysis can include the following features, although for some situations you may choose to do only the first three:
1. Identifying who are the stakeholders involved
2. List out the EoT issues that we wish to address.
3. Establishing the views of stakeholders towards EoT initiatives.
4. Mapping the power of a stakeholder to take action against the dynamism or predictability of their willingness to use that power.
5. Mapping the power of a stakeholder to take action against the interest they have to do so.
1. Identifying Stakeholders
The first step is to be clear who are the stakeholders involved in a training programme and any EoT initiative being considered. As well as participants and trainers delivering the programme there are the managers of the participants, those who designed the programme, and then those funding the programme, the managers of the training institutes, the departments employing the staff and the DoPT. Some of these will be easily accessible to the EoT team and others difficult to reach to gain their views or to seek information from for EoT. Stakeholders can be treated as individuals or groups. If there are large numbers of stakeholders it may be preferable to treat some as groups and only identify individuals in the analysis when they have high importance for the particular initiative.
2. EoT Issues
The issues to be addressed through an EoT initiative need to be identified. This may involve use of the EoT Matrix in the development of your EoT initiatives.
3. Stakeholder Views
This analysis records stakeholder views and they may take one of four different stances to your EoT initiatives. Their stance can be:
Supportive – seeing the benefits of EoT and giving whatever help is required.
Neutral – perhaps also indifferent to EoT, neither helping nor hindering your evaluation.
Opposing – possibly seeing EoT as a threat.
Uncertainty – unsure of what to expect or divided opinion about the consequences.
The analysis can be presented in the form of a matrix, as illustrated in Fig 1. Attention here is on the attitudes of stakeholders towards EoT initiatives. Three of the stakeholders shown are ‘internal’, for example, members of staff delivering the programme at a training institute, the trainer faculty and the institute management. The other two ‘external’ stakeholders are say participants and staff at a client organisation.
Using the following key, we can record their views to a list of issues concerning EoT.
Key + = Support
0 = Neutral
- = Oppose
? = Uncertainty, showing divided opinion
Internal External
CHANGES / ISSUES ‘V’ ‘W’ ‘X’ ‘Y’ ‘Z’
1. Revise IRQ + - - + +
2. Introduce product/process checklists + - 0 + +
3. Re-design objective test items + - - ? 0
4. Introduce learning logs 0 0 - ? ?
5. Involve managers in evaluation ? + - 0 +
6.Propose cost/benefit analysis + - 0 + +
4. Mapping Power/Dynamism
This analysis uses another matrix to examine the power individual stakeholders have against the likelihood of them using it. One axis looks at their power, which could be direct – they have seniority or status or control the budget, or indirect, due to their position within their organisation giving access to information or providing influence. The other axis estimates their predictability to use their power.
PREDICTABILITY
This may be applied to each stakeholder in turn to provide a profile of those we need to consider carefully and those who can be expected to not present any challenges.
So for stakeholder V we may rate them as B
So for stakeholder V we may rate them as C
So for stakeholder W we may rate them as D
So for stakeholder Y we may rate them as D
So for stakeholder Z we may rate them as A
So for stakeholder X we may rate them as B
The stakeholders who need to be treated most carefully are X & Y and one of them is internal and the other external.
5. Mapping Power/Interest
This is a development of the power/dynamism matrix and attempts to clarify a stakeholder’s attitude on two axes - the power they hold and the extent to which they are likely to show interest in proposed EoT initiatives.
LEVEL OF INTEREST
This may be applied to each stakeholder in turn to provide a profile of those we need to consider carefully and those who can be expected to not present any challenges.
So for stakeholder V we may rate them as P
So for stakeholder V we may rate them as Q
So for stakeholder W we may rate them as S
So for stakeholder Y we may rate them as R
So for stakeholder Z we may rate them as S
So for stakeholder X we may rate them as P
So the stakeholder who needs to be treated most carefully is Y and V needs to be kept informed.
HOW TO USE
Having introduced stakeholder analysis we can now consider how they can be used.
Let us return to Fig. 1, where issues relevant to the process of EoT are listed against five stakeholders - both internal and external. As an example, take one issue: ‘introduce product/process checklists’ from the matrix. It shows three stakeholders willing to support that issue, one who may oppose it and one who is neutral.
The example shows a completed matrix, with each cell having a key indicator. However, in reality, you’ll have a blank matrix requiring answers to the following questions:
• Who are the stakeholders likely to have influence on the EoT initiative? Are some more important than others? Which of them should be included in the matrix?
• Which issues within the EoT process are likely to present problems – either to you or to some of the stakeholders?
• How do you propose making decisions about a stakeholder’s attitude? As far as possible, base your judgement by talking to the person, or with peers or documentary evidence that reveals their attitude.
• Having identified stakeholder attitudes to specific issues how are you going to take this information forward? What strategy will you employ to convince the doubters of the benefits of EoT? How can you use the positive attitudes of some stakeholders to influence others? A tactic could be to gather a network of like-minded people as your allies, who may help to shift opinions.
• Stakeholders who are neutral probably need less attention, although you should keep them informed. Perhaps by becoming involved in discussing issues, they will develop a more positive attitude.
It really comes back to the old saying “To maximise the positive and minimise the negative.” We might build on this stakeholder analysis by using the mapping power/dynamism matrix, shown in Fig.2. Using the same example - to introduce product/process checklist and internal – where stakeholder ‘W’ is opposed to this issue. However, the potential negative effect that this attitude will have will depend on other factors - critically the power that the stakeholder has in the organisation. This can be considered alongside the dynamism or predictability of this person to take action. Obviously, segment D has the potential to be of concern in that the stakeholder has high power and low predictability. The value of placing stakeholders in any segment is that you can plan where you put your effort and resources to ensure they allow you to implement EoT initiatives.
Another development of the use of this tool is to also use the power/interest matrix tool as illustrated in Fig.3. Segment D, for example, highlights key players who have both high power and a high interest in EoT initiatives. It will be important to develop your EoT initiatives from the sound basis that such people provide.
SWOT Analysis
PURPOSE OF THE TOOL
Although EoT is often concerned with details of training - reactions to learning, improved performance and so on - there’s also the need to take a more strategic view. For example, taking a typical course being run at a training institution, what value is being obtained from a considerable investment? Should the course continue to be run, or are there perhaps better options to meet the same training need? Should a training institute, or indeed a funding agency, continue to support the provision of this course, or could resources be better directed elsewhere? Lot’s of strategic questions may need evaluating before decisions can be taken.
SWOT analysis is a well-known tool of management, often used to help clarify business issues. It’s a tool we’ve adapted for use with the TNA Toolkit (Tool 2.1), and one that can be adapted again for EoT.
The purpose of the EoT version of SWOT analysis is to examine strategic issues concerning training and development. It has particular significance following the government’s declaration of providing ‘training for all’ and the introduction of IT-based learning systems.
DESCRIPTION
'SWOT' is a mnemonic for an analysis of four factors related to training performance:
Strengths What are the strengths of a training institute or function, or of a particular training strategy that is or could be used? What are trainers or other contributing to the institute or function proud of? What evidence is there that existing training has pleased client organisations, trainees and funding agencies? What is the potential to improve either the effective or efficient use of available resources?
Weaknesses Where is a training institute, or a particular training function weak? What are the weaknesses of an organisation’s training function? In the opinion of trainers, training administrators, stakeholders and trainees, at all levels from senior officials to lower cadre workers - and from customers and beneficiaries, where are the deficiencies?
Opportunities In what sort of ways can training and development, generally, or training institutions or organisational training functions take advantage of opportunities created by technological changes, or government initiatives?
Threats What changes to technology, or government policies are likely to be dangerous to the existing strategies and resources being used to provide training in the public sector? Are expectations or demands likely to present threats, especially to training institutions, or the roles and responsibilities of trainers?
The analysis can be focussed in different ways to suit the purpose of the analysis. For example:
• External issues influencing success of a training institute.
• Internal issues determining how well a training institute can utilise available resources to satisfy client organisations, funding agencies or trainees.
• A combination of both internal and external issues that together provide a basis for analysing aspects of a training institute’s performance - or of an organisation’s training function.
At an early stage in an EoT, you may choose to use this tool to obtain answers to such questions as:
Developments in Society
How will changes in society influence the organisation?
What changes in government policy or legislation can be expected?
Customers
How big is the market into which training institute provides its services?
Will this market change, and in which way?
What will be the purchasing habits of its customers?
Which tendencies will affect an institute’s market opportunities?
How can these opportunities be exploited?
Will there be a change in the volume of business?
What level of quality are customers willing to pay for?
Funding
How many funding agencies does the institute have?
How will the most important agencies develop resource provision in the future?
Is there likely to be an increase or decrease the number of funding agencies?
Competitors
How many competitors are there?
Who are the most important?
How will they affect the market for the institute’s services?
How are the market for the institute's products and services likely to change?
What is the level of price, quality and supply compared to competitors?
Services
What demands are customers likely to make on the organisation's services?
What changes are likely to occur?
To what extent is the institute able to meet customer requirements?
Which services are likely to be more in demand in the future?
What are the consequences of falling demand for some institutional services?
Technology
What technological changes will affect the institute’s operations and business?
What is the institute strategic planning in terms of investment in programmes or resources?
Are any of the present programmes are likely to become redundant in the near future?
Workers
What is present staff levels compared to the business the institute is currently attracting?
Is this likely to change in the near future?
What are the current levels of competence relative to what is desirable?
Which category of worker has the greatest potential to improve performance?
Are workers sufficiently flexible to enable them to respond to business demands?
Is the institute able to retain its best, most competent workers?
Are workers willing to be trained?
Would they respond positively to opportunities for professional development?
What factors may cause workers to resist opportunities for training and development?
Although these questions are directed toward training institutions, ones very similar could be asked of an organisations’s training function. Nor should a funding agency feel absolved from such analysis.
HOW TO USE IT
Decide the issue(s) to be analysed. The examples given above indicate typical issues that can be used.
You may decide to use several SWOT analyses, each dealing with a major issue. Remember, your consultancy is for training purposes so don't get distracted into issues where you have no expertise or authority.
You may choose to compile a SWOT analysis yourself using information obtained from interviews.
Alternatively, you could invite stakeholders, trainees and others with an interest in training and development to contribute.
Focus attention exclusively on the issue being analysed. Otherwise, the quality of the analysis will be impaired by trying to include too many disparate factors.
Use the attached form to compile the SWOT analysis. Initially, move from one factor to another and then, finally 'brainstorm' to obtain additional items.
Issue: STRENGTHS, OPPORTUNITIES, WEAKNESSES, THREATS
EoT Report
PURPOSE OF THE TOOL
As part the Terms of Reference for an EoT assignment, which has been agreed with stakeholders, may be the requirement to submit a written report. Reports are an important part of the EoT process as it contains information and recommendations that are need for decision-making. Although you can outline your findings and recommendations during an oral presentation, it is usual and more effective to provide a written report. This will enable decision-makers to discuss the information obtained and your
recommendations.
The purpose of this tool is to suggest guidelines for presenting your report.
DESCRIPTION
To enable your stakeholders to benefit from your EoT assignment, it is essential that your report is readable. Information should be presented in a way that is easily understood by those involved in taking decisions that are based on your findings and recommendations. So, however relevant and detailed the information, and however sound your recommendations, it is essential that you write the report with style and substance. Since reports are often formal, official documents, they are usually written using formal language. Nevertheless, taking into account the roles, responsibilities and status of the stakeholders who have to read it, you can:
• Use a style that is well structured and simple to read. A lack of jargon and using familiar vocabulary will help readers’ understanding. They may be reading the report in a second language.
• Be tactful when describing problems and their causes. Don’t offend people who are likely to be involved in either taking decisions, or implementing them. You need their active, willing participation - not indifference and resentment.
• Help people to feel a sense of ownership of your findings and recommendations. Include in your report references and contributions by stakeholders, members of staff, beneficiaries, customers, etc.
• Rather than using formality to present your findings, include examples likely to be of interest to readers of your report. Consider using analogies to illustrate what might be a complex, sensitive situation.
• Be aware of your relationship to the stakeholders. Although you are acting on their behalf, as an EoT specialist, your status will be an important factor. Avoid allowing your style to become subservient, when dealing with senior people. Equally, don’t become patronising or authoritarian to others subordinate to you.
• Use positive, objective and direct recommendations, based on substantive findings. Each recommendation should be a basis for action that is acceptable, practical, realistic and motivating to all concerned.
• Wherever possible, write your report on a computer. This will allow you to compile the report where sections can be patched together in an acceptable structure. Use grammar checks available with word processing programmes to help simplify content.
• Initially, submit a draft report so that your stakeholders can comment on the content. This will enable you to allow for their views when carrying out a final edit.
HOW TO USE IT
Although there’s no standard format fro an EoT report, the following headings are likely to feature in a
report structure:
Title Page Clearly state the subject of the report, also references to stakeholders, associated client organisations, authors and date.
Contents Page List headings, subheadings and appendices.
Summary of Recommendations
Refer to numbered paragraphs in the text. Give particular emphasis to recommendations, where these are described in detail.
Terms and Abbreviations
Inevitably, the report will include terms, abbreviations and jargon unfamiliar to some readers. List these with a brief explanation.
Introduction The purpose of an introduction is to describe the reasons for carrying out the EoT assignment and what it is intended to achieve. It may be appropriate to outline how information was gathered and what limitations, if any, were imposed.
Foreword For some reports a foreword, written by a senior person can add authority and support to findings and recommendations. However, there is a risk that a foreword includes irrelevancies that may distract readers’ attention.
Acknowledgements
Record the names and designation of the people involved. Particular reference should be given to stakeholders, followed by others who have helped.
Terms of Reference
These should be stated, as agreed with stakeholders, whose names and designations should be included. Note any changes made during the assignment.
Main Text This is the most important part of the report. It must be organised so that it is accessible, well presented, and easy to read. Use main headings and subheadings to highlight significant features or chapters of the text. Consider linking these to specific performance issues, functions, working groups or stakeholders. Remember that people may only read those parts of the report directly relevant to them.
It is important to note that the main text is presenting an explanation of the process you’ve used, along with your findings, conclusions and recommendations. The emphasis should be on the description of significant issues to which you wish to draw attention, rather than the tools you’ve used. Any reference you make to EoT tools should be brief or, when considered necessary, details included as an appendix.
Executive Summary
Busy decision makers usually want to grasp the point of a report without having to read the whole thing. They are likely to become impatient if a summary is not provided and may not bother reading the report at all. The summary should consist of one page and done after the main section has been compiled, not before.
Appendices Also known as attachments, annexes or exhibits, appendices allow you to cater for a variety of readers. You should aim to keep the main text as concise as possible, with details presented separately where they will not interfere with the main flow of the report. Appendices can include completed EoT tools, data and information obtained from interviews.
Appendices can be referred to as ‘see Appendix A’, or ‘details are shown in Appendix B’. Normally, appendices are designated by letter rather than numbers.
Numbering As EoT reports are likely to accumulate within an organisation, where cross-referencing is an advantage, a standard number notation should be used. The most common system is paragraph numbering. Each paragraph has a sequential number, starting with 1 and carrying on through the text. The advantage of this system is that any paragraph can easily be identified and located, and thus discussed at meetings, or by telephone, or emailed.
Numbering is best done after the main text is written and a summary of recommendations drafted.
Eot Tools Course Review Questionnaire
Course Review Questionnaire Directions: At the start of the course complete column A by placing an ‘X’ at the point which most reflects your opinion On completion of the course complete column B by placing an ‘X’ at the point that now reflects your opinion A. How effective are you at performing each skill listed
LIST OF SKILLS
Improvement Teams
PURPOSE OF THE TOOL
EoT often identifies that performance remains unsatisfactory and needs further improvement. Several reasons could be responsible, organisational inertia, lack of management commitment, lack of expertise in identifying stakeholders, and barriers to change and ability to project manage.
However, evidence has been gathered about the importance of forming effective improvement teams. This requires using an improvement process that is rigorous and user-friendly.
DESCRIPTION
Improvement is the process of making organisational performance better than it is even if it is good already.
Successful learning organisations never say “that’s good enough” or ‘that’ll do”. They follow the aims of continuous or never ending improvement by saying “how could we make our performance even better?"
Improvement in terms of the training and development function could apply to any activity from improving feedback from learners to improving the organisation’s performance in the market place.
It has been proved that however it is more effective to tackle one problem at a time than to try and improve everything at once.
Most improvement is made little by little, as each problem is identified, qualified and then tackled. Many problems do not require huge investments of money. What they do demand is systematic analysis by people who know the jobs well and can use some straightforward improvement techniques.
Improvement teams are an effective way to tackle the improvement process. They usually involve people with good practical experience from different functions - a different perspective can often do wonders to help unravel the complex causes of things that go wrong.
Senior officers must help to identify problems to be tackled provide training and support to the teams and listen attentively to their recommendations when solutions have been worked out.
Improvement Teams
HOW TO USE
Firstly the improvement process / flow chart can be illustrated generically:
1.COLLECT FACTS ON PRESENT PERFORMANCE
2. SET A CLEAR THAT YOU WANT TO ACHIEVE
3. SET OUT IMPROVEMENT TARGETS
4. PUT TOGETHER AN IMPROVEMENT PLAN
One example of how this could be used is the improvement needed to test skill acquisition, which is not
happening very well on a training programme.
1. Collect facts on present performance
On investigation through interviews and data collection it is evident that on the present courses lecturers are used to giving knowledge but no skill development is formally undertaken through role plays / case studies. This obviously results in no objective measures of skill development being possible.
2. Set a clear standard that you want to achieve
This could be a pilot on one course to introduce:
Briefed role play.
Use of process checklist to measure skills.
Both formative and summative feedback of results within the course.
Development of trainers with requirement skill to administer.
3. Set out improvement targets
Briefed role play - three page document within three weeks.
Design process checklist - in-house pro forma within 1 month.
Redesign course to allow time for role play and feedback - 2 months.
Three trainers to be trained to use checklist - 2 months.
4. Put together an improvement plan
External consultancies to be engaged with the task to develop with master trainer a three page briefed task. To be agreed with centre manager.
Research an effective process checklist design with TVU, pilot with at least two role players to ensure validity and reliability.
Committee to form of three centre managers, master trainer, original designer of course to redesign objectives, timings to allow feedback on role plays. Internal training of trainers to practice feedback using checklist.
IMPROVEMENT TEAMS
Whilst individuals can implement improvement processes it is undoubtedly best if a team approach is used. Some of the features of improvement teams are listed below:
• Teams can be from 3-7 people hopefully with a range of technical and interpersonal skills together with experience of training issues.
• The team leader is always a manager but could be someone who has maturity, experience and respect from colleagues and management.
• One member should be someone who has decision-making abilities at a strategic level, and an interest in the team.
• Team members need to have training in basic improvement techniques and working in a group.
• The team ideally works on one problem at a time.
• The team leader keeps the organisations senior officers informed of progress and any difficulties.
• The team analyses the problem, collects additional data, considers alternative solutions and finally puts forward recommendations to management.
The team must be allowed adequate time to work on the problem and a realistic time for completing it. Presenting a case to management is a critical phase of the teamwork. Recommendations must be carefully justified preferably on cost grounds.
Teams may be set up to tackle just one evaluation problem or work on a series of problems.
The team’s success must be publicised so that others in the organisation recognise their contribution to improvement and co-operate as much as possible.
CONSTRUCTION OF CHECKLISTS AND RATING SCALES TO MEASURE SKILLS
INTRODUCTION
Checklists and rating scales are useful and usable instruments to measure learners Process or Product Skills, in many cases unless automatic testing and recording devices are to be used they are the only way.
However, there are two major difficulties to overcome before they can be used with any degree of confidence:
• They must be constructed with considerable care or a great deal of error in the instrument will occur.
• The biases and preferences of the observers or judges must be reduced - a good instrument will go some way to remedy this.
These notes are concerned with:
• Constructing checklists;
• Constructing the major types of rating scales;
• Checking the constructed checklists and rating scales.
Constructing checklists
There are six steps to follow. If they are followed, a usable, valid and reliable checklist should result.
Step 1: Specify an appropriate process or product.
Step 2: List the important behaviours or characteristics or criteria.
Step 3: Add any common errors.
Step 4: Arrange the list of characteristics or criteria.
Step 5: Provide a way to use the checklist.
Step 6: Pilot it.
Step 1: Specify an appropriate process or product
Checklists are not too difficult to construct and to use, but that should not be the sole reason for using them as internal validation instruments. It is naturally important to make certain that the information needed about the process and/or product that is being trained for and validated is included on the checklist. Hopefully, the analysis of the task and/or job will have provided this; if it has not, it is necessary to look again.
Process checklists provide information of the ‘It’s there - it’s not there’ type.
The following question should be asked about the process:
‘Are there some performance characteristics of this kind of process which are so important that it is valuable simply to know whether or not they occur?’
If the answer is yes, then a checklist is an appropriate measuring instrument to use.
A similar question should be asked about any product for which one wishes to construct an internal validation instrument to check learners and their learning.
‘Are there some characteristics typical of this kind of product which are so important that it is valuable to find out about whether or not they occur?’
Again, if the answer is yes, then a checklist is appropriate.
Below are a few examples of processes which can be internally validated. For each process some of the characteristics or criteria that might and could be included in a checklist have been listed:
Processes
• Playing a clarinet:
Holds instrument properly
Has a satisfactory arm position
Uses correct alternate fingering where necessary
Checks the key signature and time
Counts correctly.
• Using an electric sewing machine:
Plugs in the machine
Threads the machine
Checks needle
Threads needle
Adjusts stitch length.
• Doing a laboratory experiment:
Writes down what he is trying to find out
Gets equipment needed
Checks over procedure
Gets pad to write down results.
Step 2: List the important characteristics or criteria
Examine the task closer and extend the list of criteria. For example, ‘Using an electric sewing machine’ might look like this:
Plug in the machine
Thread the machine
Remove the bobbin
Wind the bobbin
Insert the bobbin
Remove the needle
Thread the needle
Insert the needle
Adjust the tension
Oil the machine
Adjust the stitch length
Use the foot control
Properly position the material to be sewn
Start the machine (insert needle into fabric first)
Stop the machine correctly (needle out of the fabric).
Step 3: Add any common errors
It is important to find out not only if desirable procedures are being followed, but also if undesirable ones are. For example, it may be important in observing someone learning to give a speech in public to find out whether the learner uses gestures - they may be desirable or undesirable ones!
It may also be important to find out if he uses any distracting words or mannerisms - for example, saying ‘Ah’, ‘Er’, or ‘Um’, or keeping his hands in his pockets. It is important not to add too many undesirable traits - add only those that are serious and very common.
Step 4: Arrange the list of characteristics or criteria
A process checklist will be much easier to use if the list of criteria to be observed is listed in the order in which the criteria are likely to occur.
In a product checklist the criteria should be arranged so that the judge can begin at one place and examine it systematically, part by part.
Step 5: Provide a way to use a checklist
There must be a convenient place to check each criteria as it occurs (or as it is seen if it is a product). The best way is to place blanks on the right hand side of the page next to each criterion being checked. A tick () or cross (x) can be put as appropriate – if sequence is important, a space should be left to number it.
The electric sewing machine could now look like this:
Criteria Yes No Sequence
Plugs in machine
Threads machine
Removes bobbin
Winds bobbin
Replaces bobbin
Removes needle
Threads needle
Replaces needle
Adjusts tension
Adjusts stitch length
Uses foot control
Positions material to be sewn properly
Starts machine (inserting needle into fabric first)
Stops machine (needle out of fabric)
Step 6: Pilot it
Try it before use – see section later, checking for checklists and rating scales
Other examples of checklists are added as Appendices:
Appendix 1: Saddling a horse
Appendix 2: Heavy Goods Driving –
Performance Monitoring Test – Side 1
Appendix 3: Heavy Goods Driving –
Performance Monitoring Test – Side 2
Appendix 4: Heavy Goods Driving –
Performance Monitoring Test –
Record card for 4 tests to be taken
during a 10 day course – Side 1
Appendix 5: Heavy Goods Driving –
Performance Monitoring Test –
Record card for 4 tests to be taken
during a 10 day course – Side 2
Constructing rating scales
In some cases rating scales have a definite advantage over checklists because they allow the ‘judge’ to make a systematic judgement about the degree to which a characteristic or criterion is present or observed.
A rating scale usually consists of a set of characteristics or criteria to be judged with some kind of scale. The judge uses the scale to indicate the quality, quantity, or level of performance observed. The points along each scale represents different degrees of the characteristic or criterion being observed. A set of directions tells the observer how the scale should be used.
Example: Rating scale for rating discussion leaders during training
Directions
Rate the discussion leader on each of the following characteristics by placing a tick () anywhere along the horizontal line under each item that represents your view of what you have observed.
1. To what extent did the leader encourage discussion?
Discouraged discussion by negative comments
Neither discouraged nor encouraged discussion
Encouraged discussion by negative comments
2. How well did the leader keep the discussion on the right track?
Let the discussion wander Only occasionally brought the discussion back on target
Did not let members wander from the point
3. How well did the leader ask controversial questions?
Never asked controversial questions
Occasionally asked controversial questions
Continually asked controversial questions
4. How well did the leader respond to inappropriate comments?
Ridiculed the person who made the comment
Treated inappropriate comments the same as appropriate ones
Discouraged inappropriate comments
In the excerpts from the rating scale above, the observer is helped by it to focus on specific observable aspects of the traits being observed and judged. In addition, if it is used on many people learning to lead discussions, they will all be judged from a common frame of reference.
This provides objectivity and improves the reliability of observation if the scale is well constructed and used.
Steps to take when constructing a rating scale
Step 1: Specify an appropriate training or enabling objective
Step 2: List the important characteristics
Step 3: Define a scale for each characteristic
Step 4: Arrange the scales
Step 5: Write the instructions
Step 6: Pilot it
Step 1: Specify an appropriate training or enabling objective
There are many training and enabling objectives which can be judged with the use of a rating scale. Here are a few from the field of education - add your own in the spaces provided.
• Processes:
Singing
Playing an instrument
Leading a discussion
Drawing a picture
Using a band saw
Solving a problem
Handwriting
• Products:
An oil painting
A woodworking project
A handwriting example
A blue print
A typed letter
A jar of jam
• Social-personal traits:
Friendliness
Politeness
Honesty
Patience
Generosity
Extroversion
Endurance
Step 2: List the important characteristics
This step is essentially the same as Step 2 for constructing checklists. Carefully analyse each objective and decide what is most important to the accomplishment of that objective.
Deciding what is important to the accomplishment of a given objective
• If the objective is a process, ask ‘What important behaviours are exhibited by individuals who can successfully perform this process?’
• If the objective is a product, ask ‘What important features are characteristic of high quality products of this type?’
• If the objective is a result of a social-personal development, ask ‘What important behaviours are most commonly associated with this social-personal trait?’
Below are specific examples of these general questions accompanied by some possible answers.
In most cases these characteristics will have to be obtained by an analysis of ‘good’ and ‘bad’ performers.
Note that each answer lists only those characteristics which are considered to be most important to the objective being rated.
‘What behaviours are exhibited by individuals who can successfully play tennis?’
They can:
• Hit a forehand drive
• Hit a forehand lob
• Hit a backhand drive
• Hit a backhand lob
• Anticipate the opponent’s next move
• Place a ball accurately, and so forth.
‘What important features are characteristic of good charcoal sketches of nature?’
They are:
• Well composed
• Contain bold as well as thin lines
• Free, not stiff
• Shaded so as to create a mood, and so forth
‘What important behaviours are usually associated with friendliness?’
They are:
• A ready smile
• A readiness to speak
• An interest in others
• A willingness to meet others
• A memory for names, and so forth
Step 3: Define a scale for each characteristic
A scale defines an underlying continuum. A ruler is a scale used for judging the physical characteristics of length. The continuum defining length (which theoretically ranges from zero to infinity) is marked off in some meaningful units (inches, centimetres, etc., in the case of a ruler).
If one things of rating scales as rulers designed to measure the quantity or quality of certain characteristics, a number of features emerge.
Whereas rulers are used to measure length, rating scales are used to measure such characteristics as:
• Participation
• Friendliness
• Smoothness
• Appropriateness
• Frequency
• Value
• Balance
• Beauty
• Flexibility
• and so forth
Like rulers, rating scales should also be marked off into meaningful units. However, the units on a ruler are all of equal value, and the distances between the points are equal. This is not true of the points along a rating scale. For example, imagine a rating scale had been constructed for rating ‘Ability to stay with a task’. The scale might look like this:
Ability to stay with a task
1 2 3 4 5
Gives up
at the first sign
of trouble
Gives up with a
minimal amount
of struggle
Keeps trying
even though
experiencing
difficulties
Stays with the
task after most
would have quit
Never quits until told to do so by trainer
All five points are drawn equidistant from each other, but in reality the significance of the difference between a rating of 1 and a rating of 2 is not known. Nor is it known how that compares with the difference between a rating of 3 and a rating of 4. Is someone who ‘stays with a task after most would have quit’ (a 4 on the scale) twice as good as someone who ‘gives up with a minimal amount of struggle’ (a 2 on the scale)? Obviously it is not possible to answer the questions.
Maybe the scale would represent reality better if it were drawn as follows:
1 2 3 4 5
Or like this:
1 2 3 4 5
Or like this:
1 2 3 4 5
Unfortunately it is not known which drawing best represents reality. Consequently, the scales might as well be drawn as if all the points along them represented equal intervals. This is not to say that they should be interpreted in that way. Nor does it mean no concern should be given about this problem when constructing the scales. The problem is obviously a major source of error. Therefore, operating from the premise that it is better to prevent error from occurring than to try to cope with it later, scales should be defined as carefully as possible. The following suggestions may help.
Establish the extremes of the continuum When attempting to define a scale, first describe the extreme ends of the continuum. If the characteristic that is being measured is most naturally rated in terms of frequency, then the task is easy. The scales could range from:
• Zero to infinity
• Never to always
• Rarely to very frequently
• 0 to 100 per cent
• 0 to 15, and so forth.
Example of a quantitative rating scale
How often does the assistant arrive late for work?
Never About half the time Always
However, when the characteristics being measured is to be rated in terms of:
• Degree to which it exists
• Quality of performance
• Level of development
• Quality of product or similar qualitative terms, then the task of defining the extreme ends is far more difficult.
Some sample scales are given below:
Examples of qualitative rating scales
How well does the discussion leader use gestures?
Very poorly –
Inappropriate and not
well timed
Average –
Appropriate but not
always well timed
Very well –
Appropriate and
well timed
What is the quality of the discussion leader’s voice?
Weak, squeaky, does
not carry well
Moderately strong, not full,
carries fairly well
Strong, full, carries well
Indicate the extent to which this drawing conveys a feeling of depth?
Appears to be flat, almost no perspective
Appears to have some depth, but lacks proper perspective
Looks quite realistic,
like a good photograph; good perspective
Looks real, almost like a 3D photograph, excellent perspective
In the above examples, the descriptions at either extreme describe the worst and the best a process or product could possibly be. Whenever defining the extreme ends of a continuum designed to be used in rating a process, analyse good and bad performers and describe the process that exemplifies the negative or low end of the scale. Next describe the process that exemplifies the positive or high end of the scale.
Example of a rating scale for a personality trait
Honesty
Always lies.
Stretches every story out
of proportion.
Exaggerates to his own advantage
Never lies, does not even
‘hide’ the truth or twist it
slightly. So honest it hurts!
(Note: If you can think of anyone who fits either end, keep it to yourself!)
This same procedure can be used when defining the ends of a continuum designed to rate products. Analyse bad and good products – ‘rejects’ and ‘ideal’. Describe each one.
Example of a scale to rate products
A typed form
Poor and confused layout;
Lines not in proportion;
Typeface illegible;
Generally sloppy.
Neat and clear layout;
Lines in proportion and straight;
Legible typeface
A pleasure to look at
Describe the point between the extremes
It is important to try to describe each point along the scale so that the difference in quantity and quality between any two points is approximately equal. Although it is not always possible to obtain equal intervals along with scale, it is possible to place the points along the scale in the correct rank order. It is important, of course, for example, to make certain that the behaviour descriptive of the fifth point of the scale does represent a higher quality performance than the behaviour described at the third point.
Examine the extremes and then try to describe two or three points representing behaviours (or features) which are not quite so extreme. Usually it is considered better to divide a scale into an odd number of points (3, 5 or 7). This is especially the case when there is a neutral point or a clear-cut mid-point between the two extremes. For this kind of scale, the first point to describe after the extreme ends is the mid-point.
Examples of behavioural rating scales
Commitment to a cause
Definitely against the cause Neutral Definitely for the cause
To what extent does this officer talk with other officers?
Always talks with others
Talks with others about
half the time
Never talks with others;
Always keeps himself to himself
There are now three points clearly defined:
• Two ends, and a
• Mid-point
This may be, and often is, enough. Sometimes, however, it is helpful to give the judge(s) a little more guidance by describing a few more points.
The procedure then, is to decide two more points on the scale, each of them as a mid-point on a ‘new and separate scale’ on the scale already created. This is done by looking at the high and low ends of the scale in turn, regarding one end as an end, but the mid-point as the other end – then establishing a new mid-point. The ‘talking’ scale above would thus look something like this:
Example of a behavioural rating scale
To what extent does this officer talk with other officers?
Always talks
with others
Talks freely with others – sometimes to the exclusion of other things, but
occasionally avoids the opportunity to talk
Talks with others about half of the time
Occasionally
talks with others,
but would rather
observe, keep to
himself or listen
to others, than
talk himself
Never talks with
others, always
keeps himself tohimself
There are several ways of describing the points on a rating scale. The three most common types of description are:
• Numerical
• Graphic, and
• Descriptive
Sometimes combinations of these are used, for example:
• Descriptive-Graphic
A numerical scale is simply a list of numbers keyed to descriptive labels which remain constant from one characteristic to the next. The judges mark in some way the number which best describes the person, process, or product, being validated (checked).
Example of numerical rating (with different marking methods)
Circle (O) the number which best describes the officers relationship with his subordinates. Use the following key:
5 = Excellent
4 = Above average
3 = Average
2 = Below average
1 = Poor
Tick () the number which best describes the officer’s relationship with his sub-ordinates. Use the following key:
5 = Excellent
4 = Above average
3 = Average
2 = Below average
1 = Poor
Tick () the box beside the number which best describes the officer’s relationship with his subordinates. Use the following key:
5 = Excellent
4 = Above average
3 = Average
2 = Below average
1 = Poor
Underline _________ the number which best describes the officer’s relationship with his subordinates. Use the following key:
5 = Excellent
4 = Above average
3 = Average
2 = Below average
1 = Poor
Numerical scales are also used when frequency of occurrence is to be judged.
Examples of numerical rating for frequency of occurrence
How many times does the speaker say ‘ah’ during a three-minute speech?
1 = less than 5
2 = 5 – 10
3 = 10 – 15
4 = 15 – 20
5 = more than 20
(The ‘barred gate’ system (//// ///) may be used to record the ‘ahs’ as well, before scoring the above). When describing points along a scale graphically, draw a horizontal (or sometimes vertical) line and divide it into equal parts. Each dividing line is accompanied by a label describing that point on the continuum. Observer(s) place their preferred mark (a or a X) anywhere on the line.
This allows marking between two points, closest to the one they feel best describes the individual.
Examples of graphic rating scales
How good is this officer’s relationship with his subordinates?
Extremely poor Poor Fair Good Excellent
How often does the speaker say ‘ah’ during a three-minute speech?
Never Rarely Occasionally Often Very frequently
A descriptive scale implies a more complete verbal description of the points along the continuum than the verbal labels used above. It is usual to construct and use a graphic arrangement, producing a description-graphic scale.
Whenever constructing a verbal description of points along a scale, be certain to be terse, crisp, and to use action, observable terms as if writing training or enabling objectives.
Example of a descriptive rating scale
How good is this assistant’s relationship with his fellow assistants?
Extremely poor; fights, argues, blames others, always wants his own way Fair; has some friends, often tries to get his own way; has some fights
Excellent; has many friends, considers others first, helpful, almost never in an argument
The advantage of a numerical over a graphic scale is that it yields numbers (which are easier to record and which can be manipulated). The graphic scale, on the other hand, has an advantage because the traits being rated are more explicitly stated in observable terms.
Step 4: Arrange the scales
For any given process, product, or social-personal trait, several scales will be constructed and used (one for each of the important behaviours, characteristics or criteria). If there is any logical arrangement for the scales it is preferable to place them in that order – for example, in the order of expected occurrence.
The ‘direction’ of the scale should be considered – it can go in one of three directions:
Poor Fair Excellent
Or, it can go from positive to negative:
Excellent Fair Poor
Or, it can go from strong (not necessarily positive) to neutral and back to strong:
Strongly disagree Neither agree nor disagree Strongly agree
Notice that to disagree is not necessarily negative. It depends with what you are disagreeing!
When judges are marking a number of scales in a row, it is easier if they are always arranged in the same direction, for example, negative to positive. However, with this arrangement it is easy to establish a response pattern – for example, ‘This officer is pretty good, I’ll mark all 4s’ – and just mark all down one side of the page, marking each scale about the same.
If this is thought as likely to be a problem, it can be avoided by occasionally changing the direction of some of the scales – for example, every so often putting in a positive to negative arrangement. The expectation is that observers will have to read and consider each scale before establishing their decision.
Step 5: Write the instructions
Whether or not the creator of the scale is going to use it exclusively himself it is vital to write instructions that clearly specify the judges’ task. This will help to reduce any error which might result from a misunderstanding of how the scale is to be used.
The instructions to the judges need not be elaborate. However, they must contain the following information:
The instructions accompanying any rating scale should contain the following information:
1. A statement naming or describing the overall process, product, or trait, being rated.
2. Directions on how to mark the scales.
3. Any special instructions, for example, add any comments at the end or omit any characteristics you do not feel qualified to judge.
Examples of instructions for rating scales
Rate the learner’s ability to present a convincing argument. Place a tick () anywhere on the horizontal line below each of the characteristics. In each case place the () at the place which you think best describes this learner’s ability.
For each feature below, circle (O) the number which best describes this apple. Make any additional comments about the apple at the bottom of this page.
The above example instructions can be adapted to fit particular situations and are the most popular. They can help produce analytical ratings – for example, for a given
• Process
• Product
• Trait
that is analysed and the most important characteristics are identified and carefully described.
Instead of trying to devise a series of descriptive scales which would define a specific attribute of a product it is better to provide actual examples of products. These sample products can be used as examples illustrating each of the points along some scale. Ratings would be made by observers by comparing the product being rated to the various sample products at the various points along the scale.
Example
Although the following is taken from a non-industrial/commercial environment, the principles hold good.
A group of art teachers were trying to rate products produced by a group of their learners. The products were creative wax sculptures, and they found it difficult to define the important characteristics. (Creativity would be one, but how would you describe the points along a scale of creativity except as ‘not creative at all’ to ‘very creative’?) Although they could not always verbalise all their reasons, the three art teachers were in almost complete agreement as to which sculptures were the best, which the worst, and which ‘about average’. Knowing that they would be having the next group of learners do the same assignment, they decided to develop a product scale using the learners’ own work as models.
The first thing these teachers did was to select the three best and the three worst sculptures.
These would be used to anchor the ends of the scale. There is nothing magical about selecting three examples at each end; often one is enough. However, they felt that by selecting three very different but equally good creative pieces as ‘best’ examples, they could demonstrate the fact that any particular form was not necessarily best. They had thus decided to use the product scale they were developing as concrete ‘objectives’ – a demonstration to future learners. By looking at the products on the scale, learners could get a fairly good idea of what would not meet the teachers’ expectations.
They identified then, three ‘about average’ sculptures as examples of the mid-point on the scale.
Next, they identified some which seemed to fit about halfway between the ‘average’ and the ‘best’ and between the ‘average’ and the ‘worst’. Now they had a five-point scale. Each point on the scale was represented by three actual, learner-produced sculptures.
The final touches were placed on the scale by writing brief descriptions of the examples at each point. In the descriptions, they tried to point out the ‘good’ and ‘bad’ features. To use this scale in the future, the teachers would simply take a piece of sculpture and compare it to the examples on the scale. If it looked most like those at position 2, it would receive a score of 2, if it looked most like those at position 5, a score of 5, and so on.
‘Process’ skills scales although more difficult to construct than ‘product’ ones, can be constructed in the same way as the above example.
Step 6: Pilot it – checking checklists and rating scales
Checking for obvious flaws
The validity of a checklist or rating scale is determined to a great extent by which behaviours, criteria, or characteristics, were selected to be checked or rated. It is vital to be certain that the most important ones are included in the instrument. An easy first step is to obtain the views of a colleague. He needs to know in general terms what the checklist or scale is trying to measure.
He needs to be asked to produce his list of important criteria and compare them with the created one. Differences need to be discussed.
Then ask him to read over the descriptions created to describe the particular behaviours listed on the checklist or placed on the scale. Ask him to ‘translate’ the descriptions into his own words.
If some of the created descriptions are ambiguous, rewrite the descriptions.
Finally, have the instructions checked and rewrite the unclear ones.
Trying out the instrument
Pilot the instrument and, where possible, have other people use it too. Notes should be made on any problems. For example:
• Were there times you wanted to put something down but had no place to put it?
• Were there characteristics you did not get an opportunity to observe?
• Were there behaviours as described typical of those you observed?
• Did the performance move too fast for the number of ratings you had to make?
These and many similar questions often arise as a checklist or rating scale is used. They signal problems with the instrument that could probably be corrected with a little rewriting.
Whenever piloting an instrument, a note pad should be kept handy so that any questions which may arise or any problems which may occur can be jotted down. Sometimes there will not be the time to make notes while you are using an instrument of observation. It may not be possible to jot down your impressions until after using the instrument. Therefore, it is advisable to plan to have extra time immediately after piloting the instrument of observation. Use this time to write down impressions while they are still fresh in the mind.
Validity
There are a number of ways to check the validity of an information-gathering instrument. The easiest way for a trainer-made test is to simply verify the information which the instruments yield with ‘work done for real’, ie in the work situation, with manager, peer, and if appropriate, quality control comment. If they confirm the information gained, there is validity; if it is not confirmed, it is invalid. But also check the instrument against some other form of source of information.
Reliability
It is highly unlikely that valid results will be obtained if the instrument is unreliable. However, there is a major source of error in checklists and rating scales:
• Judge’s error
The procedure to establish this is relatively simple:
Two or more judges obtain information about the same group of individuals, processes, or products, at the same time.
The information obtained by one judge is compared with the information obtained by another judge.
A large amount of agreement means high inter-judge reliability, a small amount of agreement means low reliability.
When correlation is used to make the comparison, the resulting coefficient gives a numerical estimate of the degree of consistency between judges.
Statistical techniques suitable for use are described in other notes.
Summary
• Checklists and rating scales are useful instruments for observing learners’ processes and products.
• There are a number of steps to take when constructing a checklist:
— Specify an appropriate process of product;
— List the important characteristics of criteria;
— Add any common errors;
— Arrange the list of characteristics or criteria;
— Provide a way to use the checklist;
— Pilot it.
• Checklists provide one with:
— ‘It’s there – it’s not there’ kind of information.
• Rating scales allow judges to make systematic judgements about the degree to which a process, product, or characteristic, is present.
• There are a number of steps to take when constructing a rating scale:
— Specify an appropriate training or enabling objective;
— List the important characteristics;
— Define a scale of each characteristic;
— Arrange the scales;
— Write the instructions;
— Pilot it.
• Product scales (and sometimes) process scales, can be constructed by using actual examples (or pictures of them) of appropriate or sample products or processes.
• The validity, reliability of each of use of checklists and rating scales should be checked before they are used as well as after they have been used. Any faults should be corrected before they are used again.
• Judges should be carefully selected and trained.
Appendix 1
Saddling a horse
Criteria Yes No
Is the saddle properly in place on the horse?
Is the saddle properly positioned on the saddle pad?
Are the stirrups at the proper height for the leg length of the rider?
Are the stirrups firmly secured to the saddle?
Is the girth strap tightened so that the saddle will not slide? and so forth
Appendix 2: Performance Monitoring Test – Marking Sheet
Name Authority Date
Course day/duration Test: 1 2 3 4
Finish
Start
Time
Start
Finish
Traffic
conditions
Weather
conditions
Total miles/ kilometres
Notes:
1 2 3 4 5
1 Fails to ensure H/B engaged1 S gear in neutral2 clutch down3
2 Moving off Procedure1 U;Hill2 D/Hill3 Blind spot4Stall5
3 Gear selection Appropriate1 two speed axle2
4 Gear production Up1 Down2 Neutral3 Exercise4
5 Clutch Control1 Riding2 Coasting3 Exercise4
6 Footbrake Late1 Fierce2 Jerky3
7 Handbrake Release1 Stopping2 Necessary3 Unnecessary4
8 Accelerator Blipping1 Jerky2 Excessive3 Insufficient4
9 Steering Hold1 Oversteer2 Understeer3
10 Overtaking Early1 Late2 Too close3
Dangerous4
11 Progress Road conditions1 Traffic conditions2
12 Roundabouts Assess1 MS2 Position3 Speed4 Observations5
13 Road junctions Assess1 MS2 Position3 Speed4 Observations5
14 Cross roads Assess1 MS2 Position3 Speed4 Observations5
15 Lane discipline Crown1 Kerb2 Straddle3 Change4
16 Following ORU Separation distance1
17 Meeting ORU Incorrect1 Dangerous2
18 Crossing ORU Incorrect1 Dangerous2
19 Ped. crossings Approach1 Invite2
20 Normal stop Selection1 Position2
21 Before Starting1 Adjusting speed2 Stopping3
22 Before Changing direction1 Signals2 General use3
23 Overtaking Before1 During2 After3
24 Indicators Wrong1 Timing2 None3 Unecessary4 Cancel5
25 Arm Wrong1 Timing2 None3 Unnecessary4
26 Horn Necessary1 Not necessary2
27 Automatic T/Sig Approach speed1 Fails to comply2
28 Observation Traffic signs1 Road markings2 Hazards3
29 Traffic cont. Fails to comply1
30 Other road users Alertness and anticipation1 Response to ORUs
actions2 Response to ORUs signals3
Eot
Appendix 3
1 2 3
31
32
33
Forward steering
Reversing
Braking
Accuracy1 control2 Observation3
Accuracy1 control2 Observation3
Accuracy1 control2 Observation3
General Comments
Name Authority Dept. Course Instructor
Day/duration Test 1 Test 2 Test 3 Test 4 D of T Test Pass/Fail
Date
Time: Start/finish
Miles
Traffic
Weather
Comments
Performance monitoring test – Record card Test 1 Test 2 Test 3 Test 4
1 2 3 4 5 1 2 3 4 5 1 2 3 4 5 1 2 3 4 5
S 1 Fails to ensure H/B engaged 1 Gear in neutral2 Clutch down3
t 2 Moving off Procedure1 U/hill2 D/hill3 Blind spot4 Stall5
3 Gear selection Appropriate1 Two speed axle2
4 Gear procedure Up1 Down2 Neutral3 Exercise4
5 Clutch Control1 Riding2 Coasting3 Doubling4
6 Footbrake Late1 Fierce2 Jerky3
7 Handbrake Release1 Stopping2 Necessary3 Unnecessary4
8 Accelerator Blipping1 Jerky2 Excessive3 Insufficient4
l 9 Steering Hold1 Oversteer2 Understeer3
10 Overtaking Early1 Late2 Too close3 Dangerous4
11 Progress Road conditions1 Traffic conditions2
12 Roundabouts Assess1 MS2 Position3 Speed4 Observation5
13 Road junctions Assess1 MS2 Position3 Speed4 Observation5
14 Cross roads Assess1 MS2 Position3 Speed4 Observation5
15 Lane discipline Crown1 Kerb2 Straddle3 Change4
16 Following ORUs Separation distance1
17 Meeting ORUs Incorrect1 Dangerous2
18 Crossing ORUs Incorrect1 Dangerous2
19 Ped. crossing Approach1 Invite2
20 Normal stop Selection1 Position2
21 Before Starting1 Adjusting speed2 Stopping3
22 Before Changing direction1 Signals2 General use3
23 Overtaking Before1 During2 After3
24 Indicators Wrong1 Timing2 None3 Unnecessary4 Cancel5
25 Arm Wrong1 Timing2 None3 Unnecessary4
26 Horn Necessary1 Not necessary2
I 27 Auto. T/signals Approach speed1 Fails to comply2
g
n
s
28 Observation Traffic signs1 Road markings2 Hazards3
A 29 Traffic cont. Fails to comply1
T
T 30 Other road users AA1 Response to actions2 Response to signals3
S
PMT Record card
Highway code and signs Vehicle Safety Vehicle check Accessories Uncoupling and recoupling
1 Theory test % Comments Theory test % Comments
2 % %
3 % %
% %
Checks and tests
4
Forward steering Reversing Braking exercise
Test No. 1 2 3 4 Test No. 1 2 3 4 Test No. 1 2 3 4
Accuracy Accuracy Accuracy
Control Control Control
Observation Observation Observation
Manoeuvres
General comments
General comments
General comments
Test 1
Test 2
Test 3
Test 4
PMT Comments
ASSESSING MANAGEMENT COMPETENCE
People in management or senior technical, administrative or professional jobs frequently perform tasks that are regarded as ‘productive’. Other people, or the same person on other occasions, will perform a similar task differently and achieve an acceptable outcome. Assessing whether people performing such tasks are competent depends on having:
1. Clearly defined standards of performance, as in a performance objective.
2. Relevant criteria specifying evidence a person should provide to confirm their competence.
3. Methods for assessing the evidence.
Although here we are concerned with how to assess competence, it should be noted that this can only be done when performance standards and relevant criteria are available. Management competences may be assessed in a variety of ways, some of which are similar to methods used elsewhere in the assessment of practical and/or cognitive skills, and by employers when selecting or evaluating staff. The following is a list of methods suitable for assessing competence using both work based evidence and activities to generate evidence upon which to make a judgement.
Assessment Centre
Assessment centres are used to judge competence using simulation exercises and other off-the-job assessment techniques. They employ observer-assessors and are used for staff selection, promotion, or development. In one programme sampled in the survey, an assessment centre used for recruiting graduates was adapted for use in assessing competences. There were five simulation exercises designed to allow managers to demonstrate behaviours associated with five basic management competences. Each exercise was intended to demonstrate several behaviours and competences.
There were two group exercises, a presentation, an in-tray exercise, and an interview simulation.
Several assessors observed the managers’ behaviours from which they inferred competence.
Assignment
An assignment consists of any problem-solving exercise set with clear guidelines and specified length. While assignments are difficult to define precisely, they are more structured and less open-ended than projects. Assignments may deal with real work problems or with hypothetical questions. In the programmes surveyed assignments were normally presented in written form.
Case Study
Case studies, consisting of a description of an event concerning a real-life or simulated situation, usually as a text, a video, a picture or a role-play exercise. People are assessed as to their competence in analysing the situation presented, drawing conclusions and making decisions or suggesting courses of action. This method of assessment may be carried out with individuals or small groups.
Group Exercises
These are designed to simulate situations where managers meet and work together. There can be various formats:
- Competitive where individuals are in competition with each other for something, say a limited budget, and therefore have to utilise their negotiating and influencing skills to get what they want;
- Co-operative where individuals have to work together as a team to achieve a group goal;
- Leadership where one person in the group is nominated to be leader for a particular task.
Interview
The personal interview is probably the oldest and best-known means of eliciting information directly from managers. An interview can be two-way, providing managers with the opportunity to question the interviewer. It is most widely used to generate evidence of process-based skills.
Interview Simulation
These exercises can be designed to simulate interview situations in which the manager typically finds him/herself (e.g. staff counselling, disciplinary interview, selection). Assessors can observe how competent the manager is at dealing with staff in these situations.
In-tray Simulation
In-tray simulations are designed to accurately simulate the contents of a manager’s in-tray. Managers work through the contents, make decisions and take appropriate action. An in-tray exercise is particularly useful in gaining evidence on planning, organising and prioritising.
Log-book/Diary/Personal Journal
A self-report technique in which managers generate evidence of their performance, progress, experiences, attitudes and personal feelings in an on-going situation.
A log-book can, if properly structured, provide a useful means of assessing the progress of a manager’s achievements or performance. To be effective, it should have a clear specification and give guidance on how essential information is to be recorded.
Portfolio
The portfolio owes its derivation to the artist who presents a selection of his/her work to the prospective employer to demonstrate his/her ability. For managers a portfolio might include a business plan, a marketing plan, an investigative report, an extended CV, a video of a presentation or an interview conducted, etc. The purpose is to present evidence of the manager’s performance.
Presentation Exercise
These are used to provide evidence of oral communication, interpersonal, and analytical skills. They provide the opportunity for managers to put forward their ideas in a face-to- face situation and to demonstrate their ability to persuade the audience to their point of view.
Project
Projects have the following characteristics:
- Are more comprehensive and open-ended than assignments;
- May be tackled by individuals or by a group;
- Usually involve a significant part of the work being carried out without close supervision;
- Involve less direction from a third party.
All projects involve the following stages:
1. Planning the project.
2. Devising the investigation strategies.
3. Conducting the investigation.
4. Analysing the results.
5. Reporting on the conclusions.
Role-play
In role-play, managers are presented with a situation, often a problem or incident, which they then have to respond to, by assuming a particular role. The enactment may be totally unrehearsed or they may be briefed in a particular role to be played. Such assessment is more open ended and person centred than simulation and can be used to assess a wide range of the behavioural and interpersonal skills required in the world of work.
Self-report Techniques
A self-report technique is any form of assessment in which managers generate the necessary evidence of their performance, experiences, attitudes and personal feelings in an ongoing situation.
Simulation
A simulation is a structured practical exercise with specific objectives involving the organisation and completion of a particular task, which is product - or resource - based that seeks to simulate real-life conditions. To be effective, simulations must succeed in recreating the atmosphere, conditions, and pressures of the real situation.
Written Examinations
Written examinations are a traditional form of time-constrained, summative assessment. They may be closed or open book, seen or unseen, examinations.
Written Exercise
These are used to provide evidence of analytical ability and written communication skills, e.g. the ability to impart information and ideas accurately and precisely when communicating in writing; the ability to construct a clear, logical and persuasive argument; the ability to present issues and problems in a broad perspective.
Constructing Questionnaires
AN OVERVIEW
Introduction
Questionnaires are instruments that all of us are familiar with:
• We receive them in the post as part of our official mail
• We see them in newspapers and journals
• We are asked to complete them as part of a survey at work, and so forth.
As trainers, we may not just have to receive, see, and complete – we may also have to consider constructing, administering, and interpreting the responses. This may be because we wish to obtain information to help in an analysis of need or validate a training activity, or some other purpose. These notes have been written to help those who may wish to construct and use questionnaires to validate a training activity; the principles should hold good for any other purpose.
There are a number of activities that need careful thought when considering whether or not to construct and use a questionnaire. They can be subdivided into four groups.
1. Deciding to use a questionnaire
• Deciding upon general area(s) that more general information is required about
• Identifying a specific problem
• Checking that a questionnaire is suitable.
2. Constructing a questionnaire
• Choosing the format for questions
• Identifying the frame of reference for the target group
• Writing the questions
• Testing the questions
• Revising the questions
• Re-testing the questions if necessary
• Revising the questions if necessary
• Designing a summary sheet
• Organising the questionnaire
3. Administering the questionnaire
• Administering the questionnaire
4. Interpreting the results of the questionnaire
• Interpreting the results of the questionnaire.
These 'activities' are briefly described and discussed below.
Deciding upon general area(s) that more information is required about
This activity will often be considered when planning the total validation system for a particular training activity. The problem areas are usually identified by a discussion between the people involved with the activity (the training designers, the direct trainers, and so forth). They should reach a consensus view that:
• A particular aspect is worth looking at in more detail and
• The information collected will enable decisions to be made either to confirm that what was done was acceptable and/or to improve the training design or implementation.
Example
A designer of a programme has discussed with some of its direct trainers a recurring concern – the project work component.
Identifying a specific problem
Having decided upon the general area, specific questions need to be formulated.
The questions asked will influence the choice of method. Different types of questions lead to different ways of collecting information. A number of interviews will often help to identify the specific problems which can be restated as a series of questions.
Example
A trainer (validator) has been asked by two training teams (X and Y) to collect information to enable them to review the project work component of their programmes. The trainer – an outsider adviser in this case – decides to interview the programme direct trainers concerned before proceeding to collect information from the learners. He interviews four direct trainers from the two training teams and obtains the following responses:
Trainer A: Amongst other things, I am always worried about enthusiasm for the project excluding all other activities – this can lead to specialisation with little breadth. This enthusiasm also leads to repeated contact with me (either by telephone, telex, or memo) which, although welcome, does cause problems.
Trainer B: Containing the project is often a problem – resources and time are not always available for elaborate project work.
Example continued
Trainer C: The project follows some of the basic skills we have tried to develop on the programme, but not others – if they do not follow, how is the learner supposed to integrate his knowledge, skills and attitudes – another question I often ask myself is – do I give enough time to the help I should give to each learner?
Trainer D: I was talking to some line managers from some of the departments recently – they seemed to be critical of the project work carried out in their departments. Their main criticisms were – too time-consuming and not really relevant.
These responses were then taken by the trainer (validator) as a guide in constructing a questionnaire, by rephrasing them as the following set of questions (these are NOT the precise questions/items to be used in the questionnaire):
• How much time do the learners spend on:
- theoretical framework
- practical skills development
- project?
• Do they think this is the best balance?
• Is the coaching system giving the correct amount of support and guidance?
• Is the timing and duration of the project such that it proceeds as smoothly as possible?
• Are the resources (time and people) available adequate for the projects that the learners work on?
• Does the project allow the learner to integrate his skills, knowledge, and attitudes, with his work activities?
Checking that a questionnaire is suitable
The final decision to use a questionnaire will often depend on the particular context of the validation. It is difficult to give precise advice. However, the following guidelines may help.
A questionnaire may be appropriate if:
• You want to ask a definite set of questions and give the respondents time to consider their replies.
• There are more respondents than you have facilities or time to interview.
• The respondents are widely geographically dispersed and you neither have the time nor the finance to visit them.
• The writing/reading skills of the respondents may possibly be inadequate.
• You have the time to construct a good instrument before you use it.
• You have the production facilities available (typing, reprographic, paper, etc.)
• You have considered the problem of omissions, ie questions answered.
• You have considered the problem of response rate, i.e. questionnaires not returned.
However, a decision has to be made in terms of what information is needed as well as practical constraints. Remember: matching – scientific soundness, administrative convenience, political acceptability.
Example
The trainer (validator) decided to use a questionnaire to collect information from the learners. The decision to use a questionnaire was based on:
• The number of learners attending programmes run by the two teams in the past nine months (X = 65, Y = 59).
• The timing of the project, ie at the end of the programme.
• Available access guide to learners.
This met the 'Is it necessary to use a questionnaire?' criterion.
1. Checklist
Example
Tick () as many boxes as necessary.
These are skilled areas in which I was already competent when I joined the programme:
Photography:
Camera work .. .. .. .. .. ..
Developing .. .. .. .. .. ..
Printing .. .. .. .. .. ..
Enlarging .. .. .. .. .. ..
2. Two-way questions
This type of format forces a choice, a better approach might probably be to add a
third choice – 'Do not know' (but this does have inherent difficulties), therefore
making sure that two-way questions give a realistic choice.
Example
Tick () one box only.
Will you choose another project based on computer-assisted learning?
Yes .. .. .. .. .. .. ..
No .. .. .. .. .. .. ..
Do not know .. .. .. .. .. ..
Although in itself a questionnaire is a valid method for collecting the information, if resources were available it would probably be cross-checked by combination with another method. The questions identified in the previous activity – Identifying a specific problem – could then be negotiated with the trainers concerned to assess suitability.
Choosing the format for questions
Two types of question are constructed for questionnaires
• Open response questions and
• Closed response questions.
The former tend to be used for more complex questions and the latter for simpler ones.
A. Open response questions
This type of question allows the person (respondent) answering the questions some freedom in the way in which they answer – but hopefully not too much!
Example
In your opinion, what new skills did you acquire through carrying out your project?
………………………………………………………………………………………………..
………………………………………………………………………………………………..
………………………………………………………………………………………………..
………………………………………………………………………………………………..
B. Closed response questions
In answering these questions the respondent has to choose from among a number of answers that have been provided. These questions can be written in a number of ways:
3. Multiple-choice questions
This type of format gives a wider range than two-way questions.
Example
Tick () one box only.
Did you enjoy the style of learning?
Not at all .. .. .. .. .. .. ..
A little .. .. .. .. .. .. ..
Fairly .. .. .. .. .. .. .. ..
Quite a lot .. .. .. .. .. .. ..
Very much .. .. .. .. .. .. ..
4. Rating scales
Some types of scale used formally in an attitude rating scale may be used less formally in small groups of questions. The question types are Likert and Semantic differential (Osgood). Both have different characteristics.
Likert scale
The Likert scale is a list of statements and the respondent makes a judgement on every item. The scales may vary from simple alternatives (eg agree/disagree) to a ten-point scale. Normally the scales used are three to six-point ones. Evennumbered scales force a choice (eradicates the 'central tendency'), odd-numbered scales allow a neutral position to be taken (allows the 'central tendency').
Examples
Ring (O) one of the numbers.
How relevant do you think this topic is to your everyday job?
very not at all relevant relevant relevant
5 4 3 2 1
Tick () one of the boxes.
How relevant do you think this topic is to your everyday job?
Very relevant .. .. .. .. .. .. ..
Quite relevant.. .. .. .. .. .. ..
Relevant .. .. .. .. .. .. ..
A little relevant .. .. .. .. .. ..
Not at all relevant .. .. .. .. .. ..
Eot Tools Constructing Questionnaires
Note: The scale numbers do not have to be disclosed; if they are not, it is important to name the unnamed ones, eg 4 = quite relevant, 2 = a little relevant.
Semantic differential scales
Semantic differential scales are thought to be useful when the respondents are likely to have strong opinions. It represents a general reaction to the person, object, or materials it is focusing on. The vocabulary must, of course, be appropriate for the respondents.
It is based on the use of paired adjectives (bi-polar opposites) written on either side of the page, with 5, 6 or 7 possible positions in between.
The pairs should be arranged so that the positive and negative adjectives are not all on the right or left. If the same pairs are used in a number of items they should be in the same order.
The score is obtained by giving 0 for the most negative response and 6 for the most positive (on a 7 position scale). 0 is used so that they can be displayed graphically with a 0 base on the graph.
The scores for each item are added to give a score for that item.
Example
Tick () one boxes on each scale.
The help and advice given to tackle the project was:
Informative Uninformative
Boring Interesting
Superficial Profound
Helpful Unhelpful
It is not usual to disclose numbers in a semantic differential scale, but if it is felt to be necessary to do so, the above example could look like this:
Example
Ring (O) one of the numbers on each scale.
The help and advice given to tackle the project was:
Informative 1 2 3 4 5 6 7 Informative
Boring 1 2 3 4 5 6 7 Interesting
Superficial 1 2 3 4 5 6 7 Profound
Helpful 1 2 3 4 5 6 7 Unhelpful
In the above example, the number 1 has been used to show graphically, this would need to be subtracted from responses.
5. Ranking scales
The respondent is asked to place a number of statements in order:
Example
The following aims have been suggested as relevant for project work after the programme you have attended. Rank them in order of importance to you as a learner. Put a 1 against the aim you consider most important, 2 against the next important, and so on. Rank all aims.
Aim Ranking
To increase your knowledge in one specific area to a high standard
To improve skill in doing critically relevant literature searches
To increase independence
To improve the ability to interpret data critically
To increase interest in studying your subject
To increase ability to present information and ideas lucidly
To increase ability to work under constraints (of time, space, and money)
To develop practical skills
To open up the ranking scale spaces can be left for respondents to enter other factors (in this example, aims) and rank them.
Example
After considering the format of the questionnaire the trainer (validator) in this case decided to use a mixture of open and closed questions because, although some of his questions are relatively simple, others need more complex answers.
Identifying the frame of reference for the target group
Before beginning to construct the questionnaire it is a good idea to draw up a frame of reference for the target group, ie list the characteristics of the respondents. The frame of reference can serve as a reminder to consider such things as:
• Vocabulary
• Knowledge
• Experience
• Bias as the questions are written.
Example
Two programmes are being considered so the frame of reference needed to check that only questions on common areas were asked or sub-sections arranged. The trainer (validator) drew up the following frame of reference.
Frame of reference:
• Post-experience programme
• Mature learners
• Three years minimum supervisory/managerial experience
• Degree or advanced diploma holders, but a mixture of academic / professional subject areas
• 50% England and Wales based 20% Scotland based 30% Overseas based
• 60% Studied for professional qualifications in last three years 40% Studied for professional qualifications over three years ago
Writing the questions
This activity is probably best carried out in parallel with the activity 'testing the question' described on page 18. The writing of questions needs considerable thought and attention. Take time to make sure that the questions are:
• Relevant
• Credible
• Relate to the identified specified problems.
Remember: The type of question asked influences the summary sheet design and the time needed to analyse and summarise.
When writing the questions use a separate piece of paper or a filing card for each.
This will help when organising the order.
Example
The numbers in circles eg 2 relate to the example listed in 'Identifying a specific problem' on page 2. The following questions (for the questionnaire) were written by the trainer (validator) to enable him to collect information to answer those specific questions.
A. Please estimate the number of days you spent over the year on these aspects of your programme. NB 8 hours = 1 day.
Theoretical studies
Practical skills
Project work
B. What is the percentage value of the total assessment mark given to the project work?
%
C. Please tick () one of the following boxes:
I feel that on the project work I spent:
Too much time
Too little time
About the correct
amount of time
1
1
1
Example continued
D. Please tick () one of the boxes and comment as appropriate.
Who chose the project topic?
You
Your manager
Both of you
Someone else Who?
Comment: …………………………………………………….
…………………………………………………….
E. Please insert the number(s) in sections (a) and (b) as
appropriate.
How many formal sessions (about your project) did you have with
your trainers while:
a) Planning it
Doing it
Writing it
b) On average, how long did each session last?
Minutes
c) Please tick () one box.
Do you think that this amount of contact was:
Too little
About right
Too much
F. Please tick () one box.
During the formal sessions you had with your trainer about your
project, was the guidance given:
Too general
About right
Too detailed
2
2
2
Example continued
G. Please describe any problems caused by (a) the timing (eg starting at a busy work time), and (b) the duration (eg six months):
(a) …………………………………………………………………………
…………………………………………………………………………
(b) …………………………………………………………………………
…………………………………………………………………………
H. Please list any difficulties eg resource shortages, bottlenecks, etc., you experienced when working on your project:
…………………………………………………………………………
…………………………………………………………………………
…………………………………………………………………………
…………………………………………………………………………
I. Please tick () one box.
Will the results of your project become a normal part of your occupational activities, e.g. managing?
(a) Yes
No
Not sure
(b) If you answered 'No' or 'Not sure', please explain why:
…………………………………………………………………
…………………………………………………………………
…………………………………………………………………
…………………………………………………………………
3
4
5
Eot Tools Constructing Questionnaires
Example continued
J. In your opinion, what is the response of your colleagues to your
project?
…………………………………………………………………………
…………………………………………………………………………
…………………………………………………………………………
Testing the questions
Each question needs to be tested against specific criteria. The following criteria can be useful:
• Does the question involve only one idea?
• Is the question worded as simply as possible?
• Is the question as brief as possible?
• Is the question as direct as possible?
• Does the question allow the respondent to admit lack of knowledge without loss of face?
• Are the words unambiguous when considering the target group?
• Do the words and phrases of the question influence the response?
• Is the question a negative question?
• Is the question loaded?
In order to see their usefulness, test the questions written in the previous section.
See whether or not they reach the criteria (some do not, or at least are doubtful).
Remember: The questions are attempting to obtain information about the specific problems formulated in 'Identifying a specific problem' (page 6).
In addition to testing the questions yourself, ask other people to test them using the same criteria. Ideally they should be tested by two groups:
• Colleagues and
• Target group.
It may well be that it is not possible to achieve the ideal of representatives of the target group to test the questions. It is then even more vital to obtain help from colleagues.
Revising the questions
Those questions that need revising should be revised, and if necessary retested and revised again until they are regarded as useable.
Designing a summary sheet
This activity can be carried out in parallel with the next activity - 'Organising the questionnaire'. The order of questions and position on the page can affect the ease with which the collected data can be processed and analysed.
In most small scale validation exercises the scoring of the data will be done manually. If larger scale ones are contemplated and access to a computer is possible, help should be sought at this time from computer specialists. As time for scoring is usually limited, careful preparation of a data summary sheet during the construction may save valuable time and frustration later.
Open response questions
Open response questions may need to be categorised and coded. Preliminary matrices may help, but space is needed for unexpected categories to be added as recognised. The information can be summarised in three main ways:
• A descriptive written summary with or without direct quotes.
• The responses can be divided into categories and direct quotes or alternatively, summaries can be recorded.
• The categories can be coded, i.e. given a numerical value.
These three ways are exclusive, combinations may be used. The three methods are briefly described below. Reports are used as the example of an open response instrument in all three cases.
A descriptive written summary
This method is suitable only if there are a few reports. The reports are read through and notes are made of what appear to be important points. From these notes a summary is made. If quotations are going to be used, potential quotes can be identified by underlining them as they are read through. The quotes used can be selected from the underlined statements.
A way of summarising a number of reports
Use file cards (or sheets of paper) as summary sheets. Alternatively an analysis matrix can be drawn - Appendix 1.
Select one of the reports, read it carefully, and each time a new opinion or event is described summarise it on a new card. In the right corner place a tally mark (1) to indicate that it has appeared once.
Read the rest of the reports making new cards as necessary.
When you come to an opinion or description of an event that seems to have occurred before, find the card that summarises it. Check to make sure that it means the same and then record an additional tally stroke (//) and so on (//// /) as more similar opinions or descriptions occur in other reports. If part of the statement is new, summarise that part on another card.
Prepare a summary, covering the most frequent statements, for your report.
If your categories at any time do not seem satisfactory, you need to begin again. It is difficult and leads to errors to adjust what you have already done.
Coding categories
Categories chosen should be relevant and useful. For example, categories can be chosen related to training method, audio-visual aids, programme content, and validation procedures. A matrix like the one below can be devised to aid summarising comments within reports relating to these aspects.
Comments
Category Favourable Unfavourable Difficult to categories
Training method //// /// //
Audio-visual aids //// // // 0
Programme content //// // 0
Validation procedures /// // //
This allows a numerical statement to be made. However, considerable information has been lost compared with direct quotes and a written summary, but statistical analysis can now be carried out.
Categories and decisions should be checked by somebody else. This can either be done by asking a colleague to review the categories and decisions or doing the whole job themselves.
In all three methods of summarising open response questions great care needs to be taken that the summary is not distorted by features the analyser has chosen to focus on, when he or she chooses the categories or main points contained in the reports.
One way of overcoming this is by two people independently analysing the reports, this then allows any discrepancies between the two summaries to be looked at again in more detail.
An example set of reports is given as Appendix 2. You may care to use them as practice – there are no provided answers!
Closed response questions
Closed response questions are easier to summarise. There are two basic types of summary sheets – tally and score.
Tally sheets
The original questionnaire can be used or a simple sheet can be designed, as below:
----------------
Examples of blank score sheets used for Likert and semantic differential scales are given as Appendices 3 and 4. These are used for recording responses to an opinion questionnaire about training activities – workshops. The item number would need to be inserted. In the case of the Likert scales a % score is obtained for displaying on a bar chart and similar information from the semantic differential scales – except in this case it is displayed on a graph.
Example
In this problem the trainer (validator) decided not to use a computer. The initial problem was to decide on the categories that were to be used to summarise the data. As explained above, in closed response questions most responses will allow a numerical value or letter code to be used, whereas open response questions have responses that may or may not be acceptably coded.
In this example questionnaire, both closed and open questions have been used. The summary sheet should allow subsequent analyses to proceed easily by having rows of scores that need to be operated on in proximity to other relevant scores. It is necessary at this stage to decide which if any statistical analysis methods will be used later.
Data summary sheet for the questions in this example is given below:
Days
0-40 41-80 81-120 121-160 161-200 Average Range
A.
Theoretical studies
Practical skills
Project work
Can't really remember
Percentage
value of total assessment work
Example continued
Data Summary Sheet continued
C. Too much time
Too little time
About the correct amount
D. You
Your manager
Both of you
Someone else
Comments See back of page
Number of sessions
Organising the questionnaire
It is important that the appearance of the pages of the questionnaire do not deter the respondent from co-operating when completing it either by not completing it, or by completing only part of it. Each page should not appear to be crammed. There should be:
• Adequate spaces between questions
• Adequate spaces between questions and responses
• Adequate spaces for open responses
• A to the right hand side is the best space for closed responses
• The question sequence should be logical, making certain that contingency questions follow the starter question and are clearly related
• Adjacent questions should not influence each other
• The instructions given to respondents must be carefully checked to ensure that they are unambiguous
• This includes sub-instructions where relevant and the spacing of sub-sections
• The introduction to the questionnaire must be clear and include a deadline for return
• The questionnaire should be tried out on a few people with knowledge of the content if possible (typical respondents) alternatively, someone who can put themselves in the position of a respondent NB. The pilot run is important. It allows for the removal of ambiguities. Any changes that are made should be tested against the criteria list in 'Testing the Questions' on page 18. If the order of the questions needs to be changed, adjustments may need to be made to the data summary sheet.
Administering the questionnaire
The questionnaire can be used as part of the validation strategy. Decide when to administer it. Estimate the time it will take to print, mail out, respond to, score, process and analyse it.
Remember: Asking the respondent's name may give answers less farther right than if the option of anonymity is given. However, the respondent's name allows for encouragement of a higher response rate to questionnaires by a follow-up request for its return.
A checklist for the construction of a questionnaire is given in Appendix 5.
Interpreting the results of the questionnaire
All constructors of questionnaires when they first start to interpret the results seem to want to find some formulas which will help them to decide whether they have got good or bad information. Unfortunately there are no magic formulas to help them.
There are, however, a few clues:
• Does the information match up with their objectives and/or what they expected to find?
• Can the information be compared with a similar or related study or investigation?
• Does the information appear complete enough to aid deciding courses of action?
Usually a report will be required and notes on this is given elsewhere.
Appendix 1
Analysis matrix
Appendix 2
Example set of reports
This set of seven reports was written by a group of learners attending a two-part Training/Educational Technology Programme. The reports were written after the first part. They were written in response to the general question:
'What is your reaction to this part of the programme?'
The reports were limited to one side of A4 paper. They illustrate the problems of analysing open response questions. You may like to summarise them, using the three methods outlined on pages 19-25. Ask a colleague to do the same, and check one another. You do not have to start at Report A, or read them in the order given.
The analysis matrix sheet (Appendix 1) can be used, or cards.
Report A
I had expected a programme oriented more towards gadgetry with a lesser theoretical content, but this was due to the fact that I had no previous experience of the subject.
The programme and the way it was presented was fascinating and stimulating to the extent that working on the three given projects was a pleasure, and time alone appeared to be the limiting factor.
In hindsight, the programme will be very valuable to me as a trainer and it is remarkable not only in the breadth of its coverage but also in the depth of its penetration in this subject, in such a short period of time.
Report B
It has been interesting, being mostly new, and stimulating. When applying for the programme in the first place I thought that we should be dealing more with hardware, that we haven't has not disappointed me since there has been so much to do that was new that to have spent more time on hardware would have meant the loss of much that will be, and has already been, useful.
The worksheets have been most helpful in guiding me through 'the darkness' as well as providing a framework in which to base the assignments.
There has seemed to be a tremendous amount of reading to do, but on reflection, I think it is preferable to being given a long reading list at the beginning to which one may refer in one final assignment. By having to consult journals etc., for references I have been made more aware of the range available.
Appendix 2 continued
Report C
Gut reaction is needed for a more media-based module; for more studio time, for directed play to investigate potentialities of media and hence for aesthetic determination of values of media-based learning systems as well as technical-based.
Report D
There was an attempt to cover too much ground in time available. I must have spent about 80% of my study time at least, on Ed.Tech. and I feel I have not completed the programme to my satisfaction as all worksheets have not been completed.
Practical work included in not assessed. Practical assessment only takes place in the following part, a part I have not opted for. So my practical work will not be assessed.
Some rationalisation is needed.
I have enjoyed the programme, in a masochistic way, and objectives stated before each section were a great aid to relevant study.
Report E
1. A fair summary of an individualised programme.
2. Value of assignments and work put in.
3. The three weeks of media works fit rather unhappily into the overall pattern.
Though Educational Technology devolved from media studies, it is by now more associated with systems thinking and actual mechanical use of media and elementary media production may be less relevant.
4. With the exception of the session on Bloom there seemed little advantage in splitting the group. The amount of interaction was rather limited and probably would have been maintained in a total group situation.
5. The wallet folder packs in the library seemed to do what they were intended to.
Report F
The course is a useful contribution to the techniques of training. It brings together a body of scattered information in the field of training such as objectives, mastery learning, evaluation, and validation, AV media, etc. This is especially so in writing up the last assignment on 'design'. The course is self-contained and comprehensive. It is well organised and carefully planned. As a direct trainer I find this a most valuable part of the whole programme.
Report G
1. It did what I wanted it to do – it gave me a good theoretical framework for reviewing my own considerable training experience and also served as an introduction for further study.
2. The work load was excessive at times, but in fairness I do not see how it could have been reduced.
3. The unavailability of reading and wallet folder packs was often inconvenient but again, it would have been difficult to overcome this.
4. I found it impossible to relate the first two assignments to my final choice of topic. It was only after I had down the first two that I was able to decide finally on a specific topic. (But I would concede that this was only difficult because I was not dealing with a specific curriculum).
5. I found it enjoyable.
6. I would have liked more time to read around subjects.
Appendix 3
----------------
Appendix 4
----------------------
Appendix 5
-----------------
This workbook is intended to illustrate some of the different types of objective test items and to provide practice in their construction.
You will be shown examples of the different test items and learn some of the pitfalls which exist in objective test construction.
At intervals throughout the workbook you will be asked to answer questions based on material previously covered. By careful reading you should be able to respond correctly before looking back to check your answers.
Types of objective test item
1. Introduction
Writing any type of examination question requires skills of the highest order.
1.1 The first essential is a very good knowledge of the subject to be tested.
1.2 The second essential is the important skill of the perfect, unambiguous communication which is demanded of an objective test item-writer.
1.3 The next essential quality is the capability to accept criticism without emotional reaction. The fact is that test items are prepared for communication to many different minds. Clear and accurate communication is likely to occur between the item-writer and only a few of the candidates. It is for this reason, above all others, that objective test items must be submitted to criticism by several other persons in the hope that the communication of the test item may be improved for a larger group of candidates.
1.4 In addition, the best item-writers will probably have experience of teaching thesubject to candidates at the same level as those expected to sit the test.
1.5 Finally the item-writer must be ready to absorb and work within the objectives which are to be tested. This will involve:
(i) A clear appreciation of instructional skills.
(ii) A thorough knowledge of the appropriate syllabus.
(iii) An ability to judge whether a question is easy, average or difficult for a given group of candidates.
2. Short-answer items
2.1 The short-answer item is the only objective item type that requires the candidate to supply, rather than select, the answer. Its make-up consists of a question or incomplete statement to which the candidate responds by providing the appropriate words, numbers or symbols.
2.2 Examples
What is the first thing you must do if your vehicle breaks down on a motorway?
It is against the law to sound your horn in built up areas, between ___ pm and ____am.
2.3 Rules for constructing short-answer items
(i) State the item so that only a single brief answer is possible.
What appears to be a simple clear question to the item-writer can frequently be answered in many different ways.
(ii) Start with a direct question and switch to an incomplete statement only when greater conciseness is possible.
The use of a direct question increases the likelihood that the problem will be clearly stated and that only one answer will be appropriate. Also incomplete statements tend to be less ambiguous when based on
problems that were first stated in question form.
(iii) The words to be supplied should relate to the main part of the statement.
Leave blanks only for key words or statements. Avoid asking the candidates to supply such words as ‘the’ and ‘an’.
(iv) Place the blanks at the end of the statement.
This permits the student to read the complete problem before he comes to the blank. If possible do not vary the length of space according to the answer, so that clues to the length of answer required are given.
(v) Keep sentences short and unambiguous.
(vi) For numerical answers indicate the degree of precision expected and the units in which they are to be expressed.
This will clarify the task to the student and make scoring easier. Where the learning outcome requires students to know the type of unit in common use and the degree of precision expected, this rule must of
course be discarded.
2.4 There are two major problems in constructing short-answer items. First it is extremely difficult to phrase the question or incomplete statement so that only one answer is correct. Second, there is the problem of spelling. If credit is given only when the answer is spelled correctly, the poor spellers will be prevented from showing their true level of achievement. On the other hand, if attempts are made to ignore spelling during the whole process, there is still the problem of deciding whether the badly spelled word represents the intended answer.
2.5 Due to the weaknesses mentioned above, the short-answer item should be reserved for those special situations where supplying the answer is a necessary part of the learning outcome to be measured - for example, where problems are used requiring computational solutions, or where the intent is to have the students ‘recall’ the information, or where a selection-type item would make the answer obvious.
2.6 Examples Comment
1. The person responsible to the Managing Director for departmental budgets is
2. To find the split half reliability of a test you correlate the odd numbered questions with the numbered questions.
3. What is the square root of 16?
4. What are the incorrect responses in a multiple-choice item called?
Satisfactory
Poor. The candidate needs only a hazy knowledge of the subject to make the correct response.
Satisfactory
Poor. The correct answer is ‘Distractors’ but any number of answers might equally well be defended, eg
incorrect alternatives, wrong answers, decoys, foils, etc.
2.7 When judging the quality of short-answer items use the following check-list.
1. Is the item realistic and practical?
2. Is the language appropriate for the level of the candidate?
3. Does the item require greater knowledge than most students could be expected to possess?
4. Does the item contain details which might reveal the correct response to the candidates?
5. Is the item free from ambiguities?
6. Is there only one correct answer?
Now write two or three short-answer items in the space provided below. Apply the check-list to your items. If you are satisfied, ask a colleague to apply the check-list to the items.
3. True/false
3.1 The true-false item is simply a statement which the candidate must judge as true or false. There are modifications of this basic form in which the student must respond yes or no, agree or disagree, right or wrong, fact or opinion, etc.
Such variations are usually given the more general name of 'alternativeresponse' items. In any event, this item-type is characterised by the fact that only two responses are possible.
3.2 Examples
Are the following statements true or false. True False Record your answer by placing a tick in the
appropriate box.
It is forbidden to overtake any vehicle within the area of the zig-zag lines before a zebra crossing. Ice on the road con sometimes be detected by a change of tyre noise.
3.3 Rules for constructing true-false items
(i) Include only one central significant idea in each statement.
The main point of the item should be an important one, and it should be in a prominent position in the statement. The true-false decision should not depend on some subordinate point or trivial detail. The use of several ideas in each statement should generally be avoided because these tend to be confusing and are more apt to measure reading ability than the intended knowledge outcome.
(ii) Word the statement so precisely that it can be judged unequivocally true or false.
True statements should be true under all circumstances, and yet free of qualifiers (words such as, may, possible, etc.) which might provide clues. This requires the use of words which are definite and precise, and the avoidance of such vague terms as ‘seldom’, ‘frequently’, and ‘often’. Can you see why the second example in 3.2 above is poor? The same care must be taken of course, when preparing false items so that their falsity is not too readily apparent from the differences in wording.
(iii) Keep the statements short and use simple language structure.
Short simple statements will increase the likelihood that the point of the item is clear, and that passing or failing it will be determined by the student’s knowledge. As noted earlier, long involved statements tend to measure reading comprehension, which, of course, defeats the intended purpose of the measurement.
(iv) Use negative statements sparingly and avoid double negatives.
Negative statements are frequently misread as positive statements because the ‘no’ or ‘not’ is so easily overlooked. Negative words should be underlined or put in capital letters. Double negatives are confusing, and such statements can usually be restated in positive form. (See 3.7 example 2).
(v) Avoid extraneous clues to the answer.
There are a number of words which provide verbal clues to the truth or falsity of an item. Statements which include such absolutes as ‘always’, ‘never’, ‘all’, ‘none’, and ‘only’ tend to be false. By the same token, statements with qualifiers like ‘usually’, ‘may’, and ‘sometimes’ tend to be true. The length and complexity of the statement might also provide a clue. True statements tend to become longer and more complex than false ones because of the need for qualifiers.
(vi) The item should be plausible enough to deceive the student whose knowledge is incomplete.
3.4 Contrary to popular belief, the true-false item is one of the most difficult items to construct. In most areas of knowledge, the more important statements must be qualified to make them absolutely true and the qualifiers provide obvious clues.
3.5 Since only two choices are possible, the uninformed student has a 50-50 chance of guessing the correct answer. This limits the range of scores on the test, and thus reduces its effectiveness as a measuring instrument.
3.6 Despite the limitations of the true-false item, there are situations where it should be used. Whenever there are only two possible responses, the truefalse item, or some adaptation of it, is likely to provide the most effective measure. The best procedure is to use true-false items only when the multiple choice form of item is inappropriate.
3.7 Examples Comment
1. The square root of 0.9 is 0.3.
TRUE/FALSE
2. Correction for guessing is NOT a practice which should NEVER be used.
TRUE/FALSE
3. Older people are more difficult to train, are less adaptable to change, and are too old to learn.
TRUE/FALSE
4. A test CANNOT be reliable unless it is valid.
TRUE/FALSE
Satisfactory. Note that this is a common fallacy.
Poor. Double negative makes the statement confusing.
Note use of capital letters.
Poor. Too many ideas in the statement.
Satisfactory. The terms ‘reliable’ and ‘valid’ when used in this context are often confused.
3.8 When judging the quality of true/false items use the following check-list.
1. Is the item realistic and practical?
2. Does the item contain one central significant idea?
3. Is the item clearly true or clearly false to the able candidate?
4. Is the language appropriate for the level of the candidate?
5. Does the item contain extraneous clues to the correct answer?
6. Could the learning outcome to be measured be better tested with a different form of test item?
Now write two or three true/false items in the space provided below. Apply the check-list to your items. If you are satisfied, ask a colleague to apply the check-list to the items.
Progress test number one (based on Sections 1, 2 and 3 of this booklet)
Are the following statements True or False? Record your answer in the appropriate box.
True / False
1. The short-answer item is the only objective item type that requires the candidate to supply, rather than select, the answer. True / False
2. When providing space for the answer to a short-answer item the space should vary according to the length of the answer.
3. List three of the essential qualities required by a writer of objective test items.
(i)
(ii)
(iii)
4, List three of the items from the check-list used to check the quality of shortanswer items.
(i)
(ii)
(iii)
5. List three of the items from the check-list used to check the quality of true/false items.
(i)
(ii)
(iii)
6. List three words which are liable to provide verbal clues when writing true/false items.
(i) (ii) (iii)
You can check your answers by referring to the back of this sheet.
Answers to progress test number one
1. True see para 2.1
2. False see para 2.3 (iv)
3. Any three see paras 1.1, 1.2, 1.3, 1.4, 1.5
4. Any three see para 2.7
5. Any three see para 3.8
6. Any three see para 3.3 (v)
4. Multiple-choice items
4.1 It is probably true that the multiple-choice test, with a minimum of four choices, is the most widely used instructionally respectable form of objective testing. Even subsets of “knowledge” such as terminology, conventions of the subject, methods of procedure or simple facts can be tested by sensitively prepared multiple-choice tests. This versatility enables the test constructor and the item-writers to operate upon one standard form of item and yet test all the various qualities as required by the instructional objectives.
4.2 The multiple-choice item consists of a stem, which presents a problem situation, and several alternatives, which provide possible solutions to the problem. The stem may be a question or an incomplete statement. The alternatives include the correct answer and several plausible wrong answers, called distractors. Their function is to distract those students who are uncertain of the answer.
4.3 The following items illustrate the use of both the question-form and the incomplete-statement form of multiple-choice item.
Incomplete-statement Question
A tachometer indicates What does a tachometer indicate?
A. road speed A. Road speed
B. oil pressure B. Oil pressure
C. engine speed C. Engine speed
D. battery charge D. Battery charge
Note in these examples that both stems pose the same problem. The question form is easiest to write and forces the item-writer to pose a clear problem but tends to result in a longer stem. An effective procedure for the beginner is to start with a question, and to shift to the incomplete statement only if greater conciseness can be obtained.
The alternatives in the above examples contain only one correct answer and the distractors are clearly incorrect.
4.4 Another type of multiple-choice item is the best-answer form in which the alternatives are all partially correct but one is clearly better than the others.
Example
A recall situation is best tested by a
A. multiple-choice type test.
B. short-answer type test.
C. true/false type test.
D. matching type test.
This type is used for more complex achievement, such as where the student must select the best method for doing something, the best reason for an action, or the best application of a principle. Thus, whether the correct-answer or best-answer form is used will depend on the learning outcomes to be measured. Since any test is likely to contain items of both types, it is important that the directions tell the student to select the best answer.
4.5 The above examples also illustrate the use of four alternatives.
Multiplechoice items typically include four or five choices. The larger number will, of course, reduce the student’s chances of obtaining the correct answer by guessing. It is frequently difficult to obtain five plausible choices, however, and an item is not improved by adding an obviously wrong answer merely to obtain five alternatives. There is no reason why the items in a given test all need to have the same alternatives. Some might contain four and some five, depending on the availability of plausible distractors.
4.6 Rules for constructing multiple-choice items
(i) Design each item to measure an important learning outcome.
Avoid testing for unimportant details, unrelated bits of information, and material that is irrelevant to the desired outcomes. Resist the temptation to increase item difficulty by resorting to the more obscure and less significant items of knowledge. Remember that each test item is expected to call forth student behaviour which will serve as evidence concerning the extent to which the instructional objectives have been achieved.
(ii) Present a single, clearly formulated problem in the stem of the item.
The task set forth in the stem should be so clear that it is understood without reading the alternatives.
(iii) State the stem of the item in simple, clear language.
The problem in the stem of a multiple choice item should be stated as precisely as possible and be free of unnecessarily complex wording and sentence structure. Complex sentence structure may make the item more a measure of reading comprehension than of the intended knowledge outcome. Another common fault in stating multiple choice items is to load the stem down with irrelevant and, thus, non-essential material.
(iv) Put as much of the wording as possible in the stem of the item.
Avoid repeating the same material over again in each of the alternatives. It is of course, impossible to streamline all items, but economy of wording and clarity of expression are important goals to strive for in test construction.
(v) State the stem of the item in positive form, wherever possible.
The use of negatively stated item stems all too often results from the ease with which such items can be constructed, rather than from the importance of the learning outcomes measured. Being able to identify answers which do not apply provides no assurance that the student possesses the desired knowledge.
(vi) Emphasise negative wording whenever it is used in the stem of an item.
There are instances when the use of negative wording is basic to the measurement of an important learning outcome. For example:
Knowing that you must not cross the street against a red light. or Knowing that you should not mix certain chemicals together are so important that they might be directly taught and directly tested.
Where negative wording is used in the stem of an item, it should be placed near the end of the statement and emphasised by underlining or capital letters.
(vii) Make certain that the intended answer is correct or clearly best.
When the correct-answer form of multiple-choice item is used, there should be only one correct answer and it should be unquestionably correct.
(viii) Make all alternatives grammatically consistent with the stem of the item and parallel in form.
The correct answer is usually carefully phrased so that it is grammatically consistent with the stem. Where the item writer is apt to slip is in writing the distractors. Unless care is taken to check them against the wording in the stem and the correct answer, they may be inconsistent in tense, article, or grammatical form. A general step that can be taken is to avoid using the article ‘a’ or ‘an’ at the end of the stem of the item.
(ix) Avoid verbal clues which might enable students to select the correct answer or to eliminate an incorrect alternative.
One of the most common sources of extraneous clues in multiplechoice items is to be found in the wording of the item. Here is a list of some of the verbal clues commonly found in items:
(a) similarity of wording in both the stem and the correct answer,
(b) stating the correct answer in textbook language or stereotyped phraseology,
(c) stating the correct answer in greater detail,
(d) including absolute terms in the distractors (enables candidates to eliminate them as possible answers, because such terms are commonly associated with false statements, e.g. always, never, none, all, only).
(x) Make the distractors plausible and attractive to the uninformed.
The distractors in a multiple-choice item should be so appealing to the student who lacks the knowledge called for by the item that he selects one of the distractors in preference to the correct answer. The art of
constructing good multiple-choice items depends heavily on the skillful development of effective distractors. There are a number of things that can be done to increase the plausibility and attractiveness of distractors. These are summarised below:
(a) use the common misconceptions, or common errors of candidates as distractors.
(b) state the alternatives in the language of the student.
(c) use ‘good’ sounding words (eg accurate, important, etc.) in the distractors, as well as the correct answer.
(d) make the distractors similar to the correct answer in both length and complexity of wording.
(e) use extraneous clues in the distractors, such as stereotyped phrasing, scientific sounding answers, and verbal associations with the stem of the item. But don’t overuse these clues, and beware of trick questions.
(f) make the alternatives homogeneous. The difficulty of an item can be controlled by making the alternatives more homogeneous, ie making the candidate make finer discriminations.
(xi) Vary the relative length of the correct answer to eliminate length as a clue.
There is a tendency for the correct answer to be longer than the alternatives because of the need to qualify statements to make them unequivocally correct. In some cases, it is more desirable to make the alternatives approximately equal in length, by adjusting the distractors rather than the correct answer.
(xii) Avoid use of the alternative “all of the above” and use “none of the above” with extreme caution.
When the item-writer is having difficulty in locating a sufficient number of distractors, he frequently resorts to the use of “all of the above” or “none of the above” as the final option. These special alternatives are seldom used correctly and almost always render the item less effective than it would be without them. The inclusion of “all of the above” as an option makes it possible to answer the item on the basis of partial information. The alternative “none of the above” is probably most widely used with computational problems when these are presented in multiple-choice form.
(xiii) Vary the position of the correct answer in a random manner.
The correct answer should appear in each alternative position approximately an equal number of times, but it should not follow a pattern that may be apparent to the person taking the test. When the alternative responses are numbers, they should always be placed in order of size, preferably in ascending order. This will eliminate the possibility of a clue, such as the correct answer being the only one that is not in numerical order.
(xiv) Control the difficulty of the item either by varying the problem in the stem or by changing the alternatives It is usually preferable to increase item difficulty by increasing the level of knowledge called for or by making the problem more complex.
(xv) Make each item independent of the other items in the test.
Occasionally information given in the stem of one item will help in answering another item. This can best be remedied by a careful review of the items before they are assembled into a test.
(xvi) Use an efficient item format.
The alternatives should be listed on separate lines, under one another, like the examples in this section. This makes the alternatives easy to read and compare. The use of letters in front of the alternatives is preferable to using numbers. This avoids possible confusion, when numerical answers are used in an item. When writing the item, follow the normal rules of grammar. If the stem of the item is a question, each alternative should begin with a capital letter and end with a full stop or other termination punctuation mark. The full stop should be omitted with numerical answers, however, to avoid confusing them with decimal points. When the stem is an incomplete statement, each alternative should begin with a lower-case letter and end with whatever terminal punctuation mark is appropriate.
4.7 The above sixteen rules for constructing multiple-choice items are stated rather dogmatically, to aid the beginner. As experience in test construction is obtained, it will soon be noticed that there are exceptions to some of the rules and that minor modifications may be desirable in others. Until such experience is gained, however, the novice will find that following these rules will assist him to produce items of an acceptable quality.
4.8 Examples Comment
1. The paucity of probable but incorrect statements that can be related to a central idea pose a problem when constructing which one of the following types of test items ?
A. Short-answer
B. True/false
C. Multiple-choice
D. Essay
Poor. The stem of the item should be stated in simple, clear language.
Examples Comment
2. In objective testing, the term objective
A. refers to the method of identifying the learning outcomes.
B. refers to the method of selecting the test content .
C. refers to the method of presenting the problem.
D. refers to the method of scoring the answers.
3. The recall of factual information can be best measured with
A. matching items.
B. multiple-choice items.
C. short-answer items.
D. essay questions.
4. Which one of the following would you first consult to locate research articles on achievement testing ?
A. Journal of Educational Psychology
B. Journal of Educational Measurement
C. Journal of Consulting Psychology
D. Review of Educational Research
5. Achievement tests aid students to improve their learning by
A. encouraging them all to study hard.
B. informing them of their progress.
C. giving them all a feeling of success.
D. preventing any of them from neglecting their assignments.
Poor. Put as much of the wording as possible into the stem. eg In objective testing, the term objective refers to the method of
A. identifying ....................
B. electing .........................
C. presenting.....................
D. scoring..........................
Satisfactory.
Poor. The word research in both stem and the correct answer is apt to provide a clue to the uninformed candidate.
Poor. Including absolute terms (eg all, any of them) in the distractors enables candidates to eliminate them as possible answers, because such terms are commonly associated with false statements.
Examples Comment
6. The function of a tachometer is to
A. indicate the engine speed in rpm.
B. indicate road speed in mph.
C. it indicates oil pressure in lbs/sq. in.
D. shows whether or not the battery is charging.
7. On a 1” OS map a path is shown as a
A. single broken line.
B. two parallel broken lines.
C. dots and dashes.
D. unbroken line.
8. You are about to enter a roundabout at 6 o’clock and wish to go forward from it at the second turning at 12 o’clock. You should
A. use left flasher entering the roundabout, then right, then left at the exit before the one to be taken.
B. use left flasher all the way through.
C. use left flasher at the exit before the one to be taken.
D. use no flashers whatsoever.
Poor. Distractors C and D as worded do not fit the grammar of the stem.
Poor. Alternative A is the only one grammatically correct. It is usually unwise to complete a stem with the words ‘a’ or ‘an’.
Satisfactory. Perhaps a little too wordy. Note the use of diagram or sketch to provide reinforcement to stem.
4.9 When judging the quality of multiple-choice items use the following check-list.
1 Is the item realistic and practical?
2. Does the item contain one central significant idea?
3. Is it written in simple, clear language appropriate for the level of the candidate?
4. Is there as much wording as possible in the stem of the item?
5. Is the stem stated in a positive form?
6. If negative wording has been used, has it been emphasised adequately?
7. Is the correct answer, unquestionably correct?
8. Are all the alternatives grammatically consistent with the stem of the item?
9. Does the item contain extraneous clues to the correct answer?
10. Are the distractors plausible to the uninformed candidate?
11. If the alternative “none of the above” has been used, has it been used correctly?
12. Is each item in the test independent of other items in the same test?
Now write two or three multiple-choice items in the space provided. Apply the checklist to your items. If you are satisfied, ask a colleague to apply the check-list to the items.
___________________________________________________________________
___________________________________________________________________
___________________________________________________________________
___________________________________________________________________
Progress test number two
(based on Sections 1, 2, 3 and 4 of this booklet)
Instructions:- Each question has a number of alternative answers only one of which is correct.
You are required to select the correct alternative.
Record your answer by placing a tick in the appropriate box.
1. The stem of a multiple choice item
A. answers the question.
B. provides the answer.
C. asks the question.
D. provides the distractors.
E. Provides the responses.
2. Which of the following describes the question-form used in Question 1?
A. Multiple-choice short-answer.
B. Multiple-choice incomplete statement.
C. Multiple-choice question.
D. Multiple-choice best answer.
3. The lack of probable wrong answers will cause the greatest difficulty when constructing
A. short-answer items.
B. true/false items.
C. essay items.
D. multiple-choice items.
4. All of the following are desirable practices when preparing multiple-choice items except
A. stating the stem in positive form.
B. using a stem that could function as a short-answer item.
C. shortening the stem by lengthening the alternatives.
D. underlining certain words in the stem for emphasis.
E. making alternatives grammatically consistent with the stem.
5. Why should negative terms be avoided in the stem of a multiple-choice item?
A. They may be overlooked.
B. They tend to increase the length of the stem.
C. They make the construction of alternatives difficult.
D. They make the scoring more difficult.
6. The alternatives in a multiple-choice item should be
A. exactly the same length.
B. as equal in length as possible.
C. longer than the correct answer.
D. shorter than the correct answer.
7. Which of these statements is most correct ?
A. Five choices is better than four.
B. If you cannot find three plausible distractors make do with two.
C. Always include ‘none of the above’ as your last choice.
D. Offering choices means that the student does not need to waste time thinking.
8. List five of the items from the check-list used to check the quality of multiplechoice items.
(i)
(ii)
(iii)
(iv)
(v)
You con check your answers by referring to the back of this sheet.
Answers to progress test number two
1. C see para 4.2
2. B see para 4.3
3. D see paras 4.5, 4.6 (x)
4. C see paras 4.6 (iv), 4. 8 Example 2
5. A see para 3.3 (iv)
6. B see para 4.6 (xi)
7. A see para 4.5, see also 3.6, 4.6 (xii)
8. Any five see para 4.9
5. Matching items
5.1 The matching item is simply a modification of the multiple-choice form. Instead of listing the possible responses underneath each individual stem, a series of stems, called premises, is listed in one column and the responses are listed in another.
5.2 Example
Directions : Column 1 lists the names of some of the symbols found on a 1/50000 Ordnance Survey Map. Column 2 lists a number of these symbols.
In the box provided write the letter of the symbol in column 2
which corresponds to each name in column 1.
5.3 In the example, it can readily be seen that seven separate multiple-choice items could have been used instead. Unless all of the responses in a matching item serve as plausible alternatives for each premise, the matching format is clearly inappropriate. In each subject-matter area there are relatively few situations where this condition can be met.
5.4 Rules for constructing matching items
(i) Include only homogeneous material in each matching item.
In the example, we included only Ordnance Survey symbols and their names. Such homogeneity is necessary to make all responses plausible.
(ii) Use a larger, or smaller, number of responses than premises, and permit the responses to be used more than once.
Both an uneven match and the possibility of using each response more than once reduces the guessing factor. Remember, correct use of the matching form requires that all responses serve as plausible alternatives for each premise. This of course, that each response is suitable for reuse within the same item.
(iii) Specify in the directions the basis for matching and indicate that each response may be used once or more than once.
This will clarify the task for all candidates and prevent any misunderstanding. Care should be taken, however, not to make the directions too long and involved. Note the instructions given before the example in 5.2.
5.5 Examples Comment
1. Match the units with the weapon calibres.
A. Missile Regiment 1. 175 mm
B. Heavy Regiment Royal Artillery 2.7.62 mm
C. Field Regiment Royal Artillery 3.155 mm
D. Medium Regiment Royal Artillery 4.8 in
E. Infantry Battalion 5.105 mm
Poor. Items are not homogeneous (infantry calibre should be easily guessed). The names Heavy and Medium are prompts. The lists are equal in length and so only four of the five need be known.
Examples Comment
2. Match the correct road marking with the instruction to the driver. All these markings are located in the centre of the carriageway.
A. No crossing
B. Lane Line
C. Centre line
D. Warning line
A B C D
Satisfactory. Perhaps the markings are not quite large enough
6. Multiple-response items
All the points that have been made about multiple-choice items apply to multipleresponse items, except that instead of one of the alternatives being correct or best, two or more of the are correct or best.
The ideal in multiple-choice items is one of five alternatives being correct or best. In multiple-response it is two out of ten or three out of fifteen, and so forth.
They are only infrequently used, either because it is not possible to think of two or more correct or best answers, and/or the number of distractors required defeats the item writer. They can be a valuable addition to a test providing that it is made abundantly clear to the respondent that he has to choose more than one alternative.
There are numerous instances of respondents failing to choose more than one alternative when asked to do so, and thus not demonstrating their learning. Usually this is because of poor written instructions in the test paper.
Example
The playwright William Shakespeare wrote:
(a) The Maid’s Tragedy;
(b) Waiting for Godot;
(c) The Lady’s not for Burning;
(d) As You Like It;
(e) The Importance of Being Earnest;
(f) Roar Like a Dove;
(g) Under Milk Wood;
(h) Romeo and Juliet
(i) Butter in a Lordly Dish;
(j) Whose Life is it Anyway?
Summary
The advantages of using multiple-response test items are that they:
Are purely objective;
Reduce guessing;
Can be designed to test a variety of learning principles;
Tend to be more reliable than some other item forms.
Their disadvantages are that they:
Tend to develop items that measure facts along;
Make it difficult to construct enough plausible alternatives;
Consume a lot of space.
Some suggestions for construction:
The ratio of 2:10, 3:15 correct or best answers in relation to the total number of alternatives is recommended;
Design alternatives so that most of them are plausible;
Avoid: - Wording that serves as clues
- Changes in parts of speech
- Mixing singular and plural
- The use of ‘none of the above’ or ‘all of the above’;
Vary the position of the correct responses to avoid creating a pattern of correct answers;
Make the alternatives as nearly equal in length as possible; be consistent; if the answer is a number, use all numbers and place them in sequence.
Some guidelines to use when judging the quality of multiple-response items:
Is the item realistic and practical?
Does the item contain one central significant idea?
Is it written in simple clear language appropriate for the level of the respondent?
Is there as much wording as possible in the stem of the item?
Is the stem stated in a positive form?
If negative wording has been used has it been emphasised adequately?
Are the correct answers unquestionably correct?
Are all the alternatives grammatically consistent with the stem of the item?
Does the item contain extraneous clues to the correct answer?
Are the distractors plausible to the uninformed respondent?
If the alternative ‘none of the above’ has been used, has it been used correctly?
Is each item in the test independent of other items in the same test?
If numbers have been used are they in sequence?
STEPS INVOLVED IN CONSTRUCTING OBJECTIVE TEST ITEMS
Objective
Select
Write test items
Review and modify test items
Design layout/instructions/scoring
Review and modify questionnaire
Pilot test
Mark, review and modify test
Further pilot of test if necessary
Mark, review and modify test
Use
STEPS INVOLVED IN CONSTRUCTING QUESTIONNAIRES
Objective
Select
Write items
Review and modify items
Design questionnaire: layout/instructions/scoring
Review and modify questionnaire
Pilot questionnaire
Score, review and modify questionnaire
Further pilot of questionnaire if necessary
Score, review and modify questionnaire
Use
STEPS INVOLVED IN CONSTRUCTING SKILL PROCESS CHECKLISTS
Objective
Select
Analyse good/bad performers/key steps and criteria
Define categories
Construct checklist
Establish profile – criteria scores for each step
Review and modify checklist
Design layout/instructions/scoring
Review and modify checklist
Pilot checklist against performers
Score, review, and modify checklist
Further pilot checklist if necessary
Score, review, and modify checklist
Use
V E N U E C H E C K L I S T
Venue address Hotel group?
Telephone: Date of 1st contact
Contact name Location map available?
N.H.B.C. event:
Function Name:
Room layout:
Boardroom Open
Boardroom
Theatre Schoolroom Horseshoe
with tables
Horseshoe
no tables
Capacity:
Needed …….. …….. …….. …….. …….. ……..
Capable …….. …….. …….. …….. …….. ……..
Location of room: GF Above GF Below GF
Access to room: Stairs or steps en route: Wide Narrow
Doors en route: Wide Narrow
Corridors en route: Straight Turns
Loading from vehicle? Under cover No cover Otherwise: Excellent Fair Poor
Natural light? None Windows 1 side 2 sides 3 sides 4 sides Roof lights
Any windows facing EAST (Problems on very sunny mornings with glare)
Artificial lights? Fluorescent Tungsten Spots fixed Spots swivel Spots adjustable track
Ornamental chandeliers
Switches? Dimmers Lectern controls Otherwise in room Outside room Control room
Blackout? (windows & roof lights) 100% 75% 50% 25% Nil Ceiling height?….. ft/m (100% necessary for 35mm slides. At least 25% room darkening useful for OHP)
Any obstruction to view or to projector beam (hanging lights, low beams?)………………
Heating? Radiators Fans Gas portable Heating controls? Good Fair Poor
Road noise when open? Acceptable NOT acceptable None (not near road)
Air conditioning A/C fans noise (run them during vetting): High Medium Low
Kitchen / catering: Too close/noisy with respect to location of function room Not a problem
Dining: In meeting room Separate from meeting room but, Close Not close Licensed bar?
If any meals served in function room, will clearing up be done ONLY during breaks? Yes No
Floor covering: Wood Plastic Carpet
Furniture: Chairs: Hard Soft Arms No arms Normal height Easy-chair height
Tables suitable for function? Good Fair Poor Table cloths available? Yes No
Table extras included in price? Top table – Water + glasses Cordials Sweets Gavel + block
Delegates – Water + glasses Cordials Sweets
Toilets: Excellent Fair Poor Proximity to meeting room: Close Not close Outside
Car parking off road: Cars How many. Car + trailer Yes No Height restriction? …… ft/m Rates:
Daily delegate rate (DDR): £……..
(includes: Morning coffee Lunch Afternoon tea Hire of meeting room:
35mm projector OHP 16mm cine proj Flipchart & easel
Lunch = Sit-down meal Finger buffet Fork buffet and includes: Sweet Coffee
24 hour delegate rate: £…….
(includes all daily delegate rate features + bed, breakfast and evening meal)
Separate rates: (for rooms, assumes bath or shower included)
B/Br* (single) £…… B/Br/EM* (single) £…..… Br (E) £…….. Br (C) £…..…
B/Br* (double /twin) £…… B/Br/EM* (double/twin) £…..… Dinner (alc) £..……
Tea/coffee making in room Colour TV B&W TV Trouser press Hair dryer
Separate MEETING ROOM hire rate (if not opting for all-in “DDR” or “24 hr” rates)
Name of room……………. 9-5 rate £……… AM/PM/Evening rate £..……
Builders: None on site at time of booking Promised none on site at time of function
NOTES: (Name of venue FUNCTIONS SUPERVISOR on the day)………………………………………………………..)
(If vetting a venue for the Training Department, a rough sketch – separate sheet? – of the room shape and dims together with an indication of door, window, power sockets and lights positions, will be helpful).
On the day venue check list
Check planned meals arrangements and timings with venue staff.
V/a equip check: All working? Projectors registered on screens Spare bulbs available
Room checks: Ventilation before meeting Heating Cooling Blackout (if required)
Seating: Correct number Can all delegates see screens?
Speakers: Water + glasses Pens, pointers etc. for visual aids Rehearse equipment
Means of escape (Check personally): Walk complete route, checking for temporary obstructions
Find out and announce to delegates the fire instructions and assembly point
Lunchtime break check list:
Drinks + glasses renewed? Ashtrays cleaned
MoE still OK (10 mins before afternoon start)?
THE TEST TEST
Name of Test
Questions
Yes
?
No
Comments / Action
1. Is the test fair?
Is it reliable?
Has it been repeated with this learner?
Has it been used with other learners?
Is the test valid?
- predictive?
- concurrent?
- content?
- construct?
- face?
2. Is the test being used with an individual?
Have you established the criteria?
Is it being used by a group?
Have you established the norms?
3. Is the test likely to help the learner?
- is it overt?
- is it explicit
- is it transparent?
- is it standardised?
- is it understandable?
- is it appropriate to learner?
- is it appropriate to objectives?
- is it specific?
- is it neutral of any bias?
OBJECTIVE TEST ITEM CHECKLIST
All items should be:
• Relevant to the objective
• In language appropriate for the trainees
• Independent of other items
• Free from clues to give away answers
• Unambiguous
• Clearly worded
• Technically accurate
• The best type of question to test the trainee=s knowledge
Sentence completion, ‘short answer’ items should:
• Have one correct answer
• Contain no irrelevant content
• Have a short stem
• Require a short answer
• Allow adequate space for the answer, on the right-hand side
Multi choice, multi-response, and matching items should:
• Contain homogeneous material
• Have real alternatives and plausible distracters
• Use directions indicating the number of possible correct answers
• Have choice of approximately the same length
• Have choices that fit the stem
• Allow adequate space for answers
True / False items should:
• Be sufficiently plausible
• Be clearly, and wholly, ‘true’ or ‘false’ to a competent trainee
• Not include extreme words, such as ‘all’, ‘always’, or ‘never’
Sequencing items should
• Give the trainee opportunity to rearrange methods procedures or sequences
