Introduction: This was my first WORC posting, having been asked to take over the task for the Spring 2012 semester. The evaluation of training was my thesis topic, and the first two WORC articles were ones I had come across during my literature review. I found them interesting but not relevant to use for my thesis, and was pleased to have the chance to write about them anyway.
The 2009 survey on evaluation conducted by the American Society for Training and Development (ASTD) found that nearly 92% of the respondents used reaction-level evaluations (Kirkpatrick Level 1) to collect data on the effectiveness of their training programs. Yet only 35% of them thought this data had a high level of value to their departments and organizations. If you’ve taken a training class, you’ve most likely filled out a survey asking what you thought about the class; that’s a Level 1 reaction evaluation. There are three basic topics covered in Level 1 surveys: class content/materials, instructor quality, and relevance of the class to one’s job needs. I think Level 1 surveys have considerable value if the questions are focused on the relevance of the training – did you learn things you can apply to your work, and do you believe you can use the new knowledge or skills effectively in your work? As someone who has been asked to fill out such surveys, however, I don’t recall being asked about relevance all that often.
This article takes a look at the survey content. As the authors pointed out, hardly any formal research has been done on Level 1 evaluations. Is it because researchers assume that these evaluations are so routine and automatic that there’s no point in looking at them in depth? What if learning professionals and researchers in the field took Level 1 evaluations more seriously instead of dismissing them as “smile sheets”?Â
*ARTICLE *
Giangreco, A., Sebastiano, A., & Peccei, R. (2009). Trainees’ reactions to training: An analysis of the factors affecting overall satisfaction with training. International Journal of Human Resource Management, 20(1), 96-111.
* SYNOPSIS *
The authors noted that academic research on the evaluation of training programs has been focused on behavior-based and results-based evaluations, as these are considered to be the most accurate measurements of the programs’ effectiveness in achieving organizational goals. However, in practice many training professionals are limited to conducting reaction-based evaluations and must rely on this level of data to gauge the quality of their training programs. In recognition of this reality, the authors examined the factors which affected the overall post-training satisfaction level of learners and how the factors related to each other.
* RESEARCH QUESTIONS *
The authors identified the following three main factors (i.e., independent variables) which impact learner satisfaction (i.e., dependent variable) after participating in a training class:
- Perception of the efficiency of the training class, including the organization and management of the class
- Perception of the usefulness of the training, including its relevance to both current job needs and personal career growth
- Perception of trainer performance, including method of delivery and mastery of content
Based on a review of the existing literature about training delivery, the authors assumed that trainer performance would have the greatest impact on learner satisfaction, and thus generated their five research hypotheses:
- The more positive the learners’ perception of the efficiency of the training class, the more positive their overall perception.
- The more positive the learners’ perception of the usefulness of the training class, the more positive their overall perception.
- The more positive the learners’ perception of the trainer performance, the more positive their overall perception.
- The perception of trainer performance will be more strongly positively related to overall learner satisfaction than the perception of efficiency or usefulness of the training class.
- The perception of trainer performance will affect the relationship between the perception of efficiency or usefulness and overall learner satisfaction, meaning that higher satisfaction with trainer performance will compensate for lower satisfaction levels with efficiency or usefulness when learners determine their overall satisfaction.
* METHODS *
The authors gathered survey data from 2,697 adults who had just completed one of the training classes offered through a joint venture between industry and trade union organizations in Italy. This program was designed to improve worker knowledge and companies’ competitiveness, and offered classes in a variety of subjects such as information technology, marketing, and foreign languages. Approximately three-fourths of the learners attended classes which were tailored to specific companies’ needs, with the others attending more generalized classes. The 14-question survey asked respondents to rate their satisfaction with the efficiency and usefulness of the class, the trainer performance, and the overall class.
The authors identified satisfaction with efficiency, usefulness, and trainer performance as the independent variables which might influence the dependent variable of overall satisfaction.
* FINDINGS *
The authors found that their respondents reported a high level of satisfaction with all four variables, with the average score for trainer performance significantly higher than for efficiency, usefulness, and overall satisfaction. They found that their first three hypotheses were supported by the data, in that a high satisfaction rating on any of the independent variables was positively correlated to high overall satisfaction.
However, their data indicated that their hypotheses about the dominance of trainer performance as a satisfaction factor were not proven. According to their analysis, the satisfaction rating for the perceived usefulness of the training class had a significantly stronger link to overall satisfaction than did the satisfaction rating for trainer performance. They also found that a higher trainer performance rating did not compensate for lower scores in either efficiency or usefulness when learners rated their overall satisfaction level.
Based on their data analysis, the authors concluded that the perception of a training class’s usefulness to the learner was the most significant factor in the learner’s overall satisfaction with the class, contrary to the common assumption that the trainer’s skill and knowledge had the greatest impact. However, all three of the independent variables did have a significant impact on the overall rating, and organizations should evaluate the learner reaction to training classes on all three factors.
* DISCUSSION FOR IPT-N MEMBERS *
When you conduct reaction-based training evaluations (Kirkpatrick Level 1), which aspects of the training class do you emphasize in your survey? Do you have a one-size-fits-all reaction survey, or do you tailor it for each specific class or program? Which theory (or theories) would you use to support the finding that the perceived degree of “usefulness†was the most significant factor in learners’ overall satisfaction with the training class?