8th International Conference on Computers in Education, Proceeding

An analysis of the structure of course evaluation items based on ARCS motivation model (2)

 

Chiharu KOGO* and Katsuaki SUZUKI**

* Toyama University, Toyama 930-8555, Japan
phone, fax: +81 76 445 6338
E-mail:kogo@edu.toyama-u.ac.jp

** Iwate Prefectural University, Iwate 020-0193, Japan
phone, fax: +81 019 694 2670
E-mail:ksuzuki@soft.iwate-pu.ac.jp


1 Introduction

John M. Keller's ARCS motivation model has been used as a framework for improving teaching strategy and making courses more appealing. It is useful if we have an evaluation sheet based on ARCS model as we can see clearly sufficient and insufficient aspects of a course. However, such a sheet has been developed only for a particular class or courseware, and we have never had a sheet for a general purpose, especially in Japanese. The objectives of this study were to make a preliminary version of the evaluation sheet and to analyze the structure of evaluation for comparing the presumed structure of motivation based on ARCS model.

 

2 Method

A questionnaire to evaluate an instructional unit/material was developed based on ARCS motivation model. It consisted of 16 items using 9-point Likert scale, each of which was written to represent one of the ARCS four motivational categories: Attention, Relevance, Confidence, and Satisfaction (see Table 1). Four classes of students enrolling in "Systems Theory", "Media Studies", "Educational Technology", and "Educational Information Processing" at college freshman through junior level filled out the questionnaire anonymously at the end of the 15-week course, based on their overall impression of the course.

 

3 Results

A factor analysis was conducted based on 209 responses, resulting in 4 factors. The eigenvalues were 5.57, 0.95, 0.80, and 0.75. We took a rotation method of VariMax, and then reached an oblique solution. Factors were named as "Interestingness", "Confidence", "Fairness of evaluation", and "Familiarity." Table 1 shows the oblique solution reference structure. Each four factors included the main items of each presumed categories of ARCS model. Factor 1 "Interestingness" included all items of "Attention" and the items of "relevant", "process enjoyable", and "satisfactory". Factor 2 "Confidence" included all items of "Confidence" and the item of "useful". Factor 3 "Fairness of Evaluation" included the items of "fairly evaluated" and "consistent evaluation". Factor 4 "Familiarity" included the items of "familiar" and "important".

 

4 Discussion

Although the final factor structure did not perfectly match the ARCS, it is suggested that the results of factor analysis supports the appropriateness of the structure of ARCS model.

The detailed observation of the results revailed the followings:

a) Factor 1 "Interestingness" mainly represented the Attention aspect. The items of "relevant", "process enjoyable", and "satisfactory" was also included in this factor, due to the result of interestingness of the instruction.

b) Factor 2 "Confidence" matched with the Confidence mostly. Item of "useful" could be judged to represent Confidence aspect of instruction, because of the way it was stated in Japanese.

c) Factor 3 "Fairness of Evaluation" should be considered as a separate factor regarding "Evaluation".

d) Factor 4 "Familiarity" was a subset of the Relevance aspect.

e) Then, the Satisfaction aspect could not be separated in this analysis. As the same tendency had been found in our earlier study in 1999, the Satisfaction aspect is hard to be distinguished from the Confidence aspect by this method of inquiry.

f) It could be speculated that Factor 1"Interestingness" described the aspect of the hot cognition, or affective and emotional side, whereas Factor 2 "Confidence" represented the cold cognition, or objective side of instruction. They may have represented two major views of the students': (1) If the instruction had emotionally appealing, and (2) If the instruction has any value when examined objectively, accordingly.

 

5 Conclusion

16-item course evaluation sheet was developed and its factor structure was analyzed with 209 responses. Many items were categorized as presumed ARCS classes. The data will be used to develop a refined version of the sheet.