Paper to be presented at 1993 Annual Meeting of the Australian
Association for Research in Education, November 22-25, Fremantle, WA
A case study of motivational design:
Multi-media courseware
"The Secret of Aunt Mariko"
Katsuaki SUZUKI
Tohoku Gakuin University, Sendai JAPAN
Masaru SAKAYAUCHI
National Institute for Educational Research, Tokyo JAPAN
Kanji AKAHORI
Tokyo Institute of Technology, JAPAN
Abstract
This paper reports on a project to develop a laser-disc based
multimedia CAI courseware on English listening comprehension skill
using a role-playing game format. It was a three-year project by RISE,
an affiliate of the International Trade and Industry Ministry of Japan.
The product is first briefly described in terms of its hardware,
software, and courseware. Then theoretical backgrounds, especially that
of motivational aspect of the courseware, will be introduced, using the
ARCS motivational design model. Finally the developmental process and
research activities, which were involved in the project, will be
summarized.
THE PROJECT
The aim of this project was to build a shared chunk of know-how
knowledge in utilizing multimedia technology to be applied in
educational material development. It was sponsored by the Mechanical
Social Systems Foundations, an affiliate of the International Trade and
Industry Ministry of Japan, in order to form a project-based research
and development team in the Foundation of Research Institute of
Software Engineering (RISE). RISE was granted roughly 100 million yen,
or 1.25 million Australian dollars, to conduct this three-year project
including prototype development (1st year), stand-alone courseware
development (2nd year), and networking (3rd year). This paper deals
mainly with the project's second year, when stand-alone courseware was
designed, developed, and pilot-tested.
Project members were from diverse sectors, ranging from hardware
manufacturers and an entertainment software house, to a public
broadcast system and university researchers. The project was chaired by
the third author of this paper, and the members were divided into five
research teams. The teams were Theory and Display Design Team (to where
the authors of this paper belong), Storyboard Team (consisted of
English Educators), Development Team (software house) , Evaluation Team
I (researchers to plan and conduct evaluation studies), and Evaluation
Team II (hardware manufacturers to assist conducting evaluation). Each
team worked individually and cooperatively throughout the project years.
Based on the basic research concerning current research trends and
technology available to develop a prototype of "new" courseware, the
project decided that the goal of this effort was to make the product
"appealing". By finding out why TV games are so popular and by
translating the reasons into educational product, the project attempted
to seek a way to keep motivation to learn to be continued. The basic
elements to be incorporated into the final product were defined as
interactive multimedia presentations, a role-playing game format, and
an affective control mechanism.
THE PRODUCT
Hardware configurations
The product "The Secret of Aunt Mariko" runs on a Macintosh (4.5MB RAM)
with a video digitizing board (e.g., RasterOps 24STV) to show
full-color motion pictures from a laser disc. A laser disc player
(e.g., Pioneer LD-V800) is to be controlled by the Macintosh using an
RS-232 cable. A pair of speakers is also required to play sounds
directly from the laser disc player. With the hardware described above,
full-motion video can be merged with texts and graphics created by the
computer and displayed on a single color monitor screen. A mouse is
used when a user makes choices.
Multimedia nature of the software
The product consists of multimedia data: live video and stereo sounds
on the laser disk; digitized video and sounds in the QuickTime format,
digitized sound narrations both in Japanese and in English, text data
in both languages, computer graphics and still pictures. About 200 cuts
of live video were filmed at University of California, Riverside, by an
executive director of NHK (Japan Broadcasting Corporation, the public
TV station), then stored on a single side of a 60 minute laser disc.
All other data are stored on a hard disk, for which a total of 62.5MB
are required.
---
Figure 1. Insert about here
---
An authoring software for multimedia presentations, Macro Mind
Director, is used to control all the data mentioned above, in
accordance with the flow of the program and with the choices of the
user. The program to run this software looks just like the script for
an ordinal CAI courseware; the difference is that not only texts and
illustrations are to be selected and displayed by the program, but also
other forms of multimedia chunks of data, such as clips of full-motion
picture, are accessed and presented to the user.
Courseware
Using the hardware and software described above, the core of this
project is to plan and produce courseware, by which the contents and
flows of the study materials are represented. Courseware follows a
grand story to investigate the secret of Aunt Mariko by finding out the
location a photo was taken. The old photo shows young Aunt Mariko
standing by a river, who was studying in the United States. Depending
upon the correctness of choices in the adventure, the user will see
different endings (solve or falsely solve the mystery) or will not be
able to see the ending at all and be forced out in the midst of the
adventure (timeout).
In the quest for the secret of aunt Mariko, the user of the program has
to listen to English conversations, in order to select a better answer
in return. The choices consist of either 'guesses' of what was heard,
or 'actions' to advance the situation. By doing so, the user is
expected to develop his or her listening comprehension skill, which is
the educational objective of this courseware in the cognitive sense.
For the choices based on a misunderstanding of what a character of the
scene said, the program does not answer by saying "No, the correct
answer is such and such". However, the program either branches to an
additional happening and branches back to the original scene, or asks
the user to try to listen again by re-presenting the question slower.
This brings the user repetitive opportunities to listen to the same
questions, with a time penalty. For those who often fail to make best
choices, they will run out of time to complete required quests. Then
they would be forced out from the program and must start over again
from the beginning, to get repetitive chances of listening.
The second educational objective is to prevent the user from becoming
bored of or feeling forced into the repetitive training of English
conversation. This objective deals with the affective aspect of
learning, especially with motivational concerns. It is expected that by
making an English student to amateur investigator, the perceived goal
of the courseware would become not to study English but to have fun of
solving the Mariko's secret. Any improvement of listening comprehension
skill would then become almost a byproduct of having fun with this
"entertainment", where boredom is no longer the issue.
In order to come closer to the mystery of the aunt Mariko, the user of
the courseware take a role of our hero Ken'ichi, who lives in a
dormitory of an American university and is asked by his uncle (Mariko's
husband and his financial supporter) to find out the secret in 5 days.
Each day is filled up with an adventure an American college student's
everyday life, such as a surprise party, a date with Sari, and making a
phone call to reserve a dinner party at a Chinese restaurant. Depending
on how well the user performed on the previous day, a chance to
challenge for a clue to the mystery will become available, such as
searching the aunt's name at the alumni association and doing an
interview with whom Mariko may have been acquainted. On the final day,
Ken'ichi will be told by his uncle the real reason of his asking
Ken'ichi to investigate the secret, the reason being different
depending on the total score of the quest, or he would be cut out the
financial support if the performance were too low.
MOTIVATIONAL DESIGN
The ARCS motivational design model
The presenter (first author) of this paper was brought into the project
because of his background in motivational design, particularly the use
of the ARCS model originated by Keller (1983; 1984). The ARCS model of
motivational design suggests four basic categories of motivational
strategies: gaining and sustaining ATTENTION, establishing perceived
RELEVANCE of instruction, building CONFIDENCE in succeeding the given
task, and providing with SATISFACTION for the consequences of the
effort put into the learning. The name of the ARCS represents these
four basic categories.
Numerous strategies have been suggested in various instructional
settings such as classroom activity planning, message design, textbook
enhancement, and computer courseware design (Keller, 1984; Keller &
Burkman, 1993; Keller & Kopp, 1987; Keller & Suzuki, 1988). The
model has become well known due to its applicability to any forms of
instruction, and its comprehensive structure addressing psychological
research literature (Newby, 1991).
Motivational Analysis of the Product
The characteristics of the product "The secret of Aunt Mariko" have
been analyzed according to the four basic categories of the ARCS model,
as shown in Table 1. This summary is based on the comments from the
originator of the ARCS model in September 1992 to a prototype product,
with some additions from an analysis of the final product in February
1993 by the Theory Team of the project.
---
Table 1. Insert about here
---
Process of Motivational Design
It is apparent in Table 1 that many motivational strategies can be
identified in all of the ARCS four categories. However, as Keller
suggests, it is not the best way of motivational enhancement to adopt
as many strategies as possible. In stead, inclusion of the strategies
should be selective, in order to fit the motivational characteristics
brought by the learner, the task, and the learning environment. The
notion of systematic motivational design comes in to play an important
role in this regard.
In the course of the systematic process of defining, designing,
developing, and piloting an instructional material, Keller (1987)
suggests 12 phases or activities of motivational design. Among these
there are the three important activities; that is, audience analysis,
selective adoption of motivational strategies, and formative evaluation
and revision of the material.
Audience analysis refers to making a learner profile as to how
appealing the instruction would be, based on the ARCS four categories.
This provides a basis for the selective adoption of motivational
strategies, where relative emphasis of motivational enhancement is
judged to match learners' profile. For example, if the learners are
expected to be anxiously waiting for the instruction, then strategies
for getting Attention will be omitted because they would be
unnecessary. If, on the other hand, the learners are forced to be
"captive audience", then Attention strategies will be necessary to
prevent the learners from thinking something else in the air.
The third of the important activities is formative evaluation and
revision of instructional materials using empirical data. It should be
noted that the summary of motivational characteristics shown in Table 1
is merely a theoretical observation, which suggests possible
motivational effects of the product. Actual effects can only be
examined with empirical data from courseware evaluation, as shown later
in this paper.
Despite of the recommended procedures of motivational design described
above, the motivational design in this project came into a play almost
post hoc, trying to interpret the ideas formulated by the project's
Development Team with expertise and experiences of producing
commercially well-known TV games. Table 2 shows some observations of
the design and development procedures of the project in terms of
motivation design.
---
Table 2. Insert about here
---
The notion of motivation design and the ARCS model's categories and
strategies did help clarifying the goal of the project and various
means to that end, it was how far the contributions from this Theory
Team could go in this particular case. In other words, the role of
theoretical notions was more descriptive than prescriptive, not
suggesting the best of the alternatives from the theory, but trying to
explain, using the theory, why the experts took certain actions .
EVALUATION RESEARCH
In order to see if the aims of the project were met or not, several
evaluation methods were examined and conducted, mainly by the
Evaluation Teams of the project. The evaluation methods included (1)
questionnaires administered prior to and after the evaluation session
to the participants from the target audience, (2) a protocol analysis
to observe and analyze the process of using the courseware, and (3) a
game-design expert interview to try to clarify the rules of judgments
in the course of game design, applied to the development of this
courseware.
Despite of a review of prior research studies, the project group did
not feel comfortable as to how to go about conducting the evaluation.
It should be noted that the methods of evaluation per se were to be
examined their validity through the usage in this study. Brief
descriptions of the methodologies and the results are as follows:
Questionnaire
Pre and post session questionnaires were administered to all of the 39
participants (mostly college students) of the evaluation study.
Pre-session questionnaire asked participants' biographical data,
experiences of learning on computer, characteristics of the most
memorable computer courseware using 17 5-point Likert scale items, and
TV game experiences and impression of the most memorable TV game, with
18 Likert items. Post-session questionnaire asked English learning
experiences, and reactions, opinions and feelings about the courseware
"The Secret of Aunt Mariko", using a total of 81 Likert items.
Among the items concerning the English learning aspects of this
courseware, two items on participants' subjective judgment of
improvement of listening comprehension (both the contents and the words
of narrations and questions) were rated higher than other items (such
as remembering dialog patterns). A comparison between game aspects of
this courseware and impression of the most memorable TV game revealed
that both were rated similarly, suggesting that this courseware was
considered to be one of the best "games" they ever played.
Many of the participants expressed their continuing motivation to spend
more time on this courseware, to recommend this courseware to their
friends, and to wish if it could be used in their schools. This verbal
commitment was supported by frequent observations of voluntarily
choosing to remain using the courseware after the evaluation session
when they were told that they could do so.
Protocol analysis
Protocol analysis, or discourse analysis, refers to recording and
categorizing learners' conversation while studying courseware (Yoden,
1991). In the process of the formative evaluation of this courseware,
some of the participants were asked to use the courseware in a pair, by
orally exchanging feelings and ideas before making any choices. The
evaluation process was videotaped, which was to be analyzed into
sentences and classified using predetermined utterance categories. The
amounts of utterance in each category were compared with each other to
see the kinds of mental activities frequently occurred during the
learning process.
The results from an analysis of two pairs of participants indicated
that (1) the operation of the courseware was clearly understood, which
was judged by few conversation concerning how to proceed to the next
screen; (2) the courseware was affectively stimulating, judged by
frequent utterance in affective nature, such as curiosity, joyfulness,
and surprises; (3) the courseware required the users to remember the
contextual information to make choices, and (4) the courseware required
them to concentrate on what the characters said in English, also judged
by frequent conversations making sure contextual information and
repeating questions in English for themselves.
This method of evaluation was adopted because it captures the process
of learning, unlike the other forms of tests or questionnaires
collecting information at either the entrance or the exit of leaning
activities. It brought useful information in a quantitative form as to
how the participants reacted to the courseware. It is noted, however,
that the time consuming processes of analyzing all the utterance for
over one-hour sessions and then manually categorizing them must be
shortened without loosing validity of the data. It should also be noted
that possibility of affecting users motivation by employing this
evaluation method (i.e., to study in a pair being directed to overtly
respond) deserves an attention, especially when enhancing motivation is
the goal of the project .
Expert interview
The Theory Team conducted an interview with the chief engineer of
Development Team, to try to extract his expertise in game design.
Because it was mostly due to his expertise if the product was
interesting and appealing, and because the theory could not play a
prescriptive role in this project, it seemed a natural step to conduct
an interview so as his decisions to be stated in as an objective form
as possible. Borrowing from the method of structured interviews in the
field of knowledge engineering, a three-hour interview was conducted
using predetermined questions and free discussion with an access to the
courseware. A part of the results from the interview is shown in Table
3.
---
Table 3. Insert about here
---
CONCLUSIONS
This paper described a case study of motivational design applied to a
development project of multimedia courseware, "The Secret of Aunt
Mariko". The usefulness of the ARCS model and motivational design was
demonstrated, at least in a descriptive sense. It helped to categorize
motivational attributes of the courseware, which were originated from a
game designer's rules of thumb, unconsciously from his own experiences
in game design. The next step of project's theoretical concern is to
narrow the focus of motivational enhancement, in order to position the
future product in light of the ARCS four categories.
Although some innovative evaluation methodologies provided indications
supporting project's success, further investigations are in order, as
to how to improve evaluation process per se. We still feel uncertain
how much or how little our success truly was, as well as the reason
why.
As Keller (1987) has pointed out about a systematic motivational
design, we should "not expect it to be completely mechanical or
algorithmic; it still requires judgment and benefits from experience,
intuition and creativity (p. 1-2)." It is apparent from this study that
we can learn many things from joint projects with entertainment
experts, if we are to make our own instruction more appealing. It can
only be said at this point that investigations in motivational design
are in order in various settings of educational research.
References
Keller, J. M. (1983). Motivational design of instruction. In C. M. Reigeluth (Ed.), Instructional-design theories and models: An overview of their current status. Lawrence Erlbaum Associates, U.S.A.
Keller, J. M. (1984). The user of the ARCS model of motivation in teacher training. In K. E. Shaw (Ed.), Aspects of educational technology volume XVII: Staff development and career updating. Kogan Page, England.
Keller, J. M. (1987). The systematic process of motivational design. Performance & Instruction, 26 (9), 1 - 8.
Keller, J. M., & Burkman, E. (1993). Motivation (Chapter 1). In M. Fleming, & W. H. Levie (Eds.), Instructional message design (2nd Ed.). Educational Technology Publications, U.S.A.
Keller, J. M., & Kopp, T. (1987). Application of the ARCS model of motivational design. In C. M. Reigeluth (Ed.), Instructional theories in action: Lessons illustrating selected theories and models. Lawrence Erlbaum Associates, U.S.A.
Keller, J. M. & Suzuki, K. (1987). Use of the ARCS motivation model
in courseware design (Chapter 16). In D. H. Jonassen (Ed.), Instructional designs for microcomputer courseware. Lawrence Erlbaum Associates, U.S.A.
Newby, T. J. (1991). Classroom motivation: Strategies of first-year teachers. Journal of Educational Psychology, 83 (2), 195-200.
Research Institute of Software Engineering (1993). Feasibility study report on the development of new computer assisted educational system. Report #4-F-9, Association for Promotion of Machine Systems, Japan.
Turn detective and learn English with new software. Daily Yomiuri, 1993.6.21., Japan.
Yoden, Y. (1991). Discourse analysis: An alternative approach to the
evaluation of educational software. A paper presented at the 3rd Joint
Conference on Educational Technology, Proceedings, 277-278, Japan.
Table 1. Motivational Analysis of "The Secret of Aunt Mariko"
<<ATTENTION>>
(+) Novelty effect by utilizing multimedia to realize a previously unencountered learning environment
(+) Sustaining attention by presenting problems frequently
(+) Curiosity aroused by the mystery of the grand story
(+) Variability by unanticipated happening such as being called on by pizza delivery
(+) Avoiding boredom by dividing a long grand story into daily substories
(-) Technological complexity to be unrecognizable so that it can be
seen as "Nothing new" or "TV without sharpness" for the users
<<RELEVANCE>>
(+) Familiarity by concrete scenes with on site filming
(+) Instrumentality by being able to listen to English conversation that leads to uncover the mystery
(+) Match to the interest of potential users by showing everyday life of studying abroad
(-) Preventing from enjoying the study process by a lengthy waiting time for accessing visuals from the videodisc
<<CONFIDENCE>>
(+) Avoiding to remember failure experiences of studying English by making the learning aspect invisible
(+) Fear exclusion by providing multiple chances after a failure
(+) Anxiety reduction by offering detailed orientation on how to play the game
(+) Personal control by letting a low score user choose either to give
up the quest, or to proceed with knowing that the possibility of
getting murdered becomes greater
(+) Personal control of pacing by frequent interactions without long presentations of learning materials
(-) Limited personal control with not being able to decide the flow and consequences of the story
(-) No direct mechanism provided to inform user of the goal or improvement relating his/her listening comprehension skill
<<SATISFACTION>>
(+) Immediate reinforcement for the responses by providing the result at once
(+) Reward mechanism by solving mystery
(+) Multiple consequences depending upon the user's performance by having different endings for different scores
Table 2. Some observations of motivational design process of this project
<<AUDIENCE ANALYSIS>>
(+) Novelty effect would enhance Attention.
(-) Because this project is innovative, it is difficult to anticipate
users' characteristics as to which of four aspects motivates potential
users.
<<SELECTIVE ADOPTION OF MOTIVATIONAL STRATEGIES>>
(+) The story with many happenings would be effective focusing on sustaining Attention.
(-) Due to difficulty of defining users' characteristics and due to the
fact that this project focuses on enhancing motivational
characteristics of the product, it is hard to avoid overdose of
motivational strategies.
(-) It is difficult to specify which of the four categories is selectively emphasized.
<<FORMATIVE EVALUATION AND REVISION>>
(+) Revisions have been made based on the inputs from within the project.
(+) Empirical data have been collected to make future revisions of the product.
(-) It is hard to judge the completeness of the final product due to the lack of existing courseware of this sort.
(-) It is possible to affect users' motivation not only by the
courseware itself, but also by the characteristics of the setting of
evaluation study.
Table 3. Rules of expert judgments on game design
<<END PRODUCT>>
(Screen Design)
#Use a simple display design with distinctive features stood up.
#Emphasize the critical factor of game advancement.
(Stories)
#Vary substories in difficulty levels, suspension factors, content seriousness, and mix them to create a rhythm.
#Show users the goal, but have deep process toward the goal.
(Rules)
#Have the user an experience of game over or they don't understand the critical rules of the game.
#Do not expect the user to have navigation skills in interactive environment. Train them at the outset or explain how to.
<<DEVELOPMENT PROCESS>>
(Planning)
#Provide user with either regularity as in series, or novelty.
#Differentiate from other products and emphasize the difference.
(Prototyping)
#Build a concrete prototype for multimedia products to facilitate communications among members with various backgrounds
#Make a visual design first to provoke images, then do interactions.
(Development)
#Use/Develop a tool to link multimedia data to computer program.
#Adjust the balance among multimedia parts, after linking them.
(Evaluation)
#Adjust the game difficulty level thoroughly during the field test.
#Check
if users are dipped into the world of game, by weaving out
contradictions, unnatural assumptions, unfriendly interface, etc.