This is an HTML rendering of a working paper draft that led to a publication. The publication should always be cited in preference to this draft using the following reference:
© ACM, 2004. This is the author's version of the work. It is posted here by permission of ACM for your personal use. Not for redistribution. The definitive version was published in Computers in Entertainment (CIE), 2(3), ISSN 1544-3574, (July 2004) http://doi.acm.org/10.1145/1027154.1027177
Affective Usability Evaluation for an Interactive Music Television Channel
Athens University of Economics & Business
Department of Management Science and Technology
47 Evelpidon & Lefkados Str., 113 62 Athens, Greece
<![if !supportEmptyParas]> <![endif]>
Computer mediated television
requirements for user interface design and evaluation, since interactive
television applications are deployed in a relaxed domestic setting and aim at
the gratification of entertainment needs. The digital
video recorders, the generation of computer graphics on each digital
and the introduction of new advertising formats are important issues for
research and practice. We explore the employment of an animated character and
the dynamic insertion of advertising in the design of an intuitive user interface
for interactive music video television. We found that the animated character
and the skippable video- track feature
seamlessly enhanced consumer satisfaction, as evidenced by affective usability
H.1.2. [User/Machine Systems]: Human factors. H.5.2 [User Interfaces]: Interaction styles, Prototyping, Screen design, User-centered design, Input devices and strategies. H.5.1 [Multimedia Information Systems]: Video. J.7 [Computers in Other Systems]: Consumer products.
Design, Human Factors.
Interactive television, set-top box, user interface,
affective usability, music video
animated character, TiVo.
Computer mediated entertainment (CME) is emerging as a major
economic force and
is of scientific discourse.
In the past, it was military and space exploration that drove scientific
innovations that later became consumer products, like the PC and the Internet.
More recently, forms of CME (e.g. video games) have been the main drivers of
innovation, which is later applied to areas like scientific visualization,
simulation, and education [ 21 ].
Besides scientific importance, CME has emerged as a major economic factor in the
media industry (cinema, video rentals, music, books, radio, television),
taking-up a large portion of consumer spending and leisure time. In particular,
television, which has traditionally occupied the largest share of consumer
leisure time, is now undergoing a process of technological transformation.
We employed the
Virtual Channel UI
toolkit [ 4 ] in the
implementation of an ITV application that allows the user to ask for
information (e.g. trivia, now playing, coming next, etc.) and to browse through
music video clips. Since traditional
HCI settings involve a task-oriented approach
where the human interacts with an application to accomplish a particular goal , computer-mediated
leisure applications require a fresh view of the current paradigms.
There is growing evidence that traditional desktop usability principles do not
account for the pleasure of the user experience [ 12 ].
Affective usability studies provide an alternative conception of user goals to
consider like enjoyment, fun, and relaxation [ 7 , 9 ].
In the light of the above findings, we employed an affective usability instrument
in the evaluation of the interactive music television prototype.
The rest of this paper is organized as follows. The next
section presents a critical review of
research about ITV applications ,
and in particular, about animated
characters , and video skipping. These
features are employed in music ITV application , which
is evaluated by consumer s ,
as described in section 3. Section 4 presents the results and the
discussion of the usability evaluation test for each one of the ITV
application’s features. In the last section, we draw the
implications of the study’s findings for the media industry.
Previous HCI research about ITV focused
the design of the Electronic Program Guide
(EPG) and did not consider the
enhancement of the TV content. In particular, previous research approached ITV
from a technological perspective [ 23 ] 2 ].
While there have been numerous approaches for ITV
design, few have focused on enhancing a traditional TV program
type with only a standard remote control and a TV screen in a relaxed setting.
We chose to study music video television, which is a commercially successful,
and widely available format of TV content. Music video clips have a large
are very few research works that concern the consumer UI for digital video
libraries, although video
navigation for digitally stored video has been an active research area from the
beginning of the 1990 ’s. The
latter research was
performed from the perspective of the TV author and producer. It is usually
the users of digital video libraries are in front of a desktop computer that
may display tens of thumbnails and video sources at the same time, by employing
a n advanced PC UI
[ 24 ].
These assumptions break down in the living room, in which the user is far away
from a low resolution TV and has remote control. A traditional comparative usability evaluation test
revealed that consumers preferred a UI that was perceived as fun and relaxing
to use, although it had the worst performance in terms of efficiency and
effectiveness [ 10 ].
Therefore, a consumer-level UI for video skipping should be evaluated with
affective usability methods. Furthermore, there is no
that studies digital
video navigation that is in close relationship with a
specific type of TV content, such as music TV.
Animated characters used in desktop computing have been
widely researched, but the respective commercial implementations (most notably
the infamous Microsoft Office Clip) are reported to be annoying to end-users [
3 ][ 22 ].
An explanation might be that the attention grabbing and interrupting nature of
animated characters is inappropriate for productivity computing. On the other
hand, television content has traditionally been about stories and character
development [ 16 ].
Therefore, animated characters might be viable in a domestic environment for
leisure activities, like television watching. In fact, there is evidence that
animated characters are suitable in the entertainment domain ,
because users liked more and
were more engaged with the UI version of an interactive game that displayed a
face to depict the opponent player [ 14 ].
Animated characters have been also applied to
set the right level of expectation and to make errors from the recommendation
engine seem more acceptable [ 8 ].
Yet, there is no affective
usability evaluation of animated characters in comparison with
traditional information presentation for TV.
Leaving aside television content quality questions,
major research questions
about interactive television (ITV) user interface design concern: (a) EPG
navigation, (b) local storage navigation and (c) presentation of related
information with animated character. In this paper, we address
the latter two
issues in an ITV prototype.
We designed and implemented an application that uses local
and allow s
a television viewer to skip a music video clip. Commercial Digital Video
Recorders (DVRs )
offer a 30 second skip button, but content and network providers are reluctant
to adopt a Set-Top Box (STB )
technology that neglects their main revenue source (advertising) and are
interested in alternative forms of television advertising. For example, TiVo is
offering a special space (named ‘TiVo Showcase’), where by
advertisers may store
their advertisement (ad) and users watch
them on-demand. Alternatively, short ads (5 to 10 seconds) or video clip-like
ads might be a choice when users choose to skip recorded content, although a
subscription service may still be ad-free. Thus, we designed track -skipping to
an ad, before the next video clip begins ( Figure 1). Unless the track-skip
button is pressed, there is a continuous flow of music video clips, just like a
normal music video television channel. <![if !vml]><![endif]>
Interactive music video television channel with dynamic advertisement insertion
In addition to the
we also used the dynamic video overlay property of the Virtual Channel
programming library to
superimpose information over the music video. MTV displays
which music video comes next, a feature that we made interactive by allowing
the user to ask for ‘what is playing now and what comes next’ on-demand. The
dynamic insertion of related information to the music video raises the issue of
the presentation style. We implemented and tested two alternative presentation
styles: (a) The traditional MTV information box and (b) an animated character
with a balloon dialog box ( Figure 3).
The central element for our experimental set-up was a
portable PC, running Microsoft Windows XP. We developed the prototype in VB.NET
programming library [ 4 ]. The PC’s ATI
display software was configured in the extended desktop setting (the desktop
area extends beyond the PC monitor to the TV connected through the TV-out port)
and the ITV prototype was set to display on the TV (second monitor). Then, the
PC’s TV -out and audio -out were
connected to the audio-visual inputs of a TV. The
ITV application was designed to run in full-screen and in window-less
mode, so that there is no
visible portion of
the underlying desktop environment. After starting the ITV prototype, we closed
the portable PC’s lid and placed it away from the TV.
<![if !vml]><![endif]> <![if !vml]><![endif]>
Low-budget set -up for interactive television
For supporting relaxed control with a normal TV remote
control, the PC’s serial port was connected to an infrared sensor
(http://www.evation.com/irman/) that receive
signals from the remote control. The sensor’s software driver and the
supporti ve applications were used to map the remote
control’s buttons to specific keyboard buttons. The whole set-up was
unobtrusive and seamless to the television viewer ( Figure 2). It also allowed the experimenter to perform
evaluations in consumers’ homes using their familiar TV and remote control.
Traditional human-computer interaction settings involve a
task-oriented approach where the human interacts with an application to
accomplish a particular goal. Accordingly, usability evaluation techniques
measure successful task completion, efficiency and
error rate parameters that are supposed to correlate positively with user
satisfaction. Most notable among the recent findings about ITV applications is
the realization that users’ subjective satisfaction is at odds with performance
metrics. For example, a usability test of three video skipping interfaces (two
commercial and one novel) revealed that user satisfaction was higher for
interface that required more time, more clicks and had
the highest error rate. In other words, the most usable interface was not the most
preferred interface. Users reasoned
their choice on the basis of how fun and relaxing an interface was [ 10 ].
Therefore, in the context of CME, there is a need to
consider an alternat
e concept ion of user goals. Several concep tions of
affective usability have been proposed: Enjoyment,
fun, trust, engagement, motivation for using [ 9 ],
hedonic quality of a user experience [ 12 ] and
consumer emotions [ 7 ]
are the most relevant. We chose to use Hassenzahl’s
et al [ 11 ]
affective usability measuring instrument, because
it is a validated, freely available, short and easy-to-understand
Furthermore, a fulfilling television experience depends on the subjective
evaluations of the entertaining value of the content [ 16 ],
a characteristic that is partially captured by the construct of hedonic
quality. We used
a seven point semantic differential scale and reversed the polarity of every
other pair: (outstanding-second rate, standard-exclusive,
impressive-nondescript, ordinary-unique, innovative-conservative,
dull-exciting, interesting-boring) .
Scores were summed and then scaled from 0 to 10.
3 The animated character (on the left, using
the Microsoft Genie) and the traditional transparent box (on the right) for
displaying dynamic video overlays
The objective of
this study was to evaluate user
preferences for an ITV application that offers track skipping
for music video clip television and an animated character for presenting
information. The experiment was designed to address two of the main issues
identified for interactive television user interface design: (a) local storage
navigation through simple video track skipping
and (b) presentation of related information through alternative presentation
styles. We were also interested to explore consumer preferences regarding the
dynamic ad insertion, when they choose
to skip a music video clip.
: the track-skipping
user interface , since
users will be able to seek preferred clips and avoid the disliked ones.
: value will be different
between the animated character and the
transparent information box
for the presentation of related information.
Consumers are exposed
to more advertisements in an ITV
application that employs a video track-skipping user interface with dynamic
advertisement insertion than in a
linear TV schedule.
Each participant received two experimental treatments
(within groups) of the user interface for interactive music video television:
1) The animated character and 2) the transparent box, while both of
offered video track skipping
with ad insertion. After the end of each session, participants evaluated
separately the hedonic quality of (a) traditional music video television (all
participants were selected to be frequent viewers of music TV), (b) music video
television with track skipping,
(c) information presentation with the transparent box and (d) information
presentation with the animated character. We ran tests with 21 users (recruited
from the post -graduate and under -graduate
departments of a European business university).
Ages were between 22 and 35 (13 men and 8 women). Users were assigned with a
random order to each treatment and the order of the songs
was randomized for each session. The music-clip
related information and the remote control were the same for all sessions.
The study was performed in a relaxed setting, using a TV
set and a remote control. We used multiple usability engineering methods:
(a) we observed user behavior during the testing session, (b) we kept record
of user actions in log files, (c) users complet
questionnaires and (d) we interviewed each user after the end of the testing
session. In the beginning, the interviews were unstructured and directed by the
users. Gradually, the interviews became more focused to
repeating issues ,
mentioned during the interviews or observed during the testing session.
The testing session contained 16 video-clips and an
advertising break with three ads every 4 songs (approximately every 15
minutes), just like a commercial music video television channel. The study was
in accordance with the selective-exposure paradigm
[ 25 ] . Users were
free to choose the music video clip they preferred to watch, like they would do
if they were outside the experiment. In order to ensure selective-exposure the
users were allowed a maximum of 1/3 of watching time, out of the total session
duration [ 12 ],
that is a maximum of approximately 20 minutes out of the 1h program duration.
Users could press the power-off button on the remote to end the testing session
and they were told to watch as much as they liked, between 10 and 20 minutes.
Users who were not involved in computer research and
development asked how video skipping was possible and whether
was a commercially available product or television station. Most of the
other users were aware of an experimental system behind the TV program, but
when asked whether they understood that there was actually a PC running the
system, users said that it looked and felt like normal TV.
The use of a normal TV and a remote control contributed to these positive
evaluations, but they can also be attributed to the employment of the Virtual Channel
UI development toolkit that delivers a familiar television experience [ 4 ].
Therefore, the combination of the Virtual Channel programming library and an
appropriate experimental set-up may be used to create high-fidelity ITV
prototypes. We had
predicted that there would be a difference in the hedonic quality value between
a traditional music video television channel and one that features track
of music videos. We found ( Table 1) that the hedonic quality score (scale is from 0 to
10; scores less/more than 5 represent negative/positive attitude) for the
traditional one is close to neutral (average 5.1/10) , which is quite expected, since music video television is a
pervasive experience and feels familiar to consumers. In contrast, track-skipping
(average 7.5/10) allowed consumers to watch their preferred music video clips and despite the
dynamic insertion of ads the hedonic quality score was significantly higher
(two tailed t–test, p=0.002, n=21). Therefore, it is argued that
simple audio CD-like track skipping enhances the
perceived television entertainment value, when compared with same fixed TV
Hedonic Quality (p=0.002, n=21)
Mean hedonic quality scores for the track-skipping
music video television are significantly higher
Consistent with the selective exposure theory, users
actively sought for the video clips and songs they preferred. This kind of
interactive behavior may be due to the experimental setting and may not have
external validity; users may have been more engaged than normal because the
to them and because they were specifically asked to use the new system. They
reported that they used the skip functionality mainly to skip a music video
that they disliked and to a lesser extent to get to a favorite one. Either way,
feature was a favorite, despite the ad insertion, and provided relaxed control
of the interactive music TV application, based on the dynamic video synthesis
of music clips. We had
also predicted that there would be a difference in the hedonic quality value
between the animated character and the transparent box for the related
information presentation style. We found that the hedonic quality
(scale is from 0 to 10; scores less/more than 5 represent negative/positive
attitude) for a music video television channel is significantly higher (two
tailed t–test, p=0.0002, n=21) when using an animated character (average
7.0/10) for presenting dynamic video overlays compared with the traditional transparent information box (average
4.4/10). Again, consumers were neutral toward the traditional information box, since it is
a widely used and familiar presentation style for information related to music
video clips ( Table 2). Therefore, we argue that the animated character
could be used to enhance the consumers’ entertainment experience with
Hedonic Quality (p=0.0002, n=21)
Mean hedonic quality scores for the animated character compared to the
traditional overlay box
Those who have been exposed before to the Office Assistant
(through the Microsoft Office
suite of application s)
recognized the similarity (due to the balloon-style dialog, which is standard
for the characters developed with the Microsoft Agent system) despite the use
of a different character (the genie). A few of these
users reacted negatively to the concept of the animated character. Therefore,
the animated character may have a carry-over effect from the desktop to the ITV
environment. Those users ,
who are already negatively predisposed to it ,
will continue to be so, at the expense of their satisfaction with the whole TV
program. For those users, we suggest that the animated character should be an
option and an alternative UI should be available to select. Nevertheless, most
of the users considered the character funny and less obtrusive compared with
human presenters, who interrupt the flow of the video clips to present related
information. Furthermore, users proposed that there should be a variety of
animated characters to select from (e.g. cartoons, heroes, personalities).
Users also asked for more control of the character, like changing its placement
on the screen. Finally, most of the users reported that the solid balloon
dialog that stands over the head of the character hides a considerable part of
the TV screen. According to their suggestions, the best form for the balloon
dialog would be a transparent one across the bottom of the screen.
We were interested in finding out how many ads users would
be willing to watch, as a consequence of using the music video clip skip
feature. In a traditional music video TV channel the proportion is
approximately 12 ads for each hour of TV video [Executive of a European music
television channel, personal communication]. The findings regarding the dynamic
advertisement insertion were very positive, despite the fact that advertising
is one of the most controversial features of commercial
During the interviews and the observation sessions, it was found
that all the co n sumers
were positive to the dynamic
advertisement insertion. Log file analysis revealed
that consumers actually
watched the double number of advertising messages compared with a normal
broadcast session. Interestingly, according to the log files, some users also
tried to skip over the ads, with no effect since the application was programmed
to ignore the video skip when inside an ad.
the trade-off between skipping to the next video clip and watching a short
advertisement positively. Yet, it is not clear which is the cause of the above
result, since the system employed two innovative features that affect the
number of advertising messages displayed. The dynamic advertisement insertion
every time the user skipped a video clip increases the number of messages
shown. At the same time, leaving aside
for a moment the
dynamic advertisement insertion, the video clip skipping action brings
the user closer to the next scheduled advertising break, thus increasing the
number of advertising messages displayed. Therefore, in order to draw cause and
effect conclusions between the video skipping feature and the number of
advertisements watched, a further study
is needed that will compare
between having dynamic advertisement insertion and not having,
for the same video-clip skipping UI.
Music video clips have a number of unique characteristics
that facilitate further ITV development. Music content can be easily classified
and filtered by employing open Internet databases and
classification schemes and adaptation models that have been
developed for the popular MP3 music format [ 18 ].
Additional metadata that describe the emotional content (e.g. from
http://www.moodlogic.com) may also be used and combined with research about
emotion in animated characters . The latter research issue merits further research
by the respective community, which has already put considerable effort in the
investigation of animated characters for consumer electronics [ 8 ].
The most interesting suggestions for future improvements
ed the augmentation of the music video
clip skip feature. Users familiar with CD players and the PC-based MP3 music
players asked for more options when skipping a music video, like repeat the
same song, play a song from the same artist or play a new song within same music genre.
Moreover, information about a longer list of the upcoming music videos would be
and it would also allow organiz ing their
since they could leave the TV on and plan to return back when their favorite
song is on.
Using the television as a time tool to structure household activities and
organize time has been documented before in an ethnographic study of an STB
trial [ 16 ].
Providing on-demand information about the upcoming video clips supports the
relaxed control of TV as a time management tool, while the ability to alter
dynamically the upcoming play-list would further support
interactive behavior. For example, the user could bring up a play-list of 10
upcoming music videos and alter it dynamically along a number of parameters
like genre , artist or
automatically create play-lists [ 18 ].
The user could also decide whether to skip directly to a music video by
pressing the corresponding button on the numeric keypad. Despite
the alternative usability evaluation methods (qualitative,
quantitative) that were used in
the course of the present study,
all of them were
employed during the limited duration span of
a n experiment, instead of spanning
through everyday living in
consumers’ homes. Previous findings regarding the
usability of everyday technology demonstrate that the consumers’ perceptions
and especially the mental models they form about new domestic technologies are
very elastic and prone to change in the
passage of time [ 19 ].
Therefore, a longitudinal study would have provided additional information
about the research
issues. A longitudinal study should employ an appropriate research framework, like
the experience sampling method [ 15 ].
In the context of ITV prototyping, adding a TV tuner would extend the
functionality to include synchronization between broadcast and local storage
and to perform more realistic TV experiments in consumers’ homes,
over longer periods of time.
In the context of the commercial implementation of the
video-skipping feature, a major open research question is who is going to
control the rules for the dynamic synthesis of video and for the dynamic
For example, on the one hand, the media industry may choose to subsidize DVR
STBs to consumers, in exchange of
having increased control of what is stored and how it is played (e.g.
targeted advertising and subscription services for personalized music/news
channels for each STB). On the other hand, the consumer electronics industry
may offer advanced general purpose DVRs with DVD- Recording
functionality and Internet connectivity for downloading meta-data and related
information. The latter are likely to be
more complex to use, but will offer increased consumer control, while the
former will be introduced by established media brands and broadcasting
services. Finally, between
these two extremes there is a continuum of alternative product and service
We would like to thank the university students who participated to the experiment and Ronald Boring for providing valuable comments on the research and for reading through early paper drafts. Parts of this work were supported by the CONTESSA (IST-2000-28567) and MUSICAL (EDC-22131) projects, partially funded by the European Commission.
1.<![endif]> J. Bates.
The role of emotion in believable agents. Communications of the ACM ,
1.<![endif]> J. Carey. Content
and services for the new digital tv environment.
In D. Gerbarg, editor, The
Economics, Technology and Content of Digital TV , pages 88–102. Kluwer
Academic Publishers, 1999.
1.<![endif]> R. Catrambone,
J. Stasko, and J. Xiao. Anthropomorphic agents as a user interface
paradigm: Experimental findings and
a framework for research. In Proceedings of CogSci 2002 ,
pages 166–171, 2002.
1.<![endif]> K. Chorianopoulos
and D. Spinellis. User interface development for interactive television:
Extending a commercial DTV platform to the virtual channel API. Computers and Graphics ,
1.<![endif]> J. Clark.
A telecomputer. In Proceedings
of the 19th annual conference on Computer graphics and interactive techniques ,
pages 19–23. ACM Press, 1992.
1.<![endif]> D. Crockford.
Integrating computers and television. In B. Laurel, editor, Art of Human-Computer
Interface Design , pages 461–466. Addison-Wesley, 1990.
1.<![endif]> P. M. Desmet.
Measuring emotions: Development and application of an instrument to
measure emotional responses to
products. In M. Blythe, A. Monk, K. Overbeeke, and
P. Wright, editors, Funology: from usability to enjoyment .
Kluwer Academic Publishers, 2003.
1.<![endif]> E. M. A.
Diederiks. Buddies in a box: animated characters in consumer electronics. In Proceedings of the
2003 international conference on Intelligent user interfaces , pages 34–38. ACM Press,
1.<![endif]> S. W. Draper.
Analysing fun as a candidate software requirement. Personal and
Ubiquitous Computing , 3(3), 1999.
1.<![endif]> S. M. Drucker,
A. Glatzer, S. D. Mar, and C. Wong. Smartskip: consumer level
browsing and skipping of digital video content. In Proceedings of the
SIGCHI conference on Human factors in computing systems , pages 219–226. ACM
1.<![endif]> M. Hassenzahl,
A. Beu, and M. Burmester. Engineering joy. IEEE Software ,
18(1):70–76, January/February 2001.
1.<![endif]> M. Hassenzahl,
A. Platz, M. Burmester, and K. Lehner. Hedonic and ergonomic
quality aspects determine a software's appeal. In Proceedings of the
SIGCHI conference on Human factors in computing systems , pages 201–208. ACM
1.<![endif]> S. Knobloch
and D. Zillmann. Mood management via the digital jukebox. Journal of
Communication , 52(2):351–366, 2002.
T. and Maes, P. Agents with faces: The effects of personification of agents. In Proceedings of
HCI'96 , London, 1996.
R. and Csikszentmihalyi, M. Television
and the Quality of Life: How Viewing Shapes Everyday Experiences .
Lawrence Erlbaum, 1990.
B. and Lee, R. S. How and why people watch tv: Implications for the future of
interactive television. Journal
of Advertising Research , 35(6):9–18, 1995.
1.<![endif]> J. O'Brien,
T. Rodden, M. Rouncefield, and J. Hughes. At home with the
technology: an ethnographic study of a set-top-box trial. ACM Transactions on
Computer-Human Interaction (TOCHI) , 6(3):282– 308, 1999.
1.<![endif]> F. Pachet.
Content management for electronic music distribution. Communications of the
ACM , 46(4):71–75, 2003.
1.<![endif]> M. G. Petersen,
K. H. Madsen, and A. Kjaer. The usability of everyday technology:
emerging and fading opportunities. ACM Transactions on Computer-Human Interaction
(TOCHI) , 9(2):74– 105, 2002.
1.<![endif]> L. Press.
Personal computing: compuvision or teleputer. Communications of the ACM ,
1.<![endif]> A. Rosenbloom.
Special section: A game experience in every application. Communications of the
ACM , 46(7), 2003.
1.<![endif]> H. Schaumburg.
Computers as tools or as social actors? - the users' perspective on anthropomorphic agents. International Journal
of Cooperative Information Systems , 10(1,2):217–234, 2001.
1.<![endif]> B. Smyth and
P. Cotter. A personalized television listings service. Communications of the
ACM , 43(8):107–111, 2000.
H. D., Christel, M. G., Gong, Y., and Hauptmann, A. G. Lessons learned from the
creation and deployment of a terabyte digital video library. IEEE
Computer , 32(2):66–73, 1999.
1.<![endif]> D. Zillmann and
J. Bryant, editors. Selective
exposure to communication . Hillsdale, NJ:
Lawrence Erlbaum Associates, 1985.
<![if !supportEmptyParas]> <![endif]>