Initial User Evaluation of Five Preview Surrogates

 

1.      Project Description.

(a)   Purpose, hypotheses, or research questions.  Five surrogates for previewing videos retrieved from a video collection have been prototyped (see Appendix A for a description of each).  Prior to their further development, it is essential to learn which will be most effective for their intended audience.  The purpose of this study is to collect information about each surrogate’s strengths and weaknesses for several purposes related to the browsing of video collections.  In this study, users’ perspectives on storyboards, storyboards augmented with spoken keywords, slide shows, slide shows augmented with spoken keywords, and fast-forwards will be collected.  The ultimate goal is to identify which of these surrogates warrants further development.

(b)   Procedures.  A small number of potential users of video collections will be recruited for participation.  Each study session will be conducted in three phases. 

In the first phase, the study participant will be asked to interact with the full video and all five surrogates for two video segments:  “Apollo, Segment 4006” (2:07 minutes) and “Chevrolet Promotional Videos: Master Hands, Segment 1” (4:54 minutes).  First, the participant will view the full video segment; then they will view the surrogates of that video. For half of the participants (randomly selected), the five surrogates will be presented in the following order:  slide show with visual keywords, slide show with audio keywords, storyboard with visual keywords, storyboard with audio keywords, fast forward; for the other half of the participants, the order of the surrogates will be:  storyboard with visual keywords, storyboard with audio keywords, slide show with visual keywords, slide show with audio keywords, fast forward.  As they work with each video and its surrogates, the participants will be interviewed using the interview guide in Appendix B.

In the second phase, the participants will be asked to interact with all five surrogates (but not the full video), for two additional video segments:  “Challenge at Glen Canyon, BOR03, Segment 2” (3:00 minutes) and “Educational Films: A Date with Your Family” (5:00 minutes).  The order of the surrogates will be counter-balanced, just as in phase one. The same interview schedule will be used.

In the third phase, the participant will be asked to complete several assigned tasks (see Appendix C) while interacting with one of the surrogates.  The participant will be able to use whichever surrogate he or she prefers to perform the tasks.  The participant may choose a different surrogate for the tasks for each of the three videos included in this phase.  The three videos are:  “Moon, Segment 2” (3:43 minutes), “Hurricanes, Segment 1” (3:54 minutes), and “New Indians, Segment 101” (2:11 minutes).  Participants will be asked to think aloud while completing the assigned tasks.

All phases of the study will be videotaped, using the usability workstation in the Interaction Design Lab (face shot, keyboard/mouse shot, and screen capture combined on the same videotape).

 


2.      Participants.

(a)   Age, sex, and approximate number.  Five to ten participants will be recruited for this initial test.  They will be recruited from among students, faculty and staff at UNC.  Personal invitations will be extended to faculty and staff identified as having an interest in video collections.  In addition, students in some of the following classes will be invited to participate (with the instructor’s permission): 

·         COMM 014, Introduction to Media Production;

·         COMM 034, Audio, Video, and Film Production and Writing;

·         COMM 135, Documentary Production;

·         COMM 139, Special Topics in Media Production:  Experimental Video;

·         COMM 146, History of Film I;

·         COMM 252, Studies in Film and TV;

·         COMM 356, Seminar in Communications Technology;

·         JOMC 120, Introduction to Video Production and Editing;

·         JOMC 121, Electronic Journalism;

·         JOMC 188, Cybercast and Publishing; or

·         JOMC 189, Multimedia Design and Production.

Care will be taken to include both genders and people with a broad range of ages (over 18).

(b)   Inclusion/exclusion criteria.  No children will be participating in this study.  No other exclusion or inclusion criteria will be applied.

(c)    Method of recruiting.  Members of the research team will personally invite appropriate faculty and staff to participate in the testing sessions.  The invitation will stress that participation is completely voluntary.

A flyer inviting student participation will be distributed in some or all of the classes listed in 2a.  A draft flyer is included in Appendix D.  Student participation is completely voluntary and outside the scope of the class requirements; the names of students participating will not be reported to the class instructors.  The classes are being used for recruitment only because the students in them have an interest in videos and video production and so are representative of the target audience for the system incorporating the video surrogates.

(d)   Inducement of participation.  Participants will be offered $25 for their participation.  Each session is expected to last approximately 2 hours.

 

3.      Are participants at risk?  There is no risk for the participants.

 

4.      Describe steps to minimize risk.  Not applicable.

 

5.      Are illegal activities involved?  No illegal activities are involved.

 

6.      Is deception involved?  No deception is involved.

 

7.      What are the anticipated benefits to participants and/or society?  The only concrete benefit to the participants is the opportunity to experience an information system that is novel and interesting.  They will also be furthering the advancement of the development of such systems, which will eventually be useful for the searching/browsing of video collections on a broader scale.

 

8.      How will prior consent be obtained?  Each testing session will be individually scheduled at the convenience of the participant.  When the participant arrives, s/he will be given an overview of the project and its rationale, and the procedures for the usability testing will be explained.  The participant will be provided with and asked to sign a consent form (see Appendix E).  Immediately after the session, the participant will be given the incentive and asked to sign a receipt, which will be stored with the consent form.

 

9.      Describe security procedures for privacy and confidentiality.  Videotapes of the testing sessions will be stored in a secure location in the Interaction Design Lab while they are needed, and destroyed when they are no longer needed for the development of the surrogates.  A “highlights” tape of the testing sessions will be maintained; the participants will be made aware of this procedure and will be asked for permission to include portions of their tapes on the consent form.


Appendix A

Descriptions of the five surrogates to be evaluated

 

 

Five different surrogates will be evaluated in this initial study:  storyboard with visual keywords, storyboard with audio keywords, slide show with visual keywords, slide show with audio keywords, and fast forward.  Each of these surrogates is briefly described here.

 

Storyboard with visual keywords.  The storyboard surrogate will display a set of key frames (individual frames selected from the target video, intended to represent the video’s content), laid out on a grid on the computer screen.  For this study, the most extensive storyboard will include 36 key frames, in a 6x6 grid.  The key frames will be augmented with a set of keywords, providing a verbal description of the video content.  The set of keywords will be displayed at the bottom of the screen, under the storyboard. 

For this study, users will be allowed to view the storyboard for a limited amount of time.  The amount of viewing time will vary with the number of key frames included in each storyboard, allowing 500 milliseconds per key frame.  For example, if a storyboard includes 20 key frames, the user will view it for 10 seconds.

 

Storyboard with audio keywords.  For this surrogate, the same key frames in the same layout will be used.  However, instead of the keywords being displayed on the screen, an audio recording of them will be played during the viewing.  The audio recording will be generated by a speech synthesizer, so that standardization of pace and pronunciation can be ensured.

            The users will view the storyboard for the same amount of time as described above.  The audio track will be repeated, if there is sufficient time.

 

Slide show with visual keywords.  The slide show surrogate will display a set of key frames, one at a time, at a prescribed pace:  250 milliseconds per frame, for this study.  The entire set will be displayed twice, with no pause between the two repetitions.  The selection of key frames will be the same as the set used for the storyboard surrogates.  The slide show will be augmented with a set of keywords (the same set as used for the storyboard surrogates) that will be displayed at the bottom of the screen.

 

Slide show with audio keywords.  The same slide show will be used for this surrogate:  the same key frames, at the same pace, shown twice.  An audio recording of the keywords will be read during the slide show; it will be the same audio track as that used with the storyboard with audio keywords surrogate.

 

Fast forward.  The fast forward surrogate mimics the fast forward function of a VCR player.  For this study, the target video segment will be played at four times its original speed.  The user will view it once.  No keywords will augment this surrogate.

 

 


Appendix B

Interview guide

 

 

Explanation of “surrogate”:  During this evaluation, we’ll be asking you to work with five alternative surrogates for videos in our collection.  We call them surrogates, because they might be used in place of viewing the whole video for certain purposes such as selecting a particular video for full viewing or sorting the videos into a certain order.

 

 

For each surrogate:

·          What would you say are two strengths of this video surrogate?

·          What would you say are two weaknesses of this video surrogate?

·          For what types of tasks do you think this surrogate will work very well?

·          What types of tasks might you like to do, for which this surrogate will not work well?  How can it be improved to serve this purpose?

·          If there are time constraints on you completing any of these tasks, does your evaluation of the usefulness of this surrogate change?  In what ways?

·          Any other suggestions for improving this surrogate?

 

After all surrogates have been viewed:

·          Which surrogate do you think was the most effective?

·          For which tasks will it work the best?  Why?

·          For which tasks will it be less effective?  Why?

·          Were some surrogates better for particular videos than for other videos?  Why?

 


Appendix C

Tasks to be completed in phase 3

 

            Five tasks (some involving two variations) will be performed with each of the three videos included in the third phase:  object recognition, action recognition, gist determination, and visual gist determination.

 

Object recognition task

Method 1:  Visual

Twelve frames will be displayed to the study participant, one at a time.  The frames will include six randomly selected from the key frames included in the storyboard surrogate, three selected from other segments of the same video, and three selected from other videos in the collection.  For each frame, the participant will be asked whether it was one of the key frames viewed in the surrogate.

Method 2:  Object names

            A list of 12 object names will be displayed to the study participant.  Six of the objects will have been viewed in the video surrogate; six will name objects not included in the video.   Three of each set of names will be concrete, e.g., “astronaut” for the “Moon” video, and three of each set will be more abstract or conceptual, e.g., “space program” for the “Moon” video.  The participant will be asked to check any member of the list that refer to obejcts seen in the surrogate.

 

Action recognition task

Six mini-segments will be displayed to the study participant.  A mini-segment will be 2-3 seconds long.  Two of the mini-segments will be from the video segment represented in the video surrogate, two will be from another segment of the same video, and two will be from a different video.  In response to each mini-segment, the participant will be asked whether s/he believes it is from the same video as represented in the surrogate.

 

Gist determination task

Method 1:  Free-form writing

            The study participant will be asked to write a brief summary (a few sentences) of the video content.

Method 2:  Multiple choice

            Five brief summaries of the video segment will be displayed simultaneously to the study participant.  The participant will be asked to select the summary that best represents the video content.

 

Visual gist task

            Twelve frames will be displayed to the study participant, one at a time.  None of them will be key frames included in the surrogate.  Three will be other frames from the video segment; three will be selected from other segments of the same video; three will be selected from other videos of a similar style (black and white versus color, etc.), and three will be selected from videos of other styles.  The study participant will be asked whether s/he believes the frame belongs in the video segment for which the surrogate was seen.  The study participant will also be asked for the basis for his/her judgment for each frame.

 

Appendix D

Recruitment flyer

Interested in video?

Help us evaluate a

computer system for

video collections!

 

 

 

We are developing some computer interfaces for interacting with large collections of video online.  We need your input in order to make these interfaces useful for their intended audience.

 

 

If you participate in this study, you will be asked to interact with five alternative interfaces and will be interviewed about your perspective on their strengths and weaknesses.  You will also be asked to perform some basic tasks with one of the interfaces.  Your interactions with the interfaces will be videotaped for later analysis.  The entire procedure should take about two hours; you will receive $25 as compensation for your effort.

 

If you’re interested in participating in this study or have questions about it, please contact: 

Gary Marchionini (966-3611, march@ils.unc.edu) or

Barbara Wildemuth (962-8072, wildem@ils.unc.edu).

 

To learn more about the Open Video Project, see http://video.ils.unc.edu/open-video/.

 


Appendix E

Informed Consent Form

 

 

 

THE UNIVERSITY OF NORTH CAROLINA

AT

CHAPEL HILL

 

School of Information and Library Science

Phone# (919) 962-8366

Fax# (919) 962-8071

CB# 3360  100 Manning Hall

Chapel Hill  NC 27599-3360

Email: info@ils.unc.edu

Http://www.ils.unc.edu

 

 

Evaluation of Alternative Video Surrogates

 

Introduction to the Study:

We are inviting you to be involved in the evaluation of several alternative interfaces, or surrogates, for browsing a video collection.  We call them surrogates, because they could be used in place of viewing the whole video. These surrogates are in the early development stages, and your comments will help to improve their usability for others.  The study is being conducted by a research team from the Interaction Design Lab at UNC’s School of Information and Library Science, and is directed by Dr. Gary Marchionini (966-3611, march@ils.unc.edu) and Dr. Barbara Wildemuth (962-8072, wildem@ils.unc.edu).

 

Purpose:

The purpose of the evaluation is to identify the strengths and weaknesses in each alternative surrogate, relative to its use for working with the videos in a video collection.  We will use what we learn from the evaluation in order to redesign the surrogates so that they are easier to use.

 

What Will Happen During the Study:

You are one of just a few people being asked to participate in the evaluation of the surrogates.  You will be asked to interact with several different video surrogates, each representing a particular video in the collection.  You will be asked to comment on the strengths and weaknesses of each surrogate, from your own perspective.  After you have had a chance to interact with all the surrogates of several videos, you will be asked to complete several tasks (such as selecting a video for a certain purpose or summarizing the content of a short video) with whichever surrogate you prefer.  A member of the research team will be with you at all times, and will be interviewing you as you work with the video surrogates.  Your interactions with the surrogates and your comments about them will be videotaped.  After you have completed the study activities, you will be offered $25 in appreciation of your participation.

 


Your Privacy is Important:

We will make every effort to protect your privacy.

We will not use your name in any of the information we get from this study or in any of the research reports.

The videotape will be viewed only by members of the research team, for the purposes of understanding usability problems that you experienced.

At the end of the study, a tape may be compiled of interactions representing the strengths and weaknesses of the surrogates being evaluated.  This “highlights” tape may be viewed by others interested in the development of similar systems, e.g., at research conferences.

The original videotape will be destroyed at the end of the study.

 

Risks and Discomforts:

We do not know of any personal risk or discomfort you will have from being in this study.

 

Your Rights:

You decide on your own whether or not you want to be in this study.

If you decide to be in the study, you will have the right to stop being in the study at any time.

 

Institutional Review Board Approval:

The Academic Affairs Institutional Review Board (AA-IRB) of the University of North Carolina at Chapel Hill has approved this study.  If you have any concerns about your rights in this study you may contact the Chair of the AA-IRB, Barbara Davis Goldman, Ph.D., at CB# 4100, 201 Bynum Hall, UNC-CH, Chapel Hill, NC 27599-4100, (919) 962-7761 email: aa-irb@unc.edu .15

 

Your Consent:

I have had the chance to ask any questions I have about this study, and they have been answered for me.  There are two copies of this form.  I will keep one copy and return the other to the investigator.

 

I have read the information in this consent form, and I agree to be in the study.

   

________________________________

(Signature of Participant)

 

I also agree that portions of the videotape made during my participation may be used in a “highlights” tape that may be shown to people outside the research team.

 

________________________________

(Signature of Participant)

 

 

________________________________

(Date)