Protocol for Usability Testing, Fall 2001

(updated 9/26/01)

I. Welcome the participant

II. Briefly describe the study

a. Use the consent form as a guide for our oral description

b. Provide the consent form to the participant to review

III. Obtain the participant’s consent

a. Keep a signed copy of the consent form

b. Give one copy to the participant

IV. Begin videotape capture of session

V. Phase One: Working with the surrogates

a. View video segment (full length version); see participant worksheet for video order

Video 1 = Apollo, Segment 4006

Video 2 = Master Hands, Segment 1

b. Explain what a surrogate is and how they are used in the study (for first video only):

“During this evaluation, we’ll be asking you to work with five alternative surrogates for videos in our collection. We call them surrogates, because they might be used in place of viewing the whole video for certain purposes such as selecting a particular video for full viewing or sorting the videos into a certain order.

You will be working with each surrogate for just a short time; the computer program controls the timing.

We are very interested in the ways that you think you might find these surrogates to be useful, and will be asking you for your ideas on their use as we work through the study.”

c. View surrogates (in order specified)

i) Slide show with visual keywords (SS-V)

ii) Slide show with audio keywords (SS-A)

iii) Storyboard with visual keywords (SB-V)

iv) Storyboard with audio keywords (SB-A)

v) Fast forward (FF; available in separate window)

d. After each surrogate, conduct interview:

§ What would you say are two strengths of this video surrogate?

§ What would you say are two weaknesses of this video surrogate?

§ For what types of tasks do you think this surrogate will work very well?

§ What types of tasks might you like to do, for which this surrogate will not work well? How can it be improved to serve this purpose?

§ If there are time constraints on you completing any of these tasks (for example, if you only had a few seconds to choose a video), does your evaluation of the usefulness of this surrogate change? In what ways?

§ Any other suggestions for improving this surrogate?

e. After all surrogates have been viewed, additional interview:

§ Which surrogate do you think was the most effective?

§ For which tasks will it work the best? Why?

§ For which tasks will it be less effective? Why?

§ Were some surrogates better for particular videos than for other videos? Why?

f. Repeat all Phase One procedures (a, c-e) for next video

VI. Phase Two: Working with the surrogates

a. View surrogates for video (do NOT view the full length segment); see participant worksheet for video order

Video 3 = Challenge at Glen Canyon, BOR03, Segment 2

Video 4 = Educational Films: A Date with Your Family

b. After each surrogate, conduct interview:

§ What would you say are two strengths of this video surrogate?

§ What would you say are two weaknesses of this video surrogate?

§ For what types of tasks do you think this surrogate will work very well?

§ What types of tasks might you like to do, for which this surrogate will not work well? How can it be improved to serve this purpose?

§ If there are time constraints on you completing any of these tasks, does your evaluation of the usefulness of this surrogate change? In what ways?

§ Any other suggestions for improving this surrogate?

c. After all surrogates have been viewed, additional interview:

§ Which surrogate do you think was the most effective?

§ For which tasks will it work the best? Why?

§ For which tasks will it be less effective? Why?

§ Were some surrogates better for particular videos than for other videos? Why?

§ Did not being allowed to view the full segment before viewing the surrogates make a difference in how you viewed the surrogates? How?

d. Repeat all Phase Two procedures (a-c) for next video

VII. Phase Three: performance of tasks with the surrogates

a. Explain that several tasks related to understanding a video will be performed; explain that the tasks are intended to test the usefulness of the surrogates, not the participants’ abilities

b. Allow the participant to select ONE surrogate to use for the tasks:

“Which surrogate would you like to use to complete the first set of tasks?”

c. View the selected surrogate for video 5; see table for video order

Video 5 = Hurricanes, Segment 1

Video 6 = Moon, Segment 2

Video 7 = New Indians, Segment 101

d. Ask the participant to complete the tasks (in the order specified), and to think aloud while completing them

i) Gist determination: free-form writing

ii) Gist determination: multiple choice

iii) Object recognition: object names

iv) Object recognition: visual/frames

v) Action recognition

vi) Visual gist/style

e. Interview about the surrogate used:

§ Were you pleased with your choice of surrogate?

§ In what ways did this surrogate help you perform the tasks? In what ways was it not as useful as some other surrogate might have been?

f. Repeat all Phase Three procedures (b-e) for next video

g. Repeat all Phase Three procedures (b-e) for next video

PROTOCOL: USABILITY STUDY, FALL 2001

Note: The table provides the order in which the videos will be presented for each participant.

Participant

Phase 1

First

Video 1

Video 2

Video 1

Video 2

Video 1

Video 2

Video 1

Video 2

Video 1

Video 2

Video 1

Video 2

Second

Video 2

Video 1

Video 2

Video 1

Video 2

Video 1

Video 2

Video 1

Video 2

Video 1

Video 2

Video 1

Phase 2

Third

Video 3

Video 4

Video 3

Video 4

Video 3

Video 4

Fourth

Video 4

Video 3

Video 4

Video 3

Video 4

Video 3

Phase 3

Fifth

Video 6

Video 5

Video 7

Video 5

Video 7

Video 6

Video 5

Video 7

Video 5

Video 7

Video 6

Sixth

Video 5

Video 7

Video 6

Video 5

Video 7

Video 5

Video 7

Video 6

Video 5

Video 7

Seventh

Video 7

Video 6

Video 5

Video 7

Video 6

Video 5

Video 7

Video 6

Video 5

Video 7

Video 6

Video 5