1. Introduction

This report summarizes work done from September 1, 1998 through July 31, 1998 pursuit to a contract that had two main objectives: a) designing and testing user interface designs for the BLS website that are rooted in user needs and tasks studies done in the previous phase of work, and b) extending the transaction log analyses done in a previous phase of this work and defining recommendations for BLS transactions logging procedures. In this document, this work will be referred to as the second phase (or current phase). As in the previous phase, Anita Komlodi and Stephan Greene were research assistants at the University of Maryland with Gary Marchionini and all worked closely with Carol Hert and her team at Indiana University and with BLS staff.

1.1. Task description

The BLS website represents an increasingly important service that the Bureau provides to the public. The website will continue to evolve in the years ahead as technology advances, public information literacy grows, and as government continues to develop its online service capabilities and personnel. During the October 1996-June 1997 period, we investigated website structure and customer usage patterns for the BLS site (phase 1). That study led to better understanding of website users and their needs and to several recommendations for improving and extending development of the website (Hert & Marchionini, 1997). The second phase reported on here built on that work with an aim toward assisting BLS staff in building advanced interfaces and tools for new iterations of the website.

In the first phase report, many design recommendations were made. Some of those recommendations are relatively easy to implement (e.g., some hierarchical reorganization of the website, adding some query enhancement features, etc.), while others require considerable effort (e.g., providing an interface geared to users rather than agency data, developing procedures and tools for website management and maintenance). One mportant theoretical as well as practical problem raised by our investigation of users in phase one was how to develop the website to serve diverse user capabilities, experiences, systems, and needs.

In the second phase work, we focused on two specific interface design problems: alternative interfaces and transaction log tools and techniques. The former is a general problem in the HCI field and our work has both practical benefits for the BLS website interface as well as theoretical import for the design community. The latter leveraged our BLS transaction log analysis experience to specify procedures that will allow BLS to systematically monitor website evolution and may be transferable to other government installations.

Thus, the main objective of the work was to inform the evolution of the BLS website in general with special emphasis on defining alternative interfaces that serve diverse sets of users. A secondary emphasis was to develop a design concept for transaction log tools and techniques.

1.1.1. Alternative interfaces

One of the grand challenges of interface design is to effectively serve the complete range of diverse users and tasks without requiring that everyone learn a single interface style. This challenge is also manifested in the goal of government and business to provide universal access to all citizens and customers. One approach to this challenge is to create an interface that adapts to users as they work. In this approach, users may define general profiles and the system makes inferences about session-specific needs as work progresses. Many years of effort have gone into creating so-called intelligent interfaces and we remain optimistic that progress will continue over the long term (the task/question taxonomy developed in our earlier work may be applicable in this regard). A second approach, however, is more practical for the immediate years ahead. This approach pre-defines a small set of interfaces that users may choose according to their specific experience and needs. This approach is no less challenging conceptually, as the choices must be made carefully, the respective interfaces must be crafted to minimize complexity while maximizing system functionality, and the alternatives must be clearly represented so that the entire range of users can quickly understand and optimally choose the interface variant that meets their information-seeking situation. The work conducted in the second phase takes this second approach to the alternative interface challenge.

Many types of users visit the BLS website. The general types identified in our phase one study were: business users, academic users, the media, the general public, government users, K-12 education users, statisticians, and library/museum and other non-profit users. Likewise, users bring many types of tasks to the website. In phase one, we developed a multidimensional task/question taxonomy. The pragmatic (contextual) dimension includes attributes in three categories: goal (learn, verify, judge, explore, refer, subtask, ongoing, plan), constraints on the task (time, volume, geographic location), and system (database match to task, search and extract tools, formats, entry point and path). The semantic dimension includes attributes of topic, level of abstraction, level of specificity, and number of facets. The syntactic dimension includes attributes in three categories: expression type (what, where, who, how, and why), goal type (closed, interpretive, accretional), and specificity of expression.

The strategy taken in the current work was to use the taxonomy developed in phase one to define a small set of views that users may select to use during their sessions. These views were based on crossing user types—the roles people play (e.g., teacher, researcher)—and common tasks—the goals people have when they visit (e.g., gather information about moving, update a contract, etc.). These views were defined, prototyped as webpages, tested with BLS staff, and revised.

1.1.2. Transaction log tools and techniques

In our previous report, we found that transaction log analysis could be used to understand users' information-seeking strategies and as a tool for identifying interface features to improve. This phase followed up that work to help BLS develop the best ways to leverage transaction log data to continuously develop the website interface. There were three main goals: confirm trends and usage patterns identified in the first phase, to identify the primary functionality and goals for summary and sequential log analysis, and to suggest prioritized development actions.

2. User Interface Design

The bulk of our efforts went into addressing the interface objective. The guiding principles are first laid out, the procedures and testing results for each of the three prototype iterations are then presented, and recommendations for adoption and adaptation in the BLS site are made.

2.1 Problems and Principles

A third general problem all websites face is giving users easy access to the information they need. There are two primary solutions: eliciting a query from the user and applying a search engine to return a ranked list of potentially relevant pages/objects, and providing a series of choices for users to "drill down" through until they find the most appropriate page/object. The former is a form of analytical search and the latter one form of browsing (Marchionini, 1995). In this project we focused on the browsing solution. BLS provides a typical keyword search facility and we included both quick (simple) and advanced search options in the interface prototype but did not specify complete mechanisms for either (this should be done in some future project; see Shneiderman, Byrd, & Croft, 1998 for general guidelines on what features should be included in search interfaces). Problems related to browsing include user disorientation (getting lost in a set of selections), distraction (becoming seduced by interesting but immediately irrelevant selections), and frustration due to too many ancillary actions before getting relevant information (this often leads to abandonment as well as dissatisfaction). Our approach to the problems of disorientation and frustration (we pose no solution for helping users maintain the self-discipline to remain on task) is to try to minimize mouse clicks and bring primary information as close to the user entry point (home page) as possible. It has become common to hear designers talk about a "three click rule" that aims to get users what they want in three clicks or less. We have applied techniques to collapse hierarchy and use information abundant widgets (Marchionini, Plaisant, & Komlodi, in press; Shneiderman, 1998) to this set of user problems. This represents our application of the information abundant design principle.

Finally, website designers must find ways to represent complex information and organization in sensible displays. We have chosen a visual metaphor rather than a textual list, array, or hierarchy. The visual methaphor is a set of paths with gateways to different types of services. This design uses the mutually reinforcing techniques of positional clustering and a physical navigation metaphor. This represents our application of the spatial display design principle.

2.2 Procedures and Results

The interface had three iterations, each of which yielded a prototype that was presented to BLS groups and revised based on their reactions. Because the prototypes and testing are integral to the each succeeding iteration, testing results for each iteration are given immediately after each iteration’s design description. The overall approach was to use discount usability testing (Nielsen, 1993) in the first iterations and laboratory usability testing in the final iteration. Suggestions for further development based on the user testing of the third iteration are included in the Summary and Recommendations section.

2.2.1. Iteration 1.

Design

Based on the user taxonomy and guided by the design principles above, a paper-based design was created and presented for discussion at a meeting with BLS staff on January 6, 1998 (Conrad, Bosley, Dippo, Tucker, Klein, Hert, Marchionini). This meeting served to provide first-level reactions to the design and raise suggestions for improvement. A second set of reactions were obtained in a meeting with Labstat staff on January 24, 1998.

This original design provided four different access modes to the website (see Figure 1). These four access methods provide access alternatives for users on a single screen rather than having users first pick an interface from a short list. These alternatives present very coarse distinctions between access methods because the system's user model is based on the general user type/task rather than a personalized user profile. Although the type/task taxonomy could be used as the basis for a template that users could complete to create personalized user profiles, the elicitation and completion process are tedious and may be difficult for novices. Additionally, hundreds of thousands of profiles must be stored and managed. A more generalized ‘just-in-time’ solution was, therefore deemed more practical. The pre-determined ‘profiles’ used in our design aimed to provide a range of access methods as well as specific instances of user type and task. To better communicate ‘type’ and ‘task’ to end users, the terms ‘role’ and ‘goal’ respectively were used in the design.

We included two approaches to the design challenge of presenting role and goal alternatives to users. The first was based on the notion that users can map their need onto the entire taxonomy as an interactive form. This allows the possibility for users to fill out the whole taxonomy and store it on the local machine as a profile (assuming some Java applet would transfer this to the server upon connection). For this prototype, we focused on developing and testing the viability of the full taxonomy approach. The second approach simplified the process of expressing user needs by providing a small set of scenarios that were based on the most frequently used cases of the taxonomy (based on our research in the first phase). This approach can save considerable user time and system resources as the most frequently needed scenarios would serve a large percentage of users at the expense of providing coverage for the full range of types and tasks.

The matrix in the upper left part of the screen represents our first attempt at implementing the full taxonomy approach. Some of the user and task categories defined in the previous phase work were collapsed for a less complex design. The two matrix dimensions (role and goal) were each divided into six categories (role categories were students, researchers.economists/teachers, business persons, statisticians, journalists, others; goal categories were explore/learn, verify/find, judge/evaluate/compare, refer to other sources, ongoing/regular lookup, and plan/forecast), yielding 36 cells representing role/goal categories. After creating the matrix, the most useful information sources and search tools for each cell in the matrix were collected. The plan was to present a set of optimal pages for each cell of the matrix. The display provides pages in a left index frame and specific content in a right frame. A key issue is whether users can successfully identify their role and goal. If so, the system can better customize search tools and interfaces to serve their needs, i.e., the system would have a better model of the user than if the user simply chose from a few alternative access modes.

The upper right corner of the first iteration screen design provides a graphical site map. Site maps provide spatial views of the organization of the website, thus providing easier navigation and better understanding of the structure and functions of the website. Most site maps are simple textual tables of contents for the site. Although these are useful, problems with scrolling, manipulating hierarchy, and showing relationships other than parent-child are well known. Additionally, graphical maps may help to overcome some of the ambiguity of linguistic expression. The site map in this iteration was organized by functional units: data, fast information and publications. The fourth region contains other items such as help, contact information, etc. This three-way organization reflects our understanding that the main focus of the BLS website is to provide statistical data, summary information compiled from this data, and publications building on the data and the information. This map provides access on two levels. It enables frequent users to go directly to the page of interest and it helps novices familiarize themselves with the site.

The lower left section of the screen presented links to scenarios. These are tasks that our previous work indicated are frequently brought to the BLS site. For each scenario, a set of useful pages was compiled to make information seeking and navigation simpler and more effective. Scenarios are well-defined cases of the user type/task matrix, but the user does not have to fill out the matrix to define his/her situation. For a large number of users, such context-based scenarios may quickly and easily match their information needs.

Keyword search is an essential service for any website (transaction log analysis shows that large portions of the requests are to keyword searches from the home page or within different BLS services). At present, BLS provides a single search service. Many web portals (e.g., Excite, Lycos) have expanded to provide simple and advanced options as well as specialized features such as relevance feedback, alternative term suggestions, etc. For the prototype, we included a quick search option. This seems reasonable since users may not be willing to spend more than few minutes at a site (as we can see from the high portion of abandoned sessions in the transaction log analysis). WWW users are also used to having a quick search box available on most sites today.

Several problems with the matrix were discussed. First, it presents a complex framework that may be overwhelming for novice users. Second, users may not be able to accurately classify themselves according to these categories since interpretation of the labels can vary from user to user and also can be very different from the page mappings built into the system. Third, these interfaces can slow down access for expert users by shielding them from direct access to sources and tools. Fourth, the set of pages relevant to each cell showed considerable overlap. The challenge of mapping cells to best sets of pages caused considerable discussion and the 36 variants were judged to not provide sufficient differences to warrant the screen real estate given to the matrix.

The BLS staff surmised that uses would easily understand the scenarios and would find them useful, although everyone recognized the need to fully develop and test the suite of scenarios with broad constituencies over time. Reactions to the site map were highly favorable and we were encouraged to enhance the site map metaphor in subsequent iterations. The general approach was discussed with respect to Labstat’s homepage content draft (dated September 5, 1997) and most of the basic functionality and user-centered perspective were found to be in agreement. Challenges related to geographic and cross-survey access, and providing vocabulary and other help to users were discussed. Based on these sets of discussions, a second iteration was initiated.

2.2.2. Iteration 2.

The second iteration focused on the site map, incorporating the comments from the first meeting with a BLS group. All four access methods remained in design but the site map became the focal point. The user task/goal matrix was changed to provide two drop-down menus (one for each of the role and goal dimensions). This saved considerable screen real estate and allowed a more detailed site map to become the centerpiece of the design. New scenarios were also added to the design. This design was mocked up in HTML with JavaScript extensions so that it could be shown online rather than on paper. This not only allows a more realistic look and feel but also allows some of the interface dynamics (e.g., screen transitions) to be demonstrated. This prototype was discussed at a meeting with BLS staff on March 24, 1997 (Conrad, Bosley, Tucker, Klein, Hert, Marchionini) in preparation for a feedback meeting with five BLS staff members not familiar with the project. That meeting took place on March 30, 1997.

A screen dump of this prototype is presented in Figure 2. The upper part of the screen contained and extended the search function with a three-line search box and search buttons customized to the different sections (topical areas) of the site (suggesting more focused, field-delimited searching). This was meant to inform users about the different types of content available at the site and also help them better focus their searches to yield fewer irrelevant materials.

The site map was greatly expanded in this iteration. The site map was made larger in size and resolution and had a more prominent place on the homepage. The spatial arrangement of four functional areas and the color coding to emphasize those regions remained in the design. To push the information-abundant design principle, many new links were added to the map representing two or three levels of the web site hierarchy. On the left side, in the data region, direct links to the search tools by content area were added. This multiple-layer access was designed to help users understand what they will see under a link before actually going there, thus enabling better navigational decisions. At the same time it provides direct access to pages two or three levels down the hierarchy of the website, thus saving time for users who know what pages and tools they want to access. The spatial metaphor of the design suggests three regions with gates leading to them and a foyer area in the forefront of the design where general tools can be found. This graphical layout was a mockup and needs more careful graphics design to better reflect the spatial metaphor.

In the Economy at a Glance section the design provided actual data extracted from the larger economy at a glance table. This brings primary data immediately to the fore, a principle we have implemented in our Library of Congress designs (Plaisant, Marchionini, Komlodi, & Bruns, 1996). The data here can be selected on the basis of most frequent use and can be updated depending on changing demands. Displaying data on the homepage would save users effort and time in navigating. Of course, this complicates the overall page layout considerably. Presenting the second level links on the site map also aimed at putting the user much closer to the data directly from the homepage. This design allows user to get to the actual content of the site in one or two clicks rather than three to five.

Wording of the labels on the map was given careful consideration. In some cases wording from the current BLS website was kept, (e.g. Handbook of Methods, Economy at a Glance, etc). In other cases new, more user-oriented wording was chosen (e.g., Quick Access, Advanced Access). Although this helps novice users to better understand what they will find behind a link, it may make it more difficult for experts and frequent users to find certain familiar tools or pages. An example compromise was the label ‘Series Report: Expert Access’ where BLS wording was supplemented with explanatory words. However, these labels can get very long and take up too much space on the map. To solve this problem, pop-up windows with short explanations were included in the map. These explanatory messages were implemented as mouse-over features that cause the messages to appear as the user moves his/her cursor over a part of the map. The goal of these messages is to help the user learn what is behind a link before actually going there. This can save user time and effort by avoiding unnecessary navigation and also better inform the user about the options of the site.

The matrix was still present on this design in a less screen-consuming form; it was presented as two drop-down lists from which users can select their own role and goal for the current visit. The actual pages behind these links have not been updated since the first iteration of the design. Selection of these pages could be greatly enhanced by the expertise of intermediaries working with users.

The prototype was demonstrated to the LabStat and Publications group and participants were invited to ask questions or make comments at any time. The group was supportive of the aim to minimize user mouse clicks and the techniques of providing alternative interfaces. The topic that generated the most discussion at the meetings was the linkage between roles and goals. One suggestion was to separate roles and goals to simplify the choices for users and the mappings to BLS services. Roles suggested as important included real estate agents and economists. Goals suggested as important included contract escalation (e.g, updating contracts to account for inflation), moving, wages, and prices (cpi and ppi seperately). The general consensus was that the matrix and the role/goal specification would be confusing for people to use as well as difficult for BLS staff or expert users to agree on the best mappings for each role/goal combination.

The overall development cycle for the BLS website was also discussed. It was agreed that the website is an evolving entity that will require regular updates in design as well as content. Although it was suggested that a 3-month development cycle should be achieved, it is unlikely that the Bureau will be to support or tolerate such change rates. Discussions with other Department of Labor agencies on common guidelines must also be taken into account when planning upgrades.able

2.2.3. Iteration 3.

The online prototype was revised based on the feedback from BLS staff at the two meetings. The third iteration was then tested with six BLS staff members from a variety of divisions and job roles. The testing was conducted in the BLS Usability Laboratory. Separate one-hour sessions were held for each subject (April 17, 1998 and May 4, 1998). A testing protocol was developed (see Appendix A) to guide the sessions. The protocol included three sections: an introduction in which subjects were introduced to the prototype and purpose of the testing and background information on their work experience and website experience was elicited; a set of eight scenarios that subjects were asked to use to guide searches with the prototype, and one self-directed task; and a debriefing section that elicited user responses to both specific and general questions about the interface. Subjects were seated at a workstation with the prototype with the investigator (Marchionini) seated next to them. Subjects used the mouse and keyboard to execute the scenarios while the investigator prompted them and took notes. Two video cameras and a microphone were used to record activity. One was trained on the subject’s face and one (ceiling mounted) was trained on the keyboard. These signals as well as a video out from the workstation display were fed into a video mixer and a three frame video composite sent to a VCR. The resulting tape showed the screen, keyboard and mouse, and user’s face. One disadvantage of this setup was the inability to see what subjects were pointing to on the screen as they discussed their actions and responded to investigator queries. A second investigator (Komlodi) controlled equipment and took notes in an adjoining room where the recording equipment was housed.

The final prototype (see Figure 3) eliminated the role/goal feature completely. The three search boxes were collapsed to a larger one with separate search buttons by content area remaining on the right. The collapsed search box suggests to users to enter longer expressions when searching. The site map remained the focus of the homepage and the scenarios were integrated onto the map and completely took over the role of the user task/role matrix (see Figures 4 and 5 for scenario pages).

To save space, the Current statistics data table was redesigned to reflect only the most used statistics (a change that was based on the March 30 meeting feedback). Some of the icons were changed and a Hot Topic area was introduced on the map. Based on the discussions in previous iterations, the user/task matrix and the scenarios were united in this design, since the scenarios are general instances of user role/task combinations and they provide coarser models of the user to the system. Based on consultation with the BLS teams, five scenarios were included: moving, writing a term paper, verifying data, exploring career prospects, and teacher. The first four represent tasks (goals) and the fifth a user type (role).

The user testing yielded a rich set of data (approximately six hours of video tape and dozens of pages of notes made by the two investigators). The subjects were all BLS employees and in the background discussion we learned that four subjects were daily users of the BLS site, used it as their default home page, and mainly used it to help users who call or email. One subject used the site occasionally and one seldom used it. This relatively high level of familiarity must be taken into account when interpreting results and applying recommendations to the general population.

The protocol was designed to elicit an overall first impression of the design before subjects actually used it. This approach has the advantage of drawing out naïve first reactions unbiased (and uninformed) by the assigned tasks, and the disadvantage of setting up first impressions that may be inaccurate but reinforced as users work (people tend to try and reinforce impressions as they work rather than correct them, [Tversky & Kahneman, 1974]). We felt that gaining the first impressions would be highly useful and asked subjects to react to the overall layout, whether it would be helpful for searching, and how they thought they might use it in practice. Paraphrased statements made included:

S1:

Map needs better graphical metaphor

Search full site should come first

S2:

Terms used too close to BLS, should be more user-friendly

S3:

Data should be most important on map

Too complex—better for expert user

Use drop-down menu instead of search buttons to save space

Design does not reflect real content well

S4:

Word ‘search’ is repeated too many times

Problems with label wording

Map too cluttered but likes graphical organization

Reduce number of links for people with slow browsers

There can be other geographical access than regions

S5:

Scenarios are good because people call in with specific problems

At first sight it’s a lot but when you look at it more it’s helpful

S6:

Does not like search on web in general, would rather just browse and click

Does not get the spatial metaphor, needs better graphical design

Likes that it is on one screen and no scrolling necessary

These initial reactions illustrate both the sophistication of the subject sample and the complexity of the design approach. Many of these themes (cluttered look and feel, terminology, potential for scenarios, and potential for a content-oriented spatial organization) reoccurred as subjects completed the tasks and in the debriefing discussion.

For the eight tasks, subjects made a total of 72 actions with the interface. In most cases, these actions ended with a page retrieval that satisfied the scenario at some level of effectiveness. Because solutions are sometimes interpretations and there are often multiple relevant pages, no attempt was made to determine an accuracy score. Our general sense was that subjects were successful in the vast majority of trials. For the eight tasks, the total number of actions taken by all six subjects in aggregate were 12, 6, 10, 7, 10, 7, 7, and 13 respectively. In five cases, there was no action taken or an action was suggested based on experience with the current BLS site (e.g., selective access, look for announcements or warnings). It is interesting to examine the very first action users took for each task. There were 43 ‘first moves’ (six subjects for eight tasks less the five special cases of no or unusable response). Table 2.1. summarizes the number of selections made for different services. Note that only geographic access, search, and data are currently on the BLS home page, thus subjects would have to make at least two mouse clicks on different screens to get to their first choice to begin answering these questions. Thus, 34 of the 43 first choices (79%) would have required at least one additional mouse click on the current homepage. Moreover, geographic access on the prototype allowed subjects to immediately select a region without going to a second screen and making another mouse click. A design that saves 80% of the clicks from a page that gets 100,000 or more hits per month yields significant savings on both the client and server side. On the client side, assume a 2 second wait for a page to load (a faster connection than home modems offer), 160,000 seconds (more than 44 hours) of collective user wait time would be saved each month. On the server side, 80,000 fewer requests per month would need to be served. For pages that save many mouse clicks the user and server savings can be substantial. One caveat on the user side is that fewer clicks may come at the expense of more thinking time. This would be an interesting hypothesis to test in an environment with virtually no response wait time. The server side savings for information-abundant designs hold in any case. Thus, user performance lends strong support to the effectiveness of the information abundant design.

Service                Frequency

News releases        11

Geographic access 6

Quick access                 6

Scenarios                 6

Handbook of Methods 5

Advanced access         3

Economy at a glance 2

Search                         2

Research papers         1

Data                          1

Total                        43

The debriefing section was meant to prompt subjects into giving frank feedback on the interface after they had some experience using it. Table 2.2. provides paraphrased statements summarizing the responses to the debriefing questions used in the protocol. Note the paraphrase lists are generally in order by subject but specific assignments of statement to subject are not included.

1. Understand difference between data and fast info?

3. Does 3-way division of content make sense? Alternatives?

use the building metaphor all over (throughout design)

divisions between different levels of the hierarchy not clear

too much emphasis on main gates

likes color coding but lines confusing

4. Is the screen too cluttered or confusing?

yes, lots of information but well organized and legible, some terms are too BLS-y

better indication of boundaries needed between sections

yes, graphical design not obvious

it is, because it gives many options, but on second look it does not take long to decide where to start

yes, at first it is, but after looking at it for a few minutes like that everything is on one screen and does not have to scroll

5. How would you comapre this to the existing site?

more artistic, more subject-oriented rather than BLS

scenarios helpful

this is quite helpful, more visual than intranet, takes less time to decide where to go

take a lot more time thinking where to go

should have BLS logo bigger

more complicated, can’t figure out the graphical metaphor

liked having everything on a screen don’t like to scroll and go back

6. Do you think this interface will be better? Why?

sectorizing very useful, especially in quickly orienting people to right neighborhood and tell them what’s there

likes geographic access and current stats table on homepage, homepage invites you back as it may be updated later

yes, tells end uses about the structure of the site, tells you where the different information is

7. Do you think this interface will be faster to use? Why?

there is a learning curve, first time users may be frustrated

being able to sort things out at first step

faster to find entry point, visual roadmaps helps, save time compared to reading

can get to a desired starting point more quickly

takes longer to understand but shorter to navigate

graphical design confusing

8.Do you think this interface will be worse? Why?

not really, might not notice the organization

need to explain levels of hierarchy to users

some labels are vague, have to tell people what’s underneath

lists would be better as pop-ups

not sure, didn’t get the arches and road at first

Will the interface serve a variety of users? Why? How?

it is more content and subject oriented

yes, even high school kids and middle school kids

What is the best thing about this prototype?

Three way layout of info

Users can easily and quickly pick out best general area to start with

Roll-over help good

Conveys flexibility, not rigid, more interesting

Up front info

Access to different data types available right from homepage

Has Occupational Outlook Handbook in front

nice to look at, more topic oriented

Visual

Its, all there

scenarios good

What is the worst thing?

Lots of data, lists are organization (BLS) driven

Glossary between BLS and user terminology needed

Too much space wasted on things people don’t look for

Looks busy, no distinction between places, assumes user knows the site

Crowded

Kind of overwhelming at first glance

Too much visually

Lines a little confusing, takes a while to figure out, icons don’t work well

Suggestions

quick access is too vague

most of the action is outside the center

get rid of the lines

add glossary, give the high level info plus explanations & glossary before the fine grained details

put news releases on the boundary of data and publications

clarify fast info and quick access

add state & area data

add search by subject, search by BLS people

advanced search too much, consolidate

BLS screen currently geared to professionals

About half of the subjects understood the distinction between ‘data’ and ‘fast information,’ others expressed concern about the clarity of the distinction, and some suggested more extensive explanation about the labels and terminology used in the design. One subject strongly preferred ‘quick’ to ‘fast’ information. Another argued for a glossary to map user vocabulary to BLS vocabulary. Clearly, for all labels we need to find a common term to use as the default and then provide mouse-over alternatives and explanations.

We gave considerable attention to the overall organization of the design. The best aspects of the design mentioned in the debriefing were the visual display, ease of navigation (once the organization/layout was understood), and availability of many links from the homepage without scrolling. The visual display and cluttered layout were also mentioned as the worst aspects of the design. This seeming contradictory reaction to the design’s organization and layout illustrates the tradeoffs between managing complex and extensive information and the desire to have intuitive interfaces with no learning curve. Perhaps a better way to think about the realities of the goal is to aim for steep learning curves for websites, i.e., we believe the principles used in this design require a very brief but highly focused and cognitively demanding learning period that will lead to much better performance due to users gaining an understanding of the availability and overall structure of the site and facilities for getting to primary data faster and more precisely.

The overall organization and the three-way distinction was clear to four of the six subjects. One subject felt that it reflected the BLS organization too much and another was unclear about the different levels of the hierarchy and why different levels of access were necessary. The spatial metaphor seems useful for them but the graphical design needs more work to present a cleaner picture that is easier to process. Compared to the existing site, some subjects questioned the clarity of the visual design, but most subjects found the new design more visual and generally organized more around subjects and topics instead of the internal organization of BLS. One subject referred to the current BLS design as the ‘block’ design—an apt description of its highly structured and neat organization. One subject thought the prototype design showed the structure of the site and where different information is located. One noted the possibility of placing the user in the right neighborhood and another found it conducive to finding a good entry point. More choices on one screen slows users down since it takes longer to understand the first screen. One subject felt there would be a detrimentally long learning curve and another thought it would take longer to understand but eventually make for easier navigation. On the other hand, one subject said " I think it would be faster in that you can at least get to a desired starting point even if you don’t get your information." The issue of quickly finding an entry point and easier navigation are important to consider in future iterations of the BLS website.

All subjects felt that the screen was cluttered. On the other hand, half of them mentioned that after looking at it for a few minutes they understood the organization and distinctions and appreciated that they did not have to scroll and everything was presented on one screen. A quote from one subject responding to the overall organization question in the debriefing illustrates this tradeoff. "Now that I’m looking at it, I’m seeing there is a correlation between these three things here, all the data stuff is in this area, under this arch, and the fast information in there, but at first glance I didn’t think of it as that. Then all the publications, now that I look at it are up there." This clearly shows the tradeoffs of the information abundant design principle, whether it is worth a short learning curve to provide a more navigable and informative website is the key design question.

Subjects liked the scenarios. One noted how useful they would be to users who might otherwise call for support. This subject noted: "This is good because I have a lot of people calling our office looking for information. They’ve been to the website but they couldn’t actually narrow down what they actually wanted. They want to talk to a person, they want to explain that I’m writing a paper or I’m moving, I’m trying to verify data, whatever---to talk to somebody to tell them that---they’re hoping that somebody can direct them to the right place without having to search all over the place. So immediately I thought, boy this is great for an outsider." Expanding and testing good scenarios for the BLS site should be a high priority for site planning and development.

2.3 Recommendations

BLS has an upgrade path defined (Labstat staff have upgraded the server infrastructure and have plans to revise the website interface in coordination with Department of Labor guidelines) but full site redesigns are both expensive and require policy actions. We recommend that Labstat continue its practice of making incremental changes while planning for periodic major revisions. Because major revisions cause significant disruptions in user work flow and the already large installed base of BLS users continues to grow dramatically, an inertia effect that will make it likely that major revisions will be infrequent. Much of the operational effort in website development will then go into occasional small changes and additions. Rather than being solely driven by technical developments, these incremental improvements should be guided by a customer-oriented continuous-quality-improvement perspective that relies on user feedback using techniques such as email content analysis, transaction log analysis, and periodic usability testing. This process may tune existing features, incorporate selected new features on an ad hoc basis, and then fully incorporate new features and revise existing features in a scheduled complete re-design. What follows is a set of recommendations for changes at both the incremental and major-revision levels. The first set (labeled A1-A4) includes incremental changes that can be easily made to the existing homepage and the second set (labeled B1-B4) includes more substantial changes more appropriate to a major redesign event.

A1. Add scenario option. Fully implementing the visual metaphor with the information abundant widgets may be too ambitious but it should be rather easy to add a scenario option to the existing design. We suggest that BLS take the scenarios developed in this project, distribute them to BLS staff for comment and suggestions for additions. It is important that as staff make suggestions for additional scenarios that they also provide a list of the best web pages that satisfy the scenario.

A2. Cluster graphic icons on home page. The current array of icons is unordered. It costs nothing in screen real estate or debugging/testing time to more consciously cluster the existing options on the home page. We suggest the clusters used in the prototype: data, summarized data (was termed fast info on the prototype, which should be renamed), and publications. One possible clustering would group data, regional information, and surveys and programs (as a middle column), economy at a glance, publications, and keyword search (the left-most column), and BLS Mission, Other statistical sites, and what’s new as a third column. This is an easily implementable incremental change.

A3. Restructure selected pages. The transaction logs in both phases of work illustrate that the Occupational Outlook Handbook and News Releases draw far more interest than most of the top-level choices on the home page. On the other hand, BLS Mission and Other Statistical Sites get relatively little attention. It may be prudent to move OOH and NR up to the home page and move BLSM and OSS down a level.

A4. Collapse hierarchy on the textual links on home page. Another change that would be easy to implement and may improve access at the cost of a bit more screen complexity is to add subchoices for each category in the text-only portion of the screen. This would not take much more screen real estate but save frequent users one click and guide new users with strong cues about where links will lead. One layout option is as follows:

Data: Most Requested Series, Selective Access, News Releases, Series Report, FTP

Regional Information: Boston, New York, Philadelphia, Atlanta, Chicago, Dallas, Kansas City, San Francisco

Surveys and Programs: Employment/Unemployment, Prices, Compensation, Productivity, Emp Projections, International, Other

Publications: Occupational Outlook, News Releases, MLRO, Research Papers, Issues, CWC

BLS Mission: Statement, Sr. Mgmt, Fellowships, Jobs, Intn’ Training, Strategic Plan

Other Statistical Sites: FedStats, National, International

Note: the main category could be displayed in a larger font than the subcategories, which could also be indented.

Note: EAG could be eliminated as a Data subcategory since it is a major category on the home page. Likewise, contacts could be eliminated from Publications as it is a major category. A full review of the labels should also be made to find more task-oriented terms.

B1. Create and disseminate development plans. Develop and publicize tactical and strategic plans for incremental and major revision levels of development. Encourage BLS staff and end users to make suggestions and comments. One caution is to not over-react to small but vocal groups--be user oriented, but not user driven. Include systematic data collection and triangulation to inform changes (e.g., transaction log data, user feedback, email content analyses, usability testing). Such planning reflects the organizational discipline present in other key parts of the corporate culture at BLS where regular surveys, reports, and releases are expected and depended upon.

B2. Implement a graphical design with primary data on the homepage. Consider a new graphical design that clusters data, summarized data, and publications, and brings key primary data to the home page. The design tested here is a good starting point for such a design. It makes sense to provide such a design as an alternative (with a separate URL) and invite selected users to try it out and provide feedback while the existing site continues to handle the mainstream activity.

B3. Improve search. Add simple and advanced search options that allow field/sector delimited options. Offer alternative terminology or ask for clarifications, and spell check entries. Provide opportunities for relevance feedback (more like these). Provide richer results displays with user-controllable options for filtering and sorting.

B4. Improve vocabulary. Conduct a complete study of site vocabulary with the aim to integrate general vocabularies with statistics-specific vocabulary. Work to develop a statistical thesaurus that all government agencies can use. Explore interface mechanisms for displaying alternative vocabulary.

B5. Add user help and education. Develop usage-sensitive, on-demand help for services that transaction logs and user feedback show are most troublesome for users. Provide glossaries, tips, and suggestions for users that help them better find and use statistical data.

B6. Work toward interoperation with other agencies. As more novice users visit the site, many of their needs will be met by statistical data at other agencies. To provide a more fluid, user-oriented environment, develop links to other agency data at the leaf level of BLS pages, develop vocabulary and table mapping services between different agencies (in lieu of cross-agency standards).

B7. Consider other new services.

B7.1 Frequently asked question (FAQ) services can be helpful to both new and experienced users. The current contact information is laudable but providing services such as FAQs that help people help themselves before contacting BLS staff seems a win-win action.

B7.2. A related service is online reference, which is difficult to provide but some examples of primitive services have emerged in other government agencies (e.g., Energy Information Administration’s Energy Data Query System leads users through a series of selection menus to define a desired data set [http://www.fedstats.gov/index20.html] and the National Agricultural Library’s AgNic services provides a reference template users complete that is then automatically processed with the possibility of professional librarian assistance [http://www.agnic.org/orsp/].)

Transaction Log Analysis

In the first phase of this work, a process for analyzing summary data and sequential data for transaction logs was developed and applied for selected months of 1996 (see Seeking Statistical Information in Federal Websites: Users, Tasks, Strategies, and Design Recommendations Final Report to the Bureau of Labor Statistics, Hert & Marchionini for details). The intention of this second phase of work was to replicate and extend the analyses to 1997 data to verify usage trends. Because web technology is developing rapidly, several changes influenced the work. First, the BLS website itself evolved (new features such as K-12 Educational Resources were added and some web pages were combined or moved). This affected a few of the sequential analysis codes as well as some summary data page counts, and the server migrated to Microsoft NT, which affected performance (adversely during the transition and positively once the transition was complete and new procedures established). Second, BLS adopted a commercial statistical tool for its website (Microsoft Usage Analyst), which provides many of the summary data recommended in the first phase but uses different bases for counting hits and requests (e.g., by calendar month rather than by weekly period as the custom BLS summaries did; slightly different domain categories than used in our previous work). See Cooper (1998) for a recent overview of considerations for instrumenting web-based retrieval systems.

3.1 Procedure

As in the previous phase, summary files were obtained from Labstat in order to compare overall web page usage over several intervals. In this phase, summary files for September and November 1997 were obtained. In a discussion with Labstat staff (January 23, 1998) it was noted that the conversion to the NT server may have influenced summary data since the server was down periodically during the September/October transition period. These files were examined manually and page counts (hits) for the top-level services and for the frequently-accessed services were added to the spreadsheet created in the first phase. Two new columns (September and November) were created for raw counts and four new columns were created for percentage changes (March to September denoted as B to C; September to November denoted as C to D; March to November denoted as B to D; and December 96 to November 97 denoted as A to D). Each of these intervals represents 31 days of activity (the March files were 32 days and were adjusted by a 31/32 factor).

A Comprehensive site analysis report for the month of February 1998 produced by Microsoft Usage Analyst was also obtained. These reports ar now made available through the BLS Intranet to BLS staff, an excellent development. This report provides much of the same type of data that the custom BLS summary reports produced, but differs enough to make direct comparisons in chart form difficult. General comparisons and observations are included in the results section.

Additionally, the complete http server log for October 1997 was ftped and the custom C programs developed for phase one were applied to parse the logs into sessions. These sessions were then examined using the Sequence program and selected comparisons were made based upon the sequential data. See the phase one final report for details on how logs were parsed, the coding scheme used, and what analytic procedures were applied.

3.2. Results

Results are divided into two groups: summary and sequential analyses. The analyses replicated most of the analyses done in the first phase work and comparisons across the various time periods were made whenever feasible.

3.2.1. Summary trends December 1966 to November 1997 (Dec 96, March 97, Sept 97, Nov 97).

The rapid growth in BLS website usage continued in the final quarter of 1997. Table 3.1 summarizes the page activity for the top-level services (pages) on the BLS home page. The data demonstrate that overall usage on these top-level sites roughly doubled over the 11 month period. For the purposes of rate of change comparisons within each page category, caching rates are unlikely to influence results (since the rates would presumably be stable for the same page over time). It is interesting to note that the highest rates of usage are for publications (deeper analysis indicates Occupational Outlook Handbook and News Releases show greatest usage as well as increases) and keyword searches, services that may be used by more general audiences; while there is a decrease or small growth in ancillary information such as BLS mission and other statistical sites (strongly suggesting that these options be moved down in the organizational hierarchy). The core data services (BLS data) show good growth but considerably less than the digested information in publications. One obvious explanation is that there are many more causal users coming to the site than sophisticated users (of course this has to be so since there is a much smaller number of statisticians, economists, researchers, journalists, accountants, and other specialists to attract as new users than the general citizen pool).

Table 3.1. BLS Top-level Service Activities December 1996 to November 1997.

These same trends can be seen in Table 3.2 that summarizes hits for high-frequency pages over the eleven-month period. This table displays page activity for pages that receive high volumes of accesses (1000 hits in the December 1996 or March 1997 periods was used as the high volume threshold). Pages with dramatic changes over the period may reflect site reorganization (e.g., oreother and contact information show huge increases from the December base and this is likely due to moving links to these pages to the BLS homepage), or out of date material (e.g., large decreases in news releases for 1996 CPI data). There are four very high-usage pages in this table (more than 10,000 hits in December 1996), cpihome, ocohome, sahome, and top20. The Occupational Outlook Handbook had the largest overall increase in usage (234%) over the eleven months, the CPI homepage had the second largest increase (152%), while the data-specific pages top20 (most requested series) and selective access showed increases below the overall average increase for all pages, 93% and 83% respectively. In addition to the increases in interest in news releases, high-profile indexes such as CPI, general descriptive information, and occupational information, there were generally large increases in access to employment information. These data tend to support the hypothesis that the BLS audience continues to broaden with many more users looking for interpreted/digested data rather than raw data. This has implications for site organization and the interface.