USER UNCERTAINTIES WITH TABULAR STATISTICAL DATA: IDENTIFICATION AND RESOLUTION

 

Carol A. Hert and Naybell Hernández

Syracuse University

 

 October 1, 2001

Final Report for Purchase Order #B9J03235

 

1.         EXECUTIVE SUMARY

1.1. Study Objectives

United States government services are increasingly becoming Web-based, creating opportunities to make potentially useful, even vital, information and services more easily accessible to the citizens than in the past. This opportunity has challenged Federal agencies as they work to provide information and services that are easy to use and understandable to an extremely diverse constituency. Federal mandates requiring agencies to provide "universally usable" information and services have added further impetus to resolving the challenges.

The statistical agencies have been addressing these issues via a variety of strategies and approaches. The FedStats website and its related planning and research development activities has been one venue. National Science Foundation funding for projects associated with statistical digital government has been another. The work reported on here was conducted in conjunction with an NSF-funded project investigating statistical information in tabular format.

Enabling universal access and usability of statistical tables can be modeled as a process in which a user with an information need comes to a system in order to locate and then use a table or tables of interest. The NSF project developed an integrated approach to that process. Several specific technologies were developed to support this process, each of which was designed to incorporate a rich understanding of user behavior that the project has developed. The specific piece funded under this BLS purchase order concerned user understanding of tables and the extent to which metadata could be used to support enhanced understanding.

Specifically, this project addressed the following questions:

·         What questions and uncertainties do users have when investigating the statistical tables used in the NSF project?

·         What are the answers to these questions?

·         To what extent is metadata available to answer the questions?

·         How do the questions, question types, and answers map to the XML DTD developed by the NSF project to support the Table Browser?

As a result of the investigation of these questions, a number of issues related to metadata creation and use were identified and a set of recommendations developed.

1.2. Findings and Recommendations

Users had a variety of uncertainties when investigating tables. The majority of these related to definitions of terms, categories of variables, etc. A second important class of uncertainties was that concerning rationales for why certain things were done, reported in certain ways, etc. Other uncertainties related to the structure of the tables and lack of information on various aspects of the tables. Users also provided a wide variety of suggestions and complaints about the tables.

Answers were found for all user uncertainties by searching relevant documentation and asking experts. Questions concerning rationales were difficult to resolve through existing documentation while other answers were found in the documentation.

The uncertainty categorization scheme developed in the project can serve to categorize questions in future studies in which the goal is to map to metadata sources and specify tool implementations.

Perhaps one of the important implications of this study for metadata design will be the provision of some notions of how to translate users’ uncertainties into metadata and metadata into functionality features of an information system. In order to scale the results of this project, it will be necessary to understand the processes by which a user uncertainty can be mapped to a potential answer and then potentially presented via the interface tools. A number of issues related to both the uncertainties and currently available metadata were identified. These include the potential uniqueness of answers needed to respond to user uncertainties, the specificity of answers provided, and the lack of easily retrieved information from documentation (due to lack of encoding within documentation).

One of the obstacles experienced during the project was the incomplete development of existing DTD’s and their lack of compatibility with project needs and this project suggests approaches to further development work.

This work might be furthered with the following additional research:

·         Expand the identification and coding of user uncertainties to additional tables in order to further validate the coding scheme, potentially begin to determine relative frequencies of uncertainty types.

·         Test the extent to which the Table Browser or other tools that incorporate relevant metadata are able to resolve user uncertainties.

·         Conduct document analyses to determine the effort involved in resolving user uncertainties with existing documentation.

The most obvious area in which further development of applications exists is in the area of metadata encoding and DTD development. As the statistical community continues to disseminate its information electronically, it will become ever more critical for the metadata behind the data to be easily available for users and applications. The most logical approach is would be to encode it in structured and standardized formats. Some metadata already exists in this form (such as data dictionaries) but technical documentation does not. XML has also become the standard of choice for encoding information. Thus the following recommendations for development and for agency action seem relevant:

·         Continue efforts to develop metadata standards.

·         Build relevant XML DTD’s for agency information.

·         Investigate mechanisms for ensuring compatibility of DTD’s across document types and agencies.

 

 2.        PROJECT OVERVIEW

United States government services are increasingly becoming Web-based, creating opportunities to make potentially useful, even vital, information and services more easily accessible to the citizens than in the past. This opportunity has challenged Federal agencies as they work to provide information and services that are easy to use and understandable to an extremely diverse constituency. Federal mandates requiring agencies to provide "universally usable" information and services have added further impetus to resolving the challenges.

The statistical agencies have been addressing these issues via a variety of strategies and approaches. The FedStats website and its related planning and research development activities has been one venue. National Science Foundation funding for projects associated with statistical digital government has been another. The work reported on here was conducted in conjunction with an NSF-funded project investigating statistical information in tabular format.

Enabling universal access and usability of statistical tables can be modeled as a process in which a user with an information need comes to a system in order to locate and then use a table or tables of interest. The NSF project developed an integrated approach to that process. Several specific technologies were developed to support this process, each of which was designed to incorporate a rich understanding of user behavior that the project has developed. Figure 1 represented the larger project. The specific piece funded under this BLS purchase order concerned user understanding of tables and the extent to which metadata could be used to support enhanced understanding. In Figure 1, the work of this project is contained within the component at the far right, entitled the Table Browser.

FIGURE 2.1: INTEGRATION

Specifically, this project addressed the following questions:

·         What questions and uncertainties do users have when investigating the statistical tables used in the NSF project?

·         What are the answers to these questions?

·         To what extent is metadata available to answer the questions?

·         How do the questions, question types, and answers map to the XML DTD developed by the NSF project to support the Table Browser?

As a result of the investigation of these questions, a number of issues related to metadata creation and use were identified and a set of recommendations developed.

2.1.  Universal Usability And The Role Of User Understanding

Shneiderman (2000, p. 85-6) has framed the universal usability challenge as having three components: 1) the need to support a diverse technology base, 2) the need to provide access to diverse users with diverse skills and tasks, and 3) the need to bridge user knowledge gaps. It is the second and third aspects of universal usability that are the focus of this project, as appropriate technological solutions rest, to some extent, on the characteristics of users and their needs.

The world of Federal statistical information is a challenging one for most users who must navigate a labyrinth of agencies (over 70 at the Federal level), interpret very distilled information (numbers, often presented in formats such as tables that are difficult to use, and, who, to use the information appropriately, may need to understand very specific details of the data collection and analysis that generated the numbers. Statistics and statistical information are not easy to use for the layperson. Most of us are not taught in school how to read or work with statistics, resulting in low statistical literacy for the general population (Moore, 1997). Statistics are often highly distilled (as a specific statistic, a table or a time-series of statistics), have been produced through complex statistical and mathematical procedures (such as sampling design, weighting), and utilize specific and sometimes arcane definitions of concepts and variables (with associated jargon). All these represent potential sources of misunderstanding and barriers to use.

The work reported here is focused on tables. There are several rationales for this focus. Although there is a substantial effort given to graphical representations of data (e.g., Carr, 1998; Wainer, 1997; Wilkinson, 1999), tabular display treatments are treated minimally at best (e.g., Hall, 1943; Walker and Dorost, 1936). Tables are a common conceptual and presentational structure by which statistical data are stored and represented. Data in tabular form are often the starting point for additional depictions (such as graphics or analytical reports) and contextualize specific numbers. Tables are, however, difficult to find, interpret and use. Most commercial search engines do not index the contents of tables nor can they retrieve that information and often do not even identify the existence of tables within a text. Once a table is found, users face succinct labels and highly distilled numbers and may wish to perform comparisons and calculations that are difficult in static tables. The ubiquity of tables along with the associated challenges suggest that research into improvement of table retrieval, interpretation, and use has the potential to significantly improve access to data produced by statistical agencies.

Providing universal access to tables is both a technical as well as a challenge concerning user understanding and modeling of that understanding. In this project, we first identified user questions then investigated how to model them as metadata, specifically in terms of metadata elements available within the NSF’s project’s DTD.

2.2. The Role Of Metadata In Location And Understanding

Metadata is an often ambiguous and nebulous term and is used variously in different communities. Dempsey and Heery (1998) define metadata as information that enables one to manage and use the data/information to which they refer. This definition highlights two key points, that metadata are defined within a context (there is no one set of metadata associated with a set of data), and that they are information that supports usage. Some of the purposes which metadata may support are information/resource discovery, administrative uses such as tracking terms and conditions of use, the context of creation, and unique identification of objects (see Bearman (1996) for a discussion).

Within the statistical domain, metadata may include subject heading schemes to support resource discovery (such as the list of headings employed by the American Statistical Index (published by the Congressional Information Service) and HASSAT (from the University of Essex), codebook information, survey instruments and related documentation, as well as reports and other documentation produced by survey methodologists about data collection strategies, analysis of past survey efforts, etc. (Dippo and Gilman, 1999).

2.2.1 Past empirical work on metadata use

The study of user interaction with metadata is not completely unknown. Within the traditional library and information science domain, there is a thread of research most commonly known as relevance judgment research that investigates how users make judgments on the relevance (variously defined and operationalized) or potential relevance of information units. Traditionally those information units have been articles and books, and users examine representations of those units (such as citations, which represent the metadata in this case) and indicate those they consider relevant or non-relevant. Users are asked about the criteria they are using in the judgments and how they make those judgments. The intent of this line of work has been to understand the phenomenon of relevance judgment, provide typologies of relevance criteria, and in some cases to suggest enhancements to the representations of the information units (See for example, Park (1993) and Barry (1994).) For example, if users indicate that having information on the chapter titles in a book is helpful, it may be suggested that such information be added to the description of the book.

The vast majority of work of this type has looked at books (using information on records in online library catalogs) or articles (using periodical databases with or without abstracts). Users may be asked to examine different representations of the same item such as a citation, a citation with an abstract, or the item itself. Only recently have other types of information entities such as maps (Gluck, 1996) and meteorological data (Schamber, 1991) been considered.

In the domain of statistical information seeking, the author and Bosley (as reported in Hert, 1999) have been investigating how experts and other users employ metadata within codebooks (in this case, from the Current Population Survey) as they choose variables for analysis. He and Gey (1996) allude to the value of the codebook data in choosing variables in a paper that discusses a system that might facilitate browsing of such data.

In general, these studies have worked from existing metadata associated with information entities back to user behavior with that metadata. Such an approach limits our ability to see what metadata might actually resolve user uncertainties since we have not begun with those uncertainties. Thus in this project we began with identifying these uncertainties then moved on to the potential of metadata to resolve them.

2.3.  Metadata and XML

To make metadata accessible in an automated environment, it needs to be represented and encoded so that software can identify appropriate metadata components and retrieve them. In the last several years, there have been a variety of efforts to encode statistical metadata. The International Organization of Standardization (ISO) has developed a standard, ISO/IEC 11179. The Inter-University Consortium for Political and Social Research’s (ICPSR) has a program entitled the Data Documentation Initiative (DDI) and an UN/ECE Work Session on Statistical Metadata (see for example: http://www.unece.org/stats/documents/2000.11.metis.htm) has been actively engaged in discussions.

For this project, the DDI encoding was used to encode tables and metadata. This choice was made because the DDI has a specific encoding designed to encode tables and project personnel had expertise with this encoding. However, the DDI encoding was not fully compatible with project needs and some modification was done. Details on the encoding of tables and metadata using the DTD can be found in Marchionini and Mu (2001).

 

3.     METHODOLOGY

3.1.  Investigating User Uncertainties

3.1.1 Overview

The investigation of user uncertainties involved several different activities. First, a set of respondents interacted with specific tables. The research team then mined transcripts of their sessions for uncertainties, questions, complaints, and suggestions. Answers to all questions were found by the research team. Questions and complaints were categorized.

Eleven people participated in the study. Each participant viewed a total of three tables in a mix of electronic and paper formats. After an initial unstructured period in which each participant was instructed to examine the tables, the researchers asked a series of questions about the participant’s understanding of each table. Demographic information on each respondent was gathered via a self-administered questionnaire at the beginning of the interview.

The team created records of each participant’s comments, responses to interview questions and other data (such as which component of a table was the focus of the comments). Analysts reviewed the records, and extracted uncertainties, suggestions, and complaints. The team coded the resultant lists using the schemes described below.

The team also searched for answers to the specific individual questions (rather than for the derived categories of questions). Answers were sought within the actual table and accompanying text (e.g., footnotes), related documentation (in both electronic and paper format), and in some cases, by consulting experts within the agencies that produced the tables.

3.1.2 Data collection

3.1.2.1 Table Selection

For the study, the team selected four tables from a set of tables nominated by agency partners in the project (tables available in Appendix 1). The four selected differed in their content, complexity, size, and formatting styles. The intent was to provide sufficient variety while still assuring that the researchers could provide the tables in multiple formats as well as be able to find answers to user questions. While this has implications for the generalizability of the results, consensus on what constitutes important differences in table format is generally lacking even among experts on statistical presentation. Additionally, it was important to show users real instances of tables, rather than artificial constructions, in order to identify actual questions.

All participants reviewed a set of three assigned tables about which they would answer questions. Some of these tables were presented in paper format, others in electronic format according to a researcher pre-defined set of combinations of the four tables and the two formats. All combinations had at least one example in each of the two formats (paper and electronic) to account for any difference that might occur when using different presentation media. One table was only available in electronic format.

3.1.2.2 Selection of Participants

Study participants were solicited through calls for participation posted in the university library’s government documents section. The researchers assumed that visitors to this section of the library would be more likely to be interested in and potentially knowledgeable about government information and statistics. Potential participants were screened for previous use of government statistical data. The study had a total of eleven participants (three males and eight females). Characteristics of the participants are indicated in Table 3.1. Each person was paid 25 dollars upon completion of participation.

Most of the participants were undergraduate students at Syracuse University and all of them reported using computers on a daily basis and to be highly experienced web searchers. One participant also reported to have been exposed to statistics and to have used tables from government websites at least occasionally.

Potential participants were recruited throughout the data collection period until the researchers determined that theoretical saturation on the uncertainties was achieved for each table (no matter in what format the table was presented). Theoretical saturation is reached, among other things, when no more relevant data seems to emerge regarding a category or variable (Glaser and Strauss, 1967). In this case, interviewing stopped when no new uncertainties were elicited for a given table. The eleven interviews is a reasonable sample; as pointed out by Schamber (2000), as few as ten interviews can be expected to provide representative results when eliciting cognitive perceptions purely for exploratory purposes.

3.1.2.3 The Interview

Once a preliminary questionnaire was developed, we started the pretest process with a total of six respondents (with their data not included in analysis). After each interview the questionnaire was revised. The final questionnaire consisted of two sections. The first session contained eleven demographic questions that asked participants about their frequency of computer use, web searching experience, statistical background, statistical packages used, as well as frequency of use of some specific statistical tables. Factual questions allowed the researchers to verify the appropriateness of each participant for the study as well as to collect data to classify each of them based on background information. The second set of questions (the loop section) contained twelve general questions intended to elicit what questions/uncertainties participants faced during exploration of government statistical tables. The underlying purpose of these questions was to determine what kind of metadata and its content would need to be accessible during table usage so that users of these types of tables could better understand the meaning and significance of the data presented. Appendix 2 presents the interview guide.

The choice of these particular questions was supported by defined characteristics of good tables such as the ones described by (UN/ECE, 1992); by the standards on the sources, methods and procedures of statistics as defined in Walker and Durost (1936); as well as by researchers’ own evaluations of each of the tables to be used in the investigation. Some of these standards point out the need for titles to be constructed as an aid to the reader in understanding the facts, for the source of the data to be indicated, as well as for the indication of unit of measure used and the methods used to compute the data. It is based on these and other standards that our specific questions emerged. Examples of such a questions include but are not limited to: ‘Does the title help you to understand the facts on the table?’, ‘Is there anything in the way the table, its rows or columns are organized that makes the table more difficult to understand?’, ‘Can you tell from the information in the table how any of the statistical measures were calculated? ’.

A member of the team conducted interviews in person at a time convenient to each participant, over a period of three weeks. All the interviews were limited to ninety minutes since during pre-test sessions this amount of time proved to be sufficient for coverage of three tables and not overly tiring.

The interviews were performed following the interview guide, but researchers exercised some flexibility in order to give the researchers more control of the situation. This control allowed the interviewer to clarify terms that were unclear for the participants and to probe for additional information (Frankfort-Nachmias & Nachmias, 1996). All the interviews were taped-recorded and the transcripts were utilized as the main source for the subsequent content analysis.

3.1.3 Data Analysis

Data analysis had two components. In one component, specific answers to each user question were found. These answers were then forwarded to the project’s system design team for inclusion in tools designed to support manipulation and usage of the tables. The second component was to categorize the questions in order to better understand user’s uncertainties and how they could be resolved.

3.1.3.1 Finding Answers to Questions

Table 3.2 lists all questions asked (by table), and the frequency of asking. Due to the length of the answers, the full table is presented as Appendix 3. Researchers searched for answers to each question in a variety of paper and online sources. They first examined the table itself for answers (e.g., the footnotes in a table), then examined associated technical documentation. For online tables, links present within the table were also followed. The researchers did not do general searches on the respective websites, as the assumption was that users of tables would not be likely to do so. If no answer was found, a member of the team contacted the tables’ experts from the government agencies that were working with the research team. These experts also confirmed the answers that had already been found.