Using Decision Making Theories to Investigate Human Errors and Difficulties in Human-Information Retrieval System Interaction

By Yan Zhang

Topic description

Human errors and difficulties with operating computer systems are of particular interests for researchers studying human computer interaction (HCI) and artificial intelligence (AI) in that achieving deeper understanding of mechanism of human errors can help researchers fulfill their commission of building user-friendly artificial systems. Information retrieval systems, as a special type of computer systems, recently become more and more prevalent in people's daily life and serving more and more purposes.

Typical information retrieval (IR) systems include library online catalogs (OPAC), commercial databases, such as Psychinfo, ERIC, and Web of Sciences, Web search engines and so on. Even the World Wide Web itself can be regarded as a huge IR system at some aspects. With too much information available, people tend to depend more on IR systems to search for information, make judgment, and then use information. Given the important role that IR systems play, how to improve human experience with various IR systems is a focus for researchers in the information science field. Researching human errors and difficulties with current IR systems will shed light on people's behavior with IR system so as to contribute to bridging the gap between people's preferences and needs with functions and services provided by IR systems.

In this paper, I will look at documented human errors and difficulties exhibited in the process of human-IR system interaction from decision making point of view. People's interaction and negotiation with IR systems can be viewed as a constant decision making process. Human beings will make similar mistakes in this process as in others, and looking at biases and heuristics that people use in regular decision making scenarios can help understand the errors that they make in the information searching process. By identifying and diagnosing errors properly, we can isolate misconceptions and missing knowledge, therefore to gain a deeper understanding of the nature of human-IR interaction (Borgman, 1987). Meanwhile, it also can inform the design of future IR systems that are intuitive for common people to use and can assist people in avoiding bias and preventing errors.

Related topics

-- Dual-Process Theory. The theory argues that human reasoning system consists of two sub-systems. System 1 encompasses primarily the process of interactional intelligence. It is automatic, largely unconscious, and relatively undemanding of computational capacity. System 2 is a controlled processing. It is analytical and requires more computational power (Stanovich, 1999). When the information structure in the environment does not match with the ecological reasoning process of system 1, people tend to make mistakes and in these cases, system 2 should be activated to do analytical thinking, thus to override system 1 when judgment and decisions need to be made. The difference and relationship between system 1 and system 2 can be used to explain people's misconception about the size of results that will bring back by IR systems by adding logic AND to search queries. A typical behavior people show in the search process is that: adding word(s) or using Boolean operator AND to resolve zero-hit problem. In other words, narrowing was used by subjects when no hits were obtained (Marchionini, 1989; Wang, Berry, and Yang, 2003). Dual-process theory can be used to explain this type of error in the sense that in daily environment, adding more words and AND often connotate the perception of "more". But in information retrieval systems, AND impose more restrictions on search conditions and thus will lead to less results.

-- Availability: is a type of bias implicated by the fact that people often judge probability by thinking of examples. When a symbol or object is more familiar and more readily available, they tend to make the judgment that it is more widely used or more prevalent in the real world. A typical example of availability in information search is: when people decide which operator to use in the current system, they often carry over operators from other system that they had used (Jasen, Spink, and Saracevic, 2000).

-- Attentional bias: is a failure to look for evidence against an initial possibility, or as failures to consider alternative possibilities. A well documented difficulty that people have with IR system is that it is difficult for them to recover quickly from errors based on system's feedback; rather they keep trying the strategies that did not work in the first place. Attentional bias can be used to make sense of this phenomenon.

-- Simple heuristics vs. multi-attribute decisions: it is observed by Mathews and colleagues (1983) that the majority of searches were simple, specifying only one filed or data type to be searched; the advanced search features were rarely used. Other studies also found that people tend to use simple quick search more than advanced search (Tenopir, et al., 2004). In information searching process, people tend to use simple heuristics, even when a multiple attribute decision making scenario (multi-entry form in advanced search) was provided. In other words, people rarely use multiple cues in retrieving information.

Research problem

I have two research goals:
1. Collect human errors and difficulties that have been identified by other researchers in the current body of IR literature. User study has been conducted in Information Science in the past 30 years and many difficulties people have during their interaction with IR systems were documented. But a systematic collection and consolidation of the findings about human errors and difficulties has not been done yet. Thus, in this paper, I will try to collect together as much as empirical findings of human errors with IR systems. Also, I am interested to see that after 30 years research of user behaviors, are people still making similar slips or mistakes and does user study really have helped improve people's experience with IR systems.

2. My second goal is to arrange the empirical findings about human errors and difficulties with IR systems into a meaningful knowledge structure so as to improve our understanding of the fundamental reasons of these fallacies. Knowledge about decision making theories would help me construct the knowledge structure on a more solid foundation.

Apologists point of views

The philosophical foundation underlying user studies in information retrieval and more broadly in the HCI field is the argument held up by Apologists, who accept the notion of human irrationality resulted from discrepancy between descriptive and normative theories but believe that adapting the world to human beings' cognitive machinery could reduce the discrepancy and enhance human performance. As outlined in the previous section, the main goals I want to achieve in this paper are collecting documented human errors and difficulties with IR systems and making sense of them based on various biases and heuristics inherent in the human reasoning process as suggested by decision making theories. I would not propose decisive solutions for these errors and difficulties per say, but the process of identifying, categorizing, and making sense of the identified errors would to a great degree inform the design of easy-to-learn and easy-to-use IR systems.

Reference

Baron, J. (2000). Thinking and Deciding. New York, USA: Cambridge University Press.

Borgman, C. L. (1987). The study of user behavior on information retrieval systems. ACM SIGCUE Outlook, 19(3-4), 35-48.

Jansen, B. J., Spink, A., & Saracevic, T. (2000). Real life, real users, and real needs: a study and analysis of user queries on the Web. Information Processing and Management, 36, 207-227.

Marchionini, G. (1989). Information-seeking strategies of novices using a full-text electronic encyclopedia. Journal of the American Society for Information Science, 40(1), 54-66.

Matthews, J. R., Lawrence, G. S., & Ferguson, D. K. (1983). Using Online Catalogs: A Nationalwide Survey. New York: Neal-Schuman.
Stanovich, K. E. (1999). Who is Rational? Studies of Individual Differences in Reasoning. New Jersey: Lawrence Erlbaum Asspciates, Publisher.

Tenopir, C., Wang, P., Pollard, R., Zhang, Y., & Simmons, B. (2004). Use of Electronic Science Journals in the Undergraduate Curriculum: An Observational Study. Proceedings of the 67th Annual Meeting of the ASIS&T-- Managing and Enhancing Information: Cultures and Conflicts (pp. 64-71). Medford, NJ: Information Today.

Wang, P., Berry, M. W., & Yang, Y. (2003). Mining longitudinal web queries: Trends and patterns. Journal of the American Society for Information Science and Technology, 54(8), 743-758.