Cognitive Reflection and Information-Seeking Behavior:
An Exploratory Study

 

 

 

 

 

 

 

 

 

 

 

Diana Ascher

IS 291B

03/17/14

Introduction

One of the most pervasive myths about information access is the idea that anyone with a connection can find, use, and learn from all of the information existing on the Internet (Papparassi, 2002; Manovich, 2010). Certainly, Internet access provides an opportunity to search for and encounter information that could be more difficult to obtain otherwise. However, we know that online information seeking is limited by constraints arising from personal, role-related, and environmental contexts (Wilson, 1999). Even if the Internet were deployed in such a way as to place the totality of the world’s information at the fingertips of every individual, there would remain limits to people’s online information-seeking abilities.

Of all the impediments to online information seeking, individual physiological, affective, and cognitive constraints have received less attention than other phenomena in the field of Information Studies. Similarly, though behavioral scientists have studied the biases and heuristics underlying concepts such as impatience and satisficing, these investigations have focused primarily on consumer choice and judgment. Among the behavioral science measures and insights to derive from studies of decision making, one relatively recent test uniquely identifies the propensity of individuals to have confidence in their impulsive “gut” responses. The three-question Cognitive Reflection Test (CRT) developed by Shane Frederick is a predictor of an individual's tendency to recognize gaps in knowledge and take action to fill them (Frederick, 2005).

Given that a large proportion of online information seeking is geared toward bridging information gaps, a measure that predicts one’s ability to recognize and act on a lack of knowledge should be correlated to information-seeking behavior in some way. If a correlation exists between, say, low CRT scores and “poor” information seeking, we will be able to apply behavioral science techniques to help people overcome the biases that constrain their information-seeking success. To date, the CRT has not been studied in conjunction with information-seeking behavior.

Therefore, this paper proposes an exploratory, quasi-experimental study to determine whether a correlation exists between a particular cognitive bias measured by the CRT and information-seeking behavior. If, indeed, such a correlation exists, then Information Studies researchers will have greater insight into attributes that underlie the information-seeking behavior of a subset of the population, which has implications for the ways in which we assess information-seeking skills and design information systems, interfaces, algorithms, and tools.

Problem Statement

The ostensible promise of the Internet—the world’s knowledge, free, and at one’s fingertips—fails to account for the myriad constraints that limit people’s access to and use of information. Much of the research in Information Studies focuses on inequalities in information access in an effort to support practices and policies geared toward more balanced discourse and free exchange of knowledge. However, even if the Internet were to place the totality of the world’s information at the fingertips of every individual, there remain limits to people’s online information-seeking abilities.

Humans’ lack of omniscience—that fact that we must seek information—necessarily demonstrates that the information search process operates within certain constraints. Constraints to contemporary online search originate from multiple domains. For example, in the political arena, laws dictate what content should be available to whom. The economics of information creates inequities in access as well as in the ability to contribute to the corpus of online content. Social norms, too, constrain engagement online. In addition, the technology employed to make information available and “findable” involves query parameters and algorithms that determine the information that is included in and excluded from search results. Similarly, cognitive biases contribute to limitations facing information seekers during the information-search process. Of the three contexts from which such barriers to successful information behavior arise—personal, role-related, and environmental—the personal context (composed of individual physiological, affective, and cognitive attributes) has received disproportionately little examination in studies of information-seeking behavior (Wilson, 1999; Case, 2012). Among these constraints is a set of cognitive biases that reside within every human to some degree, and which have been the subject of numerous studies in the fields of psychology, neuroscience, and behavioral science.

This paper proposes an exploratory, quasi-experimental study to determine whether a correlation exists between a particular cognitive bias measured by the Cognitive Reflection Test (CRT) and information-seeking behavior. If, indeed, such a correlation exists, then Information Studies researchers will have greater insight into attributes that underlie the information-seeking behavior of a subset of the population, which has implications for the ways in which we assess information-seeking skills and design information systems, interfaces, algorithms, and tools.

Literature Review

Researchers in Information Studies have studied the ways people go about their Internet searches, devising myriad models of information-seeking behavior, identifying user characteristics, and developing techniques to make visible information that otherwise tends to escape the notice of the vast majority of online information seekers. Much of the focus of these studies has been on access, information literacy, and user skills. While some of these studies have acknowledged the effects that time pressure, cognitive load, and other psychobiological constraints have on information processing, there has been little—if any—exploration of how information seekers’ awareness of information gaps affects their confidence in having retrieved the information necessary to cease the search process and turn attention to task execution or information reporting, nor how this awareness affects their search strategies. Further, most user studies in the field have been idiographic ethnographies with small sample sizes, which provide deep insight into a specific context, but are more difficult to generalize to other contexts.

Behavioral scientists have studied the biases and heuristics underlying concepts such as discounting, satisficing, risk assessment, and choice architecture, particularly with respect to consumer choice and judgment. However, the effects of cognitive biases on information-seeking behavior have been the subject of very few investigations outside the realm of consumer purchasing decisions.

Scope

This literature review focuses on information-seeking behavior studies conducted in the field of Information Studies, as well as information-processing studies in the domain of Behavioral Science. It is selective for research conducted in Information Studies and Behavioral Science on cognitive attributes of information seekers and decision makers, with a focus on online information seeking. The scope has been defined narrowly to highlight the ways in which Behavioral Science insights into and interventions to counter cognitive biases may inform research on information-seeking behavior, with the aim of encouraging further research.

Information

Competing perspectives on an appropriate definition of information result from varied assumptions about (1) intentionality of the user or communicator, (2) utility of the information, (3) physicality of the information (material, conceptual, observable), and (4) truth or accuracy (Case, 2012). However, Bateson’s description of “any difference that makes a difference to a conscious, human mind” remains above the fray (1972, p. 453). (For a thorough review of the various conceptions of information, see Case, 2012.) For the purposes of this research proposal, information is defined broadly, drawing on the characterizations offered by Bateson and Bates—and endorsed by Case—as that which is perceived and to which meaning is ascribed.

Information-seeking behavior

Information-seeking behavior begins with the recognition of the existence of an information need and ends when that need is believed to have been satisfied (Krikelas, 1983), or the search is abandoned. The information seeker may employ both formal and informal information sources to resolve this need, and is either satisfied or dissatisfied with the end result (Wilson, 1999). Theories of information-seeking behavior tend to classify the activities of information-seeking behavior as concerning immediate needs, whereas the activities that aim to resolve deferred needs are described as “information gathering” (Krikelas, 1983). Much of the research on information-seeking behavior has focused on specific user groups, such as high school students or CEOs, which makes it difficult to compare information-seeking theories (Leckie & Given, 2005). However, some general concepts have been noted.

For example, the types of information people seek are situational, as information needs are linked to time and place (Krikelas, 1983). People of different backgrounds have various levels of prior knowledge and a high degree of subjectivity (Weiler, 2005), which contributes to distinct information-seeking behavior to meet dissimilar needs (Prentice, 1980). Thus, a universal information-seeking behavior model is problematic to define (Leckie & Given, 2005), but overarching concepts regarding situational constraints can inform our understanding.

For the purposes of this paper, a definition of information-seeking behavior is drawn from Marchionini’s (1995) characterization, “a process in which humans purposefully engage in order to change their state of knowledge,” and the concept of a recognized lack of knowledge. Information-seeking behavior comprises the activities instigated by the recognition of a knowledge gap.

Cognitive reflection

Behavioral scientists rarely address the possible effects of cognitive abilities in their research on biases and heuristics such as ambiguity aversion, anchoring, endowment effects, time preference, risk preference, and probability weighting (Frederick, 2005). However, cognitive reflection may be a contributor to these behaviors. Frederick (2005) defines cognitive reflection as the “ability or disposition to resist reporting the response that first comes to mind.” The Cognitive Reflection Test (CRT) is a measure of this capacity. This concept derives from the work of behavioral scientists to describe a dual-system theory of cognition, in which humans have two cognitive systems: System 1 (quick, effortless, intuitive, and heuristic processes) and System 2 (slow, effortful, reflective, and rule-based processes) (Kahneman & Frederick, 2002). The CRT measures the ability of System 2 to monitor and, if necessary, override System 1 functioning. Loewenstein’s conceptualization of visceral decisions relates closely to Frederick’s notion of impulsive gut responses. “The defining characteristics of visceral factors are, first, a direct hedonic impact (which is usually negative), and second, an effect on the relative desirability of different goods and actions” (Loewenstein, 1996).  In the most comprehensive CRT study thus far, Toplak, West, & Stanovich (2014) explain that “this strong bias to default to the simplest cognitive mechanism—to be a cognitive miser—means that humans are often less than rational,” and indicate that the CRT explained more unique variance and was a better predictor of rational thinking than a set of measures of intelligence and measures of executive functioning (2014).

Knowledge gaps

Information-seeking behavior, then, implies an interaction between a person (or group) and the situated environment with the aim of ameliorating a state of uncertainty arising from the recognition of missing knowledge, or a lack of confidence in knowledge. Dervin describes an individual encounter with a discrepancy or lack of “sense” in a person’s environment as a state that arises within a person that might be filled by “information” or by some manner of “help” or “bridge” (1983, p. 156). Belkin (2005) refers to the knowledge gap at an individual level as a recognized anomaly in a person’s state of knowledge regarding a situation or topic, which he calls an anomalous state of knowledge (ASK). An individual may attempt to address the uncertainty by seeking information and subsequently determining whether the anomaly has been resolved; if it is not resolved, an updated ASK may prompt further search, or the search may be abandoned (Case, 2012; Belkin, 2005).

Knowledge gaps have also been identified at the group level, when a group of people is considered systematically deficient in what its members know compared to other groups (Chatman & Pendleton, 1995; Dervin, 1983; Savolainen, 1993). While this paper concerns individual information-seeking behavior, future research will examine cognitive reflection in the context of collaborative groups.

In Information Studies, the concept of the knowledge gap has been discussed in terms of “distressing ignorance” by Wilson, “rhetorical situation” by Bitzer, “anomalous states of knowledge” by Belkin, and “information gap” by Dervin. All of these descriptions concern a state of uncertainty that may motivate information seeking. To extend this thinking, lack of knowledge or lack of confidence in existing knowledge can be thought of as a precursor to the active pursuit of information. It is this premise that ties information-seeking behavior to cognitive reflection.

As Donald Rumsfeld famously explained at a U.S. Defense Department briefing in 2002,  “As we know, there are known knowns; there are things we know that we know. There are known unknowns; that is to say, there are things that we now know we don’t know. But there are also unknown unknowns; there are things we do not know we don’t know.^[1] While Rumsfeld received a fair amount of chiding for his explanation of the lack of evidence of weapons of mass destruction in Iraq, it is interesting to note that his comments followed a meeting with Nassim Taleb, a self-proclaimed “epistemologist of probability” and expert in risk engineering.^[2]

Linking decision making to information search, March (1994) explains that “the study of decision making is, in many ways, the study of search and attention.” His examination of decision making delves into the concept of frame rationality—that actors make rational choices based on both cognitive and evaluative beliefs. Differences in framing and access, as well as the varied ways in which decision makers judge information and incorporate it into their worldviews, result in different ideas about information relevance and value. Urbany et al. (1989) looked at consumers’ information-seeking behavior in terms of states of uncertainty. The researchers were surprised to discover that buyers with low “knowledge uncertainty” engaged in more effortful information-seeking behavior, and posited that this was “likely due to their knowledge about trade-offs between alternatives creating more uncertainty about which product to choose.” This finding is in line with recent work on metacognitive knowledge and decisions under uncertainty being conducted at UCLA.

Zipf’s Principle of Least Effort

People prioritize convenience over accuracy when they select sources for information search (Krikelas, 1983). Connaway et al. (2011) highlight the concept of convenience as the primary driver throughout the information-seeking process, and note the effects of time pressure on an individual’s choice of information source. According to Zipf (1949), people naturally manage their behavior to expend the least average amount of effort to accomplish their tasks. This idea has been extended into the realms of information seeking and problem solving as a form of “satisficing.” While Zipf’s principle is often considered in terms of “laziness,” the notion that cognitive reflection contributes to this prioritization may influence perspectives on whether laziness is an appropriate term for behavior influenced by a cognitive trait, as well as indicate opportunities to intervene in situations that would be best served by System 2 thinking, but in which people tend to act as “cognitive misers.” Kuhlthau (2004b) alludes to this idea: “The zone of intervention is that area in which an information user can do with advice and assistance what he or she cannot do alone or can do only with great difficulty. Uncertainty indicates a zone of intervention in the information search process.”

Correlating CRT and information-seeking behavior

In sum, both fields have zeroed in on a state of uncertainty as the feeding ground for impulsive behavior. However, neither group of researchers has tied the motivation for information seeking to the cognitive reevaluation of gut responses. The ability to “gut-check”—measured by the CRT—may be correlated to information-seeking behavior.

In Behavioral Science, researchers have long studied the role of cognitive bias in decision making. Because decisions are influenced by available information, this research translates well to concepts in information-seeking behavior. In an examination of avenues for improving decision making, Milkman et al. explain that “the most fruitful directions for researchers seeking to reduce heuristics and biases may be those predicated upon ‘some understanding of and hypotheses about people’s cognitive processes’ (2008).

Pertinent Models and Theory

Until recently, information-seeking behavior research was based on the idea of the “needy” seeker. A need was considered the precursor to active pursuit of information to meet it. Many of the information-seeking behavior models reflect this goal-oriented approach to understanding humans’ information behavior.

Kuhlthau’s model of the information-search process (ISP) depicts several stages of activity and affect, including identification of a source to meet a perceived need, recognition that information is needed, identification of the general topic, confusion and uncertainty, the gaining of confidence about the search; the collection of information; and closure, which can include either satisfaction or dissatisfaction with what is obtained. ASK and sense making approaches are both cognitivist in their origins and orientation.

Dervin & Nilan’s analysis of the paradigm shift from a system-oriented vision of Information Studies toward a focus on user experience summarizes the tensions in the field:  objective vs. subj
ective, active vs. passive users, 
transituationality (positivist) vs. situationality (context-dependent), 
atomism vs. holism, qualitative vs. quantitative. Taylor’s User-Values approach, Dervin’s Sense-Making approach, and Belkin Anomalous States of Knowledge approach are representative of this paradigm shift.

In addition, Sonnenwald’s framework for understanding information-seeking behavior suggests a shift from designing systems and structures to eliminate human intermediaries to creating densely populated information horizons to serve as rich sources of information. Sonnenwald notes the importance of thinking of human information behavior as a process involving cognitive, affective, and contextual factors. From this basis, Sonnenwald notes that information behavior is constructed in the midst of reflections on and evaluations of change. Unlike prior models, her framework includes the notable provision of information in contexts where no information need has been expressed, allowing for information sharing in anticipation of a yet-to-be-identified lack of knowledge. Information needs and the resources to sate them are determined socially and individually within what Sonnenwald calls an information horizon. Within information horizons, communication and collaboration tools are needed to share meaning and resolve knowledge gaps.

Significance

Given the abundance of content on the Internet and its perceived openness and accessibility, contemplation that engages System 2 cognition is essential to accurate assessment of online information. This study aims to address empirical gaps in information-seeking behavior research by investigating the whether there is a correlation between CRT score and the evaluation of information encountered during the information-seeking process. If the results of the proposed exploratory, quasi-experimental study indicate that such a correlation exists, then Information Studies researchers will have greater insight into attributes that underlie the information-seeking behavior of a subset of the population, which has implications for the ways in which we design information systems, assess online information-seeking skills, and devise techniques to overcome cognitive constraints to the information search process.

Research Question

To date, no test has been predictive of whether people are able to resist behavioral biases that may constrain information search in unintended ways. This void is particularly interesting given the timeline of paradigm shifts in Information Studies. This study is designed to answer the following exploratory question:

What is the nature of the relationship between cognitive reflection and information-seeking behavior?]

Methods

As mentioned in the literature review, most of the studies of information-seeking behavior in Information Studies were conducted with ethnographic observation and small sample sizes. While the investigations into cognitive reflection in Behavioral Science tended to be experimental or quasi-experimental in design, they were rarely focused on information-seeking behavior. In an effort to build on the ethnographic insights of Information Studies and to meet the expectations of fields in which experimental and quasi-experimental methods are the norm, this study is designed in a quasi-experimental fashion.

Given that the objective of this exploration is to determine whether a correlation exists between cognitive reflection and information-seeking behavior, there is no control group. However, correlated samples design is a standard protocol in cognitive psychology. The chief advantage of correlated samples design is the reduction of random error due to individual differences. The primary disadvantage, of course, is that a correlation does not signify causation. “Although the presence of thinking biases might be universal, their ability to result in nonnormative choices varies from individual to individual because heuristic responses are sometimes overridden by a nonautonomous analytic system of thought” (Stanovich & West, 2008). Fortunately, the CRT has been shown to be a better predictor of this ability to override gut-impulse responses compared to multiple intelligence measures (Frederick, 2005; Toplak, Stanovich, & West, 2014). Therefore, knowing whether this System 2 capacity is correlated to information-seeking behavior is useful because it provides insight into the potential utility of tools to engage System 2 thinking in the information search process.

The proposed quasi-experiment protocol leverages the reach and variation afforded by Amazon’s Mechanical Turk (MTurk) to recruit a sample large enough to demonstrate the aggregate characteristics of the broad population.

Conceptualization and Operationalization of Variables

This study aims to determine whether a correlation exists between cognitive reflection and information-seeking behavior. Certainly, these notions are constituted by numerous elements, which must be unpacked to determine any sort of causal relationship. Since it is unknown whether a correlation exists, any assertion of cause will be deferred for future research.

Therefore, this study employs the Cognitive Reflection Test (CRT) as a measure of the ability to resist impulsive response and apply System 2 thinking to a situation of uncertainty. The CRT comprises three questions, and is scored on a scale from 0 (no correct answers) to 3 (all correct answers). Its predictive validity has been shown to equal or exceed that of other cognitive tests involving up to 215 items and taking up to 3½ hours to complete (Frederick, 2005).

Information-seeking behavior has been operationalized variously in user studies. The measures for the proposed study are drawn from prior investigations, because “measures with a long history of use usually have known degrees of validity and reliability” (Babbie, 2012). To approximate indicators of System 2 contemplation, the following measures and their underlying assumptions were considered in the operationalization of information-seeking behavior variables:

1. Search duration. System 2 thinking is deliberative, and takes longer than System 1 thinking. Therefore, duration of search may be longer for respondents engaging System 2 thinking to override prepotent responses.
2. Number of sources. System 2 thinking involves the active consideration of alternatives to the intuitive response. Therefore, the number of URL addresses visited for each task may be greater for respondents engaging System 2 thinking.
3. Number of search-term permutations. Another way respondents might demonstrate an effort to consider alternatives may be related to how they select and configure search terms. Therefore, a greater number of search-term permutations may indicate engagement of System 2 thinking.
4. Confidence in response. Cognitive reflection is the ability to monitor and override prepotent responses deriving from System 1 thinking. Participants’ confidence in their answers as measured on a Likert scale with a range of 0 (not at all confident) to 5 (extremely confident) may demonstrate a few things. At first blush, one could envision that a respondent who has engaged System 2 thinking and deliberated over alternatives with adequate information might exhibit increased confidence. However, the recognition of a knowledge gap—that there are known unknowns—is more likely to decrease the confidence level of subjects with high CRT scores.
5. Answers. For each task, the study participants are instructed to provide an answer. A content analysis of the responses to this open-ended question may provide insight into whether the respondents performed cursory versus thorough searches. However, this is not a formal measure to test for a correlation between CRT and information-seeking behavior.

The predictor variable in this design is cognitive reflection as measured by respondents’ CRT scores. Information-seeking behavior is the broad criterion variable, with these specific criterion variables comprising information-seeking behavior: search duration, number of sources, number of search-term permutations, and confidence in response.

Recruitment and sampling

“A sample is representative of the population from which it is selected if the aggregate characteristics of the sample closely approximate those same aggregate characteristics in the population” (Babbie, 2012). The decision to conduct the study using MTurk was based on a review of academic papers concerning the validity of online research. Over time, MTurk has been noted to offer randomly assigned samples that are generally more diverse than the samples recruited for research reported in top academic journals with respect to gender, socioeconomic status, geographic region, and age. In addition, the system allows for the collection of large samples, which would be cost-prohibitive using traditional recruiting methods. A summary of the tradeoffs among recruiting methods is available in the Appendix (Paolacci et al., 2010).

A pilot test will be conducted to allow for refinement of the study design if indicated. A pilot test of the recruitment protocol has already been performed to see how a randomly assigned EPSEM sample is constructed in MTurk. The process proved uncomplicated.

Once any refinements are implemented as a result of pilot testing, a sample of 500 adults will be recruited from MTurk in return for a small cash payment. Using the service’s built-in screening criteria, only U.S. users will be eligible to participate.  Exclusion criteria will include the failure to answer correctly a comprehension check asking the subjects to identify a default option; should any exclusions occur, it will be determined whether including those participants in the analysis would change the results substantively. The sampling frame will be a list of unique ID numbers issued by MTurk and associated with the responses to demographic questions and the aforementioned measures of cognitive reflection and information-seeking behavior. The final sample will be analyzed for relative representativeness of gender, age, education level, household income, and self-reported level of online search expertise.

The ethical principle of autonomy contained in the Belmont Report (1979) requires informed consent so participants can judge for themselves the relevance and weight of risks. The consent form to be used in this study is derived from the UCLA IRB template and available in the Appendix. The consent form will appear on respondents’ screens prior to the start of the survey, and requires an acceptance action and submission of the form to begin the survey.

Apart from the ability to recruit large samples, perhaps the greatest benefit of using MTurk for this study is that procedures are already in place for protecting anonymity and confidentiality. Shadish, Cook & Campbell admonish researchers to use “research procedures that can ensure confidentiality, such as using randomized response methods or determining not to gather any data about potential identifiers, and, just as important, they should ensure that such procedures are followed” (Shadish et al., 2002). The only identifying information available to the researcher will be the unique MTurk Worker ID number, which will be associated with each respondent’s answers to the survey questions.

MTurk respondents can complete experiments without interacting with researchers. This circumvents concerns about experimenter obtrusion, subject cross-communication, and reactance.  “Mechanical Turk is a reliable source of experimental data in judgment and decision-making. Results obtained in Mechanical Turk did not substantially differ from results obtained in a subject pool at a large Midwestern U.S. university. Moreover, response error was significantly lower in Mechanical Turk than in Internet discussion boards” (Paolacci et al., 2010).

The nomothetic correlational design is intended to be highly replicable. Further, its internal validity is relatively high compared to other quasi-experiments, because the study is designed to determine whether a correlation exists between a measure and observed behavior, not to assert causality. Note that two aspects of internal validity are compromised due to the nature of the correlational design. First, the distribution of high- and low-CRT respondents within groups defined according to gender, age, education, household income, and search expertise will not be determined until after the quasi-experiment. Therefore, differences between these groups may not necessarily be helpful in post-experimental analysis. Second, the respondents may circumvent the MTurk platform to answer the CRT questions, which are available on the Internet. Generally, the amount of time spent on the CRT question should be less than the duration of the search tasks. However, this will not necessarily be the case. However, given that the survey will be administered in a standard format, the study will have few threats to internal validity.

To ensure high external validity, the search tasks were drawn from the Repository of Assigned Search Tasks (RepAST), a collaborative project being conducted at the University of North Carolina at Chapel Hill, the University of British Columbia, and the University of Sheffield. The tasks are constructed in such a way as to minimize situational, confounding variables. Further, the participant sample is constrained to U.S.-based participants between the ages of 18 years and 74 years. Therefore, though this study may or may not be generalizable to a global population, it will be applicable to numerous scenarios in the United States. One threat to external validity may be that the workers on MTurk may be more technology-savvy than the general online information-seeking population. This aptitude threat is not unlike that which occurs with academic research using undergraduate student pools.

Thus, the study will attain internal validity by measuring behavior and aptitude in a way that can be easily replicated, and will attain external validity equivalent to current quasi-experimental research conducted in information-seeking behavior research in demonstrating whether or not a correlation exists between CRT and information-seeking behavior.

Data collection

After submitting the consent form, participants will be presented with three sets of questions: demographic, CRT, and search tasks. The instructions will not indicate these categories. The order in which each set of questions appears will be random to control for ordering effects, but the questions within each set will be presented in the same order for all participants. The survey is available in the Appendix.  The demographic and CRT questions are presented without the ability to search for information online. The search tasks, however, instruct the respondents to use the Google search engine to find the answers to five questions.

The respondents’ search behavior will be tracked using the Wrapper, an embedded deployment of an open-source software program within the MTurk interface, which has been used to good effect in other information-seeking behavior studies (Jensen, 2006). Data for each variable will be stored in the MTurk database for 120 days, during which time the researcher may download a spreadsheet of the survey results and associated MTurk Worker IDs in CSV format. The study protocol is designed to minimize the need to collect and maintain identifiable information about research participants. Data will be collected anonymously and access to research data is based on a “need to know” and "minimum necessary" standard.

Data analysis

Results will be considered using exploratory data analysis to gain insight into how data are distributed. To assess the degree of linear association between cognitive reflection and the aspects of information-seeking behavior measured in the study, results will be analyzed using Pearson's product-moment correlation coefficient and Spearman's rank-order correlation coefficient to estimate of the degree to which the variables are correlated. The findings of these analyses will inform subsequent study design, particularly with regard to the presence of confounding factors that should be controlled, if possible, in future experiments.

Summary

This paper proposes an exploratory, quasi-experimental study to determine whether a correlation exists between a particular cognitive bias measured by the CRT and information-seeking behavior. If, indeed, such a correlation exists, then Information Studies researchers will have greater insight into attributes that underlie the information-seeking behavior of a subset of the population, which has implications for the ways in which we assess information-seeking skills and design information systems, interfaces, algorithms, and tools.

 

 

References

Alter, A. L., Oppenheimer, D., Epley, N., & Eyre, R. N. (2007). Overcoming intuition: Metacognitive difficulty activates analytic reasoning. Journal of Experimental Psychology 136(4), 569-576.

Babbie, E. R., (2012). The practice of social research, (13^th Ed.). Stamford, CT: Cengage Learning.

Bates, M. J., Wilde, D.N., & Siegfried, S.L. (1993). An analysis of search terminology used by humanists: the Getty Online Search Project report number 1. Library Quarterly, 63(1), 1-39.

Bateson, G. (1972). Steps to an ecology of mind. Chicago, IL: University of Chicago Press.

Belkin, N. J. (2005). Anomalous state of knowledge. In K. E. Fisher, S. Erdelez, & L. McKechnie (Eds.). Theories of information behavior. (pp. 44-48). Medford, NJ: Information Today.

Belmont Report (1979). The Belmont Report: Ethical principles and guidelines for the protection of human subjects of research. Retrieved from: hhs.gov/ohrp/humansubjects/guidance/belmont.html

Bosch-Domènech, A., Brañas-Garza, P., & Espín, A. M. (2014). Can exposure to prenatal sex hormones (2D:4D) predict cognitive reflection? Psychoneuroendocrinology, 43, 1-10.

Byström, K, & Järvelin, K. (1995). Task complexity affects information seeking and use. Information Processing & Management, 31, 191-213.

Case, D. O. (2012). Looking for information: A survey of research on information seeking, needs, and behavior, (3rd Ed.). Bingley, UK: Emerald Publishing Group.

Chatman, E.A. (1996). The impoverished life-world of outsiders. Journal of the American Society for Information Science and Technology, 47(3), 193-206.

Chatman, E. A. & Pendleton, V. (1995). Knowledge gap, information seeking and the poor. The Reference Librarian, 49(50) 135-145.

Connaway L.S., Dickey T.J., & Radford M.L., (2011). If it is too inconvenient, I’m not going after it: Convenience as a critical factor in information-seeking behaviors. Library & Information Science Research, 33(3), 179-190.

Dervin, B. (1976). Strategies for dealing with human information needs: Information or communication. Journal of Broadcasting, 20(3) 324-333.

Dervin, B. (1983). Information as a user construct: The relevance of perceived information needs to synthesis and interpretation. In S.A. Ward & L.J. Reed, (Eds.), Knowledge structure and use: Implications for synthesis and interpretation. Philadelphia, PA: Temple University Press, (pp. 153-184).

Dervin, B., & Nilan, M. (1986). Information needs and uses. In M. E. Williams (Ed.), Annual Review of Information Science and Technology, 21 (pp. 3–33). White Plains, NY: Knowledge Industry, Inc.

Ellis, D., Cox, D., & Hall, K. (1993). A comparison of the information seeking patterns of researchers in the physical and social sciences. Journal of Documentation, 49(4), 356-369.

Finucane, M. L., & Gullion, C. M., (2010). Developing a Tool for Measuring the Decision-Making Competence of Older Adults, Psychology and Aging, 25(2): 271–288. doi: 10.1037/a0019106

Fischoff, B. (1982). Debiasing. In Kahneman, D., Slovic, P., & Tversky, A., (Eds.), Judgment under uncertainty: Heuristics and biases. Cambridge, MA: Cambridge University Press.

Frederick, S. (2005). Cognitive reflection and decision making. Journal of Economic Perspectives, 19(4) 25-42.

Gorman, P.N. (1995). Information needs of physicians. Journal of the American Society for Information Science, 46(10), 729-736.

Gosling, S. D., Sandy, C. J., John, O. P., & Potter, J. (2010). Wired but not WEIRD: The promise of the Internet in reaching more diverse samples. Behavioral and Brain Sciences, 33, pp 94-95. doi:10.1017/S0140525X10000300.

Jansen, B. J. (2006). The Wrapper: An open source application for logging user-system interactions during search studies. From the proceedings of the WWW 2006 Conference. Retrieved from: http://wwwconference.org/proceedings/www2006/torch.cs.dal.ca/~www2006/jansen-www2006_MechanicsDataCollection.pdf.

Kahneman, D., & Frederick, S. (2002). Representativeness revisited: Attribute substitution in intuitive judgment. In T. Gilovich, D. Griffin, & D. Kahneman (Eds.), Heuristics and biases: The psychology of intuitive judgment, pp. 49–81. New York, NY: Cambridge University Press.

Kahneman, D., Slovic, P., & Tversky, A. (1982). Judgment under uncertainty: Heuristics and biases. Cambridge, MA: Cambridge University Press.

Krikelas, J. (1983) Information seeking behavior: patterns and concepts. Drexel Library Quarterly, 19, 5-20.

Kuhlthau, C. C. (1993). A principle of uncertainty for information seeking. Journal of Documentation, 49(4), 339-355.

Kuhlthau, C. C. (2004). Seeking meaning: A process approach to library and information services, (2nd ed.). Westport, CT: Greenwood Press.

Kuhlthau, C. C. (September 2004) Toward collaboration between information seeking research and information retrieval. Keynote address, Information Seeking in Context (ISIC) Conference, Dublin, Ireland.

Leckie, G. J. & Given, L. M. (2005). Understanding information seeking: The public library context. Advances in Librarianship, 29, 1-72.

Leckie, G. J., Pettigrew, K.E., & Sylvain, C. (1996). Modeling the information seeking of professionals: A general model derived from research on engineers, health care professionals, and lawyers. Library Quarterly, 66(2), 161-193.

Loewenstein, G. (1996). Visceral decisions. Organizational behavior and human decision processes, 65(3), 272 –292.

Mandelbrot, B., & Taleb, N. N. (2010). Mild vs. wild randomness: Focusing on risks that matter. In F. Diebold, N. Doherty, & R. Herring (Eds.), The known, the unknown and the unknowable in financial institutions. Princeton, NJ: Princeton University Press.

March, J. G., & Heath, C. (1994). A primer on decision making: How decisions happen. New York, NY: Free Press.

Marchionini, G. (1995). Information seeking in electronic environments. New York, NY: Cambridge University Press.

Milkman, K. L., Chugh, D., & Bazerman, M. H. (2009). How can decision making be improved? Perspectives on Psychological Science, 4(4) 379-383.

Morozov E. (2011). The net delusion: The dark side of Internet freedom. New York, NY: Public Affairs.

Obal, M. (February 2013), Utilizing Amazon Mechanical Turk to gather data: Pros and cons. Marketing Research Association Philadelphia Chapter Newsletter.

Papparassi, Z. (2002). The virtual sphere: The Internet as a public sphere. New Media & Society, 4(1), 9-27. doi: 10.1177/14614440222226244

Paolacci, G., Chandler, J., & Ipeirotis, P. (2010). Running experiments on Amazon Mechanical Turk. Judgment and Decision Making, 5(5) 411-419.

Rubin, R., & Babbie, E.R. (2010). Essential research methods for social work (2nd Ed.). Belmont, CA: Brooks-Cole.

Sairanen, A. & Savolainen R, (2010). Avoiding health information in the context of uncertainty management. Information Research 15(4).

Savolainen, R. (1993). The sense-making theory: reviewing the interests of a user-centered approach to information seeking and use. Information Processing & Management, 29(1), 13-28.

Savolainen, R. (1995). Everyday life information seeking: approaching information seeking in the context of 'Way of Life.' Library and Information Science Research, 17(3), 259-294.

Schacter, J., Chung, G. & Dorr, A. (1998). Children's Internet searching on complex problems: performance and process analyses. Journal of the American Society for Information Science and Technology, 49(9), 840-849.

Shadish, W. R., Cook, T. D., & Campbell, D. T. (2002). Experimental and quasi-experimental designs for generalized causal inference. Boston, MA: Houghton Mifflin.

Sonnenwald, D.H. (1999). Evolving perspectives of human information behavior: Contexts, situations, social networks and information horizons. In T.D. Wilson and D. Allen (Eds.), Exploring the contexts of information behaviour, pp. 176-190. London: Taylor Graham.

Stanovich, K. E., & West. R. F. (2008). On the relative independence of thinking biases and cognitive ability. Journal of Personality and Social Psychology, 94, 672-695.

Taleb, N. N. (2009). Errors, robustness, and the fourth quadrant. International Journal of Forecasting, 25(4), 744–759.

Taleb, N. N., & Goldstein, D. (2012). The problem is beyond psychology: The real world is more random than regression analyses, International Journal of Forecasting, 28(3), 715-716.

Toplak, M. E., West, R. F., & Stanovich, K. E. (2011). The Cognitive Reflection Test as a predictor of performance on heuristics-and-biases tasks. Memory & Cognition, 39, 1275-1289.

Urbany, J. E., Dickson, P. R., & Wilkie, W. L. (1989). Buyer uncertainty and information search. Journal of Consumer Research, 16(2), 208-15.

Vicente, K.J. (1999). Cognitive Work Analysis. Mahwah, NJ: Lawrence Erlbaum Associates.

Weiler, A. (2005). Information‐seeking behavior in generation Y students: Motivation, critical thinking, and learning theory. Journal of Academic Librarianship, 31(1), 46‐53.

Wilson, T.D. (1997). Information behavior: an interdisciplinary perspective. Information Processing & Management, 33(4), 551-572.

Wilson, T.D. (1999). Models in information behaviour research. Journal of Documentation, 55(3), 249-270.

Winograd, T., & Flores, F. (1986). Using computers: A direction for design. In Understanding Computers and Cognition: A New Foundation for Design, pp. 163-79. Norwood, NJ: Ablex.

Zipf, G. K. (1949). Human behavior and the principle of least effort. Cambridge, MA: Addison-Wesley.

 

 

 

 

Appendix

1. Tradeoffs of different recruiting methods
2. Online Eligibility Consent Form
3. Survey
4. Journal Call
5. IRB Certification

 

 

I.          Tradeoffs of different recruiting methods

Paolacci, G., Chandler, J., & Ipeirotis, P. (2010). Running experiments on Amazon Mechanical Turk. Judgment and Decision Making, 5(5) 414.

 

 

 

II.        Online Eligibility Consent Form

 

Online Information Seeking Eligibility Form

 

This qualification will allow you to participate in our research study HIT, “Online Information Seeking.”

 

Do you live in the United States?

¨ Yes

¨ No

 

Please read this information

 

You are invited to participate in a survey, entitled “Online Information Seeking.” The study is being conducted by Diana Ascher and Leah Lievrouw in the Information Studies department of The University of California, Los Angeles.

 

UCLA OHRP

11000 Kinross Avenue, Suite 211

Box 951694

Los Angeles, CA 90095-1694

 

The purpose of this study is to examine how people in the United States search for information online. Your participation in the survey will contribute to a better understanding of the online information seeking. We estimate that it will take an average of 15 minutes of your time to complete each search task. You are free to contact the investigators at the above address to discuss the survey.

 

Risks to participants are considered minimal. There will be no costs for participating. You will be paid for each HIT you complete, but will not otherwise benefit from participating. Your Amazon Mechanical Turk ID number will be kept while we collect data for tracking purposes only. A limited number of research team members will have access to the data during data collection. This information will be stripped from the final dataset.

 

Your participation in this survey is voluntary. You may decline to answer any question and you have the right to withdraw from participation at any time without penalty. If you wish to withdraw from the study or have any questions, contact the investigator listed above.

 

If you have any questions, please email Diana Ascher at dianaascher(at)ucla.edu. You may also request a hard copy of the survey from the contact information above.

 

This study has been reviewed and approved by The University of California, Los Angeles Institutional Review Board. If you have questions about your rights as a study participant, or are dissatisfied at any time with any aspect of this study, you may contact—anonymously, if you wish—the Institutional Review Board by phone at (310) 825-7122 or email at  gcirb(at)research.ucla.edu.

 

IRB Approval Number: 2014-tk-tk

 

¨ I understand want to participate in this study.

 

 

 

 

 

III. Survey

Please answer these preliminary questions.

1. What is your gender?

 

1. What is your age range?

 

1. Which of the following best describes your highest achieved education level?
2. What is the total income of your household?

 

 

1. How good are you at searching online for information? (1 = No Experience; 5 = Expert Searcher)

 

 

 

Below are three items that vary in difficulty. Answer as many as you can.

1. A bat and a ball cost $1.10 in total. The bat costs $1.00 more than the ball.

How much does the ball cost?

1. If it takes 5 machines 5 minutes to make 5 widgets, how long would it take 100 machines to make 100 widgets?

 

1. In a lake, there is a patch of lily pads. Every day, the patch doubles in size. If it takes 48 days for the patch to cover the entire lake, how long would take for the patch to cover half of the lake?

 

 

Please use Google Search to answer the following questions. For each question, please indicate every source(s) you use by entering the full URL for every page, the amount of time you spent on each page, and your confidence in your answer.

1. We hear of people going on low carbohydrate and high protein diets, such as the Atkins diet, to lose weight. Is there evidence to support that low carbohydrate, high protein diets result in greater long-term weight loss than conventional low energy, low fat diets?

URLs	Time	Your Answer	Confidence
				Not at all confident
				Somewhat confident
				Confident
				Very Confident
				Absolutely confident

 

1. Compare the pros and cons of three models of hybrid cars.

 

URLs	Time	Your Answer	Confidence
				Not at all confident
				Somewhat confident
				Confident
				Very Confident
				Absolutely confident

 

1. What are the most important skills for young adults to put on their resumes when searching for a job?

 

URLs	Time	Your Answer	Confidence
				Not at all confident
				Somewhat confident
				Confident
				Very Confident
				Absolutely confident

 

1. Breast cancer is one of the most common types of cancer found in women. Is there evidence indicating an increased chance of developing breast cancer for women who have a family history of breast cancer?

 

URLs	Time	Your Answer	Confidence
				Not at all confident
				Somewhat confident
				Confident
				Very Confident
				Absolutely confident

 

 

1. Your best friend purchased an old powder horn at a flea market. The seller told your friend that it had been used in connection with hunting. However, another friend is certain it was used in the military. How was the powder horn used?

 

URLs	Time	Your Answer	Confidence
				Not at all confident
				Somewhat confident
				Confident
				Very Confident
				Absolutely confident

 

 

 

 

 

 

 

 

 

 

IV.       Journal Call

2013 ASIS&T SIG-USE Symposium: Call for Participants

Theme: Information Behavior on the Move: Information Needs, Seeking, and Use in the Era of Mobile Technologies
Date: November 2, 2013 (Saturday)
Time: 1:30 to 6:30 pm
Location: Centre Sheraton, Montreal, Quebec, Canada
Keynote Speaker: TBA

ABOUT THE 2013 SIG-USE SYMPOSIUM:
We live in a time when mobile technologies are becoming more ubiquitous within people’s everyday lives, facilitating new forms of information seeking, sharing, creation, and re-use of information and data. Personal computer ownership has been far surpassed by mobile phone ownership and nearly equaled by smartphone ownership. In many developing countries, mobile computing has leapfrogged over personal computing in order to provide online access where none existed. Such prevalent use of cellphone and mobile technologies to access information has a significant impact on the study of information needs, seeking and use. These dramatic changes in technology preferences and use can be seen as altering the very ecology for the study of information behavior, and indeed, blur the contexts of systems and users. It can be argued that even those interactions that do not involve mobile technologies are influenced by the increasing mobility of information use.

The 13th Annual SIG-USE Research Symposium at the 2013 Annual Meeting of the American Society for Information Science & Technology (ASIS&T) will explore the ways in which information “on the move” is transforming or changing the nature of people’s information behaviors and the ways in which people’s information behaviors are, in turn, shaping technologies, services, work and leisure. We will also consider the impact of this transformation on the general assumptions and premises informing the research domain of information behavior. In this Symposium, we will explore the above themes through a keynote speech, short presentations, and small and large-group discussions.

Proposed themes for submissions include:

• Theoretical perspectives/frameworks that constitute the social, cultural and technological underpinnings of information behavior “on the move”;
• The extent to which new information and communication technologies (e.g., social media, mobile technologies) are revolutionary vs. evolutionary within various information behavior contexts;
• How environments in which information is literally “on the move” are providing a technological platform that facilitates the kinds of informal, social information sharing practices desired by users;
• How a better understanding of people’s changing information behaviors might be used to inform the future development of information and communication technologies, services, practices, management and education;
• What innovative strategies of inquiry and methods of data collection might be used to systematically investigate and fully capture and map “information behavior on the move”;
• Any other themes that relate to the Symposium topic of “information behavior on the move”.

Submission guidelines for Ignite talk proposals:

• Include your name, title, and institutional affiliation at the top of your proposal
• Proposal text must not exceed 1 page (max. 500 words)
• Submit proposal in pdf format
• Name your file according to the following convention: ‘2013_SIGUSESymposium_Lastname.pdf’
• E-mail your proposal to Beth St. Jean ([email protected]) by midnight EST on September 13, 2013. Please use “SIGUSE Ignite Talk Proposal” as the subject line of your e-mail.
• When you e-mail your submission, please be sure to indicate in the text of your e-mail whether or not we may post your proposal to the public SIG-USE website both before and after the Symposium is conducted.