How does missing info (a known unknown) impact confidence?
Rumsfeld
Known unknowns:
info we know we don't know
awareness of missing info or unknown info
prediction: greater known unknowns=?
confidence = s(focal H)
s(focal H) + s(alt H)
confidence in choice bt two
present info on three attributes
this evidence would suggest that audi gets better gas mileage with high level of confidence
then realizes there's info they don't have
called known unknown
including these known unknowns shouldn't affect confidence, but hypothesis is that conf decreases
holding support constant, confidence also varies as fct of info known to be missing
indicator of knowledge completeness
MT:
awareness reduces confidence, but not valence
MT: would be interesting to compare to skepticism
Danny: ambiguity aversion; if we knew the info on one but not the other-->you might have a base rate, but if we assume we set the values equal to the midpoint of the distribution, then you would have awareness of known unknowns, but you'd also have ambiguity
he's asking how that fits together
MT: I should talk to Danny with regard to misinformation in this scenario
weight on the known unknowns changes in two conditions-->does shifting the weight affect confidence? fewer or more unknowns?
ambiguity effects
if you make a particular known unknown more salient, how does that affect confidence
may also have scenarios with more info that the subjects don't know how to use
awareness (cont from above)
won't change choice
won't change accuracy
should change confidence
study 1 (process data--correlational)
2 (experimental)
3 (exp) priming in consumer judgments
4 (exp) extend findings to kaufman's intervals
- confidence task abt products with reason listing
which has more calories? list all reasons for the answer; can change choices
rate reasons on 7-pt likert scale
degree known unknowns
support for focal vs. alt
want to make the confidence rating measure explicit in your slide
sample known unknowns:
i don't know how much fat is in a meatball
i don't know how much bigger a ftlong is than a big mac
known support (rated high on the known side)
mcdonalds has unhealthy food
a ftlong has a lot of meat and bread
results p< .0001
low known unknowns (2 avg rating) are more confident
need more explanation on the graph
stratifying the group
a fitted line as the rating changes how does the confidence change
these are coeffs out of predicted mean values of a regression
floodlight analysis
taking an average of 7-pt scale answers
should there be a monotinicity case? (Keith)
number of things above and below the midpoint of the scale: confounds those who elaborate more
MT: do you categorize the reasons at all? do you ask how much this affected the decision?
Helen asked whether the data on changes is captured
what is the reason?
what is your confidence?
did this to measure the effects of known unknowns and ?
maybe ppl justify their low confidence
people gave reasons more about their confidence than their choices
also added support into the regression
there are diff variances at diff levels of conf
this is not just a measure of accuracy: as ppl list more unknowns, it trends toward them being more accurate/better calibrated, driven by lower confidence
were some items easy and some hard? yes
also looked at diffs by question
as ppl list more unknowns, their confidence decreases
significance levels are within subject, but not the unknown:confidence by question
Keith: everything so far is consistent with Bayes rule
accuracy rates: looked at difficulty by accuracy
the most difficult question, where ppl had least accuracy, ppl listed the lowest # of unknowns
MT: how does this match up to the CRT measure?
MT: I think if ppl don't know, they don't bother to reason
the type of question may have bearing on this: if it's something that could be estimated, more contemplation goes into it
the main conclusion is that as ppl become more aware on unknowns, they show lower confidence
the measure of known unknowns is less reliable for difficult questions
MT: how does this account for someone just not feeling like making the effort?
there are other things that can affect confidence, such as sitting next to a professor (comparative ignorance) novel in the sense of being on the belief side
reliance on ignorance prior
when you control for support, there's something else there: confidence doesn't explain it all
salience, availability manipulations are of more interest to Danny and Keith
Keith: causality; "big animals tend to be heavy, but I'm not sure what causes which"
diving into a mechanism discussion, but prior to that you should be ??
want to compare to existing theory
the term "hardest" --> perhaps change it to something having to do with accuracy
Study 2
pull out the correlational aspects and experimentally manipulate
coin flips, told outcome of first 15 flips
varied the number of unknown flips
varied the level of evidential strength
then completed the same tasks from study 1
regressed confidence judgment on objective P
sensitivity to known unknowns
sensitivity to evidence strength
ppl were well calibrated in sensitivity to KUks
undersensitive to evidence strength (to the proportion)
want to be more clear about what you mean about evidence strength
this flies in the face of the law of small numbers
what's diff here is that you're making more salient these known unknowns, rather than the sample size
Benjamin and Rabin have run similar studies with diff results
sensitivity to evidence strength did not predict
Keith: it's a difference in levels, right?
so this shows the overconfidence, unsystematic departure of Bayesian sensitivity
Danny: anything that lowers confidence improves calibration
either you need to be careful about how you say calibration, or you need some other questions
Craig: is this biased by tendency for 50%? No, bc it should be covered in the support
Danny: you could say high sens leads to less confidence or to
study 3 priming ppl to consider KUnkowns
compare to support
consumer fin dm mult choice questions
estimate probability of correct answer
three conditions: control, KUK diff question; known knowns answer an easy question
open response ppl said they didn't know on the hard question
no diff in accuracy across the conditions suggests they didn't introduce more info
no diff in conf for 2, but signif lower conf in the KUN ~12%
next study pits opposite alternatives
Study 4
prior experiments in AFC
extend to overconfidence domains
recruited on mTurk
point estimate and 90% conf intvl
meta-knowledge=interval width/point estimate accuracy
1 is perfect calibration, <1 is overconf
KUK prime improves meta knowledge p< .05
not making ppl less confident, it's making ppl more sensitive to what they don't know
this also motivates why you go to confidence intervals
be careful about the term "determinant"
hard-easy effect may be driven by neglect of known unknowns
implications
KUKs provides a new mechanism behind overconf and conf
prescriptive fix for overconf
most unaware when making difficult decisions
salience may affect confidence in consumer judgments
(typo in last bullet point in this slide)
start out telling why confidence matters in consumer decisions
real knowledge is to kno...