[messaging] Let's run a usability study (was Useability of public-key fingerprints)

[Joseph Bonneau]

>
>
> > I agree.  Let's run one.  I've participated in them - it's really not
> > that hard, especially if we can find a professor in the field who's
> > willing to advise/review our proposal.
>

I disagree, good usability studies are very hard to design and execute. The
primary difficulty is avoiding Hawthorne effects and priming effects-people
behave very differently when they know they're being observed for their
security behavior, and they're likely to perform far better on tasks like
matching a hex string in isolation than when they're in the middle of
another task like trying to get OTR to work with their friend for the first
time. That's not to say it's completely useless to study them in isolation
and you might still get a sense of how different methods compare to each
other at a high level. But doing this experiment correctly would require an
elaborate study with deception so that users don't realize that checking a
fingerprint is part of the experiment. That's a lot of work to set up.

Of Tom's list the SSH experiments seem most adaptable here, as you could
plausibly convince users the study is about something else and they need to
SSH into their home machine to complete the task. Although the filter
"knows what SSH is" eliminates 99% of the population, so that's already a
niche study.

Another approach is to take a popular client and push experimental UI
changes which fiddle with the fingerprint display. Chrome has used this
approach, for example, to test people's reaction to certificate warning
messages (http://www.robreeder.com/pubs/sslExperimentCHI2014.pdf). The rate
at which people see incorrect fingerprints is probably so low though that
you'd need to do this on an enormous scale to see a change in people's
behavior (ie they're more likely to close the window and reject a
connection if a visual fingerprint is shown), and even then you probably
don't know the ground truth so it wouldn't be clear if your new fingerprint
display was actually helping them spot a real attack or if it just freaked
them out. Also this requires people opting in to the telemetry
experiment-Chrome has a chunk of users who do but I don't know that say,
Pidgin does, and ssh most definitely does not.

One other note-getting a large population of users to study is not free.
You're probably talking thousands of dollars in budget just for that, which
is probably worthwhile, but it means you have to be very sure about your
experimental design before you launch anything.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://moderncrypto.org/mail-archive/messaging/attachments/20140213/dd103614/attachment.html>