[messaging] Let's run a usability study (was Useability of public-key fingerprints)

[Tom Ritter]

As promised, here's a first-pass at a proposal:
https://github.com/tomrittervg/crypto-usability-study

On 9 March 2014 20:10, Joseph Bonneau <jbonneau at gmail.com> wrote:
> I'm interested in helping out as well if I can, glad to see there's some
> movement here! In particular I can help with data analysis/significance
> testing if that's something needed, though sounds like the sample size will
> be low.
>
>>
>> For the tests, I could imagine giving users pairs of fingerprints which
>> are either identical or a close match, and have them choose same/different
>> after X seconds, where X is tuned to produce a significant error rate.  I'd
>> also try having one value on a screen, and the other in different formats
>> that might be used for fingerprint exchange:  e.g. printed on the front of a
>> business card, displayed on a separate screen, read aloud, written on a
>> napkin, etc.
>
>
> I think I've made this point before but I think the main challenge is seeing
> how users perform not just in a quick check time wise, but one in which they
> have no reason to suspect an error, because most of the time most users
> don't think they're being attacked so they just check the beginning for a
> gross error then click through. If you tell users to check for errors, it
> may not represent very well how they'd do in practice. Perhaps the only way
> around this is to show users fingerprints which match in 99% of cases and
> see if they catch the 1% when they are mind-numbingly bored and their prior
> is low. But that probably has to be an mTurk study...