[messaging] Fingerprint usability study (experiment design)

[Daniel Kahn Gillmor]

On 06/20/2014 04:53 AM, Michael Rogers wrote:
> On 20/06/14 00:22, David Leon Gil wrote:
>> E.g., (a non-MT) experiment might be: Subject is told that the
>> study is about the user interface for playing some game* via
>> ChatSecure; will play 10 games. As part of instructions, are told
>> to verify fingerprint for games; if fingerprint doesn't verify,
>> won't be able to play. Do a few sessions, presenting valid
>> fingerprints; then present invalid fingerprint (before, e.g.,
>> session 5).
> 
>> The idea is to make verifying the fingerprint purely instrumental,
>> as it is in real life, and an invalid fingerprint an obstacle.
> 
> But in real life, you can proceed if the fingerprint's invalid - you
> just get owned. If we stop the test subject from proceeding when the
> fingerprint's invalid, I'm not sure we're reproducing the real life
> incentive to "just hit OK so I can get on with what I'm doing".
> 
> Perhaps we should instruct the subject to check the fingerprint, but
> let them proceed regardless of whether they actually check it. That
> seems fairly close to what we're asking people to do in real life. :-/

I think simulating these incentives is useful for the experimental
design if we can do it.  I don't know if we can do it, but it's worth a
bit of brainstorming.

One observation in the real world is that if a fingerprint doesn't
match, it doesn't mean you can't do *anything* -- it just means that you
can't make that one connection safely.  You still have other things you
could be doing instead, even if you'd set your mind on doing that one
unavailable thing.

Here's another proposal, but one that might only work for written
fingerprints:

 * user learns the (simple) game -- maybe it's online tic-tac-toe?

 * user is given a list of opponents to play against, with their
fingerprints.  they are told that some of the opponents may be offline
or unavailable, and that there are other opponents available to play
against, but who are "not worthy" somehow -- maybe this list of
opponents are ones who are ranked at the "master" level.

 * user is told that their objective is "win (or draw) against as many
of these players as you can in X minutes to increase your ranking."

 * in the game, they're presented with a list of names of available
opponents.  they pick one that is in their list, are presented with the
fingerprint, which they can approve or reject; if they approve it, they
proceed to play a quick game with that player.

This provides some incentive to verify the fingerprints (since only
games against the actual "ranked" opponents are worthy) and a little
time pressure (because the subject wants to play more games in the
allotted time) while also giving them something else to do if they
decide to reject the fingerprint (play another player).

It's entirely possible that this is too subtle, though, and we could
find that no one ever rejects any of the bad fingerprints because eh,
why not go ahead and just play the game with this non-ranked player
anyway :/

	--dkg

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 1010 bytes
Desc: OpenPGP digital signature
URL: <http://moderncrypto.org/mail-archive/messaging/attachments/20140620/f4d5e372/attachment.sig>