[messaging] Fingerprint usability study (experiment design)

[Joseph Bonneau]

This is a great breakdown, thank you David! A few comments:

On Mon, Jun 16, 2014 at 9:59 AM, David Leon Gil <coruus at gmail.com> wrote:
>
>  ## Summary
>
> The overall goal: Determine whether fingerprint format affects the
> reliability of user comparison of fingerprints.
>

I think our real-world goal is "helps users ensure they're communicating
with the intended party" We can specify that we're only looking at
solutions here that involve fingerprint comparison but it's worth keeping
the real-world goal in mind.

*Factor B.* Incentive to reject fakes:
>
> 1. None
> 2. Desire to "do well" or please experimenter
> 3. Game-like incentive (e.g., Mechanical Turk performance-based
> compensation)
> 4. 'Real-world' privacy-preservation-like incentive (e.g., belief that
> security of answers to personally sensitive questions rests on correct
> performance)
>

Not sure if #3 encapsulates this, but you can use monetary incentives in an
experiment.


> ### Factors that are measurable, but hard to select for
>
> *Factor E.* Subject type:
>
> 1. Pure novice subjects (e.g., an Internet user who doesn't know what a
> fingerprint is, doesn't understand the cost of generating collisions, and
> has never attempted this tasks)
> 2. Educated novice subjects
> 3. Experienced subjects
> 4. Educated and experienced subjects
>

This can be approximated by giving users more instructions in one treatment
than another, so perhaps this belongs above.

 *Factor F.* Learning style:
>
> (Needs research; likely needs to be measured and results normalized to
> population prevalences. Note that I believe that there is substantial
> evidence that a one-size-fits-all fingerprint verification format will be
> inferior to allowing users to choose a preferred fingerprint format. Here,
> it might be interesting to do an experiment with feedback; e.g., have a
> subject choose a fingerprint format to verify, provide feedback on
> accuracy, then allow choosing another format, etc.)
>

I would discourage any attempt to vary this factor. For the conceivable
future a one-size-fits-all format is necessary.

Missing factor H? The similarly of the fake fingerprints to the expected
genuine ones.

## The proposed experiment
>
> As I understand it, the consensus is that an experiment that is likely to
> have discriminatory power among fingerprint types is infeasible to conduct
> in a realistic setting. (I.e., the 'head fake' type scenarios.) I'd tend to
> agree.
>
> So, the proposed experiment is, approximately: A1/B3/C1.
>
> For that experiment, I'd note that the actual probability of a fake
> fingerprint (and perhaps the 'goodness' of the fake) has to vary so to
> allow extrapolation to the zero-cheater case. (Though I'd expect that very
> few participants will cheat unless the compensation scheme is extremely
> imbalanced.)
>

Why are participants cheating? I think all errors should be introduced by
experiments rather than have subjects insert their own error.s

 ## The gold-standard experiment
>
> A large trial among users of messaging software that requires fingerprint
> verification, in which errors are introduced (with some small probability)
> in fingerprints.
>

Personally I think this is the way to go.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://moderncrypto.org/mail-archive/messaging/attachments/20140616/febdecf8/attachment.html>