[messaging] Fingerprint usability study (experiment design)

[David Leon Gil]

# Studying verification of fingerprints


## Note


My goal is to prepare a brief summary of the fingerprint usability study suitable for presenting to behavioral economists / cognitive scientists for review. I'd very much appreciate any comments, suggestions, or corrections


(I used to design and review similar experiments -- howbeit in rather different subject areas -- in experimental philosophy/cogsci and behavioral economics in my law school days; my intention is to flog this study around some former colleagues, and see if anyone has time to review or comment.)


## Summary


The overall goal: Determine whether fingerprint format affects the reliability of user comparison of fingerprints.


It's obvious that there a lot of interesting studies that can be carried out in this area. To try to summarize some of the prior discussion (and perhaps add some thoughts of my own), the (indirect) factors we'd expect to influence performance:


## Experiment background factors and metrics


### Factors controllable by experiment design:


*Factor A.* 'Type' of memory:


1. Short term
2. Medium/long-term single shot
3. Medium/long-term with rehearsals


*Factor B.* Incentive to reject fakes:


1. None
2. Desire to "do well" or please experimenter
3. Game-like incentive (e.g., Mechanical Turk performance-based compensation)
4. 'Real-world' privacy-preservation-like incentive (e.g., belief that security of answers to personally sensitive questions rests on correct performance)


*Factor C.* Psychological incentive to accept fakes:


1. None
2. Game-like (e.g., performance compensation + directive to answer as quickly as possible)
3. Realistic pressure (e.g., pressure to please experimenter)


*Factor D.* Expected baseline error rate. (Approximately continuous variable on a repeated task? On a single-shot task, likely highly correlated with other experimental  parameters.)


### Factors that are measurable, but hard to select for


*Factor E.* Subject type:


1. Pure novice subjects (e.g., an Internet user who doesn't know what a fingerprint is, doesn't understand the cost of generating collisions, and has never attempted this tasks)
2. Educated novice subjects
3. Experienced subjects
4. Educated and experienced subjects


*Factor F.* Learning style:


(Needs research; likely needs to be measured and results normalized to population prevalences. Note that I believe that there is substantial evidence that a one-size-fits-all fingerprint verification format will be inferior to allowing users to choose a preferred fingerprint format. Here, it might be interesting to do an experiment with feedback; e.g., have a subject choose a fingerprint format to verify, provide feedback on accuracy, then allow choosing another format, etc.)
 
*Factor G.* General memory capacity. For short-term multi-shot tests, easy to control for by, e.g., digit-span tests administered to (a portion of) the experimental population. (This is important to measure because Mech. Turk subjects taking the study may not be representative of users.)


## The proposed experiment


As I understand it, the consensus is that an experiment that is likely to have discriminatory power among fingerprint types is infeasible to conduct in a realistic setting. (I.e., the 'head fake' type scenarios.) I'd tend to agree.


So, the proposed experiment is, approximately: A1/B3/C1.


For that experiment, I'd note that the actual probability of a fake fingerprint (and perhaps the 'goodness' of the fake) has to vary so to allow extrapolation to the zero-cheater case. (Though I'd expect that very few participants will cheat unless the compensation scheme is extremely imbalanced.)


## The gold-standard experiment


(The above is obviously a useful preliminary towards a realistic experiment; the following is my idea of what a 'gold-standard' experiment on this would look like.)


A large trial among users of messaging software that requires fingerprint verification, in which errors are introduced (with some small probability) in fingerprints.


If this is set up so that (1) users give some form of consent to the experiment and (2) the experiment never causes a user to falsely accept a forgery (i.e., if a fake fingerprint is accepted, the user is reprompted suitably*), are there any ethical objections?


- David


*(This would probably require highlighting the position of the introduced error.)




PS. And apologies for the post about ring signatures last night; as Trevor was kind enough to point out to me, the curves list is a much more appropriate place for discussion of that.
—
Sent using alpine: an Alternatively Licensed Program for Internet News and Email
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://moderncrypto.org/mail-archive/messaging/attachments/20140616/e6dea6f5/attachment.html>