[messaging] Test Data for the Usability Study

[Michael Rogers]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA256

On 28/05/14 20:47, Trevor Perrin wrote:
> Ideally there would be initial testing to identify good parameters
> for each method.  Since these tests should be a lot simpler (with a
> single variable, like: upper vs lowercase; size of char groups;
> etc), maybe they're easier to design and run on M-Turk?

There are at least three properties that could make one variant better
than another: false positive rate, false negative rate and comparison
time. One variant could be better than another in one respect and
worse in another respect, so I'm not sure we can say a priori that
there will be a single best variant for each method. We might have to
compare the variants of each method, eliminate any that are strictly
dominated, and pass the rest through to the next round.

> Simulating 2^80 work-factor "fuzzy match" attacks is also going to 
> involve a bunch of decisions.

Those decisions should be based on evidence. As far as I can see, that
means we should start by making random modifications, then see whether
the data show that some modifications are less noticeable than others
for each method. If so, we have an objective definition of "fuzzy
match" for that method.

> I think that for text methods maybe we can come up with visual / 
> phonetic similarity metrics that are reasonably comparable.  But I 
> dunno about visual fingerprints, that seems like a research project
> in itself - unless someone has a lot of time to work on it, maybe
> the visual methods are too much to tackle.

If the difficulty of designing comparable fuzzy matches is what's
causing you to say that, I think it's better to postpone studying
fuzzy matches than to compare a narrower range of methods. The false
positive rate, false negative rate and comparison time of a wide range
of methods would be interesting in their own right, even without
considering fuzzy matches. If the data supports the concept of a fuzzy
match for some methods then we could do a follow-up study on those
methods.

Cheers,
Michael
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.12 (GNU/Linux)

iQEcBAEBCAAGBQJTi5BrAAoJEBEET9GfxSfMqFoIALBwfd87T99g4ByPiBkYso+1
4eRIsUwqxM+69oN7mUFxSfpe2duDSf2o390HR08pIgiDAqj5NbKvvFPvc4LfWdir
zPbQTxZOheq4zYdStCjneRebgNbu8esFzkiRsSfyiKI6d6qyuP9jvFW6on8qmJqT
1UwHQ5sp2JzkHfUHHquImWmWJDXM2PPhbGjFTL4w0x0la//twxo9pu1by2Pl+5co
+p/FYB1Wj9vUb3VgZWnDRD9MBK3ny6TI5dRpXWKYuAeZNtFrauOBKwn+GAzh2IWN
QkdTe/kKhnHmQC70DYvBSnnyE/oAs7US17Gd7vrPBSE0fJbthpyHXs3txZiTGTs=
=hwJy
-----END PGP SIGNATURE-----