[messaging] Matrix.. is Federation at odds with Privacy?

[Matthew Hodgson]

On 17/04/2015 17:15, carlo von lynX wrote:
> On Fri, Apr 17, 2015 at 12:49:02PM +0100, Matthew Hodgson wrote:
>> However, I'm afraid the world is not so black and white - there are
>> also scenarios where it's frustrating for users that their contacts
>> are stuck in different apps.  For instance,
>> Viber/WhatsApp/Messenger/Kik/WeChat/LINE are all incredibly similar,
>> and I have friends on all of them.  Forcing everyone to install all
>> apps and having group chats scattered across them is creating a
>> completely horrid UX for end users.  Users should have freedom to
>> choose their preferred UX if they desire.
>
> But again, the companies you mention have no interest in
> making themselves interchangeable unless they have a strong
> reason to believe they will be the winners of such a change.

Federation makes it easy for communication apps to gain and lose users.

You gain the users who are willing to try your new UX if their 
conversations and contacts remain intact, and it increases their 
stickiness if they like it, as you remove the barrier to entry of 
migrating their existing social graph into the app.  You also gain users 
who discover app-specific content/features via federation, and actually 
have a good reason to install the specific app to participate in these 
specific features... without losing their conversation history, 
contacts, etc.

You obviously risk losing users who discover an app with a better 
UX/UI/features and promptly jump ship, never to look back.  However, we 
think the likelihood of gaining users is higher than losing them, plus 
this serves as a worthwhile incentive to maintain a competitive 
UI/UX/featureset in your app.

>>   * Facebook could easily move from dominance, just like the many
>> waves of change that have preceded it.  There's absolutely room for
>> new comms technology, whether it obfuscates metadata or federates or
>> whatever - so long as it actually differentiates and builds a
>> community.
>
> Yes, either by a new platform with outstanding new features that
> everyone goes wild about, or because a law changes the rules of the
> game, or because all competitors join forces.

You don't need *all* competitors to join forces to provide an 
alternative to the incumbent(s).  You just need some of the 
emerging/secondary ones to join forces, who have something to gain by 
pooling their userbases, and then snowball.

> But not because a small
> company wrote code to implement a proposed open standard - that is
> more likely to become the 16th competing open standard.

In the interests of full disclosure: whilst Matrix.org is ~15 people 
currently, most of us have day jobs at a very large company (Amdocs) who 
is kindly funding us to work on it.  This seems to help the longer tail 
of messaging/voip apps to take us seriously when we suggest joining Matrix.

>>   * Facebook may be replacing email for social contact (alongside a
>> whole range of technologies like SMS, WhatsApp etc), but email is
>> here to stay just as much as the PSTN as a genuinely ubiquitous
>> federated technology for a huge range of other use cases - for
>> instance, it's literally still the only standard federated
>> technology out there for businesses to exchange data!
>
> Sure, even IRC isn't dead yet, although the usage numbers are
> below 0.01% of the worldwide chat market. I know twenty-year-olds
> that have *FIVE* addresses in their email address book. They only
> use it because it is being forced upon them by *FIVE* external
> authorities. Everything else happens elsewhere.

You seem to be ignoring what will happen when those 20 year olds leave 
school and get a job and find themselves needing to have conversations 
(like this one) with people in other organisations.  I'm afraid that 
email is still living on, despite the spam.

And I hope that Slack or other closed proprietary systems don't end up 
replacing its function for cross-organisational collaboration and 
communication, just because we sat around arguing about ideology rather 
than getting our act together and providing a good open solution, 
federated or otherwise.

>> Meanwhile, Matrix's mission is to provide a simple extensible
>> platform to allow realtime data interchange with decentralised
>> persistence: basically a read/write Web with pubsub.  And just as
>> the Web itself is huge vehicle for social justice and democracy
>> (even without metadata obfuscation and even without crypto!!), I
>> believe Matrix can make the world a better place too.
>
> But, if it is without metadata obfuscation and without crypto,
> then it doesn't need a complicated multicast pubsub scheme over
> federation. It can simply use the cloud! You can use two clouds
> for redundancy, but you really don't need to re-invent cloud
> technology in a federated manner if you have nothing to hide.

It's not all about privacy!!

The reason to federate is to give the users the freedom to pick the 
clients they use (in terms of UX and features) and the servers they use 
(in terms of location, features, trustworthiness), etc.

Sure, we don't give metadata privacy yet.  But this is still a *massive* 
improvement over a closed proprietary cloud architecture.

>> This is a failure of old-school federation systems like
>> SMTP/XMPP/PSYC.  To be honest, we've also punted on the
>> migration/porting question in v1 of Matrix (but it's on the radar
>> for v2 or v3).  It's worth noting that it's a *really* useful
>> feature (however painful the process is) for consumers on GSM.
>
> Wait wait... please consider the migration scenario in a distributed
> system. You move your configuration files from one device to another.
> The new device announces your public key to the network. You have
> already completed the migration.
>
> Now explain again how ANY federation migration will ever be as
> simple as that.

Agreed that it's simple in that architecture.  But it's also quite 
possible in a federated architecture too - the fact it's a bit more 
hassle is irrelevant.

>> It's worth noting that Matrix's semantics of federation really are
>> nothing like SMTP and XMPP.  The building block in Matrix is
>> *synchronising history*, not message passing.  This makes both
>> interoperability and federation much more compelling: if I'm
>> basically using different apps as different UI/UX for viewing the
>> same decentralised conversation database, the motivation to move
>> between apps (or services, in future) and pick the best app becomes
>> much stronger. Just like users love picking their preferred email
>> client or GSM handset, I expect them to love picking their favourite
>> Matrix client... without losing their identity or conversation
>> history.
>
> Yes, same here with distributed pubsub and state. We are coding the
> same stuff. In our case it doesn't run on company servers and users
> can change UIs as they like. In your case they depend on companies
> to play by the rules as there probably are several possibilities to
> practice lock-in.

I really hope we've avoided this.  The core protocol simply doesn't work 
at all if you don't play by the rules.  However, the data you sync over 
matrix is entirely extensible JSON in the end, so we expect to see apps, 
services and possibly servers go and gleefully fragment away at the 
application layer.

And that's okay, so long as the core interoperable IM and VoIP works. 
It's no different to email supporting arbitrary attachments, so long as 
the mail body is intact.

>> Have you ever seen how painful it is to migrate users to new
>> intranet tools?  Not to mention that the world of business (by which
>
> You mean against their will?

Yup, which is 99% of the time the case, as folks already have a workflow 
up and running with their current tools, no matter how blunt those tools 
are - and thus have massive inertia.

>> i basically mean 'professional interaction') is dedicated to
>> millions of different islands (organisations) desperately trying to
>> collaborate on different projects, and discovering that email is
>> still about the only thing they actually have in common which lets
>> them own and manage their own IP/data.
>
> Yes. And since humanity hasn't solved this one, it should try
> with metadata protection in place from the start...  ;)

In practice, companies often need to assert precisely (physically) where 
their data lies, and need to run it on specific hardware with a specific 
SLA etc.

I know this sucks, but this is the reality of legislation or braindead 
IT departments, and a federated architecture with exposed metadata fits 
this better.  In time the necessary legal/social engineering may fix 
this problem and open the route to ubiquitous metadata-protected 
infrastructure, but it's not going to happen on day 1 (at least for this 
use case).

>>>> Well, Hangouts & FB Messenger are both WebRTC-based these days, so I
>>>> don't have to wait for a new Faceboogle :)
>>>
>>> Why on Earth should they care to interop? One of them is the stronger
>>> and has an interest in draining the other. And they both have no
>>> interest in letting small businesses have a piece of the cake.
>>
>> The monomania of trying to compete with FB via G+ seems to be
>> fading.
>
> No no, it's not about social networks. It's about the time you
> spend on *their* web pages with *their* ads embedded. So why
> should they let anyone else have a piece of the web-telephony
> market? You sound like you're trying to defy basic laws of
> capitalism.

As I said above, it can open the way to increase the amount of time 
spent on their services.  It's just creating a free market for 
communication, and there's nothing more capitalist than a free market. 
May the best service win the most users!  (Rather than locking them in 
against their will).

>> Sure, everyone and their dog can design their own signalling scheme.
>> And if you categorically will never need persistence, federation,
>> e2e crypto, or any of the other goodies that Matrix provides, then
>> perhaps it'll be quicker to write your own rather than use an
>> existing library.  Arguing that "anyone can invent their own library
>> if it serves their purpose" seems a little specious ;)
>
> Well, I am arguing that either Faceboogle will always opt for
> offering *all* of those features out of their own cloud backbone
> rather than interopping with strangers that might suddenly attract
> attention and steal a piece of the cake. And I especially like
> the "e2e crypto" there in your list, knowing that no stateless
> federation (aka the web and its webrtc) architecture can guarantee
> REAL end-to-end crypto to people.

Well, matrix is decentralised stateful, and the axolotl-based e2e 
crypto's looking pretty good.  Once we have the first draft finished 
we'll post it here for folks to pick holes in.  I'm hoping it'll be REAL 
end-to-end, even if metadata is exposed.

>> So, the reason I came across PSYC however many years ago was
>> (briefly) running a psyced for precisely this purpose.  I think the
>> main reason why it had limited uptake is that the UI/UX of the app
>
> What UI?

The general feel of running the psyced daemon from a sysadmin 
perspective, or the perspective of someone used to running ircds or 
jabberds.  It's a fairly subtle thing, but it just felt a bit too fringe 
and weird - might have been the lack of doc, might have been the 
unfamiliarity of the native PSYC experience, etc.  I wasn't talking 
about actual client UI.

>> felt aggressively non-mainstream: an extreme poweruser tool without
>> any reassuring visible end-user facing benefits/community/glossiness
>
> Well it was federation at its max: letting you idle on as many servers
> as you liked, allowing everyone to run their own server and host chats.
> Like IRC, but without the oligarchy and the chanwars - like XMPP, but
> with better chatroom control as IRCers expect.
>
> As I said, stuff nobody really needs. Federation. We've been there,
> we bought the t-shirt, now we know it was badly spent time.

As far as I can see, PSYC was focused on building an IRC-killer network 
targeted at poweruser geeks, with some forays in using the tech to power 
commercial services.  The IRC/XMPP federation was experimental, and in 
the end the community probably didn't have that much motivation to 
actually make the product mainstream and risk it being flooded with 
newbies and destroying the SnR for the overall network.

These constraints may have limited its success far more than the 
shortcomings of federation as a technology.  And as you can see, Matrix 
is coming from a totally different direction: corporate-sponsored 
opensource, and we're addressing a totally different world: one where 
users' communications are getting fragmented over hundreds if not 
thousands of different silos.

I think the constraints are sufficiently different in Matrix's instance 
that it's worth us having another stab at the problem.  If not, you get 
to say 'I told you so!' :)

>> I agree that regulation is one way to force folks to use an open standard.
>>
>> But the assertion that open standards formed to attack market
>> leaders will always suck and lose relevance is just bogus.  If a
>
> Oh yeah? ISPs formed an alliance and kicked out Compuserve & co.

Okay, so the modern internet is an example of an alliance of open 
standards /not/ sucking, surely - at least in the context of kicking out 
Compuserve and all the other non-federated networks.

> When Netscape was bossing the web, Microsoft got in bed with
> everyone at W3C until the web was p0wned. They stopped chanting
> the open standards refrain on the day IE took over the lead.

...and yet the standards prevailed, and you could argue that the IE6 
disaster formed the necessary impetus for the rest of the industry to 
get off its ass and ditch XHTML and sort out HTML5.  It didn't make the 
standards suck.

> XMPP came up when AIM/ICQ and MSN were dominating the chat market.
> Funny that Facebook later won that race by adding a webchat. Its
> support for XMPP clients is merely political cosmetics.

Precisely. It wasn't remotely trying to push XMPP forwards, it just 
implemented it as a box-checking exercise.  This is irrelevant to the 
actual question of whether open standards formed to attack market 
leaders will always suck.

> In the meantime Google has locked in millions of email users into
> their Gmail cloud offering. Even if people later decide to get out
> of Gmail again, will all of their friends seriously stop sending
> mail to gmail.com?

Thank goodness it's table stakes to implement forwarding rules in free 
email services these days.

> So when exactly did an open standard ever get anywhere if there
> wasn't a reason for a number of actors to form an alliance against
> some market leaders?

Well, I think that pretty much every communication solution who isn't 
FB, Google, MS or possibly Slack right now has a great reason to form an 
alliance in order to compete more effectively.  And Google has other 
priorities (expand the internet), as does Microsoft (entrench Windows), 
which could actually be served by participating in the alliance too.

> I can't think of any such scenario. Ah right.
> Whenever some legislator imposed a standard by law.

I don't think we'll need to fall back onto that one, which is just as 
well, given what a hail mary it is.

>> standard exists, works, and it brings value to members of a
>> community, they will use it.  Whether it takes over the whole world
>
> Community? What is the community?

Users who just want to be able to use their preferred app & service to 
communicate.  They do exist.

> Idealist developers that can talk
> their companies into doing something for a while? Independent idealists
> that spend their spare time? You chose to compete in a capitalist
> market - how does your "community" have any influence in that?
> Companies make the market move and they are tied to its rules -
> the "community" isn't even a large number of consumers capable of
> having some power as consumers, and they aren't organized enough to
> have much influence on the actual mass of consumers, either. They
> may at best influence media coverage for a bit.

The community are the consumers; they drive the market by choosing what 
services to use.

>> or not is another story of course, but the whole internet owes its
>> existence to IP federation between organisations, just as much of
>> the modern Web exists thanks to HTTP API federation between
>> organisations.
>
> Sure, the Internet was created in times when it wasn't under the
> harsh rule of capitalism yet. I know, because I've felt the shift.

Haven't we all :)

> HTTP is not a federation as 99.x% webservers do not interact among
> each other. Some people have done some apps that use HTTP as a
> federation platform, but I don't know of any that are of relevance to
> the market. RSS pings? No. OStatus? No.

Erm, there's an entire industry of server-facing HTTP APIs: PaaS, IaaS, 
Telco APIs, things like IFTTT, Matrix... HTTP works great for 
server<->server interaction, and the whole modern web exploits that 
enormously.

>> Providing competent tools for building extensible decentralised
>> federated platforms like IP and the web is something worth fighting
>> for.
>
> Even if it will not affect much, like it hasn't since 1995?

Sure, nobody's got it to work before.  But that's not a reason not to 
try something.

>>> And who is the market leader that would motivate Google and Apple to
>>> team up with a common standard?
>>
>> Perhaps Facebook.  Perhaps the long-tail of next-generation
>> WebRTC-backed solutions which happily interoperate via Matrix,
>> leaving Google & Apple to really look like dinosaurs who refuse to
>> join in the party.
>
> Yes, maybe. Unless they stick to the XMPP which is already doing
> the job.

Alas XMPP doesn't do the job, unless you add layers upon layers of new 
XEPs which may or may not be supported in your particular blend of 
clients & servers in order to get decentralised history, e2e crypto, 
etc.  At which point why not just use a relatively clean HTTP API which 
provides that in the baseline: Matrix.

> Or maybe they come up with something new because they don't
> want to put any small company in power of defining the "open" standard.
> Again, the companies involved are making the rules and the protocols
> they will pick are their playthings. Of course you could just be lucky
> by being in the right place at the right time. There is a fringe chance
> there.

The main motivation for folks to write their own signalling protocols 
seems to be the bad habit of developers liking to reinvent wheels, as it 
lets them hone their skills and learn the tech and feel accomplishment. 
  Hopefully by providing a sufficiently obvious and fully-featured 
alternative, folks may adopt it.  If they don't, that's good too, given 
we can just federate in to whatever proprietary thing they come up with 
anyway, when they get that far :)

>> Just as even Microsoft eventually ditched X.400 in favour of SMTP
>> when it became apparent that SMTP had won the mail protocol war,
>
> Yes, before commerce was on the Internet. Internet just once was able
> to take some folks by surprise and impose itself over ISO standards.

Then I think it's time to take back the internet ;) - and not in the 
context of privacy, for once, but in terms of fighting for 
interoperability and federation.


>>> Who is the regulatory body that
>>> would enforce such a standard on them?
>>
>> A forward-looking government who wants to avoid monopolies or
>> protect citizens from vendor-lock-in might consider enforcing
>> interoperability.  Just as telco regulators enforce GSM portability.
>
> Yes, and if they understand the implications for the future of
> democracy they should also mandate metadata protection with it
> as the law proposal on http://youbroketheinternet.org suggests.

I suspect the regulation approach has some hope with the metadata 
protection problem.  Good luck.  The message probably needs to be made 
less hysterical for the mainstream to pay attention, and given the 
negative association the mainstream has with pirate parties, you may 
need to shake the association there too.

Meanwhile, I hope that projects like Matrix can game the capitalist 
system sufficiently to claim back our aspects of the internet without 
having to resort to legislation :)

>> Otherwise, hopefully, the ecosystem of smaller and newer players
>> will make the most of Matrix and showing that consumers /can/ care
>> about interoperability if it's clearly linked to improved UX, and
>> eventually Google/Apple will reconsider.
>
> I don't see standards improving UX. Since you are forced to decide
> upon certain standard behaviours you are giving up flexibility.

No. Standards are layers. Just as the IP federated standard opens up a 
whole world of flexible behaviours, as does HTTP, so does Matrix.

> Whatever you manage to improve over existing silo apps can be retro-
> actively integrated by them - but it is much harder for you to
> retroactively integrate cool features the silos have come up with
> when you already made some choices and have a codebase spread over
> many different places.

Not if the extensibility can be encapsulated as data.  If some silo app 
comes up with a new feature, then that's great.  They can chose not to 
federate it, or they can federate it as any kind of JSON message-passing 
API they like, or they can map it to an existing 'official' dialect of 
Matrix data types.

We don't care.  As long as the core communication (IM and VoIP) works, 
then Matrix has done its job.  Hopefully the value of federation and the 
matrix transport will then be clear enough that folks may use it for 
their differentiating features too, but of course we can't force that.

>>> Get friends with regulatory bodies. I don't think anything else can
>>> force companies into playing this game.
>>
>> It certainly wouldn't hurt.  But the other body who can force
>> companies into playing this game are the end-users. *IF* one can
>> find a way to show clear benefit to doing so.  And yes, this is
>> hard, but just because everyone else has failed so far doesn't mean
>> that we shouldn't try :)
>
> You can lure consumers into focusing on certain features of a product,
> but not really into the abstract notion of federation. Especially if
> certain promises like being independent and in control of your data
> just aren't true at all.

They are true.  You get to pick the server your account sits on, and the 
clients you use to access it.  You get to pick who to communicate with, 
and so you know which servers will also get copies of your conversation. 
  And you can run your own server and be independent, and then you 
control the reliability and behaviour of your server entirely.

> Whoever has been trying that won't believe
> this lie again. Consumers see their data going to servers and having
> the choice between the devil and the beelzebub. So to them there just
> isn't a real difference between a silo or a company that says it is
> openly federated. It's always folks trying to sell them something.

If a user logged into WhatsApp and Facebook with the same phone number 
and saw that their conversations were magically synced, I think they'd 
notice somehow.  And they might even consider it an improvement.

>> Perhaps we'll get there in the end :)
>
> You can spare yourself a whole lot of work if you skip the
> dead-end-street called federation.
>
> But hey, I'm still curious if anyone on this list can come up with a
> credible scenario for the federation architecture other than as an
> alliance against a market leader or as a governmental regulation.

Do you seriously think that anyone else on the list has been reading us 
ranting at each other?! ;D

M

-- 
Matthew Hodgson
matrix.org