[noise] My compromise for dealing with DoS

[Jason A. Donenfeld]

Hi folks,

Here's the solution I've worked out:

The usual structures now get a "struct macs" attached:

handshake_initiation {
    u8 message_type
    u16 sender_index
    u8 unencryped_ephemeral[32]
    u8 encrypted_static[AEAD_LEN(32)]
    u8 encrypted_timestamp[AEAD_LEN(12)]
    struct macs
}

handshake_response {
    u8 message_type
    u16 sender_index
    u16 receiver_index
    u8 unencrypted_ephemeral[32]
    u8 encrypted_nothing[AEAD_LEN(0)]
    struct macs
}

struct macs {
    u8 mac1[32]
    u8 mac2[32]
}

As we know, computing DH is CPU intensive. In order to fend off a
CPU-exhaustion attack, if the server is under load, it may choose to
not process handshake messages, but instead respond with a cookie
reply packet. In order for the server to remain silent unless it
receives a valid packet, while under load, all messages are required
to have a MAC that combines the receiver's public key and optionally
the PSK as the MAC key. When the server is under load, it will only
accept packets that additionally have a second MAC of the prior bytes
of the message that utilize the cookie as the MAC key. Cookies expire
after two minutes and are a MAC of the sender's IP address and source
port using a changing (every two minutes) server secret as the MAC
key. This allows for proof of IP ownership, which can then be rate
limited properly using the usual token bucket mechanisms.

    macs.mac1 = MAC(HASH(public_key || psk), msg[0:offsetof(msg.mac1)])[0:31]
    if (last_received_cookie is empty or expired)
        macs.mac2 = [empty]
    else
        macs.mac2 = MAC(last_received_cookie, msg[0:offsetof(msg.mac2)])[0:31]

The server, after computing these MACs as well and comparing them to
the ones received in the message, must reject messages with an invalid
`macs.mac1` and when under load must reject messages with an invalid
`macs.mac2`.

As mentioned above, when a message with a valid `msg.macs.mac1` is
received, but `msg.macs.mac2` is empty or invalid and the server is
under load, the server may send a cookie reply packet as follows:

    msg = packet_cookie_reply {
        u8 message_type
        u16 receiver_index
        u8 salt[32]
        u8 encrypted_cookie[AEAD_LEN(32)]
    }

    msg.message_type = 4
    msg.receiver_index = little_endian(initiator.sender_index)
    msg.salt = SALT_OF_32_RANDOM_BYTES
    cookie = MAC(responder.changing_secret_every_two_minutes,
initiator.ip_address || initiator.src_port)[0:31]
    msg.encrypted_cookie = AEAD(MAC(salt, responder.static_public ||
(using_psk ? psk : [empty])), 0, cookie, last_msg.macs.mac1)

This fixes a lot of authentication issues with DTLS's cookie scheme.
Relying on the static public key isn't perfect, but using mac1 as AD
helps bind things. Not perfect, but a huge improvement on the existing
work, I think.

Comments, suggestions?

Jason