Mse
The MSE/PE (Message Stream Encryption / Protocol Encryption) handshake. This is the live automaton libtorrent runs in bt_peer_connection.cpp (write_pe1_2_dhkey, write_pe3_sync, write_pe4_sync, read_pe_*). It performs the Diffie-Hellman exchange, the sync-hash and VC obfuscation steps, and crypto negotiation, then returns a transport (MseByteStream when RC4 is selected, the raw stream when plaintext is) over which the normal BitTorrent handshake proceeds.
Wire layout (https://wiki.vuze.com/w/Message_Stream_Encryption):
A->B: Ya, PadA
B->A: Yb, PadB
A->B: HASH('req1', S), HASH('req2', SKEY) xor HASH('req3', S),
ENCRYPT(VC, crypto_provide, len(PadC), PadC, len(IA), IA)
B->A: ENCRYPT(VC, crypto_select, len(PadD), PadD), ENCRYPT2(payload)S is the DH shared secret, SKEY the v1 info-hash. We send an empty IA (len 0) and run the BitTorrent handshake over the established channel afterwards. That costs one extra round-trip compared with sending it inside IA, but it is simpler and fully interoperable.
Properties
Allowed encryption level (settings_pack::allowed_enc_level), expressed as the crypto bitmask the spec puts on the wire: plaintext = 0x01 (MseCrypto.CRYPTO_PLAINTEXT), rc4 = 0x02 (MseCrypto.CRYPTO_RC4), both = 0x03 (MseCrypto.CRYPTO_BOTH). These values intentionally equal libtorrent's enc_level enum (pe_plaintext=1, pe_rc4=2, pe_both=3), so a io.github.yuroyami.kitetorrent.settings.EncLevel value can be passed straight through.
Functions
Passive open for a single torrent infoHash. Returns the negotiated transport. Throws MseException if the obfuscated SKEY doesn't match or verification fails.
Passive open (we accepted a connection that began with a DH key rather than the plaintext BitTorrent handshake): run the MSE handshake as the receiver, resolving which torrent the peer wants from the obfuscated SKEY against infoHashes. Returns the negotiated transport plus the matched info-hash. Throws MseException if no candidate matches or verification fails.
Active open (we dialed): run the MSE handshake as the initiator, offering the crypto methods permitted by allowedEncLevel. Returns the negotiated transport (MseByteStream for RC4, raw for plaintext). Throws MseException on any verification failure.
True if bytes begin with a plaintext BitTorrent handshake (0x13 + "BitTorrent protocol"). The engine uses this to tell a plaintext peer from an MSE peer (whose first bytes are a random DH public key).