MseCrypto
Message Stream Encryption / Protocol Encryption (MSE/PE). This is the pure-crypto half of the obfuscated BitTorrent handshake.
This is a port of the key-derivation and framing constants that libtorrent spreads across src/pe_crypto.cpp (io.github.yuroyami.kitetorrent.crypto.Rc4, rc4_init's 1024-byte discard) and the init_pe_rc4_handler / write_pe3_sync / write_pe_vc_cryptofield helpers in src/bt_peer_connection.cpp. Only the byte transforms and key schedule live here. Reading and writing the handshake over a socket needs I/O, so it lives in the session artifact instead (peer/Mse.kt).
The MSE handshake (see the spec at https://wiki.vuze.com/w/Message_Stream_Encryption): after a Diffie-Hellman exchange (see DhKeyExchange) both peers share a 96-byte secret S. The torrent's v1 info-hash acts as the stream key SKEY. From (S, SKEY) each side derives two RC4 keys and obfuscation hashes:
keyA = SHA1("keyA" + S + SKEY): the outgoing peer's encrypt keykeyB = SHA1("keyB" + S + SKEY): the outgoing peer's decrypt key
The roles swap by direction: the initiator (outgoing) encrypts with keyA and decrypts with keyB; the receiver (incoming) does the reverse, so the two RC4 streams line up. Each RC4 stream discards its first 1024 bytes after keying (libtorrent's rc4_handler::set_*_key).
The handshake is located/obfuscated on the wire with three SHA-1 hashes keyed by the literal prefixes "req1", "req2", "req3":
sync hash,
SHA1("req1" + S), from reqSyncHashobfuscated SKEY,
SHA1("req2" + SKEY) xor SHA1("req3" + S), from obfuscatedStreamKeyHash
The verification constant (VERIFICATION_CONSTANT) is 8 zero bytes, RC4-encrypted and matched by the peer to confirm the keys agree.
Properties
Both plaintext and RC4 offered: CRYPTO_PLAINTEXT or CRYPTO_RC4 = 3.
Plaintext (no stream encryption) allowed: bit value 1.
Full RC4 stream encryption allowed: bit value 2.
Number of leading RC4 keystream bytes discarded immediately after keying. libtorrent's rc4_handler::set_incoming_key / set_outgoing_key run (and throw away) exactly 1024 bytes of output before any real data.
Size in bytes of the DH shared secret S and each public key (768 bits).
The MSE verification constant (VC): 8 zero bytes. After the crypto field is negotiated, both peers send this RC4-encrypted; finding 8 decrypted zero bytes confirms the derived keys match (libtorrent's sh_vc / errors::invalid_encryption_constant). A fresh array is returned each call so callers may not mutate shared state.
Functions
The obfuscated stream-key hash written on the wire: SHA1("req2" + SKEY) xor SHA1("req3" + S) (libtorrent write_pe3_sync's obfsc_hash). The receiver recovers SHA1("req2" + SKEY) by XOR-ing the received value with req3XorMask and matches it against each candidate torrent's info-hash.
SHA1("req2" + SKEY): the un-obfuscated stream-key hash.
The XOR mask used to obfuscate the stream-key hash: SHA1("req3" + S). This is libtorrent's dh_key_exchange::m_xor_mask (computed in compute_secret) and is reproduced by DhKeyExchange as well; provided here so the whole MSE derivation can be done from raw S bytes.
The sync hash SHA1("req1" + S) (libtorrent write_pe3_sync). The initiator writes this 20-byte hash so the receiver can find the start of the encrypted handshake by scanning for it in the byte stream.