Encryption Protocols

Off-the-Record Messaging: Where Deniable Encryption Started

July 16, 2026 7 min read Haven Team

In 2004, three researchers published a paper with a pointed subtitle: "Why Not To Use PGP." Their argument was that encrypting instant messages with PGP gets the security properties backwards, and their replacement protocol, Off-the-Record Messaging, introduced ideas that every serious messenger now treats as mandatory. OTR itself is nearly gone. Its ideas run inside billions of phones.


The paper was "Off-the-Record Communication, or, Why Not To Use PGP," by Nikita Borisov, Ian Goldberg, and Eric Brewer, presented at the Workshop on Privacy in the Electronic Society in 2004. Its core observation: a private conversation in person has two properties that PGP-encrypted email destroys.

First, once the conversation is over, it is over. Nobody can later produce a transcript and prove what was said. Second, if someone steals your house keys tomorrow, they cannot retroactively hear what you said in your kitchen yesterday. PGP fails both tests. A signed message is durable, transferable proof that you wrote those exact words. And a message encrypted to a long-term key stays decryptable for as long as that key exists, so a key stolen years later unlocks years of archived ciphertext.

Borisov, Goldberg, and Brewer wanted digital conversations to behave like spoken ones. That requirement produced two properties with names that are now standard vocabulary: forward secrecy and deniability.

Forward secrecy: keys with a shelf life

OTR never encrypts messages directly to a long-term key. Instead, both parties run a Diffie-Hellman key exchange with fresh, ephemeral values, and they keep re-running it as the conversation continues. Each time a message flows in a new direction, the parties advertise new DH values and derive new encryption keys, discarding the old ones.

The consequence is that no single stolen key unlocks the conversation history. The long-term key in OTR exists only to authenticate the initial handshake, never to encrypt content. Compromise it and you can impersonate someone in future conversations, which is bad, but you cannot decrypt anything already said. In 2004, when the norm for "secure" chat was a static key or a plain TLS tunnel to the server, this was a genuinely unusual design decision.

Deniability: encryption that refuses to testify

The stranger idea was deniability, and OTR implemented it with two mechanisms that still read as clever twenty years later.

Malleable encryption. OTR encrypts with AES in counter mode, a stream cipher construction. Stream ciphers are malleable: anyone who can guess the plaintext can flip bits in the ciphertext to produce any other plaintext of the same length. For data integrity this sounds like a defect, and inside a live conversation OTR prevents tampering with authentication codes. But as forensic evidence, malleability is a feature. A ciphertext that anyone could have forged proves nothing about who wrote it.

Published MAC keys. Messages are authenticated in transit with a MAC (a symmetric integrity check) rather than a digital signature. Both parties hold the MAC key, so each party already knows the other could have forged any given message. Then OTR goes one step further: once a MAC key is no longer needed, the protocol broadcasts it in plaintext as part of the conversation. After the fact, literally anyone who captured the traffic could have forged messages with valid MACs. The transcript authenticates the conversation while it happens and is worthless as proof afterwards.

The design principle

Signatures prove authorship to anyone, forever. MACs prove authorship only to someone who already holds the key, and only until the key is public. OTR chose MACs precisely because they make weaker promises. In a private conversation, the weaker promise is the correct one.

The socialist millionaires check

Version 2 of the protocol added a verification method that remains one of the more user-friendly ideas in applied cryptography. Comparing key fingerprints over the phone works, but most people never do it. OTR's alternative, the Socialist Millionaire Protocol, lets two people verify each other by answering a shared question.

The name comes from a variant of a classic puzzle: two millionaires want to learn whether they are equally rich without revealing their fortunes. The protocol lets two parties learn whether their secrets match without revealing the secrets. In OTR, you would pick something only the two of you know ("the name of the restaurant where we met") and both type the answer. If the answers match, the protocol confirms it and binds the result to the session keys. If they differ, each side learns only that they differ, and an active attacker in the middle learns nothing usable for a second attempt.

This is key verification for people who will never read a hex fingerprint, and it deserved more adoption than it got.

Why OTR faded

Through the late 2000s, OTR was the tool for encrypted chat. It shipped as a plugin for Pidgin, came built into Adium on the Mac, and rode on top of any messaging network the client supported, XMPP especially. Journalists and activists relied on it for years. Its limits were structural, and they became fatal when messaging moved to phones:

The protocol that solved these problems is the one you are probably carrying right now. TextSecure, the app that became Signal, started from OTR's ratcheting idea and asked what it would take to make it work asynchronously. The answer involved precomputed key bundles fetched from a server (X3DH) and a two-part ratchet that can advance even when only one side is talking (the Double Ratchet). Signal's own protocol documentation credits OTR as the ancestor of its DH ratchet. The deniability goal survived the translation too: Signal authenticates messages without producing transferable cryptographic proof of authorship, the same property OTR pioneered, achieved with different machinery.

A fourth version of OTR was drafted in the late 2010s with modernized primitives, but by then the momentum belonged to Signal-descended protocols and to MLS for groups. OTR's libraries still exist and still work. Almost nobody starts a new project on them.

What OTR settled

Protocol design arguments rarely get settled this cleanly. OTR won three of them:

Claim (2004) Status (2026)
Conversations need forward secrecy, so keys must be ephemeral and constantly rotated Standard in Signal, WhatsApp, MLS (RFC 9420), TLS 1.3
Message authentication should not create durable proof of authorship Deniable authentication is the default in modern messaging protocols
Long-term keys should authenticate, never encrypt content Universal in modern protocol design

The 2004 paper's title argument also holds up. PGP remains the right tool where its properties fit: durable, storable, asynchronous messages to a known key, which is to say, email. It was never the right shape for conversation. The messaging world took OTR's path, and email encryption and conversation encryption evolved into different protocols for different jobs. Any tool that handles both has to hold both models at once, which is a design constraint we think about a great deal.

Try Haven free for 15 days

Encrypted email and chat in one app. No credit card required.

Get Started →