BitTorrent: The Protocol

Abstract

BitTorrent began on 2 July 2001 as one program by Bram Cohen and a specification short enough to read in an afternoon. Over the next two decades other people rebuilt most of it. The central tracker gave way to a distributed hash table (2005), the .torrent file gave way to the magnet link (2008 to 2012), TCP gave way to a delay-sensitive UDP transport that yields to everything else on the line (2009 to 2010), SHA-1 gave way to SHA-256 (specified 2008, shipped 2020), and a JavaScript port moved the swarm into the web browser (2013). Much of that work came from outside Cohen’s company, much of it was driven by conflict with internet providers, and the protocol that came out the other end was harder to shut down than the one that went in.

A seed (bottom) and peers exchanging pieces of one file; each peer uploads what it already holds. Image: Wikiadd, CC BY-SA 3.0, via Wikimedia Commons.

The Original Design

The first version had three parts. A .torrent file held the file names, the piece size, and a SHA-1 hash for every piece. A tracker, an ordinary web server, kept a list of which IP addresses were currently downloading or seeding which torrent. The peer wire protocol ran over TCP between clients and carried the pieces themselves, governed by the rarest-first and tit-for-tat rules described in Cohen’s biography. The specification, first posted on Cohen’s bitconjurer.org site, was later filed as BEP 3 when the project started its BitTorrent Enhancement Proposal series in 2008. It hashes the “info” section of the torrent file with SHA-1, and that 20-byte infohash became the name of the swarm, a detail that mattered more than anyone expected.

The weak point was the tracker. Everything else was spread across the swarm, but a torrent without a reachable tracker could not find its peers. Trackers also carried legal weight: they were the one place where an operator knowingly coordinated the transfer of a specific file, which made them the target of takedowns and raids. Suprnova, the largest early index, closed on 19 December 2004, days after Hollywood studios announced lawsuits against tracker operators, and when a big tracker went down, every torrent pointing at it stalled.

Removing the Tracker

The fix came from two directions within a month. On 2 May 2005 the Java client Azureus (later renamed Vuze) released version 2.3.0 with a distributed hash table, written by Paul Gardner, that let peers find each other without any central server. About three weeks later BitTorrent’s own “Mainline” client, version 4.1, shipped a different DHT, which Andrew “Drue” Loewenstern had started writing in the summer of 2002. Both were based on Kademlia, a design published by Petar Maymounkov and David Mazières in 2002, in which every node picks a random 160-bit ID and “distance” between IDs is their bitwise XOR. The 160 bits were no accident: they are the same width as a SHA-1 infohash, so a torrent’s name is also a point in the address space, and the nodes closest to that point store the list of its peers.

The two DHTs were incompatible, and remain so. Mainline’s version, documented as BEP 5 (authors Loewenstern and Arvid Norberg), won through adoption: BitComet supported it in June 2005, µTorrent and most other clients followed. Measurements around 2013 counted 16 to 28 million concurrent nodes in it on a given day.

Peer exchange (PEX), added to clients in 2006, filled the remaining gap: once a client had a single peer, it could ask that peer for the addresses of the others it knew about. Between DHT and PEX, a tracker became optional.

The .torrent file was the last central piece. In January 2008 Greg Hazel and Arvid Norberg specified BEP 9, which let peers send each other the torrent’s metadata. A client now needed only the infohash to join a swarm, and the infohash fits into a magnet link:

magnet:?xt=urn:btih:<40 hex characters>&dn=<name>

In November 2009 The Pirate Bay shut down its tracker for good and told users to rely on DHT, PEX, and magnet links. On 29 February 2012 it stopped serving .torrent files for all torrents with more than ten peers. The site then consisted of links containing hashes; the whole catalogue of 1,643,194 torrents was offered as a download of about 90 MB.

The Fight with the Providers

By June 2004 the network-caching firm CacheLogic estimated that peer-to-peer made up about 62 percent of internet traffic and BitTorrent 53 percent of that, which is where the often-repeated “one third of all internet traffic” comes from. TorrentFreak pointed out in 2006 that the figure was a single study’s snapshot and that eDonkey had since overtaken BitTorrent in most countries. The direction was clear anyway: upload-heavy traffic on residential lines built for downloads, and providers began to shape it.

Client developers answered with obfuscation. BitComet shipped a partial “protocol header encryption” in September 2005. In January 2006 the Azureus developers published Message Stream Encryption (MSE), an RC4 scheme keyed on the infohash; after a few days of negotiation µTorrent adopted a compatible version, and Azureus 2.4.0.0 (10 February 2006), µTorrent 1.5 and BitComet 0.63 (both 7 March 2006) shipped it as stable. Cohen opposed the move, suggesting that “some developer has gotten rate limited by his ISP, and is more interested in trying to hack around his ISP’s limitations than in the performance of the internet as a whole.”

In October 2007 the Associated Press, and then the Electronic Frontier Foundation in test results it published on 19 October, showed what the largest US cable provider was doing. Comcast’s equipment watched for BitTorrent seeding and injected forged TCP reset packets into the connection, so that each side believed the other had hung up. When one of the EFF’s test users routed the traffic through a VPN, the resets disappeared. The FCC ordered Comcast in 2008 to stop; on 6 April 2010 the D.C. Circuit ruled in Comcast Corp. v. FCC that the agency lacked the authority to do so, which moved the dispute into the net-neutrality fight where it stayed.

µTorrent and uTP

µTorrent, written in C++ by the Swede Ludvig Strigeus and first released on 18 September 2005, was a Windows client whose small executable gave it its name (µ for “micro”). It claimed 52 million users by November 2009. BitTorrent, Inc. bought it on 7 December 2006. The company’s own client was rebuilt on µTorrent’s code and went proprietary with version 6.0 in 2007.

The acquisition gave BitTorrent, Inc. the installed base to change the transport. The problem it chose to solve was one that users felt directly: a BitTorrent client uploading at full speed filled the oversized send buffers in DSL and cable modems, which, as the specification put it, “can hold several seconds worth of packets,” and every web page and online game on the same line lagged. The usual advice was to cap uploads at about 80 percent by hand. The µTorrent Transport Protocol (uTP, BEP 29, June 2009, by Norberg with design credits to Strigeus, Greg Hazel, Stanislav Shalunov and Cohen) moved the peer connection to UDP and measured one-way delay with timestamps. When queueing delay rose toward a target of 100 milliseconds, uTP slowed down, so it yielded to any TCP traffic on the same link and used only the capacity nobody else wanted.

When the plan became public in late 2008, some commentators predicted that a UDP-based BitTorrent would ignore congestion control and melt the network. It did the opposite. µTorrent 1.8 betas had preliminary support, 2.0 (February 2010) made it the default, and the code was released as the open-source library libutp in May 2010; KTorrent 4.0 was the first free client to use it that month, and Transmission followed. The congestion algorithm was taken to the IETF and published in December 2012 as RFC 6817, LEDBAT (Low Extra Delay Background Transport), with two BitTorrent, Inc. engineers among its four authors.

The Legal Swarm

The same properties made BitTorrent useful to organisations with nothing to hide. Twitter’s engineers Larry Gadea and Matt Freels wrote Murder, released in 2010, which pushed new code to the company’s servers over BitTorrent; Twitter reported that a 40-minute deployment dropped to 12 seconds. Facebook used BitTorrent for server updates, and Blizzard used it for World of Warcraft and StarCraft II patches. On 7 August 2012 the Internet Archive made over a million items available as torrents, each seeded from two of its own data centres plus whoever else held a copy. Linux distributions, which had faced the flash-crowd problem first (see The Warez and Filesharing Era for the other side of the ledger), kept their torrent mirrors.

Version 2 and the Browser

The SHA-1 infohash was both BitTorrent’s name system and its integrity check. Cohen had written a successor specification, BEP 52, in January 2008, and it sat mostly unused until Google announced the first practical SHA-1 collision in 2017. BitTorrent v2 replaces SHA-1 with SHA-256 and hashes each file separately as a Merkle tree over 16 KiB blocks, so a corrupt block can be identified and fetched again on its own, and identical files in different torrents share a hash. It shipped in the widely used libtorrent 2.0 library in September 2020, with “hybrid” torrents that carry both formats so old and new clients can share a swarm.

A browser could not join a swarm at all, because web pages cannot open raw TCP or UDP connections. In October 2013 Feross Aboukhadijeh released WebTorrent, a JavaScript implementation that runs the BitTorrent wire protocol over WebRTC data channels, the peer-to-peer channel browsers built for video calls. Browser peers can only talk to other WebRTC peers, so the project added a desktop client that joins both networks and relays between them; the Brave browser bundled WebTorrent to open magnet links directly.

Dead Ends

Two efforts in this history failed at what they set out to do.

The first was protocol encryption as a shield against throttling. MSE was deliberately light (RC4, keyed on a value every peer already knew) because it was meant to hide traffic, not secure it. That was enough to defeat simple port- and signature-based filters in 2006, but academic studies later showed that the sizes and directions of the first hundred or so packets identify an obfuscated BitTorrent connection with more than 96 percent accuracy. The Comcast case was settled in the courts and regulators’ offices, not by the encryption.

The second was the attempt to turn the protocol into a business. BitTorrent, Inc. built a content-delivery network (BitTorrent DNA), a live-streaming product, a file-sync tool (continued as Resilio Sync), and a legal download store, while its most popular product, µTorrent, drifted toward bundled toolbars. In March 2015 µTorrent installed a cryptocurrency miner called Epic Scale alongside the client; after the backlash it was removed within the month. In 2018 the company, by then renamed Rainberry, was sold to Justin Sun’s Tron. Meanwhile the protocol’s share of downstream traffic fell as streaming took over: Sandvine measured 2.46 percent of global downstream traffic in 2019, though still 27.58 percent of upstream. The swarm survived by becoming a tool for the people who ship large files, and the company that owned the name was not needed for that.