Netflix is one of the major companies to utilize FreeBSD servers. Their streaming services account for a large amount of internet traffic in many countries.
Netflix will soon use the HTTPS protocol to authenticate and encrypt customer streams, a move that helps ensure what users watch stays secret. The move now leaves Amazon as one of the most noticeable no-shows to the Web encryption party.
Flipping on the HTTPS switch on Netflix’s vast network of OpenConnect Appliances (OCAs) has been anything but effortless. That’s because the demands of mass movie streaming can impose severe penalties when transport layer security (TLS) is enabled. Each Netflix OCA is a server-class computer with a 64-bit Xeon CPU running the FreeBSD operating system. Each box stores up to 120 terabytes of data and serves up to 40,000 simultaneous, long-lived connections, a load that requires as much as 40 gigabits per second of continuous bandwidth. Like Amazon, Netflix has long encrypted log-in pages and other sensitive parts of its website but has served movie streams over unsecured HTTP connections. Netflix took the unusual step of announcing the switch in a quarterly earnings letter that company officials sent shareholders Tuesday.
Netflix first experimented with TLS-protecting customer streams six months ago when it dedicated several servers to deliver only HTTPS traffic to a subclass of users and compared the results to similarly situated servers serving HTTP streams. The results weren’t encouraging. There was as much as a 53-percent capacity hit. The penalty was the result of the additional computational requirements of the encryption itself and the lost ability to use certain Netflix streaming optimizations. The optimizations involve avoiding data copies to and from a server’s user space, something that’s not possible with HTTPS turned on.
“This is not a capacity hit we can absorb in the short term, and we estimate the costs over time would be in the $10s to $100’s of millions per year,” Netflix Director of Streaming Standards Mark Watson wrote in an October 2014 e-mail to W3C public listservs. Netflix decided to forgo the HTTPS rollout until it could get costs in line.
On Wednesday, Watson was back to say Netflix had made enough progress that it was ready to begin rolling out HTTPS for both the entire site and the content itself. Desktop browser tests will be at scale in the next three months, and the job should be completed in the coming year. The performance hit was stemmed by the some TLS optimizations Netflix engineers developed for high-bandwidth FreeBSD applications. The work was presented at this year’s Asia BSD conference.
“We now believe we can deploy HTTPS at a cost that, whilst significant, is well justified by the privacy returns for our users,” Watson wrote in a follow-up e-mail Wednesday. He didn’t quantify the current performance hit or cost that’s incurred now.
Watson’s account casts a new light on the conventional wisdom often cited by encryption advocates that the costs of switching to full-blown HTTPS are negligible. Netflix’s experiments suggest that the costs can be driven down by engineering, but the savings don’t come without a considerable amount of work.
“It’s not clear why that was, but I’m guessing it had to do with the way their servers were configured, the types of cipher suites they were using, lack of hardware, etc.,” Matt Green, a Johns Hopkins University professor and encryption expert, told Ars. “The fact that they’ve made so much progress in only six months probably means that the improvements were probably not so hard to make.”
In a paper that accompanied the Netflix presentation at the Asia BSD conference, engineers from Netflix and FreeBSD laid out a wealth of technical details that helped them realize the performance gains. They wrote:
In order to retain the benefits of the sendfile model but also implement TLS functionality, we designed a hybrid TLS scheme whereby session management would stay in the application space, but the bulk encryption would be inserted into the sendfile data pipeline in the kernel (see Fig 4). TLS session negotiation and key exchange messages are passed from Nginx to the TLS library, and session state resides in the library’s application space. Once the TLS sessions is set up and appropriate keys are generated and exchanged with the client, those keys become associated with the communication socket for the client and are shared into the kernel. For sockets that are flagged for encryption, the TLS bulk encryption happens, via a trip through the Open Crypto Framework, as the data pages are added to the socket buffer. For sessions that require encryption services not available in the Open Crypto Framework, Nginx reverts to the traditional read+send model and performs the encryption in the application space. This ultimately allows for a modular selection of encryption protocols and algorithms that best suit the requirements of the client and the available resources of the server.
Netflix’s entry into the HTTPS party comes as privacy and security advocates are calling on all websites to encrypt all their traffic. The rationale behind the request is that continuous and complete HTTPS protection thwarts state-sponsored attacks that countries like the US and China launch from the Internet backbone. Web encryption is also useful against man-in-the-middle attacks that hijack huge chunks of Internet traffic. In both cases, HTTPS prevents the attacker from surreptitiously injecting malicious packets into the targeted data stream.
Of course, always-on encryption also makes it much harder for anyone with a man-in-the-middle vantage point to log a user’s viewing habits. In December, Netflix officials said company engineers obscured certain URL structures to protect members from deep packet inspection tools that gather data about what they watch online. The rollout of full-blown HTTPS is a much more meaningful way for Netflix to protect member privacy.