30 votes

Inside Netflix’s bet on advanced video encoding

9 comments

  1. [8]
    nul
    Link
    TL;DR Netflix developed new encoding techniques for videos, and now they must redevelop encoding techniques for live streaming and streaming video games

    TL;DR Netflix developed new encoding techniques for videos, and now they must redevelop encoding techniques for live streaming and streaming video games

    Aaron has spent the past 13 years optimizing the way Netflix encodes its movies and TV shows. The work she and her team have done allows the company to deliver better-looking streams over slower connections and has resulted in 50 percent bandwidth savings for 4K streams alone, according to Aaron. Netflix’s encoding team has also contributed to industry-wide efforts to improve streaming, including the development of the AV1 video codec and its eventual successor.

    Now, Aaron is getting ready to tackle what’s next for Netflix: Not content with just being a service for binge-watching, the company ventured into cloud gaming and livestreaming last year. So far, Netflix has primarily dabbled in one-off live events like the SAG Awards. But starting next year, the company will stream WWE RAW live every Monday. The streamer nabbed the wrestling franchise from Comcast’s USA Network, where it has long been the No. 1 rated show, regularly drawing audiences of around 1.7 million viewers. Satisfying that audience week after week poses some very novel challenges.

    To achieve that, Aaron and her team have to basically start all over and disregard almost everything they’ve learned during more than a decade of optimizing Netflix’s streams — a decade during which Netflix’s video engineers re-encoded the company’s entire catalog multiple times, began using machine learning to make sure Netflix’s streams look good, and were forced to tweak their approach when a show like Barbie Dreamhouse Adventures tripped up the company’s encoders.

    When Aaron joined Netflix in 2011, the company was approaching streaming much like everyone else in the online video industry. “We have to support a huge variety of devices,” said Aaron. “Really old TVs, new TVs, mobile devices, set top boxes: each of those devices can have different bandwidth requirements.”

    To Aaron and her eagle-eyed awareness of encoding challenges, that approach seemed inadequate. Why spend the same bandwidth to stream something as visually complex as an action movie with car chases (lots of motion) and explosions (flashing lights and all that noisy smoke) as much simpler visual fare? “You need less bits for animation,” explained Aaron.

    My Little Pony, which was a hit on the service at the time, simply didn’t have the same visual complexity as live-action titles. It didn’t make sense to use the same encoding recipes for both. That’s why, in 2015, Netflix began re-encoding its entire catalog with settings fine-tuned per title. With this new, title-specific approach, animated fare could be streamed in 1080p with as little as 1.5 Mbps.

    Per-title encoding has since been adopted by most commercial video technology vendors, including Amazon’s AWS, which used the approach to optimize PBS’s video library last year. But while the company’s encoding strategy has been wholeheartedly endorsed by streaming tech experts, it has been largely met with silence by Hollywood’s creative class.

    However, Netflix’s new encoding struggled with one particular scene. “There’s this guy with a very sparkly suit and a sparkly water fountain behind him,” said Aaron. The scene looked pretty terrible with the new encoding rules, which made her realize that they needed to be more flexible. “At (other) parts of the title, you need less bits,” Aaron said. “But for this, you need to increase it.”

    The solution to this problem was to get a lot more granular during the encoding process. Netflix began to break down videos by shots and apply different encoding settings to each individual segment in 2018. Two people talking in front of a plain white wall were encoded with lower bit rates than the same two people taking part in a car chase; Barbie hanging out with her friends at home required less data than the scene in which Mr. Sparklesuit shows up.

    As Netflix adopted 4K and HDR, those differences became even more stark. “(In) The Crown, there’s an episode where it’s very smokey,” said Aaron. “There’s a lot of pollution. Those scenes are really hard to encode.” In other words: they require more data to look good, especially when shown on a big 4K TV in HDR, than less visually complex fare.

    Slicing up a movie by shot and then encoding every slice individually to make sure it looks great while also saving as much bandwidth as possible: all of this work happens independently of the video codecs Netflix uses to encode and compress these files. It’s kind of like how you might change the resolution or colors of a picture in Photoshop before deciding whether to save it as a JPEG or a PNG. However, Netflix’s video engineers have also actively been working on advancing video codecs to further optimize the company’s streams.

    Netflix is a founding member of the Alliance for Open Media, whose other members include companies like Google, Intel, and Microsoft. Aaron sits on the board of the nonprofit, which has spearheaded the development of the open, royalty-free AV1 video codec. Netflix began streaming some videos in AV1 to Android phones in early 2020 and has since expanded to select smart TVs and streaming devices as well as iPhones. “We’ve encoded about two-thirds of our catalog in AV1,” Aaron said. The percentage of streaming hours transmitted in AV1 is “in the double digits,” she added.

    And while the roll-out of AV1 continues, work is already underway on its successor. It might take a few more years before devices actually support that next-gen codec, but early results suggest that it will make a difference. “At this point, we see close to 30 percent bit rate reduction with the same quality compared to AV1,” Aaron explained. “I think that’s very, very promising.”

    In 2016, Netflix released a 12-minute 4K HDR short film called Meridian that was supposed to remedy this. Meridian looks like a film noir crime story, complete with shots in a dusty office with a fan in the background, a cloudy beach scene with glistening water, and a dark dream sequence that’s full of contrasts. Each of these shots has been crafted for video encoding challenges, and the entire film has been released under a Creative Commons license. The film has since been used by the Fraunhofer Institute and others to evaluate codecs, and its release has been hailed by the Creative Commons foundation as a prime example of “a spirit of cooperation that creates better technical standards.”

    Netflix launched its very first livestream in March of 2023, successfully broadcasting a Chris Rock comedy special to its subscribers. A month later, it tried again with a live reunion event for its reality show Love Is Blind — and failed miserably, with viewers waiting for over an hour for the show to start.

    Netflix wants to avoid further such failures, which is why the company is playing it safe and moving slowly to optimize encoding for live content. “We’re quite early into livestreaming,” Aaron said. “For now, the main goals are stability, resilience of the system, and being able to handle the scale of Netflix.” In practice, this means that Aaron’s team isn’t really tweaking encoding settings for those livestreams at all for the time being, even if it forces her to sit through the livestream of the SAG Awards show without being able to improve anything. “We’re starting with a bit more industry-standard ways to do it,” she told me. “And then from there, we’ll optimize.”

    Live is unpredictable: that’s not just true for encoding but also for Netflix’s business. The company just inked a deal to show two NFL games on Christmas Day and will begin streaming weekly WWE matches in January. This happens as sports as a whole, which has long been the last bastion of cable TV, is transitioning to streaming. Apple is showing MLS games, Amazon is throwing tons of money at sports, and ESPN, Fox, and Warner Bros. are banding together to launch their own sports streaming service. Keeping up with these competitors doesn’t just require Netflix to spend heavily on sports rights but also actually get good at livestreaming.

    20 votes
    1. [7]
      lou
      (edited )
      Link Parent
      I'm sometimes surprised to see that many people compare Netflix's service unfavorably in comparison to its competitors when it comes to how it serves video. I just don't care about bells and...

      I'm sometimes surprised to see that many people compare Netflix's service unfavorably in comparison to its competitors when it comes to how it serves video. I just don't care about bells and whistles: I need my videos to play just as I hit play, stop just as I hit stop, and promptly reach whatever time code I want. Netflix is, consistently, the absolute best at the basics of giving me the video I want when I want. It is the only one that, sometimes, gives a similar feeling to playing a local file. I find it interesting to notice that this dedication is reflected by new developments and the history preceding them.

      23 votes
      1. [3]
        whs
        (edited )
        Link Parent
        I paid for 4k Netflix, and I realize Netflix has no quality selector like YouTube. Now I don't know if Netflix actually give me the quality I paid for. (I don't have 4k screen, at most 1440p) Some...

        I paid for 4k Netflix, and I realize Netflix has no quality selector like YouTube. Now I don't know if Netflix actually give me the quality I paid for. (I don't have 4k screen, at most 1440p) Some of the problems I found are:

        • You can't get 1080p on Linux.
        • You can't get 720p on Firefox on Windows either. The only fully supported browser for some reason is Edge (and not other Chromium-based browsers) or the Windows app which is way worse than the web.
        • My Chromecast Gen 3 is capped to 720p. I end up buying Chromecast with GTV
        • I moved the old Gen 3 to my old Dell monitor used as TV. I think it selected 5.1 audio (without option to change to stereo except changing to other languages without 5.1 audio) which make dialogue inaudible. I had to buy a HDMI audio splitter that forcefully cap to 2.1 audio.
        • Now that I bothered to check ctrl-alt-shift-d, apparently my Edge was using 1080p all along. I might have to buy the HEVC decoder from Microsoft, change the cable from DP to HDMI, etc. There's not even a check utility provided

        And all these there's no indicator that I'm not using optimal settings for the setup. I just have to feel that the content isn't right and workaround it outside of Netflix settings.

        12 votes
        1. [2]
          Crespyl
          Link Parent
          It used to be possible to get Netflix to send up to 1080p to Firefox, even on Linux, by "asking the server nicely" with an addon that would modify the player JS to add 1080p as a supported profile...

          It used to be possible to get Netflix to send up to 1080p to Firefox, even on Linux, by "asking the server nicely" with an addon that would modify the player JS to add 1080p as a supported profile for the device type. The addon I used to use is here: https://github.com/TheGoddessInari/netflix-1080p-firefox, not sure if it still works, there's a handful of other forks and different implementations of the same idea.

          This link should still go to their streaming test video that will display a bunch of information about the streaming parameters, which is handy for debugging.

          Agreed about the other frustrations, it's not a great experience to have to keep debugging why resolution or audio aren't working right. I had similar problems with Prime Video, but without any (known) addons to fix the streaming resolution. In that case I've just resorted to piracy in a handful of cases.

          3 votes
          1. whs
            Link Parent
            I tried that on Brave a few months ago, and it seemed to work. I believe it still doesn't do HDR or spatial audio (at most 5.1) so I end up adding the Netflix site on Edge as PWA to hide that it...

            I tried that on Brave a few months ago, and it seemed to work. I believe it still doesn't do HDR or spatial audio (at most 5.1) so I end up adding the Netflix site on Edge as PWA to hide that it is Edge.

            1 vote
      2. [2]
        DeaconBlue
        Link Parent
        I have a ton of complaints about Netflix, but delivery of the video itself hasn't been one of the complaints (at least not for years). Most complaints I hear are about their hair trigger on...

        I have a ton of complaints about Netflix, but delivery of the video itself hasn't been one of the complaints (at least not for years).

        Most complaints I hear are about their hair trigger on flagging your account for password sharing (which is what got me to get rid of it, even though I wasn't password sharing), the less than stellar choice of things to actually watch, and the app itself being really annoying (opening to their mobile game advertisements ???).

        8 votes
        1. Gummy
          Link Parent
          Netflix really has no chill about password sharing. My wife and I had a shared account while I was still working on my degree. Her work and campus were in opposite directions. I guess two people...

          Netflix really has no chill about password sharing. My wife and I had a shared account while I was still working on my degree. Her work and campus were in opposite directions. I guess two people 40 miles apart during the day was too much for them.
          Now I just host a jellyfin server backed up by sonarr/radarr since I probably won't ban myself.

          8 votes
      3. CptBluebear
        Link Parent
        As far as I can tell that's usually bitrate grouches complaining about the quality of the video file and not necessarily the experience itself. I'm sure there's people that dislike the player too,...

        As far as I can tell that's usually bitrate grouches complaining about the quality of the video file and not necessarily the experience itself.

        I'm sure there's people that dislike the player too, because of course, but the bitrate argument is the one I've heard most often.

        7 votes
  2. tomf
    Link
    with the more traditional method of an m38u serving ts segments, this totally makes sense and would be relatively easy to set up. it’ll be interesting to see how this is implemented.

    with the more traditional method of an m38u serving ts segments, this totally makes sense and would be relatively easy to set up. it’ll be interesting to see how this is implemented.