Our live sports benchmarking analysis continues with the English Premier League opening weekend, Aug 5-7, when we monitored three streaming services across several games. Similar to our Formula 1 analysis we found that there is one service that stands out in terms of the viewer experience. Peacock delivered all three matches with an average SSIMPLUS Viewer Score (SVS) just above 81, which is not ideal but better than most sports streaming services. The overall score delta versus the worst performing provider was 7 points, which is readily noticeable, especially by avid sports viewers. For some parts of the games, the gap widened to 20+pts between Peacock and other streaming providers, increasing the risk of subscriber churn in this competitive environment. Please note that the analysis scores and ranks, top profiles created by streaming providers, on a 0-100 linear scale using the two-time Emmy-award winning SSIMPLUS algorithm.* Actual Viewer Experience of subscribers could be worse due to content delivery and playback issues.
NBC, with its Peacock service, outbid other media giants for the Premier League rights, renewing them for an estimated $2 billion until 2028. Being committed to the PL, which according to Nielsen attracts close to 3 billion viewers worldwide across all season matches, also shows in the viewer experience that Peacock delivered on the opening weekend of the 2022/2023 season.
Peacock was the only streaming service that consistently delivered viewer experience quality above 80 SSIMPLUS Viewer Score (SVS) for all 3 games that we measured (Arsenal vs. Crystal Palace - 80.5 SVS, Liverpool vs. Fulham - 80.8 SVS, Man United vs. Brighton - 81.6 SVS. 81.6 SVS is the highest overall score we have measured for any of the sports so far).
Premier League is the third sport we have monitored in the past 2 months (besides MLB and F1) and the live sports video quality benchmarking of multiple providers shows that:
- In many cases, there are large gaps between the SVS scores of the best and the worst streaming services, and sometimes even between the best and the next best. F1 TV and Peacock, for example, have been far ahead of any other F1 or Premier League providers.
- Even the services that performed better with an average SVS above 80 had a lot of variability inside their delivery - video quality ranging from the low 50s to high 80s SVS. This can cause lower subscriber engagement as the stark quality contrast between different parts of the live stream can annoy and frustrate sports fans.
- Not enough is being done to prevent impairments that can seriously damage viewing experiences. For example, de-interlacing impairment is observed very commonly across sports streaming services. The impairment significantly distorts the shape of fast-moving objects making it hard to assess their position in the video and therefore causing visual discomfort. Dе-interlacing artifacts were present for half of the F1 race in one of the provider’s streams, making it unwatchable. We also observed this impairment in Premier League games as you can see from images #3 and #4 below.
Image 1: Who are the players? This snippet with a SVS of 50 shows severe compression impairments present uniformly across the video other than the graphics overlays. Please note that no player in the snippet is recognizable.
Image 2: Provider 2 - it is very frustrating for viewers when the most important shots are of such low quality - this is a 48 SVS from Provider 2 of the Arsenal vs. Crystal Palace match. Compression impairments are abundantly obvious.
Image 3: Another crucial part of the match is being totally destroyed for viewers as faces are hardly recognizable - was there a foul? Was there any jostle between the Liverpool and Fulham players? SVS of 31.
Image 4: How many legs do you see? If you are seeing 4 legs per player this is due to de-interlacing issues.
With the FIFA World Cup in Qatar coming around the corner services that stream soccer can use the time to ensure their Viewer Experience is up to the expected soccer hype.
To learn more about our live sports benchmarking, visit our booth 1.B10 at IBC 2022 in Amsterdam, Sep 9-12. Book your meeting now.
Subscribe to our blog to receive the next sports streaming report.
What and how we are monitoring
- How we perform the test:
The goal of the report is to assess content processing and encoding performance across a sports workflow. The performance of content delivery and playback is "excluded" by ensuring ideal conditions for delivery and playback while scoring the content. When the content was streamed, we made sure that we were fetching top profiles at all times - which means that the performance of ISP(s), content delivery, and players did not negatively affect viewer experience. As the video encoding and processing performance have huge room for improvement, assessing content delivery and playback performance is part of future work. ABR profile was fixed to the top profile only and the results (including visual examples) are provided just for the top profile. The content is captured and scored in raw form using HDMI capture cards installed on a computer.
SSIMPLUS Scale: what's a good score?
The SSIMPLUS Viewer Score is a linear scale from 0-100 where 0 means bad (a worse quality cannot be imagined) and 100 excellent (without any visible impairments). SVS is highly correlated to how an actual human being - an average viewer watching on their device - will evaluate the video quality. An overall score of 74.5 (the average we measured for one of the MLB weekends) across the streaming platforms monitored is "good enough", even if not pleasing to the eye and mildly annoying, which is why they continue to serve millions of viewers. The viewing experience (and objective scores) are significantly worse than premium SVOD platforms (generally above an SVS of 80). Also, there is significant variation in quality across various streaming options. In addition to that, there is significant variation in the quality of each streaming platform across a game as evident from the visual examples.
The SSIMPLUS Viewer Score (SVS) is the most accurate measurement of how an actual human being watching video content on their device would rate the quality of the video. It is based on more than 20 years of research into the human visual system by some of the most renowned experts in the field, including the inventor of the original SSIM, Professor Zhou Wang, who is also our co-founder with more than 80,000 citations in the field. Our team has won numerous awards for this work including two Emmys. Currently, our products SSIMPLUS LIVE Monitor and SSIMPLUS VOD Monitor are deployed by 5 of the top streaming services in the world. You can read more here.
Impairment identification and localization (where in the delivery chain are the impairments originating)
We can localize any video quality issue if we are at the appropriate points. The current monitoring setup for benchmarking streaming services "watches" content with software probes like viewers do at the last point of a delivery chain. The score does not localize drops in video quality (if the drops are due to bad sources or poor compression, to do that we need to monitor at upstream points in the workflow) but it does identify what viewers are perceiving, which is a great first step towards improving viewer experience. Based on our vast experience in content delivery chains and perceptual quality assessment, we can predict where an issue may be originating from after a deeper analysis of impairments.
- Technical specs (resolutions, codec, bitrate)
The services we compared are monitored at the highest resolution they are capable of delivering and forcing that resolution in respective apps while ensuring that the setup supports very high bitrates. Sports streaming services generally deliver at 1280x720, 1920x1080, and 3840x2160 resolutions at 59.94 or 60 frames per second. The most common codec used is H.264/AVC. The most common ABR standard is MPEG-DASH. Typically bitrates are between 5 and 8 Mbps using variable bitrate approaches.
- Other measurement considerations
SSIMPLUS scores are device adaptive. The data presented in the blog posts correspond to viewer experience on a 65" 4K TV. Higher resolutions do benefit from higher content detail, if available. 4K resolution has a much higher likelihood to get to a 100 SVS and it would get to a 100 when there are no perceptually visible impairments in the content.
The scores provided in this blog are overall scores across the event. We have in-depth per-second and per-frame scores that show quality variation across content.
- How does the score compare from game to game or content to content?
The score considers saliency by modeling visual attention when assessing perceptual quality. Generally speaking, impairments in areas that matter more to viewers drag the score down further than impairments in areas that do not matter as much. As a result of this unique ability of the score, baseball scores are comparable to F1 scores and to any other sport we monitor.
- How we take the screen captures and what their measurements actually show
Content is scored in raw format in real-time using HDMI capture cards. Content shared in this blog are frames extracted from the captures triggered based on quality thresholds. The scores for the frames presented in this blog post are “video scores” measuring overall viewer experience when a specific frame is played on a 4K 65” TV screen and not the score just for the exact frame, which can also be provided.