Adaptive Media Streaming and Quality of Experience Evaluations using Crowdsourcing

Prof. Christian Timmerer (Alpen-Adria-Universität Klagenfurt, Austria) and Prof. Tobias Hossfeld (University of Duisburg-Essen, Germany)

Summary:

Recently, traditional TV services, Internet TV and mobile streaming services have started converging, and it is expected that this convergence trend will continue with other services. Additionally, new emerging multimedia services are being introduced. These developments in the multimedia arena mean that various content and services will be delivered over different networks, and the users expect to consume these services using those networks, depending on the availability and reach of the network at the time of consumption. This massive heterogeneity in terms of terminal/network capabilities and user expectations requires efficient solutions for the transport of modern media in an interoperable and universal fashion. In particular, in recent years, the Internet has become an important channel for the delivery of multimedia. The Hypertext Transfer Protocol (HTTP) is widely used on the Internet and it has also become a primary protocol for the delivery of multimedia content.

Additionally, standards developing organizations (SDOs) such as MPEG have developed various technologies for multimedia transport and encapsulation, e.g., MPEG2-TS (Transport Stream) and MPEG4 file format. These technologies have been widely adopted and are heavily deployed by various providers and in different applications and services, such as digital broadcasting, audio and video transport over the Internet and streaming to mobile phones, etc. At the same time, many other SDOs such as the IETF, IEEE, and 3GPP have provided various protocols to deliver multimedia content packetized or packaged by such MPEG transport technologies.

In practice, however, these multimedia services are typically deployed over best effort networks utilizing existing infrastructures without any quality guarantees. Thus, the quality as perceived by the end user (denoted as Quality of Experience, QoE) becomes more and more important for the success of multimedia services and the corresponding stakeholders. Insufficient network resources may lead to an increased initial delay before service startup or interruptions during the media playout. The resulting waiting and stalling times are major influence factors on the QoE and, consequently, may significantly decrease the QoE. For online video streaming, both live and on-demand, there are however several possibilities how to adapt the video quality to the actual network conditions to avoid strong quality degradations, e.g., by managing the playout buffer and changing the bit-rate, resolution, and/or frame rate. Thus, the influence of those adaptation possibilities on the QoE and the comparison to stalling or waiting times is important for the design of AMS systems.

The aim of this tutorial is to provide an overview of adaptive media streaming in the context of the recently ratified MPEG-DASH standard. One of the essential differences between DASH and earlier streaming solutions is that the client is in charge of adapting its demands to the bandwidth share that is available for serving it, thus, directly influences the QoE. This client behavior is not standardized and differentiates players. We are going to present how current industry standard solutions, including Microsofts’ Smooth Streaming, Netflixs’ variant, Apples’ HTTP Live Streaming, and Adobe’s Dynamic HTTP Streaming perform; we explain how this performance comes to pass and where each of the solutions is applicable. It may not be obvious, but widely different behavior is desirable for on-demand streaming to wireless devices, on-demand streaming to home cinemas, joining continuous live streams, or waiting for a live event. We present the differences and the best-known adaptation policies. Obviously, the strategies matter only if network performance varies. We will therefore also explain how different bottlenecks in the network can be approached.

Viewers of a video stream observe a good QoE when they watch video in high quality immediately after choosing to do so, without stalls and with good audio-visual quality. Adaptive behavior may provide this without requiring the user to make choices, which makes the service provided more attractive. However, there is a problem in increasing and decreasing qualities as networks’ bandwidth presents itself. The quality change itself is disruptive to user experience; encodings can be changed along several parameters (bit-rate, frame rate, picture resolution, sharpness) which have different properties; enhancing quality at the price of an increased startup delay improves user experience in some situations. We discuss the variety of options and present state-of-the-art knowledge for making these choices from a user-centric point of view. To this end, the influence factors and features of QoE for AMS are discussed and existing QoE results will be presented which may be used for the design but also for the user-centric evaluation of AMS mechanisms.

Additionally, a closer look is given to crowdsourcing as tool for researchers to assess QoE and to conduct their own subjective user studies for the evaluation of multimedia systems. The challenges and key issues of crowdsourcing are outlined and the best practices for the design and evaluation of crowdsourcing tests are summarized.

Outline:

The tutorial can be roughly clustered into three parts. In part I we will provide an introduction to Adaptive Media Streaming (AMS) with a strong focus on Dynamic Adaptive Streaming over HTTP (DASH), part II introduces Quality of Experience (QoE) aspects with respect to AMS/DASH, and part III provides evaluation results and best practices in order to enable QoE for AMS/DASH.

1. Adaptive Media Streaming

• Motivation: what is adaptive media streaming and why is streaming over HTTP so popular?
• Background on MPEG-DASH standardization, the enabler for interoperable streaming over HTTP
• Media Presentation Description (MPD), segment formats (ISOBMFF, M2TS, [WebM]), profiles, deployments (DASH Industry Forum)

2. Quality of Experience

• What is Quality of Experience in general and for Adaptive Media Streaming?
• Influence factors and features of QoE in general for Adaptive Media Streaming
• QoE assessment with crowdsourcing:

3. Evaluation Results and Best Practices

• This section will cover recent results from the presenters and others
• Framework for deriving the optimal QoE for adaptive streaming
• QoE results related to AMS as guideline for designing user-centric adaptation logics: adaptation of frame rate, image quality, resolution, and impact of quality switches on the QoE

Biographies of the presenters:

Christian Timmerer received his M.Sc. (Dipl.-Ing.) in January 2003 and his Ph.D. (Dr.techn.) in June 2006 (for research on the adaptation of scalable multimedia content in streaming and constraint environments) both from the Alpen-Adria-Universität Klagenfurt. He joined the Alpen-Adria-Universität Klagenfurt in 1999 (as a system administrator) and is currently a Senior Assistant Professor at the Institute of Information Technology (ITEC) within the Multimedia Communication Group. His research interests include immersive multimedia communication, streaming, adaptation, Quality of Experience, and Sensory Experience.

Tobias Hoßfeld is full professor and head of the Chair “Modeling of Adaptive Systems” at the University of Duisburg-Essen, Germany, since 2014. He finished his PhD in 2009 and his professorial thesis (habilitation) “Modeling and Analysis of Internet Applications and Services” in 2013 at the University of Würzburg, Chair of Communication Networks, where he was also heading the “Future Internet Applications & Overlays” research group. He has been visiting senior researcher at FTW in Vienna with a focus on Quality of Experience research. His main research interests cover social networks, crowdsourcing platforms, content distribution networks and clouds, as well as investigations on Quality of Experience and Traffic Management for Internet applications like Skype, YouTube, Web Browsing or cloud applications in general. The chair researches in the area of measurement, modeling, evaluation and optimization of adaptive systems in order to solve real world problems.