With the advent of technology characterized by heterogeneity and the increasing offers of services and applications, a user’s Quality of Experience (QoE) has become a crucial factor for the success or failure of new applications and services. Content adaptation is the process of selecting, generating, or modifying content to suit the user’s preferences, consumption style, computing and communications environment, and usage context. This can be applied within media types, such as reducing image size or resolution, and across media types, like converting video from one codec to another. Content adaptation can be implemented through three general approaches: Client-side approach (user); server-side approach (media source); or, intermediate-proxy approach between the client and the server.10
Key Insights
- Today, online video services like mobile TV, IPTV, or WebTV, are providing content adaptation techniques under the form of adaptive streaming protocols, which mostly rely on network status, CPU usage, and screen resolution.
- The proliferation of connected devices stresses the need to address the increase in the end-user’s expectations on quality of experience together with network-friendly approaches. For this purpose, adaptation of content and services are required to meet users’ preferences, application expectations, and network characteristics, among others.
- To improve user satisfaction, we need to consider contextual information about the user, the terminal, the networks, and the content.
The client-based approach is performed by the client’s device, therefore providers can create one version of content and the same content can always be displayed on different devices. The terminal selects the content, which is matching the most of its capacities and the available bandwidth. Examples are Scalable Video Coding (SVC), where the delivered content is coded in different versions in the form of different layers and the client terminal selects the layers that best match the terminal capacity and network status; and HTTP Adaptive Streaming (HAS), where content is coded considering different quality levels and the terminal requests the best matching content based on device capacity and available bandwidth.
In the server-based approach, the content is adapted before being delivered to the client, thus reducing the transmission time and the bandwidth consumption since the delivered content is in an adapted form (only the content users need will be sent). This also reduces the processing time at the client side.11 Examples include Windows Media Services, Adobe Flash Media Server, and QuickTime Streaming Server, where they store multiple variants of the same content in the server and the content best matching to the client content is selected through the client identification sent in the client request to the client profile.1
In the intermediate proxy approach, content adaptation is done by an intermediate element. For example, in Wegner et al.,19 the Media Aware Network Element (MANE) is a proxy between the server and the client that gathers the terminal capacity in the client side and the network characteristics and selects the best adapted content based on the terminal and network status. On the other hand, several research contributions exist on content adaptation methods that adapt the content and its delivery based on information related to network congestion, terminal capacity, measured QoE. However, to our knowledge, there is no method that combines all these factors for both content adaptation and delivery. In our work, we consider a multidimensional approach that adapts the content and its delivery means for enhancing the user’s experience. Adapting the content is mainly in terms of changing bit rate, changing codec, while adapting the content delivery means is related to how content will be delivered (unicast, multicast, through which access network). The adaptation process should consider the measured QoE coupled with context information on the user, devices, network, and the content itself to take the adequate adaptation decision. Our notion of QoE is explained in Diallo et al.,4 where the user experience is evaluated as a function of different context concerning the user (preferences, content consumption style, level of interest, location), his environment including the terminal used (for example, capacity or screen size) and network (available bandwidth, delay, jitter, packet loss rate).
The adaptation process should consider the measured QoE coupled with context information on the user, devices, network, and the content itself to take the adequate adaptation decision.
In this article, we examine the existing video streaming techniques and compare the HTTP-based techniques with other techniques. We then focus on the content delivery means, present the content adaptation, and discuss the research contributions for content adaptation and the adaptation of content delivery means.
Video Streaming Techniques
Today there are several streaming methods to deliver audiovisual content, including: HTTP streaming, Real Time Streaming Protocol (RTSP, developed by IETF), Microsoft Media Services (MMS), and Real Time Messaging Protocol (RTMP) for Adobe Systems. Here, we review these different techniques.
HTTP-based streaming techniques carry out a dynamic content adaptation before and/or during the session following a client-based approach. The adaptation is managed by the player. The different HTTP-based streaming techniques are:
- HTTP download. Audiovisual content is delivered through the HTTP protocol just as classical Web objects/pages. To view the content, the media player on the terminal must wait until the video is downloaded in the full disk of the terminal.
- HTTP progressive download is based on HTTP and allows the player to begin playback of the media before the download is complete. The key difference between media streaming and the Progressive Download is in how the digital media content is received and stored by the end user’s device that is accessing the digital media. The media player for the Progressive Download playback makes use of metadata located in the header of the file and a buffer in the user device. When a specified amount of content becomes available in the buffer, the playback is started. This specified amount of buffered content is embedded in the media file by the content producer in the encoder settings and is reinforced by additional buffer settings imposed by the media player. It can also be computed by the player using encoding profile (codec, bitrate, and so on).
Adaptive streaming based on HTTP (HAS)13,15 allows a multi-rate client-based streaming for multimedia content, where different bitrates and resolutions are available for the same content and the streaming is managed by the client. As explained in Oyman and Singh,16 HAS deployment presents opportunities for services and content providers when compared with traditional adaptive streaming techniques. The server sends a manifest file containing the description of content pieces namely chunks (supported codec, minimum bandwidth, resolution, bitrates, URL). The Media Player then selects the most adequate fragment based on the available bandwidth and device capacity (CPU load, supported codec, bitrate). The following are examples of Adaptive Streaming methods based on HTTP.
- Apple HLS (HTTP Live Streaming): This solution was very quickly adopted by Over The Top (OTT) players and is currently available on all Apple devices (iPhone, iPad, iPod) as well as some set-top boxes (AirTies, Netgem, Amino) and software players (VLC media player release 1.2.0, QuickTime X Player).
The native codec chosen for HLS is MPEG H.264 for video and Advanced Audio Coding (AAC) for audio. HLS is implemented as follows: Encoding video in H.264/TS format at different bitrates. For each encoding profile, a Stream Segmenter cuts the content into pieces called chunks, typically 10 seconds each, and generates a play list file (m3u or m3u8) with a URL for each chunk of this encoding profile. Generating a general index file (manifest) indicating each available encoding profiles (bitrate, codec) and the URL of the corresponding playlist files. Distributing content chunks, playlists, and manifest through an HTTP server (origin or cache). Looking for the most suitable bitrate by the receiving device, which can be a laptop, set-top boxes, or mobile phone and selects the most suitable chunk based on the device capabilities and CPU status.
- Google WebM is the Google HTTP-based solution for content adaptation proposed in 2010 to provide an OTT solution that is royalty-free. It uses VP8 video codec for video and Vorbis for audio and does not require segmentation of the media into chunks. However, one media stream is seen as one file. To stream a video through WebM, the following procedure takes places. Encoding the video and audio content in VP8 and Vorbis respectively, in different bitrates (such as, quality profiles). Multiplexing them into a WebM file, which must be automatically refreshed in real time in the case of a live video. Using a Web server (origin or cache) to deliver the WebM files. The adaptive bitrate process mainly relies on the server that selects the audio/video streaming bitrates before multiplexing and pushes in an output buffer all the content packets ready to be sent. While sending the buffer content to the network, the server detects if there is enough bandwidth toward the client, otherwise it scales down to a lower-quality profile (lower bitrate).
- Microsoft Smooth Streaming is the protocol released by Microsoft in 2009 as an extension of Silverlight 3.0,a an application for writing and running rich Internet applications. It supports the H.264 and AAC codecs respectively for video and audio. The general principle behind Smooth Streaming is quite similar to HLS streaming: that is, encoding video and audio in different bitrates (such as quality profiles); using a Stream Segmenter to generate content fragments (chunks) and multiplexing them into a container; distributing video content through an HTTP server (origin or cache); generating and distributing a manifest file that lists the available profiles (bitrates, resolution), languages, and corresponding URLs for chunks.
Once the client receives the manifest, it is able to request some indexed fragments according to its environment (available bandwidth, screen resolution, supported codec). Like HLS, Smooth Streaming leaves the client the choice of the relevant bitrate.
- Adobe HTTP Dynamic Streaming (HDS). Adobe’s solution for streaming media over HTTP is a comprehensive open source video delivery. The Adobe HDS principle is similar to Microsoft Smooth Streaming. The differences, however, include: creation of manifest files (.f4m); creation of segmented files (.f4f) that correspond to chunks (fragments); creation of index files (.f4x) containing specific information about the fragments inside the segmented files (available bitrates, codecs, URLs to stream content). All these files are multiplexed into a single stream and sent to the client device. The supported codecs for HDS are H.264 and VP6 (video), and AAC or MP3 (audio). The terminal in the manifest file has many quality choices and selects the most suitable.
- Dynamic Adaptive Streaming over HTTP (DASH). MPEG DASH is a promising ISO Standard for video streaming services over HTTP published in April 2012 and is gaining popularity.18 DASH has the potential to replace existing proprietary technologies like Microsoft Smooth Streaming, Adobe Dynamic Streaming, and Apple’s HLS. A unified standard is needed because it will help rationalize cost storage, development, maintenance, support, and evolution for all DASH devices. All HTTP-based adaptive streaming technologies have two components: the pure encoded A/V streams, and manifest files that indicate to the player which streams are available (bitrates, codecs, resolutions, among others) and how to access them (for example, chunk URL). For DASH, the AV streams are called the Media Presentation, while the manifest file is called the Media Presentation Description that is encoded in an XML format.
Like other adaptive streaming techniques, this manifest identifies alternative streams, their respective URLs, network bandwidth, and CPU utilization. On this basis, the player chooses the most adapted stream. Two types of file segment types are allowed in DASH: MPEG2 TS (currently used by HLS), and ISO Base media file format (ISO BMFF, currently used by Smooth Streaming and HDS). This simplifies potential migration of existing adaptive streaming platforms to MPEG DASH, as the media segments can often remain the same, and only the index files need to be migrated to the Media Presentation Description (MPD) format.
Adaptive streaming techniques not based on HTTP. RTSP,b RTMP,c and MMSd streaming techniques are not HTTP based and the adaptation (if it is enabled) is managed by the server following a server-centric approach. In RTSP, the content adaptation can take place making use of the Real Time Control Protocol (RTCP) reports sent between the clients to the server. These reports contain information such as packet loss, jitter, RTT (measured/estimated at the client side) that can help the server adapt the content to network conditions. For example, if the connection deteriorates and the transfer rate decreases, the content is streamed with a lower quality so that playback interruptions are avoided, and stream quality is increased if the connection becomes more fluid.
An advantage of this type of streaming technique is the fast start ability, that is, to start content streaming without delay. On the other hand, the drawback mainly lies in the need of a dedicated server with non-negligible license cost (example are Xiph, Icecast, Real Helix Streaming Server, Windows Media Services, Adobe Flash Media Server, QuickTime Streaming Server, and so on) and in difficulties of passing through firewalls. Consequently, this type of streaming is more adapted for users who are connected in managed networks where operators/content providers have a mastery of end-to-end links and application needs in terms of resource consumption. Then they can anticipate the network congestion because most of these streaming techniques use the User Datagram Protocol (UDP) and this transport protocol does not retransmit lost data. Here, we discuss these streaming techniques.
Audiovisual delivery based on RTSP: RTSP was developed by Real Networks, Netscape, and Columbia University. It is available on the products of these companies. The RTSP is a network control protocol designed for use in entertainment and communications systems to control media streaming. The stream consists of cutting the data into packets (fragments) whose size is adapted to the available bandwidth between the client and the server. When the client receives enough packets (buffering), the client application begins to play a package.
Audiovisual delivery based on RTMP: Initially a proprietary protocol developed by Macromedia, RTMP is now Adobe proprietary and free to use. This protocol is used for streaming audio, video, and data over the Internet, between Adobe player (Flash) and a server. Audiovisuals delivery based on MMS, Microsoft’s proprietary network streaming protocol used to transfer data in Windows Media Services. MMS can be transported via UDP or TCP.
Table 1 compares the advantages and disadvantages of streaming techniques through the standard player and the origin server.
Content Adaptation Means for Enhanced Delivery
Here, we describe some means of performing content adaptation for mobile TV, IPTV, and WebTV services.14
Content delivery adaptation in mobile TV. The most common way to adapt content in mobile TV relies on using different encoding profiles according to the Radio Access Technology (or RAT, such as EDGE, UMTS, HSPA, LTE, or Wi-Fi) and to the client’s device (form factor, OS, hardware specificities, screen resolution). While device information can be obtained indirectly through the standard User-Agent HTTP header, an enrichment process adds an extra header containing the client’s identity (MSISDN) and its RAT information. Generally, most of the operators are constrained to use different video profiles for addressing software, hardware, and legal limitations (profiles for 3GPP mobiles, HLS, Windows media). Each TV channel is coded in a different profile and each profile has its own characteristic (video encoding rate, resolution, video codec, frame rate, audio encoding rate, audio codec, which corresponds to different quality level. For example the following video profiles could be proposed as a function of the device and the access type.
Table 2 illustrates fictive profiles that depend on the terminal screen and the access type.
Content delivery adaptation in WebTV and IPTV. For IPTV Live and Video on Demand (VoD) service, the user generally has two choices: Standard Definition (SD) streams and High-Definition (HD) streams. Nevertheless, the operator may enforce a given content quality based on the available network bandwidth and on the end user’s subscription type. WebTV contains TV and/or VoD services offered by a third party available from any Internet access. This method is thus by default available on unmanaged networks, where the best-effort is the unique possible QoS traffic class. Some content providers like French TF1 or France 24 adapt content to the network conditions by using HTTP Adaptive Streaming for their live TV channels. With this technology, users do not care about whatever video quality to choose in order to match their available bandwidth: the video player will automatically request content pieces that are the most adapted to its network status and the device capacity. On the contrary, some OTT content providers for the VoD let the client choose the type of delivery as follows: Streamed mode available in SD (for example, 620kbps stream) for “Instant Viewing;” Progressive Download mode available for both HD (for example, 1500kbps/1GB/WMV file); and in SD (for example, 620kbps/450MB/WMV file), with a possible non-negligible start-up delay for HD that depends on the client’s bandwidth.
Limitation of content adaptation means in current deployment. The current content adaptation for mobile TV, IPTV, and Web TV does not consider a sufficiently large set of parameters to fully enable optimal QoS and QoE. The user context is considered in a limited manner through mainly considering the characteristics of the device used and the network used. The user’s localization (physical/geographical location, but also location in terms of proximity to the access network entities) and the user’s consumption style are not considered. In addition, network context is considered only in terms of bandwidth availability while ignoring the cost of using this bandwidth instead of allocating it to monetized services. Neither is considered the matching degree of the content to the users’ preferences. Consequently, context awareness and user centricity need more consideration in content adaptation through studying users’ localization including their physical location and in terms of proximity to the access network entities; or considering the users’ consumption style for a given type of video service through, for instance, a trade-off between start-up delay and quality content. For example, a fast start is the most important factor for news and live streams, while content quality is the most important one for a movie.
The current content adaptation for mobile TV, IPTV, and Web TV does not consider a sufficiently large set of parameters to fully enable optimal QoS and QoE.
Related Research Contributions
The classification of existing research contributions on content adaptation show three main categories: Content adaptation: Which version of a given content shall we transmit? (codec, bit rate, video resolution). This aspect is related to the encoding of the content information. Content delivery adaptation: How the content is delivered through the network (unicast, multicast, from which server(s), from a cloud, through which access network)? This aspect is related to the service and network aspect of the content transmission. And adaptation of content and delivery: Integrating adaptation of both content and its delivery.
Content adaptation. Several research contributions exist about adapting content based on terminal capacity, network congestion, user profile, and service requirements. For instance, the method in Kim et al.6 provides a QoE-guaranteed service that maximizes the visual expectation of the viewer by considering the size of the LCD panel size on his device. In Mohan and Agarwal,12 users are allowed to define their preferences (user profile) during service subscription, according to some categories based on QoS requirements (streaming, conversational, interactive, background). For example, the streaming traffic class is sensitive to packets losses. The user can also select different types of subscription (Bronze, Silver, Gold, Platinum) for each profile and traffic class. There are maximum and minimum QoS parameters where an adaptation is needed for each type of service and user profile.
The method introduced in Koo et al.8 adjusts the quality level and transmission rate of video streaming on the basis of the wireless channel status (Modulation Coding Scheme, Signal to Interference-Ratio level), location, and client buffer status. The transmission rate is determined as a function of the network context (for example, packet loss, jitter) and some player buffer ratios. In Guo et al.,8 the adaptation of the transmission rate is done on the basis of the pre-buffering time and of the available bandwidth (network status/context), so that QoE is maximized even in case of network congestion.
Khan et al.7 adapts content using two parameters: the “congestion” (C) and the “degradation” (D). The congestion is defined as the fraction of the number of video blocks lost (BL) divided by the total number of video blocks sent (BS) within an interval of time. After predicting the estimated quality of experience (denoted MOSt), the degradation is defined as the difference between the maximum achievable Mean Opinion Score (MOS) and the estimated MOS (MOSt). The Sender Bit Rate (SBR) is computed by on an algorithm using congestion and degradation. W3C proposed by Bakhtiar,1 content adaptation techniques within the Composites Capabilities Preferences Profiles (CC/PP) for Web content and User Agent Profiles (UAProf) for mobile phones. These frameworks can be used to deliver devices contexts (screen size, audio/video capabilities) and users preferences (language, type of content) and allow devices to communicate their capabilities and preferences to servers. The server can then accurately adapt content according to this information.
Adaptation of content delivery. Research contributions on the adaptation of content delivery adaptation focus on two main approaches: the network-centric approach, in which decisions are made at the network side (mainly by network operators) and principally based on the network operator’s benefits; and the user-centric approach, where the decision is based on the user’s benefit, generally, without considering network load-balancing or other users. It should be noted that the user-centric approach has the main drawback from a load balancing perspective, since users generally consider only their own benefit while making decisions. This could result in bad performance of the overall network and service.2 Ksentini et al.9 considers QoE measurements over different access types. After predicting a MOS with Pseudo Subjective Quality Assessment (PSQA), a vertical handover (change in access network and/or technology) is carried out toward the network offering the best MOS; that is, the best QoE.
A new algorithm is called Smooth Adaptive Soft-Handover Algorithm. Its goal is to improve the user-perceived quality while roaming through a heterogeneous wireless network environment.
A new algorithm is defined in Ciubotaru and Muntean3 called Smooth Adaptive Soft-Handover Algorithm (SASHA). Its goal is to improve the user-perceived quality while roaming through a heterogeneous wireless network environment. The score of each connection is evaluated based on a comprehensive Quality of Multimedia Streaming (QMS) score including the following metrics: QoS, QoE, cost, power efficiency, and user preferences. The idea is to adapt delivery in the network that has the best QMS score. Each metric is weighted with a proper coefficient.
Adaptation of content and delivery means. Pichon et al.17 examine the most suitable content to be delivered to the user and selects the best delivery mean. Two decision entities are considered, namely the service manager responsible for the service delivery, and the mobility manager responsible for the network connectivity. For content adaptation, the service management entity will be notified when a terminal requests the streaming of a new video (contents encoding is done with SVC), and decides which version should be sent according to the user rights, to his preferences, to his terminal capabilities, and to the network congestion. The service management entity then provides its decision to the service execution entity that sends the corresponding signalization. For content delivery adaptation, the mobility manager is notified about the network-related events and service requirements and retrieves network-related information and decides which possible network connection(s) must be used for every service based on information such as cost, network load, and user preferences.
Comparison of content adaptation and content delivery adaptation techniques. Here, we provide a comparative study regarding different issues that should be addressed in content adaptation techniques, mainly considering user context, user satisfaction, network congestion, and required resources. There are a lot of research contributions on content adaptation and its delivery. We have therefore classified them into several categories:
Terminal capacity has a lot of advantages among which we can mention the consideration of user context by using the terminal capacity. The disadvantage of this method is not considering the dynamic variation of user’s needs, network resources optimization, and user satisfaction.
Network congestion. The main advantage of this method is the network resources optimization. The lack is no consideration of others context information like user context (his location, his preferences, his profile), user measured quality of experience.
User profile and service requirements. The advantage is the consideration of user context by reviewing his profile. This adaptation technique is easy to implement because the user profile and service needs are known by the server platform. The disadvantage is the no consideration of user context, network congestion in the adaptation process.
Network congestion and terminal capacity considers some context information like terminal capacity and aids resource optimization. The needed resource is a centralized server for gathering network status and terminal feedback. The user location, the measured QoE, is not considered in the adaptation technique.
Network congestion and measure QoE. This type of adaptation considers the user satisfaction (QoE) and optimizes the network use. Some context information is missed in the adaptation technique like user location or terminal capacity. In the literature we notice some limitations in the existing work as follows: The method in Kim et al.6 does not consider the dynamic variation of users’ needs and the network resources optimization. The solution presented by Mohan and Agarwal12 could not adequately enhance the user’s experience since the media source is not aware of the user context (terminal capacity, user preferences, and localization). The method presented by Koo and Chung8 can be difficult to execute because it is not easy to ask each user to implement his profile when he subscribes to a service. Some important context information is missed in the proposed method by Guo et al.,5 like the user localization and terminal capacity.
A general notice is the lack of contributions considering the adaptation of both the content and their delivery means.
Conclusion
This article surveyed content adaptation techniques considering the application-level adaptation and the adaptation of the delivery methods. Firstly, a review on video streaming techniques was given comparing them through several operational, performance, and user-centric criteria. Secondly, the content delivery means were reviewed and compared considering both operational solutions and research contributions.
We notice several research contributions on content adaptation based on terminal capacity, user preferences, network congestion, and so on; however there is a lack of fully contextual methods making use of context information on the terminals, network, and users. In addition, there are limited contributions considering adaptation of both contents and their delivery means. Consequently, we solicit the need to consider the user context (environment, location), terminal context, network context, content context for better content adaptation, and optimized content delivery aiming to improve the user’s satisfaction.
Join the Discussion (0)
Become a Member or Sign In to Post a Comment