WO2016008131A1

WO2016008131A1 - Techniques for separately playing audio and video data in local networks

Info

Publication number: WO2016008131A1
Application number: PCT/CN2014/082390
Authority: WO
Inventors: Yunchao CHEN; Lu Jiang
Original assignee: 21 Vianet Group, Inc.
Priority date: 2014-07-17
Filing date: 2014-07-17
Publication date: 2016-01-21

Abstract

An audio transmission rate and an audio rendering latency are determined for an audio rendering device.Similarly, a video transmission rate and a video rendering latency are determined for a video rendering device.Based on these transmission rates and latencies, a combined audio and video rendering latency is determined.Audio content of audiovisual data is transmitted to the audio rendering device at the audio transmission rate, and comprises timestamps for rendering the audio content.The timestamps are determined based in part on the combined audio and video rendering latency.Video content of the audiovisual data is transmitted to the video rendering device at the video transmission rate and comprises the same timestamps for rendering the video content.The audio or video rendering device decodes content data units from the audio or video content, and renders the content data units at time points based on the timestamps.

Description

TECHNIQUES FOR SEPARATELY PLAYING AUDIO AND VIDEO DATA

IN LOCAL NETWORKS

TECHNOLOGY

[0001] The present invention relates generally to playing audio and video data, and in particular, to separately playing audio and video data in local networks.

BACKGROUND

[0002] In operating scenarios in which a single multimedia player is used to play multimedia content, it is relatively easy to synchronize the playing of audio content in the multimedia content with the playing of video content in the multimedia content. For example, the multimedia device may render a portion of the audio content with a corresponding portion of the video content at the same time point as determined based on a timestamp for playing the portion of the audio content.

[0003] On the other hand, in operating scenarios in which multiple audio and video devices are used to play multimedia content, it becomes relatively easy to be out of sync in the playing of audio content and the video content among the audio and video devices. For example, the audio content may be delivered to an audio device over a network that is entirely different from another network over which the video content is delivered to a display device. Many differences and variations in device types, underlying network types, data sizes, network conditions, processing powers, memory spaces, load conditions on devices, etc., tend to generate a large amount of time differences in the playing of the audio content and the video content of the same media program by these different devices.

[0004] The approaches described in this section are approaches that could be pursued, but not necessarily approaches that have been previously conceived or pursued. Therefore, unless otherwise indicated, it should not be assumed that any of the approaches described in this section qualify as prior art merely by virtue of their inclusion in this section. Similarly, issues identified with respect to one or more approaches should not assume to have been recognized in any prior art on the basis of this section, unless otherwise indicated.

BRIEF DESCRIPTION OF DRAWINGS

[0005] The present invention is illustrated by way of example, and not by way of limitation, in the figures of the accompanying drawings and in which like reference numerals refer to similar elements and in which:

[0006] FIG. 1A and FIG. IB illustrate example system configurations;

[0007] FIG. 2 illustrates an example media device;

[0008] FIG. 3 illustrates an example audio rendering device;

[0009] FIG. 4 illustrates an example video rendering device;

[0010] FIG. 5A and FIG. 5B illustrate example process flows; and

[0011] FIG. 6 illustrates an example hardware platform on which a computer or a computing device as described herein may be implemented. DESCRIPTION OF EXAMPLE EMBODIMENTS

[0012] Example embodiments, which relate to separately playing audio and video data in local networks, are described herein. In the following description, for the purposes of explanation, numerous specific details are set forth in order to provide a thorough understanding of the present invention. It will be apparent, however, that the present invention may be practiced without these specific details. In other instances, well-known structures and devices are not described in exhaustive detail, in order to avoid unnecessarily occluding, obscuring, or obfuscating the present invention.

[0013] Example embodiments are described herein according to the following outline:

1. GENERAL OVERVIEW

2. STRUCTURE OVERVIEW-

3. MEDIA DEVICES

4. AUDIO RENDERING DEVICES

5. VIDEO RENDERING DEVICES

6. EXAMPLE PROCESS FLOW

7. IMPLEMENTATION MECHANISMS - HARDWARE OVERVIEW

8. EQUIVALENTS, EXTENSIONS, ALTERNATIVES AND

MISCELLANEOUS

1. GENERAL OVERVIEW

[0014] This overview presents a basic description of some aspects of an example embodiment of the present invention. It should be noted that this overview is not an extensive or exhaustive summary of aspects of the example embodiment. Moreover, it should be noted that this overview is not intended to be understood as identifying any particularly significant aspects or elements of the example embodiment, nor as delineating any scope of the example embodiment in particular, nor the invention in general. This overview merely presents some concepts that relate to the example embodiment in a condensed and simplified format, and should be understood as merely a conceptual prelude to a more detailed description of example embodiments that follows below.

[0015] A multitude of content rendering devices may be available for rendering multimedia content at a given location such as a home, an office, a classroom, a theater, etc. Each of these content rendering devices may have its respective set of capabilities different from other content rendering devices. For example, individual content rendering devices may be configured to receive their respective media content over different networks, different data links, different end-to-end network paths, etc. Examples of networks may include, but are not limited to only, any of: Ethernet networks, Wi-Fi networks, Bluetooth networks, point-to-point networks, HDMI video links/networks, etc. An earphone device may be able to receive audio content over a Bluetooth network. An audio receiver system may be able to receive audio content over an audio cable, a radio frequency link, a Wi-Fi link, etc. A display device may be able to receive video content over a video link such as an HDMI link, etc. A tablet computer or a wearable computer may be able to receive audio content and video content wirelessly via a public or private wireless network.

[0016] Transmission rates, transmission delays, content processing delays, network-specific latencies, device-specific latencies, component-specific latencies, etc., may vary greatly among individual content rendering devices. For example, a display device may receive video content over an HDMI link at a high data transmission rate and render the video content with a relatively low or moderate latency. A Bluetooth earphone may receive audio content with a relatively low data transmission rate and render the audio content with a different latency as compared with the display device.

[0017] Techniques as described herein can be used to stream and playing audio and video content of a media program with different content rendering devices relatively smoothly and uniformly, and to prevent respective content renderings of the same media program by the different content rendering devices from becoming out-of-sync. In some embodiments, a media device is used to separate audiovisual data of a received media program into video content and audio content and streamed the video content and audio content separately in different data streams to different content rendering devices (e.g., display devices, audio devices, etc.) for rendering.

[0018] For example, video chunks, decoded from the audiovisual data received by the media device, which comprise video content to be rendered by a specific video rendering device, can be compressed, transcoded or transrated into a video data stream such that the video data stream can be transmitted to the video rendering device at a specific video transmission rate supported by the video rendering device. The media device can adaptively adjust the specific video transmission rate based on transmission and rendering statistics (e.g., related to an end-to-end network path between the media device and the video rendering device, the processing speed of the video rendering device, etc.) collected while the media program is being rendered collectively by the different audio and video rendering devices.

[0019] Similarly, audio chunks, decoded from the audiovisual data as received by the media data, which comprise audio content to be rendered by a specific audio rendering device can be compressed, transcoded or transrated into an audio data stream such that the audio data stream can be transmitted to the audio rendering device at a specific audio transmission rate supported by the audio rendering device. The media device can adaptively adjust the specific audio transmission rate based on transmission and rendering statistics (e.g., related to an end-to-end network path between the media device and the audio rendering device, the processing speed of the audio rendering device, etc.) collected while the media program is being rendered collectively by the different audio and video rendering devices.

[0020] To synchronize the playing (e.g., playback, trick plays, etc.) of the video content and audio content on the different content rendering devices, the media device may be configured to perform clock synchronization operations with individual content rendering devices, in order to synchronize clocks used by the content rendering devices for content rendering to a common clock for content rendering.

[0021] Each data stream of one or more audio data streams and one or more video data streams transmitted by the media device to one or more audio rendering devices and one or more video rendering devices carries the same set of timestamps. A timestamp indicates a specific time point in reference to the common clock for rendering a specific audio data unit or a video frame. A video frame and an audio data unit that are to be rendered at the same time point share the same timestamp.

[0022] Based on observed rendering characteristics, the media device (102) can adaptively determine a combined audio and video rendering latency at any given time while processing the audiovisual data. The combined audio and video rendering latency may be a combination of a maximum rendering latency among the individual rendering devices and a safe margin. The combined audio and video rendering latency may be used to determine the above-mentioned set of timestamps to indicate specific time points at which video frames or audio data units are to be rendered by the individual rendering devices such that all the rendering devices are enabled to render their respective video frames or audio data units at these indicated time points.

[0023] In some example embodiments, mechanisms as described herein form a part of an information processing system, including but not limited to any of: a handheld device, game machine, television, laptop computer, netbook computer, tablet computer, cellular radiotelephone, electronic book reader, point of sale terminal, desktop computer, computer workstation, computer kiosk, plug-in devices, media accessing devices, or various other kinds of terminals and media processing units.

[0024] Various modifications to the preferred embodiments and the generic principles and features described herein will be readily apparent to those skilled in the art. Thus, the disclosure is not intended to be limited to the embodiments shown, but is to be accorded the widest scope consistent with the principles and features described herein.

2. STRUCTURE OVERVIEW

[0025] FIG. 1A depicts an example configuration 100 comprising a media device 102, an audio rendering device 104, a video rendering device 106, one or more (e.g., cloud-based, on premise, internet-based, etc.) media content servers 110, etc. As illustrated in FIG. 1 A, the media device (102) can be communicatively and/or operatively linked with one or more computer networks (108). The one or more computer networks (108) may include, but are not limited to only, any of: one or more of the Internet, intranets, wireless or wire-based local area networks, telecom networks, wireless or wire-based wide area networks (WANs), etc.

[0026] In some embodiments, the media device (102) can be communicatively and/or operatively linked with a first (e.g., local, non-local, etc.) network (112-1) for communicating with the audio rendering device (104), and operatively linked with a second (e.g., local, non-local, etc.) network (112-2) for communicating with the video rendering device (106). Examples of (e.g., local, non-local, etc.) networks between a media device (e.g., 102, etc.) and a rendering device (e.g., 104, 106, etc.) include, but are not limited to only, any of: Wi-Fi networks, Bluetooth networks, infrared links, CDMA networks, GSM networks, HDMI video links, non-HDMl video links, radio frequency (RF) networks, optical links, etc.

[0027] In some embodiments, the first network (112-1) may be the same as the second network (112-2). In some other embodiments, the first network (112-1) may be different from the second network (112-2); for example, the first network (1 12- ) may be a Bluetooth network, whereas the second network (112-2) may be a non-Bluetooth network.

[0028] As used herein, a media device (e.g., 102, a multimedia device, etc.) refers to a computing device configured to separate audiovisual data (e.g., a combination or mixture of audio data and video data, etc.) into audio data component and video data component. The media device (102) can be further configured to synchronize the playing of an audio data stream (generated based on the audio data component) by the audio rendering device (104) with the playing of a video data stream (generated from the video data component) by the video rendering device (106).

[0029] An audio rendering device (e.g., 104, etc.) refers to a content rendering device for rendering audio content, which is a computing device specifically configured to receive an audio data stream from a media device (e.g., 102, etc.) and operate in conjunction with the media device (102) to synchronize the playing the audio data stream by the audio rendering device (104) with the playing of a corresponding video data stream by a video rendering device (e.g., 106, etc.). Likewise, the video rendering device (106) refers to a content rendering device for rendering video content, which is a computing device specifically configured to receive a video data stream from the media device (102) and operate in conjunction with the media device (102) to synchronize the playing the video data stream by the video rendering device (106) with the playing of a corresponding audio data stream by the audio rendering device (104).

[0030] The media device (1 2) can retrieve audiovisual data as described herein from one or more of a variety of audiovisual data sources. For example, the media device (102) can retrieve the audiovisual data from one or more storage media accessible to the media device (102). Additionally, optionally, or alternatively, the media device (102) can retrieve audiovisual data from one or more of local computing devices or remote media content servers (e.g., 110, etc.) accessible to the media device (102). The audiovisual data received by the media device (102) can be formatted in one of a variety of standard-based or proprietary content formats, and received in one of a variety of packages including but not limited to media files, streams, broadcast signals, cable signals, analog signals, digital signals, etc.

[0031] FIG. IB depicts an example configuration 100-1 comprising a media device 102, two audio rendering devices 104-1 and 104-2, a video rendering device 106-1, one or more (e.g., cloud-based, on premise, internet-based, etc.) media content servers 110-1 and 110-2, etc. As illustrated in FIG. IB, the media device (102) can be communicatively and/or operatively linked with one or more computer networks (108).

[0032] In some embodiments, the media device (102) can be communicatively and/or operatively linked with a third (e.g., local, non-local, etc.) network (112-3) for communicating with the audio rendering devices (104-1 and 114-2), and operatively linked with a fourth (e.g., local, non-local, etc.) network (112-4) for communicating with the video rendering device (106-1). Through these networks, the media device (102) establishes an individual end-to-end network path (e.g., 114-1, 114-2, 1 14-3, etc.) with each of the audio rendering devices (104-1 andl04-2) and the video rendering device (106-1).

[0033] In some embodiments, the third network (112-3) may be the same as the fourth network (112-4). In some other embodiments, the fourth network (112-3) may be different from the second network (1 12-4).

[0034] In some embodiments, the media device (102) is configured to synchronize (e.g., a portion of, all, etc.) clocks (used for rendering content) of the audio rendering devices (104-1 and 104-2) and the video rendering device (106-1) to a common clock (used for rendering content). The media device (102) is further configured to separate audiovisual data (e.g., a combination or mixture of audio data and video data, etc.) into audio data component and video data component; generate, based at least in part on the audio data component, two audio data streams for the audio rendering devices (104-1 and 104-2), respectively; generates, based at least in part on the video data component, a video data stream for the video rendering device (106-1); etc. Examples of audiovisual data include, but are not limited to, one or more of: movies, media programs, live TV programs, etc.

[0035] At the same time, while the media device (102) is separating the audio visual data into the audio data component and the video data component and generating the audio data streams and the video data stream, the media device (102) communicates with the audio rendering devices (104-1 andl04-2) and the video rendering device (106-1) to determine individual rendering characteristics of each of the audio rendering devices (104-1 andl 04-2) and the video rendering device (106-1) over a respective end-to-end network path (e.g., 114-1, 114-2, 114-3, etc.). These rendering characteristics can be determined as functions of time based on bandwidth tests, delay or latency tests, latency tests, analyses of statistics collected for operations involved in rendering audio or video content by each of the audio rendering devices (104-1 and 104-2) and the video rendering device (106-1), etc.

[0036] Based on the rendering characteristics that may or may not vary over time, the media device (102) is configured to adaptively determine an individual transmission rate (e.g., a combination of a maximum transmission rate and a safe margin, etc.) that can be supported by each of the audio rendering devices (104-1 andl04-2) and the video rendering device (106-1) at any given time while processing the audiovisual data. A safe margin for setting transmission rate as described herein can be set either with absolute values (e.g., 0.5Mbps, 1 Mbps, etc., over the maximum transmission rate) or relative values (e.g., 5%, 10%, 15%, etc., of the maximum transmission rate). In some embodiments, a safe margin for setting transmission rate can be dynamically and/or adaptively set. For example, if a network, a data link, an end-to-end network path, etc., provides a relatively constant bit rate support, the safe margin for setting transmission rates may be set relatively low (e.g., 0.5 Mbps, 5%, etc.). On the other hand, if a network, a data link, an end-to-end network path, etc., provides a relatively variable bit rate support, the safe margin for setting transmission rates may be set relatively high (e.g., 2 Mbps, 30%, etc.). In various embodiments, network characteristics such as bit error rates, delays, jitters, etc., can be taken into consideration in determining a safe margin for setting transmission rates.

[0037] Based on the rendering characteristics, the media device (102) is also configured to adaptively determine a combined audio and video rendering latency at any given time while processing the audiovisual data. The combined audio and video rendering latency may be a combination of a maximum rendering latency and a safe margin. The maximum rendering latency can be set to be as great as any individual rendering latency among individual rendering latencies of all content rendering devices. In the present example, the maximum rendering latency can be set to be the greatest rendering latency among the individual audio rendering latencies of the audio rendering device 104-1 and the audio rendering device 104-2 and the individual video rendering latency of the video rendering device 106-1. Here, an individual audio or video rendering latency may account for a number of delays and latencies such as transmission delays, content processing delays, processor loads, processor ratings, network-specific latencies, device-specific latencies, component-specific latencies, etc., related to content rendering by an individual audio or video rendering device. In some embodiments, the individual audio or video rendering latency excludes idle time spent by the audio or video rendering device for synchronizing content rendering. A safe margin for setting delays or latencies as described herein can be set either with absolute values (e.g., 50 milliseconds, 100 milliseconds, etc., over the maximum delay or latency) or relative values (e.g., 5%, 10%, 15%, etc., of the maximum delay or latency). In some embodiments, a safe margin for setting delays or latencies can be dynamically and/or adaptively set. For example, if a network, a data link, an end-to-end network path, etc., provides a relatively low delay or latency, the safe margin for setting delays or latencies may be set relatively low (e.g., 10 milliseconds, 5%, etc.). On the other hand, if a network, a data link, an end-to-end network path, etc., provides a relatively high delay or latency, the safe margin for setting delays or latencies may be set relatively high (e.g., 500 milliseconds, 30%, etc.). In various embodiments, one or more of network characteristics such as bit error rates, delays, jitters, etc., cm be taken into consideration in determining a safe margin for setting delays or latencies.

[0038] The media device (102) comprises rate conversion units, modules, etc., to adapt the audio data streams to respective audio transmission rates supported by the audio rendering devices (104-1 and 104-2) and the video rendering device (106-1), and to adapt the video data stream to the video transmission rate supported by the video rendering device (106-1).

[0039] To synchronize the playing rendering of audio content in the audio data streams by each of the audio rendering devices (104-1 and 104-2), of video content in the video data stream by the video rendering device (106-1), the media device (102) includes a set of timestamps in reference to the common clock in each of the audio data streams and the video data stream to the audio rendering devices (104-1 and 104-2) and the video rendering device (106-1).

[0040] In some embodiments, timestamps in the set of timestamps are generated based on timestamps decoded from the audiovisual data and the combined audio and video rendering latency determined at a given time, and are to be interpreted by a recipient device in reference to the common clock used for synchronized rendering by the audio rendering devices (104-1 and 104-2) and the video rendering device (106-1).

[0041] A video frame decoded from the video data stream may share a specific timestamp with respective decoded audio data units in the audio data streams. In some embodiments, the video rendering device (106-1) is configured to render the video frame at a specific time point as indicated by the specific timestamp, while the audio rendering devices (104-1 and 104-2) are configured to render the respective decoded audio data units at the same specific time point as indicated by the specific timestamp.

[0042] As the rendering characteristics associated with the end-to-end network path (114-1, 1 14-2 and 1 14-3), the individual audio and video transmission rates can be dynamically (and individually) adjusted continuously, from time to time. Similarly, the combined audio and video rendering latency can also be dynamically adjusted continuously, from time to time, etc., to adapt to real-time or near-real-time changes occurring in the rendering characteristics involving the audio rendering devices (104-1 and 104-2) and the video rendering device (106-1).

[0043] When the rendering characteristics change relatively fast, smoothing algorithms may be implemented to transition gracefully from a first individual transmission rate at a first time to a second individual transmission rate a second time subsequent to the first time for a content rendering device (e.g., audio rendering device 104-1, audio rendering device 104-2, video rendering device 106-1, etc.). Additionally, optionally, alternatively, jitter buffers, a threshold to limit rate changes per unit time, etc., can be implemented by one or more of the media device (102), the audio rendering devices (104-1 and 104-2), or the video rendering device (106-1).

[0044] The media device ( 102) can retrieve the audiovisual data from one or more of a variety of audiovisual data sources (e.g., 110-1, 110-2, etc.) accessible to the media device (102).

3. MEDIA DEVICES

[0045] FIG. 2 illustrates an example media device (e.g., 102 of FIG. 1A, etc.) comprising a content separation unit 202, a content synchronization unit 204, an audio transmission unit 206, a video transmission unit 208, etc.

[0046] In some embodiments, the content separation unit (202) compdses software, hardware, a combination of software and hardware, etc., configured to retrieve audiovisual data 210 from one or more of internal or external content sources such as media content server 110 of FIG. 1, locally or remotely accessible storage media, media files, media containers, bitstreams from one or more media streaming servers, etc.

[0047] In some embodiments, the audiovisual data (210) as received by the media device (102) comprises multiplexed audio and video data (or a mixture of audio or video data). In an example, the audiovisual data (210) may comprise a combination of a set of video frames (or video chunks) and a set of corresponding audio data units (e.g., audio samples, audio data blocks, audio chunks, etc.). In another example, the audiovisual data (210) may comprise a combination of links (e.g., local links, hyperlinks, etc.) to audio and/or video data.

[0048] In some embodiments, a set of (input) timestamps can be extracted from the audiovisual data (210). Examples of timestamps as described herein include integers indicating time points from a reference time point such as the beginning of a media program, etc.; frame indexes such as video frame indexes, etc.; integers indicating multitudes of a set time interval such as 20 milliseconds, etc.,; logical time values relative to a reference time point (e.g., a particular scheduled time, the beginning, etc.) in a media program, etc.; etc.

[0049] A video frame (in the set of video frames) that is to be rendered at the same time as an audio data unit (in the set of audio data units) is to be rendered may share or comprise the same timestamp (in the set of input timestamps) with the audio data unit.

[0050] The content separation unit (202) is further configured to separate or de-multiplex the audiovisual data (210) into audio data component 216-1 and video data component 216-2; forward the audio data component (216-1) to the audio transmission unit (206) for further processing; forward the video data component (216-2) to the video transmission unit (208) for further processing; etc. In some embodiments, the content separation unit (202) includes the same set of input timestamps extracted from the audiovisual data (210) with each of the audio data component (216-1) and the video data component (216-2).

[0051] In some embodiments, the content synchronization unit (204) comprises software, hardware, a combination of software and hardware, etc., configured to synchronize (e.g., a portion of, all, etc.) clocks (used for content rendering) of the one or more audio rendering devices and the one or more video rendering devices to a common clock (used for content rendering). The common clock can be based on an external clock external to the media device (102), an internal clock of the media device (102), a clock derived from one or more clocks of the media device (102), the audio rendering devices, or video rendering devices, etc.

[0052] In some example implementations, some or all of the media device (102), the audio rendering devices, or the video rendering devices, may be synchronized to an external clock sources provided or supported by a satellite based navigational system (e.g., the BeiDou navigational system, the GPS navigational system, etc.). In some other example implementations, some or all of the media device, the audio rendering devices, or the video rendering devices, may be synchronized to a terrestrial radio clock.

[0053] Additionally, optionally, or alternatively, some or all of the media device (102), the audio rendering devices, or the video rendering devices, may be synchronized to a clock (used for content rendering) derived based on one or more internal clocks of one of the media device (102), the audio rendering devices, or video rendering devices. An algorithm based on one or more of a network time protocol, etc., can be implemented by some or all of the media device (102), the audio rendering devices, or video rendering devices, for the purpose of synchronizing the clocks (used for content rendering) of the media device (102) and/or the content rendering systems.

[0054] In some embodiments, the media device (102), or the content synchronization unit (204) therein, can be configured to communicate over one (e.g., 212-1 of FIG. 3, etc.) of one or more data paths 212 with each audio rendering device (e.g., 104, etc.) of one or more audio rendering devices to determine audio rendering characteristics pertaining to that audio rendering device (104); communicate with each video rendering device (e.g., 106, etc.) of one or more video rendering devices to determine video rendering characteristics pertaining to that video rendering device (106); etc.

[0055] Examples of rendering characteristics pertaining to a content rendering device (either audio rendering device or video rendering device) as described herein may include, but are not limited to only, any of: (time- varying or time-constant) transmission rates (e.g., supported bit rates, etc.) between the media device (102) and the content rendering device, (time- varying or time-constant) round-trip times between the media device (102) and the content rendering device, (time- varying or time-constant) transmission delays between the media device (102) and the content rendering device, content processing delays for rendering operations performed internally by the content rendering device, other delays or latencies associated with the underlying network, devices, components, etc.

[0056] To obtain some or all of the rendering characteristics related to a content rendering device (either audio rendering device or video rendering device), one or both of the media device (102) and the content rendering device may perform one or more specific operations.

[0057] In an example, one or both of the media device (102) and a content rendering device (e.g., 104, 106, etc.) can run one or more bandwidth tests (e.g., by transmitting test data, by transmitting test files, test packets, etc.) on an end-to-end network path between the media device (102) and the content rendering device (e.g., 104, 106, etc.) to determine or estimate a transmission rate from the media device (102) to the content rendering device (e.g., 104, 106, etc.), continuously, from time to time, periodically, on-demand, over a plurality of time points, through polling, by emitting reporting events, etc.

[0058] In another example, one or both of the media device (102) and a content rendering device (e.g., 104, 106, etc.) can run one or more delay or latency tests (e.g., by transmitting test data, by transmitting test files, test packets, etc.) on the end-to-end network path between the media device (102) and the content rendering device (e.g., 104 or 106, etc.) to determine or estimate delays or latencies from the media device (102) to the content rendering device (e.g., 104 or 106, etc.), continuously, from time to time, periodically, on-demand, over a plurality of time points, through polling, by emitting reporting events, etc.

[0059] In some embodiments, one or both of the media device (102) and a content rendering device (e.g., 104 or 106, etc.) can determine some or all of the rendering characteristics based on statistics collected by the media device (1 2) and the content rendering device (e.g., 104, 106, etc.). For example, real-time or non-real-time statistics collected for processing received audio or video content can be analyzed to determine transmission rates, transmission delays, content processing delays, other delays or latencies, etc., as functions of time. The transmission rates, transmission delays, content processing delays, other delays or latencies, etc., as determined based on analyzing statistics can be updated on a real-time or near-real-time basis, at the same time when the media device (102) and the content rendering device (e.g., 104, 106, etc.) are processing and rendering audio or video content.

[0060] Rendering characteristics such as transmission rates, transmission delays, processing delays, other delays or latencies, etc., are highly dependent on the types of underlying (e.g., local, non-local, etc.) networks and possibly time-varying conditions of the underlying networks. In some embodiments, the delivery/transmission of audio content (e.g., a portion of 214, etc.) from the media device (102) to an audio rendering device (e.g., 104 of FIG. 1A, etc.) is made over a different local network from one used for the delivery/transmission of video content (e.g., 222, etc.) from the media device (102) to a video rendering device (e.g., 106 of FIG. 1A, etc.).

[0061] For example, the audio data (214) can be delivered over one or more first networks (e.g., 112-1 of FIG. 1 A, etc.), which may be Bluetooth based, Wi-Fi based, etc., whereas the video data (222) can be delivered over a second network (e.g., 112-2 of FIG. 1A, etc.), which may be non-Bluetooth based, HDMI video link based, etc.

[0062] In some embodiments, the content synchronization unit (204) can be configured to determine individual rendering characteristics related to individual underlying networks used to transport audio data and video data to individual content rendering devices (e.g., the audio rendering device 104, the video rendering device 106, etc.).

[0063] From the individual rendering characteristics, the content synchronization unit (204) determines a combined audio and video rendering latency to be used for some or all of the content rendering devices that are to receive media content for rendering. In some embodiments, the content synchronization unit (204) can compute or estimate an individual content rendering latency (e.g., summing all of the transmission delay, the content processing delay, other delays or latencies, a safety margin, etc.) for each of all of the audio rendering devices and the video rendering devices, and select the largest of all of audio rendering latencies of the audio rendering devices and video rendering latencies of the video rendering devices as the combined audio and video rendering latency.

[0064] Based on the individual rendering characteristics, the content synchronization unit (204) further determines an individual transmission rate (e.g., a percentile such as 60% of the individual maximum transmission rate supported by the underlying network used to deliver content from the media device 102, etc.) for each of all the content rendering devices.

[0065] Individual audio transmission rates for individual audio rendering devices (e.g., 104, etc.), the combined audio and video rendering latency, etc., may be provided to the audio transmission unit (206) by the content synchronization unit (204) as one or more audio content transmission parameters (218-1) to be used in processing the audio data component (216-1). Similarly, individual video transmission rates for individual video rendering devices (e.g., 106, etc.), the combined audio and video rendering latency, etc., can be provided to the video transmission unit (208) by the content synchronization unit (204) as one or more video content transmission parameters (218-2) to be used in processing the video data component (216-2).

[0066] In some embodiments, the audio transmission unit (206) comprises software, hardware, a combination of software and hardware, etc., configured to receive the audio data component (216-1) from the content separation unit (202); receive the audio content transmission parameters (218-1) from the content synchronization unit (204); based at least in part on the audio content transmission parameters (218-1), generate output audio data 214; transmit the output audio content (214) to the one or more audio rendering devices.

[0067] The audio transmission unit (206) may be configured to generate one or more individual input audio data portions from the audio data component (216-1); each of the individual input audio data portions comprises audio content to be rendered by a specific audio rendering device of the one or more audio rendering devices. An individual input audio data portion as described herein may refer to specific audio content designated for one or more of: a left front audio channel, a right front audio channel, a center audio channel, a left surround audio channel, a right surround audio channel, etc. In some embodiments, the audio data component (216-1) comprises multi-channel audio data (e.g., for a 5.1 configuration, for a 7.1 configuration, etc.). An individual input audio data portion may comprise a sub-mix of the multi-channel audio data (e.g., a stereo mix for left front and right front audio speakers, etc.).

[0068] Based on the one or more individual input audio data portions, the audio transmission unit (206) generates the output audio data (214) as a collection of one or more individual audio data streams. Each individual audio data stream in the one or more audio data streams is generated from a corresponding individual input audio data portion in the one or more input audio data portions. An individual audio data stream as described herein refers to an output audio data portion that is delivered, transmitted, outputted, etc., at a transmission rate (e.g., a bit rate, a bandwidth, etc.) from the media device (102) to an audio rendering device for rendering. [0069] In some embodiments, the audio transmission unit (206) is configured to determine/select an individual audio rendering device to which an individual audio data stream in the output audio data (214) is to be transmitted; etc. An audio rendering device (e.g., 104 of FIG. 1 A, etc.) represents a content rendering device that renders an audio data stream transmitted at a particular transmission rate by a media device (e.g., 102 of FIG. 1 A, etc.), and may comprise one or more of: a left front audio speaker, a right front audio speaker, a center audio speaker, a left surround audio speaker, a right surround audio speaker, a Bluetooth earphone that can be used to render a stereo audio mix, a mobile phone with one or more of speakers or earphones to render audio content, a tablet computer with one or more of speakers or earphones to render audio content, a home entertainment system, a multi-channel car-based audio content rendering system, a multi-speaker audio rendering device, etc.

[0070] In some embodiments, the audio transmission unit (206) comprises one or more audio rate conversion sub-units 220. The audio transmission unit (206) can use the one or more audio rate conversion sub-units (220) to transcode or transrate the individual input audio data portions in the audio data component (216-1) into the individual audio data streams in the output audio content (214) to be transmitted at respective individual audio transmission rates to respective individual audio rendering devices.

[0071] For example, an audio rate conversion sub-unit as described herein can be used by the audio transmission unit (206) to transcode or transrate an individual input audio data portion (e.g., for a left front audio channel, etc.) in the audio data component (216-1) into an individual audio data stream in the output audio content (214) to be transmitted at an individual audio transmission rate to an individual audio rendering device (e.g., a content rendering device that renders audio content through a left front speaker, etc.); the individual audio transmission rate for the individual audio rendering device may be determined by the audio transmission unit (206) based on the audio content transmission parameters (218-1). Additionally, optionally, or alternatively, the audio rate conversion sub-unit may perform one or more operations such as filtering, quantization, noise reduction, mixing (e.g., down-mixing audio data for N channels to audio data in M channels, up-mixing audio data for M channels to audio data in N channels, where N is an integer greater than a positive integer M, etc.), encryption, compression/encoding, buffering, etc., to generate the individual audio data stream.

[0072] In some embodiments, the audio transmission unit (206) is configured to extract the set of input timestamps from the audio data component (216-1) as received from the content separation unit (202). This set of input timestamps is derived from the audiovisual data (210) and also embedded in the video data component (216-2) for synchronization purposes. The audio transmission unit (206) is further configured to obtain/extract/determine the combined audio and video rendering latency based on the audio content transmission parameters (218-1) as received from the content synchronization unit (204). Based at least in part on the set of input timestamps and the combined audio and video rendering latency, the audio transmission unit (206) generates a set of (output) timestamps. In some embodiments, each timestamp in the set of output timestamps is derived from a combination (e.g., sum, etc.) of the combined audio and video rendering latency and a corresponding timestamp in the set of input timestamps. In some embodiments, the audio transmission unit (206) includes or embeds (e.g., a copy of, etc.) the set of output timestamps in each individual audio data stream, in the individual audio data streams, for transmission to a respective audio rendering device. A timestamp in the set of output timestamps can be used by a recipient device to determine a specific time point with reference to the common clock at which a video frame or an audio data unit is to be rendered by the recipient device.

[0073] In some embodiments, the video transmission unit (208) comprises software, hardware, a combination of software and hardware, etc., configured to receive the video data component (216-2) from the content separation unit (202); receive the video content transmission parameters (218-2) from the content synchronization unit (204); based at least in part on the audio content transmission parameters (218-2), generate output video data 222; transmit the output audio content (222) to the one or more video rendering devices.

[0074] The video transmission unit (208) may be configured to generate one or more individual input video data portions from the video data component (216-2); each of the individual input video data portions comprises video content to be rendered by a specific video rendering device of the one or more video rendering devices.

[0075] Based on the one or more individual input video data portions, the video transmission unit (208) generates the output video data (222) as a collection of one or more individual video data streams. Each individual video data stream in the one or more video data streams is generated from a corresponding individual input video data portion in the one or more input video data portions. An individual audio data stream as described herein refers to an output video data portion that is delivered, transmitted, outputted, etc., at a transmission rate (e.g., a bit rate, a bandwidth, etc.) from the media device (102) to a video rendering device for rendering.

[0076] In some embodiments, the video transmission unit (208) is configured to determine/select an individual video rendering device to which an individual video data stream in the output audio data (214) is to be transmitted; etc. A video rendering device (e.g., 106 of FIG. 1 A, etc.) represents a content rendering device that renders a video data stream transmitted at a particular transmission rate by a media device (e.g., 102 of FIG. 1 A, etc.), and may comprise one or more of: a mobile phone, a television, a HDMI display, a mobile phone with an integrated display to render video content, a tablet computer with a display to render video content, a home entertainment system, a car-based video content rendering system, etc.

[0077] In some embodiments, the video transmission unit (208) comprises one or more video rate conversion sub-units 224. The video transmission unit (208) can use the one or more video rate conversion sub-units (224) to transcode or transrate the individual input video data portions in the input video content (216-2) into the individual video data streams in the output video content (222) to be transmitted at respective individual video transmission rates to respective individual video rendering devices.

[0078] For example, a video rate conversion sub-unit as described herein can be used by the video transmission unit (208) to transcode or transrate an individual input video data portion in the video data component (216-2) into an individual video data stream in the output video content (222) to be transmitted at an individual video transmission rate to an individual video rendering device (e.g., a content rendering device that renders video content through a display, etc.); the individual video transmission rate for the individual video rendering device maybe determined by the video transmission unit (208) based on the video content transmission parameters (218-2). Additionally, optionally, or alternatively, the video rate conversion sub-unit may perform one or more operations such as filtering, quantization, noise reduction, encryption, compression/encoding, buffering, etc., to generate the individual video data stream.

[0079] In some embodiments, the video transmission unit (208) is configured to extract the set of input timestamps from the video data component (216-2) as received from the content separation unit (202). This set of input timestamps is derived from the audiovisual data (210) and also embedded in the audio data component (216-1) for synchronization purposes. The video transmission unit (208) is further configured to obtain/extract/determine the combined audio and video rendering latency based on the audio content transmission parameters (218-2) as received from the content synchronization unit (204). Based at least in part on the set of input timestamps and the combined audio and video rendering latency, the video transmission unit (208) generates a set of (output) timestamps. In some embodiments, each timestamp in the set of output timestamps is derived from a combination (e.g., sum, etc.) of the combined audio and video rendering latency and a corresponding timestamp in the set of input timestamps. In some embodiments, the video transmission unit (208) includes or embeds (e.g., a copy of, etc.) the set of output timestamps in each individual video data stream, in the individual video data streams, for transmission to a respective video rendering device. A timestamp in the set of output timestamps can be used by a recipient device to determine a specific time point with reference to the common clock at which a video frame or an audio data unit is to be rendered by the recipient device.

4. AUDIO CONTENT RENDERING DEVICES

[0080] FIG. 3 illustrates an example audio rendering device (e.g., 104 of FIG. 1A, etc.) comprising an audio stream decoder 302, an audio synchronization module 304, an audio rendering module 306, etc.

[0081] In some embodiments, the audio synchronization module (304) comprises software, hardware, a combination of software and hardware, etc., configured to cooperate with a media device (e.g., 102 of FIG. 1 A or FIG. 2, etc.) to synchronize (e.g., all, etc.) a clock used by the audio rendering device (104) for rendering its received audio content with other clocks used for content rendering by other content rendering devices to which the media device (102) transmits respective audio content or video content for synchronized rendering in reference to a common clock (used for content rendering). As discussed, the common clock can be based on an external clock external to the media device (102), an internal clock of the media device (102), a clock derived from one or more clocks of the media device (102), the audio rendering devices, or the video rendering devices, etc.

[0082] For example, the audio rendering device (104), or the audio synchronization module (304) therein, may implement its part of a time synchronization algorithm based on one or more of a network time protocol, etc., used to synchronize the clocks (used for content rendering) of the media device (102) and/or the content rendering systems.

[0083] In some embodiments, the audio rendering device (104), or the audio synchronization module (304) therein, is configured to communicate over a data path 212-1 with the media device (102) or the content synchronization unit (204) therein, so that the media device (102) can determine audio rendering characteristics pertaining to the audio rendering device (104).

[0084] In an example, the audio rendering device (104), or the audio synchronization module (304) therein, can perform operations in conjunction with the media device (102) to run one or more bandwidth tests (e.g., by transmitting test data, by transmitting test files, test packets, etc.) on an end-to-end network path between the media device (102) and the audio rendering device (104) for the purpose of determining or estimating a transmission rate from the media device ( 102) to the audio rendering device (104), continuously, from time to time, periodically, on-demand, over a plurality of time points, through polling, by emitting reporting events, etc.

[0085] In an example, the audio rendering device (104), or the audio synchronization module (304) therein, can perform operations in conjunction with the media device (102) to run one or more delay or latency tests (e.g., by transmitting test data, by transmitting test files, test packets, etc.) on the end-to-end network path between the media device (102) and the audio rendering device (104) for the purpose of determining or estimating delays or latencies from the media device (102) to the audio rendering device (104), continuously, from time to time, periodically, on-demand, over a plurality of time points, through polling, by emitting reporting events, etc.

[0086] In some embodiments, the audio rendering device (104) can be configured to collect statistics related to audio rendering operations performed by the audio rendering device (104). For example, real-time or non-real-time statistics can be collected for processing received audio content and analyzed, by one or both of the media device (102) and the audio rendering device (104), to determine transmission rates, transmission delays, content processing delays, other delays or latencies, etc., as functions of time. The transmission rates, transmission delays, content processing delays, other delays or latencies, etc., determined based on analyzing statistics can be updated, by one or both of the media device (102) and the audio rendering device (104), on a real-time or near-real-time basis, at the same time when the media device (102) and the audio rendering device (104) are processing and rendering audio content with the audio rendering device (104).

[0087] In some embodiments, the audio stream decoder (302) comprises software, hardware, a combination of software and hardware, etc., configured to receive, from the media device (102), an audio data stream 214-1 comprising specific audio content to be rendered by the audio rendering device (104); decode a set of audio data units (e.g., audio samples, audio data blocks, audio chunks, etc.) from the audio data stream (214-1); extract a set of corresponding timestamps from the audio data stream (214-1), which are included in the audio data stream (214-1) by the media device (102) for the purpose of synchronizing the playing/rendering of the audio data stream (214-1) by the audio rendering device (104) with the playing/rendering of other content data streams (e.g., other audio data streams, video data streams, etc.) by other content rendering devices (e.g., other audio rendering devices, video rendering devices, etc.); etc. The audio stream decoder (302) is further configured to send/forward the set of decoded audio data units and the set of corresponding timestamps to the audio rendering module (306) for rendering.

[0088] In some embodiments, the audio rendering module (306) comprises software, hardware, a combination of software and hardware, etc., configured to queue or store up to a certain number of decoded audio data units in memory (e.g., an audio jitter buffer, etc.) at any given time before rendering these decoded audio data units.

[0089] The audio rendering device (104) may be configured to generate digital drive values based on audio data in an audio data unit that is to be rendered at a specific time point (e.g., with reference to the common clock, etc.) specified by an individual timestamp in the set of timestamps, and drives one or more audio transducers 308 (e.g., earphones, audio speakers, etc.) to generate, at the specific time point, sound waves that represent an audio content portion carried by the audio data unit.

5. VIDEO CONTENT RENDERING DEVICES

[0090J FIG. 4 illustrates an example video rendering device (e.g., 106 of FIG. 1A, etc.) comprising a video stream decoder 402, a video synchronization module 404, a video rendering module 406, etc.

[0091] In some embodiments, the video synchronization module (404) comprises software, hardware, a combination of software and hardware, etc., configured to cooperate with a media device (e.g., 102 of FIG. 1A or FIG. 2, etc.) to synchronize (e.g., all, etc.) a clock used by the video rendering device (106) for rendering its received video content with other clocks for content rendering by other content rendering devices to which the media device (102) transmits respective audio content or video content for synchronized rendering in reference to a common clock (used for content rendering). As discussed, the common clock can be based on an external clock external to the media device (102), an internal clock of the media device (102), a clock derived from one or more clocks of the media device (102), the audio rendering devices, or the video rendering devices, etc.

[0092] For example, the video rendering device (106) may implement its part of a time synchronization algorithm based on one or more of a network time protocol, etc., used to synchronize the clocks (used for content rendering) of the media device (102) and/or the content rendering systems.

[0093] In some embodiments, the video rendering device (106), or the video synchronization module (404) therein, is configured to communicate over a data path 212-2 with the media device (102) or the content synchronization unit (204) therein, so that the media device (102) can determine video rendering characteristics pertaining to the video rendering device (106).

[0094] In an example, the video rendering device (106), or the video synchronization module (404) therein, can perform operations in conjunction with the media device (102) to run one or more bandwidth tests (e.g., by transmitting test data, by transmitting test files, test packets, etc.) on an end-to-end network path between the media device (102) and the video rendering device (106) for the purpose of determining or estimating a transmission rate from the media device ( 102) to the video rendering device (106), continuously, from time to time, periodically, on-demand, over a plurality of time points, through polling, by emitting reporting events, etc.

[0095] In an example, the video rendering device (106), or the video synchronization module (404) therein, can perform operations in conjunction with the media device (102) to run one or more delay or latency tests (e.g., by transmitting test data, by transmitting test files, test packets, etc.) on the end-to-end network path between the media device (102) and the video rendering device (106) for the purpose of determining or estimating delays or latencies from the media device (102) to the video rendering device (106), continuously, from time to time, periodically, on-demand, over a plurality of time points, through polling, by emitting reporting events, etc.

[0096] In some embodiments, the video rendering device (106) can be configured to collect statistics related to video rendering operations performed by the video rendering device (106). For example, real-time or non-real-time statistics can be collected for processing received video content and analyzed, by one or both of the media device (102) and the video rendering device (106), to determine transmission rates, transmission delays, content processing delays, other delays or latencies, etc., as functions of time. The transmission rates, transmission delays, content processing delays, other delays or latencies, etc., determined based on analyzing statistics can be updated, by one or both of the media device (102) and the video rendering device (106), on a real-time or near-real-time basis, at the same time when the media device (102) and the video rendering device (106) are processing and rendering video content with the video rendering device (106).

[0097] In some embodiments, the video stream decoder (402) comprises software, hardware, a combination of software and hardware, etc., configured to receive, from the media device (102), a video data stream 214-2 comprising specific video content to be rendered by the video rendering device (106); decode a set of video frames (or video chunks) from the video data stream (214-2); extract a set of corresponding timestamps from the video data stream (214-2), which are included in the video data stream (214-2) by the media device (102) for the purpose of synchronizing the playing rendering of the video data stream (214-2) by the video rendering device (106) with the playing/rendering of other content data streams (e.g., audio data streams, other video data streams, etc.) by other content rendering devices (e.g., audio rendering devices, other video rendering devices, etc.); etc. The video stream decoder (402) is further configured to send/forward the set of decoded video frames and the set of corresponding timestamps to the video rendering module (406) for rendering.

[0098] In some embodiments, the video rendering module (406) comprises software, hardware, a combination of software and hardware, etc., configured to queue or store up to a certain number of decoded video frames in memory (e.g., a video jitter buffer, etc.) at any given time before rendering these decoded video frames.

[0099] The video rendering device (106) may be configured to generate digital drive values based on video data in a video frame that is to be rendered at a specific time point (e.g., with reference to the common clock, etc.) specified by an individual timestamp in the set of timestamps, and drives a display 408 (e.g., a liquid crystal display, a plasma display, etc.) to generate, at the specific time point, an image that represents a video content portion in the video frame.

6. EXAMPLE PROCESS FLOWS

[0100] FIG. 5A illustrates an example process flow according to an example embodiment of the present invention. In some example embodiments, one or more computing devices or components may perform this process flow. In block 502, a media device (e.g., 102 of FIG. 1A, FIG. IB or FIG. 2, etc.) determines, for an audio rendering device, an audio transmission rate and an audio rendering latency. [0101] In block 504, the media device determines, for a video rendering device, a video transmission rate and a video rendering latency.

[0102] In block 506, the media device determines, based at least in part on the audio rendering latency and the video rendering latency, a combined audio and video rendering latency for both the audio rendering device and the video rendering device.

[0103] In block 508, the media device transmits a portion of audio content of audiovisual data to the audio rendering device at the audio transmission rate. The portion of the audio content comprises a set of timestamps for rendering audio data in the portion of the audio content; the set of timestamps is determined based as least in part on the combined audio and video rendering latency.

[0104] In block 510, the media device transmits a portion of video content of the audiovisual data to the video rendering device at the video transmission rate. The portion of the video content comprises the set of timestamps for rendering video data in the portion of the video content.

[0105] In an embodiment, the media device is further configured to perform: separating the audiovisual data into an audio data component as the audio content and a video data component as the video content; encoding the audio data component into one or more audio data streams; encoding the video data component into one or more video data streams; etc. An audio data stream in the one or more audio data streams comprises the portion of the audio content, and is streamed to the audio rendering device at the audio transmission rate. A video data stream in the one or more video data streams comprises the portion of the video content, and is streamed to the video rendering device at the video transmission rate.

[0106] In some embodiments, the portion of the audio content and the portion of the video content may be transmitted over the same network such as a local network, a non-local network, etc. In some embodiments, the portion of the audio content and the portion of the video content may be transmitted over two different networks such as two different local networks, one local network and one non-local network, two non-local networks, etc.

[0107] In an embodiment, the portion of the audio content is transmitted from a media device to the audio rendering device over a Bluetooth connection, whereas the portion of the video content is transmitted from the media device to the video rendering device over a non-Bluetooth connection.

[0108] In an embodiment, the audio rendering device and the video rendering device are synchronized to a common clock for content rendering; the set of timestamps comprises timestamps to be interpreted in reference to the common clock.

[0109] In an embodiment, the audio rendering device is among a plurality of audio rendering devices for each of which an individual audio transmission rate and an individual audio rendering latency are determined; the combined audio and video rendering latency is determined for the plurality of audio rendering devices and the video rendering device based at least in part on individual audio rendering latencies of the plurality of audio rendering devices and the video rendering latency; the media device is further configured to transmit a second portion of the audio content of the audiovisual data to a second audio rendering device in the plurality of audio rendering devices at a second audio transmission rate, wherein the second portion of the audio content comprises the set of timestamps.

[0110] In an embodiment, the video rendering device is among a plurality of video rendering devices for each of which an individual video transmission rate and an individual video rendering latency are determined; the combined audio and video rendering latency is determined for the audio rendering device and the plurality of video rendering devices based at least in part on individual video rendering latencies of the audio rendering device and the plurality of video rendering devices; the media device is further configured to transmit a second portion of the video content of the audiovisual data to a second video rendering device in the plurality of video rendering devices at a second video transmission rate, wherein the second portion of the video content comprises the set of timestamps.

[0111] In an embodiment, the audiovisual data comprises one of movies, media programs, television programs, etc.

[0112] In an embodiment, the audiovisual data is received from one or more of media files stored in locally or remotely accessible storage media, or media data streams from one or more media streaming servers.

[0113] In an embodiment, the media device is further configured to perform: determining a set of input timestamps from the audiovisual data; generating, based on the set of input timestamps and the combined audio and video rendering latency, the set of timestamps.

[0114] In an embodiment, the audio rendering latency for the audio rendering device comprises one or more of transmission delays, content processing delays, network-specific latencies, device-specific latencies, or component-specific latencies, as related to the audio rendering device.

[0115] In an embodiment, the audio transmission rate, the video transmission rate, and the combined audio and video rendering latency are determined based on contemporaneous rendering characteristics for a first time point; the media device is further configured to perform: determining, for the audio rendering device, a subsequent audio transmission rate and a subsequent audio rendering latency for a second time point subsequent to the first time; determining, for the video rendering device, a subsequent video transmission rate and a subsequent video rendering latency for the second time point; determining, based at least in part on the subsequent audio rendering latency and the subsequent video rendering latency, a subsequent combined audio and video rendering latency for both the audio rendering device and the video rendering device; transmitting a subsequent portion of the audio content of the audiovisual data to the audio rendering device at the subsequent audio transmission rate; transmitting a subsequent portion of the video content of the audiovisual data to the video rendering device at the subsequent video transmission rate. The subsequent portion of the audio content comprises a subsequent set of timestamps for rendering audio data in the subsequent portion of the audio content. The subsequent set of timestamps is determined based as least in part on the subsequent combined audio and video rendering latency. The subsequent portion of the video content comprises the subsequent set of timestamps for rendering video data in the subsequent portion of the video content. [0116] In an embodiment, the audio transmission rate, the video transmission rate, and the combined audio and video rendering latency are determined, while the audio content of the audiovisual data is being streamed to one or more audio rendering devices that include the audio rendering device and the video content of the audiovisual data is being streamed to one or more video rendering devices that include the video rendering device.

[0117] In an embodiment, the method is performed by a media device; the portion of the audio content is transmitted to the audio rendering device in a single media data stream between the media device and the audio rendering device.

[0118] In an embodiment, the method is performed by a media device; the portion of the video content is transmitted to the video rendering device in a single media data stream between the media device and the video rendering device.

[0119] In an embodiment, the method is performed by a media device; the portion of the video content is transmitted to the video rendering device in a first media data sub-stream of a media data stream between the media device and the video rendering device; and a second media data sub-stream of the media data stream comprises a second portion of the audio content streamed from the media device to the video rendering device.

[0120] FIG. 5B illustrates an example process flow according to an example embodiment of the present invention. In some example embodiments, one or more computing devices or components may perform this process flow. In block 522, a content rendering device (e.g., 104 of FIG. 1A or FIG. 3, 104-1 or 104-2 of FIG. IB, 106 of FIG. 1A or FIG. 4, 106-1 of FIG. IB, etc.) communicates with a media device, by a content rendering device, to determine a content transmission rate and an individual content rendering latency.

[0121] In block 524, the content rendering device receives a portion of media content of audiovisual data at the content transmission rate. The portion of the media content comprises a set of timestamps for rendering media data in the portion of the media content. The set of timestamps is determined based at least in part on a combined audio and video rendering latency determined by the media device.

[0122] In block 526, the content rendering device decodes one or more content data units from the portion of the media content.

[0123] In block 528, the content rendering device renders the one or more content data units at one or more time points as determined based on one or more time stamps in the set of timestamps.

[0124] In an embodiment, the content rendering device represents an audio rendering device, and wherein the media content comprises audio-only content. In an embodiment, the content rendering device represents a video rendering device, and wherein the media content comprises video-only content. In an embodiment, the content rendering device represents a combined audio and video rendering device, and wherein the media content comprises audio and video content.

[012S] In an embodiment, the content rendering device communicates with the media device over an end-to-end network path over one or more of Wi-Fi networks, Bluetooth networks, infrared links, CDMA networks, GSM networks, HDMI video links, non-HDMI video links, radio frequency (RF) networks, optical links, etc. [0126] In an embodiment, the content rendering device is an audio rendering device in a plurality of audio rendering devices, wherein the plurality of audio rendering devices together forms a multi-channel audio rendering system, and wherein the media content as received by the audio rendering device is a sub-mix of a mix of multi-channel audio data.

[0127] In an embodiment, the one or more time points are determined in reference to a common clock (used for content rendering) based on the one or more time stamps in the set of timestamps; the content rendering device is further configured to communicate with the media device, by the content rendering device, to determine the common clock to be used by the content rendering device for synchronized content rendering with one or more other content rendering devices. The one or more other content rendering devices use the common clock for synchronized content rendering.

[0128] In various example embodiments, a system, an apparatus, or one or more other computing devices performs any or a part of the foregoing methods as described.

[0129] In an embodiment, a non-transitory computer readable storage medium, storing software instructions, which when executed by one or more processors cause performance of any of the foregoing methods.

[0130] In an embodiment, a computing device comprising one or more processors and one or more storage media storing a set of instructions which, when executed by the one or more processors, cause performance of any of the foregoing methods. Note that, although separate embodiments are discussed herein, any combination of embodiments and/or partial embodiments discussed herein may be combined to form further embodiments.

7. IMPLEMENTATION MECHANISMS - HARDWARE OVERVIEW

[0131] According to one embodiment, the techniques described herein are implemented by one or more special-purpose computing devices. The special-purpose computing devices may be hard-wired to perform the techniques, or may include digital electronic devices such as one or more application-specific integrated circuits (ASICs) or field programmable gate arrays (FPGAs) that are persistently programmed to perform the techniques, or may include one or more general purpose hardware processors programmed to perform the techniques pursuant to program instructions in firmware, memory, other storage, or a combination. Such special-purpose computing devices may also combine custom hard-wired logic, ASICs, or FPGAs with custom programming to accomplish the techniques. The special-purpose computing devices may be desktop computer systems, portable computer systems, handheld devices, networking devices or any other device that incorporates hard-wired and/or program logic to implement the techniques.

[0132] For example, FIG. 6 is a block diagram that illustrates a computer system 600 upon which an example embodiment of the invention may be implemented. Computer system 600 includes a bus 602 or other communication mechanism for communicating information, and a hardware processor 604 coupled with bus 602 for processing information. Hardware processor 604 may be, for example, a general purpose microprocessor.

[0133] Computer system 600 also includes a main memory 606, such as a random access memory (RAM) or other dynamic storage device, coupled to bus 602 for storing information and instructions to be executed by processor 604. Main memory 606 also may be used for storing temporary variables or other intermediate information during execution of instructions to be executed by processor 604. Such instructions, when stored in non-transitory storage media accessible to processor 604, render computer system 600 into a special-purpose machine that is customized to perform the operations specified in the instructions.

[0134] Computer system 600 further includes a read only memory (ROM) 608 or other static storage device coupled to bus 602 for storing static information and instructions for processor 604. A storage device 610, such as a magnetic disk or optical disk, is provided and coupled to bus 602 for storing information and instructions.

[0135] Computer system 600 may be coupled via bus 602 to a display 612, such as a liquid crystal display, for displaying information to a computer user. An input device 614, including alphanumeric and other keys, is coupled to bus 602 for communicating information and command selections to processor 604. Another type of user input device is cursor control 616, such as a mouse, a trackball, or cursor direction keys for communicating direction information and command selections to processor 604 and for controlling cursor movement on display 612. This input device typically has two degrees of freedom in two axes, a first axis (e.g., x) and a second axis (e.g., y), that allows the device to specify positions in a plane.

[0136] Computer system 600 may implement the techniques described herein using customized hard-wired logic, one or more ASICs or FPGAs, firmware and/or program logic which in combination with the computer system causes or programs computer system 600 to be a special-purpose machine. According to one embodiment, the techniques herein are performed by computer system 600 in response to processor 604 executing one or more sequences of one or more instructions contained in main memory 606. Such instructions may be read into main memory 606 from another storage medium, such as storage device 610. Execution of the sequences of instructions contained in main memory 606 causes processor 604 to perform the process steps described herein. In alternative embodiments, hard-wired circuitry may be used in place of or in combination with software instructions.

[0137] The term "storage media" as used herein refers to any non-transitory media that store data and/or instructions that cause a machine to operation in a specific fashion. Such storage media may comprise non-volatile media and/or volatile media. Non-volatile media includes, for example, optical or magnetic disks, such as storage device 610. Volatile media includes dynamic memory, such as main memory 606. Common forms of storage media include, for example, a floppy disk, a flexible disk, hard disk, solid state drive, magnetic tape, or any other magnetic data storage medium, a CD-ROM, any other optical data storage medium, any physical medium with patterns of holes, a RAM, a PROM, and EPROM, a FLASH-EPROM, NVRAM, any other memory chip or cartridge.

[0138] Storage media is distinct from but may be used in conjunction with transmission media. Transmission media participates in transferring information between storage media. For example, transmission media includes coaxial cables, copper wire and fiber optics, including the wires that comprise bus 602. Transmission media can also take the form of acoustic or light waves, such as those generated during radio-wave and infra-red data communications.

[0139] Various forms of media may be involved in carrying one or more sequences of one or more instructions to processor 604 for execution. For example, the instructions may initially be carried on a magnetic disk or solid state drive of a remote computer. The remote computer can load the instructions into its dynamic memory and send the instructions over a telephone line using a modem. A modem local to computer system 600 can receive the data on the telephone line and use an infra-red transmitter to convert the data to an infra-red signal. An infra-red detector can receive the data carried in the infra-red signal and appropriate circuitry can place the data on bus 602. Bus 602 carries the data to main memory 606, from which processor 604 retrieves and executes the instructions. The instructions received by main memory 606 may optionally be stored on storage device 610 either before or after execution by processor 604.

[0140] Computer system 600 also includes a communication interface 618 coupled to bus 602. Communication interface 618 provides a two-way data communication coupling to a network link 620 that is connected to a local network 622. For example, communication interface 618 may be an integrated services digital network (ISDN) card, cable modem, satellite modem, or a modem to provide a data communication connection to a corresponding type of telephone line. As another example, communication interface 618 may be a local area network (LAN) card to provide a data communication connection to a compatible LAN. Wireless links may also be implemented. In any such implementation, communication interface 618 sends and receives electrical, electromagnetic or optical signals that carry digital data streams representing various types of information.

[0141] Network link 620 typically provides data communication through one or more networks to other data devices. For example, network link 620 may provide a connection through local network 622 to a host computer 624 or to data equipment operated by an Internet Service Provider (ISP) 626. ISP 626 in turn provides data communication services through the world wide packet data communication network now commonly referred to as the "Internet" 628. Local network 622 and Internet 628 both use electrical, electromagnetic or optical signals that carry digital data streams. The signals through the various networks and the signals on network link 620 and through communication interface 618, which carry the digital data to and from computer system 600, are example forms of transmission media.

[0142] Computer system 600 can send messages and receive data, including program code, through the network(s), network link 620 and communication interface 618. In the Internet example, a server 630 might transmit a requested code for an application program through Internet 628, ISP 626, local network 622 and communication interface 618.

[0143] The received code may be executed by processor 604 as it is received, and/or stored in storage device 610, or other non- volatile storage for later execution.

8. EQUIVALENTS, EXTENSIONS, ALTERNATIVES AND MISCELLANEOUS

[0144] In the foregoing specification, example embodiments of the invention have been described with reference to numerous specific details that may vary from implementation to implementation. Thus, the sole and exclusive indicator of what is the invention, and is intended by the applicants to be the invention, is the set of claims that issue from this application, in the specific form in which such claims issue, including any subsequent correction. Any definitions expressly set forth herein for terms contained in such claims shall govern the meaning of such terms as used in the claims. Hence, no limitation, element, property, feature, advantage or attribute that is not expressly recited in a claim should limit the scope of such claim in any way. The specification and drawings are, accordingly, to be regarded in an illustrative rather than a restrictive sense.

Claims

CLAIMS What is claimed is:

1. A method for synchronizing audio rendering and video rendering, comprising:

determining, for an audio rendering device, an audio transmission rate and an audio rendering latency;

determining, for a video rendering device, a video transmission rate and a video rendering latency;

determining, based at least in part on the audio rendering latency and the video rendering latency, a combined audio and video rendering latency for both the audio rendering device and the video rendering device;

transmitting a portion of audio content of audiovisual data to the audio rendering device at the audio transmission rate, wherein the portion of the audio content comprises a set of timestamps for rendering audio data in the portion of the audio content, wherein the set of timestamps is determined based as least in part on the combined audio and video rendering latency;

transmitting a portion of video content of the audiovisual data to the video rendering device at the video transmission rate, wherein the portion of the video content comprises the set of timestamps for rendering video data in the portion of the video content.

2. The method of Claim 1 , further comprising:

separating the audiovisual data into an audio data component as the audio content and a video data component as the video content;

encoding the audio data component into one or more audio data streams, wherein an audio data stream in the one or more audio data streams comprises the portion of the audio content, and wherein the audio data stream is streamed to the audio rendering device at the audio transmission rate;

encoding the video data component into one or more video data streams, wherein a video data stream in the one or more video data streams comprises the portion of the video content, and wherein the video data stream is streamed to the video rendering device at the video transmission rate.

3. The method of Claim 1, wherein the portion of the audio content is transmitted from a media device to the audio rendering device over a Bluetooth connection, and wherein the portion of the video content is transmitted from the media device to the video rendering device over a non-Bluetooth connection.

4. The method of Claim 1, wherein the audio rendering device and the video rendering device are synchronized to a common clock, and wherein the set of timestamps comprises timestamps to be interpreted in reference to the common clock.

5. The method of Claim 1, wherein the audio rendering device is among a plurality of audio rendering devices for each of which an individual audio transmission rate and an individual audio rendering latency are determined; wherein the combined audio and video rendering latency is determined for the plurality of audio rendering devices and the video rendering device based at least in part on individual audio rendering latencies of the plurality of audio rendering devices and the video rendering latency; further comprising transmitting a second portion of the audio content of the audiovisual data to a second audio rendering device in the plurality of audio rendering devices at a second audio transmission rate, wherein the second portion of the audio content comprises the set of timestamps.

6. The method of Claim 1 , wherein the video rendering device is among a plurality of video rendering devices for each of which an individual video transmission rate and an individual video rendering latency are determined; wherein the combined audio and video rendering latency is determined for the audio rendering device and the plurality of video rendering devices based at least in part on individual video rendering latencies of the audio rendering device and the plurality of video rendering devices; further comprising transmitting a second portion of the video content of the audiovisual data to a second video rendering device in the plurality of video rendering devices at a second video transmission rate, wherein the second portion of the video content compnses the set of timestamps.

7. The method of Claim 1, wherein the audiovisual data comprises one of movies, media programs, or television programs.

8. The method of Claim 1, wherein the audiovisual data is received from one or more of media files stored in locally or remotely accessible storage media, or media data streams from one or more media streaming servers.

9. The method of Claim 1 , further comprising:

determining a set of input timestamps from the audiovisual data;

generating, based on the set of input timestamps and the combined audio and video rendering latency, the set of timestamps.

10. The method of Claim 1, wherein the audio rendering latency for the audio rendering device comprises one or more of transmission delays, content processing delays, network-specific latencies, device-specific latencies, or component-specific latencies, as related to the audio rendering device.

11. The method of Claim 1 , wherein the audio transmission rate, the video transmission rate, and the combined audio and video rendering latency are determined based on contemporaneous rendering characteristics for a first time point, further comprising: determining, for the audio rendering device, a subsequent audio transmission rate and a subsequent audio rendering latency for a second time point subsequent to the first time;

determining, for the video rendering device, a subsequent video transmission rate and a subsequent video rendering latency for the second time point;

determining, based at least in part on the subsequent audio rendering latency and the subsequent video rendering latency, a subsequent combined audio and video rendering latency for both the audio rendering device and the video rendering device; transmitting a subsequent portion of the audio content of the audiovisual data to the audio rendering device at the subsequent audio transmission rate, wherein the subsequent portion of the audio content comprises a subsequent set of timestamps for rendering audio data in the subsequent portion of the audio content, wherein the subsequent set of timestamps is determined based as least in part on the subsequent combined audio and video rendering latency;

transmitting a subsequent portion of the video content of the audiovisual data to the video rendering device at the subsequent video transmission rate, wherein the subsequent portion of the video content comprises the subsequent set of timestamps for rendering video data in the subsequent portion of the video content.

The method of Claim 1 , wherein the audio transmission rate, the video transmission rate, and the combined audio and video rendering latency are determined, while the audio content of the audiovisual data is being streamed to one or more audio rendering devices that include the audio rendering device and the video content of the audiovisual data is being streamed to one or more video rendering devices that include the video rendering device.

The method of Claim 1, wherein the method is performed by a media device; and wherein the portion of the audio content is transmitted to the audio rendering device in a single media data stream between the media device and the audio rendering device.

The method of Claim 1, wherein the method is performed by a media device; and wherein the portion of the video content is transmitted to the video rendering device in a single media data stream between the media device and the video rendering device.

The method of Claim 1, wherein the method is performed by a media device; wherein the portion of the video content is transmitted to the video rendering device in a first media data sub-stream of a media data stream between the media device and the video rendering device; and wherein a second media data sub-stream of the media data stream comprises a second portion of the audio content streamed from the media device to the video rendering device.

A method for synchronizing audio rendering and video rendering, comprising:

communicating with a media device, by a content rendering device, to determine a content transmission rate and an individual content rendering latency;

receiving a portion of media content of audiovisual data at the content transmission rate, wherein the portion of the media content comprises a set of timestamps for rendering media data in the portion of the media content, wherein the set of timestamps is determined based as least in part on a combined audio and video rendering latency determined by the media device;

decoding one or more content data units from the portion of the media content;

rendering the one or more content data units at one or more time points as determined based on one or more time stamps in the set of timestamps.

The method of Claim 16, wherein the content rendering device represents an audio rendering device, and wherein the media content comprises audio-only content.

18. The method of Claim 16, wherein the content rendering device represents a video rendering device, and wherein the media content comprises video-only content.

19. The method of Claim 16, wherein the content rendering device represents a combined audio and video rendering device, and wherein the media content comprises audio and video content.

20. The method of Claim 16, wherein the content rendering device communicates with the media device over an end-to-end network path over one or more of Wi-Fi networks, Bluetooth networks, infrared links, CDMA networks, GSM networks, HDMI video links, non-HDMI video links, radio frequency (RF) networks, or optical links.

21. The method of Claim 16, wherein the content rendering device is an audio rendering device in a plurality of audio rendering devices, wherein the plurality of audio rendering devices together forms a multi-channel audio rendering system, and wherein the media content as received by the audio rendering device is a sub-mix of a mix of multi-channel audio data.

22. The method of Claim 16, wherein the one or more time points are determined in reference to a common clock based on the one or more time stamps in the set of timestamps; further comprising:

communicating with the media device, by the content rendering device, to determine the common dock to he used by the content rendering device for synchronized content rendering with one or more other content rendering devices, wherein the one or more other content rendering devices use the common clock for synchronized content rendering.

23. A system performing any of the methods as recited in Claims 1-22.

24. A non-transitory computer readable storage medium, storing software instructions, which when executed by one or more processors cause performance of the method recited in any of Claims 1-22.

25. A computing device comprising one or more processors and one or more storage media, storing a set of instructions, which when executed by one or more processors cause performance of the method recited in any of Claims 1-22.