RU2798414C2 - Audio device and audio processing method - Google Patents

Abstract

FIELD: computer technology for processing audio data.

SUBSTANCE: invention is aimed to increase the accuracy of matching the perception of audio and visual scenes. It is achieved by choosing between a transformation for reproducing at least the first part of the first audio element for a set of acoustic systems and for headphones in response to the first index of the transformation property for audio reproduction, and the index of the transformation property for audio reproduction indicates whether the first part of the first audio element represents an audio source with a spatial property whose orientation is head-bound and intended to be a fixed position relative to the user's head, or representing an audio source with a spatial property whose orientation is not head-bound and intended to be a fixed position in the environment.

EFFECT: increase the accuracy of matching the perception of audio and visual scenes.

14 cl, 2 dwg

Description

FIELD OF TECHNOLOGY

The invention relates to an audio device and method for processing audio, and in particular, but not exclusively, to the use of such a device and method to support an augmented/virtual reality application.

BACKGROUND OF THE INVENTION

Thanks to the continuous development and introduction of new services and ways of using and consuming audiovisual content, there has been a significant increase in the diversity and range of perception of such content in recent years. In particular, many spatial and interactive experiences, applications and approaches are being developed to provide users with a more engaging and immersive experience.

Examples of such applications are virtual reality (Virtual Reality, VR), augmented reality (Augmented Reality, AR) and mixed reality (Mixed Reality, MR) applications, which are rapidly becoming mainstream, with a number of solutions targeted at the consumer market. In addition, a number of standards are being developed by many standards bodies. As part of this standardization activity, standards are being actively developed for various aspects of VR/AR/MR systems, including, for example, streaming, broadcast, conversion for playback, etc.

VR applications typically provide user experiences corresponding to the user being in a different world/environment/scene, while AR applications (including Mixed Reality (MR)) typically provide user experiences corresponding to the user being in the current environment , but with the addition of additional information or virtual objects or information. Thus, VR applications typically provide a fully immersive, synthetically generated world/scene, while AR applications typically provide a partially synthetic world/scene overlaid on the real scene in which the user is physically present. However, these frequency terms are used interchangeably and have a high degree of overlap. In the following, the term virtual reality/VR will be used to refer to both virtual reality and augmented reality.

As an example, a rapidly growing service is to provide images and audio in such a way that the user is able to actively and dynamically interact with the system to change the rendering settings for playback so that the images and audio will adapt to the movement and changes in the position and orientation of the user. A very attractive feature of many applications is the ability to change the actual viewing position and viewing direction of the viewer so that, for example, the viewer can move and "look around" in the scene being presented.

Such a function may in particular provide the user with a virtual reality experience. Due to this, the user can (relatively) freely move around in the virtual environment and dynamically change his position and the direction in which he looks. Typically, such virtual reality applications are based on a 3D scene model, with the model being dynamically evaluated to provide the particular requested look. This approach is well known, for example, from gaming applications for computers and consoles, for example, from the category of "shooters", or first-person shooting games.

It is also desirable, particularly for virtual reality applications, that the displayed image be a three-dimensional image. Indeed, in order to optimize viewer immersion, it is generally preferred that the user experience the presented scene as a three-dimensional scene. After all, the virtual reality experience should preferably allow the user to choose their own position, camera viewpoint, and point in time relative to the virtual world.

Generally, virtual reality applications are inherently limited in that they are based on a predetermined scene model and usually an artificial model of the virtual world. In some applications, a virtual reality experience can be provided based on a capture of the real world. In many cases, this approach is typically based on building a virtual model of the real world from captured real world data. In such a case, the feeling of virtual reality is formed by evaluating this model.

Many current approaches are generally sub-optimal and often tend to place high demands on computing and communication resources and/or provide a sub-optimal user experience, eg due to reduced quality or limited freedom.

As an example of application, virtual reality glasses have appeared on the market, which allow viewers to perceive the captured 360° (panoramic) or 180° video. Such 360° videos are often pre-filmed using a multi-camera setup in which individual images are stitched together into a single spherical display. The usual stereo formats for 180° or 360° video are top/bottom and left/right. As with non-panoramic stereoscopic video, the left eye and right eye images are compressed, for example, as part of a single H.264 video stream.

In addition to visual reproduction, most VR/AR applications also provide an appropriate audio experience. In many applications, audio preferably provides a spatial perception of sound, with audio sources perceived as emitting sound from positions that correspond to the positions of corresponding objects in the visual scene. Thus, the audio and video scenes are preferably perceived to be consistent, with both providing a complete spatial experience.

In terms of audio, up to now, the focus has been on playback through headphones using conversion technology to reproduce binaural audio. In many cases, playback through headphones enables a highly immersive personalized experience for the user. With head tracking, the transformation can be made responsive to the user's head movements, greatly enhancing the sense of immersion.

Recently, both in the marketplace and in the discussion of standards, use cases have begun to be offered that include the "social" or "general" aspect of VR (and AR), i.e. the ability to share perceptions with other people. It can be people in different places, but also people in the same place (or a combination of both). For example, several people in the same room may have the same overall experience with the presence of a projection (audio or video) of each participant in the VR content/scene. For example, in a multi-person game, each player may have a different location in the game scene and hence a different projection of the audio and video scene.

As a specific example, MPEG is attempting to standardize the bitstream and decoder for realistic, immersive AR/VR experiences with six degrees of freedom. Social VR is an important feature and allows users to interact in a shared environment (games, conference calls, online shopping, etc.). The concept of social VR also contributes to making virtual reality a more social experience for users who are physically in the same place, but in the case, for example, when a head-mounted display or other VR headset provides isolation from the perception of the physical environment.

The disadvantage of playing through headphones in such use cases of "social" or "general" AR (or VR) is that when each user wears individual headphones, located in the same place (for example, a room), users are at least partially acoustically isolated from each other, which reduces the “social” part of perception (for example, it becomes difficult or uncomfortable for people standing next to each other to conduct a natural conversation).

This problem can be solved by using speakers instead of headphones for audio playback. However, this approach has the disadvantage that audio playback cannot be as easily adapted and tailored to a particular user. For example, it is difficult to dynamically adapt audio playback to head movements and, in particular, to changes in the orientation of the head of each individual user. This effect is very important for the immersive experience, and therefore loudspeakers are generally not optimal for delivering an optimized user experience.

Therefore, an improved audio processing approach would be useful, in particular for perceptual/virtual/augmented/mixed reality applications. In particular, an approach would be useful to improve performance, increase flexibility, reduce complexity, facilitate implementation, improve audio experience, improve audio and visual scene perception matching, improve adaptation, improve personification, improve virtual reality experience, and/or improve performance and /or work.

DISCLOSURE OF THE INVENTION

Accordingly, the present invention is directed to preferably weakening, alleviating or eliminating one or more of the above disadvantages, individually or in any combination.

In accordance with an aspect of the present invention, an audio device is provided, comprising: a receiver for receiving data describing an audio scene, the data containing audio data for a set of audio elements corresponding to audio sources in the specified scene, and metadata containing at least a first index of a transform property for playing audio for the first audio element from a set of audio elements; a first playback converter for converting to reproduce audio elements by generating a first set of audio signals for a set of speakers; a second playback converter for converting to reproduce audio elements by generating a second set of headphone signals; and a selector configured to select between the first playback converter and the second playback converter for converting to reproduce at least the first part of the first audio element in response to the first audio playback transform property indicator; wherein the audio playback transform property pointer indicates whether the first part of the first audio element is associated with a position-dependent listener pose or a position-independent listener pose.

This approach may provide an improved user experience in many implementations, and may in particular provide an improved user experience for many virtual reality applications (including augmented and mixed reality), including but not limited to social or general experiences. The present approach can provide improved performance with hybrid rendering. For example, in many implementations, it may allow users in the same room to facilitate face-to-face conversations with each other while providing a targeted and personalized transformation for audio scene reproduction.

The audio playback transform property pointer may indicate whether the first audio element represents an audio source with a spatial property that is head-orientation-bound or head-orientation-free (corresponding to a listener's pose-dependent position and a listener's pose-independent position, respectively). This approach can reduce complexity and resource requirements.

In some embodiments, the device may include a first driver for driving a set of speakers with a first set of audio signals and a second driver for driving headphones with a second set of audio signals. The first set of audio signals may in particular be a set of surround sound signals and the second set of audio signals may be in particular a binaural stereo signal.

The first audio playback transform property pointer may indicate a playback transform property to be applied to the first audio element, or a property of the first audio element.

In accordance with an optional feature of the present invention, the apparatus also comprises a listener position receiver for receiving a listener posture indicative of the listener posture, wherein the first playback transducer is configured to generate a first set of audio signals independent of the listener's posture, and the second playback transducer is configured to generate a second set of audio signals. audio signals in response to the listener's posture.

An audio device can provide a highly favorable and flexible user experience, allowing for a tight match between, for example, the user's movement and the perceived audio scene. Pose may refer to position and/or orientation data and may also be referred to as position. The listener's posture may be an indication of the position of the listener, an indication of the orientation of the listener, or a combination of the indication of the position and orientation of the listener. A pose/position may be represented by one or more values providing an indication of position and/or direction.

In accordance with an optional feature of the present invention, the apparatus is configured to generate audio signals for a plurality of listeners, wherein the first playback converter is configured to generate the first set of audio signals as a common set of audio signals for the plurality of listeners; and the second playback converter is configured to generate a second set of first listener headphone audio signals from the plurality of listeners and generate a third set of second listener headphone audio signals from the plurality of listeners.

The audio device may provide effective support for multiple users. In many applications, improved support for reducing complexity and resource usage can be achieved while still providing an engaging user experience, often with a consistent and natural feel to the audio space.

The second set of audio signals may be generated in response to the first listener position for the first listener, and the third set of audio signals may be generated in response to the second position for the second listener. The first set of signals can be generated independently of the listeners' postures.

In accordance with an optional feature of the present invention, the first part is the frequency subband of the first audio element.

This may provide improved performance in many implementations.

In accordance with an optional feature of the present invention, the selector is configured to select different playback transducers from a first playback transducer and a second playback transducer for the first part of the first audio element and for the second part of the first audio element.

This may provide an improved user experience in many implementations. The selector may in particular be configured to select different transducers for reproduction for different frequency ranges of the first audio element.

This can provide an efficient approach in many applications. The audio playback transform property pointer may indicate whether the first audio element is diegetic or not.

According to an optional feature of the present invention, the audio playback transform property pointer indicates the audio format of the first audio element.

This may provide an improved user experience in many implementations. The transform property pointer for audio playback may indicate an audio format from a set of audio formats containing at least one audio format from the group: an audio object format, a higher order ambiophonic audio format, and an audio channel signal audio format.

According to an optional feature of the present invention, the audio playback transform property pointer specifies the audio source type for the first audio element.

This may provide an improved user experience in many implementations. The audio playback transform property pointer may indicate an audio source type from an audio source type set containing at least one audio source type from the group: speech audio, music audio, foreground audio, background audio, voice-over audio, and audio of an actor reading text from an author.

According to an optional feature of the present invention, the audio playback transform property pointer indicates a playback transform guide property for the first audio element's transform to reproduce.

This may provide improved user experience and/or performance in many implementations.

According to an optional feature of the present invention, the audio playback conversion property indicator indicates whether the first part of the first audio element is to be converted for playback through speakers or headphones.

This may provide improved user experience and/or performance in many implementations.

In accordance with an optional feature of the present invention, the receiver is also configured to receive visual data indicative of a virtual scene corresponding to an audio scene, and an audio playback transform property pointer indicating whether the first audio element represents an audio source corresponding to an audio scene object.

This may provide improved user experience and/or performance in many implementations.

In some embodiments, the audio playback transform property pointer may indicate whether the first audio element represents an audio scene corresponding to a scene object that is within the viewport defined for the listener's current pose.

In accordance with an optional feature of the present invention, the apparatus also comprises a user input for receiving a user input, wherein the selector is configured to select between a first playback transducer and a second playback transducer for converting to play at least a first portion of the first audio element in response to the user input.

This may provide an improved user experience in many implementations.

In accordance with an optional feature of the present invention, the selector is configured to determine an audio property of the first audio element and select between a first playback transducer and a second playback transducer to transform to reproduce at least a first portion of the first audio element in response to the audio property.

This may provide improved user experience and/or performance in many implementations.

In accordance with an aspect of the present invention, an audio processing method is provided, including: receiving data describing an audio scene, the data containing audio data for a set of audio elements corresponding to audio sources in the specified scene, and metadata containing at least a first index of a transform property for playing audio for the first audio element from a set of audio elements; converting to reproduce audio elements by generating a first set of audio signals for a set of speakers; transformations for reproducing audio elements by generating a second set of headphone signals; and selecting between a transform to reproduce at least a first portion of the first audio element for the speaker set and the headphone set in response to the first audio transform property pointer; wherein the audio playback transform property pointer indicates whether the first part of the first audio element is associated with a position-dependent listener pose or a position-independent listener pose.

These and other aspects, features and/or advantages of the present invention will become apparent from the embodiments described later in this document and will be explained with reference to the embodiments.

BRIEF DESCRIPTION OF THE DRAWINGS

Embodiments of the invention will be described by way of example only, with reference to the drawings, in which:

in FIG. 1 shows an example of a virtual reality system based on a client-server architecture; And

in FIG. 2 shows an example of elements of an audio device in accordance with some embodiments of the present invention.

IMPLEMENTATION OF THE INVENTION

Virtual reality experiences (including augmented and mixed reality) that allow the user to navigate in a virtual or augmented world are becoming increasingly popular, and services are being developed to meet this demand. In many such approaches, visual data and audio data can be generated dynamically to reflect the user's (or viewer's) current position.

In the art, the terms "location" and "posture" are used as a general term for position and/or direction/orientation. A combination of position and direction/orientation, such as an object, camera, head, or view, may be referred to as a posture or position. Thus, a position or pose indication may include up to six values/components, or components/degrees of freedom, with each value/component/degree of freedom describing a separate property of the position/location or orientation/direction of the respective object. Of course, in many situations, a location or pose can be represented using fewer components, for example, if one or more components is considered fixed or irrelevant (for example, if all objects are considered to be located at the same height and have a horizontal orientation, then four components can provide a complete representation of the object's pose). Hereinafter, the term "pose" is used to refer to the position and/or orientation, which can be represented by one to six values (corresponding to the maximum possible degrees of freedom).

Many applications of virtual reality are based on a posture that has a maximum degree of freedom, i.e. three degrees of freedom each of position and orientation, resulting in a total of six degrees of freedom. Thus, a pose may be represented by a set or vector of six values representing six degrees of freedom, and therefore a pose vector may provide an indication of a 3D position and/or a 3D direction. However, it is understood that in other implementations, the pose may be represented by fewer values.

A system or object based on providing maximum degrees of freedom for the viewer is usually referred to as having 6 degrees of freedom. Many systems and objects only provide orientation or position and are commonly referred to as having 3 degrees of freedom.

Typically, a virtual reality application generates 3D output as separate left and right eye view images. They can then be presented to the user by suitable means, typically such as separate displays for the left and right eyes of a virtual reality headset. In other embodiments, one or more view images may, for example, be presented on an autostereoscopic display, or in fact, in some embodiments, only one 2D image may be generated (eg, using a conventional 2D display).

Similarly, for a given viewer/user/listener pose, an audio representation of the scene can be provided. The audio scene is typically rendered to provide a spatial experience in which audio sources are perceived as originating from desired positions. Audio sources can be static in a scene, and changes in the user's pose will change the relative position of the audio source relative to the user's pose. Accordingly, the spatial perception of the audio source should be changed to reflect the new position relative to the user. The audio playback transform can be adapted accordingly depending on the user's posture.

Viewer or user pose input can be defined in different ways in different applications. In many implementations, the physical movement of the user can be tracked directly. For example, a camera capturing a user's area can detect and track the head (or even eyes (eye tracking)) of the user. In many implementations, the user may be wearing a virtual reality headset that can be monitored by external and/or internal means. For example, the headset may contain accelerometers and gyroscopes that provide information about the movement and rotation of the headset and hence the head. In some examples, the virtual reality headset may transmit signals or contain (eg, visual) identifiers that allow an external sensor to determine the position of the virtual reality headset.

In some systems, the viewer's pose may be provided by manual means, such as by the user manually operating a joystick or similar manual input. For example, the user may manually move the virtual viewer around the virtual stage by operating the first analog stick with one hand, and manually control the direction the virtual viewer is facing by manually moving the second analog stick with the other hand.

In some applications, a combination of manual and automated approaches may be used to generate the viewer's entrance pose. For example, the headset can track the orientation of the head, and the movement/position of the viewer in the scene can be controlled by the user using a joystick.

In some systems, the virtual reality application may be provided to the viewer locally, such as by a stand-alone device that does not use or even have any access to any remote virtual reality data or processing. For example, a device such as a game console may include a store for storing scene data, an input for receiving/shaping a viewer's pose, and a processor for generating appropriate images from the scene data.

In other systems, the virtual reality application may be implemented and executed remotely from the viewer. For example, a device local to the user may detect/receive motion/pose data transmitted to a remote device that processes the data to form the viewer's pose. Thereafter, the remote device may generate appropriate view images for the viewer's pose based on the scene data describing the scene. The view images are then transferred to a device local to the viewer where they are presented. For example, the remote device may directly generate a video stream (typically a stereo/3D video stream) that is directly presented by the local device.

Similarly, the remote device may generate an audio scene that reflects the virtual audio environment. In many implementations, this can be done by generating audio elements that correspond to the relative position of different radio sources in the virtual audio environment, and are converted for playback so that they are perceived in their respective positions.

For example, a remote device may generate audio data representing an audio scene and may transmit audio components/audio objects/audio signals or other audio elements corresponding to different audio sources in the audio scene together with position information indicating the position of those sources (which may, for example, change dynamically for moving objects) . Audio elements may include position-specific audio elements, but may also include elements for more distributed or diffuse audio sources. For example, audio elements may be provided representing a general (non-localized) background sound, ambient sound, diffuse reverb, and so on.

In such a case, the local VR device can render the audio elements appropriately, for example by applying appropriate binaural processing that reflects the relative position of the audio sources for the audio components.

With respect to audio in a VR service, in some embodiments, a central server may appropriately generate audio data representing an audio scene, and may specifically represent that adioscene with a set of audio elements that can be rendered by a local client/device.

FIG. 1 shows an example of a VR system in which a central server 101 communicates with a number of remote clients 103, such as through a network 105 such as the Internet. The central server 101 may be configured to support a potentially large number of remote clients 103 at the same time.

Such an approach can provide an improved trade-off between, for example, complexity and resource requirements for different devices, communication requirements, etc., in many scenarios. For example, the viewer's pose and corresponding scene data may be transmitted at longer intervals, with the local device processing the viewer's pose and the received scene data locally to provide real-time perception with low latency. This can, for example, significantly reduce the required communication bandwidth while providing low latency perception while allowing centralized storage, generation and maintenance of scene data. This may, for example, be suitable for applications in which a virtual reality experience is provided on multiple remote devices.

FIG. 2 shows the elements of an audio device that can provide improved conversion for audio playback in many applications and scenarios. In particular, the audio device can provide improved playback conversion for many VR applications, and the audio device can specifically be configured to perform audio playback processing and conversion for the VR client 103 shown in FIG. 1.

The audio device shown in FIG. 2 is configured to be converted for audio scene playback by generating a hybrid output set, wherein the first (sub-)set of outputs is generated for conversion for playback by a set of speakers, and the second (sub-)set of outputs is generated for conversion for playback by headphones. The first set of audio signals may in particular be a set of surround sound signals to be converted for playback on a surround sound installation. The second set of audio signals may in particular be a binaural stereo signal to be converted for playback in headphones.

The audio device in FIG. 2 may be part of a hybrid audio playback system for VR/AR that uses a combination of headphone and speaker playback to provide an audio scene presentation.

This approach can work effectively in many implementations. For example, in many scenarios, using a combination of headphone and speaker playback, instead of either playback in one or the other, can provide a highly immersive AR (or VR/MR) experience for each individual user while not compromising for the "social" or "general" aspect of perception. For example, due to this, the audio being converted for playback can be adapted to individual users and the current context for the user. For example, this may allow the positions of the audio sources to be precisely adapted to match the movements/turns of the user's head. At the same time, this can reduce the complexity required for binaural processing since significant portions of the audio scene can be rendered for playback with less complex audio/surround processing. In addition, this may be based, for example, on the use of headphones with low external sound attenuation, which, for example, facilitates direct interaction between users in the same environment/room.

The following description will focus on implementations in which the system converts an audio scene for playback using a combination of a surround sound setup (e.g., a 5.1 or 7.1 system) that is common to all local users, and individual (open or semi-open) headphones for individual users (where "individual headphones" means headphones that convert for reproduction a signal that has been generated or adapted for the user wearing the headphones).

In particular, the device will be described with reference to an example of using the "social" or "general" aspect of a VR/AR/MR application with a common perception by a plurality of people. They can be in different locations, but more interestingly in this example, they can also be in the same location (for example, in the same room). As a specific use case, multiple people are in the same room and have the same overall AR experience that is "projected" within their shared real world environment. For example, a couple sitting together on the couch watching an immersive movie projected virtually onto the wall of their living room. They may wear transparent glasses to allow them to see each other and their environment, as well as open headphones to allow both specially designed personalized conversion for playback and hear audio in the environment, including that generated by the surround sound setup.

In particular, the device in FIG. 2 includes a receiver 201 that is configured to receive data describing a virtual scene. The data may include data that provides a visual description of a scene and may contain data that provides an audio description of a scene. Thus, the received data can provide an audio scene description and a visual scene description.

The receiver 201 is connected to a rendering renderer 203 which proceeds to render rendering of images corresponding to the viewer's current viewing position. For example, the data may contain spatial 3D image data (e.g., images and depth or description of a scene model), and from these, the rendering renderer 203 may generate stereo images (an image for the user's left and right eyes), as is known to one skilled in the art. Images can be presented to the user, for example, using separate displays for the left and right eyes of a VR headset.

The received data contains audio data describing the scene. In particular, the audio data contains audio data for a set of audio elements corresponding to audio sources in the scene. Some audio elements may represent localized audio sources in the scene that are associated with a specific position in the scene (of course, the position may change dynamically in the case of a moving object). Often, an audio element may represent audio generated by a particular scene object in a virtual scene, and therefore may represent an audio source at a position corresponding to the position of the scene object (eg, a speaking person).

Other elements may represent more distributed or diffuse audio sources such as, for example, ambient or background noise, which may be diffuse. As another example, some audio elements may wholly or partly represent non-spatialized audio components from localized audio sources, such as, for example, diffuse reverberation from a well-defined audio source.

The audio elements may be encoded audio data, such as encoded audio signals. The audio elements may be different types of audio elements, including different types of signals and components, and even in many implementations, the first receiver 201 may receive audio data that defines different types/formats of audio. For example, the audio data may contain audio represented by audio channel signals, individual audio objects, Higher Order Ambisonics (HOA), and so on.

The audio may, for example, be represented as encoded audio for a given audio component to be converted for playback. The audio data may also contain position data that indicates the position of the source of the audio component. The positional data may, for example, contain absolute position data specifying the position of the audio source in the scene.

The device also contains two transducers 205, 207 for reproduction.

The first playback transducer 205 is configured to convert to reproduce audio elements through a set of speakers. In particular, the first playback converter 205 may generate a first set of audio signals for a set of speakers, the first set of audio signals being, for example, a set of surround signals for a surround sound speaker setup.

Thus, the first playback converter 205 can generate audio signals that are intended to be converted for playback by a speaker system of a particular configuration. The first playback transducer 205 may generate a signal for each speaker of the surround configuration, and hence for conversion for playback from a specific location corresponding to the position of the speaker in the configuration.

The first playback transducer 205 may be configured to generate the audio signals so that the transformation to reproduce a given audio element is performed in such a way that the cumulative effect gives the impression of an audio element being transformed to be played from a desired position. Typically, the received data may, for at least some of the audio elements, contain specific position indications, and the first playback transform 205 may render the audio elements such that they are perceived as originating from the specified position. Other audio elements may be distributed and scattered, for example, and may be rendered as such.

It is understood that many conversion algorithms and approaches are known to those skilled in the art for reproducing spatial audio using loudspeakers, and in particular surround sound systems, and that any suitable approach may be used without prejudice to the present invention.

For example, the first playback converter 205 may generate audio signals for five speakers in a surround configuration with a center speaker, a left front speaker, a right front speaker, a left surround speaker, and a right surround speaker. The first transducer 205 for reproduction may generate an audio signal set containing an audio signal for each speaker system. The signals can then be amplified to form drive signals for a particular speaker system.

In some implementations, the audio element being converted for speaker playback may be taken as stereo downmix, for example, and the first playback converter 205 may be upmixed to generate surround signals, which in some cases may be directly converted for playback. This approach can be useful, for example, for audio elements that represent diffuse sound that is not directly related to the user's posture. For example, an audio element representing the overall diffuse surround audio may be provided as a downmix stereo that is directly upmixed to provide proper surround audio channels. Each of the upmix signals can be combined with corresponding speaker signals generated from other audio elements to form a set of output signals.

Some audio elements that are rendered for playback through an acoustic setup may be provided, for example, in the form of audio objects. Such an audio object may be represented by audio data describing a particular audio and associated with position data that describes the position of the audio source. Based on the position data and speaker positions (whether actual positions or nominal positions for a surround speaker setup), the first playback converter 205 can determine coefficients for a matrix or vector mapping the audio signal to different surround channels.

In some embodiments, the first playback transducer 205 may also be configured to adapt the generated audio signals based on the acoustic environment data. For example, if the provided data indicates that the current environment is a highly reflective environment (e.g., a bathroom or similar highly reflective acoustic environment), then the first rendering transducer 205 may generate and apply a filter having an impulse response corresponding to transfer function for the environment (first reflections, etc.). In some embodiments, the filter may be applied to each of the generated audio signals for the individual surround channels, or in some embodiments, may be applied to the audio element before the different audio channels are upmixed.

In some embodiments, the first playback transducer 205 may alternatively or additionally be configured to add reverb, which may in particular be based on environmental data received along with the audio element. For example, the first transducer 205 for playback may apply a synthetic reverb, such as a Jot reverb, with parameters set depending on the acoustic environment data (eg, reverb duration as specified by the data). Typically a reverb can be applied to an audio element prior to any up-mixing or mapping to the surround channels. The second playback converter 207 is configured to generate a second set of headphone audio signals. The second set of audio signals may in particular be a binaural stereo signal.

In many embodiments, the playback transform by the second playback transducer 207 is a binaural playback transform process using suitable binaural transfer functions to provide the desired spatial effect to the user wearing the headphones. For example, the second playback transducer 207 may be configured to generate an audio component that is to be perceived by binaural processing as coming from a particular position.

As is known, binaural processing is used to provide spatial perception through the virtual positioning of sound sources using separate signals for the listener's ears. With proper binaural rendering processing, the signals needed at the eardrums for the listener to perceive sound from any desired direction can be computed, and these signals can be rendered to produce the desired effect. These signals are then played back at the eardrum using either headphones or crosstalk cancellation (suitable for conversion for playback on speakers placed close together). Binaural rendering for playback can be thought of as a signal generation approach to the listener's ears that gives the human auditory system the false impression that the sound is coming from the desired positions.

The binaural conversion for playback is based on binaural transfer functions, which vary from person to person due to the acoustic properties of the head, ears, and reflective surfaces such as the shoulders. For example, binaural filters can be used to create a binaural recording that simulates multiple sources at different locations. This can be done by convolving each sound source with a pair of Head Related Impulse Response (HRIR) that correspond to the position of the sound source.

A well-known way to determine binaural transfer functions is binaural recording. This is a sound recording method that uses a specially designed microphone layout and is designed to be played back with headphones. Recording is carried out either by placing microphones in the subject's ear canal or by using a headform with built-in microphones - a bust containing auricles (outer ears). The use of such a headform containing auricles provides a spatial experience very similar to that of the person listening to the recording being present at the time of recording.

Suitable binaural filters can be determined by measuring, for example, the responses from sound sources at a specific location in 2D or 3D space on microphones placed at or near the person's ears. Based on such measurements, binaural filters can be generated that reflect the acoustic transfer functions to the user's ears. Binaural filters can be used to create a binaural recording that simulates multiple sources at different locations. This can be done by convolving each sound source with a pair of measured impulse responses for the desired sound source position. To create the illusion of a sound source moving around the listener, as a rule, a large number of binaural filters with sufficient spatial resolution, such as 10 degrees, are required.

Binaural head transfer functions can be represented, for example, as Head Related Impulse Response (HRIR), or equally as Head Related Transfer Function (HRTF), or as binaural impulse responses. room (Binaural Room Impulse Response, BRIR), or binaural transitional room functions (Binaural Room Transfer Function, BRTF). The (estimated or implied) transfer function from a given position to the ears (or eardrums) of a listener may, for example, be given in the frequency domain, in which case it is usually called HRTF or BRTF, or in the time domain, in which case it is usually called HRIR or BRIR. In some scenarios binaural head transfer functions are defined to include aspects or properties of the acoustic environment and in particular the room in which measurements are taken, while in other examples only user characteristics are taken into account. Examples of functions of the first type are BRIR and BRTF.

The second playback transducer 207 contains a store with binaural transfer functions, typically for a large number of different positions, with each binaural transfer function providing information on how the audio signal should be processed/filtered to be perceived as originating from that position. To generate an audio scene with multiple audio sources located at appropriate positions in sound space, binaural processing individually applied to multiple audio signals/audio sources and combining the result can be used.

For a given audio element to be perceived as originating from a given position relative to the user's head, the second playback transducer 207 may select and retrieve the stored binaural function that most closely matches the desired position (or, in some case, may generate it by interpolation between a set of close binaural transfer functions). After that, it can apply the selected binaural transfer function to the audio signal of the audio element, thereby generating an audio signal for the left ear and an audio signal for the right ear.

The resulting left ear and right ear stereo signal output is then suitable for conversion for headphone playback and may be amplified to generate excitation signals that are fed into the user's headset. The user will then perceive the audio element as originating from the desired position.

It is understood that in some embodiments, the audio element may also be processed, for example to add acoustical ambient effects. For example, as described for the first transducer 205 for playback, the audio element can be processed to add reverb or, for example, decorrelation/spaciousness. In many implementations, this processing may be performed on the generated binaural signal rather than on the element audio signal itself.

Thus, the second playback transducer 207 can be configured to generate audio signals such that a given audio element is converted for playback such that a user wearing headphones perceives the audio element to be perceived from a desired position. Typically, the second renderer 207 may render the audio elements such that they are perceived as originating from a position indicated in the positional data included in the audio data. Possibly other audio elements may be distributed and scattered, for example, and may be rendered as such.

The device may suitably be part of a client 103 that receives data containing audio data describing an audio scene from a central server 101. In many implementations, the central server 101 may provide a number of audio elements in the form of audio objects, audio channels, audio components, HOAs, audio signals, and so on. In many situations, some audio elements may correspond to a single audio source having a particular position. Other audio elements may correspond to more diffuse and less well-defined and more distributed audio sources.

It is understood that many algorithms and conversion approaches are known to those skilled in the art for spatial audio playback using headphones, and in particular for binaural playback conversion, and that any suitable approach can be used without prejudice to the present invention.

In such a case, the device shown in FIG. 2 may be used in the client 103 to process received audio data for conversion to reproduce the desired audio scene. In particular, it can process each audio element based on the required position data (when appropriate) and then combine the results.

Accordingly, in order to generate audio representing a scene, the device in FIG. 2 uses two different conversion methods for playback. Different conversion methods for reproduction may have different properties, and the device shown in FIG. 2 includes a selector 209 configured to select which audio elements are rendered for playback by the first renderer 205 and which audio elements are rendered by the second renderer 207. In particular, for a given first audio element, the selector 211 may select which playback transform 205, 207 should be used for the playback transform. The selector 209 may respectively receive the first audio element and supply it to the first playback converter 205 or the second playback converter 207 depending on the selection.

In the system, receiver 201, in addition to audio data (and possibly visual data), is configured to receive metadata containing transform property pointers for playing audio for at least one of the audio elements, and often for most or even all of the audio element. In particular, for the first audio element, at least a first transform property pointer is included for audio playback.

The selector 209 is configured to select which playback transform to use depending on the received metadata and property pointers of the audio playback transform. In particular, the selector 209 is configured to consider the first audio playback transform property pointer and decide whether the first audio element is to be converted by the first playback transformer 205 or the second playback transformer 207, i. whether it should be converted for playback using speakers or headphones.

As an example of low complexity, the data for an audio element may include encoded audio data as well as metadata containing an indication of a position (typically the position of the audio source corresponding to the audio element) and an indication of an audio playback transform property for the audio element, wherein the indication of the audio playback transform property in a particular example is may simply be a binary indication of whether the first audio element should be rendered with the first renderer 205 or with the second renderer 207. The selector 209 may then evaluate this binary indication and select the indicated transducer 205, 207 for playback. The playback converter 205, 207 can then generate the proper output signals for the speakers and headphones, respectively, so that the audio elements are perceived as coming from the position indicated by the position indicator. The contributions of each of the audio elements that are indicated to be converted for playback using the playback converter 205 can then be combined to form a first set of speaker audio signals, and the contributions of each of the audio elements that are indicated to be converted for playback using the second converter 207 for playback, then can be combined to form a second set of audio signals for headphones.

Thus, the audio device in FIG. 2 can render an audio scene through a hybrid audio rendering transform system comprising both speakers and headphones. In addition, the control of the distribution of audio elements to headphones and speakers can be controlled/managed remotely. For example, the VR experience provider may also control and decide how the audio elements should be rendered for playback. Since the service provider can typically have additional information about the specific nature of the audio source for each audio element, this may allow one to choose how to render each audio element based on the additional information and knowledge of what may not be available on the client side. This approach may provide improved rendering for playback in many situations and may provide an improved user experience in many scenarios. This approach can, for example, provide an accurate and natural conversion to reproduce an audio scene while, for example, allowing people in the same room to talk to each other more naturally.

Thus, in many implementations, the audio playback transform property pointer can provide guidance to the client and audio device on how to transform received audio data for playback. The audio playback transform property pointer may indicate a playback transform guide property for the first audio element's rendering transform. In many implementations, a rendering transform guiding property may be a preferred, suggested, or nominal rendering transform property that is recommended for use by a local rendering transform. Thus, the playback transform guide property may be control data that can be used by the client to set a transform parameter for playing the playback transform.

In some implementations, the playback transform guiding property may be intended as a required playback transform property to be used in the audio element's playback transform, but in other implementations, the playback transform guiding property may be a suggested property that can be used or not used by the client. Thus, in many implementations, an audio device may choose whether to tailor its playback transform to conform to the playback transform's guide property, or may choose to use a different value. However, this approach provides an approach that allows the device to tailor its operation under the guidance of a remote server/service provider. Due to this, improved performance can be achieved in many implementations because the remote server/service provider may have additional information. For example, it may also allow for centralized manual optimization or analysis to potentially improve the rendering transformation while allowing the client to retain freedom and flexibility in the rendering transformation.

In the specific example mentioned above, the audio playback transform property pointer indicates whether the first audio element is intended to be converted for playback through speakers, or whether it is intended to be converted for playback through headphones. The selector 209 may be configured to select, for the first audio element, the first transducer 205 for playback for transposition for playback if the first transducer for playback for the first audio element indicates that the first audio element is intended for transposition for playback by loudspeakers, and selecting the second transducer 207 for playback. of the first audio element, if the first playback conversion pointer indicates that the first audio element is to be converted for headphones playback. Thereafter, the selector 209 may provide it to the selected renderer 205, 207 for rendering.

Thus, in many implementations, the audio playback transform property pointer indicates the playback transform property to be applied to the first audio element, and in particular, the playback transform pointer for the audio element may indicate whether the audio element is intended to be rendered for playback by loudspeakers. or headphones.

In some implementations, respectively, using metadata in the content stream, it is possible to explicitly indicate whether the audio element should be converted for playback through speakers or through headphones in the case of using a hybrid playback system. This may be a clear artistic choice made by the content producer, and therefore may provide improved rendering control/guidance.

In the device of FIG. 2, the transformation for audio playback (as well as the transformation for visual playback) may depend on the pose of the viewer. Specifically, the device includes a listener posture receiver 211 that is configured to receive a listener posture indicative of the listener posture. The pose of the listener may in particular be represented by the pose of the headset, for example determined by tracking the VR headset being worn by the user/listener. It is understood that any suitable method of generating, evaluating, receiving and delivering a listener's posture may be used without prejudice to the present invention.

The receiver 211 of the listener's posture is connected to the visual transducer 203 for playback and is used to generate a visual output corresponding to a particular posture. In addition, the receiver 211 of the listener's posture is connected to the second transducer 207 for reproduction, and is used for conversion for reproducing headphone audio elements. Thus, the second playback transducer 207 is configured to generate a second set of audio signals in response to the listener's posture.

The second playback transducer 207 may specifically perform binaural playback transformation such that the audio elements are rendered as originating from appropriate positions relative to the listeners' current orientation and position. For example, for a first audio element, the second playback transformer 207 may first determine a position in scene space indicated by a position indicator received from the first audio element in the stream data. The relative position of the first audio element relative to the user can then be determined by analyzing the listener's current pose and the corresponding pose in scene space. The second renderer 207 may then extract the HRTFs corresponding to this relative position and filter the first audio signal using the extracted HRTFs to generate a binaural stereo signal for the first audio element. After that, the components can be added to the corresponding components formed from other audio elements to form output binaural stereo signals.

It is understood that many other approaches are known for generating headphone signals (and in particular binaural signals) corresponding to audio sources at spatial positions, and that such a suitable approach or algorithm can be used by the second transducer 207 for playback.

Unlike the second playback transducer 207, the playback transformation of the first playback transducer 205 (i.e., the playback transformation for speakers) is independent of the listener's posture, and therefore the first playback transducer 205 in the example of FIG. 2 is configured to generate the first set of audio signals regardless of the listener's posture.

The first playback transducer 205 may specifically take into account the indication of a position for an audio element to be rendered by the first playback transducer 205 and map it to a position in the speaker rendering transform space. The first transducer 205 for reproduction may then generate signals for the speakers to provide a spatial perception of the audio element corresponding to a certain position.

It is understood that many other approaches are known for generating speaker signals (and in particular surround signals) corresponding to audio sources at spatial positions, and that such a suitable approach or algorithm can be used by first transducer 205 for playback.

Thus, in this example, the headphone signals are continuously shaped to reflect the movement and rotation of the listener's head, thereby providing a continuous and consistent user experience. At the same time, the transformation for speaker playback remains the same with respect to the movements and rotation of the listeners' heads, further providing a consistent approach. This approach can provide an approach in which different rendering transformation approaches provide a consistent representation of the audio scene with respect to a non-static listener.

The previous examples have focused on the situation in which a device generates an audio scene representation for a single user. However, in many implementations, the device may generate an audio scene representation for multiple users, such as, for example, two or more users in the same room.

In such a case, the first playback converter 205 may be configured to generate a common set of audio signals for multiple users, while the second playback converter 207 is configured to generate separate headphone signals for each user.

Thus, for the audio elements that are selected for playback conversion by the first playback converter 205, only one set of output signals is generated for all users, for example, only one speaker signal is generated for each speaker in the configuration, and they can generally not depend on any user-specific properties. In particular, the first set of audio signals generated for conversion for playback by loudspeakers is generated without considering any listener postures. The same transformation for displaying the audio scene is generated for all users.

However, for audio elements that are rendered by the second render converter 207, a different set of audio signals may be generated for each user. In particular, a binaural stereo signal can be generated for each user. These individual cues may be shaped to reflect properties or specific characteristics for an individual listener, and may in particular be shaped to reflect the listener's posture of an individual listener. Thus, binaural signals can be generated that reflect the user's current position and orientation.

Therefore, the device can, in particular, provide very efficient support for multi-user scenarios. The required audio processing to support multiple users can be substantially reduced. For example, binaural processing is typically relatively complex and resource intensive, and the number of signals that need to be generated using binaural processing can be substantially reduced, greatly reducing complexity and computational burden in many implementations.

Thus, in an example where the device supports two users in the same room, the first playback converter 205 may be configured to generate a common first set of audio signals for conversion for playback using speakers, and the second playback converter 207 may be configured to generate a second set of headphone audio signals for the first listener and generate a third set of headphone audio signals for the second listener. The first set of audio signals may be generated independently of the listener posture of the first and second listeners, and the second set of audio signals may be generated in response to the listener posture of the first listener, and the third set of audio signals may be generated in response to the listener posture of the second listener.

The audio playback transform property pointer provided in the received data stream may represent different data in different implementations.

The audio playback transform property pointer indicates whether the first part of the first audio element is associated with a listener pose dependent position or a listener pose independent position. The transform property pointer for audio playback may specifically indicate whether the first audio element is diegetic or not.

As a specific example, in some implementations, the selector 209 may be configured to allocate audio elements between the first playback transformer 205 and the second playback transformer 207 based on whether the audio playback transform property pointer for the first audio element indicates that it is "anchored to head orientation" or "not tied to head orientation" according to MPEG terminology"

The audio element specified by the audio playback transform property pointer as "attached to the head" is an audio element that should have a fixed location relative to the user's head. Such audio elements may be rendered using the second renderer 207 and may be rendered regardless of the listener's posture. Therefore, the transformation for playing such audio elements does not take into account the orientation (orientation changes) of the user's head, in other words, such audio elements are audio elements for which the relative position does not change when the user turns his head (for example, non-spatial audio such as ambient noise or, for example , music, is designed to follow the user without changing relative position).

The audio element specified by the audio playback transform property pointer as "not head-bound" is an audio element that should have a fixed location in the (virtual or real) environment and therefore its playback transform is dynamically adapted to (changes in) the orientation of the user's head . In many implementations, realism can be enhanced when such an audio element is rendered as a binaural headphone signal that is adapted based on the listener's current posture. For example, the perception of the position of an audio source being converted for playback by a surround speaker setup may depend on the position and orientation of the user, and therefore, converting to play back an audio element specified as "headless" by the speaker setup may result in the audio source being perceived as moving when the user's head movement.

Thus, in some embodiments, the "head-orientation-free" elements can be converted for playback through the user's headphones, with their positions adapted for each individual user according to the tracked user's head orientation. On the other hand, "head-orientation-bound" elements can be converted for playback through speakers and are not adapted to users' head movements.

The advantage of this implementation is that the "head-orientation-bound" elements, which are now mostly presented through speakers (rather than headphones), are primarily responsible for the acoustic isolation that is felt when all elements are converted for playback through headphones. . This is because “head orientation-related” sounds (mostly music and atmospheric sounds such as crowd sounds, wind, rain, thunder, etc.) are often continuous and ubiquitous in nature, resulting in a sound "veil" that isolates the user from their physical surroundings. On the other hand, "head-independent elements" are often more localized and dispersed in space and time, and therefore mask the user's physical acoustic environment much less.

In some practical implementations, the user's perception of "head-orientation-bound" sounds that are converted for playback through loudspeakers may be somewhat different from how they are typically perceived when played back through headphones. However, this usually does not pose a problem, since the "head orientation-bound" sounds that are converted for playback by loudspeakers are generally non-directional or non-critical in terms of spatial localization.

Which audio elements are "non-head-orientated" and which are "head-orientated" can be explicitly communicated via metadata in the audio content stream.

Typically in the context of AR (and VR) audio playback, the term "diegetic" is also used to describe whether an audio element should be "locked to head orientation" or not. The term "diegetic" describes elements that must remain in the same virtual position when the user moves their head (meaning that the rendered position relative to the user's head must be changed). The term "non-diegetic" describes elements for which it is not important or even preferable that their positions do not take into account the movements of the user's head (i.e., they will move with the user's head or "attached" to it).

In some implementations, the audio playback transform property pointer for an audio element may indicate the audio format of the audio element. The selector 209 may be configured to select whether to use the first playback converter 205 or the second playback converter 207 to convert to reproduce the audio element based on the audio element's audio format. The transform property pointer for audio playback may, for example, indicate that the audio element has an audio format from the group: audio object format, higher order ambiophonic audio format, and audio channel signal audio format.

In some implementations, the selector 209 may be configured to distinguish between elements to be converted by headphones or speakers based on the format of the audio elements.

For example, channel-based or Higher-Order Ambiophonic (HOA) elements that are often used to convey background sounds such as music and atmospheric sounds can be converted for playback through loudspeakers, while object elements that are typically used for transmission the main audio elements of a scene (often representing audio sources with well-defined positions) can be converted for playback through headphones on a per-user basis. This will also allow the user not only to change the orientation of their head, but also to interact with individual audio objects (if the content producer intended the objects to be interactive).

This implementation can be considered as an alternative to, or addition to, providing pointers to audio playback transform properties that directly specify which playback transform should be used. For example, in situations where no explicit signaling element is included as to whether the audio element is head-orientation-free/head-orientation-related, the selector 209 may evaluate the audio format to determine which transducer 205, 207 should be used for playback. .

Approaches and different transform property pointers for audio playback can be combined, e.g. channel elements, HOA elements, or elements that are explicitly reported as "attached to head orientation" transform for playback through loudspeakers, while objects and "unattached to head orientation" elements are converted for playback through headphones.

In some embodiments, the audio playback transform property pointer may indicate the source type for the first audio element. For example, a transform property pointer for audio playback may indicate whether the audio element is an audio source type from a set including, for example, one or more of: speech audio; foreground audio; background audio; voiceover audio; and audio of an actor reading text from the author.

In some implementations, the allocation of audio elements to speakers and headphones may be based on indications in the content stream of source types for the audio elements, such as metadata such as "speech" or "music" or "foreground" or "background sounds" . In this example, "speech" sources may be converted for playback through headphones, while "music" and "background" sources may be converted for playback through speakers. A special case would be a speech that is labeled as being "voiceover" or "actor reading from the author", which would be best transcribed for playback through loudspeakers (because it is not intended to be in a specific location in space, but, rather, for the presence "everywhere").

In some implementations, the receiver 201 may, as previously described, also receive visual data indicative of a virtual scene corresponding to an audio scene. This data may be provided to the rendering renderer 203 for rendering rendering using a suitable rendering rendering technique, such as rendering stereo images corresponding to the user's current posture.

In some implementations, the audio playback transform property pointer for an audio element may indicate whether the first audio element represents an audio source corresponding to a scene visual. A scene visual object may be an object for which the visual data contains a visual representation.

In an example where the visual data provides visual data to the viewport, the audio playback transform property pointer may indicate whether the audio element is associated with an object within the viewport.

If the audio playback transform property pointer indicates that the object corresponding to the audio element is visible in the scene, the selector 209 may decide to convert it for playback using headphones, otherwise it may convert the audio element for playback using headphones. In some embodiments, the audio playback transform property pointer may directly indicate whether the object is visible. However, in other implementations, a transform property pointer for audio playback may provide an indirect indication of whether an audio element corresponds to a visible scene object.

For example, the transform property pointer for audio playback may contain an indication of the scene object that is represented by the received visual data. The selector 209 may then proceed to evaluate whether the object associated with the audio element is visible for the current listener pose. If yes, then you can proceed to convert it for playback using headphones, otherwise the object can be converted for playback by speakers.

In some implementations, the allocation of audio elements to speakers and headphones may be based on an indication in the received content stream of whether the audio element is associated with a visual element/objects in the content stream. If the pointer indicates that this is the case, the audio element is converted for playback through headphones. If the pointer indicated that this is not the case, the audio elements are converted for playback through loudspeakers.

In the previous examples, the selector 209 has been configured to select the appropriate transducer 205, 207 for playback based on received data only. However, it is understood that in many implementations, additional considerations and, in particular, other data may be taken into account.

In many implementations, the device may include a user input function that can accept user input. In such implementations, the selector 209 may be configured to select between the first renderer 205 and the second renderer 207 based on user input. For example, the user input may be a direct indication of the preferred conversion for playback, such as, for example, an explicit indication that a particular audio element should be converted through headphones rather than speakers. In other implementations, the user input may be more indirect and may, for example, change the selection criterion or incline the selection to one of the renderers 205, 207. For example, user input may indicate that it is desirable to convert a majority of the audio elements for playback with headphones, and selector 209 may change the decision criteria to do so.

Thus, in some implementations, the user may be able to directly influence the distribution of elements across speakers and headphones. One example is to allow users to manually assign individual items for playback through headphones or speakers.

Yet another example of user control of distribution is to provide the user with two or more modes from which to select, for example, an "individual experience" mode and a "common experience" mode. In the case where the user selects the "general perception" mode, any of the above implementations for determining which audio elements should be converted for playback through speakers and headphones, respectively, can be used in combination.

In some implementations, the selector 209 may itself be configured to parse the audio elements and determine based thereon which transducer 205, 207 should be used. For example, if a transform property pointer for audio playback is not received for a given audio element, the selector 209 may proceed to parse the audio elements to determine an audio property, such as, for example, the number of audio elements in the scene, the number of channels per audio element, the position of the audio elements, the distances of the audio elements to listeners ( or before each speaker) or the movement of audio elements. Thereafter, the selector 209 may proceed to decide which converter 205, 207 to use based on that audio property or a plurality of them.

In a specific configuration example, hereinafter referred to as the X configuration, the selector 209 may select the renderer for each audio element in order to create the most accurate spatial representation of the audio scene. For example, if an audio element is in a position fairly close to that of one of the physical speakers, then it can be converted to play on that particular speaker. Conversely, if an audio element falls into an area that is not covered by any speaker system, then it can be converted for playback through headphones. The fact that the audio element has the same direction as the loudspeaker (from the listener's point of view) can also be used in the same way for a single listener as for multiple listeners, provided they are all on the same line. with speaker. However, this is generally not practical as the user may change positions over time. In this particular X configuration, the angular accuracy of the (binaural) headphone transducer 207 can be taken into account by the selector 209 to make this decision.

Thus, in some embodiments, the selection of the appropriate transducer 205, 207 for playback may be further based on an analysis of the audio signals. For example, an acoustic property estimator of audio signals may be used to determine properties such as the distance (or speed) of an audio object/audio source (particularly in the case of multi-channel signals) or reverberation time. Audio signal classifiers such as speech/music classifiers, music genre classifiers, or audio event classifiers may also be used. Also, specific type classifiers can be used to determine which type of microphone (HOA, lavalier, omnidirectional, XY…) was used to record a given signal. Audio frequency distribution analysis can also be used to decide which audio system (headphones or speakers) is more suitable for conversion to reproduce the entire audio element.

In the previous example, the selector 209 is configured to select either the first playback transformer 205 or the second playback transformer 207 for the audio elements on an element-by-element basis. However, it is clear that this is not necessary or essential. For example, in some implementations, the selector 209 may be configured to select which render converter 205, 207 to use for a group of audio elements.

In addition, in some implementations, the selector 209 may be configured to separately select between the converters 205, 207 for different parts of the same audio element. For example, for some audio elements, one part may be rendered by the first renderer 205 and another portion may be rendered by the second renderer 207.

It is understood that an audio element can be divided into different parts in different ways depending on the requirements and preferences of the individual implementation. For example, in some implementations, the audio element may be received as a combination or group of different parts, and the selector 209 may individually select the transducer 207 to be converted for each part. For example, an audio element may represent a particular audio source with a first component that represents an audio source with a well-defined position (eg, corresponding to direct audio) and a second component that represents a more diffuse and distributed sound (eg, corresponding to a reverb sound). In such a scenario, the selector 209 may be configured to convert to reproduce the first component and the second component using speakers.

In other implementations, the selector 209 may be configured to separate the audio element into different parts for conversion for playback. For example, the received audio element may correspond to an audio signal that can be analyzed to be split into different parts, which can then be converted to be played back separately.

In particular, in many implementations, different parts of an audio element may correspond to different frequency ranges. For example, the selector 209 may be configured to select, for a given first part corresponding to a particular frequency band, which transducer 205, 207 to use for playback. The same can be done for a different frequency range, which may result in using different transducers 205, 207 for reproduction for the first and second frequency ranges.

In some implementations, different parts of an audio element may be provided with different audio playback transform property pointers, and the selector 209 can take into account the specific audio playback transform property pointer for a given part when deciding how to transform it for playback. In other implementations, a transform property pointer for audio playback may be provided for the audio element as a whole, but using different decision criteria for different parts. For example, for the mid-to-high frequency range, the choice between headphones and speakers is made depending on the received transform property indicator for audio playback for the audio element, while for the very low frequency range, the first playback transducer 205 is used to convert to reproduce the signal through the speakers. whatever the transform property pointer for audio playback specifies (this reflects the fact that low frequencies tend to provide much less significant spatial cues).

For example, the signal may be divided into a low frequency part and a high frequency part using high pass filtering, with the low frequency part sent to the speakers and the high frequency part sent to the headphones depending on the transform property pointer for audio reproduction. In some implementations, advanced audio source separation may be used (eg, separation of each time-frequency point between transducers for playback).

The use of filtering that conserves energy at each time-frequency point may allow a physical hybrid rendering system to reduce the possible errors created by the filtering.

The described approach can provide a number of advantageous effects, including the previously described enabling perceptually accurate spatial transformation to reproduce an audio scene while enabling/facilitating direct interaction with users in the same location.

This approach can reduce complexity and resource usage in many scenarios by potentially reducing the amount of binaural processing required. Another often achievable benefit is the reduction in energy used by the headphone playback system, e.g. in terms of amplifier power and/or processing load for the built-in playback converter, which can be critical in the case of untethered headphones (e.g. battery powered headphones). ).

Another interesting property of a hybrid audio playback system for a VR application is that it generally provides improved security. Indeed, unlike wearing closed headphones, participants are not completely cut off from the potential dangers of the real environment around them. This can be an important factor in many practical situations.

Another advantage of a hybrid system, such as the systems described, is that a portion of the audio content is converted for playback through a common acoustic setup, which typically enhances the user's sense of the overall experience. This approach generally provides an improved user experience.

It is understood that in the foregoing description, embodiments of the present invention have been set forth for clarity with reference to various functional circuits, blocks, and processors. However, it is understood that any suitable distribution of functionality between different functional circuits, blocks or processors can be used without prejudice to the present invention. For example, functionality shown to be implemented by separate processors or controllers may be implemented by the same processor or controllers. Therefore, references to specific functional blocks or diagrams should only be considered as references to suitable means to provide the functionality described, and not as an indication of a strict logical or physical structure or organization.

The present invention may be implemented in any suitable form, including hardware, software, firmware, or any combination thereof. The present invention may optionally be implemented, at least in part, as computer software executing on one or more processors and/or digital signal processors. The elements and components of an embodiment of the present invention may be physically, functionally and logically implemented in any suitable manner. Indeed, functionality may be implemented in a single block, in multiple blocks, or as part of other functional blocks. Because of this, the present invention may be implemented in a single unit, or may be physically or functionally distributed among different units, circuits, and processors.

Although the present invention has been described in connection with certain embodiments, this should not be construed as a limitation to the particular form set forth herein. Rather, the scope of the present invention is only limited by the appended claims. In addition, while it may appear that a feature is described in connection with particular embodiments, one skilled in the art will appreciate that various features of the described embodiments may be combined in accordance with the present invention. In the claims, the term "comprising/comprising" does not exclude the presence of other elements or steps.

In addition, although a plurality of means, elements, circuits, or method steps are listed individually, they may be implemented, for example, using a single circuit, block, or processor. Further, although individual features may be included in different claims, they may be effectively combined, and inclusion in different claims does not mean that the combination of features is not feasible and/or disadvantageous. In addition, the inclusion of a feature in one category of claims does not imply a limitation to that category, but rather indicates that the feature is equally applicable to other categories of claims when appropriate. In addition, the order of the features in the claims does not indicate the specific order in which the features should be worked out, and in particular the order of the individual steps in the claims per method does not mean that the steps must be performed in that order. Rather, the steps may be performed in any suitable order. In addition, references in the singular do not exclude the plural. Therefore, references using means of indicating the singular, numerals in the singular "first", "second", etc. do not exclude sets. Reference positions in the claims are provided solely as a clarifying example and should not be construed as limiting the scope of the claims in any way.

Claims (24)

1. An audio processing device for audio data, comprising: a receiver (201) for receiving data describing an audio scene, the data comprising audio data for a set of audio elements corresponding to audio sources in the specified scene, and metadata containing at least a first index of a transformation property for playing audio for a first audio element from the set of audio elements; a first converter (205) for converting to reproduce audio elements by generating a first set of audio signals for a set of speakers; a second converter (207) for converting to reproduce audio elements by generating a second set of headphone audio signals; and at the same time, the audio device is characterized in that it contains: a selector (209) configured to select between the first converter (205) for playback and the second converter (207) for playback for conversion to reproduce at least the first part of the first audio element in response to the first a transform property pointer for audio playback; wherein the first transform property pointer for audio playback indicates whether the first part of the first audio element represents an audio source with a spatial property whose orientation is head-bound and is intended to be a bound position relative to the user's head, or represents an audio source with a spatial property whose orientation is not head-bound, and which is designed for a fixed position in the environment. 2. The apparatus of claim 1, further comprising a listener position receiver (211) for receiving a listener posture indicative of the listener posture, wherein the first transducer (205) for playback is configured to generate a first set of audio signals independent of the listener posture, and the second transducer (207) ) for playback is configured to generate a second set of audio signals in response to the listener's posture. 3. The apparatus of claim 1, configured to generate audio signals for a plurality of listeners, wherein the first playback converter (205) is configured to generate the first set of audio signals as a common set of audio signals for the plurality of listeners; and the second playback converter (207) is configured to generate a second set of first listener headphone audio signals from the plurality of listeners and generate a third set of second listener headphone audio signals from the plurality of listeners. 4. The device according to any preceding claim, wherein the first part is a frequency subband of the first audio element. 5. The device according to any previous claim, in which the selector (209) is configured to select different transducers (205, 207) for playback from the first transducer (205) for playback and the second transducer (207) for playback for the first part of the first audio element and for second part of the first audio element. 6. The apparatus of claim 1, wherein the audio playback transform property pointer indicates the audio format of the first audio element. 7. The apparatus of claim 1, wherein the audio playback transform property pointer indicates the audio source type of the first audio element. 8. The apparatus of any preceding claim, wherein the audio reproduction transform property pointer indicates a playback transform guide property for the first audio element to be converted to reproduce. 9. The apparatus of claim 8, wherein the audio reproduction conversion property indicator indicates whether the first portion of the first audio element is intended to be converted for playback through speakers or headphones. 10. The apparatus of any preceding claim, wherein the receiver (201) is also configured to receive visual data indicative of a virtual scene corresponding to an audio scene, and the audio playback transform property pointer indicates whether the first audio element represents an audio source corresponding to an audio scene object. 11. The device according to any previous claim, also containing a user input for receiving user input, and wherein the selector (211) is configured to select between the first converter (205) for playback and the second converter (207) for playback to convert to play at least at least the first part of the first audio element in response to user input. 12. The device according to any previous claim, in which the selector (209) is configured to determine the audio property of the first audio element and select between the first transducer (205) for playback and the second transducer (207) for playback for converting to reproduce at least the first part of the first audio element in response to the audio property. 13. An audio processing method, including: receiving data describing an audio scene, the data comprising audio data for a set of audio elements corresponding to audio sources in the specified scene, and metadata containing at least a first index of a transformation property for playing audio for a first audio element of the set of audio elements; converting to reproduce audio elements by generating a first set of audio signals for a set of speakers; converting to reproduce audio elements by generating a second set of headphone audio signals, and the method differs in that it includes: selecting between a transformation to reproduce at least the first part of the first audio element for a set of speakers and for headphones in response to the first index of the transformation property for audio reproduction, wherein the transform property pointer for audio playback indicates whether the first part of the first audio element represents an audio source with a spatial property whose orientation is bound to the orientation of the head and which is intended to be a bound position relative to the user's head, or represents an audio source with a spatial property whose orientation is not bound to the head, and which is designed for a fixed position in the environment. 14. A computer-readable storage medium on which program code means are stored, configured to carry out all the steps of the method according to claim 13 when said program code means are executed on a computer.

Images