CN102804264A - Apparatus and method for extracting a direct/ambience signal from a downmix signal and spatial parametric information - Google Patents

Abstract

An apparatus for extracting a direct/ambient signal from a downmix signal and spatial parameter information representing a multi-channel audio signal having more channels than the downmix signal, wherein the The spatial parameter information includes an inter-channel relationship of the multi-channel audio signal. The device includes a direct/surround estimator and a direct/surround extractor. The direct/surround estimator is configured for estimating level information of the direct part and/or the surrounding part of the multi-channel audio signal based on the spatial parameter information. The direct/surround extractor is configured for extracting the direct signal part and/or the ambient signal part from the downmix signal based on the level information of the direct part or the surrounding part.

Description

Apparatus and method for extracting direct/surrounding signal from downmix signal and spatial parameter information

technical field

The present invention relates to audio signal processing, and more particularly to an apparatus and method for extracting direct/surrounding signals from downmix signals and spatial parameter information. Other embodiments of the invention relate to utilizing direct/ambient separation for boosting binaural reproduction of audio signals. Yet other embodiments relate to binaural reproduction of multi-channel sound, wherein multi-channel audio means audio having two or more channels. Typical audio content with multi-channel sound are movie soundtracks and multi-channel music recordings.

Background technique

The human spatial auditory system tends to process sound roughly in two parts. On the one hand the localizable or immediate part and on the other hand the non-localizable or surrounding part. There are many audio processing applications, such as binaural sound reproduction and multi-channel upmixing, where it is desired to access both audio components.

A direct/ambient separation method is known in the art, e.g. in "One-time ambient signal decomposition and vector-based localization for spatial audio coding and enhancement", Goodwin, Jot, IEEE International Acoustics, Speech, and Signal Processing Conference, April 2007; "Correlation-based surround extraction from stereo recordings", Merimaa, Goodwin, Jot, AES Journal 123, New York 2007; "Multi-loudspeaker playback of stereo signals", C. Faller, AES Conference , October 2007; "One-time Surrounding Decomposition of Stereo Audio Signals Using Complex Similarity Pointers", Goodwin et al., publication number: US2009/0198356A1, August 2009; A METHOD FOR AUDIO SIGNALS", Inventor: Christof Faller, Attorney: FISH&RICHARDSON P.C., Assignee: LG Electronics Corporation, Origin: Minneapolis, MN, USA, IPC8 Class: AH04R500FI, USPC Class: 3811; and " Ambient Generation of Stereo Signals", Avendano et al., Issue Date: July 28, 2009, Application No.: 10/163,158, Application Date: June 4, 2002, these methods can be used in a number of applications. State-of-the-art direct/surround separation algorithms are based on inter-band signal comparisons between stereo and frequency bands.

Also, in "Binaural 3-D Audio Rendering Based on Spatial Audio Scene Coding", Goodwin, Jot, AES123 Session, New York 2007, addressing binaural playback using ambient extraction. Surrounding extraction for correlated binaural reproduction is also described in J. Usher and J. Benesty, "Enhancement of spatial sound quality: a novel reverberant audio upmixer", IEEE Transactions on Audio, Speech, and Language Processing, Issue 15, Issue 2141 - 2150 pages September 2007. The report that follows focuses on extraction around stereo microphone recordings using adaptive least mean square cross-channel filtering of the direct components of each channel. Spatial audio codecs such as MPEG Surround typically consist of a one- or two-channel audio stream combined with spatial side information, which extends the audio into multiple channels, as described in ISO/IEC 23003-1-MPEG Surround and Breebaart, J., Herre, J., Villemoes, L,, Jin, C., Kjorling, K., Plogsties, J., Koppens, J. (2006), "Multichannel access to mobile devices: MPEG Surrounding Binaural Presentation", Proceedings of the 29th AES Conference, Seoul, South Korea.

But modern parametric audio coding techniques such as MPEG Surround (MPS) and Parametric Stereo (PS) provide only a few audio downmix channels, and in some cases only one channel with additional spatial side information. Comparisons between "old" input channels are only possible after first decoding the sound into the desired output format.

Therefore, a concept is required to extract the direct signal part or the surrounding signal part from the downmix signal and the spatial parameter information. But using parameter side information for direct/surrounding extraction is not an existing solution.

It is therefore an object of the present invention to provide a concept for extracting direct or ambient signal parts from a downmix signal by using spatial parameter information.

This object is achieved by an apparatus according to claim 1 , a method according to claim 15 , or a computer program according to claim 16 .

Contents of the invention

The basic concept based on the present invention is when estimating the level information (level information, level information) of the direct part or the surrounding part of the multi-channel audio signal based on the spatial parameter information and from the downmixing based on the estimated level information When the signal extracts the direct signal part or the surrounding signal part, the aforementioned direct/surrounding extraction can be realized. Here, the downmix signal and the spatial parameter information represent the multi-channel audio signal having more channels than the downmix signal. Such a solution allows direct and/or ambient extraction from a downmix signal with one or more input channels by using spatial parameter side information.

According to an embodiment of the present invention, an apparatus for extracting direct and/or ambient signals from a downmix signal and spatial parameter information includes a direct/ambience estimator and a direct/ambience estimator. The downmix signal and the spatial parameter information represent a multi-channel audio signal having more channels than the downmix signal. In addition, the spatial parameter information includes an inter-channel relational expression of the multi-channel audio signal. The direct/surround estimator is configured for estimating level information of the direct part or the surrounding part of the multi-channel audio signal based on the spatial parameter information. The direct/surrounding extractor is configured for extracting the direct signal part or the surrounding signal part from the downmix signal based on the estimated level information of the direct part or the surrounding part.

According to another embodiment of the present invention, a device for extracting direct and/or ambient signals from a downmix signal and spatial parameter information includes a binarural direct sound rendering device (binarural direct sound rendering device), a binarural ambient sound rendering device and combiners. The binaural direct sound rendering device is configured for processing the direct signal portion to obtain a first binaural output signal. The binaural ambient sound rendering device is configured to process the ambient signal portion to obtain a second binaural output signal. The combiner is configured to combine the first binaural output signal and the second binaural output signal to obtain a combined binaural output signal. Thus, binaural reproduction of an audio signal can be provided, wherein the direct signal part and the surrounding signal part of the audio signal are processed separately.

Description of drawings

1 shows a block diagram of an embodiment of an apparatus for extracting direct/surrounding signals from a downmix signal and spatial parameter information representing a multi-channel audio signal;

Figure 2 shows a block diagram of an embodiment of an apparatus for extracting direct/ambient signals from a mono downmix signal and spatial parameter information representing a parametric stereo audio signal;

Fig. 3a shows a schematic illustration of spectral decomposition of a multi-channel audio signal according to an embodiment of the present invention;

Figure 3b shows a schematic illustration of the inter-channel relationship for calculating a multi-channel audio signal based on the spectral decomposition of Figure 3a;

Figure 4 shows a block diagram of an embodiment of a direct/surround extractor with estimated level information downmix;

Figure 5 shows a block diagram of yet another embodiment of a direct/surround extractor by applying gain parameters to the downmix signal;

Figure 6 shows a block diagram of yet another embodiment of a direct/surround extractor based on a Least Mean Square (LMS) solution using channel mixing;

Figure 7a shows a block diagram of an embodiment of a direct/ambient estimator using stereo ambient estimators;

Figure 7b shows a graph of an example of direct-to-total energy ratio versus inter-channel coherence;

Figure 8 shows a block diagram of an encoder/decoder system according to an embodiment of the present invention;

Figure 9a shows a block diagram of an overview of binaural direct sound presentation according to an embodiment of the invention;

Figure 9b shows a block diagram of details of the binaural direct sound presentation of Figure 9a;

Figure 10a shows a block diagram of an overview of binaural ambient sound presentation according to an embodiment of the invention;

Figure 10b shows a block diagram of binaural ambient sound rendering details of the binaural ambient sound rendering details of Figure 10a;

Figure 11 shows a conceptual block diagram of an embodiment of binaural reproduction of a multi-channel audio signal;

Figure 12 shows a general block diagram of an embodiment of direct/surrounding extraction including binaural rendering;

Figure 13a shows a block diagram of an embodiment of an apparatus for extracting a direct/surrounding signal from a mono downmix signal in the filterbank domain;

Figure 13b shows a block diagram of an embodiment of the immediate/surrounding extraction block of Figure 13a; and

FIG. 14 shows a schematic illustration of an example of an MPEG surround decoding scheme according to yet another embodiment of the present invention.

Detailed ways

FIG. 1 shows a block diagram of an embodiment of an apparatus 100 for extracting direct/surrounding signals 125 - 1 , 125 - 2 from a downmix signal 115 and spatial parameter information 105 . As shown in FIG. 1 , the downmix signal 115 and the spatial parameter information 105 represent the multi-channel audio signal 101 having more channels Ch ₁ . . . Ch _N than the downmix signal 115 . The spatial parameter information 105 may include an inter-channel relationship of the multi-channel audio signal 101 . More specifically, the device 100 includes a direct/surrounding estimator 110 and a direct/surrounding extractor 120 . The direct/surround estimator 110 may be configured to estimate the level information 113 of the direct part or the surrounding part of the multi-channel audio signal 101 based on the spatial parameter information 105 . The direct/surround extractor 120 may be configured to extract the direct signal portion 125-1 or the ambient signal portion 125-2 from the downmix signal 115 based on the estimated level information 113 of the direct portion or the ambient portion. .

FIG. 2 shows a block diagram of an embodiment of an arrangement for extracting direct/ambient signals 125 - 1 , 125 - 2 from a mono downmix signal 215 and spatial parameter information 105 representing a parametric stereo audio signal 201 . The device 200 of FIG. 2 generally includes the same blocks as the device 100 of FIG. 1 . Therefore, identical blocks having identical implementations and/or functions are labeled with identical reference symbols. Furthermore, the parametric stereo audio signal 201 of FIG. 2 may correspond to the multi-channel audio signal 101 of FIG. 1 , and the mono downmix signal 215 of FIG. 2 may correspond to the downmix signal 115 of FIG. 1 . In the embodiment of FIG. 2 , the mono downmix signal 215 and the spatial parameter information 105 represent the parametric stereo audio signal 201 . A parametric stereo audio signal may include a left channel indicated with an "L" and a right channel indicated with an "R". Here, the direct/surround extractor 120 is configured to extract the direct signal portion 125-1 or the ambient signal portion 125-2 from the mono downmix signal 215 based on the estimated level information 113; 113 may be derived from the spatial parameter information 105 through the use of the direct/surrounding estimator 110 .

In fact, the spatial parameters (spatial parameter information 105 ) in the embodiment in FIG. 1 or FIG. 2 especially refer to MPEG surround (MPS) or parametric stereo (PS) side information. These two technologies are low bit rate stereo or surround audio coding methods in the prior art. Referring to FIG. 2, the PS provides one downmix audio channel with spatial parameters, and referring to FIG. 1, the MPS provides one, two or more downmix audio channels with spatial parameters.

In particular, the embodiments of Figs. 1 and 2 clearly show that the spatial parameter side information 105 can be easily used to generate directly and/or around the extracted field.

或以虚线指示。此处，指数i表示声道Ch_i，而R表示其余声道的线性组合，而指数n及k对应于某些滤波器排组时槽（filter banktime slot）307和滤波器排组子频带303。基于这些时间/频率片（tile）

或

例如定位在相对于时间/频率轴310、320的相同时间/频率点（t₀,f₀），如图3b所示，可在步骤330中求出声道间关系式335，诸如所检查的声道Ch_i的声道间相干性（ICC_i）或声道位准差（CLD_i）。此处，声道间关系式ICC_i及CLD_i的计算可通过使用下列关系式进行：The estimation of direct and/or ambient level (level information 113 ) is based on information about inter-channel relationships or inter-channel differences, such as level differences and/or correlations. These values can be calculated from stereo or multi-channel signals. Fig. 3a shows a schematic illustration of a spectral decomposition 300 of a multi-channel audio signal (Ch ₁ ...Ch _N ) used to calculate the inter-channel relationship of the respective Ch ₁ ...Ch _N . As can be seen in Fig. 3a, the spectral decomposition of the inspected channel _Chi or the linear combination R of other channels of a multi-channel audio signal (Ch ₁ ... Ch _N ) respectively includes a plurality of sub-bands 301, wherein these multiple sub-bands Each sub-band 303 in 301 extends along a horizontal axis (time axis 310 ) with sub-band values 305 , as indicated by the small boxes of the time/frequency grid. Furthermore, sub-bands 303 are positioned consecutively along the vertical axis (frequency axis 320 ) to correspond to different frequency regions of a filter bank. In Figure 3a, the corresponding time/frequency tiles (tiles)

or indicated by dashed lines. Here, the index i denotes the channel Ch _i , while R denotes the linear combination of the remaining channels, while the indices n and k correspond to certain filter banktime slots 307 and filter bank subbands 303 . Based on these time/frequency tiles

or

For example positioned at the same time/frequency point (t ₀ , f ₀ ) relative to the time/frequency axes 310, 320, as shown in FIG. Inter-channel coherence (ICC _i ) or channel level difference (CLD _i ) of channel Ch _i . Here, the calculation of the inter-channel relations ICC _i and CLD _i can be performed by using the following relations:

${\mathrm{<span\; class="notranslate">\; ICC</span>}}_{\mathrm{<span\; class="notranslate">\; i</span>}}<span\; class="notranslate">\; =</span>\frac{<span\; class="notranslate">\; <</span>{\mathrm{<span\; class="notranslate">\; Ch</span>}}_{\mathrm{<span\; class="notranslate">\; i</span>}}{\mathrm{<span\; class="notranslate">\; R</span>}}^{<span\; class="notranslate">\; *</span>}<span\; class="notranslate">\; ></span>}{\sqrt{<span\; class="notranslate">\; <</span>{\mathrm{<span\; class="notranslate">\; Ch</span>}}_{\mathrm{<span\; class="notranslate">\; i</span>}}{{\mathrm{<span\; class="notranslate">\; Ch</span>}}_{\mathrm{<span\; class="notranslate">\; i</span>}}}^{<span\; class="notranslate">\; *</span>}<span\; class="notranslate">\; ></span><span\; class="notranslate">\; <</span>{\mathrm{<span\; class="notranslate">\; RR</span>}}^{<span\; class="notranslate">\; *</span>}<span\; class="notranslate">\; ></span>}}$

${\mathrm{<span\; class="notranslate">\; \&sigma;</span>}}_{\mathrm{<span\; class="notranslate">\; i</span>}}<span\; class="notranslate">\; =</span>\frac{<span\; class="notranslate">\; <</span>{\mathrm{<span\; class="notranslate">\; Ch</span>}}_{\mathrm{<span\; class="notranslate">\; i</span>}}{{\mathrm{<span\; class="notranslate">\; Ch</span>}}_{\mathrm{<span\; class="notranslate">\; i</span>}}}^{<span\; class="notranslate">\; *</span>}<span\; class="notranslate">\; ></span>}{<span\; class="notranslate">\; <</span>{\mathrm{<span\; class="notranslate">\; RR</span>}}^{<span\; class="notranslate">\; *</span>}<span\; class="notranslate">\; ></span>}$

where Ch _i is the checked channel, and R is the linear combination of the remaining channels, and <...> denotes the temporal average. An example of a linear combination R of the remaining channels is their energy-normalized sum. In addition, the channel level difference (CLD _i ) (channel level difference) is usually the decibel value of the parameter σ _i .

Referring to the aforementioned equations, the channel level difference (CLD _i ) or parameter σ _i may correspond to the level P _i of the channel Ch _i normalized to the level _PR of the linear combination R of the remaining channels. _Here , the level P _i or P _R can _be obtained _from the linear combination ICLD _R Calculated in the guide.

Here, ICLD _i and ICLD _j are respectively related to a reference channel Ch _ref . In an additional embodiment, the inter-channel level difference parameters ICLD _i and ICLD _j may also be related to any other channel of the multi-channel audio signal (Ch ₁ . . . Ch _N ) belonging to the reference channel Ch _ref . In this way, it will eventually lead to the same result of channel level difference (CLD _i ) and parameter σ _i .

According to other embodiments, the inter-channel relationship 335 of Fig. 3b can also be derived by operating on different or all pairs of Ch _i , Ch _j input channels of a multi-channel audio signal (Ch ₁ ... Ch _N ) . In this case, the inter-channel coherence parameter ICC _{i, j} or channel level difference (CLD _{i, j} ) or parameter σ _ij (or ICLD _{i, j} ), index (i, j) represent a certain pair of channels Ch _i and Ch _j respectively.

FIG. 4 shows a block diagram of an embodiment 400 of a direct/surround extractor 420 that includes downmixing of estimated level information 113 . The embodiment of FIG. 4 contains substantially the same blocks as the embodiment of FIG. 1 . Accordingly, identical blocks with similar implementation and/or functionality are marked with the same reference numerals. However, the direct/surround extractor 420 of FIG. 4 corresponding to the direct/surround extractor 120 of FIG. 1 is configured to downmix the estimated level information 113 of the direct part or the surrounding part of the multi-channel audio signal to obtain The downmixed level information of the direct part or the surrounding part, and extracting the direct signal part 125 - 1 or the surrounding signal part 125 - 2 from the downmixed signal 115 based on the downmixed level information. As shown in Fig. 4, the spatial parameter information 105 can be derived from the multi-channel (Ch ₁ ... Ch _N ) audio signal 101 in Fig. 1, for example, and can include the inter-channel relationship of Ch ₁ ... Ch _N introduced in Fig. 3b Formula 335. The spatial parameter information 105 of FIG. 4 also contains downmix information 410 to be fed to the direct/surround extractor 420 . In an embodiment, downmix information 410 may characterize a downmix of an original multi-channel audio signal (eg, multi-channel audio signal 101 of FIG. 1 ) into downmix signal 115 . Downmixing can eg be performed using a downmixer (not shown in the figure) operating in any coding domain, eg time domain or frequency domain.

According to other embodiments, the direct/surrounding extractor 420 is further configured to perform Downmixing of the estimated level information 113 of the immediate part or the surrounding part of the multi-channel audio signal 101 .

It should be noted that the estimated level information may represent the energy level or power level of the immediate part or the surrounding part, respectively.

More specifically, downmixing of the estimated direct/surrounding energy (ie level information 113 ) can be performed by assuming full incoherence or full coherence between channels. In the case of performing downmixing based on incoherence sum or coherence sum respectively, the following two formulas can be applied.

For incoherent signals, the downmixed energy or downmixed level information can be given by ${\mathrm{E.}}_{\mathrm{DMX}}=\underset{i=1}{\overset{N}{\mathrm{\&Sigma;}}}{g}_i^2{\mathrm{E.}}_{{\mathrm{Ch}}_i}$ calculate.

For coherent signals, the downmixed energy or downmixed level information can be given by ${\mathrm{E.}}_{\mathrm{DMX}}={\left(\underset{i=1}{\overset{N}{\mathrm{\&Sigma;}}}{g}_i\sqrt{{\mathrm{E.}}_{{\mathrm{Ch}}_i}}\right)}^2$ calculate.

Here, g is the downmix gain, which can be obtained from the downmix information, and E(Ch _i ) represents the energy of the immediate/surrounding part of a channel _Chi in the multi-channel audio signal. As for the typical example of incoherent downmixing, in the case of downmixing 5.1 channels into two channels, the energy of the left downmixing can be:

E _{L_DMX} =E _Left +E _{Left_surround} +0.5*E _Center

FIG. 5 shows a further embodiment of a direct/surround extractor 520 by applying gain parameters g _D , g _A to the downmix signal 115 . The direct/surround extractor 520 of FIG. 5 may correspond to the direct/surround extractor 420 of FIG. 4 . First, estimated level information for either the direct portion 545-1 or the surrounding portion 545-2 may be received from a direct/surrounding estimator as previously described. The received level information 545-1, 545-2 may be combined/downmixed in step 550 to obtain downmixed level information for the immediate portion 555-1 or surrounding portion 555-2, respectively. Then in step 560, gain parameters g _D 565-1 , g _A 565-2 may be derived from the downmix level information 555-1 , 555-2 for the immediate part or surrounding part, respectively. Finally, the direct/surround extractor 520 can be used to apply the derived gain parameters 565-1, 565-3 to the downmix signal 115 (step 570), so that either the direct signal portion 125-1 or the ambient portion 125-2 will be obtained.

Here, it should be noted that in the embodiments of FIGS. 1 , 4 , and 5 , the downmix signal 115 may be composed of a plurality of downmix channels (Ch ₁ ... Ch _N ) composition.

In other embodiments, the direct/ambient extractor 520 is configured to determine the direct-to-total (DTT) energy ratio or the ambient-to-total (ATT ) energy ratio, and use the extracted parameters based on the measured DTT energy ratio or ATT energy ratio as gain parameters 565-1, 565-2.

In yet other embodiments, the direct/surround extractor 520 is configured to multiply the downmix signal 115 with the first extraction parameter sqrt(DTT) to obtain the direct signal portion 125-1, and multiply the downmix signal 115 with the second extraction parameter sqrt(ATT ) are multiplied to obtain the surrounding signal portion 125-2. Here, the downmix signal 115 may correspond to the mono downmix signal 215 , as shown in the embodiment of FIG. 2 (“mono downmix case”).

In the case of a mono downmix, surround extraction can be performed by applying sqrt(ATT) and sqrt(DTT). But to be more specific, by applying sqrt(ATT _i ) and sqrt(DTT _i ) to each channel Ch _i , the same method is also effective for a multi-channel downmix signal.

According to other embodiments, in the case where the downmix signal 115 contains multiple channels (“mono downmix case”), the direct/surround extractor 520 may be configured to apply a first plurality of extraction parameters such as sqrt( DTT _i ) to the downmix signal 115 to obtain the direct signal portion 125 - 1 , and apply a second plurality of extraction parameters such as sqrt(ATT _i ) to the downmix signal 115 to obtain the ambient signal portion 125 - 2 . Here, the first plurality of extraction parameters and the second plurality of extraction parameters may form a diagonal matrix.

In general, the direct/surround extractor 120, 420, 520 can also be configured to extract the direct signal portion 125-1 or the ambient signal portion 125-2 by applying a square M×M extraction matrix to the downmix signal 115, wherein, The size (M) of the square M×M extraction matrix corresponds to the number (M) of the downmix channels (Ch ₁ . . . Ch _N ).

Thus, applying ambient extraction can be described as applying a square M×M extraction matrix, where M is the number of downmix channels (Ch ₁ . . . Ch _N ). This can include all possible ways to manipulate the input signal to obtain direct/surrounding output, including sqrt(ATT _i ) and sqrt(DTT _i ) parameters, or the LMS hybrid approach structured as a full matrix. The latter will be explained later, and here, it should be noted that the aforementioned method of applying an M×M extraction matrix covers any number of channels, including one.

According to other embodiments, the extraction matrix may not necessarily be a square matrix of MxM matrix size because the inventors have a smaller number of output channels. Therefore, the extraction matrix has reduced rows. An example of this could be to extract a single direct signal instead of M.

It is also not necessary to always take all M downmix channels as input to M columns with extraction matrices. More specifically, it may be relevant to the application use that it is not necessary here to have all channels as input signals.

Fig. 6 shows a block diagram of yet another embodiment 600 of a direct/surround extractor 620 based on an LMS (least mean square) solution using channel mixing. The direct/surround extractor 620 of FIG. 6 may correspond to the direct/surround extractor 120 of FIG. 1 . In the embodiment of FIG. 6 , the same blocks having similar implementations and/or functions as those of the embodiment of FIG. 1 are therefore denoted by the same component symbols. But the downmix signal 615 of FIG. 6 corresponding to the downmix signal 115 of FIG. ₁ comprises a plurality 617 of downmix channels Ch _{1 .} The number of channels Ch ₁ ... Ch _N (N), that is, M<N. More specifically, direct/surround extractor 620 is configured to extract either direct signal portion 125-1 or ambient signal portion 125-2 by using a least mean square (LMS) solution of channel mixing, which does not require equal to the surrounding level. Such an LMS solution is provided as follows, which does not require equal ambient levels, and is also extendable to any number of channels. The LMS solution just mentioned is not mandatory, but represents a more precise alternative to the previous method.

The component notation used for the LMS solution of blended weights for direct/surrounding extraction is:

Ch _i channel i

α _i direct sound gain in channel i

声音的直接部分及其估值D and

The immediate part of the sound and its valuation

A _i and Surrounding part of channel i and its evaluation

P _X = E[XX*] Estimated energy of X

E[] expected value

X的估算误差

X's estimation error

声道i对直接部分的LMS交混权值

The LMS blending weight of channel i to the direct part

声道n对声道i的周围部分的LMS交混权值

The LMS blending weight of channel n to the surrounding part of channel i

In this context, it is noted that the derivation of the LMS solution may be based on the spectral representation of each channel of a multi-channel audio signal, which represents each function in the frequency band.

The signal model is expressed as

Ch _i =a _i D+A _i

The derivation first deals with a) the immediate part, then, b) the surrounding part. Finally, a solution to the weights is derived and a method for normalizing the weights is described.

a) direct part

The direct part of the weight is estimated as

$\stackrel{<span\; class="notranslate">\; ^</span>}{\mathrm{<span\; class="notranslate">\; D.</span>}}<span\; class="notranslate">\; =</span>\underset{\mathrm{<span\; class="notranslate">\; i</span>}<span\; class="notranslate">\; =</span>\mathrm{<span\; class="notranslate">\; 1</span>}}{\overset{\mathrm{<span\; class="notranslate">\; N</span>}}{\mathrm{<span\; class="notranslate">\; \&Sigma;</span>}}}{\mathrm{<span\; class="notranslate">\; \&omega;</span>}}_{\stackrel{<span\; class="notranslate">\; ^</span>}{\mathrm{<span\; class="notranslate">\; D.</span>}}\mathrm{<span\; class="notranslate">\; i</span>}}{\mathrm{<span\; class="notranslate">\; Ch</span>}}_{\mathrm{<span\; class="notranslate">\; i</span>}}<span\; class="notranslate">\; =</span>\underset{\mathrm{<span\; class="notranslate">\; i</span>}<span\; class="notranslate">\; =</span>\mathrm{<span\; class="notranslate">\; 1</span>}}{\overset{\mathrm{<span\; class="notranslate">\; N</span>}}{\mathrm{<span\; class="notranslate">\; \&Sigma;</span>}}}{\mathrm{<span\; class="notranslate">\; \&omega;</span>}}_{\stackrel{<span\; class="notranslate">\; ^</span>}{\mathrm{<span\; class="notranslate">\; D.</span>}}\mathrm{<span\; class="notranslate">\; i</span>}}<span\; class="notranslate">\; (</span>{\mathrm{<span\; class="notranslate">\; a</span>}}_{\mathrm{<span\; class="notranslate">\; i</span>}}\mathrm{<span\; class="notranslate">\; D.</span>}<span\; class="notranslate">\; +</span>{\mathrm{<span\; class="notranslate">\; A</span>}}_{\mathrm{<span\; class="notranslate">\; i</span>}}<span\; class="notranslate">\; )</span>$

Estimate Error Read

${\mathrm{<span\; class="notranslate">\; E.</span>}}_{\stackrel{<span\; class="notranslate">\; ^</span>}{\mathrm{<span\; class="notranslate">\; D.</span>}}}<span\; class="notranslate">\; =</span>\mathrm{<span\; class="notranslate">\; D.</span>}<span\; class="notranslate">\; -</span>\stackrel{<span\; class="notranslate">\; ^</span>}{\mathrm{<span\; class="notranslate">\; D.</span>}}<span\; class="notranslate">\; =</span>\mathrm{<span\; class="notranslate">\; D.</span>}<span\; class="notranslate">\; -</span>\underset{\mathrm{<span\; class="notranslate">\; i</span>}<span\; class="notranslate">\; =</span>\mathrm{<span\; class="notranslate">\; 1</span>}}{\overset{\mathrm{<span\; class="notranslate">\; N</span>}}{\mathrm{<span\; class="notranslate">\; \&Sigma;</span>}}}{\mathrm{<span\; class="notranslate">\; \&omega;</span>}}_{\mathrm{<span\; class="notranslate">\; \&sigma;i</span>}}<span\; class="notranslate">\; (</span>{\mathrm{<span\; class="notranslate">\; a</span>}}_{\mathrm{<span\; class="notranslate">\; i</span>}}<span\; class="notranslate">\; +</span>{\mathrm{<span\; class="notranslate">\; A</span>}}_{\mathrm{<span\; class="notranslate">\; i</span>}}<span\; class="notranslate">\; )</span>$

与输入信号正交To obtain the LMS solution, the inventors require

Orthogonal to the input signal

E[E _σ Ch _i ]=0, for all k

$\mathrm{<span\; class="notranslate">\; E.</span>}<span\; class="notranslate">\; [</span><span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; D.</span>}<span\; class="notranslate">\; -</span>\underset{\mathrm{<span\; class="notranslate">\; i</span>}<span\; class="notranslate">\; =</span>\mathrm{<span\; class="notranslate">\; 1</span>}}{\overset{\mathrm{<span\; class="notranslate">\; N</span>}}{\mathrm{<span\; class="notranslate">\; \&Sigma;</span>}}}{\mathrm{<span\; class="notranslate">\; w</span>}}_{\stackrel{<span\; class="notranslate">\; ^</span>}{\mathrm{<span\; class="notranslate">\; D.</span>}}\mathrm{<span\; class="notranslate">\; i</span>}}<span\; class="notranslate">\; (</span>{\mathrm{<span\; class="notranslate">\; a</span>}}_{\mathrm{<span\; class="notranslate">\; i</span>}}\mathrm{<span\; class="notranslate">\; D.</span>}<span\; class="notranslate">\; +</span>{\mathrm{<span\; class="notranslate">\; A</span>}}_{\mathrm{<span\; class="notranslate">\; i</span>}}<span\; class="notranslate">\; )</span><span\; class="notranslate">\; )</span>{<span\; class="notranslate">\; (</span>{\mathrm{<span\; class="notranslate">\; a</span>}}_{\mathrm{<span\; class="notranslate">\; k</span>}}\mathrm{<span\; class="notranslate">\; D.</span>}<span\; class="notranslate">\; +</span>{\mathrm{<span\; class="notranslate">\; A</span>}}_{\mathrm{<span\; class="notranslate">\; k</span>}}<span\; class="notranslate">\; )</span>}^{<span\; class="notranslate">\; *</span>}<span\; class="notranslate">\; ]</span>$

$<span\; class="notranslate">\; =</span><span\; class="notranslate">\; (</span>{\mathrm{<span\; class="notranslate">\; a</span>}}_{\mathrm{<span\; class="notranslate">\; k</span>}}<span\; class="notranslate">\; -</span>\underset{\mathrm{<span\; class="notranslate">\; i</span>}<span\; class="notranslate">\; =</span>\mathrm{<span\; class="notranslate">\; 1</span>}}{\overset{\mathrm{<span\; class="notranslate">\; N</span>}}{\mathrm{<span\; class="notranslate">\; \&Sigma;</span>}}}{\mathrm{<span\; class="notranslate">\; w</span>}}_{\stackrel{<span\; class="notranslate">\; ^</span>}{\mathrm{<span\; class="notranslate">\; D.</span>}}\mathrm{<span\; class="notranslate">\; i</span>}}{\mathrm{<span\; class="notranslate">\; a</span>}}_{\mathrm{<span\; class="notranslate">\; i</span>}}{\mathrm{<span\; class="notranslate">\; a</span>}}_{\mathrm{<span\; class="notranslate">\; k</span>}}<span\; class="notranslate">\; )</span>{\mathrm{<span\; class="notranslate">\; P</span>}}_{\mathrm{<span\; class="notranslate">\; D.</span>}}<span\; class="notranslate">\; -</span>{\mathrm{<span\; class="notranslate">\; w</span>}}_{\stackrel{<span\; class="notranslate">\; ^</span>}{\mathrm{<span\; class="notranslate">\; D.</span>}}\mathrm{<span\; class="notranslate">\; k</span>}}{\mathrm{<span\; class="notranslate">\; P</span>}}_{\mathrm{<span\; class="notranslate">\; Ak</span>}}<span\; class="notranslate">\; =</span>\mathrm{<span\; class="notranslate">\; 0</span>}$

$<span\; class="notranslate">\; \&DoubleLeftRightArrow;</span>\underset{\mathrm{<span\; class="notranslate">\; i</span>}<span\; class="notranslate">\; =</span>\mathrm{<span\; class="notranslate">\; 1</span>}}{\overset{\mathrm{<span\; class="notranslate">\; N</span>}}{\mathrm{<span\; class="notranslate">\; \&Sigma;</span>}}}{\mathrm{<span\; class="notranslate">\; w</span>}}_{\stackrel{<span\; class="notranslate">\; ^</span>}{\mathrm{<span\; class="notranslate">\; D.</span>}}\mathrm{<span\; class="notranslate">\; i</span>}}{\mathrm{<span\; class="notranslate">\; a</span>}}_{\mathrm{<span\; class="notranslate">\; i</span>}}{\mathrm{<span\; class="notranslate">\; a</span>}}_{\mathrm{<span\; class="notranslate">\; k</span>}}{\mathrm{<span\; class="notranslate">\; P</span>}}_{\mathrm{<span\; class="notranslate">\; D.</span>}}<span\; class="notranslate">\; +</span>{\mathrm{<span\; class="notranslate">\; w</span>}}_{\stackrel{<span\; class="notranslate">\; ^</span>}{\mathrm{<span\; class="notranslate">\; D.</span>}}\mathrm{<span\; class="notranslate">\; k</span>}}{\mathrm{<span\; class="notranslate">\; P</span>}}_{\mathrm{<span\; class="notranslate">\; AK</span>}}<span\; class="notranslate">\; =</span>{\mathrm{<span\; class="notranslate">\; a</span>}}_{\mathrm{<span\; class="notranslate">\; k</span>}}{\mathrm{<span\; class="notranslate">\; P</span>}}_{\mathrm{<span\; class="notranslate">\; D.</span>}}$

In matrix form, the aforementioned relation is read as

b) Surrounding part

The inventors start from the same signal model and estimate the weights according to

$\stackrel{<span\; class="notranslate">\; ^</span>}{{\mathrm{<span\; class="notranslate">\; A</span>}}_{\mathrm{<span\; class="notranslate">\; i</span>}}}<span\; class="notranslate">\; =</span>\underset{\mathrm{<span\; class="notranslate">\; no</span>}<span\; class="notranslate">\; =</span>\mathrm{<span\; class="notranslate">\; 1</span>}}{\overset{\mathrm{<span\; class="notranslate">\; N</span>}}{\mathrm{<span\; class="notranslate">\; \&Sigma;</span>}}}{\mathrm{<span\; class="notranslate">\; w</span>}}_{\stackrel{<span\; class="notranslate">\; ^</span>}{\mathrm{<span\; class="notranslate">\; A</span>}}\mathrm{<span\; class="notranslate">\; i</span>}<span\; class="notranslate">\; ,</span>\mathrm{<span\; class="notranslate">\; no</span>}}{\mathrm{<span\; class="notranslate">\; Ch</span>}}_{\mathrm{<span\; class="notranslate">\; i</span>}}\underset{\mathrm{<span\; class="notranslate">\; no</span>}<span\; class="notranslate">\; =</span>\mathrm{<span\; class="notranslate">\; 1</span>}}{\overset{\mathrm{<span\; class="notranslate">\; N</span>}}{\mathrm{<span\; class="notranslate">\; \&Sigma;</span>}}}{\mathrm{<span\; class="notranslate">\; w</span>}}_{\stackrel{<span\; class="notranslate">\; ^</span>}{\mathrm{<span\; class="notranslate">\; A</span>}}\mathrm{<span\; class="notranslate">\; i</span>}<span\; class="notranslate">\; ,</span>\mathrm{<span\; class="notranslate">\; no</span>}}<span\; class="notranslate">\; (</span>{\mathrm{<span\; class="notranslate">\; a</span>}}_{\mathrm{<span\; class="notranslate">\; i</span>}}\mathrm{<span\; class="notranslate">\; D.</span>}<span\; class="notranslate">\; +</span>{\mathrm{<span\; class="notranslate">\; A</span>}}_{\mathrm{<span\; class="notranslate">\; i</span>}}<span\; class="notranslate">\; )</span>$

The estimated error is

${\mathrm{<span\; class="notranslate">\; E.</span>}}_{\stackrel{<span\; class="notranslate">\; ^</span>}{\mathrm{<span\; class="notranslate">\; A</span>}}\mathrm{<span\; class="notranslate">\; i</span>}}<span\; class="notranslate">\; =</span>{\mathrm{<span\; class="notranslate">\; A</span>}}_{\mathrm{<span\; class="notranslate">\; i</span>}}<span\; class="notranslate">\; -</span>{\stackrel{<span\; class="notranslate">\; ^</span>}{\mathrm{<span\; class="notranslate">\; A</span>}}}_{\mathrm{<span\; class="notranslate">\; i</span>}}<span\; class="notranslate">\; =</span>{\mathrm{<span\; class="notranslate">\; A</span>}}_{\mathrm{<span\; class="notranslate">\; i</span>}}<span\; class="notranslate">\; -</span>\underset{\mathrm{<span\; class="notranslate">\; no</span>}<span\; class="notranslate">\; =</span>\mathrm{<span\; class="notranslate">\; 1</span>}}{\overset{\mathrm{<span\; class="notranslate">\; N</span>}}{\mathrm{<span\; class="notranslate">\; \&Sigma;</span>}}}{\mathrm{<span\; class="notranslate">\; w</span>}}_{\stackrel{<span\; class="notranslate">\; ^</span>}{\mathrm{<span\; class="notranslate">\; A</span>}}\mathrm{<span\; class="notranslate">\; i</span>}<span\; class="notranslate">\; ,</span>\mathrm{<span\; class="notranslate">\; no</span>}}<span\; class="notranslate">\; (</span>{\mathrm{<span\; class="notranslate">\; a</span>}}_{\mathrm{<span\; class="notranslate">\; i</span>}}\mathrm{<span\; class="notranslate">\; D.</span>}<span\; class="notranslate">\; +</span>{\mathrm{<span\; class="notranslate">\; A</span>}}_{\mathrm{<span\; class="notranslate">\; i</span>}}<span\; class="notranslate">\; )</span>$

and orthogonality

$\mathrm{E.}\left[{\mathrm{E.}}_{\hat{A}i}{\mathrm{Ch}}_k\right]=0,$ for all k

In matrix form, the preceding relation reads as

solution of weights

The weights can be solved by inverting the matrix A, which is the same for the calculation of the immediate part and the surrounding part. In the stereo case, the solution is:

${\mathrm{<span\; class="notranslate">\; w</span>}}_{\mathrm{<span\; class="notranslate">\; D.</span>}\mathrm{<span\; class="notranslate">\; 1</span>}}<span\; class="notranslate">\; =</span>\frac{{\mathrm{<span\; class="notranslate">\; a</span>}}_{\mathrm{<span\; class="notranslate">\; 1</span>}}{\mathrm{<span\; class="notranslate">\; P</span>}}_{\mathrm{<span\; class="notranslate">\; D.</span>}}{\mathrm{<span\; class="notranslate">\; P</span>}}_{\mathrm{<span\; class="notranslate">\; A</span>}\mathrm{<span\; class="notranslate">\; 2</span>}}}{{\mathrm{<span\; class="notranslate">\; a</span>}}_{\mathrm{<span\; class="notranslate">\; 2</span>}}{\mathrm{<span\; class="notranslate">\; a</span>}}_{\mathrm{<span\; class="notranslate">\; 2</span>}}{\mathrm{<span\; class="notranslate">\; P</span>}}_{\mathrm{<span\; class="notranslate">\; D.</span>}}{\mathrm{<span\; class="notranslate">\; P</span>}}_{\mathrm{<span\; class="notranslate">\; A</span>}\mathrm{<span\; class="notranslate">\; 1</span>}}<span\; class="notranslate">\; +</span>{\mathrm{<span\; class="notranslate">\; a</span>}}_{\mathrm{<span\; class="notranslate">\; 1</span>}}{\mathrm{<span\; class="notranslate">\; a</span>}}_{\mathrm{<span\; class="notranslate">\; 1</span>}}{\mathrm{<span\; class="notranslate">\; P</span>}}_{\mathrm{<span\; class="notranslate">\; D.</span>}}{\mathrm{<span\; class="notranslate">\; P</span>}}_{\mathrm{<span\; class="notranslate">\; A</span>}\mathrm{<span\; class="notranslate">\; 2</span>}}<span\; class="notranslate">\; +</span>{\mathrm{<span\; class="notranslate">\; P</span>}}_{\mathrm{<span\; class="notranslate">\; A</span>}\mathrm{<span\; class="notranslate">\; 1</span>}}{\mathrm{<span\; class="notranslate">\; P</span>}}_{\mathrm{<span\; class="notranslate">\; A</span>}\mathrm{<span\; class="notranslate">\; 2</span>}}}<span\; class="notranslate">\; =</span>\frac{{\mathrm{<span\; class="notranslate">\; a</span>}}_{\mathrm{<span\; class="notranslate">\; 1</span>}}{\mathrm{<span\; class="notranslate">\; P</span>}}_{\mathrm{<span\; class="notranslate">\; D.</span>}}{\mathrm{<span\; class="notranslate">\; P</span>}}_{\mathrm{<span\; class="notranslate">\; A</span>}\mathrm{<span\; class="notranslate">\; 2</span>}}}{\mathrm{<span\; class="notranslate">\; div</span>}}$

${\mathrm{<span\; class="notranslate">\; w</span>}}_{\mathrm{<span\; class="notranslate">\; D.</span>}\mathrm{<span\; class="notranslate">\; 2</span>}}<span\; class="notranslate">\; =</span>\frac{{\mathrm{<span\; class="notranslate">\; a</span>}}_{\mathrm{<span\; class="notranslate">\; 2</span>}}{\mathrm{<span\; class="notranslate">\; P</span>}}_{\mathrm{<span\; class="notranslate">\; D.</span>}}{\mathrm{<span\; class="notranslate">\; P</span>}}_{\mathrm{<span\; class="notranslate">\; A</span>}\mathrm{<span\; class="notranslate">\; 1</span>}}}{\mathrm{<span\; class="notranslate">\; div</span>}}$

${\mathrm{<span\; class="notranslate">\; w</span>}}_{\stackrel{<span\; class="notranslate">\; ^</span>}{\mathrm{<span\; class="notranslate">\; A</span>}}\mathrm{<span\; class="notranslate">\; 1,1</span>}}<span\; class="notranslate">\; =</span>\frac{{\mathrm{<span\; class="notranslate">\; a</span>}}_{\mathrm{<span\; class="notranslate">\; 2</span>}}{\mathrm{<span\; class="notranslate">\; a</span>}}_{\mathrm{<span\; class="notranslate">\; 2</span>}}{\mathrm{<span\; class="notranslate">\; P</span>}}_{\mathrm{<span\; class="notranslate">\; D.</span>}}{\mathrm{<span\; class="notranslate">\; P</span>}}_{\mathrm{<span\; class="notranslate">\; A</span>}\mathrm{<span\; class="notranslate">\; 1</span>}}<span\; class="notranslate">\; +</span>{\mathrm{<span\; class="notranslate">\; P</span>}}_{\mathrm{<span\; class="notranslate">\; A</span>}\mathrm{<span\; class="notranslate">\; 1</span>}}{\mathrm{<span\; class="notranslate">\; P</span>}}_{\mathrm{<span\; class="notranslate">\; A</span>}\mathrm{<span\; class="notranslate">\; 2</span>}}}{\mathrm{<span\; class="notranslate">\; div</span>}}$

${\mathrm{<span\; class="notranslate">\; w</span>}}_{\stackrel{<span\; class="notranslate">\; ^</span>}{\mathrm{<span\; class="notranslate">\; A</span>}}\mathrm{<span\; class="notranslate">\; 1,2</span>}}<span\; class="notranslate">\; =</span>\frac{{\mathrm{<span\; class="notranslate">\; a</span>}}_{\mathrm{<span\; class="notranslate">\; 1</span>}}{\mathrm{<span\; class="notranslate">\; a</span>}}_{\mathrm{<span\; class="notranslate">\; 2</span>}}{\mathrm{<span\; class="notranslate">\; P</span>}}_{\mathrm{<span\; class="notranslate">\; D.</span>}}{\mathrm{<span\; class="notranslate">\; P</span>}}_{\mathrm{<span\; class="notranslate">\; A</span>}\mathrm{<span\; class="notranslate">\; 1</span>}}}{\mathrm{<span\; class="notranslate">\; div</span>}}$

${\mathrm{<span\; class="notranslate">\; w</span>}}_{\stackrel{<span\; class="notranslate">\; ^</span>}{\mathrm{<span\; class="notranslate">\; A</span>}}\mathrm{<span\; class="notranslate">\; 2,1</span>}}<span\; class="notranslate">\; =</span>\frac{{\mathrm{<span\; class="notranslate">\; a</span>}}_{\mathrm{<span\; class="notranslate">\; 1</span>}}{\mathrm{<span\; class="notranslate">\; a</span>}}_{\mathrm{<span\; class="notranslate">\; 2</span>}}{\mathrm{<span\; class="notranslate">\; P</span>}}_{\mathrm{<span\; class="notranslate">\; D.</span>}}{\mathrm{<span\; class="notranslate">\; P</span>}}_{\mathrm{<span\; class="notranslate">\; A</span>}\mathrm{<span\; class="notranslate">\; 2</span>}}}{\mathrm{<span\; class="notranslate">\; div</span>}}$

${\mathrm{<span\; class="notranslate">\; w</span>}}_{\stackrel{<span\; class="notranslate">\; ^</span>}{\mathrm{<span\; class="notranslate">\; A</span>}}\mathrm{<span\; class="notranslate">\; 2,2</span>}}<span\; class="notranslate">\; =</span>\frac{{\mathrm{<span\; class="notranslate">\; a</span>}}_{\mathrm{<span\; class="notranslate">\; 1</span>}}{\mathrm{<span\; class="notranslate">\; a</span>}}_{\mathrm{<span\; class="notranslate">\; 1</span>}}{\mathrm{<span\; class="notranslate">\; P</span>}}_{\mathrm{<span\; class="notranslate">\; D.</span>}}{\mathrm{<span\; class="notranslate">\; P</span>}}_{\mathrm{<span\; class="notranslate">\; A</span>}\mathrm{<span\; class="notranslate">\; 2</span>}}<span\; class="notranslate">\; +</span>{\mathrm{<span\; class="notranslate">\; P</span>}}_{\mathrm{<span\; class="notranslate">\; A</span>}\mathrm{<span\; class="notranslate">\; 1</span>}}{\mathrm{<span\; class="notranslate">\; P</span>}}_{\mathrm{<span\; class="notranslate">\; A</span>}\mathrm{<span\; class="notranslate">\; 2</span>}}}{\mathrm{<span\; class="notranslate">\; div</span>}}$

Here, div is the divisor a ₂ a ₂ P _D P _A1 +a ₁ a ₁ P _D P _A2 +P _A1 P _A2 .

Standardization of weights

The weights are used for the LMS solution, but since the energy level must be preserved, the weights are standardized. This also makes the division by the div term in the above formula unnecessary. Normalization is done by ensuring that the output direct and surrounding channels are P _D and P _Ai , where i is the channel index.

其为从第一输入声道及第二输入声道产生第一周围声道的增益。步骤如下：It is straightforward to assume that the inventor knows the inter-channel coherence, mixing factor and channel energy. For simplicity, the inventors focus on the case of two channels, and specifically for a pair of weights and

It is the gain to generate the first ambient channel from the first input channel and the second input channel. Proceed as follows:

Step 1: Calculate the output signal energy (where the coherent part sums amplitude by amplitude and the non-coherent part sums energy by energy)

${\mathrm{<span\; class="notranslate">\; P</span>}}_{\stackrel{<span\; class="notranslate">\; ^</span>}{\mathrm{<span\; class="notranslate">\; A</span>}}\mathrm{<span\; class="notranslate">\; 1</span>}}<span\; class="notranslate">\; =</span>{<span\; class="notranslate">\; (</span>{\mathrm{<span\; class="notranslate">\; w</span>}}_{\stackrel{<span\; class="notranslate">\; ^</span>}{\mathrm{<span\; class="notranslate">\; A</span>}}\mathrm{<span\; class="notranslate">\; 1,1</span>}}\sqrt{<span\; class="notranslate">\; |</span>\mathrm{<span\; class="notranslate">\; ICC</span>}<span\; class="notranslate">\; |</span><span\; class="notranslate">\; \&CenterDot;</span>{\mathrm{<span\; class="notranslate">\; P</span>}}_{\mathrm{<span\; class="notranslate">\; 1</span>}}}<span\; class="notranslate">\; +</span>\mathrm{<span\; class="notranslate">\; sign</span>}<span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; ICC</span>}<span\; class="notranslate">\; )</span>{\mathrm{<span\; class="notranslate">\; w</span>}}_{\stackrel{<span\; class="notranslate">\; ^</span>}{\mathrm{<span\; class="notranslate">\; A</span>}}\mathrm{<span\; class="notranslate">\; 1,2</span>}}\sqrt{<span\; class="notranslate">\; |</span>\mathrm{<span\; class="notranslate">\; ICC</span>}<span\; class="notranslate">\; |</span><span\; class="notranslate">\; \&Center\; Dot;</span>{\mathrm{<span\; class="notranslate">\; P</span>}}_{\mathrm{<span\; class="notranslate">\; 2</span>}}}<span\; class="notranslate">\; )</span>}^{\mathrm{<span\; class="notranslate">\; 2</span>}}<span\; class="notranslate">\; +</span><span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; 1</span>}<span\; class="notranslate">\; -</span><span\; class="notranslate">\; |</span>\mathrm{<span\; class="notranslate">\; ICC</span>}<span\; class="notranslate">\; |</span><span\; class="notranslate">\; )</span>{\mathrm{<span\; class="notranslate">\; P</span>}}_{\mathrm{<span\; class="notranslate">\; 1</span>}}{\mathrm{<span\; class="notranslate">\; w</span>}}_{\stackrel{<span\; class="notranslate">\; ^</span>}{\mathrm{<span\; class="notranslate">\; A</span>}}\mathrm{<span\; class="notranslate">\; 1,1</span>}}^{\mathrm{<span\; class="notranslate">\; 2</span>}}<span\; class="notranslate">\; +</span><span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; 1</span>}<span\; class="notranslate">\; -</span><span\; class="notranslate">\; |</span>\mathrm{<span\; class="notranslate">\; ICC</span>}<span\; class="notranslate">\; |</span><span\; class="notranslate">\; )</span>{\mathrm{<span\; class="notranslate">\; P</span>}}_{\mathrm{<span\; class="notranslate">\; 2</span>}}{\mathrm{<span\; class="notranslate">\; w</span>}}_{\stackrel{<span\; class="notranslate">\; ^</span>}{\mathrm{<span\; class="notranslate">\; A</span>}}\mathrm{<span\; class="notranslate">\; 1,2</span>}}^{\mathrm{<span\; class="notranslate">\; 2</span>}}$

Step 2: Calculate Normalized Gain Factor

$\mathrm{<span\; class="notranslate">\; g</span>}<span\; class="notranslate">\; =</span>\sqrt{\frac{{\mathrm{<span\; class="notranslate">\; P</span>}}_{\mathrm{<span\; class="notranslate">\; A</span>}\mathrm{<span\; class="notranslate">\; 1</span>}}}{{\mathrm{<span\; class="notranslate">\; P</span>}}_{\stackrel{<span\; class="notranslate">\; ^</span>}{\mathrm{<span\; class="notranslate">\; A</span>}}\mathrm{<span\; class="notranslate">\; 1</span>}}}}$

在步骤1中，ICC的绝对值和符号操作数被包括为也考虑输入声道为负相干的情况。其余权值因子也以相同方式被标准化。and apply this result to the blend weight factor and

In step 1, the absolute value and sign operands of ICC are included to also consider the case where the input channels are negatively coherent. The remaining weight factors are also normalized in the same way.

More specifically, referring to the foregoing description, the direct/surround extractor 620 can be configured to derive the LMS solution by assuming a stable multi-channel signal model, so that the LMS solution is not limited to the stereo channel downmix signal.

Fig. 7a shows a block diagram of an embodiment 700 of a direct/ambient estimator 710, which is based on a stereo ambient estimation formula. The direct/surround estimator 710 of FIG. 7a may correspond to the direct/surround estimator 110 of FIG. 1 . More specifically, direct/ambient estimator 710 of FIG _. The formula can be expressed as a functional dependency as

DTT _i =f _DTT [σ(Ch _i′ R) _′ ICC _i (Ch _i′ R)]

ATT _i ＝1-DTT _i

It explicitly shows the dependence on the channel level difference (CLD _i ) or parameter σ _i of the channel Ch _i and the inter-channel coherence (ICC _i ) parameter. As shown in Fig. 7a, the spatial parameter information 105 is fed to the direct/surrounding estimator 710 and may include inter-channel relational parameters ICC _i and σ _i for each channel Ch _i . After applying this stereo ambient estimation formula by using the direct/ambient estimator 710, a direct-to-total (DTT _i ) or ambient-to-total (ATT _i ) energy ratio will be obtained at its output 715 , respectively. It should be noted that the aforementioned stereo surround estimation formulas for estimating each DTT energy ratio or ATT energy ratio are not based on the condition of equal ambient.

More specifically, direct/surround ratio estimation is performed in such a way that the ratio of the direct energy of a channel to the total energy of that channel (DTT) can be formulated as

及

Ch为检查声道，并且R为其余声道的线性组合。<>为时间平均值。当声道及其余声道的线性组合的周围位准假设为相等，并且其相干性为零时遵照此一公式。here,

and

Ch is the check channel, and R is a linear combination of the remaining channels. <> is the time average. This formula is followed when the ambient levels of a channel and a linear combination of other channels are assumed to be equal and their coherence is zero.

Figure 7b shows a line graph 750 of an example DTT (direct-to-total) energy ratio 760 as a function of the inter-channel coherence parameter ICC 770 . In the embodiment of Fig. 7b, the channel level difference (CLD) or the parameter σ is set to 1 (σ=1), for example, so that the linear combination of the level P(Ch _i ) of the channel Ch _i and the remaining channels R levels P(R) will be equal. In this case, the DTT energy ratio 760 will be linearly proportional to the ICC parameter, as indicated by the straight line 775 labeled DTT~ICC. From Fig. 7b, in the case of ICC=0, which may correspond to a fully decoherent inter-channel relationship, the DTT power ratio 760 will be 0, which may correspond to a fully ambient case (case "R ₁ "). But in the case of ICC=1, which may correspond to a perfectly coherent inter-channel relationship, the DTT energy ratio 760 will be 1, which may correspond to a completely direct case (case “R ₂ ”). Thus, there is substantially no direct energy in case _R1 and substantially no ambient energy in case _R2 relative to the total energy in the vocal tract.

FIG. 8 shows a block diagram of an encoder/decoder system 800 according to other embodiments of the present invention. On the decoder side of the encoder/decoder system 800, an embodiment of a decoder 820 is shown, which may correspond to the device 100 of FIG. 1 . Due to the similarity between the embodiments of Fig. 1 and Fig. 8, the same blocks with similar implementations and/or functions in these two embodiments are denoted by the same component symbols. As shown in the embodiment of FIG. 8 , the direct/surround extractor 120 may operate on the downmix signal 115 having a plurality of downmix channels Ch ₁ . . . Ch _M . The direct/surrounding estimator 110 of FIG. 8 is further configured to receive at least two downmix channels 825 of the downmix signal 815 (optional), such that the level information 113 of the direct part or the surrounding part of the multi-channel audio signal 110 will be The estimation is based on the received spatial parameter information 105 of at least two downmix channels 825 . Finally, after extraction by the direct/surround extractor 120, either the direct signal portion 125-1 or the ambient signal portion 125-2 will be obtained.

On the encoder side of the encoder/decoder system 800, an embodiment of an encoder 810 is shown, which may include a downmixer 815 for downmixing multi-channel audio signals (Ch ₁ ...Ch _N ) to The downmix signal 115 has a number of downmix channels Ch ₁ . . . Ch _M , where the number of channels is reduced from N to M. The down-mixer 815 may also be configured to output the spatial parameter information 105 by calculating an inter-channel relationship from the multi-channel audio signal 101 . In the encoder/decoder system 800 of FIG. 8 , the downmix signal 115 and the spatial parameter information 105 may be transmitted from the encoder 810 to the decoder 820 . Here, the encoder 810 may derive an encoded signal based on the downmix signal 115 and the spatial parameter information 105 for transmission from the encoder end to the decoder end. Furthermore, the spatial parameter information 105 is based on channel information of the multi-channel audio signal 101 .

On the other hand, the inter-channel relationship parameters σ _i (Ch _i ,R) and ICC _i (Ch _i ,R) can be calculated between the linear combination R of the channel Ch _i of the encoder 810 and the remaining channels, and in Internal transmission of coded signals. The decoder 820 may in turn receive the encoded signal and operate on the transmitted inter-channel relationship parameters σ _i (Ch _i ,R) and ICC _i (Ch _i ,R).

On the other hand, the encoder 810 can also be configured to calculate the inter-channel coherence parameter ICC _i,j between different pairs of channels (Ch _i , Ch _j ) to be transmitted. In this case, the encoder 810 should be able to derive the parameter ICC i between the channel Ch _i and the linear combination R of the remaining channels from the transmitted pairwise calculated ICC _i,j (Ch _{i ,} Ch _j ) (Ch _i , R), such that the corresponding embodiments already described above are realized. It should be noted in this context that the decoder 820 cannot reconstruct the parameters ICC _i (Ch _i ,R) from the knowledge of the downmix signal 115 alone.

In an embodiment, the transmitted spatial parameters are not only about pair-wise channel comparisons.

For example, the most typical MPS situation is to have two downmix channels. The first set of spatial parameters in MPS coding makes two channels into three channels: center, left and right. The set of parameters that guide this mapping is called the Central Prediction Coefficient (CPC) and the ICC parameters that are specific for two-to-three configurations.

The second set of spatial parameters is split in two: the side channels are split into corresponding front and rear channels, while the center channel is split into center and Lfe channels. This mapping relationship is related to the ICC and CLD parameters introduced above.

It is not practical to find calculation rules for all downmix configuration classes and all kinds of spatial parameter classes. However, it is practical to follow the downmixing steps virtually. The reason is that the inventor knows that two channels become three channels, and three channels become six channels. Finally, the inventor finds out how to arrange the path of two input channels to become the input-output relational expression of six output channels. The output signal is just the linear combination of the downmix channels plus the linear combination of their decorrelated versions. It is not necessary to actually decode the output signal and measure it, but knowing such a "decoding matrix" the inventors can computationally efficiently calculate the ICC and CLD parameters for any channel or combination of channels in the parameter domain.

Independently of the downmix signal configuration and the multi-channel signal configuration, each output of the decoded signal is a linear combination of the downmix signal plus a linear combination of their respective decorrelated versions.

${\mathrm{<span\; class="notranslate">\; Ch</span>}<span\; class="notranslate">\; \_</span>\mathrm{<span\; class="notranslate">\; out</span>}}_{\mathrm{<span\; class="notranslate">\; i</span>}}<span\; class="notranslate">\; =</span>\underset{\mathrm{<span\; class="notranslate">\; k</span>}<span\; class="notranslate">\; =</span>\mathrm{<span\; class="notranslate">\; 1</span>}}{\overset{\mathrm{<span\; class="notranslate">\; dmx</span>}<span\; class="notranslate">\; \_</span>\mathrm{<span\; class="notranslate">\; channels</span>}}{\mathrm{<span\; class="notranslate">\; \&Sigma;</span>}}}<span\; class="notranslate">\; (</span>{\mathrm{<span\; class="notranslate">\; a</span>}}_{\mathrm{<span\; class="notranslate">\; k</span>}<span\; class="notranslate">\; ,</span>\mathrm{<span\; class="notranslate">\; i</span>}}\mathrm{<span\; class="notranslate">\; Ch</span>}<span\; class="notranslate">\; \_</span>{\mathrm{<span\; class="notranslate">\; dmx</span>}}_{\mathrm{<span\; class="notranslate">\; k</span>}}<span\; class="notranslate">\; +</span>{\mathrm{<span\; class="notranslate">\; b</span>}}_{\mathrm{<span\; class="notranslate">\; k</span>}<span\; class="notranslate">\; ,</span>\mathrm{<span\; class="notranslate">\; i</span>}}\mathrm{<span\; class="notranslate">\; D.</span>}<span\; class="notranslate">\; [</span>\mathrm{<span\; class="notranslate">\; Ch</span>}<span\; class="notranslate">\; \_</span>{\mathrm{<span\; class="notranslate">\; dmx</span>}}_{\mathrm{<span\; class="notranslate">\; k</span>}}<span\; class="notranslate">\; ]</span><span\; class="notranslate">\; )</span>$

where the operand D[] corresponds to a decorrelator, that is, a process that makes an incoherent replica of the input signal. The factors a and b are known since they can be directly derived from the parametric side information. Because by definition, the parameter information instructs the decoder how to form the multi-channel output signal from the downmix signal. The above formula can be simplified to

${\mathrm{<span\; class="notranslate">\; Ch</span>}<span\; class="notranslate">\; \_</span>\mathrm{<span\; class="notranslate">\; out</span>}}_{\mathrm{<span\; class="notranslate">\; i</span>}}<span\; class="notranslate">\; =</span>\underset{\mathrm{<span\; class="notranslate">\; k</span>}<span\; class="notranslate">\; =</span>\mathrm{<span\; class="notranslate">\; 1</span>}}{\overset{\mathrm{<span\; class="notranslate">\; dmx</span>}<span\; class="notranslate">\; \_</span>\mathrm{<span\; class="notranslate">\; channels</span>}}{\mathrm{<span\; class="notranslate">\; \&Sigma;</span>}}}<span\; class="notranslate">\; (</span>{\mathrm{<span\; class="notranslate">\; a</span>}}_{\mathrm{<span\; class="notranslate">\; k</span>}<span\; class="notranslate">\; ,</span>\mathrm{<span\; class="notranslate">\; i</span>}}{\mathrm{<span\; class="notranslate">\; Ch</span>}<span\; class="notranslate">\; \_</span>\mathrm{<span\; class="notranslate">\; dmx</span>}}_{\mathrm{<span\; class="notranslate">\; k</span>}}<span\; class="notranslate">\; )</span><span\; class="notranslate">\; +</span>{\mathrm{<span\; class="notranslate">\; D.</span>}}_{\mathrm{<span\; class="notranslate">\; i</span>}}$

The reason is that all decorrelation parts can be combined for energy/coherence comparison. The energy of D is known because the factor b is also known in the first equation.

In light of this, it should be noted that the inventors can make any kind of coherence and energy comparisons between output channels or between different linear combinations of output channels. In the simple case of two downmix channels and a set of output channels, channel numbers 3 and 5 are compared against each other and the sum is calculated as follows:

${\mathrm{<span\; class="notranslate">\; \&sigma;</span>}}_{\mathrm{<span\; class="notranslate">\; 3,5</span>}}<span\; class="notranslate">\; =</span>\frac{\mathrm{<span\; class="notranslate">\; E.</span>}<span\; class="notranslate">\; [</span>{\mathrm{<span\; class="notranslate">\; Ch</span>}<span\; class="notranslate">\; \_</span>\mathrm{<span\; class="notranslate">\; out</span>}}_{\mathrm{<span\; class="notranslate">\; 3</span>}}^{\mathrm{<span\; class="notranslate">\; 2</span>}}<span\; class="notranslate">\; ]</span>}{\mathrm{<span\; class="notranslate">\; E.</span>}<span\; class="notranslate">\; [</span>\mathrm{<span\; class="notranslate">\; Ch</span>}<span\; class="notranslate">\; \_</span>{\mathrm{<span\; class="notranslate">\; out</span>}}_{\mathrm{<span\; class="notranslate">\; 5</span>}}^{\mathrm{<span\; class="notranslate">\; 2</span>}}<span\; class="notranslate">\; ]</span>}$

where E[] is the expected (actually: average) number of operands. The two terms can be expressed as follows

All the aforementioned parameters are known or measurable from the downmix signal. The cross term E[Ch_dmx*D] is defined as zero and thus the lower column in the formula. Similarly, the coherence formula is

${\mathrm{<span\; class="notranslate">\; ICC</span>}}_{\mathrm{<span\; class="notranslate">\; 3,5</span>}}<span\; class="notranslate">\; =</span>\frac{\mathrm{<span\; class="notranslate">\; E.</span>}<span\; class="notranslate">\; [</span>{\mathrm{<span\; class="notranslate">\; Ch</span>}<span\; class="notranslate">\; \_</span>\mathrm{<span\; class="notranslate">\; out</span>}}_{\mathrm{<span\; class="notranslate">\; 3</span>}}{\mathrm{<span\; class="notranslate">\; Ch</span>}<span\; class="notranslate">\; \_</span>\mathrm{<span\; class="notranslate">\; out</span>}}_{\mathrm{<span\; class="notranslate">\; 5</span>}}<span\; class="notranslate">\; ]</span>}{\sqrt{\mathrm{<span\; class="notranslate">\; E.</span>}<span\; class="notranslate">\; [</span>\mathrm{<span\; class="notranslate">\; Ch</span>}<span\; class="notranslate">\; \_</span>{\mathrm{<span\; class="notranslate">\; out</span>}}_{\mathrm{<span\; class="notranslate">\; 3</span>}}^{\mathrm{<span\; class="notranslate">\; 2</span>}}<span\; class="notranslate">\; ]</span>\mathrm{<span\; class="notranslate">\; E.</span>}<span\; class="notranslate">\; [</span>\mathrm{<span\; class="notranslate">\; Ch</span>}<span\; class="notranslate">\; \_</span>{\mathrm{<span\; class="notranslate">\; out</span>}}_{\mathrm{<span\; class="notranslate">\; 5</span>}}^{\mathrm{<span\; class="notranslate">\; 2</span>}}<span\; class="notranslate">\; ]</span>}}$

Furthermore, since all parts in the above formula are linear combinations of the input signal plus the decorrelation signal, the solution can be obtained directly.

The above example compares two output channels, but comparisons between linear combinations of output channels can similarly be performed, such as using the example handlers described in detail later.

Combining the aforementioned previous embodiments, the presented technique/idea includes the following steps:

1. Obtain inter-channel relationships (coherence, level) for a "original" set of channels that may be higher than the number of downmix channels.

2. Estimate the ambient and direct energies of the "old" channel set.

3. Downmix the ambient and direct energies of the "original" channel set to a smaller number of channels.

4. The downmix energy is used to extract the direct and surrounding signals in the provided downmix channel by applying gain factors or gain matrices.

The use of spatial parameter side information is best explained and summarized by the embodiment of FIG. 2 . In the embodiment of FIG. 2, the inventors have a parametric stereo stream that includes a single audio channel and spatial side information about the inter-channel differences (coherence, level) of the stereo it represents. Now since the inventor knows the inter-channel difference, the above stereo ambient estimation formula can be applied to the inter-channel difference, and the direct energy and the ambient energy of the original channel set are known. The inventors can then "downmix" the channel energy by summing the direct energy (using coherent addition) and the ambient energy (using incoherent addition), and derive the direct-to-total energy ratio and The surrounding pairs can always be compared.

Referring to the embodiment in Fig. 2, the spatial parameter information generally includes inter-channel coherence parameters (ICC _L , ICC _R ) and channel level difference parameters (CLLD _L , _CLDR ), which are respectively related to the left Channel (L) and right channel (R) correspond. Here, it should be noted that the inter-channel coherence parameters ICC _L and ICC _R are equal (ICC _L =ICC _R ), and the channel level difference parameters CLD _L and CLD _R are related by CLD _L = _-CLDR . Correspondingly, the channel level difference parameters CLD _L and CLD _R are typically the decibel values of the parameters σ _L and σ _R respectively, so the parameters _σ L and σ _R of the left (L) and right (R) channels pass through σ _L = 1/σ _R and related. These inter-channel difference parameters can be easily used to calculate the individual direct-to-total energy ratios (DTT _L , DTT _R ) and ambient-to-total energy ratios (ATT _L ,ATT _R ). In this stereo surround estimation formula, the direct-to-total-energy ratio and the ambient-to-total-energy ratio (DTT _L , ATT _L ) of the left channel (L) depend on the inter-channel difference parameters (CLD _L , ICC _L ), while the direct to total energy ratio and ambient to total energy ratio (DTT _R , ATTR ) of the right channel ( _R ) depend on the inter-channel difference parameters ( _CLDR , ICC _R ) of the right channel R. In addition, the energy (E _L , E _R ) of the two channels L and R of the parametric stereo audio signal can be based on the channel level difference parameters (CLD _L , _CLDR ) to derive the calculation. Here, the energy of the left channel L (E _L ) can be obtained by applying the channel level difference parameter (CLLD _L ) of the left channel L to the mono downmix signal, while the energy of the right channel R ( E _R ) can be obtained by applying the channel level difference parameter (CLDR) of the right channel _R to the mono downmix signal. Then by multiplying the energy (E _L , E _R ) of the two-channel (L, R) with the corresponding parameters based on DTT _L , DTT _R , and ATT _L , ATT _R , the two-channel (L ,R) direct energy (E _DL , E _DR ) and surrounding energy (E _AL , E _AR ). The direct energies (E _DL , E _DR ) of the two channels (L,R) can then be combined/added using the coherent downmixing law to obtain the downmix energy (E _{D ,mono} ); while the ambient energies (E _AL ,E _AR ) of the two-channel (L,R) can be combined/added using incoherent downmixing rules to obtain the downmix of the ambient part of the mono downmix signal Can (E _{A, mono} ). Then, by finding the relationship between the downmix energy (E _D,mono ,EA _,mono ) of the direct signal part and the surrounding signal part and the total energy (E _mono ) of the mono downmix signal, the Direct to Total Energy Ratio (DTT _mono ) and Ambient to Total Energy Ratio (ATT _Mono ) of the mono downmix signal. Finally, based on these DTT _mono energy ratios and ATT _mono energy ratios, either direct signal parts or ambient signal parts can roughly be extracted from the mono downmix signal.

In audio reproduction, it is often necessary to reproduce sound through headphones. Headphone listening has unique characteristics that make it very different from speaker listening and also from any natural sound environment. Audio is set directly to the left and right ear. The reproduced audio content is typically reproduced for speaker playback. Therefore, audio signals do not contain the properties and cues that the human auditory system uses for spatial sound perception. This is the case unless binaural processing is imported into the system.

Basically, binaural processing can be called a processing procedure that takes an input sound and modifies it so that the sound contains only those perceptually correct (in terms of the human auditory system processing spatial sound) these interaural and monaural properties . Binaural processing does not work directly and is still not optimal according to state-of-the-art existing solutions.

There are numerous applications where binaural processing of audio and movie playback has been included, such as media players and processing devices designed to transform multi-channel audio signals into binaural counterparts for headphones. The typical approach is to use head-related transfer functions (HRTF) to make virtual headphones and add room effects to this signal. In theory, this could be the equivalent of listening with headphones in a special room.

However, it has been shown repeatedly in practice that this approach has not consistently satisfied listeners. There seems to be a tradeoff, good spatialization using this direct approach sacrifices audio quality, such as poor timbre or timbre changes, annoying perception of room effects, and loss of dynamics. Other problems include inaccurate localization (eg, intrahead positioning, front-to-back confusion), lack of spatial distance to sound sources, and inter-aural mismatch, that is, hearing close to the ear due to wrong interaural cues.

Different listeners respond to adjudicated questions very differently. Sensitivity also varies depending on the input material, such as music (strict quality standards in terms of timbre), movies (less stringent) and games (even less stringent, but positioning is important). There are also typically different design purposes depending on the content.

Therefore, the details that follow deal as successfully as possible with ways of overcoming the aforementioned problems to maximize the average perceptual overall quality.

Fig. 9a shows a block diagram of an overview 900 of a binaural direct sound presentation device 910 according to other embodiments of the present invention. As shown in FIG. 9a, the binaural direct sound rendering device 910 is configured for processing the direct signal portion 125-1 which may be present at the output of the direct/surround extractor 120 of the embodiment of FIG. 1 to obtain a first binaural Ear output signal 915. The first binaural output signal 915 may include a left channel indicated by L and a right channel indicated by R.

Here, the binaural direct sound rendering device 910 may be configured to feed the direct signal portion 125-1 through a head related transfer function (HRTF) to obtain a transformed direct signal portion. Furthermore, the binaural direct sound rendering device 910 may be configured to apply a room effect to the transformed direct signal portion to finally obtain the first binaural output signal 915 .

Fig. 9b shows a block diagram of a detail 905 of the binaural direct sound rendering device 910 of Fig. 9a. The binaural direct sound rendering device 910 may comprise a "HRTF transformer" indicated at block 912 and a room effects processing device (early reflection parallel reverberation or analog) at block 914 . As shown in Figure 9b, the HRTF converter 912 and the room effect processing device 914 can operate on the direct signal part 125-1 by applying the head-related transfer function (HRTF) and the room effect in parallel, thereby obtaining the first dual Ear output signal 915.

More specifically, referring to Fig. 9b, such room effects processing may also provide an incoherent reverberant direct signal 919, which may be processed by a subsequent cross filter 920 to adapt this signal to the interaural coherence of the diffuse sound field. Here, the filter 920 and the HRTF transformer 912 compose the first binaural output signal 915 . According to other embodiments, the processing of the direct sound by room effects may also be a parametric representation of early reflection.

Therefore, in an embodiment, the indoor effect may preferably be applied in parallel with the HRTF rather than in series (ie, the indoor effect is applied after the signal is fed through the HRTF). More specifically, only sound propagating directly from the source passes through or is transformed by the corresponding HRTF. Indirect/reverberant sound can be approximated, ie statistically (by employing coherence control instead of HRTF) to enter the ear. This can also be done serially, but a parallel approach is preferred.

Fig. 10a shows a block diagram of an overview 1000 of a binaural ambient sound presentation device 1010 according to other embodiments of the present invention. As shown in FIG. 10a, the binaural ambient sound rendering device 1010 may be configured to process the ambient signal portion 125-2 which may be present at the output of the direct/ambient extractor 120 of the embodiment of FIG. 1 to obtain a second binaural Ear output signal 1015. The second binaural output signal 1015 may include a left channel (L) and a right channel (R).

Fig. 10b shows a block diagram of a detail 1005 of the binaural ambient sound rendering device 1010 of Fig. 10a. As can be seen in Fig. 10b, the binaural ambient sound rendering device 1010 may be configured to apply a room effect to the ambient signal portion 125-2 as indicated by the box 1012 labeled "Room Effects Processing" such that an incoherent mix is obtained. Ring the surrounding signal 1013. Furthermore, the binaural ambient sound rendering device 1010 may be configured to process the incoherent reverberant ambient signal 1013 by applying a filter, such as a cross filter represented by block 1014, thereby providing a second binaural output signal 1015 , the second binaural output signal 1015 is adapted to the interaural coherence of the actual diffuse sound field. Block 1012 labeled "Room Effects Processing" may also be structured such that it directly produces the interaural coherence of the actual diffuse sound field. In this case, block 1014 is not used.

According to other embodiments, the binaural ambient sound rendering device 1010 is configured to apply room effects and/or filters to the ambient signal portion 125-2 for providing the second binaural output signal 1015, such that the second binaural output signal 1015 will Interaural coherence for practical diffuse sound fields.

In the foregoing embodiments, decorrelation and coherence control may be performed in two consecutive steps, but this is not necessary. It is also possible to achieve the same result in a single-step process without going through an intermediate incoherent signal finding formula. Both methods are equally valid.

FIG. 11 shows a conceptual block diagram of an embodiment 1100 of binaural reproduction of a multi-channel audio signal 101 . More specifically, the embodiment of Fig. 11 represents an apparatus for binaural reproduction of a multi-channel audio signal 101 comprising a first converter 1110 ("frequency conversion"), a separator 1120 ("direct-surround Separation"), binaural direct sound rendering 910 ("Direct source rendering"), binaural ambient sound rendering 1010 ("Ambient sound rendering"), combiner 1130 as indicated by "+" and second transducer 1140 ( "Inverse Frequency Transformation"). More specifically, the first transformer 1110 may be configured for transforming the multi-channel audio signal 101 into a spectral representation 1115 . Separator 1120 may be configured to extract direct signal portion 125 - 1 or ambient signal portion 125 - 2 from spectral representation 1115 . Here, the separator 1120 may correspond to the apparatus 100 of FIG. 1 , and particularly includes the direct/surrounding estimator 110 and the direct/surrounding extractor 120 of the embodiment of FIG. 1 . As explained before, the binaural direct sound rendering device 910 is operable on the direct signal portion 125 - 1 to obtain the first binaural output signal 915 . Correspondingly, the binaural ambient sound presentation device 1010 may operate on the ambient signal portion 125 - 2 to obtain the second binaural output signal 1015 . The combiner 1130 may be configured to combine the first binaural output signal 915 and the second binaural output signal 1015 to obtain a combined signal 1135 . Finally, a second transformer 1140 may be configured to transform the combined signal 1135 into the time domain to obtain a stereo output audio signal 1150 ("stereo output signal for headphones").

The frequency transform operation of the FIG. 11 embodiment illustrates system functionality in the frequency transform domain, which is the natural domain in auditory processing of spatial audio. If the system is used as an add-on on a system already functioning in the frequency translation domain, then the system itself does not necessarily have frequency translation.

The aforementioned direct/surround separation method can be subdivided into two distinct parts. In the direct/surrounding estimation part, the level and/or ratio of the immediate surrounding part is estimated based on a combination of signal models and properties of the audio signal. In the direct/surrounding extraction part, the known ratio and input signal can be used to form the direct output of the ambient signal.

Finally, Figure 12 shows a generalized block diagram of an embodiment 1200 including direct/surround estimation/extraction of binaural rendering situations. In particular, the embodiment 1200 of FIG. 12 may correspond to the embodiment 1100 of FIG. 11 . In embodiment 1200, however, details of separator 1120 of FIG. 11 are shown corresponding to blocks 110, 120 of the embodiment of FIG. 1, which include an estimation/extraction process based on spatial parameter information 105. Furthermore, in contrast to the embodiment 1100 of FIG. 11 , no transformation process between different domains is shown in the embodiment 1200 of FIG. 12 . The blocks of the embodiment 1200 also explicitly operate on the downmix signal 115 , which can be derived from the multi-channel audio signal 101 .

Fig. 13a shows a block diagram of an embodiment of an apparatus 1300 for extracting a direct/ambient signal from a mono downmix signal in the filter bank domain. As shown in Figure 13a, the device 1300 comprises an analysis filter bank 1310, a synthesis filter bank 1320 for the immediate part, and a synthesis filter bank 1322 for the surrounding part.

More specifically, the analysis filter bank 1310 of the apparatus 1300 may be implemented to perform a short-term Fourier transform (STFT), or may be configured, for example, as an analysis QMF filter bank, while the synthesis filter bank of the apparatus 1300 1310 may be implemented to perform an inverse short-term Fourier transform (ISTFT), or may be constructed, for example, as a bank of synthesized QMF filters.

Analysis filter bank 1310 is configured to receive mono downmix signal 1315, which may correspond to mono downmix signal 215 as shown in the embodiment of FIG. Transform into multiple filter bank sub-bands 1311 . As can be seen in Fig. 13a, a plurality of 1311 filter group sub-bands are respectively connected to a plurality of direct/surrounding extraction frames 1350, 1352, wherein the plurality of direct/surrounding extraction frames 1350, 1352 are configured to apply DTT _mono parameters or ATT _mono Parameters 1333, 1335 to filter bank subbands.

DTT _mono or ATT _mono based parameters 1333, 1335 may be provided by a DTT _mono , ATT _mono calculator 1330, as shown in FIG. 13b. More specifically, the DTT _mono ,ATT _mono calculator 1330 of FIG. 13b may be configured to calculate the DTT _mono ,ATT _mono energy ratio, or from the left The inter-channel coherence and channel level difference parameters (ICC _L , CLD _L , ICC _R , _CLDR ) of the channel and the right channel (L, R) are provided, and the derivation is based on DTT _mono or ATT _mono The parameters have been correspondingly as described above. Here, for a single filter group sub-band, the corresponding parameters 105 and DTT _mono or ATT _mono based parameters 1333, 1335 can be used. In this context, it is pointed out that these parameters are not constant with respect to frequency.

As a result of applying DTT _mono or ATT _mono based parameters 1333, 1335, a plurality of modified filter bank sub-bands 1353, 1355 respectively are available. Subsequently, the plurality of modified filter bank sub-bands 1353, 1355 are respectively fed to a synthesis filter bank 1320, 1322, which may be configured to synthesize the plurality of modified filter bank sub-bands 1353, 1355, Thereby the direct signal part 1325-1 or the ambient signal part 1325-2 of the mono downmix signal 1315 respectively is obtained. Here, the direct signal portion 1325-1 of FIG. 13a corresponds to the direct signal portion 125-1 of FIG. 2, and the ambient signal portion 1325-2 of FIG. 13a corresponds to the direct signal portion 125-2 of FIG.

Referring to FIG. 13b, a direct/surrounding extraction block ₁₃₈₀ of the plurality 1350, ₁₃₅₂ direct/surrounding extraction blocks of FIG. The multiplier 1360 may be configured to multiply a single filter bank (FB) subband 1301 of the plurality of filter bank subbands 1311 by a corresponding DTT _mono or ATT _mono based parameter 1333, 1335 such that multiple Modified single filter bank sub-band 1365 of filter bank sub-bands 1353 , 1355 . More specifically, where block 1380 belongs to multiple blocks 1350, direct/surrounding extraction block 1380 is structured to apply DTT _mono based parameters; whereas, where block 1380 belongs to multiple blocks 1352, it is structured to Apply ATT _mono based parameters. Furthermore, modifying a single filterbank sub-band 1365 may provide a corresponding synthesis filterbank 1320, 1322 for the immediate part or the surrounding part.

According to an embodiment, the spatial parameters and derived parameters are provided at a frequency resolution according to key frequency bands of the human auditory system (eg band 28), typically lower than the resolution of the filter bank.

Therefore, the direct/surrounding extraction according to the embodiment of Fig. 13a is substantially based on inter-channel coherence and channel level difference parameters calculated sub-band by sub-band (may correspond to the inter-channel relation parameter 335 of Fig. 3b) Instead, operate on different subbands of the filter bank domain.

FIG. 14 shows a schematic illustration of an example of an MPEG Surround coding scheme 1400 according to yet another embodiment of the invention. More specifically, the embodiment of FIG. 14 describes decoding from a stereo downmix signal 1410 into six output channels 1420 . Here, the signal marked "res" is the reverberation signal, which is a selective permutation of the decorrelated signal (obtained from the box marked "D"). According to the embodiment of FIG. 14, spatial parameter information or inter-channel relationship parameters (ICC, CLD) are transmitted within the MPS stream from an encoder, such as the encoder 810 in FIG. 8, to a decoder, such as the decoder in FIG. 8 820, can be used to generate decoding matrices 1430, 1440 denoted as "pre-decorrelator matrix M1" and "mixing matrix M2", respectively. Unique to the embodiment of Fig. 14 is that output channels 1420 (i.e. upmix channels) are generated from side channels (L, R) and center channel (C) (L, R, C 1435) by using mixing matrix M2 1440 L, LS, R, RS, C, LFE) are roughly determined by the spatial parameter information 1405, which may correspond to the spatial parameter information 105 in FIG. 1, including special inter-channel relationship parameters (ICC, CLD ).

Here, the left channel (L) is divided into corresponding output channels L, LS, the right channel (R) is divided into corresponding output channels R, RS, and the center channel (C) is divided into corresponding Output channels C, LFE, this division can be represented by a one-to-two (OTT) configuration of the respective input signals with corresponding ICC, CLD parameters.

In particular, an example of the MPEG surround decoding scheme 1400 corresponding to the "5-2-5 configuration" may include the following steps, for example. In a first step, spatial parameters or parametric side information can be deployed into decoding matrices 1430, 1440, which are shown in Fig. 14 according to the existing MPEG Surround standard. In a second step, the decoding matrices 1430, 1440 may be used to provide inter-channel information for the upmix channels 1420 in the parametric domain. In a third step, using the inter-channel information thus provided, the direct/ambient energy of each upmix channel can be calculated. In a fourth step, the direct/ambient energy thus obtained may be downmixed to a number of downmix channels 1410 . In a fifth step, the weights to be applied to the downmix channel 1410 are calculated.

Before going any further, it should be pointed out that the processing just described requires a measurement of

E[|L _dmx | ² ], E[|R _dmx | ² ].

which is the average power of the downmix channels, and

$\mathrm{<span\; class="notranslate">\; E.</span>}<span\; class="notranslate">\; [</span>{\mathrm{<span\; class="notranslate">\; L</span>}}_{\mathrm{<span\; class="notranslate">\; max</span>}}<span\; class="notranslate">\; ,</span>{\mathrm{<span\; class="notranslate">\; R</span>}}_{\mathrm{<span\; class="notranslate">\; dmx</span>}}^{<span\; class="notranslate">\; *</span>}<span\; class="notranslate">\; ]</span>$

It may be referred to as the cross-spectrum from the downmix channel. Here, the average power of the downmix channels is purposely called energy, since the term "average power" is not a commonly used term.

Expected operands indicated by square brackets may be replaced by time averaging, recursion, or non-recursion in practice. Energy and cross-spectrum are directly measurable from the downmix signal.

It should also be noted that the linear combination energy of two channels can be derived from the formula of channel energy, mixing factor, and cross spectrum (all in the parameter domain, here, no signal operation is required).

linear combination

Ch=aL _dmx +bR _dmx

Has the following energies:

$\mathrm{<span\; class="notranslate">\; E.</span>}<span\; class="notranslate">\; [</span>{<span\; class="notranslate">\; |</span>\mathrm{<span\; class="notranslate">\; Ch</span>}<span\; class="notranslate">\; |</span>}^{\mathrm{<span\; class="notranslate">\; 2</span>}}<span\; class="notranslate">\; ]</span><span\; class="notranslate">\; =</span>\mathrm{<span\; class="notranslate">\; E.</span>}<span\; class="notranslate">\; [</span>{<span\; class="notranslate">\; |</span>{\mathrm{<span\; class="notranslate">\; aL</span>}}_{\mathrm{<span\; class="notranslate">\; dmx</span>}}<span\; class="notranslate">\; +</span>\mathrm{<span\; class="notranslate">\; b</span>}{\mathrm{<span\; class="notranslate">\; R</span>}}_{\mathrm{<span\; class="notranslate">\; dmx</span>}}<span\; class="notranslate">\; |</span>}^{\mathrm{<span\; class="notranslate">\; 2</span>}}<span\; class="notranslate">\; ]</span><span\; class="notranslate">\; =</span>{\mathrm{<span\; class="notranslate">\; a</span>}}^{\mathrm{<span\; class="notranslate">\; 2</span>}}\mathrm{<span\; class="notranslate">\; E.</span>}<span\; class="notranslate">\; [</span>{<span\; class="notranslate">\; |</span>{\mathrm{<span\; class="notranslate">\; L</span>}}_{\mathrm{<span\; class="notranslate">\; dmx</span>}}<span\; class="notranslate">\; |</span>}^{\mathrm{<span\; class="notranslate">\; 2</span>}}<span\; class="notranslate">\; ]</span><span\; class="notranslate">\; +</span>{\mathrm{<span\; class="notranslate">\; b</span>}}^{\mathrm{<span\; class="notranslate">\; 2</span>}}\mathrm{<span\; class="notranslate">\; E.</span>}<span\; class="notranslate">\; [</span>{<span\; class="notranslate">\; |</span>{\mathrm{<span\; class="notranslate">\; R</span>}}_{\mathrm{<span\; class="notranslate">\; dmx</span>}}<span\; class="notranslate">\; |</span>}^{\mathrm{<span\; class="notranslate">\; 2</span>}}<span\; class="notranslate">\; ]</span><span\; class="notranslate">\; +</span>\mathrm{<span\; class="notranslate">\; ab</span>}<span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; E.</span>}<span\; class="notranslate">\; [</span>{\mathrm{<span\; class="notranslate">\; L</span>}}_{\mathrm{<span\; class="notranslate">\; dmx</span>}}{\mathrm{<span\; class="notranslate">\; R</span>}}_{\mathrm{<span\; class="notranslate">\; dmx</span>}}^{<span\; class="notranslate">\; *</span>}<span\; class="notranslate">\; ]</span><span\; class="notranslate">\; +</span>\mathrm{<span\; class="notranslate">\; E.</span>}<span\; class="notranslate">\; [</span>{\mathrm{<span\; class="notranslate">\; R</span>}}_{\mathrm{<span\; class="notranslate">\; dmx</span>}}{\mathrm{<span\; class="notranslate">\; L</span>}}_{\mathrm{<span\; class="notranslate">\; dmx</span>}}^{<span\; class="notranslate">\; *</span>}<span\; class="notranslate">\; ]</span><span\; class="notranslate">\; )</span>$

$<span\; class="notranslate">\; =</span>{\mathrm{<span\; class="notranslate">\; a</span>}}^{\mathrm{<span\; class="notranslate">\; 2</span>}}\mathrm{<span\; class="notranslate">\; E.</span>}<span\; class="notranslate">\; [</span>{<span\; class="notranslate">\; |</span>{\mathrm{<span\; class="notranslate">\; L</span>}}_{\mathrm{<span\; class="notranslate">\; dmx</span>}}<span\; class="notranslate">\; |</span>}^{\mathrm{<span\; class="notranslate">\; 2</span>}}<span\; class="notranslate">\; ]</span><span\; class="notranslate">\; +</span>{\mathrm{<span\; class="notranslate">\; b</span>}}^{\mathrm{<span\; class="notranslate">\; 2</span>}}\mathrm{<span\; class="notranslate">\; E.</span>}<span\; class="notranslate">\; [</span>{<span\; class="notranslate">\; |</span>{\mathrm{<span\; class="notranslate">\; R</span>}}_{\mathrm{<span\; class="notranslate">\; dmx</span>}}<span\; class="notranslate">\; |</span>}^{\mathrm{<span\; class="notranslate">\; 2</span>}}<span\; class="notranslate">\; ]</span><span\; class="notranslate">\; +</span>\mathrm{<span\; class="notranslate">\; 2</span>}\mathrm{<span\; class="notranslate">\; ab</span>}<span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; Re</span>}<span\; class="notranslate">\; \{</span>\mathrm{<span\; class="notranslate">\; E.</span>}<span\; class="notranslate">\; [</span>{\mathrm{<span\; class="notranslate">\; L</span>}}_{\mathrm{<span\; class="notranslate">\; dmx</span>}}{\mathrm{<span\; class="notranslate">\; R</span>}}_{\mathrm{<span\; class="notranslate">\; dmx</span>}}^{<span\; class="notranslate">\; *</span>}<span\; class="notranslate">\; ]</span><span\; class="notranslate">\; \}</span><span\; class="notranslate">\; )</span>$

The individual steps of the processing procedure (ie, the decoding scheme) are described below.

First step (spatial parameters of the mixing matrix)

As mentioned earlier, the M1 and M2 matrices are formed according to the MPEG Surround standard. The element in column a and row b of M1 is M1(a, b).

Second step (mixing matrix with energy and crossover spectrum of inter-channel information from downmixed to upmixed channels)

The inventor now has mixing matrices M1 and M2. The inventor needs to formulate how the output channels are created from the left downmix channel (L _dmx ) and the right downmix channel (R _dmx ). The inventors assumed the use of a decorrelator (Fig. 14, gray area). The decoding/upmixing of the MPS standard basically ends up providing the following formula for the total input/output relationship in the whole process:

L=a _L L _dmx +b _L R _dmx +c _L D ₁ [S ₁ ]+d _L D ₂ [S ₂ ]+e _L D ₃ [S ₃ ]

The previous section illustrates an example of a front left channel that has been upmixed. Other channels can derive formulas in the same way. The D component is a decorrelator, and a-e are weights that can be obtained from the M1 and M2 matrix entries.

Specifically, the factors a-e can be directly expressed in the formula according to the matrix entries:

${\mathrm{<span\; class="notranslate">\; a</span>}}_{\mathrm{<span\; class="notranslate">\; L</span>}}<span\; class="notranslate">\; =</span>\underset{\mathrm{<span\; class="notranslate">\; i</span>}<span\; class="notranslate">\; =</span>\mathrm{<span\; class="notranslate">\; 1</span>}}{\overset{\mathrm{<span\; class="notranslate">\; 3</span>}}{\mathrm{<span\; class="notranslate">\; \&Sigma;</span>}}}{\mathrm{<span\; class="notranslate">\; m</span>}\mathrm{<span\; class="notranslate">\; 1</span>}}_{\mathrm{<span\; class="notranslate">\; i</span>}<span\; class="notranslate">\; ,</span>\mathrm{<span\; class="notranslate">\; 1</span>}}{\mathrm{<span\; class="notranslate">\; m</span>}\mathrm{<span\; class="notranslate">\; 2</span>}}_{\mathrm{<span\; class="notranslate">\; 1</span>}<span\; class="notranslate">\; ,</span>\mathrm{<span\; class="notranslate">\; i</span>}}$

${\mathrm{<span\; class="notranslate">\; b</span>}}_{\mathrm{<span\; class="notranslate">\; L</span>}}<span\; class="notranslate">\; =</span>\underset{\mathrm{<span\; class="notranslate">\; i</span>}<span\; class="notranslate">\; =</span>\mathrm{<span\; class="notranslate">\; 1</span>}}{\overset{\mathrm{<span\; class="notranslate">\; 3</span>}}{\mathrm{<span\; class="notranslate">\; \&Sigma;</span>}}}{\mathrm{<span\; class="notranslate">\; m</span>}\mathrm{<span\; class="notranslate">\; 1</span>}}_{\mathrm{<span\; class="notranslate">\; i</span>}<span\; class="notranslate">\; ,</span>\mathrm{<span\; class="notranslate">\; 2</span>}}{\mathrm{<span\; class="notranslate">\; m</span>}\mathrm{<span\; class="notranslate">\; 2</span>}}_{\mathrm{<span\; class="notranslate">\; 1</span>}<span\; class="notranslate">\; ,</span>\mathrm{<span\; class="notranslate">\; i</span>}}$

c _L = M2 _1,4

d _L = M2 _1,5

e _L =M2 _1,6

and correspondingly for other channels.

The S signal is

S _n =M1 _n+3,1 L _dmx +M1 _n+3,2 R _dmx

These S signals are the inputs from the matrix on the left side of Fig. 14 to the decorrelator. the energy

E[|D[S _n ]| ² ]=E[|S _n | ² ]

can be calculated as explained above. The decorrelator does not affect this energy.

An intuitively motivated way to do multi-channel surround extraction is by comparing one channel to the sum of all other channels (note that this is only one option out of multiple). Now, for example considering the case of the vocal tract L, the rest of the vocal tract reads:

${\mathrm{<span\; class="notranslate">\; x</span>}}_{\mathrm{<span\; class="notranslate">\; L</span>}}<span\; class="notranslate">\; =</span>\underset{\mathrm{<span\; class="notranslate">\; Ch</span>}<span\; class="notranslate">\; =</span><span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; REST</span>}<span\; class="notranslate">\; )</span>}{\mathrm{<span\; class="notranslate">\; \&Sigma;</span>}}{\mathrm{<span\; class="notranslate">\; a</span>}}_{\mathrm{<span\; class="notranslate">\; Ch</span>}}{\mathrm{<span\; class="notranslate">\; L</span>}}_{\mathrm{<span\; class="notranslate">\; dmx</span>}}<span\; class="notranslate">\; +</span>\underset{\mathrm{<span\; class="notranslate">\; Ch</span>}<span\; class="notranslate">\; =</span><span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; REST</span>}<span\; class="notranslate">\; )</span>}{\mathrm{<span\; class="notranslate">\; \&Sigma;</span>}}{\mathrm{<span\; class="notranslate">\; b</span>}}_{\mathrm{<span\; class="notranslate">\; Ch</span>}}{\mathrm{<span\; class="notranslate">\; R</span>}}_{\mathrm{<span\; class="notranslate">\; dmx</span>}}<span\; class="notranslate">\; +</span>\underset{\mathrm{<span\; class="notranslate">\; Ch</span>}<span\; class="notranslate">\; =</span><span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; REST</span>}<span\; class="notranslate">\; )</span>}{\mathrm{<span\; class="notranslate">\; \&Sigma;</span>}}{\mathrm{<span\; class="notranslate">\; c</span>}}_{\mathrm{<span\; class="notranslate">\; Ch</span>}}{\mathrm{<span\; class="notranslate">\; D.</span>}}_{\mathrm{<span\; class="notranslate">\; 1</span>}}<span\; class="notranslate">\; [</span>{\mathrm{<span\; class="notranslate">\; S</span>}}_{\mathrm{<span\; class="notranslate">\; 1</span>}}<span\; class="notranslate">\; ]</span><span\; class="notranslate">\; +</span>\underset{\mathrm{<span\; class="notranslate">\; Ch</span>}<span\; class="notranslate">\; =</span><span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; REST</span>}<span\; class="notranslate">\; )</span>}{\mathrm{<span\; class="notranslate">\; \&Sigma;</span>}}{\mathrm{<span\; class="notranslate">\; d</span>}}_{\mathrm{<span\; class="notranslate">\; Ch</span>}}{\mathrm{<span\; class="notranslate">\; D.</span>}}_{\mathrm{<span\; class="notranslate">\; 2</span>}}<span\; class="notranslate">\; [</span>{\mathrm{<span\; class="notranslate">\; S</span>}}_{\mathrm{<span\; class="notranslate">\; 2</span>}}<span\; class="notranslate">\; ]</span><span\; class="notranslate">\; +</span>\underset{\mathrm{<span\; class="notranslate">\; Ch</span>}<span\; class="notranslate">\; =</span><span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; REST</span>}<span\; class="notranslate">\; )</span>}{\mathrm{<span\; class="notranslate">\; \&Sigma;</span>}}{\mathrm{<span\; class="notranslate">\; e</span>}}_{\mathrm{<span\; class="notranslate">\; Ch</span>}}{\mathrm{<span\; class="notranslate">\; D.</span>}}_{\mathrm{<span\; class="notranslate">\; 3</span>}}<span\; class="notranslate">\; [</span>{\mathrm{<span\; class="notranslate">\; S</span>}}_{\mathrm{<span\; class="notranslate">\; 3</span>}}<span\; class="notranslate">\; ]</span>$

The inventors used an "X" here because the use of an "R" for "the rest of the channels" might be confusing.

Then, the energy of the channel L is

$\mathrm{<span\; class="notranslate">\; E.</span>}<span\; class="notranslate">\; [</span>{<span\; class="notranslate">\; |</span>\mathrm{<span\; class="notranslate">\; L</span>}<span\; class="notranslate">\; |</span>}^{\mathrm{<span\; class="notranslate">\; 2</span>}}<span\; class="notranslate">\; ]</span><span\; class="notranslate">\; =</span>{\mathrm{<span\; class="notranslate">\; a</span>}}_{\mathrm{<span\; class="notranslate">\; L</span>}}^{\mathrm{<span\; class="notranslate">\; 2</span>}}\mathrm{<span\; class="notranslate">\; E.</span>}<span\; class="notranslate">\; [</span>{<span\; class="notranslate">\; |</span>{\mathrm{<span\; class="notranslate">\; L</span>}}_{\mathrm{<span\; class="notranslate">\; dmx</span>}}<span\; class="notranslate">\; |</span>}^{\mathrm{<span\; class="notranslate">\; 2</span>}}<span\; class="notranslate">\; ]</span><span\; class="notranslate">\; +</span>{\mathrm{<span\; class="notranslate">\; b</span>}}_{\mathrm{<span\; class="notranslate">\; L</span>}}^{\mathrm{<span\; class="notranslate">\; 2</span>}}\mathrm{<span\; class="notranslate">\; E.</span>}<span\; class="notranslate">\; [</span>{<span\; class="notranslate">\; |</span>{\mathrm{<span\; class="notranslate">\; R</span>}}_{\mathrm{<span\; class="notranslate">\; dmx</span>}}<span\; class="notranslate">\; |</span>}^{\mathrm{<span\; class="notranslate">\; 2</span>}}<span\; class="notranslate">\; ]</span><span\; class="notranslate">\; +</span>{\mathrm{<span\; class="notranslate">\; c</span>}}_{\mathrm{<span\; class="notranslate">\; L</span>}}^{\mathrm{<span\; class="notranslate">\; 2</span>}}\mathrm{<span\; class="notranslate">\; E.</span>}<span\; class="notranslate">\; [</span>{<span\; class="notranslate">\; |</span>{\mathrm{<span\; class="notranslate">\; S</span>}}_{\mathrm{<span\; class="notranslate">\; 1</span>}}<span\; class="notranslate">\; |</span>}^{\mathrm{<span\; class="notranslate">\; 2</span>}}<span\; class="notranslate">\; ]</span><span\; class="notranslate">\; +</span>{\mathrm{<span\; class="notranslate">\; d</span>}}_{\mathrm{<span\; class="notranslate">\; L</span>}}^{\mathrm{<span\; class="notranslate">\; 2</span>}}\mathrm{<span\; class="notranslate">\; E.</span>}<span\; class="notranslate">\; [</span>{<span\; class="notranslate">\; |</span>{\mathrm{<span\; class="notranslate">\; S</span>}}_{\mathrm{<span\; class="notranslate">\; 2</span>}}<span\; class="notranslate">\; |</span>}^{\mathrm{<span\; class="notranslate">\; 2</span>}}<span\; class="notranslate">\; ]</span><span\; class="notranslate">\; +</span>{\mathrm{<span\; class="notranslate">\; e</span>}}_{\mathrm{<span\; class="notranslate">\; L</span>}}^{\mathrm{<span\; class="notranslate">\; 2</span>}}\mathrm{<span\; class="notranslate">\; E.</span>}<span\; class="notranslate">\; [</span>{<span\; class="notranslate">\; |</span>{\mathrm{<span\; class="notranslate">\; S</span>}}_{\mathrm{<span\; class="notranslate">\; 3</span>}}<span\; class="notranslate">\; |</span>}^{\mathrm{<span\; class="notranslate">\; 2</span>}}<span\; class="notranslate">\; ]</span><span\; class="notranslate">\; +</span>\mathrm{<span\; class="notranslate">\; 2</span>}\mathrm{<span\; class="notranslate">\; abRe</span>}<span\; class="notranslate">\; \{</span>\mathrm{<span\; class="notranslate">\; E.</span>}<span\; class="notranslate">\; [</span>{\mathrm{<span\; class="notranslate">\; L</span>}}_{\mathrm{<span\; class="notranslate">\; dmx</span>}}{\mathrm{<span\; class="notranslate">\; R</span>}}_{\mathrm{<span\; class="notranslate">\; dmx</span>}}^{<span\; class="notranslate">\; *</span>}<span\; class="notranslate">\; ]</span><span\; class="notranslate">\; \}</span>$

Then, the energy of channel X is

$\mathrm{<span\; class="notranslate">\; E.</span>}<span\; class="notranslate">\; [</span>{<span\; class="notranslate">\; |</span>{\mathrm{<span\; class="notranslate">\; x</span>}}_{\mathrm{<span\; class="notranslate">\; L</span>}}<span\; class="notranslate">\; |</span>}^{\mathrm{<span\; class="notranslate">\; 2</span>}}<span\; class="notranslate">\; ]</span><span\; class="notranslate">\; =</span>{<span\; class="notranslate">\; (</span>\underset{\mathrm{<span\; class="notranslate">\; Ch</span>}<span\; class="notranslate">\; =</span><span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; REST</span>}<span\; class="notranslate">\; )</span>}{\mathrm{<span\; class="notranslate">\; \&Sigma;</span>}}{\mathrm{<span\; class="notranslate">\; a</span>}}_{\mathrm{<span\; class="notranslate">\; Ch</span>}}<span\; class="notranslate">\; )</span>}^{\mathrm{<span\; class="notranslate">\; 2</span>}}\mathrm{<span\; class="notranslate">\; E.</span>}<span\; class="notranslate">\; [</span>{<span\; class="notranslate">\; |</span>{\mathrm{<span\; class="notranslate">\; L</span>}}_{\mathrm{<span\; class="notranslate">\; dmx</span>}}<span\; class="notranslate">\; |</span>}^{\mathrm{<span\; class="notranslate">\; 2</span>}}<span\; class="notranslate">\; ]</span><span\; class="notranslate">\; +</span>{<span\; class="notranslate">\; (</span>\underset{\mathrm{<span\; class="notranslate">\; Ch</span>}<span\; class="notranslate">\; =</span><span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; REST</span>}<span\; class="notranslate">\; )</span>}{\mathrm{<span\; class="notranslate">\; \&Sigma;</span>}}{\mathrm{<span\; class="notranslate">\; b</span>}}_{\mathrm{<span\; class="notranslate">\; Ch</span>}}<span\; class="notranslate">\; )</span>}^{\mathrm{<span\; class="notranslate">\; 2</span>}}\mathrm{<span\; class="notranslate">\; E.</span>}<span\; class="notranslate">\; [</span>{<span\; class="notranslate">\; |</span>{\mathrm{<span\; class="notranslate">\; R</span>}}_{\mathrm{<span\; class="notranslate">\; dmx</span>}}<span\; class="notranslate">\; |</span>}^{\mathrm{<span\; class="notranslate">\; 2</span>}}<span\; class="notranslate">\; ]</span><span\; class="notranslate">\; +</span>{<span\; class="notranslate">\; (</span>\underset{\mathrm{<span\; class="notranslate">\; Ch</span>}<span\; class="notranslate">\; =</span><span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; REST</span>}<span\; class="notranslate">\; )</span>}{\mathrm{<span\; class="notranslate">\; \&Sigma;</span>}}{\mathrm{<span\; class="notranslate">\; c</span>}}_{\mathrm{<span\; class="notranslate">\; Ch</span>}}<span\; class="notranslate">\; )</span>}^{\mathrm{<span\; class="notranslate">\; 2</span>}}\mathrm{<span\; class="notranslate">\; E.</span>}<span\; class="notranslate">\; [</span>{<span\; class="notranslate">\; |</span>{\mathrm{<span\; class="notranslate">\; S</span>}}_{\mathrm{<span\; class="notranslate">\; 1</span>}}<span\; class="notranslate">\; |</span>}^{\mathrm{<span\; class="notranslate">\; 2</span>}}<span\; class="notranslate">\; ]</span><span\; class="notranslate">\; +</span>{<span\; class="notranslate">\; (</span>\underset{\mathrm{<span\; class="notranslate">\; Ch</span>}<span\; class="notranslate">\; =</span><span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; REST</span>}<span\; class="notranslate">\; )</span>}{\mathrm{<span\; class="notranslate">\; \&Sigma;</span>}}{\mathrm{<span\; class="notranslate">\; d</span>}}_{\mathrm{<span\; class="notranslate">\; Ch</span>}}<span\; class="notranslate">\; )</span>}^{\mathrm{<span\; class="notranslate">\; 2</span>}}\mathrm{<span\; class="notranslate">\; E.</span>}<span\; class="notranslate">\; [</span>{<span\; class="notranslate">\; |</span>{\mathrm{<span\; class="notranslate">\; S</span>}}_{\mathrm{<span\; class="notranslate">\; 2</span>}}<span\; class="notranslate">\; |</span>}^{\mathrm{<span\; class="notranslate">\; 2</span>}}<span\; class="notranslate">\; ]</span>$

$<span\; class="notranslate">\; +</span>{<span\; class="notranslate">\; (</span>\underset{\mathrm{<span\; class="notranslate">\; Ch</span>}<span\; class="notranslate">\; =</span><span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; REST</span>}<span\; class="notranslate">\; )</span>}{\mathrm{<span\; class="notranslate">\; \&Sigma;</span>}}{\mathrm{<span\; class="notranslate">\; e</span>}}_{\mathrm{<span\; class="notranslate">\; Ch</span>}}<span\; class="notranslate">\; )</span>}^{\mathrm{<span\; class="notranslate">\; 2</span>}}\mathrm{<span\; class="notranslate">\; E.</span>}<span\; class="notranslate">\; [</span>{<span\; class="notranslate">\; |</span>{\mathrm{<span\; class="notranslate">\; S</span>}}_{\mathrm{<span\; class="notranslate">\; 3</span>}}<span\; class="notranslate">\; |</span>}^{\mathrm{<span\; class="notranslate">\; 2</span>}}<span\; class="notranslate">\; ]</span><span\; class="notranslate">\; +</span>\mathrm{<span\; class="notranslate">\; 2</span>}<span\; class="notranslate">\; (</span>\underset{\mathrm{<span\; class="notranslate">\; Ch</span>}<span\; class="notranslate">\; =</span><span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; REST</span>}<span\; class="notranslate">\; )</span>}{\mathrm{<span\; class="notranslate">\; \&Sigma;</span>}}{\mathrm{<span\; class="notranslate">\; a</span>}}_{\mathrm{<span\; class="notranslate">\; Ch</span>}}\underset{\mathrm{<span\; class="notranslate">\; Ch</span>}<span\; class="notranslate">\; =</span><span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; REST</span>}<span\; class="notranslate">\; )</span>}{\mathrm{<span\; class="notranslate">\; \&Sigma;</span>}}{\mathrm{<span\; class="notranslate">\; b</span>}}_{\mathrm{<span\; class="notranslate">\; Ch</span>}}<span\; class="notranslate">\; )</span>\mathrm{<span\; class="notranslate">\; Re</span>}<span\; class="notranslate">\; \{</span>\mathrm{<span\; class="notranslate">\; E.</span>}<span\; class="notranslate">\; [</span>{\mathrm{<span\; class="notranslate">\; L</span>}}_{\mathrm{<span\; class="notranslate">\; dmx</span>}}{\mathrm{<span\; class="notranslate">\; R</span>}}_{\mathrm{<span\; class="notranslate">\; dmx</span>}}^{<span\; class="notranslate">\; *</span>}<span\; class="notranslate">\; ]</span><span\; class="notranslate">\; \}</span>$

and the cross spectrum is:

$\mathrm{<span\; class="notranslate">\; E.</span>}<span\; class="notranslate">\; [</span>{\mathrm{<span\; class="notranslate">\; LX</span>}}_{\mathrm{<span\; class="notranslate">\; L</span>}}^{<span\; class="notranslate">\; *</span>}<span\; class="notranslate">\; ]</span><span\; class="notranslate">\; =</span>\underset{\mathrm{<span\; class="notranslate">\; Ch</span>}<span\; class="notranslate">\; =</span><span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; REST</span>}<span\; class="notranslate">\; )</span>}{\mathrm{<span\; class="notranslate">\; \&Sigma;</span>}}{\mathrm{<span\; class="notranslate">\; a</span>}}_{\mathrm{<span\; class="notranslate">\; Ch</span>}}{\mathrm{<span\; class="notranslate">\; a</span>}}_{\mathrm{<span\; class="notranslate">\; L</span>}}\mathrm{<span\; class="notranslate">\; E.</span>}<span\; class="notranslate">\; [</span>{<span\; class="notranslate">\; |</span>{\mathrm{<span\; class="notranslate">\; L</span>}}_{\mathrm{<span\; class="notranslate">\; dmx</span>}}<span\; class="notranslate">\; |</span>}^{\mathrm{<span\; class="notranslate">\; 2</span>}}<span\; class="notranslate">\; ]</span><span\; class="notranslate">\; +</span>\underset{\mathrm{<span\; class="notranslate">\; Ch</span>}<span\; class="notranslate">\; =</span><span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; REST</span>}<span\; class="notranslate">\; )</span>}{\mathrm{<span\; class="notranslate">\; \&Sigma;</span>}}{\mathrm{<span\; class="notranslate">\; b</span>}}_{\mathrm{<span\; class="notranslate">\; Ch</span>}}{\mathrm{<span\; class="notranslate">\; b</span>}}_{\mathrm{<span\; class="notranslate">\; L</span>}}\mathrm{<span\; class="notranslate">\; E.</span>}<span\; class="notranslate">\; [</span>{<span\; class="notranslate">\; |</span>{\mathrm{<span\; class="notranslate">\; R</span>}}_{\mathrm{<span\; class="notranslate">\; dmx</span>}}<span\; class="notranslate">\; |</span>}^{\mathrm{<span\; class="notranslate">\; 2</span>}}<span\; class="notranslate">\; ]</span><span\; class="notranslate">\; +</span>\underset{\mathrm{<span\; class="notranslate">\; Ch</span>}<span\; class="notranslate">\; =</span><span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; REST</span>}<span\; class="notranslate">\; )</span>}{\mathrm{<span\; class="notranslate">\; \&Sigma;</span>}}{\mathrm{<span\; class="notranslate">\; c</span>}}_{\mathrm{<span\; class="notranslate">\; Ch</span>}}{\mathrm{<span\; class="notranslate">\; c</span>}}_{\mathrm{<span\; class="notranslate">\; L</span>}}\mathrm{<span\; class="notranslate">\; E.</span>}<span\; class="notranslate">\; [</span>{<span\; class="notranslate">\; |</span>{\mathrm{<span\; class="notranslate">\; S</span>}}_{\mathrm{<span\; class="notranslate">\; 1</span>}}<span\; class="notranslate">\; |</span>}^{\mathrm{<span\; class="notranslate">\; 2</span>}}<span\; class="notranslate">\; ]</span><span\; class="notranslate">\; +</span>\underset{\mathrm{<span\; class="notranslate">\; Ch</span>}<span\; class="notranslate">\; =</span><span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; REST</span>}<span\; class="notranslate">\; )</span>}{\mathrm{<span\; class="notranslate">\; \&Sigma;</span>}}{\mathrm{<span\; class="notranslate">\; d</span>}}_{\mathrm{<span\; class="notranslate">\; Ch</span>}}{\mathrm{<span\; class="notranslate">\; d</span>}}_{\mathrm{<span\; class="notranslate">\; L</span>}}\mathrm{<span\; class="notranslate">\; E.</span>}<span\; class="notranslate">\; [</span>{<span\; class="notranslate">\; |</span>{\mathrm{<span\; class="notranslate">\; S</span>}}_{\mathrm{<span\; class="notranslate">\; 2</span>}}<span\; class="notranslate">\; |</span>}^{\mathrm{<span\; class="notranslate">\; 2</span>}}<span\; class="notranslate">\; ]</span>$

$<span\; class="notranslate">\; +</span>\underset{\mathrm{<span\; class="notranslate">\; Ch</span>}<span\; class="notranslate">\; =</span><span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; REST</span>}<span\; class="notranslate">\; )</span>}{\mathrm{<span\; class="notranslate">\; \&Sigma;</span>}}{\mathrm{<span\; class="notranslate">\; e</span>}}_{\mathrm{<span\; class="notranslate">\; Ch</span>}}{\mathrm{<span\; class="notranslate">\; e</span>}}_{\mathrm{<span\; class="notranslate">\; L</span>}}\mathrm{<span\; class="notranslate">\; E.</span>}<span\; class="notranslate">\; [</span>{<span\; class="notranslate">\; |</span>{\mathrm{<span\; class="notranslate">\; S</span>}}_{\mathrm{<span\; class="notranslate">\; 3</span>}}<span\; class="notranslate">\; |</span>}^{\mathrm{<span\; class="notranslate">\; 2</span>}}<span\; class="notranslate">\; ]</span><span\; class="notranslate">\; +</span>\underset{\mathrm{<span\; class="notranslate">\; Ch</span>}<span\; class="notranslate">\; =</span><span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; REST</span>}<span\; class="notranslate">\; )</span>}{\mathrm{<span\; class="notranslate">\; \&Sigma;</span>}}{\mathrm{<span\; class="notranslate">\; a</span>}}_{\mathrm{<span\; class="notranslate">\; L</span>}}{\mathrm{<span\; class="notranslate">\; b</span>}}_{\mathrm{<span\; class="notranslate">\; Ch</span>}}\mathrm{<span\; class="notranslate">\; E.</span>}<span\; class="notranslate">\; [</span>{\mathrm{<span\; class="notranslate">\; L</span>}}_{\mathrm{<span\; class="notranslate">\; dmx</span>}}{\mathrm{<span\; class="notranslate">\; R</span>}}_{\mathrm{<span\; class="notranslate">\; dmx</span>}}^{<span\; class="notranslate">\; *</span>}<span\; class="notranslate">\; ]</span><span\; class="notranslate">\; +</span>\underset{\mathrm{<span\; class="notranslate">\; Ch</span>}<span\; class="notranslate">\; =</span><span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; REST</span>}<span\; class="notranslate">\; )</span>}{\mathrm{<span\; class="notranslate">\; \&Sigma;</span>}}{\mathrm{<span\; class="notranslate">\; a</span>}}_{\mathrm{<span\; class="notranslate">\; Ch</span>}}{\mathrm{<span\; class="notranslate">\; b</span>}}_{\mathrm{<span\; class="notranslate">\; L</span>}}\mathrm{<span\; class="notranslate">\; E.</span>}{<span\; class="notranslate">\; [</span>{\mathrm{<span\; class="notranslate">\; L</span>}}_{\mathrm{<span\; class="notranslate">\; dmx</span>}}{\mathrm{<span\; class="notranslate">\; R</span>}}_{\mathrm{<span\; class="notranslate">\; dmx</span>}}^{<span\; class="notranslate">\; *</span>}<span\; class="notranslate">\; ]</span>}^{<span\; class="notranslate">\; *</span>}$

The inventors can now formulate the ICC

${\mathrm{<span\; class="notranslate">\; ICC</span>}}_{\mathrm{<span\; class="notranslate">\; L</span>}}<span\; class="notranslate">\; =</span>\frac{\mathrm{<span\; class="notranslate">\; Re</span>}<span\; class="notranslate">\; \{</span>\mathrm{<span\; class="notranslate">\; E.</span>}<span\; class="notranslate">\; [</span>{\mathrm{<span\; class="notranslate">\; LX</span>}}_{\mathrm{<span\; class="notranslate">\; L</span>}}^{<span\; class="notranslate">\; *</span>}<span\; class="notranslate">\; ]</span>}{\sqrt{\mathrm{<span\; class="notranslate">\; E.</span>}<span\; class="notranslate">\; [</span>{<span\; class="notranslate">\; |</span>\mathrm{<span\; class="notranslate">\; L</span>}<span\; class="notranslate">\; |</span>}^{\mathrm{<span\; class="notranslate">\; 2</span>}}<span\; class="notranslate">\; ]</span>\mathrm{<span\; class="notranslate">\; E.</span>}<span\; class="notranslate">\; [</span>{<span\; class="notranslate">\; |</span>{\mathrm{<span\; class="notranslate">\; x</span>}}_{\mathrm{<span\; class="notranslate">\; L</span>}}<span\; class="notranslate">\; |</span>}^{\mathrm{<span\; class="notranslate">\; 2</span>}}<span\; class="notranslate">\; ]</span>}}$

and sum the sum

${\mathrm{<span\; class="notranslate">\; \&sigma;</span>}}_{\mathrm{<span\; class="notranslate">\; L</span>}}<span\; class="notranslate">\; =</span>\frac{\mathrm{<span\; class="notranslate">\; E.</span>}<span\; class="notranslate">\; [</span>{<span\; class="notranslate">\; |</span>\mathrm{<span\; class="notranslate">\; L</span>}<span\; class="notranslate">\; |</span>}^{\mathrm{<span\; class="notranslate">\; 2</span>}}<span\; class="notranslate">\; ]</span>}{\mathrm{<span\; class="notranslate">\; E.</span>}<span\; class="notranslate">\; [</span>{<span\; class="notranslate">\; |</span>{\mathrm{<span\; class="notranslate">\; x</span>}}_{\mathrm{<span\; class="notranslate">\; L</span>}}<span\; class="notranslate">\; |</span>}^{\mathrm{<span\; class="notranslate">\; 2</span>}}<span\; class="notranslate">\; ]</span>}$

The third step (the inter-channel information of the upmix channel to the DTT parameter of the upmix channel) now the inventor can calculate the channel L according to the following formula

${\mathrm{<span\; class="notranslate">\; DTT</span>}}_{\mathrm{<span\; class="notranslate">\; L</span>}}<span\; class="notranslate">\; =</span>\frac{\mathrm{<span\; class="notranslate">\; 1</span>}}{\mathrm{<span\; class="notranslate">\; 2</span>}}<span\; class="notranslate">\; [</span><span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; 1</span>}<span\; class="notranslate">\; -</span>\frac{\mathrm{<span\; class="notranslate">\; 1</span>}}{{\mathrm{<span\; class="notranslate">\; \&sigma;</span>}}_{\mathrm{<span\; class="notranslate">\; L</span>}}}<span\; class="notranslate">\; )</span><span\; class="notranslate">\; +</span>\sqrt{{<span\; class="notranslate">\; (</span>\frac{\mathrm{<span\; class="notranslate">\; 1</span>}}{{\mathrm{<span\; class="notranslate">\; \&sigma;</span>}}_{\mathrm{<span\; class="notranslate">\; L</span>}}}<span\; class="notranslate">\; -</span>\mathrm{<span\; class="notranslate">\; 1</span>}<span\; class="notranslate">\; )</span>}^{\mathrm{<span\; class="notranslate">\; 2</span>}}<span\; class="notranslate">\; +</span>\mathrm{<span\; class="notranslate">\; 4</span>}\frac{{\mathrm{<span\; class="notranslate">\; ICC</span>}}_{\mathrm{<span\; class="notranslate">\; L</span>}}^{\mathrm{<span\; class="notranslate">\; 2</span>}}}{{\mathrm{<span\; class="notranslate">\; \&sigma;</span>}}_{\mathrm{<span\; class="notranslate">\; L</span>}}}}<span\; class="notranslate">\; ]</span>$

The direct energy of L is

E[|D _L | ² ]＝DTT·E[|L| ² ]

The surrounding energy of L is

E[|A _L | ² ]=(1-DTT)·E[|L| ² ]

Step Four (Downmix Direct/Ambient Energy)

If an example of the incoherent downmixing method is used, the energy around the left downmix channel is

$\mathrm{<span\; class="notranslate">\; E.</span>}<span\; class="notranslate">\; [</span>{<span\; class="notranslate">\; |</span>{\mathrm{<span\; class="notranslate">\; A</span>}}_{\mathrm{<span\; class="notranslate">\; wxya</span>}}<span\; class="notranslate">\; |</span>}^{\mathrm{<span\; class="notranslate">\; 2</span>}}<span\; class="notranslate">\; ]</span><span\; class="notranslate">\; =</span>\mathrm{<span\; class="notranslate">\; E.</span>}<span\; class="notranslate">\; [</span>{<span\; class="notranslate">\; |</span>{\mathrm{<span\; class="notranslate">\; A</span>}}_{\mathrm{<span\; class="notranslate">\; L</span>}}<span\; class="notranslate">\; |</span>}^{\mathrm{<span\; class="notranslate">\; 2</span>}}<span\; class="notranslate">\; ]</span><span\; class="notranslate">\; +</span>\mathrm{<span\; class="notranslate">\; E.</span>}<span\; class="notranslate">\; [</span>{<span\; class="notranslate">\; |</span>{\mathrm{<span\; class="notranslate">\; A</span>}}_{\mathrm{<span\; class="notranslate">\; ls</span>}}<span\; class="notranslate">\; |</span>}^{\mathrm{<span\; class="notranslate">\; 2</span>}}<span\; class="notranslate">\; ]</span><span\; class="notranslate">\; +</span>\frac{\mathrm{<span\; class="notranslate">\; E.</span>}<span\; class="notranslate">\; [</span>{<span\; class="notranslate">\; |</span>{\mathrm{<span\; class="notranslate">\; A</span>}}_{\mathrm{<span\; class="notranslate">\; C</span>}}<span\; class="notranslate">\; |</span>}^{\mathrm{<span\; class="notranslate">\; 2</span>}}<span\; class="notranslate">\; ]</span><span\; class="notranslate">\; +</span>\mathrm{<span\; class="notranslate">\; E.</span>}<span\; class="notranslate">\; [</span>{<span\; class="notranslate">\; |</span>{\mathrm{<span\; class="notranslate">\; A</span>}}_{\mathrm{<span\; class="notranslate">\; LF</span>}}<span\; class="notranslate">\; |</span>}^{\mathrm{<span\; class="notranslate">\; 2</span>}}<span\; class="notranslate">\; ]</span>}{\mathrm{<span\; class="notranslate">\; 2</span>}}$

, and the same for the direct part and the direct and surrounding part of the left channel. Note that the previous description is only a downmixing rule. Other downmixing rules are also possible.

Step 5 (calculate weights for surrounding extraction in the downmix channel)

The DTT ratio of the lower left mix is

${\mathrm{<span\; class="notranslate">\; DTT</span>}}_{\mathrm{<span\; class="notranslate">\; wxya</span>}}<span\; class="notranslate">\; =</span>\mathrm{<span\; class="notranslate">\; 1</span>}<span\; class="notranslate">\; -</span>\frac{\mathrm{<span\; class="notranslate">\; E.</span>}<span\; class="notranslate">\; [</span>{<span\; class="notranslate">\; |</span>{\mathrm{<span\; class="notranslate">\; A</span>}}_{\mathrm{<span\; class="notranslate">\; wxya</span>}}<span\; class="notranslate">\; |</span>}^{\mathrm{<span\; class="notranslate">\; 2</span>}}<span\; class="notranslate">\; ]</span>}{\mathrm{<span\; class="notranslate">\; E.</span>}<span\; class="notranslate">\; [</span>{<span\; class="notranslate">\; |</span>{\mathrm{<span\; class="notranslate">\; L</span>}}_{\mathrm{<span\; class="notranslate">\; dmx</span>}}<span\; class="notranslate">\; |</span>}^{\mathrm{<span\; class="notranslate">\; 2</span>}}<span\; class="notranslate">\; ]</span>}$

Then the calculation of the weight factor can be as described in the embodiment of Figure 5 (that is, using the sqrt (DTT) or sqrt (1-DTT) method) or as described in the embodiment of Figure 6 (that is, using the mixing matrix method) calculate.

Basically, the foregoing processing example is concerned with the CPC, ICC, and CLD parameters of the MPS stream-to-ambient ratio in the downmix channel.

According to other embodiments, there are typically other means to achieve similar ends and other circumstances. For example, there may be other algorithms than those described above for downmixing, other speaker layouts, other coding methods, and other ways of doing multi-channel ambient estimation where a specific channel is compared to the rest.

Although the invention has been described in the context of block diagrams, the invention can also be implemented by computer-implemented methods, where the blocks represent actual or logical hardware components. In the latter case, the blocks represent corresponding method steps, wherein these steps represent functions performed by corresponding logical or physical hardware blocks.

The examples are presented merely to illustrate the principles of the invention. It should be understood that modifications and variations in the arrangements and details described herein will be apparent to those skilled in the art. It is therefore the intention to be limited only by the scope of the appended claims rather than by the specific details presented in the illustration and explanation of the embodiments herein.

According to some implementation requirements of the inventive method, the inventive method can be implemented in hardware or in software. The implementation can be performed using a digital storage medium, in particular a disc, DVD or CD with readable control signals stored thereon, which can cooperate with a programmable computer system to carry out the inventive method. In general, the invention can thus be implemented as a computer program product having a program code stored on a machine-readable carrier, which program code is operable to carry out the inventive method when the computer program product is run on a computer. In other words, the inventive method is thus a computer program with a program code which can be used to perform at least one of the inventive methods when the computer program is run on a computer. The encoded audio signal of the present invention may be stored on any machine-readable storage medium, such as a digital storage medium.

The advantages of this novel concept and technique are the foregoing embodiments described in this application, ie a device, a method or a computer program allowing estimation and extraction of direct and/or surrounding components from an audio signal by means of parametric spatial information. More specifically, the novel processing of the present invention functions in frequency bands as typically in the field of ambient extraction. The concepts presented are relevant to audio signal processing, since several applications require separation of the immediate and surrounding components from the audio signal.

Contrary to the surrounding extraction methods of the prior art, the present concept is not based only on stereo input signals, it can also be applied to the mono downmix case. For single-channel downmixing, there is usually no channel-to-channel difference to calculate. But by considering spatial side information, surrounding extraction becomes possible in this case as well.

An advantage of the present invention is that it uses spatial parameters to estimate the ambient level of the "old" signal. It is based on the idea that the spatial parameters already contain relevant information about the inter-channel differences of the "original" stereo or multi-channel signal.

Once the ambient level of the original stereo or multi-channel signal is estimated, direct and ambient levels can also be derived in the provided downmix channels. This can be done from a linear combination (ie a weighted sum) of the ambient energy of the surrounding part and the direct energy or amplitude of the immediate part. Therefore, embodiments of the present invention provide surrounding estimation and extraction by means of spatial side information.

Extending from this idea of side information based processing, there are the following advantageous properties or advantages.

Embodiments of the present invention provide ambient estimation by means of spatial side information and the provided downmix channel. These and surrounding estimates are important when more than one downmix channel is provided along with side information. The side information and the information measured from the downmix channel volume can be used together for ambient estimation. In MPEG Surround with stereo downmix, these two sources of information together provide complete information on the inter-channel relations of the original multi-channel sound, and the surrounding estimation is based on these relations.

Embodiments of the present invention also provide downmixing of direct energy and ambient energy. In the case of the side information based ambient extraction, there is an intermediate step above the multi-channel estimation of the provided downmix channel. Therefore, such ambient information has to be mapped to the number of downmixed audio channels in an efficient manner. Such a process may be referred to as downmixing, since it corresponds to the downmixing of audio channels. This is most straightforwardly done by combining the direct energy and the ambient energy in the same way as the downmix channel downmix provided.

The downmixing law does not have an ideal solution, but may depend on the application. For example, in MPEG Surround, each channel (center, front speaker, rear speaker) is advantageously processed differently since typically the signal content is different.

Furthermore, embodiments provide multi-channel ambient estimation that is independent for each channel with respect to other channels. This property/approach allows pure use of the stereo surround estimation formula presented for each channel relative to all other channels. By this means, there is no need to assume that the ambient levels of all channels are equal. The presented approach is based on the assumption regarding spatial perception that the surrounding components in each channel have incoherent counterparts for that component's parts in all other channels. An example suggesting that such an assumption is valid is that one of the noisy two channels (surroundings) can be further divided into two channels each with half volume energy, without noticeable impact on the received sound scene.

As far as signal processing is concerned, it is advantageous that by applying the presented ambient estimation formula to each channel compared to a linear combination of all other channels, the actual direct/ambient ratio estimation can be done.

Finally, an embodiment provides for applying the estimated immediate ambient energy to extract the actual signal. Once the ambient level of the downmix channel is known, two inventive methods can be applied to obtain the ambient signal. The first method is based on simple multiplication, where the direct and ambient parts of each downmix channel can be generated by multiplying the signal by sqrt (direct to total energy ratio) and sqrt (surround to total energy ratio). In this way, two signals that are coherent to each other are provided for each downmix channel, but the two signals have estimated energies of the immediate part and the surrounding part.

The second method is based on a least mean square solution with channel mixing, where channel mixing (possibly also with a negative sign) allows a better estimation of the immediate surrounding signal than the previous solution. and in "Multi-Speaker Playback of Stereo Signals", C. Faller, AES Conference, October 2007; and "Patent Application Title: Method for Producing Multi-Channel Audio Signals from Stereo Signals", Inventor: Christof Faller, Attorney By: FISH & RICHARDSON P.C., Assignee: LG Electronics, Origin: Minneapolis, Minnesota, USA, IPC8 Class: AH04R500FI, USPC Class: 3811 Minimum Average of Stereo Input and Equivalent Ambient Levels for Channels Provided Solution In contrast, the present invention provides a least mean square solution which does not require equal ambient levels and which can be extended to any number of channels.

Additional properties of the novel treatment are as follows. In ambient processing for binaural rendering, the ambient may be processed using a filter having the property of providing an interaural coherence in frequency bands similar to that of an actual diffuse sound field, wherein the filter also includes room effects. In direct part processing for binaural rendering, the direct part may be fed through a head related transfer function (HRTF) possibly adding room effects such as early reflections and/or reverberation.

Besides, "separation level" control corresponding to dry/wet control can be implemented in other embodiments. More specifically, full separation may not be desired in many applications, since it may lead to auditory artifact defects, such as sudden changes, modulation effects, and the like. Thus, all relevant portions of the process can be implemented with "separation level" control to control the desired and useful amount of separation. As for FIG. 11 , such separation level control is indicated by the dashed box controlling the direct/surround separation 1120 and/or the control input signal 1105 of the binaural rendering means 910 , 1010 . This control works similarly to the Dry/Wet control for audio effects processing.

The main effects of the provided solution are as follows. The system is valid in all cases and can also use parametric stereo and MPEG surround with a mono downmix signal, unlike previous solutions which relied only on the downmix information. Furthermore, the system can exploit the spatial side information transmitted with the audio signal in the spatial audio bitstream to more accurately estimate direct and ambient energy compared to purely inter-channel analysis using downmix channels. Therefore, many applications such as binaural processing can benefit by applying different processing for the immediate part and the surrounding part of the sound.

The embodiments are based on the following psychoacoustic assumptions. The human auditory system localizes sound sources based on interaural cues in time-frequency tiles (regions limited to certain frequency and time ranges). If two or more irrelevant parallel sound sources overlapping in time and frequency are presented at different positions at the same time, the auditory system cannot perceive the position of the sound source. The reason is that the sum of these sources does not produce reliable ear cues in the listener. The auditory system may thus be described as picking up audio scenes close to the time-frequency slice to provide reliable localization information, but treating the rest as unlocalizable. This means that the auditory system can locate the sound source in a complex sound environment. At the same time coherent sources have different effects, forming the same ear cue that a single source among coherent sources would form.

This point is also a property utilized by the embodiments. Localizable (direct) and non-localizable (surrounding) sound levels can be estimated and these components extracted. Spatialization signal processing is applied to localizable/direct parts only, while diffusion/spatial/enveloping processing is applied to non-localizable/surrounding parts. This has a significant effect on the design of binaural processing systems, since multiple processing is only applied where needed, leaving the rest of the signal unaffected. All processing occurs in frequency bands that approximate the frequency resolution of human hearing.

Embodiments maximize perceptual quality but minimize perceived problems based on a decomposition of the signal. By using this decomposition, the immediate and surrounding components of the audio signal can be obtained separately. The two components are then further processed to achieve the desired effect or expression.

More specifically, embodiments of the present invention allow surrounding estimation in the coding domain with the aid of spatial side information.

The invention is also advantageous in that it reduces the typical problems of headphones reproducing audio signals by separating the direct signal from the surrounding signal. Embodiments allow improving the existing direct/ambient extraction methods applied to binaural sound presentation for headphone reproduction.

The main use cases for processing based on spatial side information are natural MPEG surround and parametric stereo (and similar parametric coding techniques). Typical application uses that benefit from ambient extraction are binaural playback, since it can apply different degrees of room effects to different parts of the sound, and upmixing to more channels, since it can localize and process parts of the sound differently different components. There may also be some application uses where the user requires modification of the direct/surrounding alignment, eg for intelligent speech enhancement.

Claims (16)

1. one kind is used for extracting a direct and/or ambient signals (125-1 from mixing signal (115) and a spatial parameter information (105) once; Device 125-2) (100), said mixed signal (115) down and said spatial parameter information (105) expression have more multichannel (Ch than said mixed signal (115) down ₁Ch _N) a multi-channel audio signal (101), wherein, said spatial parameter information (105) comprises relational expression between the sound channel of said multi-channel audio signal (101), said device (100) comprises:

One directly/and estimate device (110) on every side, be used for estimating a direct definite message or answer breath (113) of part and/or a peripheral part of said multi-channel audio signal (101) based on said spatial parameter information (105); And

One direct/on every side extraction apparatus (120) is used for extracting said direct signal part (125-1) and/or said ambient signals part (125-2) based on the position definite message or answer breath (113) of the said estimation of said direct part or said peripheral part and from said mixed signal (115) down.

2. device according to claim 1; Wherein, The position definite message or answer breath (113) that said directly/on every side extraction apparatus (420) is constructed to down to mix the said estimation of said direct part or said peripheral part obtains the position definite message or answer breath that mixes down of said direct part or said peripheral part, and extracts said direct signal part (125-1) or said ambient signals partly (125-2) based on the said position definite message or answer breath that mixes down and from said mixed signal (115) down.

3. device according to claim 2; Wherein, Said directly/on every side extraction apparatus (420) further is constructed to have through combination the position definite message or answer breath of position definite message or answer breath and the said estimation of the said peripheral part with incoherence summation of said estimation of the said direct part of coherence's summation, and the following of position definite message or answer breath (113) of carrying out the said estimation of said direct part or said peripheral part mixes.

4. according to claim 2 or 3 described devices; Wherein, Said directly/on every side extraction apparatus (520) further be constructed to from said direct part or said peripheral part the said position definite message or answer breath that mixes down (555-1, lead in 555-2) calculate gain parameter (565-1,565-2); And lead the gain parameter calculated (565-1 565-2) is applied to said mixed signal (115) down and obtains said direct signal part (125-1) or said ambient signals part (125-2) said.

5. device according to claim 4; Wherein, Said directly/on every side extraction apparatus (520) further is constructed to the said position definite message or answer (555-1 that mixes down according to said direct part or said peripheral part; 555-2) measure one directly to total (DTT) can than or on every side can ratio to total (ATT), and use based on the DTT that is measured can than or ATT can than extracting parameter as said gain parameter (565-1,565-2).

6. according to each described device in the claim 1 to 5; Wherein, Said directly/on every side extraction apparatus (520) is constructed to extract said direct signal part (125-1) or said ambient signals part (125-2) through one M * M square of extraction matrix is applied to said mixed signal (115) down; Wherein, said M * M square of size (M) and following mixing sound road (Ch of extracting matrix ₁Ch _M) number (M) corresponding.

7. device according to claim 6; Wherein, Said directly/on every side extraction apparatus (520) is constructed to that further one first many extracting parameters are applied to said mixed signal (115) down and obtains said direct signal part (125-1); And one second many extracting parameters are applied to said mixed signal (115) down obtain said ambient signals part (125-2), said more than first extracting parameter and said more than second the extracting parameter linea angulata matrix that partners.

8. according to each described device in the claim 1 to 7; Wherein, Said directly/estimate on every side device (110) be constructed to based on said spatial parameter information (113) and by said directly/estimate said at least two following mixing sound roads (825) of mixed signal (115) down that device (110) is received on every side, estimate the said direct part of said multi-channel audio signal (101) or institute's rheme definite message or answer breath (113) of said peripheral part.

9. according to each described device in the claim 1 to 8, wherein, said directly/estimate that on every side device (710) is constructed to each the sound channel (Ch to said multi-channel audio signal (101) _i) through use said spatial parameter information (105) apply one stereo around the estimation formula, wherein, said stereo around the estimation formula given by following formula

DTT _i=f _DTT[σ(Ch _i′R) _′ICC _i(Ch _i′R)]

ATT _i＝1-DTT _i

Said formula depends on the accurate poor (CLD in sound channel position _i), said sound channel position is accurate poor to be σ _iDecibel value, and be sound channel Ch _iInter-channel coherence (ICC _i) parameter, wherein, R is the linear combination of all the other sound channels.

10. according to each described device in the claim 1 to 9; Wherein, Said directly/on every side extraction apparatus (620) is constructed to hand over the lowest mean square that mixes to separate by the use sound channel and extracts said direct signal part (125-1) or said ambient signals part (125-2), and said LMS separates and do not require that equal position on every side is accurate.

11. device according to claim 9, wherein, said directly/on every side extraction apparatus (620) is constructed to calculate said LMS and separate through supposing that a signal model is led, and makes said LMS separate and is not limited to mixed signal under the stereo channel.

12. according to each described device in the claim 1 to 11, wherein, said device also comprises:

One ears direct voice presents device (910), is used to handle said direct signal part (125-1) and obtains one first ears output signal (915);

One ears ambient sound presents device (1010), is used to handle said ambient signals part (125-2) and obtains one second ears output signal (1015); And

One combiner (1130) is used to make up said first ears output signal (915) and said second ears output signals (1015) obtain the ears output signal (1135) once combination.

13. device according to claim 12; Wherein, Said ears ambient sound presents device (1010) and is constructed to that a room effect and/or a wave filter are applied to said ambient signals part (125-2) said second ears output signal (1015) is provided, and said second ears output signals (1015) are applicable to coherence between the ears of actual dispersion sound field.

14. according to claim 12 or 13 described devices; Wherein, said ears direct voice presents device (910) and is constructed to present said direct signal part (125-1) based on head related transfer function (HRTF) through wave filter and obtains said first ears output signal (915).

15. one kind is used for extracting a direct and/or ambient signals (125-1 from mixing signal (115) and a spatial parameter information (105) once; Method 125-2) (100), said mixed signal (115) down and said spatial parameter information (105) expression have more multichannel (Ch than said mixed signal (115) down ₁Ch _N) a multi-channel audio signal (101), wherein, said spatial parameter information (105) comprises relational expression between the sound channel of said multi-channel audio signal (101), said method (100) comprises:

Estimate direct part of (110) said multi-channel audio signal (101) and/or a definite message or answer breath (113) an of peripheral part based on said spatial parameter information (105); And

Based on the position definite message or answer breath (113) of the said estimation of said direct part or said peripheral part, extract (120) said direct signal parts (125-1) and/or ambient signals part (125-2) from said mixed signal (115) down.

16. the computer program with program code is used for enforcement of rights and requires 15 described methods (100) when said computer program is carried out on a computing machine.

Images