KR20230067427A - Electronic device for controlling beamforming and operating method thereof - Google Patents

Abstract

An electronic device according to one embodiment comprises: an input module comprising multiple microphones for receiving external sound signals to determine a customized beamformer filter; a memory in which computer-executable instructions and an initial value of a voice parameter for beamforming of the external sound signals are stored; and a processor for accessing the memory to execute the instructions. The instructions may be configured to: estimate a feature value of the external sound signals; calculate the initial value of the voice parameter for beamforming on the basis of the external sound signals received from the multiple microphones; determine, according to the feature value, whether to store the calculated initial value; determine, according to the feature value, an initial value to be used between the calculated initial value and the initial value stored in the memory; and acquire, according to the feature value, a target voice parameter for beamforming of the external sound signals, on the basis of the determined initial value. The electronic device can adaptively determine the beamformer filter according to differences in a user's wearing style and ear shape.

Description

Electronic device for controlling beamforming and its operation method

The disclosure below relates to an electronic device for controlling beamforming and a method for controlling the same.

Electronic devices may provide functions related to audio signal processing. For example, the electronic device may provide a call function for collecting and transmitting an audio signal, a recording function for recording an audio signal, and the like.

Electronic devices that output audio, such as earphones and headphones, may be equipped with various noise cancellation and suppression technologies to distinguish audio signals. For example, headphones may acquire ambient noise through a microphone connected to a noise canceling circuit, and output an anti-noise signal of an inverse phase with respect to the acquired noise. The user hears the ambient noise and the noise of the opposite phase together, and through this, the effect of removing the noise can be obtained. In addition, in order to obtain a more improved user voice in an audio output device, a method of beamforming signals received through a plurality of microphones is being researched.

Electronic devices such as earphones or headphones worn on the ears have a large distance between the microphone mounted on the electronic device and the user's mouth in the worn state due to the limitation of the form factor, so the influence of ambient noise is large and it may be difficult to acquire the user's voice. .

If there is a lot of noise in the surrounding environment while performing a function such as a call or recording with an electronic device such as earphones or headphones, it may be difficult to obtain a good voice signal.

If the user's voice cannot be properly acquired through earphones or headphones, the quality of voice calls and voice recognition performance may deteriorate.

An electronic device according to an embodiment includes an input module including a plurality of microphones for receiving external sound signals, computer-executable instructions, and beamforming of the external sound signals. A memory in which initial values of voice parameters are stored, and a processor accessing the memory to execute the instructions, wherein the instructions estimate a characteristic value of the external sound signal, and the plurality of microphones Calculate an initial value of a voice parameter for beamforming based on an external sound signal received as , determine whether to store the calculated initial value in the memory according to the characteristic value, and determine whether to store the calculated initial value in the memory according to the characteristic value. It may be configured to determine which of an initial value or an initial value stored in the memory to use, and obtain a target voice parameter for beamforming of the external sound signal based on the determined initial value according to the characteristic value. .

A method for acquiring voice parameters for beamforming according to an embodiment includes an operation of estimating characteristic values of an external sound signal received through a plurality of microphones, and a voice parameter for beamforming of the external sound signal. An operation of calculating an initial value, an operation of determining whether to store the calculated initial value according to the characteristic value, an operation of determining whether to use the calculated initial value or a stored initial value according to the characteristic value, and and acquiring a target voice parameter for beamforming of the external sound signal based on the determined initial value according to the characteristic value.

An operating method of an electronic device according to an embodiment includes estimating a characteristic value of an external sound signal received through a plurality of microphones and calculating an initial value of a voice parameter for beamforming of the external sound signal. An operation of determining whether to store the calculated initial value according to the characteristic value, an operation of determining whether to use the calculated initial value or a stored initial value according to the characteristic value, Obtaining a target voice parameter for beamforming the external sound signal based on the determined initial value, determining a filter for beamforming the external sound signal based on the target voice parameter, and beamforming with the filter An operation of estimating a residual noise level for the generated signal, and an operation of performing noise processing on the beamformed signal according to the estimated residual noise level.

According to various embodiments, an electronic device for adjusting voice parameters for beamforming according to characteristic values of external sound signals may be provided.

According to various embodiments, an electronic device that adaptively determines a beamformer filter according to a difference in a user's wearing style and ear shape may be provided.

In addition to this, various effects identified directly or indirectly through this document may be provided.

1 is a block diagram of an electronic device in a network environment, according to various embodiments.
2 is a block diagram of an audio module according to various implementations.
3 is a diagram illustrating an example of an audio signal processing system according to various embodiments.
4 is a block diagram for explaining a beamformer according to various embodiments.
5A and 5B are diagrams for explaining a process of performing beamforming and noise processing in an electronic device and a configuration of the electronic device according to an exemplary embodiment.
6 is a flowchart illustrating an operating method of an electronic device according to various embodiments.
7 is a flowchart illustrating a method of processing noise in an electronic device according to various embodiments of the present disclosure.
8 to 11 are flowcharts for explaining a method of operating an electronic device according to a characteristic value of an external sound signal.

Hereinafter, embodiments will be described in detail with reference to the accompanying drawings. In the description with reference to the accompanying drawings, the same reference numerals are given to the same components regardless of reference numerals, and overlapping descriptions thereof will be omitted.

<Electronic device>

1 is a block diagram of an electronic device in a network environment, according to various embodiments.

1 is a block diagram of an electronic device 101 within a network environment 100, according to various embodiments. Referring to FIG. 1 , in a network environment 100, an electronic device 101 communicates with an electronic device 102 through a first network 198 (eg, a short-range wireless communication network) or through a second network 199. It may communicate with at least one of the electronic device 104 or the server 108 through (eg, a long-distance wireless communication network). According to an embodiment, the electronic device 101 may communicate with the electronic device 104 through the server 108 . According to an embodiment, the electronic device 101 includes a processor 120, a memory 130, an input module 150, an audio output module 155, a display module 160, an audio module 170, a sensor module ( 176), interface 177, connection terminal 178, haptic module 179, camera module 180, power management module 188, battery 189, communication module 190, subscriber identification module 196 , or the antenna module 197 may be included. In some embodiments, in the electronic device 101, at least one of these components (eg, the connection terminal 178) may be omitted or one or more other components may be added. In some embodiments, some of these components (eg, sensor module 176, camera module 180, or antenna module 197) are integrated into a single component (eg, display module 160). It can be.

The processor 120, for example, executes software (eg, the program 140) to cause at least one other component (eg, hardware or software component) of the electronic device 101 connected to the processor 120. It can control and perform various data processing or calculations. According to one embodiment, as at least part of data processing or operation, processor 120 transfers instructions or data received from other components (e.g., sensor module 176 or communication module 190) to volatile memory 132. , processing commands or data stored in the volatile memory 132 , and storing resultant data in the non-volatile memory 134 . According to one embodiment, the processor 120 includes a main processor 121 (eg, a central processing unit or an application processor) or a secondary processor 123 (eg, a graphic processing unit, a neural network processing unit ( NPU: neural processing unit (NPU), image signal processor, sensor hub processor, or communication processor). For example, when the electronic device 101 includes the main processor 121 and the auxiliary processor 123, the auxiliary processor 123 may use less power than the main processor 121 or be set to be specialized for a designated function. can The secondary processor 123 may be implemented separately from or as part of the main processor 121 .

The secondary processor 123 may, for example, take the place of the main processor 121 while the main processor 121 is in an inactive (eg, sleep) state, or the main processor 121 is active (eg, running an application). ) state, together with the main processor 121, at least one of the components of the electronic device 101 (eg, the display module 160, the sensor module 176, or the communication module 190) It is possible to control at least some of the related functions or states. According to one embodiment, the auxiliary processor 123 (eg, an image signal processor or a communication processor) may be implemented as part of other functionally related components (eg, the camera module 180 or the communication module 190). there is. According to an embodiment, the auxiliary processor 123 (eg, a neural network processing device) may include a hardware structure specialized for processing an artificial intelligence model. AI models can be created through machine learning. Such learning may be performed, for example, in the electronic device 101 itself where the artificial intelligence model is performed, or may be performed through a separate server (eg, the server 108). The learning algorithm may include, for example, supervised learning, unsupervised learning, semi-supervised learning or reinforcement learning, but in the above example Not limited. The artificial intelligence model may include a plurality of artificial neural network layers. Artificial neural networks include deep neural networks (DNNs), convolutional neural networks (CNNs), recurrent neural networks (RNNs), restricted boltzmann machines (RBMs), deep belief networks (DBNs), bidirectional recurrent deep neural networks (BRDNNs), It may be one of deep Q-networks or a combination of two or more of the foregoing, but is not limited to the foregoing examples. The artificial intelligence model may include, in addition or alternatively, software structures in addition to hardware structures.

The memory 130 may store various data used by at least one component (eg, the processor 120 or the sensor module 176) of the electronic device 101 . The data may include, for example, input data or output data for software (eg, program 140) and commands related thereto. The memory 130 may include volatile memory 132 or non-volatile memory 134 .

The program 140 may be stored as software in the memory 130 and may include, for example, an operating system 142 , middleware 144 , or an application 146 .

The input module 150 may receive a command or data to be used by a component (eg, the processor 120) of the electronic device 101 from the outside of the electronic device 101 (eg, a user). The input module 150 may include, for example, a microphone, a mouse, a keyboard, a key (eg, a button), or a digital pen (eg, a stylus pen).

The sound output module 155 may output sound signals to the outside of the electronic device 101 . The sound output module 155 may include, for example, a speaker or a receiver. The speaker can be used for general purposes such as multimedia playback or recording playback. A receiver may be used to receive an incoming call. According to one embodiment, the receiver may be implemented separately from the speaker or as part of it.

The display module 160 may visually provide information to the outside of the electronic device 101 (eg, a user). The display module 160 may include, for example, a display, a hologram device, or a projector and a control circuit for controlling the device. According to an embodiment, the display module 160 may include a touch sensor configured to detect a touch or a pressure sensor configured to measure the intensity of force generated by the touch.

The audio module 170 may convert sound into an electrical signal or vice versa. According to an embodiment, the audio module 170 acquires sound through the input module 150, the sound output module 155, or an external electronic device connected directly or wirelessly to the electronic device 101 (eg: Sound may be output through the electronic device 102 (eg, a speaker or a headphone).

The sensor module 176 detects an operating state (eg, power or temperature) of the electronic device 101 or an external environmental state (eg, a user state), and generates an electrical signal or data value corresponding to the detected state. can do. According to one embodiment, the sensor module 176 may include, for example, a gesture sensor, a gyro sensor, an air pressure sensor, a magnetic sensor, an acceleration sensor, a grip sensor, a proximity sensor, a color sensor, an infrared (IR) sensor, a bio sensor, It may include a temperature sensor, humidity sensor, hall sensor, or light sensor.

The interface 177 may support one or more designated protocols that may be used to directly or wirelessly connect the electronic device 101 to an external electronic device (eg, the electronic device 102). According to one embodiment, the interface 177 may include, for example, a high definition multimedia interface (HDMI), a universal serial bus (USB) interface, an SD card interface, or an audio interface.

The connection terminal 178 may include a connector through which the electronic device 101 may be physically connected to an external electronic device (eg, the electronic device 102). According to one embodiment, the connection terminal 178 may include, for example, an HDMI connector, a USB connector, an SD card connector, or an audio connector (eg, a headphone connector).

The haptic module 179 may convert electrical signals into mechanical stimuli (eg, vibration or motion) or electrical stimuli that a user may perceive through tactile or kinesthetic senses. According to one embodiment, the haptic module 179 may include, for example, a motor, a piezoelectric element, or an electrical stimulation device.

The camera module 180 may capture still images and moving images. According to one embodiment, the camera module 180 may include one or more lenses, image sensors, image signal processors, or flashes.

The power management module 188 may manage power supplied to the electronic device 101 . According to one embodiment, the power management module 188 may be implemented as at least part of a power management integrated circuit (PMIC), for example.

The battery 189 may supply power to at least one component of the electronic device 101 . According to one embodiment, the battery 189 may include, for example, a non-rechargeable primary cell, a rechargeable secondary cell, or a fuel cell.

The communication module 190 is a direct (eg, wired) communication channel or a wireless communication channel between the electronic device 101 and an external electronic device (eg, the electronic device 102, the electronic device 104, or the server 108). Establishment and communication through the established communication channel may be supported. The communication module 190 may include one or more communication processors that operate independently of the processor 120 (eg, an application processor) and support direct (eg, wired) communication or wireless communication. According to one embodiment, the communication module 190 may be a wireless communication module 192 (eg, a cellular communication module, a short-range wireless communication module, or a global navigation satellite system (GNSS) communication module) or a wired communication module 194 (eg, a : a local area network (LAN) communication module or a power line communication module). Among these communication modules, a corresponding communication module is a first network 198 (eg, a short-range communication network such as Bluetooth, wireless fidelity (WiFi) direct, or infrared data association (IrDA)) or a second network 199 (eg, a legacy communication module). It may communicate with the external electronic device 104 through a cellular network, a 5G network, a next-generation communication network, the Internet, or a telecommunications network such as a computer network (eg, a LAN or a WAN). These various types of communication modules may be integrated as one component (eg, a single chip) or implemented as a plurality of separate components (eg, multiple chips). The wireless communication module 192 uses subscriber information (eg, International Mobile Subscriber Identifier (IMSI)) stored in the subscriber identification module 196 within a communication network such as the first network 198 or the second network 199. The electronic device 101 may be identified or authenticated.

The wireless communication module 192 may support a 5G network after a 4G network and a next-generation communication technology, for example, NR access technology (new radio access technology). NR access technologies include high-speed transmission of high-capacity data (enhanced mobile broadband (eMBB)), minimization of terminal power and access of multiple terminals (massive machine type communications (mMTC)), or high reliability and low latency (ultra-reliable and low latency (URLLC)). -latency communications)) can be supported. The wireless communication module 192 may support a high frequency band (eg, mmWave band) to achieve a high data rate, for example. The wireless communication module 192 uses various technologies for securing performance in a high frequency band, such as beamforming, massive multiple-input and multiple-output (MIMO), and full-dimensional multiplexing. Technologies such as input/output (FD-MIMO: full dimensional MIMO), array antenna, analog beam-forming, or large scale antenna may be supported. The wireless communication module 192 may support various requirements defined for the electronic device 101, an external electronic device (eg, the electronic device 104), or a network system (eg, the second network 199). According to one embodiment, the wireless communication module 192 may be used to realize peak data rate (eg, 20 Gbps or more) for realizing eMBB, loss coverage (eg, 164 dB or less) for realizing mMTC, or U-plane latency (for realizing URLLC). Example: downlink (DL) and uplink (UL) each of 0.5 ms or less, or round trip 1 ms or less) may be supported.

The antenna module 197 may transmit or receive signals or power to the outside (eg, an external electronic device). According to an embodiment, the antenna module 197 may include an antenna including a radiator formed of a conductor or a conductive pattern formed on a substrate (eg, PCB). According to one embodiment, the antenna module 197 may include a plurality of antennas (eg, an array antenna). In this case, at least one antenna suitable for a communication method used in a communication network such as the first network 198 or the second network 199 is selected from the plurality of antennas by the communication module 190, for example. can be chosen A signal or power may be transmitted or received between the communication module 190 and an external electronic device through the selected at least one antenna. According to some embodiments, other components (eg, a radio frequency integrated circuit (RFIC)) may be additionally formed as a part of the antenna module 197 in addition to the radiator.

According to various embodiments, the antenna module 197 may form a mmWave antenna module. According to one embodiment, the mmWave antenna module includes a printed circuit board, an RFIC disposed on or adjacent to a first surface (eg, a bottom surface) of the printed circuit board and capable of supporting a designated high frequency band (eg, mmWave band); and a plurality of antennas (eg, array antennas) disposed on or adjacent to a second surface (eg, a top surface or a side surface) of the printed circuit board and capable of transmitting or receiving signals of the designated high frequency band. can do.

At least some of the components are connected to each other through a communication method between peripheral devices (eg, a bus, general purpose input and output (GPIO), serial peripheral interface (SPI), or mobile industry processor interface (MIPI)) and signal ( e.g. commands or data) can be exchanged with each other.

According to an embodiment, commands or data may be transmitted or received between the electronic device 101 and the external electronic device 104 through the server 108 connected to the second network 199 . Each of the external electronic devices 102 or 104 may be the same as or different from the electronic device 101 . According to an embodiment, all or part of operations executed in the electronic device 101 may be executed in one or more external electronic devices among the external electronic devices 102 , 104 , or 108 . For example, when the electronic device 101 needs to perform a certain function or service automatically or in response to a request from a user or another device, the electronic device 101 instead of executing the function or service by itself. Alternatively or additionally, one or more external electronic devices may be requested to perform the function or at least part of the service. One or more external electronic devices receiving the request may execute at least a part of the requested function or service or an additional function or service related to the request, and deliver the execution result to the electronic device 101 . The electronic device 101 may provide the result as at least part of a response to the request as it is or additionally processed. To this end, for example, cloud computing, distributed computing, mobile edge computing (MEC), or client-server computing technology may be used. The electronic device 101 may provide an ultra-low latency service using, for example, distributed computing or mobile edge computing. In another embodiment, the external electronic device 104 may include an internet of things (IoT) device. Server 108 may be an intelligent server using machine learning and/or neural networks. According to one embodiment, the external electronic device 104 or server 108 may be included in the second network 199 . The electronic device 101 may be applied to intelligent services (eg, smart home, smart city, smart car, or health care) based on 5G communication technology and IoT-related technology.

2 is a block diagram 200 of an audio module 170 according to various implementations.

Referring to FIG. 2 , the audio module 170 includes, for example, an audio input interface 210, an audio input mixer 220, an analog to digital converter (ADC) 230, an audio signal processor 240, and a DAC. (digital to analog converter) 250, an audio output mixer 260, or an audio output interface 270 may be included.

The audio input interface 210 is a part of the input module 150 or through a microphone configured separately from the electronic device 101 (eg, a dynamic microphone, a condenser microphone, or a piezo microphone), obtained from the outside of the electronic device 101. An audio signal corresponding to sound may be received. For example, when an audio signal is obtained from an external electronic device 102 (eg, a headset or a microphone), the audio input interface 210 directly connects the external electronic device 102 through a connection terminal 178. , or may be connected wirelessly (eg, Bluetooth communication) through the wireless communication module 192 to receive an audio signal. According to an embodiment, the audio input interface 210 may receive a control signal related to an audio signal acquired from the external electronic device 102 (eg, a volume control signal received through an input button). The audio input interface 210 includes a plurality of audio input channels, and can receive different audio signals for each corresponding audio input channel among the plurality of audio input channels. According to an embodiment, additionally or alternatively, the audio input interface 210 may receive an audio signal from other components (eg, the processor 120 or the memory 130) of the electronic device 101 .

The audio input mixer 220 may synthesize a plurality of input audio signals into at least one audio signal. For example, according to one embodiment, the audio input mixer 220 may synthesize a plurality of analog audio signals input through the audio input interface 210 into at least one analog audio signal.

The ADC 230 may convert an analog audio signal into a digital audio signal. For example, according to one embodiment, ADC 230 converts an analog audio signal received via audio input interface 210, or an analog audio signal synthesized via audio input mixer 220 additionally or alternatively, into a digital audio signal. can be converted into signals.

The audio signal processor 240 may perform various processes on the digital audio signal received through the ADC 230 or the digital audio signal received from other components of the electronic device 101 . For example, according to one embodiment, the audio signal processor 240 changes the sampling rate of one or more digital audio signals, applies one or more filters, performs interpolation processing, amplifies or attenuates all or some frequency bands, It can perform noise processing (eg, noise or echo reduction), channel change (eg, switching between mono and stereo), mixing, or specified signal extraction. According to one embodiment, one or more functions of the audio signal processor 240 may be implemented in the form of an equalizer.

The DAC 250 may convert a digital audio signal into an analog audio signal. For example, according to one embodiment, the DAC 250 is a digital audio signal processed by the audio signal processor 240, or other components of the electronic device 101 (eg, processor 120 or memory 130). )) to convert the digital audio signal obtained from the analog audio signal.

The audio output mixer 260 may synthesize a plurality of audio signals to be output into at least one audio signal. For example, according to one embodiment, the audio output mixer 260 includes an audio signal converted to analog through the DAC 250 and another analog audio signal (eg, an analog audio signal received through the audio input interface 210). ) into at least one analog audio signal.

The audio output interface 270 transmits the analog audio signal converted through the DAC 250 or the analog audio signal synthesized by the audio output mixer 260 additionally or alternatively to the electronic device 101 through the sound output module 155. ) can be output to the outside. The sound output module 155 may include, for example, a speaker or receiver such as a dynamic driver or a balanced armature driver. According to one embodiment, the sound output module 155 may include a plurality of speakers. In this case, the audio output interface 270 may output an audio signal having a plurality of different channels (eg, stereo or 5.1 channels) through at least some of the plurality of speakers. According to one embodiment, the audio output interface 270 is directly connected to the external electronic device 102 (eg, an external speaker or headset) through a connection terminal 178 or wirelessly through a wireless communication module 192. and output an audio signal.

According to one embodiment, the audio module 170 does not separately include the audio input mixer 220 or the audio output mixer 260, and uses at least one function of the audio signal processor 240 to generate a plurality of digital audio signals. At least one digital audio signal may be generated by synthesizing them.

According to one embodiment, the audio module 170 is an audio amplifier (not shown) capable of amplifying an analog audio signal input through the audio input interface 210 or an audio signal to be output through the audio output interface 270. (e.g. speaker amplification circuit). According to one embodiment, the audio amplifier may be configured as a separate module from the audio module 170.

Electronic devices according to various embodiments disclosed in this document may be devices of various types. The electronic device may include, for example, a portable communication device (eg, a smart phone), a computer device, a portable multimedia device, a portable medical device, a camera, a wearable device, or a home appliance. An electronic device according to an embodiment of the present document is not limited to the aforementioned devices.

Various embodiments of this document and terms used therein are not intended to limit the technical features described in this document to specific embodiments, but should be understood to include various modifications, equivalents, or substitutes of the embodiments. In connection with the description of the drawings, like reference numbers may be used for like or related elements. The singular form of a noun corresponding to an item may include one item or a plurality of items, unless the relevant context clearly dictates otherwise. In this document, "A or B", "at least one of A and B", "at least one of A or B", "A, B or C", "at least one of A, B and C", and "A Each of the phrases such as "at least one or two of , B, or C" may include any one of the items listed together in that phrase, or all possible combinations thereof. Terms such as "first", "second", or "first" or "secondary" may simply be used to distinguish that component from other corresponding components, and may refer to that component in other respects (eg, importance or order) is not limited. A (eg, first) component is said to be "coupled" or "connected" to another (eg, second) component, with or without the terms "functionally" or "communicatively." When mentioned, it means that the certain component may be connected to the other component directly (eg by wire), wirelessly, or through a third component.

The term "module" used in various embodiments of this document may include a unit implemented in hardware, software, or firmware, and is interchangeable with terms such as, for example, logic, logical blocks, parts, or circuits. can be used as A module may be an integrally constructed component or a minimal unit of components or a portion thereof that performs one or more functions. For example, according to one embodiment, the module may be implemented in the form of an application-specific integrated circuit (ASIC).

Various embodiments of this document provide one or more instructions stored in a storage medium (eg, internal memory 136 or external memory 138) readable by a machine (eg, electronic device 101). It may be implemented as software (eg, the program 140) including them. For example, a processor (eg, the processor 120 ) of a device (eg, the electronic device 101 ) may call at least one command among one or more instructions stored from a storage medium and execute it. This enables the device to be operated to perform at least one function according to the at least one command invoked. The one or more instructions may include code generated by a compiler or code executable by an interpreter. The device-readable storage medium may be provided in the form of a non-transitory storage medium. Here, 'non-temporary' only means that the storage medium is a tangible device and does not contain a signal (e.g. electromagnetic wave), and this term refers to the case where data is stored semi-permanently in the storage medium. It does not discriminate when it is temporarily stored.

According to one embodiment, the method according to various embodiments disclosed in this document may be provided by being included in a computer program product. Computer program products may be traded between sellers and buyers as commodities. A computer program product is distributed in the form of a device-readable storage medium (e.g. compact disc read only memory (CD-ROM)), or through an application store (e.g. Play Store™) or on two user devices (e.g. It can be distributed (eg downloaded or uploaded) online, directly between smart phones. In the case of online distribution, at least part of the computer program product may be temporarily stored or temporarily created in a device-readable storage medium such as a manufacturer's server, an application store server, or a relay server's memory.

According to various embodiments, each component (eg, module or program) of the above-described components may include a single object or a plurality of entities, and some of the plurality of entities may be separately disposed in other components. there is. According to various embodiments, one or more components or operations among the aforementioned corresponding components may be omitted, or one or more other components or operations may be added. Alternatively or additionally, a plurality of components (eg modules or programs) may be integrated into a single component. In this case, the integrated component may perform one or more functions of each of the plurality of components identically or similarly to those performed by a corresponding component of the plurality of components prior to the integration. . According to various embodiments, the actions performed by a module, program, or other component are executed sequentially, in parallel, iteratively, or heuristically, or one or more of the actions are executed in a different order, or omitted. or one or more other actions may be added.

3 is a diagram illustrating an example of an audio signal processing system 10 according to various embodiments.

Referring to FIG. 3 , an audio signal processing system 10 according to an embodiment may include a first electronic device 101 and a second electronic device 102 . The first electronic device 101 and the second electronic device 102 may include at least a part of the configuration of the electronic device 101 described above with reference to FIG. 1 . According to an embodiment, the first electronic device 101 may be connected to the second electronic device 102 by wire or wirelessly and output an audio signal transmitted by the second electronic device 102 . The first electronic device 101 may collect external sound signals using a plurality of microphones and transfer the collected audio signals to the second electronic device 102 .

According to an embodiment, the first electronic device 101 may be a wireless earphone capable of forming a short-range communication channel (eg, a communication channel based on a Bluetooth module) with the second electronic device 102 . For example, the first electronic device 101 may be any one of true-wireless stereo (TWS), wireless headphones, and wireless headsets. In FIG. 3 , the first electronic device 101 is illustrated as a kernel-type wireless earphone, but is not limited thereto. For example, the first electronic device 101 may be a stem-type wireless earphone in which at least a part of the housing protrudes in a specific direction in order to collect a good user voice signal. According to another embodiment, the first electronic device 101 may be a wired earphone connected to the second electronic device 102 in a wired manner.

According to an embodiment, the earphone-type first electronic device 101 includes an insertion unit 301a that can be inserted into the user's ear, and a device connected to the insertion unit 301a and at least partially mounted on the user's auricle. It may include a housing 301 (or case) having a mounting portion 301b. The first electronic device 101 may include a plurality of microphones 150-1 and 150-2.

According to various embodiments, the electronic device 101 may include an input interface 377 capable of receiving a user's input. The input interface 377 may include, for example, a physical interface (eg, physical button, touch button) and a virtual interface (eg, gesture, object recognition, voice recognition). In one embodiment, the electronic device 101 may include a touch sensor capable of detecting contact with a user's skin. For example, an area where a touch sensor is disposed (eg, the input interface 377) may be located in a part of the electronic device 101. An input may be applied by a user touching this area using a body part. Such touch input may include, for example, a single touch, multiple touches, a swipe, and/or a flick.

The microphones 150-1 and 150-2 may perform the functions of the input module 150 described above with reference to FIG. 1, and descriptions overlapping those described with reference to FIG. 1 will be omitted. Among the microphones 150-1 and 150-2, the first microphone 150-1 is disposed on the holder 301b to collect external ambient sound while the first electronic device 101 is worn on the user's ear. At least a portion of the sound hole may be exposed to the outside based on the inner side of the ear. Among the microphones 150-1 and 150-2, the second microphone 150-2 may be disposed in the insertion portion 301a. The second microphone 150-2 is based on the opening toward the auricle of the outer ear canal so that the signal transmitted to the inside of the outer ear canal (or ear canal) can be collected while the first electronic device 101 is worn on the user's ear. As such, at least a portion of the tone hole may be exposed toward the inside of the outer ear passage, or may be arranged such that at least a portion of the tone hole is in contact with an inner wall of the outer ear passage. For example, when a user wears the first electronic device 101 and utters a voice, at least a part of the tremors caused by the utterance is transmitted through the user's skin, muscles, bones, etc., and the transmitted tremors are transmitted to the inside of the ear. Ambient sound may be collected by the second microphone 150-2.

According to various embodiments, the second microphone 150 - 2 may be various types of microphones (eg, an in-ear microphone, an inner microphone, or a bone conduction microphone) capable of collecting sound from the inner cavity of the user's ear. For example, the second microphone 150 - 2 may include at least one air conduction microphone and/or at least one bone conduction microphone for detecting voice. The air conduction microphone may detect voice transmitted through air (eg, user's speech) and output a voice signal corresponding to the detected voice. The bone conduction microphone may measure vibration of a bone (eg, skull) caused by a user's voice and output a voice signal corresponding to the measured vibration. A bone conduction microphone may be called a bone conduction sensor or other various names. The voice detected by the air conduction microphone may be a voice mixed with external noise while the user's speech is transmitted through the air. Since the voice detected by the bone conduction microphone is the voice detected according to the vibration of the bone, it may be a voice with little external noise (eg, the effect of noise).

In FIG. 3, it is illustrated that one first microphone 150-1 and one second microphone 150-2 are mounted on the electronic device 101, but are not limited thereto, and the first microphone 150-2, which is an external microphone, 1) and a plurality of second microphones 150 - 2 that are in-ear microphones may be mounted in the electronic device 101 . Although omitted in FIG. 3 , the electronic device 101 may further include an accelerator and a vibration sensor (eg, a voice pickup unit (VPU) sensor) for voice activity detection (VAD).

According to an embodiment, the first electronic device 101 may include the audio module 170 described above with reference to FIGS. 1 and 2 . Descriptions overlapping those described with reference to FIGS. 1 and 2 will be omitted. The first electronic device 101 performs noise processing (eg, noise suppressing process), frequency band adjustment, and Audio signal processing such as gain adjustment may be performed. The configuration of the first electronic device 101 will be described in detail with reference to FIG. 5B. The first electronic device 101 may be referred to as the electronic device 101 in FIGS. 4 to 11 .

According to an embodiment, the electronic device 101 may include a sensor capable of detecting a state in which the electronic device 101 is worn on the user's ear. For example, the sensor may include a sensor capable of detecting a distance to an object (eg, an infrared sensor or a laser sensor) and a sensor capable of detecting contact with an object (eg, a touch sensor). As the electronic device 101 is worn on the user's ear, the sensor may detect a distance from or contact with the skin to generate a signal. The processor 120 of the electronic device 101 may recognize whether the electronic device 101 is currently worn by detecting a signal generated by a sensor.

According to an embodiment, the second electronic device 102 forms a communication channel with the first electronic device 101, transmits a designated audio signal to the first electronic device 101, or An audio signal can be received from For example, the second electronic device 102 may form a communication channel (eg, a wired or wireless communication channel) with the first electronic device 101, such as a portable terminal, a terminal device, a smartphone, a tablet PC, and pads. , may be various electronic devices such as wearable electronic devices. The second electronic device 102 may include components identical to or corresponding to those of the electronic device 101 described above with reference to FIG. 1 , and include fewer or more components than the electronic device 101 of FIG. 1 depending on implementation. can do. The second electronic device 102 may be referred to as the electronic device 102 in FIGS. 4 to 11 .

According to an embodiment, in the audio signal processing system 10, the first electronic device 101 may perform beamforming to obtain an enhanced user voice signal. For example, the first electronic device 101 may perform beamforming on external sound signals received through the plurality of microphones 150-1 and 150-2. A beamformer performing beamforming according to various embodiments will be described in detail with reference to FIG. 4 .

4 is a block diagram for explaining a beamformer according to various embodiments.

Referring to FIG. 4 , a block diagram of a signal-dependent beamformer according to an example is shown. Referring to FIG. 4, beamforming and types of beamformers will be briefly described.

는 음성 신호

와 노이즈 신호

로 구성되어,

로 표현될 수 있다. 방향성이 있는 신호

는 복수의 마이크에 각각 다른 위상을 가지고 유입되고, 위상 차이는 음원의 위치에 따라 결정된다. 일반적으로 음성 신호와 노이즈 신호의 음원 위치가 다르므로, 아래 [수학식 1]과 같이 M개의 마이크 중 첫번째 마이크로 입력된 음성 신호와 i번째 마이크와의 음성 신호의 위상 차이

와, 첫번째 마이크로 입력된 노이즈 신호와 i번째 마이크의 노이즈 신호의 위상 차이

가 서로 다를 수 있다.As shown in FIG. 4, the signal coming into the ith microphone among the M microphones

is a voice signal

and noise signal

consists of,

can be expressed as a directional signal

is introduced into a plurality of microphones with different phases, and the phase difference is determined according to the position of the sound source. In general, since the position of the sound source of the voice signal and the noise signal is different, as shown in [Equation 1] below, the phase difference between the voice signal input to the first microphone among the M microphones and the voice signal between the ith microphone

And, the phase difference between the noise signal input to the first microphone and the noise signal of the ith microphone

may be different from each other.

는 아래 [수학식 2]와 같이 표현될 수 있다.Beamforming technology is a technology that improves the characteristic value of a voice signal, for example, signal-to-noise ratio (SNR), by compensating for the phase difference of input signals input to different microphones, and outputs it through beamforming. output signal

Can be expressed as in [Equation 2] below.

는 음성 신호에 대한 위상 차이를 보상하기 위한 것으로, 빔포머 벡터, 빔포머 필터 등으로 지칭될 수 있다.

에 의해 노이즈 성분이 상쇄되고, 음성 성분 대비 노이즈 성분의 크기가 상대적으로 작아질 수 있으므로, 음성 신호의 SNR이 향상될 수 있다.

is for compensating for a phase difference with respect to a voice signal, and may be referred to as a beamformer vector, a beamformer filter, and the like.

Since the noise component is canceled by and the size of the noise component can be relatively small compared to the voice component, the SNR of the voice signal can be improved.

를 구하는 방법에 따라, 신호 독립 빔포머에는 DSBF(Delay-and-sum Beamformer), GSC(Generalized sidelobe canceller) 빔포머, MVDR(Minimum variance distortion response) 빔포머 등이 포함될 수 있다. 음성 신호의 방향 정보와 마이크의 위치 정보를 미리 알 수 있거나, 높은 정확도로 추정할 수 있는 환경(예: 사용자의 위치가 변하지 않는 환경)에서, 신호 독립 빔포머는 SNR에 상관없이 일정한 성능을 유지할 수 있다. 다만 사용자 위치가 변하는 경우 사용자 입의 방향(DoA)을 추정해야 하므로, 노이즈 환경에서 사용자 입의 방향 추정에 오차가 발생하거나 실내에서 반향(reverberation)이 발생하는 경우 신호 독립 빔포머의 성능이 저하될 수 있다.Beamformers may be classified into signal-independent beamformers and signal-dependent beamformers depending on whether characteristics of an input signal are used. The signal-independent beamformer performs beamforming by estimating the direction of arrival (DoA) of the voice through localization and compensating for the phase difference between signals input to each microphone using direction information. can do. Beamformer filter to compensate for phase difference

Depending on the method for obtaining , signal independent beamformers may include a delay-and-sum beamformer (DSBF), a generalized sidelobe canceller (GSC) beamformer, a minimum variance distortion response (MVDR) beamformer, and the like. In an environment where the direction information of the voice signal and the position information of the microphone can be known in advance or can be estimated with high accuracy (e.g., an environment where the user's position does not change), the signal independent beamformer maintains constant performance regardless of SNR. can However, since the direction of the user's mouth (DoA) must be estimated when the user's location changes, the performance of the signal independent beamformer may deteriorate if an error occurs in estimating the direction of the user's mouth in a noisy environment or if reverberation occurs indoors. can

A signal-dependent beamformer is a beamformer that performs beamforming based on spatial characteristics of microphone input signals, and FIG. 4 is a block diagram of the signal-dependent beamformer according to an example. In a noisy environment, the input signal of the microphone can be divided into a section where voice and noise are mixed and a section where only noise exists without voice. Cn, which is a noise covariance matrix for noise, can be obtained in a section where only . Cx may include a spatial feature of voice, and Cn may include a spatial feature of noise. Various embodiments in which Cx and Cn are obtained will be described with reference to FIG. 5A.

Based on Cx and Cn, a beamformer filter that steers in the direction of the user's mouth and a beamformer filter that generates a null vector in the direction of the user's mouth can be determined. The signal-dependent beamformer may reduce noise in the voice signal based on a beamformer filter that steers toward the user's mouth, or acquire only noise components based on a beamformer filter that generates a null vector toward the user's mouth.

A method of determining a beamformer filter based on the covariance matrices Cx and Cn may be possible in various embodiments. For example, the MaxSNR beamformer may determine a beamformer filter in a direction that maximizes the SNR of a signal based on Equation 3 below.

의 가장 큰 eigenvalue

를 갖는 eigen vector인

를 빔포머 필터로 결정할 수 있다.MaxSNR beamformer

the largest eigenvalue of

An eigen vector with

can be determined by the beamformer filter.

As another example, an MVDR beamformer (Minimum Variance Distortionless Response Beamformer) may perform beamforming that removes noise while minimizing voice distortion. There may be various ways to obtain a beamformer filter in the MVDR beamformer, and for example, the beamformer filter can be obtained by using the MaxSNR beamformer as shown in [Equation 4] below.

의 null space를 이용하여 GSC(Generalized Sidelobe Canceler)의 BM(Blocking Matrix)

가 아래 [수학식 5]에 기초하여 획득될 수 있다. As another example, obtained from the MaxSNR beamformer

BM (Blocking Matrix) of GSC (Generalized Sidelobe Canceler) using null space of

Can be obtained based on [Equation 5] below.

를 통해 노이즈와 음성이 섞인 신호에서 음성 성분이 제거될 수 있고, 음성 성분이 제거된 신호에 기초하여 노이즈와 음성이 섞인 신호에서 노이즈가 보다 정교하게 제거될 수 있다.

Through this, a voice component can be removed from a signal where noise and voice are mixed, and noise can be more elaborately removed from a signal where noise and voice are mixed based on the signal from which the voice component has been removed.

Unlike the signal independent beamformer, the signal-dependent beamformer has high robustness to the voice direction, but in an environment where the SNR is low, performance may deteriorate due to a decrease in accuracy of the covariance matrix Cx of the voice. A method of detecting voice activity based on an in-ear microphone or vibration sensor and then estimating the location of voice using a signal independent beamformer may be used, but an error in estimating voice location may increase in an environment with a low SNR. In addition, the method of controlling beamforming based on the voice position can only compensate for the deviation due to the wearing angle, and cannot reflect the deviation due to the difference in the user's wearing style or the internal structure of the ear. An ideal beamforming direction and a beamformer filter may be different due to differences in wearing styles or internal structures of ears for each user.

Referring to FIGS. 5A to 11 , a method for preventing performance deterioration due to a difference in a user's wearing style and an internal structure of the ear in a low SNR environment will be described in detail.

5A and 5B are diagrams for explaining a process of performing beamforming and noise processing in the electronic device 101 and a configuration of the electronic device 101 according to an exemplary embodiment.

Referring to FIG. 5A , a noise processing system 500 performing beamforming and noise processing in an electronic device 101 according to an exemplary embodiment is illustrated. As described with reference to FIG. 3 , the electronic device 101 includes an external microphone (eg, the first microphone 150-1 of FIG. 3 ), an in-ear microphone (eg, the second microphone 150-2 of FIG. 3 ) )) and an accelerator 502 may be included.

The beamformer 510 may perform beamforming on an external sound signal received through a plurality of microphones (eg, an external microphone and an in-ear microphone) of the electronic device 101 . The electronic device 101 loads parameters (for example, the speech covariance matrix Cx described with reference to FIG. 4) related to determining the beamformer filter from a memory or stores it in the memory according to the characteristic values of the external sound signal (520). ), the beamformer filter can be determined. An operation in which beamforming is performed according to characteristic values of the external sound signal will be described in detail with reference to FIGS. 6 to 11 .

Voice activity may be detected based on the in-ear microphone and accelerator 502 of the electronic device 101 (530). Based on voice activity detection (VAD), a mask m(t,f) corresponding to the fth frequency bin in the tth frame can be estimated, and the covariance matrix Cx(f) of the voice and noise The covariance matrix Cn(f) of can be determined based on [Equation 6] below. However, the method for obtaining Cx and Cn is not limited to the following [Equation 6], and various generally known methods may be used.

는 t번째 프레임에서 f번째 주파수 빈(bin)에 대응하는 마이크 입력 신호일 수 있다. 도 4를 참조하여 전술한 바와 같이, Cx와 Cn의 대각 행렬 성분에는 각 신호의 크기에 대한 정보가, 대각 행렬이 아닌 성분에는 각 신호의 공간에 대한 정보가 포함될 수 있다. Cx와 Cn은 도 4를 참조하여 전술한 바와 같이 빔포머(410)의 필터를 결정하는 데 사용될 수 있다. At this time,

may be a microphone input signal corresponding to the f-th frequency bin in the t-th frame. As described above with reference to FIG. 4 , diagonal matrix elements of Cx and Cn may include information about the amplitude of each signal, and non-diagonal matrix elements may include information about the space of each signal. Cx and Cn may be used to determine the filter of the beamformer 410 as described above with reference to FIG. 4 .

When the beamformer filter is determined, the residual noise size is estimated according to the result of the VAD 530 (540), and the residual noise can be removed from the beamforming result according to the estimated noise size information (550). A deep neural network (DNN) may be used in the residual noise removal process.

Through the series of operations of the noise processing system 500 described above, the electronic device 101 may output an enhanced voice audio signal. Hereinafter, the configuration of the electronic device 101 will be described with reference to FIG. 5B, and specific operations of the electronic device 101 will be described with reference to FIGS. 6 to 11.

5B is a block diagram for explaining a configuration of an electronic device 101 according to an exemplary embodiment.

The electronic device 101 of FIG. 5B may be the first electronic device 101 described above with reference to FIG. 3 , and the electronic device 102 of FIG. 5B may be the second electronic device 102 described above with reference to FIG. 3 . .

The electronic device 101 includes an input module 150 for receiving ambient sound, a sound output module 155 for outputting sound obtained by processing the ambient sound, an audio module 170 for processing ambient sound, and a computer. A memory 130 in which computer-executable instructions and voice parameter initial value information 580 are stored, and a processor 120 that accesses the memory 130 and executes the instructions may be included. An electronic device 101, an electronic device 102, a processor 120, a memory 130, an input module 150, an audio output module 155, an audio module 170, and a communication module 190 are shown in FIGS. The electronic device 101, the electronic device 102, the processor 120, the memory 130, the input module 150, the audio output module 155, the audio module 170, and the communication module described above with reference to FIG. 4 (190), and duplicate descriptions are omitted. As described above with reference to FIG. 3 , the electronic device 101 may be an audio output device such as a wireless earphone, and the electronic device 102 may be an electronic device that transmits and receives audio signals to and from the electronic device 101 such as a smartphone. .

The processor 120 may estimate a characteristic value of an external sound signal received through the input module 150 . For example, the processor 120 may estimate characteristic values of external sound signals received through a plurality of microphones (eg, the external microphone 150-1 and the in-ear microphone 150-2 of FIG. 3 ). Hereinafter, the characteristic values are described based on the SNR, but are not limited thereto, and noise power of an external sound signal may be used as the characteristic value, for example.

The processor 120 calculates an initial value of a voice parameter for beamforming based on an external sound signal, and determines whether to store the initial value calculated according to the characteristic value in the initial value information 580 of the voice parameter of the memory 130. Then, it is possible to determine which initial value among the calculated initial value or the stored initial value is to be used, and obtain a voice parameter based on the determined initial value. The speech parameter may be a speech covariance matrix Cx used to determine a beamformer filter in the signal dependent beamformer described with reference to FIG. 4 .

According to an embodiment, a program (eg, the program 140 of FIG. 1 ) adjusting a beamformer filter according to a characteristic value of an external sound signal to obtain an enhanced voice signal may be stored in the memory 130 as software. there is. A specific operation of the processor 120 will be described in detail with reference to FIGS. 6 to 11 .

<Operation method of electronic device>

6 is a flowchart illustrating an operating method of an electronic device according to various embodiments.

Operations 610 to 650 may be performed by the processor 120 of the electronic device 101 described above with reference to FIG. 5B, and for the sake of a concise description, the contents overlap with those described with reference to FIGS. 1 to 5 may be omitted. Operations 610 to 650 may correspond to the beamforming operation 510 described above with reference to FIG. 5A and the operation 520 of loading or storing parameters related to a beamformer filter.

According to an embodiment, in operation 610, the processor 120 may estimate a characteristic value of an external sound signal. External sound signals may be received through the input module 150 described above with reference to FIG. 5B, for example, the plurality of microphones 150-1 and 150-2 of FIG.

According to an embodiment, the processor 120 processes an external sound signal received through a main microphone among a plurality of microphones mounted on the electronic device 101 (eg, a microphone closest to a mouth among a plurality of external microphones). The characteristic value of can be estimated. The characteristic value may be SNR or noise power. The processor 120 may estimate the feature value using the voice activity detection 530 technique described above with reference to FIG. 5A. For example, as described above with reference to FIG. 5A , the processor 120 may more accurately estimate the SNR of an external sound signal by estimating the SNR when voice activity is detected based on a vibration sensor, an accelerator, an in-ear microphone, or the like. there is.

According to an embodiment, in operation 620, the processor 120 may calculate an initial value of an audio parameter for beamforming of an external sound signal. The processor 120 may calculate an initial value of the voice covariance matrix Cx described above with reference to FIGS. 4 and 5A based on signals input by a plurality of microphones. For example, the processor 120 may obtain an initial value of a voice covariance matrix between audio signals input to each of a plurality of external microphones. In another example, the processor 120 receives an audio signal input to an external microphone (eg, the external microphone 150-1 of FIG. 3) and an internal microphone (eg, the in-ear microphone 150-2 of FIG. 3). An initial value of a speech covariance matrix between audio signals may be obtained.

According to an embodiment, in operation 630, the processor 120 may determine whether to store the calculated initial value of the voice parameter according to the characteristic value. When the SNR of the external sound signal exceeds the first threshold (eg, 15 dB), the processor 120 stores the calculated initial value of the speech covariance matrix in the initial speech parameter information 580 of the memory 130. can When the SNR of the external sound signal is less than or equal to the first threshold (eg, 15 dB), the processor 120 does not store the calculated initial value of the speech covariance matrix in the initial speech parameter information 580 of the memory 130. can decide not to.

When the SNR exceeds, for example, 15 dB, a highly accurate speech covariance matrix can be estimated with an initial value of the speech covariance matrix because the SNR is high, and beamforming performance can be maintained. An operation of the processor 120 when the characteristic value of the external sound signal exceeds the first threshold will be described in detail with reference to FIG. 9 .

According to an embodiment, in operation 640, the processor 120 may determine a voice parameter to be obtained based on which initial value among the calculated initial value and the stored initial value is determined according to the characteristic value. The processor 120 uses the initial value of the voice parameter calculated in operation 620 if the SNR of the external sound signal exceeds the second threshold (eg, 5 dB), and if it is less than the second threshold, the memory 130 It may be determined to load and use the initial values stored in the voice parameter initial value information 580 .

When the SNR exceeds, for example, 5 dB, the processor 120 may obtain the target speech parameter by updating based on the calculated initial value. When the SNR is, for example, 5 dB or less, a speech covariance matrix with high accuracy can be estimated (or updated) if a good initial value is given, and beamforming performance can be maintained. When the SNR is 5 dB or less, the processor 120 loads the initial value of the speech covariance matrix stored when the SNR exceeds 15 dB (eg, the initial value stored in operation 630), and updates the speech covariance matrix based on this. A target speech covariance matrix used for determining the beamformer filter may be determined. An operation of the processor 120 when the characteristic value of the external sound signal is equal to or less than the first threshold value and exceeds the second threshold value will be described in detail with reference to FIG. 10 . An operation of the processor 120 when the characteristic value of the external sound signal is equal to or less than the second threshold and exceeds the third threshold will be described in detail with reference to FIG. 11 .

According to an embodiment, in operation 650, the processor 120 may obtain a target voice parameter for beamforming based on the determined initial value according to the characteristic value. If the SNR of the external sound signal exceeds the third threshold (eg, -5 dB), the processor 120 loads as described with reference to operation 640 or updates the speech covariance matrix based on the calculated initial value. A target speech covariance matrix for determining a beamformer filter may be determined.

The processor 120 uses the initial value stored in the initial voice parameter information 580 of the memory 130 as a voice parameter without an update process when the SNR of the external sound signal is less than or equal to the third threshold (eg, -5 dB). can decide to do it. When the SNR is, for example, −5 dB or less, estimation of the speech covariance matrix may be impossible because the noise component is dominant over the speech component. Therefore, the processor 120 loads good initial values stored in the voice parameter initial value information 580 of the memory 130 when the SNR is -5 dB or less, and the target for determining the beamformer filter without updating the initial values. It can be used as a negative covariant matrix. An operation of the processor 120 when the characteristic value of the external sound signal is equal to or less than the third threshold will be described in detail with reference to FIG. 8 .

A target voice covariance matrix for determining a custom beamformer filter may be determined through operations 610 to 650 of the processor 120, and even in an environment with a low SNR, beamforming performance degradation occurs due to differences in wearing style and internal structure of the ear. may not

In operations 610 to 650 described above, the first threshold value may be greater than the second threshold value, and the second threshold value may be greater than the third threshold value. The first threshold value may be 15 dB, the second threshold value may be 5 dB, and the third threshold value may be -5 dB, but is not limited thereto.

7 is a flowchart illustrating a method of processing noise in an electronic device according to various embodiments of the present disclosure.

Operations 710 to 730 may be performed by the processor 120 of the electronic device 101 described above with reference to FIG. 5B, and for the sake of a concise description, contents overlapping those described with reference to FIGS. 1 to 6 are may be omitted. Operations 710 to 730 may correspond to the noise estimation operation 540 and the noise removal operation 550 after the beamforming described above with reference to FIG. 5A .

According to an embodiment, the processor 120 of the electronic device 101 performs operations 710 to 730 after obtaining a target voice parameter based on a characteristic value of an external sound signal (eg, operation 650 of FIG. 6 ). can do.

According to an embodiment, in operation 710, the processor 120 may determine a beamformer filter for beamforming an external sound signal based on target voice parameters acquired through operations 610 to 650 of FIG. 6 . For example, the processor 120 may obtain a highly accurate target speech covariance matrix Cx by performing the above-described operation with reference to FIG. 6, and obtain the target covariance matrix Cx as described above with reference to FIGS. 4 and 5A. through the beamformer filter. For example, as described above with reference to FIG. 4 , the processor 120 may determine a beamformer filter that steers in the direction of the user's mouth or a beamformer filter that generates a null vector in the direction of the user's mouth based on the obtained Cx. .

According to an embodiment, in operation 720, the processor 120 estimates a residual noise level for the beamformed signal with the determined filter, and in operation 730, performs noise processing on the beamformed signal according to the residual noise level (eg, noise suppression).

The electronic device 101 may obtain a voice signal of improved quality by determining a beamformer filter through operations 710 to 730 and performing noise processing based on the target voice covariance matrix, which is a target voice parameter.

According to another embodiment, the electronic device 101 may perform a separate training mode for acquiring good initial values of voice parameters. The processor 120 calculates the initial value of the voice parameter to be stored in the voice parameter initial value information 580 of the memory 130 in an environment where the characteristic value (eg SNR) exceeds the first threshold value (eg 15 dB). To do so, a guide interface requesting utterance to the user may be output. For example, an environment with a high characteristic value (eg, a quiet environment) through a display of the electronic device 101 or the electronic device 102 (eg, the electronic device 102 of FIG. 3) interlocked with the electronic device 101. ), a UI requesting to be ignited may be output. For example, a UI requesting to utter a specific sentence or an arbitrary sentence in a quiet environment may be audibly output to the user through the electronic device 101 . As another example, a similar UI may be visually output to the user through the electronic device 120 interworking with the electronic device 101 .

As described with reference to FIG. 6 , the processor 120 may calculate an initial value of a voice parameter based on user speech received through a plurality of microphones, and store the calculated initial value of the voice parameter in the memory 130. . The processor 120 calculates an initial value of a voice covariance matrix according to user speech received through a plurality of microphones, and operates the processor 120 when the external sound signal exceeds a first threshold (eg, operation 630 of FIG. 6 ). ), the calculated initial value may be stored in the voice parameter initial value information 580 of the memory 130. According to the above-described training mode embodiment, the processor 120 obtains an initial value that is certainly better than the embodiment of storing an initial value when a characteristic value is high in a general situation, such as operation 630 of FIG. can be stored in

According to another embodiment, the processor 120 of the electronic device 101 may more specifically control beamforming based on the voice volume and voice direction of the external sound signal. For example, the voice parameter initial value information 580 of the memory 130 described above with reference to FIG. 5B may further include index information according to voice direction and voice volume information. The processor 120 may determine speech power and direction of arrival (DoA) of external sound signals input by a plurality of microphones, and may further consider speech power information and speech direction information.

For example, when the processor 120 stores the calculated voice parameter initial value information in the memory 130 (eg, when the characteristic value exceeds the first threshold value in operation 630 of FIG. 6), the loudness and direction of the voice It can be classified according to the information about and stored in the voice parameter initial value information 580 of the memory 130. As another example, the processor 120 loads the voice parameter initial value information stored in the memory 130 rather than the calculated voice parameter initial value information (eg, in operation 640 of FIG. 6, when the characteristic value is less than the second threshold value) When the voice is heard, it can be loaded from the voice parameter initial value information 580 of the memory 130 by classifying it according to the information on the volume and direction of the voice.

8 to 11 are flowcharts for explaining a method of operating an electronic device according to a characteristic value of an external sound signal.

The operations of FIGS. 8 to 11 (eg, operations 810 to 850, 910 to 920, 1010, and 1110) are performed by the processor 120 of the electronic device 101 described above with reference to FIG. 5B. For concise description, contents overlapping those described with reference to FIGS. 1 to 7 may be omitted. The operations of FIGS. 8 to 11 classify the operations (eg, operations 610 to 650) of the processor 120 described above with reference to FIG. 6 according to characteristic values of external sound signals received by a plurality of microphones. can be actions.

As described above with reference to FIG. 6, in FIGS. 8 to 11, the characteristic value may be SNR or noise power, and the voice parameter may be a voice for determining a beamformer filter in the signal-dependent beamformer described above with reference to FIG. 4. It can be a covariance matrix Cx. 8 to 11, the first threshold may be greater than the second threshold, and the second threshold may be greater than the third threshold. The first threshold value may be 15 dB, the second threshold value may be 5 dB, and the third threshold value may be -5 dB, but is not limited thereto.

FIG. 8 is a flowchart illustrating an operation of the processor 120 when a characteristic value of an external sound signal is equal to or less than a third threshold value in the operation of the processor 120 described above with reference to FIG. 6 .

According to an embodiment, in operation 810, the processor 120 may estimate characteristic values of external sound signals received through a plurality of microphones. Since operation 810 overlaps operation 610 described with reference to FIG. 6 , a description thereof is omitted.

According to an embodiment, in operations 820 to 840, the processor 120 may classify characteristic values based on a first threshold value, a second threshold value, and a third threshold value.

According to an embodiment, in operation 850, when the characteristic value of the external sound signal is equal to or less than the third threshold, the processor 120 sets the initial value of the voice parameter stored in the memory 130 as a voice parameter for beamforming the external sound signal. can be obtained

As described above with reference to operation 630 of FIG. 6 , in operation 850, if the SNR of the external sound signal is less than or equal to the third threshold value (eg, -5dB), the processor 120 determines the first threshold value (eg, -5dB). , 15 dB), the initial value calculated based on the external sound signal received by the plurality of microphones (eg, the initial value calculated in operation 620 of FIG. 6) is the voice parameter initial value information of the memory 130 ( 580) may be determined not to be stored.

As described above with reference to operation 640 of FIG. 6, if the SNR of the external sound signal is less than or equal to the third threshold (eg, -5dB), the processor 120 is less than or equal to the second threshold (eg, 5dB). ) is in the initial value information 580 of the memory 130 rather than the initial value calculated based on the external sound signal received by the plurality of microphones (eg, the initial value calculated in operation 620 of FIG. 6). It can be decided to use the stored initial value.

As described above with reference to operation 650 of FIG. 6 , if the SNR of the external sound signal is less than or equal to the third threshold (eg, -5 dB), the processor 120 determines the voice parameter initial value information of the memory 130 ( 580), the initial value may be determined as a target speech parameter for determining the beamformer filter without performing an update on the initial value stored in the . The processor 120 may determine a beamformer filter based on the target speech covariance matrix obtained in operation 850 and perform noise processing.

FIG. 9 is a flowchart illustrating an operation of the processor 120 when a characteristic value of an external sound signal exceeds a first threshold value in the operation of the processor 120 described above with reference to FIG. 6 .

According to an embodiment, the processor 120 estimates a characteristic value of an external sound signal received through a plurality of microphones (eg, operation 810 of FIG. 8 ), and when the characteristic value exceeds a first threshold (eg, operation 810 of FIG. 8 ). In operation 820 of FIG. 8 , Yes) operations 910 and 920 may be performed.

According to an embodiment, in operation 910, the processor 120 may store in the memory 130 an initial value of a voice parameter calculated when a characteristic value of an external sound signal exceeds a first threshold value. As described above with reference to operation 630 of FIG. 6 , in operation 910, the processor 120 performs the external sound received through the plurality of microphones when the SNR of the external sound signal exceeds a first threshold (eg, 15 dB). The initial value calculated based on the signal (eg, the initial value calculated in operation 620 of FIG. 6 ) may be stored in the voice parameter initial value information 580 of the memory 130 .

According to an embodiment, in operation 920, the processor 120 updates the voice parameter based on the calculated initial value of the voice parameter when the characteristic value of the external sound signal exceeds the first threshold, so that the beam of the external sound signal is updated. A target voice parameter for forming may be acquired.

As described above with reference to operation 640 of FIG. 6 , in operation 920, the processor 120 determines that the SNR of the external sound signal exceeds the first threshold value (eg, 15 dB), so that the second threshold value (eg, 15 dB) 5 dB), and it may be determined that the calculated initial value (eg, the initial value calculated in operation 620 of FIG. 6) rather than the stored initial value is used.

As described above with reference to operation 650 of FIG. 6, when the SNR of the external sound signal exceeds the first threshold (eg, 15 dB), it exceeds the third threshold (eg, -5 dB), so the processor ( 120) may obtain a target voice parameter for determining a beamformer filter by performing an update on the calculated initial value. For example, the processor 120 may determine a target speech covariance matrix for determining a beamformer filter by updating an initial value of the calculated speech covariance matrix Cx. The processor 120 may determine a beamformer filter based on the target speech covariance matrix obtained in operation 920 and perform noise processing.

FIG. 10 is for explaining the operation of the processor 120 when the characteristic value of an external sound signal is less than a first threshold value and exceeds a second threshold value in the operation of the processor 120 described above with reference to FIG. 6 It is a flow chart.

According to an embodiment, the processor 120 estimates a characteristic value of an external sound signal received by a plurality of microphones (eg, operation 810 of FIG. If it exceeds (eg, Yes in operation 830 of FIG. 8 ), operation 1010 may be performed.

According to an embodiment, in operation 1010, the processor 120 updates the voice parameter based on the initial value of the calculated voice parameter when the characteristic value of the external sound signal is less than or equal to the first threshold and exceeds the second threshold, , a target voice parameter can be obtained.

As described above with reference to operation 630 of FIG. 6 , in operation 1010, the processor 120 performs the calculated initial value (eg, FIG. It may be determined not to store the initial value calculated in operation 620 of step 6 in the voice parameter initial value information 580 of the memory 130 .

As described above with reference to operation 640 of FIG. 6 , in operation 1010, the processor 120 calculates the initial value instead of the stored initial value because the SNR of the external sound signal exceeds the second threshold value (eg, 5 dB). (For example, the initial value calculated in operation 620 of FIG. 6) may be determined to be used.

As described above with reference to operation 650 of FIG. 6, when the SNR of the external sound signal exceeds the second threshold (eg, 5 dB), it exceeds the third threshold (eg, -5 dB), so the processor 120 ) may obtain a target voice parameter for determining a beamformer filter by performing an update on the calculated initial value. For example, the processor 120 may determine a target speech covariance matrix for determining a beamformer filter by updating an initial value of the calculated speech covariance matrix Cx. The processor 120 may determine a beamformer filter based on the target speech covariance matrix obtained in operation 1010 and perform noise processing.

FIG. 11 is for explaining the operation of the processor 120 when the characteristic value of the external sound signal is less than the second threshold and exceeds the third threshold in the operation of the processor 120 described above with reference to FIG. 6 . It is a flow chart.

According to an embodiment, the processor 120 estimates a characteristic value of an external sound signal received by a plurality of microphones (eg, operation 810 of FIG. If it exceeds (eg, Yes in operation 840 of FIG. 8 ), operation 1110 may be performed.

According to an embodiment, in operation 1110, the processor 120 determines the voice parameters based on the initial values of the voice parameters loaded from the memory when the characteristic value of the external sound signal is less than or equal to the second threshold and exceeds the third threshold. By updating, the target voice parameters can be obtained.

As described above with reference to operation 630 of FIG. 6 , in operation 1110, the processor 120 determines the first threshold value (eg, 15 dB) if the SNR of the external sound signal is less than or equal to the second threshold value (eg, 5 dB). ) or less, it may be determined not to store the calculated initial value (eg, the initial value calculated in operation 620 of FIG. 6 ) in the voice parameter initial value information 580 of the memory 130 .

As described above with reference to operation 640 of FIG. 6, in operation 1110, the processor 120 determines that the stored initial value is used because the SNR of the external sound signal is less than or equal to the second threshold value (eg, 5 dB), An initial value may be loaded from the voice parameter initial value information 580 .

As described above with reference to operation 650 of FIG. 6, if the SNR of the external sound signal exceeds the third threshold (eg, -5 dB), the processor 120 updates the initial value to update the beamformer filter. A target voice parameter to determine may be obtained. For example, the processor 120 may obtain a target speech covariance matrix for determining a beamformer filter by updating an initial value of the speech covariance matrix Cx loaded from the speech parameter initial value information 580 of the memory 130. there is. The processor 120 may determine a beamformer filter based on the target speech covariance matrix obtained in operation 1110 and perform noise processing.

An electronic device 101 according to an embodiment includes an input module 150 including a plurality of microphones 150-1 and 150-2 for receiving external sound signals, computer-executable instructions (computer- A memory 130 in which executable instructions and an initial value 580 of a voice parameter Cx for beamforming of an external sound signal are stored, and a processor accessing the memory 130 to execute instructions ( 120), and the instructions estimate a characteristic value (SNR or noise power) of the external sound signal, and an initial value of a voice parameter for beamforming based on the external sound signal received by a plurality of microphones. Calculate , determine whether to store the initial value calculated according to the characteristic value in the memory 130, determine which initial value of the initial value calculated according to the characteristic value or the initial value stored in the memory 130 is to be used, It may be configured to acquire a target voice parameter for beamforming of an external sound signal based on an initial value determined according to the characteristic value.

According to one embodiment, the instructions may be further configured to determine a filter for beamforming of the external sound signal based on the target speech parameter.

According to an embodiment, the electronic device 101 further includes an audio module 170 that performs noise processing on an audio signal, and instructions include estimating a residual noise level for a signal beamformed with a filter, and estimating It may be further configured to perform noise processing on the beamformed signal according to the size of the residual noise.

According to one embodiment, the commands store the initial value of the calculated voice parameter in the memory 130 when the characteristic value exceeds the first threshold (eg, 15 dB), and based on the initial value of the calculated voice parameter. It may be configured to obtain a target voice parameter by updating the voice parameter with .

According to one embodiment, the instructions update the voice parameter based on the initial value of the calculated voice parameter when the characteristic value is less than or equal to a first threshold value (eg, 15 dB) and exceeds a second threshold value (eg, 5 dB). By doing so, it may be configured to obtain a target voice parameter.

According to an embodiment, the commands are based on the initial value of the voice parameter stored in the memory 130 when the characteristic value is less than or equal to the second threshold (eg, 5dB) and exceeds the third threshold (eg, -5dB). It may be configured to obtain a target voice parameter by updating the voice parameter with .

According to one embodiment, the commands may be configured to acquire an initial value of a voice parameter stored in the memory 130 as a target voice parameter if the characteristic value is less than or equal to a third threshold (eg, -5dB).

According to an embodiment, the commands output a guide interface requesting speech from the user to calculate initial values of voice parameters to be stored in the memory 130, and voice based on the user speech received through a plurality of microphones. It may be further configured to calculate initial values of parameters and store the calculated initial values of voice parameters in memory 130 .

According to one embodiment, the initial value 580 of the voice parameter stored in the memory 130 is classified according to the magnitude of the sound signal, the instructions calculate the magnitude of the external sound signal, and the characteristic value is set to a second threshold ( Example: 5 dB) or less, it may be configured to load an initial value stored in the memory 130 according to the level of the external sound signal, and acquire a target voice parameter based on the loaded initial value.

According to one embodiment, the initial value of the voice parameter stored in the memory 130 is classified according to the direction of the sound signal, the instructions determine the direction of the external sound signal, and the characteristic value is set to a second threshold value (eg, 5 dB). ) or less, load the initial value stored in the memory 130 according to the direction of the external sound signal, and acquire a target voice parameter based on the loaded initial value.

According to an embodiment, the characteristic value may be any one of a signal-to-noise ratio (SNR) value and noise power, and the speech parameter may be a speech covariance vector for speech.

According to an embodiment, the plurality of microphones include an external microphone 150-1 located on one side of the electronic device 101, and the electronic device 101 includes an in-ear microphone 150-2. and an accelerator 502 may be further included.

According to an embodiment, the electronic device 101 may be any one of true-wireless stereo (TWS), headphones, and headsets.

A method for obtaining voice parameters for beamforming by the electronic device 101 according to an embodiment includes an operation of estimating characteristic values of external sound signals received through a plurality of microphones, and beamforming of external sound signals. An operation of calculating an initial value of a voice parameter for , an operation of determining whether to store an initial value calculated according to a characteristic value, an operation of determining whether to use an initial value of an initial value calculated according to a characteristic value or a stored initial value, and acquiring a target voice parameter for beamforming of an external sound signal based on an initial value determined according to the characteristic value.

According to an embodiment, the operation of determining whether to store the initial value calculated according to the characteristic value includes: storing the initial value of the calculated voice parameter when the characteristic value exceeds a first threshold value (eg, 15 dB); and If the characteristic value is less than or equal to the first threshold value (eg, 15 dB), an operation of not storing the calculated initial value of the voice parameter may be included.

According to an embodiment, the operation of determining which of the initial value calculated according to the characteristic value or the stored initial value to use is the calculated initial value when the characteristic value exceeds the second threshold value (eg, 5 dB). An operation of using, and an operation of using the stored initial value when the characteristic value is less than or equal to the second threshold (eg, 5 dB) may be included.

According to an embodiment, the operation of obtaining a target voice parameter for beamforming of an external sound signal based on an initial value determined according to a characteristic value is determined when the characteristic value exceeds a third threshold value (eg -5dB). An operation of obtaining a target voice parameter by updating the voice parameter based on an initial value, and an operation of acquiring the determined initial value as the target voice parameter when the characteristic value is less than or equal to a third threshold (eg, -5 dB). .

According to an embodiment, an operating method of the electronic device 101 includes an operation of estimating characteristic values of external sound signals received through the plurality of microphones 150-1 and 150-2, and An operation of calculating an initial value of a voice parameter for beamforming, an operation of determining whether to store an initial value calculated according to a characteristic value, and an operation of determining whether to use an initial value of an initial value calculated according to a characteristic value or a stored initial value operation, obtaining a target voice parameter for beamforming of an external sound signal based on an initial value determined according to a characteristic value; determining a filter for beamforming of an external sound signal based on the target voice parameter; An operation of estimating a residual noise level of the beamformed signal, and an operation of performing noise processing on the beamformed signal according to the estimated residual noise level.

According to an embodiment, the electronic device 101 may be any one of true-wireless stereo (TWS), headphones, and headsets.

101: electronic device
120: processor
130: memory

Claims (20)

In electronic devices,
an input module including a plurality of microphones for receiving external sound signals;
a memory in which computer-executable instructions and initial values of voice parameters for beamforming of the external sound signal are stored; and
A processor that accesses the memory and executes the instructions
including,
These commands are
Estimating a characteristic value of the external sound signal;
Calculate initial values of voice parameters for beamforming based on external sound signals received by the plurality of microphones;
Determine whether to store the calculated initial value in the memory according to the characteristic value;
determining which of the calculated initial value or the initial value stored in the memory is to be used according to the characteristic value;
Obtaining a target voice parameter for beamforming of the external sound signal based on the initial value determined according to the characteristic value
configured to
electronic device.
According to claim 1,
These commands are
Determining a filter for beamforming of the external sound signal based on the target speech parameter
further configured to
electronic device.
According to claim 2,
The electronic device further includes an audio module that performs noise processing on an audio signal,
These commands are
For a signal beamformed with the filter, a residual noise magnitude is estimated,
Noise processing is performed on the beamformed signal according to the estimated residual noise level.
further configured to
electronic device.
According to claim 1,
These commands are
If the characteristic value exceeds the first threshold value,
Storing initial values of the calculated voice parameters in the memory;
Obtaining the target voice parameters by updating voice parameters based on the initial values of the calculated voice parameters
configured to
electronic device.
According to claim 1,
These commands are
If the characteristic value is less than or equal to the first threshold and exceeds the second threshold,
Obtaining the target voice parameters by updating voice parameters based on the initial values of the calculated voice parameters
configured to
electronic device.
According to claim 1,
These commands are
If the characteristic value is less than or equal to the second threshold and exceeds the third threshold,
Obtaining the target voice parameters by updating voice parameters based on initial values of voice parameters stored in the memory
configured to
electronic device.
According to claim 1,
These commands are
If the characteristic value is less than or equal to the third threshold value,
Obtaining the initial value of the voice parameter stored in the memory as the target voice parameter
configured to
electronic device.
According to claim 1,
These commands are
To calculate initial values of voice parameters to be stored in the memory, outputting a guide interface requesting speech to the user;
Calculate an initial value of a voice parameter based on user speech received by the plurality of microphones;
Store the initial values of the calculated voice parameters in the memory
further configured to
electronic device.
According to claim 1,
The initial values of the voice parameters stored in the memory are classified according to the magnitude of the sound signal;
These commands are
Calculate the magnitude of the external sound signal;
If the characteristic value is less than or equal to a second threshold, loading an initial value stored in the memory according to the magnitude of the external sound signal;
Obtaining the target voice parameter based on the loaded initial value
configured to
electronic device.
According to claim 1,
The initial values of the voice parameters stored in the memory are classified according to the direction of the sound signal;
These commands are
determine the direction of the external sound signal;
If the characteristic value is less than or equal to a second threshold, loading an initial value stored in the memory according to the direction of the external sound signal;
Obtaining the target voice parameter based on the loaded initial value
configured to
electronic device.
According to claim 1,
The characteristic value is any one of a signal-to-noise ratio (SNR) value and noise power,
The speech parameter is a speech covariance vector for speech,
electronic device.
According to claim 1,
The plurality of microphones include an external microphone located on one side of the electronic device,
The electronic device further comprises an in-ear microphone and an accelerator,
electronic device.
According to claim 1,
The electronic device,
Any one of a true-wireless stereo (TWS), a headphone, and a headset,
electronic device.
A method for obtaining voice parameters for beamforming,
estimating characteristic values of external sound signals received by a plurality of microphones;
calculating initial values of voice parameters for beamforming of the external sound signal;
determining whether to store the calculated initial value according to the characteristic value;
determining whether to use the calculated initial value or the stored initial value according to the characteristic value; and
Acquiring a target voice parameter for beamforming of the external sound signal based on the determined initial value according to the characteristic value
including,
method.
According to claim 14,
The operation of determining whether to store the calculated initial value according to the characteristic value,
storing an initial value of the calculated voice parameter when the characteristic value exceeds a first threshold value; and
If the characteristic value is less than or equal to a first threshold, not storing the calculated initial value of the voice parameter
including,
method.
According to claim 14,
The operation of determining which initial value of the calculated initial value or the stored initial value is to be used according to the characteristic value,
using the calculated initial value when the characteristic value exceeds a second threshold; and
Operation of using the stored initial value when the characteristic value is less than or equal to the second threshold value
including,
method.
According to claim 14,
The operation of obtaining a target voice parameter for beamforming of the external sound signal based on the determined initial value according to the characteristic value,
acquiring the target voice parameter by updating the voice parameter based on the determined initial value when the characteristic value exceeds a third threshold; and
Acquiring the determined initial value as the target voice parameter when the characteristic value is equal to or less than a third threshold value
including,
method.
A computer program stored in a medium to execute the method of any one of claims 14 to 17 in combination with hardware.
In the operating method of the electronic device,
estimating characteristic values of external sound signals received by a plurality of microphones;
calculating initial values of voice parameters for beamforming of the external sound signal;
determining whether to store the calculated initial value according to the characteristic value;
determining whether to use the calculated initial value or the stored initial value according to the characteristic value;
obtaining a target voice parameter for beamforming of the external sound signal based on the determined initial value according to the characteristic value;
determining a filter for beamforming the external sound signal based on the target speech parameter;
estimating a residual noise level for a signal beamformed by the filter; and
Performing noise processing on the beamformed signal according to the estimated residual noise level
including,
Methods of operating electronic devices.
According to claim 19,
The electronic device,
Any one of a true-wireless stereo (TWS), a headphone, and a headset,
Methods of operating electronic devices.

Images