CN105073073B - Apparatus and method for for sound visualization and auditory localization - Google Patents
Apparatus and method for for sound visualization and auditory localization Download PDFInfo
- Publication number
- CN105073073B CN105073073B CN201480004961.8A CN201480004961A CN105073073B CN 105073073 B CN105073073 B CN 105073073B CN 201480004961 A CN201480004961 A CN 201480004961A CN 105073073 B CN105073073 B CN 105073073B
- Authority
- CN
- China
- Prior art keywords
- sound
- sound source
- signal
- hearing
- wearer
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
- 238000000034 method Methods 0.000 title claims abstract description 71
- 238000012800 visualization Methods 0.000 title claims abstract description 71
- 230000004807 localization Effects 0.000 title claims abstract description 20
- 230000005236 sound signal Effects 0.000 claims abstract description 50
- 230000000007 visual effect Effects 0.000 claims abstract description 25
- 238000007794 visualization technique Methods 0.000 claims abstract description 5
- 238000001914 filtration Methods 0.000 claims description 20
- 230000013707 sensory perception of sound Effects 0.000 claims description 18
- 210000001508 eye Anatomy 0.000 claims description 15
- 206010011878 Deafness Diseases 0.000 claims description 13
- 210000005252 bulbus oculi Anatomy 0.000 claims description 8
- 239000007787 solid Substances 0.000 claims description 7
- 230000008859 change Effects 0.000 claims description 6
- 238000003759 clinical diagnosis Methods 0.000 claims description 2
- 238000001514 detection method Methods 0.000 claims description 2
- 238000012549 training Methods 0.000 claims description 2
- 230000009466 transformation Effects 0.000 claims description 2
- 210000004556 brain Anatomy 0.000 abstract description 8
- 238000013507 mapping Methods 0.000 abstract description 4
- 238000012545 processing Methods 0.000 description 36
- 238000003860 storage Methods 0.000 description 14
- 238000001228 spectrum Methods 0.000 description 13
- 239000011521 glass Substances 0.000 description 10
- 238000004891 communication Methods 0.000 description 9
- 230000008569 process Effects 0.000 description 9
- 238000010586 diagram Methods 0.000 description 7
- 208000032041 Hearing impaired Diseases 0.000 description 6
- 238000005516 engineering process Methods 0.000 description 5
- 230000008447 perception Effects 0.000 description 5
- 230000033001 locomotion Effects 0.000 description 4
- 230000003287 optical effect Effects 0.000 description 4
- 230000003190 augmentative effect Effects 0.000 description 3
- 238000013500 data storage Methods 0.000 description 3
- 230000000694 effects Effects 0.000 description 3
- 230000006870 function Effects 0.000 description 3
- 208000016354 hearing loss disease Diseases 0.000 description 3
- 230000008878 coupling Effects 0.000 description 2
- 238000010168 coupling process Methods 0.000 description 2
- 238000005859 coupling reaction Methods 0.000 description 2
- 210000000613 ear canal Anatomy 0.000 description 2
- 230000007340 echolocation Effects 0.000 description 2
- 230000010370 hearing loss Effects 0.000 description 2
- 231100000888 hearing loss Toxicity 0.000 description 2
- 230000006872 improvement Effects 0.000 description 2
- 239000004973 liquid crystal related substance Substances 0.000 description 2
- 238000012986 modification Methods 0.000 description 2
- 230000004048 modification Effects 0.000 description 2
- 210000000162 simple eye Anatomy 0.000 description 2
- 230000000638 stimulation Effects 0.000 description 2
- 238000002604 ultrasonography Methods 0.000 description 2
- 230000005788 Cochlea function Effects 0.000 description 1
- 241000209140 Triticum Species 0.000 description 1
- 235000021307 Triticum Nutrition 0.000 description 1
- 238000013459 approach Methods 0.000 description 1
- 230000005540 biological transmission Effects 0.000 description 1
- 230000015572 biosynthetic process Effects 0.000 description 1
- 239000000571 coke Substances 0.000 description 1
- 238000013461 design Methods 0.000 description 1
- 238000006073 displacement reaction Methods 0.000 description 1
- 230000008921 facial expression Effects 0.000 description 1
- 235000013305 food Nutrition 0.000 description 1
- 238000001093 holography Methods 0.000 description 1
- 230000001771 impaired effect Effects 0.000 description 1
- 239000007943 implant Substances 0.000 description 1
- 238000003780 insertion Methods 0.000 description 1
- 230000037431 insertion Effects 0.000 description 1
- 238000007689 inspection Methods 0.000 description 1
- 230000002452 interceptive effect Effects 0.000 description 1
- 230000000873 masking effect Effects 0.000 description 1
- 239000000463 material Substances 0.000 description 1
- 238000005259 measurement Methods 0.000 description 1
- 230000007246 mechanism Effects 0.000 description 1
- 238000010295 mobile communication Methods 0.000 description 1
- 229920000642 polymer Polymers 0.000 description 1
- 239000000843 powder Substances 0.000 description 1
- 238000007781 pre-processing Methods 0.000 description 1
- 239000000047 product Substances 0.000 description 1
- 230000006798 recombination Effects 0.000 description 1
- 238000005215 recombination Methods 0.000 description 1
- 230000004044 response Effects 0.000 description 1
- 230000014860 sensory perception of taste Effects 0.000 description 1
- 230000004936 stimulating effect Effects 0.000 description 1
- 238000006467 substitution reaction Methods 0.000 description 1
- 239000013589 supplement Substances 0.000 description 1
- 238000012546 transfer Methods 0.000 description 1
- 230000001755 vocal effect Effects 0.000 description 1
- 239000011800 void material Substances 0.000 description 1
Classifications
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04R—LOUDSPEAKERS, MICROPHONES, GRAMOPHONE PICK-UPS OR LIKE ACOUSTIC ELECTROMECHANICAL TRANSDUCERS; DEAF-AID SETS; PUBLIC ADDRESS SYSTEMS
- H04R25/00—Deaf-aid sets, i.e. electro-acoustic or electro-mechanical hearing aids; Electric tinnitus maskers providing an auditory perception
- H04R25/40—Arrangements for obtaining a desired directivity characteristic
- H04R25/407—Circuits for combining signals of a plurality of transducers
-
- G—PHYSICS
- G02—OPTICS
- G02C—SPECTACLES; SUNGLASSES OR GOGGLES INSOFAR AS THEY HAVE THE SAME FEATURES AS SPECTACLES; CONTACT LENSES
- G02C11/00—Non-optical adjuncts; Attachment thereof
- G02C11/06—Hearing aids
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T11/00—2D [Two Dimensional] image generation
- G06T11/60—Editing figures and text; Combining figures or text
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T19/00—Manipulating 3D models or images for computer graphics
- G06T19/006—Mixed reality
-
- G—PHYSICS
- G09—EDUCATION; CRYPTOGRAPHY; DISPLAY; ADVERTISING; SEALS
- G09B—EDUCATIONAL OR DEMONSTRATION APPLIANCES; APPLIANCES FOR TEACHING, OR COMMUNICATING WITH, THE BLIND, DEAF OR MUTE; MODELS; PLANETARIA; GLOBES; MAPS; DIAGRAMS
- G09B19/00—Teaching not covered by other main groups of this subclass
- G09B19/04—Speaking
-
- G—PHYSICS
- G09—EDUCATION; CRYPTOGRAPHY; DISPLAY; ADVERTISING; SEALS
- G09B—EDUCATIONAL OR DEMONSTRATION APPLIANCES; APPLIANCES FOR TEACHING, OR COMMUNICATING WITH, THE BLIND, DEAF OR MUTE; MODELS; PLANETARIA; GLOBES; MAPS; DIAGRAMS
- G09B5/00—Electrically-operated educational appliances
- G09B5/02—Electrically-operated educational appliances with visual presentation of the material to be studied, e.g. using film strip
-
- G—PHYSICS
- G10—MUSICAL INSTRUMENTS; ACOUSTICS
- G10L—SPEECH ANALYSIS TECHNIQUES OR SPEECH SYNTHESIS; SPEECH RECOGNITION; SPEECH OR VOICE PROCESSING TECHNIQUES; SPEECH OR AUDIO CODING OR DECODING
- G10L21/00—Speech or voice signal processing techniques to produce another audible or non-audible signal, e.g. visual or tactile, in order to modify its quality or its intelligibility
- G10L21/06—Transformation of speech into a non-audible representation, e.g. speech visualisation or speech processing for tactile aids
- G10L21/10—Transforming into visible information
- G10L21/14—Transforming into visible information by displaying frequency domain information
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04R—LOUDSPEAKERS, MICROPHONES, GRAMOPHONE PICK-UPS OR LIKE ACOUSTIC ELECTROMECHANICAL TRANSDUCERS; DEAF-AID SETS; PUBLIC ADDRESS SYSTEMS
- H04R3/00—Circuits for transducers, loudspeakers or microphones
- H04R3/005—Circuits for transducers, loudspeakers or microphones for combining the signals of two or more microphones
Landscapes
- Engineering & Computer Science (AREA)
- Physics & Mathematics (AREA)
- Health & Medical Sciences (AREA)
- General Physics & Mathematics (AREA)
- General Health & Medical Sciences (AREA)
- Acoustics & Sound (AREA)
- Otolaryngology (AREA)
- Theoretical Computer Science (AREA)
- Signal Processing (AREA)
- Business, Economics & Management (AREA)
- Educational Technology (AREA)
- Audiology, Speech & Language Pathology (AREA)
- Educational Administration (AREA)
- Ophthalmology & Optometry (AREA)
- Optics & Photonics (AREA)
- General Engineering & Computer Science (AREA)
- Computational Linguistics (AREA)
- Computer Graphics (AREA)
- Computer Hardware Design (AREA)
- Entrepreneurship & Innovation (AREA)
- Software Systems (AREA)
- Data Mining & Analysis (AREA)
- Neurosurgery (AREA)
- Quality & Reliability (AREA)
- Human Computer Interaction (AREA)
- Multimedia (AREA)
- Circuit For Audible Band Transducer (AREA)
- Stereophonic System (AREA)
- User Interface Of Digital Computer (AREA)
Abstract
Present applicant proposes for handling voice signal, positioning corresponds to the voice signal of one or more sound sources, and in the acoustics visualization method and equipment for now corresponding to localization of sound source in wearable display equipment.Wearable visualization device may include the microphone that two or more are used to detect the sound issued from one or more sound sources, and for being chosen as showing that acoustics visually shows equipment in a manner of three-dimensional.By handling by voice signal that microphone record to position the voice signal for corresponding to given sound source, and handle location sound signal to identify sound source position, it can be with localization of sound source.Acoustics visualization can be such that frequency domain visualizes, and may include the mapping from frequency to color.The acoustics visualization device and method that the application proposes can help to train human brain that sound visual signals are interpreted as voice signal itself.
Description
The cross reference of related application
It is entitled this application claims enjoying in the U.S. Provisional Patent Application No. 61/756535 submitted on January 25th, 2013
The priority of " visualization of the colored localization of sound source in real time of 3D ", entire contents are hereby incorporated by reference.
Background technique
Present application relates generally to hearing-aid devices and augmented reality equipment.
There is 80,000 people in Canada, and the U.S. has more than 1,000,000 people, and the people that the whole world has more than 20,000,000 has been
It becomes deaf entirely.For these people, conventional hearing-aid device is not helpful.Cochlear implant be to some it is effective,
But they are that insertion is intracorporal, and operating cost is very high.
5 percent world population suffers from the puzzlement of hearing loss.Existing hearing-aid device can mention in loud environment
The performance of confession is poor, especially when user wishes that performance is poor when hearing the remote sound source for short distance sound source.
Summary of the invention
Present applicant proposes a kind of for handling voice signal, positioning correspond to one or more sound sources voice signal,
And in the acoustics visualization method and equipment of sound source after now corresponding to positioning in wearable display equipment.It is wearable visual
Changing equipment may include that two or more are used to detect the microphone of the sound from one or more sound sources, and be chosen as with
Three-dimensional mode shows that acoustics visually shows equipment.By handle by voice signal that microphone record with position corresponding to
Determine the voice signal of sound source, and handles the location sound signal to identify sound source position, it can be with localization of sound source.Acoustics visualization can
It also may include the mapping of frequency to color to be frequency domain visualization.
Therefore, on the one hand, present applicant proposes one kind to be presented acoustics visualization method in acoustics visualization device,
Middle acoustics visualization device includes wearable display equipment and two or more microphones, this method comprises:
The voice signal from one or more sound sources is recorded by microphone；
Voice signal is handled, to extract location sound signal relevant at least one sound source；
The acoustic picture of the visable representation of location sound signal is presented in acoustics visualization device.
On the other hand, present applicant proposes a kind of acoustics visualization devices, comprising:
For showing the wearable display equipment to the visible image of at least one eye eyeball of its wearer；
The two or more and associated microphone of wearable display equipment；With
It is connected to the computing hardware of display equipment and microphone, wherein the computing hardware is configured that
The voice signal from one or more sound sources is recorded by microphone；
Voice signal is handled, to extract location sound signal relevant at least one sound source；
The acoustic picture of the visable representation of location sound signal is presented in acoustics visualization device.With reference in detailed below
Description and attached drawing can function to the application and advantageous aspect be further understood.
Detailed description of the invention
Next embodiment will only reference will be made to the accompanying drawings in an illustrative manner, in which:
Fig. 1 shows the example acoustic visualization device including semi-transparent 3D stereoprojection glasses.
Fig. 2 is to be integrated with the example of the earphone of earpiece and microphone.
Fig. 3 shows the exemplary diagram of the processing unit for handling audio and video frequency signal.
Fig. 4 shows the block diagram of exemplary processing unit.
Fig. 5 shows the four module and their mutual relationships and between user of example software system
Relationship.
Fig. 6 is the flow chart of the exemplary execution of auditory localization module.
Fig. 7 A shows tool, and there are three the example acoustic visualization devices of the microphone for auditory localization.
Fig. 7 B shows three spatial relationships between microphone and acoustic sound source.
Fig. 7 C is the flow chart for the illustrative methods of sound positioning.
Fig. 8 is the flow chart of the exemplary execution of Acoustic focusing/filtering module.
Fig. 9 shows the diagram of Acoustic focusing/filtering module example and algorithm, including the frequency determined by sound source position
It is grouped, and the focusing/filtering for the sound source selected based on user.
Figure 10 is the flow chart of the exemplary execution of acoustics visualization model.
Figure 11 shows the diagram of the exemplary output of acoustics visualization model.
Figure 12 is the flow chart for illustrating the exemplary execution of 3D stereoscopic display and subscriber interface module.
Figure 13 shows the exemplary realization side presented by solid performed by 3D stereoscopic display and subscriber interface module
Formula.
Specific embodiment
The various embodiments and various aspects of the application will be described with reference to details discussed below.The following description and drawings
Effect is explained to the application, should not be construed as the limitation to the application.A variety of details are described below to propose to this Shen
The thorough understanding of various embodiments please.However, in some cases, in order to simplify the discussion of embodiments herein, for
Well known and common details is not described.
It is that term " includes " used herein is interpreted to be included and open-ended and inexclusive.Particularly,
When using in the specification and in the claims, term " includes " and their deformation mean to include specific feature, step
Rapid or component.These terms are not construed as excluding the presence of other features, step or component.
" exemplary " meaning of term used in this application is " being used as example, example or illustration ", be should not be construed as opposite
It is preferred or advantageous in other configurations disclosed in the present application.
Term " about " and " approximation " used in this application is used to be covered between the upper limit value of numberical range and lower limit value
Existing variable, such as performance variable, parametric variable and volume variable displacement.In one non-limiting embodiment, term " about " and
" approximation " indicates increase and decrease 10% or less.
Term " basic " used in this application refers to the complete of movement, feature, performance, state, structure, entry or result
Range or degree entirely or almost.For example, " basic " closed object refers to that the object is completely enclosed or almost
Completely enclosed.In some cases, the exact tolerance degree for deviateing absolute integrity depends on specific environment.However, one
As for, the degree of approach of completion is whole result identical with the result obtained when absolute and whole completion in order to obtain.
The use of " basic " is no in completely or almost completely lacking for expression movement, feature, performance, state, structure, entry or result
Determine equally applicable when meaning.
Embodiments herein proposes wearable display equipment, is configured to detect and handles audio signal, and is then in
Now with display audio signal visualization.In some embodiments, audio signal is handled to identify the position of one or more sources of sound
It sets, the position of the source of sound is exposed to equipment wearer over the display.Display equipment can be can for the display in the form of 3D
Virtual reality (or augmented reality) wearable stereoscopic display device depending on changing and positioning audio representation (acoustics visualization).Pass through inspection
The visualization of the audio signal measured, embodiments herein propose can effectively simulate human cochlea's function for by acoustics
The method and apparatus that analog signal is sent to brain.
The method disclosed in the present application and equipment can be interpreted as using with the brain of training of human by acoustic signal is visualized
Thus sound itself restores or facilitates to restore to become deaf or at least part hearing of hearing impaired persons.As hereafter into one
The description of step, present aspects can also help to train that those are become deaf or hearing impaired people says based on visualization sound
Words ability, such as the raising for property accurate in one's statement.Pass through stimulating human view similar by way of sonic stimulation auditory system
The mode of feel system shows that acoustic signal, human brain can more easily receive visual stimulus as receiving auditory stimulation.
Referring now to fig. 1, illustrative acoustics visualization device 10 is shown.Example devices 10 include frame 110, portion
Divide transparent display element 120, Miniature HDMI port 130, the earphone 150 of integrated microphone 140 and 142 and additional Mike
Wind 145.Display element 120 can be refractor, or the optical element substantially without refraction (plane).
In the exemplary embodiment shown in fig.1, equipment 10 is configured to optical perspective and shows equipment, is used for image projection
On eyes of user, while allowing user through the viewing of display element 120 (such as display equipment is at least partly transparent).Example
Such as, which can be formed by light engine (being loaded in frame or on frame), which is optically coupled to shape for display
At in the waveguide in display element 120, wherein the display light being guided is directed toward the eyes of wearer by waveguide, in wearer
Eyes on form image.In the exemplary embodiment shown in Figure 13, left and right display element 120 together provides sound source
3D 3 d visualization (visable representation).
As detailed below, the processing of audio signal may include the positioning of one or more audio signals, the audio signal
The microphone 140,142 and optional additional microphone 145 being arranged on earphone 150 or in earphone 150 (hereinafter can be into one
Step is described in detail) it detects, these microphones are connected to frame 110 by line 170.
The audio signal collected from microphone can be by line (such as voice-frequency cable) or equipment or agreement by wireless communication
(such as bluetooth) is transmitted to processing unit (being discussed further below).
Power supply can be integrated into frame 110 to power to display unit 10.Rechargeable battery can be used
Power supply.
Referring now to Fig. 2, earphone 150 be configurable to include external microphone wind 140 and earplug 160 In-Ear equipment.
By the way that earphone is placed in human ear, microphone 140 can be set by with it is conventional listen attentively to it is identical in a manner of collect sound.One
In a little embodiments, multiple microphones can be set, for example, by using the reflection (such as auricle, cheek) of human body with distinguish from top and/
Or the sound transmitted from below.It should be understood that Fig. 2, which is used to show, uses human ear showing as nature acoustics collection mode
Example property equipment, and can use and place the microphone in ear canal or reach close to the position of ear canal same or similar result
Substitution deformation.
With continued reference to Fig. 1, although the example apparatus 10 shown is the glasses that a pair is adapted to provide for 3D (solid) display, answer
When understanding, scope of the present application to be defined in this exemplary embodiment, not can be used various
The wearable audio/video devices of virtual reality or the helmet in a manner of being suitble to positioning to record audio signal, and by audio signal
Or a part of audio signal carries out suitable visualization display to equipment wearer.In some embodiments, this as shown in Figure 1
The shape of the embodiment of sample, acoustics visualization device is similar to 3D game glasses, which provides hearing disabilities people
Deaf child's acceptable in scholar especially school hides design.
In some embodiments, virtual reality device, which can be, any is suitable for carrying out audio signal record and space is determined
Position, and the wearable display equipment of the space orientation visable representation to equipment wearer display audio signal.For example, at other
In non-limiting alternate embodiment, display equipment can be worn using other virtual realities, such as goggles, the helmet and other are virtual
Real cap or earphone or head-mounted display.For example, the equipment can be the display of the new line with near-to-eye display unit system
System, wherein near-to-eye display unit includes display screen.Other non-limiting example embodiments include display, which includes
Liquid crystal display (LCD), light emitting diode (LEDs), molded polymeric object show (molded polymer display) or freely empty
Between reflection display (free space reflection display) or other image formation mode.Another example is implemented
Example is the wearable computer for including the head-mounted display that Binocular displays or monocular are shown.
In another example embodiment, acoustics visualization device may include at least one holographic display device, this is complete
Breath display equipment provides holographic display on one eye eyeball.Holography by real time, dynamic and it is translucent in a manner of show that 3D is vertical
Body image and/or video provide the visualization and positioning of acoustic signal.
It is not needed in the embodiment for positioning (such as not needing depth information) with 3D in sound source, acoustics visualization device can be with
It is configured to visualize audio signal with simple eye display.For example, in this embodiment, acoustics visualization device can be only included and is used for
The single display equipment or projection device of visualization source of sound are shown on one eye eyeball.
Although the tool of example apparatus 10 is shown in FIG. 1 to be connected thereto there are three microphone, two of them microphone 140
It is arranged on insert earphone, but it is to be understood that microphone can be configured according to extensive configuration, such as microphone quilt
It is suitably arranged on earplug at least two dimensions.In some embodiments, two Mikes with stereo sound configuration are proposed
Wind, for example it is located at the two sides of equipment wearer.Although providing two microphones on insert earphone to be advantageous, because of wheat
Gram wind has used the natural sound capacity gauge of human ear, microphone can be arranged in other positions, be such as but not limited to, can wear
It wears on the glasses of display equipment or the frame of goggles, on headband or cap or on dress ornament (such as waistband).
In some embodiments, wearable device includes one or more additional microphones, and the sum of such microphone is just
It has been more than two.In the embodiment shown in fig. 1, provided with the additional microphone 145 for supporting acoustics positioning.The additional microphone
Allow the acoustics on 3D to position, is satisfactory in the application chosen for being related to the positioning of 3D sound, or allow to be selected
Hearing-impaired people help is provided.
In some embodiments, one or more additional microphones can also will be supplied to processing unit referring to voice signal
(described below) performance objective selects filter algorithm.One or more additional microphones can be located at any appropriate provide and enable
The position of the acoustic information of full 3D acoustics positioning.
In other embodiments, one or more additional microphones can be set in other positions.For example, in a kind of example
In embodiment, two additional microphones be can be set before frame 110.It in some embodiments, can be to disguise oneself as
Multiple microphones are arranged in the mode of the decorative characteristics of equipment 10.
In some embodiments, the processing of the audio of record and the presentation of audio signal visable representation with equipment 10 by connecing
The processing unit of conjunction executes.Referring now to Fig. 3, the example embodiment of processing unit 200 is shown.Exemplary processing unit 200 wraps
Include shell 202, battery or battery pack 204, ON/206 and Miniature HDMI connector 208.Such as system on chip (SoC) is set
The computer or processing hardware setting of standby/circuit are engaged in connector 208 in shell 202.
As described above, processing unit 200 can be by be directly connected to (as by optical cable or cable) or be wirelessly connected with
The standalone computing device that acoustics visualization device 10 engages.In some embodiments, be arranged to can with acoustics for processing unit 200
The dedicated equipment engaged depending on changing equipment 10.In other example embodiments, processing unit can be such as smart phone, plate
The mobile computing device of computer or laptop.In another example embodiment, processing unit 200 can be general-purpose computations
Machine.In another embodiment, processing unit 200 or in which a part can connect to display equipment 10, be formed in display and set
In standby 10 or shown equipment 10 is supported.
Fig. 4 shows the example embodiment for the computing hardware connecting with processing unit 200.As shown, example process list
Member 200 includes one or more processors (CPUs) 422 being connect by bus 424 with mass storage 430.Processing unit
200 further include power supply 426, audio interface 452, video interface 454, optional keypad 456 or other input equipments, can
The display 459 of choosing and optional additional input/output interface 460.
Power supply 426 provides electric power to processing unit 200.Chargeable or non-rechargeable battery can be used to mention
For electric power.It can also be mentioned by such as AC adapter or supplement battery and/or to the external power supply of the charging base of battery charging
For electric power.
As described above, audio interface 452 can be connected to the acoustics visualization device 10 for generating and receiving audio signal.
Video interface 454 is arranged as being engaged in acoustics visualization device 10 and shows generation in acoustics visualization device 10
Video image.For example, as described above, video interface 454 may include one or more with HDMI interface (such as Miniature HDMI
Connector) and support wireless display method (such as WiDi and Miracast).Video interface 454 can also be optionally and for recording
The coupling of the DV of video etc., so as to which acoustics visual image and/or other virtual realities can be superimposed on video
Information or image.
Mass storage 430 includes RAM432, ROM434 and other storage devices.Mass storage 430 is shown
Another computer for being used to store the information such as computer readable instructions, data structure, program module or other data is deposited
The example of storage media.Mass storage 430 be stored with the low-level operation for controlling processing unit 200 it is basic input/it is defeated
System (" BIOS ") 440 out.Mass storage also accommodates the operating system 441 for controlling the operation of processing unit 200.It answers
When will be appreciated that the component may include the version or LINUX or dedicated client communication of the general-purpose operating system such as UNIX
Operating system such as IOS, Android or Windows mobile phone operating system.Operating system, which may include, or interface is connected to lead to
Cross the java virtual machine module of java application control hardware component and/or operating system operation.
Memory 430 further includes one or more data storages 444, and data storage 444 being capable of unit 200 processed
For storing software module and/or other data.For example, it is also possible to store description processing unit using data storage 444
The information of 200 various functions.
Software module 442 may include computer executable instructions, which is held by processing unit 200
When row, execute movement following but not limited to the following: processing receives the audio signal from acoustics visualization device 10 to identify
And/or positioning source of sound, the audio signal received is filtered, generates the visual effect figure of the audio signal received to pass through audio
Visualization device 10 is shown to user.Software module 442 will be described in more detail in Fig. 5.
Optional keypad 456 may include any input equipment for being arranged as receiving from user's input.For example, small key
Disk 456 may include button numeric dial or keyboard.Keypad 456 can also include associated with selecting and sending image
Multiple order buttons.Keypad 456 can be showed virtually by touch-sensitive display screen.
Optional display 459 can be liquid crystal display (LCD), gaseous plasma, light emitting diode (LED) or with
Calculate the display for any other type that equipment is used together.Display 459 can also include be arranged as from such as stylus or
The object of the finger of people receives the touch sensitive display of input.
Processing unit 200 can also include for setting with external equipment not shown in FIG. 4 or other inputs or output
The input/output interface 460 of standby communication.Input/output interface 460 is able to use one or more communication technologys, such as USB, red
Outside line, bluetooth, Wi-Fi, Zigbee etc..
Optional GPS transceiver 464 can determine the physical coordinates of processing unit 200, usually with latitude value and longitude
Value output.GPS transceiver 464 can also use other geo-positioning mechanisms, including but not limited to triangulation, assistant GPS
(AGPS), E-OTD, CI, SAI, ETA, BSS etc., with the physical location on further determining that processing unit 200 at the earth's surface.
It should be understood that at different conditions, GPS transceiver 464 can determine geographical location within the scope of millimeter for processing unit 200；
And in other cases, identified physical location may be less accurate, such as the error range or significantly larger error of meter level.
However, in one embodiment, calculating equipment can provide other by other component and may be used to determine equipment physical location
Information, for example including MAC Address, IP address etc..
Optionally, processing unit 200 can be communicated with base station (not shown), or directly with another computing device communication.
Network interface 462 includes the loop for coupling processing unit 200 on one or more networks, and is configured to and one or more
A communication protocol is used together with technology, and communication protocol and technology include but is not limited to global system for mobile communications (GSM), code point
Multiple access (CDMA), time division multiple acess (TDMA), User Datagram Protocol (UDP), transmission control protocol/Internet protocol (TCP/
IP), the worldwide interoperability for microwave of SMS, general packet radio service technology (GPRS), WAP, ultra wide band (UWB), IEEE 802.16
Access any one of (WiMax), SIP/RTP, bluetooth, infrared ray, Wi-Fi, Zigbee or various other wireless communication protocols.
Network interface 450 is known as transceiver, transceiver or network interface card (NIC) sometimes.
In another embodiment, processing unit 200, which can be played, calculates equipment by network and other Local or Remote
The effect of interactive client device.For example, in one embodiment, processing unit 200 may be configured to support to pass through because
The network interface of special net and remote computing device communication, remotely to execute Audio Signal Processing and/or acoustics visable representation
At least partially.Therefore long-range treated acoustic signal and/or acoustics visual signals or data can be transferred to acoustics
Visualization device is for showing and/or being further processed.
Multiple embodiments of the application can be realized by (one or more) microprocessor and/or memory.For example, can
To be realized above partially by the execution of the hardware logic in microprocessor and in part with instruction in memory is stored in
Described function.Some embodiments are to be stored in the extra instruction of memory without using using microprocessor and realized
's.Some embodiments be using storage in memory for by instruction that (one or more) general purpose microprocessor executes come
It realizes.Therefore, the application is not limited to the specific structure of hardware and/or software.
While some embodiments can be realized in multiple functional computer and computer system, many embodiments are can quilt
Be distributed as it is existing in a variety of manners calculate product, no matter and certain types of machine or the whether practical shadow of computer-readable medium
The distribution is rung, these embodiments can be applied.
At least some aspects of the application can at least partly embody in software.I.e. the technology can be in computer system
Or in response to executing in other data processing systems of its processor, the microprocessor of instruction sequence in memory is such as executed, such as only
Read memory (ROM), volatile Random Access Memory (RAM), nonvolatile memory, caching or remote storage device.
Computer readable storage medium can be used for storing software and data, when the software and data are held by data processing system
When row, system is caused to execute various methods.The executable software and data can store in various positions, including for example read-only
Memory, volatile Random Access Memory, nonvolatile storage and/or caching.Each section of the software and/or data can be deposited
Storage is in either one or two of these storage equipment.Phrase " computer readable-material " used herein and " computer-readable storage medium
Matter " refers to all computer-readable mediums other than instantaneous transmitting signal itself.
Fig. 5 is shown can be between the several software modules and each module of unit 200 processed execution according to process flow
Relationship illustrate block diagram.As shown, the software module includes but is not limited to auditory localization module 480, acoustics visualization mould
Block 482 (such as executing sensual pleasure transfer algorithm), 3D stereo display module 484 and optional Acoustic focusing/filtering module
486.In one embodiment, module 486 can be used as preprocessing module with hearing-aid device (for example, for helping in loud environment
Help the non-personage that becomes deaf by equipment that hearing is concentrated to target) it is used together to realize.Modules and associated example side
Method will be described below.
Referring now to Fig. 6, the process proposed shows the exemplary method for carrying out auditory localization, and this method can pass through
The auditory localization module 480 of Fig. 5 executes.In step 500, the voice signal from three microphones is to be recorded respectively,
And realize by Fast Fourier Transform (FFT) (FFT) operation of the time of the moving window based on time data in set point
Short Time Fourier Transform (STFT) to obtain the real time spectrum of each signal.The frequency spectrum can be shown as the sound under different frequency
Loudness of a sound degree (decibel).
In step 505, the frequency spectrum from one or more microphones is treated as and the intensity of sound under different frequency
It is compared to find one or more frequencies (" most noisy frequency ") with highest intensity of sound." frequency " used herein is suitable
For frequency range, which has associated finite bandwidth.
In some embodiments, the intensity of sound for giving frequency range is stronger (such as more noisy), and the bandwidth of the frequency components is wider.
This is the auditory masking effect in order to imitate mankind's hearing.
It should be understood that two or more most noisy frequencies may be related to identical sound source or different sound sources.It calculates
Position (as described below) afterwards may be used to determine whether sound is generated by a sound source or more than one sound source.
In step 510, the signal for the most strong frequency range that step 505 identifies is obtained and is processed to determine by each microphone
The position (exemplary method is described below referring to Fig. 7 A-7C) of sound source.As shown in step 515, the step of repeating front is to find
The position for the additional frequency bands that step 505 identifies.In one embodiment, the quantity for the frequency range that step 505 identifies can be one, with
Identify single sound source.In other examples, the quantity for the frequency range that step 505 identifies can be two or more, such as
2-10 frequency range.In other embodiments, the quantity for the frequency range that step 505 identifies can be 10 or 10 or more.
Fig. 7 A-7C describes the exemplary execution of the method for location sound.Shown method can be used to carry out
The step 510 of Fig. 6 and 515.
Fig. 7 A shows the exemplary execution of audio visualization equipment 10 (as shown in Figure 1), and which use three Mikes
Wind carries out echolocation.T at any one time0, it is given by the following formula away from sound source distance for the acoustic power W of r:
Or
Wherein a is constant.Therefore, the relational expression of distance r and the intensity of measurement isCorrelation, and sound source is to three
The distance of each of microphone microphone is given by the following formula:
Or
Wherein i={ 1,2,3 } indicates that each microphone in three microphones, b are constants, and WiIt is by giving Mike
The acoustic power (acoustic strength) that wind is measured.As shown in Figure 7 B, due on frame 110 microphone 140,142 and 145 be in not
Same position, propagation duration of the sound from sound source 180 to microphone may be different each microphone.Therefore, for
A given secondary microphone i and j, the distance between microphone and sound source are provided by the time delay between arrival time, as follows
It is shown:
rj-ri=vs(tj-tj)
Wherein vsIt is the aerial speed of sound.
The exemplary method of localization of sound source position is proposed referring now to Fig. 7 C.
As described above, and as shown in the step 525 in Fig. 7 C, three microphone detections to intensity of sound be used to calculate
Distance r1, r2And r3Ratio.
Then the time shift between microphone 1 and microphone 2 is determined in step 530, as described below:
Due to having carried out real-time Fast Fourier Transform (FFT) (FFT), the acoustic strength of each frequency at every point of time
It is available.
To each microphone, calculate in a frequency from T0To maximum thresholding (the i.e. W of next acoustic strengthmax-i)
Time；
The time is denoted as T to microphone 11, T is denoted as to microphone 22。
Therefore | T1-T2| it is the time shift between two microphones.
Then 1 to 3 (r are calculated in step 5351,r2And r3) distance, it is as described below:
Range difference is by r2-r1=vs|T1-T2| --- (2) obtain
Distance r can be obtained by formula (1) and (2)1And r2；
Referring again to formula (1), can by with r1Ratio obtain distance r3；
Therefore the position of sound source passes through distance r1,r2And r3It determines.
Finally, in step 540 calculate sound source position (upper 180) Fig. 7 B, as described below:
Based on preceding method, distance r is obtained1,r2And r3。
Known to the position (coordinate) of microphone (1 to 3) (referring to Fig. 7 B), wherein the position of microphone is shown in glasses 10
On.Such as: the coordinate of microphone 1 to 3 can be (0 ,-y1,0),(0,y1, 0) and (x1,0,0)；
The coordinate of sound source 180 can then be calculated from the information, which proposes 3 containing 3 unknown numbers
Equation:
1) vector (x, y, z)-vector (0 ,-y1, 0) and=r1；
2) vector (x, y, z)-vector (0, y1, 0) and=r2；
3) vector (x, y, z)-vector (x1, 0,0) and=r3；
The coordinate of sound source 180 (x, y, z) can be obtained by solving these equations.
Referring now to Fig. 8, what the process proposed showed that measured audio signal is focused and/or filters shows
Example method.In step 600, (one or more) of sound source is proposed for the frequency (most noisy frequency) with highest acoustic strength
Position.
As shown in step 605, frequency related from different sound source positions is grouped according to their distances relative to each other,
So that the multiple frequencies for being relevant to public sound source position are merged as single sound source.For example, if estimation every two sound
Average distance between source is close to 1 meter, then all frequency/sound sources in 0.5 meter are divided into one group.That is, based on tool
There is the sound source of the space interval of the threshold value less than pre-selection to be divided into one group.It is possible thereby to calculate the new position for merging sound source
It sets.For example, in one embodiment, sound source position can be determined as the position with the frequency of the acoustic signal of maximum intensity.?
In another embodiment, sound source position can be by taking the average value of the related position of different frequency in one group to obtain.
The grouping can have multiple relevant frequencies as shown in figure 9, wherein giving sound source.As shown, can be by true
There is which fixed frequency public sound source position to identify given sound source.In example as shown in the figure, three sound sources, which produce, to be caused
Acoustic signal at the identified sound of eight frequencies and these frequencies is related to the particular sound source based on sound source position.In order into
Row is further processed given sound source, for example, coming from and given sound source phase to generate the time-variable data for visual representation
The frequency-region signal for each frequency closed can be superimposed, and the acoustic signal of time-varying relevant to given sound source can be inverse by carrying out
Fourier transformation (such as inverse FFT) obtains, thus to obtain the when domain representation of positioning signal.It should be noted that the grouping of frequency is also such as
Shown in Fig. 5.
These positions that sound source then can be used are used to handle and present the visable representation of sound, as follows further
It is described.For example, the position of sound source can be proposed to acoustics visualization model 482.
In some embodiments, particular sound source can be identified by user (such as wearer of acoustics visualization device)
Come.For example, in step 610, position/sound source can be easily selected by a user to be absorbed in this referring again to Fig. 8, other sound are come from
Source/position/frequency range signal can be filtered.
As shown in FIG. 8 and 9, the acoustic signal of time-varying by the superposition of frequency component relevant to selected sound source and can be
Domain representation carries out inverse Fourier transform (such as inverse FFT) quilt from original (net) acoustic signal to the frequency component of superposition when acquisition
It filters out.The signal of the time-varying illustrates the filtering version for the voice signal for selecting sound source absorbed based on user.The filtering version
Originally user can be played back to by one or more loudspeakers for being located at (as shown in Figure 1) in earphone 150.
The filter method can be used in various applications, and wherein particular sound source passes through visually based on various sound sources
The user of visable representation is selected, and is then filtered original (total；Merge) voice signal with generate include and selected sound
The filtering voice signal of the relevant sound in source.It is, for example, possible to use such embodiments and its deformation to be used as hearing-aid device, to permit
Perhaps equipment wearer focuses particular sound source and carries out playback to the filtering version of the sound detected and listens to.It should be understood that using
Family or equipment wearer can choose the additional sound source of one or more filterings, so that user obtains based on a variety of sound sources (as talked
Multiple talkers in words) voice signal filtering version.
Referring now to fig. 10, the process proposed shows the example side for handling acoustic signal relevant to given sound source
Method, using can be as acoustics visualization display.It, can for (being positioned according to aforementioned localization method) sound source in step 700
To obtain the frequency data (frequency spectrum) of time-varying.The data are obtained by above-mentioned audio focusing/filter method, and with given position phase
The spectrum component of the multiple frequency range of pass is simply combined.
In step 705, the data of time-varying it is processed with generate be suitable for be plotted as sonograph time-varying indicates (such as 2D seat
Mark system).Can by upper (using suitable time window) at given time point to it is relevant to given sound source when the change of voice
It learns signal and carries out Fast Fourier Transform (FFT), and one group of frequency domain data point is calculated for given time point, to prepare the data,
In each frequency domain data point include frequency logarithm and sound source acoustic signal intensity.That is, the data be stored in it is suitable
For drawing in the format as the sonograph of time-varying, wherein the Y-axis of sonograph is that the logarithm of sound frequency (is chosen as being mapped to
Color, i.e., relevant to color, as indicated by step 710) and X-axis is the time, and the wherein sound of given frequency and time point
The acoustic strength of sound is shown by the brightness of display pixel (pixel).
Sonograph can be drawn according to such as following example embodiment.In given point in time, the first of pixel is perpendicular
Column are drawn, and wherein pixel value is related to the logarithm of sound frequency (being obtained by Fast Fourier Transform (FFT)).As described above, sound audio
Rate can map to color, and each frequency and the acoustic strength on time point can be expressed by the brightness of display pixel
Out.At next time point, another perpendicular column of pixel are plotted in the position adjacent with the first perpendicular column.The treatment process can
In real time repeat or near real-time repeat, and when the new file of frequency data with the time is added into when sonograph can be by moving horizontally
File is drawn.
In an example embodiment, sound frequency (is such as mapped to color (such as from red from 0Hz to close to 44100Hz)
Color is to purple), each frequency is indicated with dedicated color, as indicated by step 710.Method that can be disclosed herein it is each
Using the one-to-one mapping relations of frequency to color in a aspect and its deformation.
In one embodiment, coloration-saturation degree-brightness HSV colour model can be used frequency is mapped in color.
According to an exemplary method, saturation value (such as to 50%) can be fixed；Chromatic value can change with frequency, such as from red
(1Hz sound frequency) arrives purple (44100Hz sound frequency)；Brightness can change with the intensity of sound of specific sound frequency ---
Thus intensity of sound is higher, and color is brighter.
As shown in step 720, for one or more additional sound sources positioned by aforementioned echolocation method, the processing
Process is repeatable.Segments sonogram data relevant to each sound source can export as the real-time video to each sound source position.
Figure 11 is to show acoustics method for visualizing how is realized to generate the diagram of the visable representation of audio signal.
The picture depicts the sonograph of phrase " going to school ", which is from left to right repeated 3 times.It can be seen from the figure that corresponding
Unique and identifiable in the pattern of the phrase, or even although difference present in the pronunciation three times being uttered in phrase very
It is small.In each sonograph, Y-axis is the logarithm (frequency to color is one-to-one mapping) of sound frequency, and X-axis is the time.
Signal is stronger, and the display in pixel is brighter.
Referring now to fig. 12, the process proposed shows the computer visualization table of the audio signal from microphone
Show the exemplary method for carrying out 3D stereoscopic display.In step 750, the position of one or more sound sources is obtained, the sound source position
Aforementioned sound localization method shown in Fig. 6 and Fig. 7 A-C (as used auditory localization module 480 shown in fig. 5) quilt is passed through before
It determines.In step 755, the segments sonogram data for obtaining the time-varying for each sound source (using method described in Fig. 8) is (suitable
Cooperation is that acoustics visualization is presented).
The segments sonogram data of the time-varying of each sound source is then shown in acoustics visualization device in step 766.
For example, the data can be shown as to 2D real-time video, so that the wearer of equipment can either see that the image at scene (such as regards
Frequency signal or the real image transmitted by transparent optical element) it can also be seen that time-varying relevant to one or more sound sources
The visable representation of frequency spectrum.As described above, using the void such as 3D stereoprojection glasses (iGlass and Google glass of such as apple)
The display of sound visable representation may be implemented in quasi- reality or augmented reality equipment.
In some embodiments, the sound visable representation from sound source can be carried out so that visable representation be presented on or
It is projected in the position in the visual field corresponding with sound source position, which is determined by auditory localization.
As shown in the step 765 of Figure 12 and 770, can be in based on the input improvement from user (wearer of such as equipment)
The visualization revealed selects specific sound source in the sound source that wherein user is identified from one group.It can be by various
One of input method makes the selection of user, such as, but not limited to eyeball tracking, finger tip recombination or simple such as mouse/touch
Plate.When receiving from the input for the user for setting sound source, can modify from selected sound source and/or other non-selected sound sources
Sound visable representation.For example, the display of sonograph relevant to selected sound source can amplify.In another embodiment, with it is non-
The display of the selected relevant sonograph of sound source can be reduced.Selected sound source can also be provided to and be used for the acoustics of subsequent processing and gather
Coke/filtering module 486.
Figure 13 is to show how to realize 3D stereo display method so that the visable representation after audio signal calculates is presented
Diagram.The example embodiment that solid is presented is therefore as follows.In step 750 and 755, obtained respectively for each sound source
The segments sonogram data of sound source position and time-varying.
In step 760, acoustics visualization is shown in for each sound source (or sound source group of identification) acoustic picture to set
On standby display screen, wherein each acoustic picture is displayed on visual field position corresponding with the position after sound source calculating.Often
One acoustic picture both provides the visualization display of sound, sonograph as shown in figure 11.To simple eye or eyes (such as solid
Ground), acoustic picture is presented in display equipment (such as 3D display glasses) by the appropriate pixels of identification display, the display is corresponding
In the position of sound source relevant to acoustic picture.
Following example embodiment is proposed for two eyes in the wearer using acoustics visualization device
The visualization method that acoustic picture is presented, referring to Fig.1 shown in 3.For giving sound source, virtual screen 800 is limited at sound source
810 position.The size and location for the acoustic picture that the wearer that the virtual screen corresponds to equipment perceives.To virtual screen
800 each pixel has carried out line between the pixel and (wearer's) right eye center.Line and the right display element phase
The point of friendship is the pixel of acoustic picture, the pixel should be shown with the pixel on virtual screen 800 content.Corresponding to
This same process is repeated on the display element of left eye so that acoustics figure is presented on the display element for corresponding to left eye and right eye
Picture.
Then preceding method can be repeated on the virtual screen for corresponding to other sound sources.The final display of acoustic picture can
To be limited on the display element of the right and left.If there is any overlapping of acoustic picture, can pay the utmost attention to only show one
Acoustic picture.For example, acoustic picture to be shown can be selected according to following example priority scheme: the sound that 1. display users select
Source；2. showing the sound source with " relatively noisy " frequency.
Therefore preceding method proposes spatial registration in the display screen of the left and right of acoustics visualization device two
(registered) with present acoustics visualization, this method by by left eye show and right eye show between difference dressing
3D stereo-picture is generated in the brain of person.It should be understood that stereoprojection can also do not had to, on one eye eyeball selectively
Execute preceding method.
In some embodiments, aforementioned acoustics visualization can be used and positioning device is become deaf for improvement or impaired hearing
The pronunciation of personage.By observing visual sound, lip motion and facial expression simultaneously using above-mentioned translucent display system,
It becomes deaf or hearing impaired persons (can such as be come from by the sound for attempting by the visualization display of their sound and correctly to say
The sound of reference database or sound from the normal people of another hearing) visualization display match, to improve him
Verbal ability.Such application, which can be improved, becomes deaf or ability that hearing impaired persons exchange with Subjects With Normal Hearing scholar.
The some aspects of the application, which also proposed to be directed to, is partially damaged personage (and hearing perfects personage) in loud environment
To the hearing-aid device of positioning and the focusing of target sound source.When compared with conventional hearing-aid device, present embodiment can be helped
Position and be absorbed in interested sound source.
In some embodiments, acoustics visualization device can be dressed by the people with hearing loss, and can show sound
Visualization is learned to substitute or make up the sense of hearing with vision.One people may be to become deaf by clinical diagnosis, then can show that acoustics is visual
Change to substitute the sense of hearing with vision.In some embodiments, acoustics visualization device can be used to train hearing impaired persons will
Sound visualization.These people can also be trained to improve their speech with voice-based visualization using acoustics visualization device
Language ability, the visual representing of the voice are to close relative to the correct sounding that such as the human hair of hearing impairment does not go out, or number
At correct spoken vocabulary reference voice visualization.
For example, in one embodiment, the sight side for the wearer that (passing through any existing eyeball tracking system) detects
To or he/her to the selection of (one or more) sound source being shown in system of the invention (referring to Figure 13) can be used to select
It filters to selecting property other sound sources or amplifies the sound source that user selectes.For example, following eyeball tracking and filter method can be carried out.Sound
It learns visualization device (such as glasses) and shows colored sonograph in each (group) sound source.When detecting direction of visual lines and one of them
When sound source position matches, the acoustic information (in the form of sonograph) from that sound source will be selected.Selected sonograph
(inverse Fourier transform will be utilized) and convert back sound.The sound (original sound of selected sound source) is exported to acoustics visualization device
Earphone (earphone 150 as shown in figure 1) in loudspeaker in, however do not export other sound, therefore filtered out and sight side
To the sound of incoherent position.
In some embodiments, the acoustical frequency spectrum bandwidth of visual acoustic signal is within the scope of human articulation.Example
Such as, acoustical frequency spectrum bandwidth can be in the range of about 300Hz to 3400Hz or a part of its range.In some embodiments
In, the acoustical frequency spectrum bandwidth of visual acoustic signal is in the range of mankind's hearing.For example, acoustical frequency spectrum bandwidth can be big
In the range of about 20Hz to 20kHz or a part of its range.In other examples, acoustical frequency spectrum bandwidth may include or
It is limited to be more than that those speak or the relevant frequency of hearing to the mankind.For example, in some embodiments, acoustical frequency spectrum bandwidth can wrap
Include ultrasound and infrasonic sound (frequency is in about 20Hz sound below).Such embodiment can obtain in military and industrial environment
Using.For example, such acoustical frequency spectrum bandwidth range can permit the wearer of acoustics visualization device in very noisy ring
Ultrasonics, infrasonics are seen in border and/or hear selected audible sound.
It should be understood that the acoustics visualization device and method that propose in the application can help to train human brain by sound
Visual signals are interpreted as sound itself, by means of the visual capacity for the personage that becomes deaf, rebuild their sense of hearing and ability of speaking.
The illustrative potential Military Application of the application will now be described.Afield, usually there is a large amount of noise, and this
The noise of sample can damage the hearing of food for powder.Method and apparatus described herein can be used block their ear with
Protect their hearing；And the same time, sound is shown on their glasses (or helmet) and is helped by their eyes
They understand sound (they need specifically to train before going to the front).In another embodiment, it (can be given birth to some additional hardware
At/receive the specific loudspeaker and specific microphone of ultrasound or infrasonic sound) can provide appear in ultrasonics on battlefield and/
Or the visable representation and filtered audio (selection according to user to sound source) of infrasonics.
When embodiment disclosed above is related to including the visual sample implementation of sound, other embodiments can make
With other non-visual perceptually relevant inputs with one or more, visualization device is perceived to from one or more to utilize
The input visualization (being chosen as the visualization except audio input) of a perception (other perception).For example, as artificial nose and smell
Relevant sensor can be set in or be engaged in perception visualization device.Alternatively or additionally, sensing relevant to the sense of taste
Device can be set in or be engaged in feeling visualization device.In some embodiments, sufficient amount of and given perception is provided
Relevant sensor is so that sound source can be at least identified on two dimensions.
Therefore, in some embodiments, present aspects are considered or play " port " of non-intrusive intracerebral
Or the effect of conduit (not only to sound).Using the principle of this application, wearer, which can be inputted any signal by eyes, is
System (such as sensor) is connected to brain, and their brain is trained to receive the input of the new type as human perception.
Above-mentioned specific embodiment has been illustrated by way of example, and should be understood that these embodiments can allow
There are various modifications and alternative form.It is to be further understood that claim is not intended to limit disclosed particular form, phase
Instead, it is intended to cover all modifications fallen in the application spirit and scope, equivalent and alternative solution.
Claims (22)
1. a kind of filtering audio signal and acoustics visualization method for generating the wearer for hearing-aid device, wearable institute
Stating hearing-aid device includes wearable display equipment, at least two microphones and loudspeaker, which comprises
The voice signal from one or more sound sources is recorded by microphone；
The frequency domain of the voice signal is handled, to extract location sound signal relevant at least one sound source；
The acoustic picture of the visable representation of the location sound signal can be provided by presenting in the wearable display equipment；
The input of the wearer from the wearable display equipment is received, the input is set from the wearable display is presented on
Selected sound source is selected in standby upper multiple identified sound sources；
By being overlapped and executing inverse fourier transform to frequency component relevant to the selected sound source, filtering audio is generated
Signal, the filtering audio signal include that the voice signal from the selected sound source makes the filtering audio signal exclude to come
The voice signal of sound source from except the selected sound source；With
Audio output is played to the wearer of the wearable display equipment using the loudspeaker, the audio output is by described
Audio signal composition is filtered, so that it is described selected to allow the wearer of hearing-aid device to focus on there are multi-acoustical
In sound source.
2. according to the method described in claim 1, further include:
The location sound signal is handled to obtain position relevant to localization of sound source, the localization of sound source is and the positioning sound
The relevant sound source of sound signal；
Wherein, it includes corresponding to the positioning sound in the visual field of the wearer of the hearing-aid device that the acoustic picture, which is presented,
The acoustic picture is presented in the position in source.
3. according to the method described in claim 2, wherein, the acoustic picture corresponds to the positioning of the localization of sound source
The frequency domain representation of voice signal.
4. the method according to shown in claim 3, wherein the acoustic picture is presented on by solid corresponding to the hearing-aid device
Wearer two eyes display element on.
5. according to the method described in claim 3, wherein, the acoustic picture is drawn the location sound using logarithmic scale and is believed
Number frequency.
6. method according to claim 3, wherein dedicated color is associated with to the location sound by the frequency domain representation to be believed
Number each frequency component.
7. according to the method described in claim 5, wherein, the acoustic picture is presented on by solid corresponding to the hearing-aid device
Wearer two eyes display element on.
8. according to the method described in claim 6, wherein, the acoustic picture is presented on by solid corresponding to the hearing-aid device
Wearer two eyes display element on.
9. method according to claim 1 to 8, wherein the hearing-aid device includes three microphones, with energy
The voice signal being enough in three dimensions of record.
10. method according to claim 1 to 8, wherein the hearing-aid device includes two microphones, with energy
The voice signal being enough in two dimensions of record.
11. the method according to any one of claim 2-6, wherein handle the location sound signal to obtain and institute
Stating the relevant position of localization of sound source includes:
Fourier transformation is carried out to the location sound signal；
Intensity based on the signal with multiple frequency bands identifies one or more frequency bands；With
To each identified frequency band:
Inverse Fourier transform is carried out to obtain the when domain representation for the location sound signal for corresponding to the identified frequency band；And
According to the location sound signal reach microphone relative time delay and by the microphone detection to the positioning sound
The relative intensity of sound signal, handle the location sound signal when domain representation to determine sound source position.
12. according to the method for claim 11, wherein identify institute according to the direction of visual lines of the wearer of the hearing-aid device
State selected sound source.
13. according to the method for claim 12, wherein when in the one or more of sound sources recorded by microphone
When at least two sound sources are identified, the method also includes having the space less than pre-selected threshold when the position of at least two sound sources
At least two sound source is merged into one group as single sound source when interval.
14. according to the method for claim 11, wherein when in the one or more of sound sources recorded by microphone
When at least two sound sources are identified, the method also includes having the space less than pre-selected threshold when the position of at least two sound sources
At least two sound source is merged into one group as single sound source when interval.
15. method according to claim 1 to 8, wherein one or more when being recorded by microphone
When at least two sound sources in a sound source are identified, the method also includes being less than pre-selection when the position of at least two sound sources has
At least two sound source is merged into one group as single sound source when the space interval of threshold value.
16. according to the method for claim 15, wherein identify institute according to the direction of visual lines of the wearer of the hearing-aid device
State selected sound source.
17. method according to claim 1 to 8, wherein according to the sight of the wearer of the hearing-aid device
Sound source is selected described in direction discernment.
18. according to the method for claim 17, wherein the hearing-aid device includes two or three microphones.
19. method according to claim 1 to 8, wherein the hearing-aid device is by the people with dysacousis
Group's wearing, and acoustics visualization is rendered as substituting or making up the sense of hearing with vision.
20. according to the method for claim 19, wherein the crowd is to become deaf, and the acoustics is visual by clinical diagnosis
Change is rendered as substituting the sense of hearing with vision.
21. according to the method for claim 20, wherein with the training crowd that sound is visual using the hearing-aid device
Change.
22. one kind for including: for its wearer generation filtering audio signal and the visual hearing-aid device of acoustics, the equipment
For showing the wearable display equipment to the visible image of at least one eye eyeball of its wearer；
At least two with the associated microphone of the wearable display equipment；
Loudspeaker；And
It is connected to the computing hardware of wearable the display equipment and the microphone, wherein the computing hardware is configured that
The voice signal from one or more sound sources is recorded by the microphone；
The frequency domain of the voice signal is handled to extract location sound signal relevant at least one sound source；
It is presented in the wearable display equipment and the acoustic picture of the visable representation of the location sound signal is provided；
The input of the wearer from the wearable display equipment is received, the input is set from the wearable display is presented on
Selected sound source is selected in standby upper multiple identified sound sources；
By being overlapped and executing inverse fourier transform to frequency component relevant to the selected sound source, filtering audio is generated
Signal, the filtering audio signal include that the voice signal from the selected sound source makes the filtering audio signal exclude institute
State the sound source except selected sound source；With
Audio output is played to the wearer of the wearable display equipment using the loudspeaker, the audio output includes institute
Filtering audio signal is stated, so that the wearer of hearing-aid device be allowed to focus on the selected sound there are multi-acoustical
On source.
Applications Claiming Priority (3)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| US201361756535P | 2013-01-25 | 2013-01-25 | |
| US61/756,535 | 2013-01-25 | ||
| PCT/CA2014/050049 WO2014113891A1 (en) | 2013-01-25 | 2014-01-24 | Devices and methods for the visualization and localization of sound |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN105073073A CN105073073A (en) | 2015-11-18 |
| CN105073073B true CN105073073B (en) | 2018-12-07 |
Family
ID=51226786
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201480004961.8A Active CN105073073B (en) | 2013-01-25 | 2014-01-24 | Apparatus and method for for sound visualization and auditory localization |
Country Status (4)
| Country | Link |
|---|---|
| US (1) | US10111013B2 (en) |
| CN (1) | CN105073073B (en) |
| CA (1) | CA2898750C (en) |
| WO (1) | WO2014113891A1 (en) |
Families Citing this family (85)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20130275873A1 (en) | 2012-04-13 | 2013-10-17 | Qualcomm Incorporated | Systems and methods for displaying a user interface |
| US10809966B2 (en) * | 2013-03-14 | 2020-10-20 | Honeywell International Inc. | System and method of audio information display on video playback timeline |
| EA032658B1 (en) * | 2013-07-15 | 2019-06-28 | Вр Электроникс Лимитид | Method for the interactive physiological synchronization of a user with a virtual environment |
| KR101968455B1 (en) | 2013-09-03 | 2019-04-11 | 토비 에이비 | Portable eye tracking device |
| US10686972B2 (en) | 2013-09-03 | 2020-06-16 | Tobii Ab | Gaze assisted field of view control |
| US10310597B2 (en) | 2013-09-03 | 2019-06-04 | Tobii Ab | Portable eye tracking device |
| US9338541B2 (en) * | 2013-10-09 | 2016-05-10 | Voyetra Turtle Beach, Inc. | Method and system for in-game visualization based on audio analysis |
| WO2015195954A1 (en) * | 2014-06-18 | 2015-12-23 | Northwestern University | Systems and methods for neuromodulation device coding with trans-species libraries |
| US9443488B2 (en) | 2014-10-14 | 2016-09-13 | Digital Vision Enhancement Inc | Image transforming vision enhancement device |
| FR3027412A1 (en) * | 2014-10-21 | 2016-04-22 | Arch | BRANCH FOR EYEWEAR. |
| WO2016088922A1 (en) * | 2014-12-05 | 2016-06-09 | 엘지전자 주식회사 | Method for providing interface using mobile device and wearable device |
| US11327711B2 (en) | 2014-12-05 | 2022-05-10 | Microsoft Technology Licensing, Llc | External visual interactions for speech-based devices |
| US9584915B2 (en) | 2015-01-19 | 2017-02-28 | Microsoft Technology Licensing, Llc | Spatial audio with remote speakers |
| US9625990B2 (en) * | 2015-03-03 | 2017-04-18 | Toyota Motor Engineering & Manufacturing North America, Inc. | Vision-assist systems including user eye tracking cameras |
| EP3286931B1 (en) * | 2015-04-24 | 2019-09-18 | Dolby Laboratories Licensing Corporation | Augmented hearing system |
| CN105205304A (en) * | 2015-06-30 | 2015-12-30 | 胡国生 | Color synaesthesia visualizing method for music |
| CN105554662A (en) * | 2015-06-30 | 2016-05-04 | 宇龙计算机通信科技(深圳)有限公司 | Hearing-aid glasses and hearing-aid method |
| US10136214B2 (en) | 2015-08-11 | 2018-11-20 | Google Llc | Pairing of media streaming devices |
| US9715366B2 (en) | 2015-09-16 | 2017-07-25 | International Business Machines Corporation | Digital map of a physical location based on a user's field of interest and a specific sound pattern |
| US9949056B2 (en) | 2015-12-23 | 2018-04-17 | Ecole Polytechnique Federale De Lausanne (Epfl) | Method and apparatus for presenting to a user of a wearable apparatus additional information related to an audio scene |
| US9905244B2 (en) * | 2016-02-02 | 2018-02-27 | Ebay Inc. | Personalized, real-time audio processing |
| US11445305B2 (en) * | 2016-02-04 | 2022-09-13 | Magic Leap, Inc. | Technique for directing audio in augmented reality system |
| EP4075826A1 (en) * | 2016-02-04 | 2022-10-19 | Magic Leap, Inc. | Technique for directing audio in augmented reality system |
| WO2017194084A1 (en) * | 2016-05-09 | 2017-11-16 | Advanced Bionics Ag | Neural stimulation system |
| EP3261367B1 (en) | 2016-06-21 | 2020-07-22 | Nokia Technologies Oy | Method, apparatus, and computer program code for improving perception of sound objects in mediated reality |
| CN105933817A (en) * | 2016-07-01 | 2016-09-07 | 歌尔科技有限公司 | Bilateral wireless headset system of virtual reality (VR) device and bilateral wireless headset of VR device |
| WO2018052776A1 (en) | 2016-09-13 | 2018-03-22 | Walmart Apollo, Llc | System and methods for identifying an action of a forklift based on sound detection |
| US10656266B2 (en) | 2016-09-13 | 2020-05-19 | Walmart Apollo, Llc | System and methods for estimating storage capacity and identifying actions based on sound detection |
| US9980078B2 (en) | 2016-10-14 | 2018-05-22 | Nokia Technologies Oy | Audio object modification in free-viewpoint rendering |
| GB2557594B (en) | 2016-12-09 | 2020-01-01 | Sony Interactive Entertainment Inc | Image processing system and method |
| US9992532B1 (en) * | 2017-01-11 | 2018-06-05 | Htc Corporation | Hand-held electronic apparatus, audio video broadcasting apparatus and broadcasting method thereof |
| US11096004B2 (en) | 2017-01-23 | 2021-08-17 | Nokia Technologies Oy | Spatial audio rendering point extension |
| US10586106B2 (en) * | 2017-02-02 | 2020-03-10 | Microsoft Technology Licensing, Llc | Responsive spatial audio cloud |
| US10154354B2 (en) * | 2017-02-10 | 2018-12-11 | Cochlear Limited | Advanced artificial sound hearing training |
| JP6788272B2 (en) * | 2017-02-21 | 2020-11-25 | オンフューチャー株式会社 | Sound source detection method and its detection device |
| IL301932B2 (en) | 2017-02-28 | 2024-08-01 | Magic Leap Inc | Virtual and real object recording in mixed reality device |
| US10531219B2 (en) | 2017-03-20 | 2020-01-07 | Nokia Technologies Oy | Smooth rendering of overlapping audio-object interactions |
| US10242486B2 (en) | 2017-04-17 | 2019-03-26 | Intel Corporation | Augmented reality and virtual reality feedback enhancement system, apparatus and method |
| US10251011B2 (en) * | 2017-04-24 | 2019-04-02 | Intel Corporation | Augmented reality virtual reality ray tracing sensory enhancement system, apparatus and method |
| US11074036B2 (en) | 2017-05-05 | 2021-07-27 | Nokia Technologies Oy | Metadata-free audio-object interactions |
| US11054272B2 (en) * | 2017-05-11 | 2021-07-06 | Disney Enterprises, Inc. | Physical navigation guided via story-based augmented and/or mixed reality experiences |
| US10165386B2 (en) | 2017-05-16 | 2018-12-25 | Nokia Technologies Oy | VR audio superzoom |
| CN107277663A (en) * | 2017-07-06 | 2017-10-20 | 京东方科技集团股份有限公司 | A kind of double conduction earphones, double conduction control methods of sound and wearable device |
| US11395087B2 (en) | 2017-09-29 | 2022-07-19 | Nokia Technologies Oy | Level-based audio-object interactions |
| CN107890673A (en) * | 2017-09-30 | 2018-04-10 | 网易（杭州）网络有限公司 | Visual display method and device, storage medium, the equipment of compensating sound information |
| CN107818793A (en) * | 2017-11-07 | 2018-03-20 | 北京云知声信息技术有限公司 | A kind of voice collecting processing method and processing device for reducing useless speech recognition |
| EP3499917A1 (en) | 2017-12-18 | 2019-06-19 | Nokia Technologies Oy | Enabling rendering, for consumption by a user, of spatial audio content |
| CN108597162B (en) * | 2018-01-29 | 2020-02-11 | 西安建筑科技大学 | Traffic safety warning device for deaf-mutes |
| US10469974B2 (en) * | 2018-03-15 | 2019-11-05 | Philip Scott Lyren | Method to expedite playing of binaural sound to a listener |
| US10542368B2 (en) | 2018-03-27 | 2020-01-21 | Nokia Technologies Oy | Audio content modification for playback audio |
| US10832537B2 (en) * | 2018-04-04 | 2020-11-10 | Cirrus Logic, Inc. | Methods and apparatus for outputting a haptic signal to a haptic transducer |
| US10659875B1 (en) * | 2018-04-06 | 2020-05-19 | Facebook Technologies, Llc | Techniques for selecting a direct path acoustic signal |
| US10715909B1 (en) * | 2018-04-06 | 2020-07-14 | Facebook Technologies, Llc | Direct path acoustic signal selection using a soft mask |
| CN108447483B (en) * | 2018-05-18 | 2023-11-21 | 深圳市亿道数码技术有限公司 | speech recognition system |
| CN108854069B (en) * | 2018-05-29 | 2020-02-07 | 腾讯科技（深圳）有限公司 | Sound source determination method and device, storage medium and electronic device |
| US11494158B2 (en) | 2018-05-31 | 2022-11-08 | Shure Acquisition Holdings, Inc. | Augmented reality microphone pick-up pattern visualization |
| US10976999B1 (en) | 2018-06-15 | 2021-04-13 | Chosen Realities, LLC | Mixed reality sensor suite and interface for physical region enhancement |
| WO2019246562A1 (en) * | 2018-06-21 | 2019-12-26 | Magic Leap, Inc. | Wearable system speech processing |
| EP3827230A1 (en) * | 2018-07-24 | 2021-06-02 | Fluke Corporation | Systems and methods for representing acoustic signatures from a target scene |
| US11287526B2 (en) * | 2018-11-21 | 2022-03-29 | Microsoft Technology Licensing, Llc | Locating spatialized sounds nodes for echolocation using unsupervised machine learning |
| TWI696946B (en) * | 2019-01-10 | 2020-06-21 | 艾維克科技股份有限公司 | Voice message processing method |
| US10424048B1 (en) * | 2019-02-15 | 2019-09-24 | Shotspotter, Inc. | Systems and methods involving creation and/or utilization of image mosaic in classification of acoustic events |
| CN113748462A (en) | 2019-03-01 | 2021-12-03 | 奇跃公司 | Determining input for a speech processing engine |
| US11071912B2 (en) | 2019-03-11 | 2021-07-27 | International Business Machines Corporation | Virtual reality immersion |
| CN109919127B (en) * | 2019-03-20 | 2021-12-17 | 胡海明 | Mute language conversion system |
| CN110097872B (en) * | 2019-04-30 | 2021-07-30 | 维沃移动通信有限公司 | Audio processing method and electronic equipment |
| WO2020243680A1 (en) | 2019-05-30 | 2020-12-03 | Gravetime Inc. | Graveside memorial telepresence method, apparatus and system |
| US11328740B2 (en) | 2019-08-07 | 2022-05-10 | Magic Leap, Inc. | Voice onset detection |
| CN112423190A (en) * | 2019-08-20 | 2021-02-26 | 苹果公司 | Audio-based feedback for head-mounted devices |
| KR102712458B1 (en) * | 2019-12-09 | 2024-10-04 | 삼성전자주식회사 | Audio outputting apparatus and method of controlling the audio outputting appratus |
| WO2021142242A1 (en) * | 2020-01-08 | 2021-07-15 | Format Civil Engineering Ltd. | Systems, and programs for visualization of auditory signals |
| JP2023511067A (en) * | 2020-01-22 | 2023-03-16 | ドルビー ラボラトリーズ ライセンシング コーポレイション | Electrooculography and eye tracking |
| US11917384B2 (en) | 2020-03-27 | 2024-02-27 | Magic Leap, Inc. | Method of waking a device using spoken voice commands |
| US11543242B2 (en) | 2020-05-20 | 2023-01-03 | Microsoft Technology Licensing, Llc | Localization and visualization of sound |
| CN114355286A (en) * | 2020-10-12 | 2022-04-15 | 阿里巴巴集团控股有限公司 | Sound source tracking method, device, equipment, system and storage medium |
| CN112367426B (en) * | 2020-11-09 | 2021-06-04 | Oppo广东移动通信有限公司 | Virtual object display method and device, storage medium and electronic equipment |
| US11205292B1 (en) | 2020-12-02 | 2021-12-21 | International Business Machines Corporation | Augmented reality visualization of soundwaves |
| CN112599144B (en) * | 2020-12-03 | 2023-06-06 | Oppo（重庆）智能科技有限公司 | Audio data processing method, audio data processing device, medium and electronic equipment |
| DE102020132254A1 (en) | 2020-12-04 | 2022-06-09 | USound GmbH | Glasses with parametric audio unit |
| US12095865B2 (en) | 2020-12-11 | 2024-09-17 | International Business Machines Corporation | Identifying a voice command boundary |
| CN112857560B (en) * | 2021-02-06 | 2022-07-22 | 河海大学 | Acoustic imaging method based on sound frequency |
| US12033656B2 (en) | 2021-06-19 | 2024-07-09 | Kyndryl, Inc. | Diarisation augmented reality aide |
| EP4420366A1 (en) * | 2021-10-22 | 2024-08-28 | Magic Leap, Inc. | Voice analysis driven audio parameter modifications |
| CN114818769A (en) * | 2022-02-16 | 2022-07-29 | 清华大学 | Man-machine symbiosis based optimization method and device for sound signal processing |
| US12020382B2 (en) | 2022-03-24 | 2024-06-25 | Kyndryl, Inc. | Physical and virtual environment synchronization |
Family Cites Families (29)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US3531595A (en) * | 1966-10-31 | 1970-09-29 | Michael S Demaree | Method and apparatus for the testing and treatment of hearing deficiencies |
| US5029216A (en) * | 1989-06-09 | 1991-07-02 | The United States Of America As Represented By The Administrator Of The National Aeronautics & Space Administration | Visual aid for the hearing impaired |
| JP3376122B2 (en) * | 1994-09-19 | 2003-02-10 | オリンパス光学工業株式会社 | Head-mounted display device |
| JPH0886974A (en) * | 1994-09-19 | 1996-04-02 | Olympus Optical Co Ltd | Video display device |
| US5901232A (en) * | 1996-09-03 | 1999-05-04 | Gibbs; John Ho | Sound system that determines the position of an external sound source and points a directional microphone/speaker towards it |
| JP3863323B2 (en) * | 1999-08-03 | 2006-12-27 | 富士通株式会社 | Microphone array device |
| US7000469B2 (en) * | 2000-04-21 | 2006-02-21 | Intersense, Inc. | Motion-tracking |
| US7039198B2 (en) * | 2000-11-10 | 2006-05-02 | Quindi | Acoustic source localization system and method |
| US6971993B2 (en) * | 2000-11-15 | 2005-12-06 | Logometrix Corporation | Method for utilizing oral movement and related events |
| US6975991B2 (en) * | 2001-01-31 | 2005-12-13 | International Business Machines Corporation | Wearable display system with indicators of speakers |
| US6967455B2 (en) * | 2001-03-09 | 2005-11-22 | Japan Science And Technology Agency | Robot audiovisual system |
| US7760898B2 (en) * | 2003-10-09 | 2010-07-20 | Ip Venture, Inc. | Eyeglasses with hearing enhanced and other audio signal-generating capabilities |
| US20060053012A1 (en) | 2004-09-03 | 2006-03-09 | Eayrs David J | Speech mapping system and method |
| JP4736511B2 (en) * | 2005-04-05 | 2011-07-27 | 株式会社日立製作所 | Information providing method and information providing apparatus |
| WO2007084122A2 (en) | 2006-01-17 | 2007-07-26 | Robert Hickling | Method and apparatus for echolocation |
| US9101279B2 (en) * | 2006-02-15 | 2015-08-11 | Virtual Video Reality By Ritchey, Llc | Mobile user borne brain activity data and surrounding environment data correlation system |
| JP5286667B2 (en) * | 2006-02-22 | 2013-09-11 | コニカミノルタ株式会社 | Video display device and video display method |
| FR2899097B1 (en) * | 2006-03-28 | 2009-02-13 | Panagiotis Pavlopoulos | HELP SYSTEM FOR A PASSING PERSON |
| US7979146B2 (en) * | 2006-04-13 | 2011-07-12 | Immersion Corporation | System and method for automatically producing haptic events from a digital audio signal |
| HUP0600540A2 (en) | 2006-06-27 | 2008-03-28 | Ave Fon Kft | System for and method of visualizing audio signals |
| NL2000510C1 (en) | 2007-02-28 | 2008-09-01 | Exsilent Res Bv | Method and device for sound processing. |
| WO2009043036A2 (en) | 2007-09-28 | 2009-04-02 | Quantum Magnifier Llc | Apparatus and method for sensors having improved angular resolution |
| JP5326934B2 (en) * | 2009-01-23 | 2013-10-30 | 株式会社Ｊｖｃケンウッド | Electronics |
| JP2011091619A (en) * | 2009-10-22 | 2011-05-06 | Sony Corp | Transmitting apparatus, transmitting method, receiving apparatus, receiving method, program, and broadcasting system |
| KR101329999B1 (en) | 2009-10-29 | 2013-11-20 | 조문경 | System for correcting english pronunciation using analysis of user's voice-information and method thereof |
| WO2011147015A1 (en) | 2010-05-26 | 2011-12-01 | Maria Karam | System and method for displaying sound as vibrations |
| KR101246831B1 (en) | 2010-06-23 | 2013-03-28 | 김보민 | Augmented reality based digital view system |
| JP2012133250A (en) * | 2010-12-24 | 2012-07-12 | Sony Corp | Sound information display apparatus, method and program |
| US9219901B2 (en) * | 2012-06-19 | 2015-12-22 | Qualcomm Incorporated | Reactive user interface for head-mounted display |
-
2014
- 2014-01-24 WO PCT/CA2014/050049 patent/WO2014113891A1/en active Application Filing
- 2014-01-24 CN CN201480004961.8A patent/CN105073073B/en active Active
- 2014-01-24 CA CA2898750A patent/CA2898750C/en active Active
- 2014-01-24 US US14/762,964 patent/US10111013B2/en active Active
Also Published As
| Publication number | Publication date |
|---|---|
| CN105073073A (en) | 2015-11-18 |
| CA2898750C (en) | 2018-06-26 |
| CA2898750A1 (en) | 2014-07-31 |
| WO2014113891A1 (en) | 2014-07-31 |
| US10111013B2 (en) | 2018-10-23 |
| US20160142830A1 (en) | 2016-05-19 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN105073073B (en) | Apparatus and method for for sound visualization and auditory localization | |
| US11805347B2 (en) | Display system having an audio output device | |
| JP7551639B2 (en) | Audio spatialization and enhancement across multiple headsets | |
| CN107430868B (en) | Real-time reconstruction of user speech in immersive visualization system | |
| US11068668B2 (en) | Natural language translation in augmented reality(AR) | |
| US10979845B1 (en) | Audio augmentation using environmental data | |
| US11758347B1 (en) | Dynamic speech directivity reproduction | |
| JP2022534833A (en) | Audio profiles for personalized audio enhancements | |
| Spagnol et al. | Current use and future perspectives of spatial audio technologies in electronic travel aids | |
| CN116134838A (en) | Audio system using personalized sound profile | |
| JP2024096996A (en) | System and method for generating head-related transfer function | |
| WO2023147038A1 (en) | Systems and methods for predictively downloading volumetric data | |
| US11816886B1 (en) | Apparatus, system, and method for machine perception | |
| Finnegan | Compensating for distance compression in virtual audiovisual environments | |
| JP2018152834A (en) | Method and apparatus for controlling audio signal output in virtual auditory environment | |
| BĂLAN et al. | COMPARATIVE RESEARCH ON SOUND LOCALIZATION ACCURACY IN THE FREE-FIELD AND VIRTUAL AUDITORY DISPLAYS. | |
| GAIN | Masters Dissertation | |
| CN117289790A (en) | Haptic system and method | |
| Feakes | Auditory Displays and Assistive Technologies: the use of head movements by visually impaired individuals and their implementation in binaural interfaces | |
| Gamper | Audio augmented reality in telecommunication | |
| CN114586375A (en) | System and method for classifying beamformed signals for binaural audio playback | |
| Abe et al. | A fundamental study on the sound pressure level that best matches with the visually perceived distance from the virtual sound source |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| C06 | Publication | ||
| PB01 | Publication | ||
| C10 | Entry into substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant | ||
| TR01 | Transfer of patent right | ||
| TR01 | Transfer of patent right |
Effective date of registration: 20190930 Address after: Ontario, Canada Patentee after: Sensory intelligence company Address before: Ontario, Canada Patentee before: Hu Hai |
|
| EE01 | Entry into force of recordation of patent licensing contract | ||
| EE01 | Entry into force of recordation of patent licensing contract |
Application publication date: 20151118 Assignee: Chongqing Zhongjia Shengshi Intelligent Technology Co.,Ltd. Assignor: Sensory intelligence Co. Contract record no.: X2021990000334 Denomination of invention: Equipment and method for sound visualization and sound source location Granted publication date: 20181207 License type: Exclusive License Record date: 20210602 |