KR102241972B1 - Answering questions using environmental context - Google Patents

Abstract

Receive speech and environmental data, obtain a transcription for the speech, identify entities using environmental data, submit a query containing at least a portion of the transcription and data identifying the entity to a natural language query processing engine, And an apparatus comprising methods, systems, and a computer program encoded on a computer storage medium for obtaining one or more results for a query.

Description

Answering questions using environment context {ANSWERING QUESTIONS USING ENVIRONMENTAL CONTEXT}

This application is filed on September 10, 2012, U.S. Provisional Application No. 61/698,934, filed on September 10, 2012, 61/698,949, U.S. Patent Application No. 13/626,439, filed on September 25, 2012, Priority is claimed to U.S. Patent Application No. 13/626,351, filed Sep. 25, 2012, and U.S. Patent Application No. 13/768,232, filed Feb. 15, 2013, all contents of which are incorporated herein by reference. Included in the specification.

The present specification relates to identifying results of a query based on a natural language query and environmental information, and is intended to answer questions using environmental information (eg, context).

In general, a search query includes one or more terms submitted to the search engine when a user requests the search engine to perform the search. Among other approaches, the user may enter the query terms of a search query by typing on the keyboard, or also speaking the query terms to the microphone of the mobile device in the context of the voice query. Speech queries can be processed using speech recognition technology.

According to some innovative aspects of the subject matter described herein, environmental information (eg, ambient noise) can help a query processing system answer natural language queries. For example, a user may ask a question about the television program he is watching (eg, "Who is the main character in this movie?"). The user's mobile device detects the user's utterance and environmental information, which may include the soundtrack audio of the television program. The mobile computing device encodes speech and environmental information as waveform data, and provides the waveform data to a server-based computing environment.

The computing environment separates the utterance from environmental data of the waveform data and then obtains a transcription for the utterance. The computing environment further identifies the environmental data and entity data related to the utterance, for example by identifying the name of the movie. From the transcription and the entity data, the computing environment can then identify one or more results (eg, results in response to the user's question). In particular, the one or more results may include an answer to the user's question about "What actor is in this movie?" (eg, the actor's name). The computing environment can provide these results to a user of a mobile computing device.

Innovative aspects of the subject matter described herein can be implemented in a method, the method comprising receiving audio data encoding utterance and environmental data, obtaining a transcription of the utterance, using the environmental data to an entity Identifying an entity, submitting a query comprising data identifying the entity and at least a portion of the transcription to a natural language query processing engine, and obtaining one or more results for the query.

Other embodiments of these aspects include corresponding systems, apparatuses, and computer programs encoded on computer storage devices and configured to perform the operations of the present invention.

These and other embodiments may each optionally include one or more of the features described below. For example, at least one representation of the result is output. The entity can be identified further using the utterance. Create the above query. Generating the query includes associating the transcription with data identifying the entity. The associating operation includes tagging the transcription with data identifying an entity. The associating operation further includes replacing a portion of the transcription with data identifying the entity. The replacing operation further includes replacing one or more words of the transcription with data identifying the entity. The operation of receiving the environmental data further includes receiving environmental audio data, environmental image data, or both. The receiving of the environmental audio data further includes receiving additional audio data including background noise.

According to some innovative aspects of the subject matter described herein, an item of media content is identified based on environmental audio data and a spoken natural language query. For example, the user may ask a question about the television program he is watching, such as "What are we watching now?"). The question may include keywords (eg, “watching”) suggesting that the question is for a television show and not for other forms of media content. The user's mobile device detects the user's speech and environmental data, which may include the background audio of the television program. A mobile computing device encodes the utterance and the environmental data into waveform data, and provides the waveform data to a server-based computing environment.

The computing environment separates the utterance from environmental data of the waveform data and then processes the utterance to obtain a transcription for the utterance. From the transcription, the computing environment detects any content type-specific keywords (eg, keyword “view”). Subsequently, the computing environment may identify items of media content based on the environmental data, and select a specific item of multi-content that matches a specific content type associated with keywords from the identified items. The computing environment provides a representation of a specific item of multimedia content to a user of a mobile computing device.

Innovative aspects of the subject matter described herein can be implemented in methods, which method comprises (i) receiving audio data encoding a spoken natural language query and (ii) environmental audio data, the spoken nature Acquiring a transcription for a language query, determining a specific content type associated with one or more keywords in the transcription, providing at least part of the environmental audio data to a content recognition engine, and by the content recognition engine And identifying a content item that has been output and matches the specific content type.

Other embodiments of these aspects include computer programs encoded in corresponding systems, apparatuses, and computing storage devices and configured to perform the operations of the methods.

These and other embodiments may each optionally include one or more of the features described below. For example, certain types of content include Movie Content Type, Music Content Type, Television Show Content Type, Audio Podcast Content Type, Book Content Type, Artwork Content Type, Trailer Content Type, Video Podcast Content Type, Internet It is a video content type, or a video game content type. The receiving of the environmental audio data further includes receiving additional audio data including background noise. The background noise is associated with a specific content type. Receive additional environmental data including video data or image data. The video data or the image data is associated with the specific content type. The operation of providing at least a portion of the environmental audio data to the content recognition engine further includes an operation of providing the portion of the environmental audio data to an audio fingerprinting engine. The determining of the specific content type includes an operation of identifying one or more keywords in which at least one of the keywords is mapped to at least one of the plurality of content types, for each of a plurality of content types, using one or more databases. Include more. The plurality of content types includes the specific content type, and the mapped operation further includes an operation in which at least one of the keywords is mapped to the specific content type. Outputs data identifying the content item.

The features include, for example, the operation of providing data identifying the specific content type to the content recognition engine, and the operation of identifying the content item is an operation of receiving data identifying the content item from the content recognition engine. It includes more. The operation of receiving two or more content recognition candidates from the content recognition system, and the operation of identifying the content item further include an operation of selecting a specific content recognition candidate based on the specific content type. Each of the two or more content recognition candidates is associated with a ranking score, and the method further includes adjusting the ranking score of the two or more content recognition candidates based on the specific content type. The two or more content recognition candidates are ranked based on the adjusted ranking scores.

One or more details of the subject matter described herein are set forth in the accompanying drawings and the detailed description below. Other potential features, aspects, and advantages of the subject matter will become apparent from the above detailed description, drawings, and claims.

1 depicts an example system for identifying content item data based on environmental audio data and spoken natural language queries.
2 depicts a flow chart for an exemplary process of identifying content item data based on environmental audio data and an issued natural language query.
3A and 3B depict portions of an example system for identifying an item of content.
4 depicts an example system for identifying media content items based on environmental image data and spoken natural language queries.
5 depicts a system for identifying one or more results based on environmental audio data and speech.
6 depicts a flowchart for an exemplary process of identifying one or more results based on environmental data and utterances.
7 depicts a computer system and mobile computer device that may be used to implement the techniques described herein.
Similar reference symbols in the various drawings indicate similar elements.

A computing environment that answers spoken natural language queries using environmental information as a context can process queries using multiple processes. In the example of some of the processes shown in FIGS. 1-4, the computing environment may identify media content based on environmental information such as ambient noise. 5 and 6, in order to provide a more satisfactory answer to the spoken natural language query, the computing environment is Language queries can be increased.

More specifically, FIG. 1 shows a system 100 for identifying content item data based on environmental audio data and spoken natural language queries. Briefly, system 100 may identify content item data that is based on environmental audio data and matches specific content types by associating them with spoken natural language queries. The system 100 includes a mobile computing device 102, a disambiguation engine 104, a speech recognition engine 106, a keyword mapping engine 108, and a content recognition engine 110. Mobile computing device 102 communicates with disambiguation engine 104 via one or more networks. Mobile device 110 may include a microphone, camera, or other detection mechanisms for detecting utterances from user 112 and/or environmental data associated with user 112.

In some examples, user 112 is watching a TV program. In the illustrated example, the user 112 wants to know who supervised the currently displayed TV program. In some examples, user 112 may not know the name of the TV program currently being displayed, and thus may ask "Who directed this show?" Mobile computing device 102 detects this utterance along with environmental audio data associated with user 112's environment.

In some examples, environmental audio data associated with the environment of the user 112 may include background noise of the environment of the user 112. For example, the environmental audio data may include sounds of a TV program. In some examples, the environmental audio data associated with the currently displayed TV program is the audio of the currently displayed TV program (e.g., the dialog of the currently displayed TV program, the soundtrack audio associated with the currently displayed TV program, etc.) It may include.

In some examples, the mobile computing device 102 may detect environmental audio data after detecting the utterance, detect environmental audio data simultaneously with detection of the utterance, or both. During operation (A), the mobile computing device 102 processes the detected speech and environmental audio data to generate waveform data 114 representing the detected speech and environmental audio data, and generates the waveform data 114 (e.g. For example, it is transmitted to the disambiguation engine 104 (via the network). In some examples, environmental audio data is streamed from mobile computing device 110.

Disambiguation engine 104 receives waveform data 114 from mobile computing device 102. During operation (B), the disambiguation engine 104 processes the waveform data 114, including separation (or extraction) of the speech from other portions of the waveform data 114, and processes the speech (e.g., via a network). ) To the speech recognition engine 106. For example, the disambiguation engine 104 separates the utterance ("Who directed this show?") from the background noise of the user 112's environment (eg, the audio of the currently displayed TV program).

In some examples, disambiguation engine 104 may be used to enable separation of speech from background noise by identifying portions of waveform data 114 that include speech activity or speech activity associated with a user of computing device 102. Use a detector. In some examples, the utterance is related to a query (eg, a query related to the currently displayed TV). In some examples, waveform data 114 includes a detected utterance. In response, the disambiguation engine 104 may request environmental audio data related to the speech from the mobile computing device 102.

The speech recognition engine 106 receives a portion of the waveform data 114 corresponding to the speech from the disambiguation engine 104. During operation (C), the speech recognition engine 106 obtains a transcription of the utterance and provides the transcription to the keyword mapping engine 108. Specifically, the speech recognition engine 106 processes the speech received from the speech recognition engine 106. In some examples, processing of the utterance by the speech recognition engine 106 includes generating a transcription of the utterance. Generating the transcription of the utterance involves transcribing the utterance into text or text related data. In other words, the speech recognition system 106 may provide a linguistic expression in the literal form of the speech.

For example, the speech recognition system 106 transcribes the utterance to create a transcription "Who directed this show?" In other embodiments, the speech recognition system 106 provides transcriptions of two or more utterances. For example, the speech recognition system 106 transcribes the utterance to "Who directed this show?" And "Who directed this shoe?" creates a warrior.

The keyword mapping engine 108 receives the transcription from the speech recognition engine 106. During operation (D), the keyword mapping engine 108 identifies one or more keywords in the transcription associated with a particular content type and provides the particular content type to the disambiguation engine 104. In some embodiments, one or more types of content are'movie','music','TV show','audio podcast','image','artwork'. ,'Book','magazine','trailer','video podcast','internet video', or'video game'.

For example, the keyword mapping engine 108 identifies the keyword "directed" from the transcription "Who directed this show?" The keyword "supervised" is related to the type of "TV show" content. In some embodiments, the transcriptional keyword identified by the keyword mapping engine 108 is associated with two or more types of content. For example, the keyword "supervised" is related to "TV show" and "movie" content types.

In some embodiments, keyword mapping engine 108 identifies two or more keywords in a transcription associated with a particular content type. For example, keyword mapping engine 108 identifies keywords "supervised" and "show" associated with a particular content type. In some embodiments, the identified two or more keywords are associated with the same content type. For example, the identified keywords "supervised" and "show" are both associated with the'TV show' content type. In some embodiments, the identified two or more keywords are associated with different types of content. For example, the identified keyword "supervised" is associated with the'movie' content type, and the identified keyword "show" is associated with the'TV show' content type. The keyword mapping engine 108 transmits the specific content type to the disambiguation engine 108 (eg, via a network).

In some embodiments, the keyword mapping engine 108 uses one or more databases that map at least one of the keywords to at least one of the plurality of content types, for each of the plurality of content types. Identifies one or more keywords in the transcription associated with. Specifically, the keyword mapping engine 108 includes (or communicates with) one database (or multiple databases). The database contains or is associated with a mapping between keywords and content types. Specifically, the database establishes a link (e.g., mapping) between keywords and content types, such as allowing the keyword mapping engine 108 to identify one or more keywords in a transcription associated with specific content types. to provide.

In some embodiments, the one or more mappings between keywords and content types may include unidirectional (e.g., unidirectional mapping (i.e., mapping from keywords to content types)). In some embodiments, one or more mappings between keywords and content types are bidirectional (eg, interactive mapping (ie mapping from keywords to content types and content types). Mapping from to keywords) In some embodiments, one or more databases map one or more keywords to two or more content types.

For example, the keyword mapping engine 108 uses one or more databases that map the keyword “supervised” to “movie” and “TV show” content types. In some embodiments, the mapping between keywords and content types includes mapping between content types and multiple varying versions (ie, word families) to a root keyword. Other forms of keywords may include other grammatical categories, such as tense (eg, past, present, future) and parts of speech (eg, nouns, verbs). For example, a database may contain one or more content types of a family of words of "directed" or "directed" root words such as "directors", "direction", and "directed". May include mapping.

The disambiguation engine 104 receives data from the keyword mapping engine 108 identifying a particular content type associated with the transcription of the utterance. Moreover, as described above, the disambiguation engine 104 receives waveform data 114 from mobile computing device 102 that includes environmental audio data associated with the speech. During operation (E), the disambiguation engine 104 then provides the environmental audio data and the specific content type to the content recognition engine 110.

For example, the disambiguation engine 104 includes audio of a currently displayed TV program (e.g., a dialog of a currently displayed TV program, soundtrack audio of a currently displayed TV program, etc.). The environmental audio data related to the program and the specific content type of transcription of the utterance (eg,'TV show' content type) are transmitted to the content recognition engine 110.

In some embodiments, the disambiguation engine 104 provides some of the environmental audio data to the content recognition engine 110. In some examples, some of the environmental audio data may include background noise detected by the mobile computing device 102 after detecting the utterance. In some examples, some of the environmental audio data may include background noise detected by the mobile computing device 102 at the same time as detection of the utterance.

In some embodiments, background noise (of waveform data 114) is associated with a particular content type associated with a keyword in the transcription. For example, the transcription keyword "Director" in the transcription "Who directed this show?" is associated with the content type of the "TV Show", and background noise (e.g., environmental audio data related to the currently displayed TV program). ) It is also related to the'TV show' content type.

Content recognition engine 110 receives environmental audio data and specific content types from disambiguation engine 104. During operation (F), the content recognition engine 110 identifies content item data, which is based on the environmental audio data and matches a particular content type, and provides the content item data to the disambiguation engine 104. Specifically, the content recognition engine 110 properly processes the environmental audio data to identify content item data associated with the environmental audio data (eg, the name of a TV show, the name of a song, etc.). Moreover, the content recognition engine 110 matches the identified content item data with a specific content type (eg, the content type of the transcription of the utterance). The content recognition engine 110 transmits the identified content item data to the disambiguation engine 104 (eg, via a network).

For example, the content recognition engine 110 is based on environmental audio data related to the currently displayed TV program, and further identifies content item data matching the'TV show' content type. To this end, the content recognition engine 110 is a content item based on a dialog of the currently displayed TV program or soundtrack audio associated with the currently displayed TV program according to a part of the environmental audio data received by the content recognition engine 110 Identify the data.

In some embodiments, the content recognition engine 110 is an audio fingerprinting engine that uses a content fingerprint that uses wavelets to identify content item data. Specifically, the content recognition engine 110 converts the waveform data 114 into a spectrogram. The content recognition engine 110 extracts spectral images from the spectrogram. Spectral images can be represented by wavelets. For each of the spectral images extracted from the spectrogram, the content recognition engine 110 extracts "top" wavelets based on the size of each of the wavelets. For each spectral image, the content recognition engine 9110 computes a wavelet signature of the image. In some examples, the wavelet signature is a truncated, quantized form of wavelet decomposition of an image.

For example, in order to describe an m × n image with wavelets, m × n wavelets are returned without compression. Moreover, the content recognition engine 110 uses a subset of wavelets that mostly characterize the song. Specifically, when t << m × n, t "top" wavelets (by size) are selected. Moreover, the content recognition engine 110 uses MinHash to compute sub-fingerprints for, for example, sparse bit vectors. Generate the expression.

In some examples, when the environmental audio data includes at least the soundtrack audio associated with the currently displayed TV program, the content recognition engine 110 is based on the soundtrack audio associated with the currently displayed TV program and is also based on the'TV Show. 'Identify content item data matching the content type. Thus, in some examples, the content recognition engine 110 identifies content item data related to the name of the currently displayed TV program. For example, the content recognition engine 110 associates a specific content item (eg, a specific TV show) with a theme song (eg, soundtrack audio), and the specific content item (eg, the specific It may be determined that the TV show) matches the specific content type (eg, a'TV show' content type). Accordingly, the content recognition engine 110 relates to the specific content item (eg, the currently displayed TV program) based on environmental audio data (eg, soundtrack audio), and additionally relates to the specific content type (eg, soundtrack audio). For example, it is possible to identify data (eg, the name of the specific TV show) matching the content type of the'TV show'.

The disambiguation engine 104 receives the identified content item data from the content recognition engine 110. In operation (G), disambiguation engine 104 then provides the identified content item data to mobile computing device 102. For example, the disambiguation engine 104 transmits the identified content item data related to the currently displayed TV program (eg, the name of the currently displayed TV program) to the mobile computing device 102.

In some examples, one or more of the mobile computing device 102, the disambiguation engine 104, the speech recognition engine 106, the keyword mapping engine 108, and the content recognition engine 110 are mobile computing device 102. ), a disambiguation engine 104, a speech recognition engine 106, a keyword mapping engine 108, and a subset (or each) of the content recognition engine 110. In some embodiments, one or more of the disambiguation engine 104, speech recognition engine 106, keyword mapping engine 108, and content recognition engine 110 are one or more computing servers, a distributed computing system. , Or using one or more computing devices, such as a single server farm or cluster.

In some embodiments, as described above, environmental audio data may be streamed from the mobile computing device 110 to the disambiguation engine 104. When the environmental audio data is streamed, the above-described processing (e.g., operations (A)-(H)) is performed when the environmental audio data is received by the disambiguation engine 104 (i.e., is performed gradually. ). In other words, when each part of the environmental audio data is received (e.g., streamed) by the disambiguation engine 104, operations (A)-(H) are repeatedly performed until the content item data is identified. do.

2 shows a flow diagram of an exemplary process 200 for identifying content item data based on environmental audio data and spoken natural language queries. The example process 200 may be executed by one or more computing devices. For example, mobile computing device 102, disambiguation engine 104, speech recognition engine 106, keyword mapping engine 108, and/or content recognition engine 110 execute exemplary process 200. Can be used to

Audio data encoding the spoken natural language query and environmental audio data are received (202). For example, the masterpiece engine 104 receives waveform data 114 from the mobile computing device 102. Waveform data 114 includes a user's spoken natural query (eg, “Who directed this show?”) and environmental audio data (eg, audio of a currently displayed TV program). The disambiguation engine 104 separates the spoken natural language query ("Who directed this show?") from the background noise of the user 112's environment (eg, the audio of the currently displayed TV program).

The transcription of the natural language query is obtained (204). For example, the speech recognition system 106 transcribes the natural language query to generate a transcription of the natural language query (eg, “Who directed this show?”).

A specific content type associated with one or more keywords in the transcription is determined (206). For example, one or more keywords within a transcription (eg, "Who directed this show?") associated with a particular content type (eg,'TV Show' content type) by the keyword mapping engine 108 (Eg, "supervised"). In some embodiments, the keyword mapping engine 108 uses one or more databases that map at least one of the keywords to at least one of the plurality of content types, for each of the plurality of content types. Or more keywords. The database provides a link (eg, a mapping) between keywords (eg, “supervised”) and content types (eg, a'TV show' content type).

At least a portion of the environmental audio data is provided to the content recognition engine 208. For example, the disambiguation engine 104 provides the content recognition engine 110 with at least a portion of the environmental audio data (eg, audio of a currently displayed TV program) encoded by the waveform data 114. In some examples, the disambiguation engine 104 also content-recognizes a particular content type (eg,'TV Show' content type) associated with one or more keywords within the transcription (eg, "supervised"). It is provided to the engine 110.

A content item that is output by the content recognition engine and matches a specific content type is identified (210). For example, the content recognition engine 110 is based on environmental audio data (eg, audio of a currently displayed TV program) and matches a specific content type (eg, a'TV show' content type). Identify the item or content item data.

3A and 3B illustrate portions 300a and 300b of a system for identifying content item data, respectively. In particular, FIGS. 3A and 3B show disambiguation engines 304a, 304b, respectively; And content recognition engines 310a and 310b, respectively. Disambiguation engines 304a, 304b are similar to disambiguation engine 104 of system 100 shown in FIG. 1; The content recognition engines 310a and 310b are similar to the content recognition engine 110 of the system 100 shown in FIG. 1.

3A shows a portion 300a including a content recognition engine 310a. The content recognition engine 310a may identify a content item matching a specific content type based on the environmental data. In other words, the content recognition engine 310a may appropriately process the environment data based on the environment data to identify the content item, and further select one or more identified content item data so that the selected content item data matches a specific content type. have.

Specifically, during operation (A), disambiguation engine 304a provides environmental data and specific content types to content recognition engine 310a. In some embodiments, disambiguation engine 304a provides some of the environmental data to content recognition engine 310a.

Content recognition engine 310a receives environmental data and specific content types from disambiguation engine 304a. Then during operation (B), the content recognition engine 310a identifies content item data matching the specific content type based on the environmental data and provides the identified content item data to the disambiguation engine 304a. Specifically, the content recognition engine 310a identifies content item data (eg, TV show name, song title, etc.) based on the environment data. The content recognition engine 310a then selects one or more of the identified content item data matching the particular content type. In other words, the content recognition engine 310a filters the identified content item data based on a specific content type. The content recognition engine 310a transmits the identified content item data (eg, via a network) to the disambiguation engine 304a.

In some examples, as described above with respect to FIG. 1, when the environment data includes at least a soundtrack audio associated with a currently displayed TV program, the content recognition engine 310a may include the currently displayed TV program and Identify the content item data based on the associated soundtrack audio. The content recognition engine 310a then filters the identified content item data based on the'TV show' content type. For example, the content recognition engine 310a identifies a'theme song name' and a'TV show name' associated with the soundtrack audio. The content recognition engine 310a filters the identified content item data such that the subsequently identified content item data also matches the'TV show' content type. For example, the content recognition engine 310a selects'TV show name' identification data, and transmits the'TV show name' identification data to the disambiguation engine 304a.

In some examples, the content recognition engine 310a selects a corpus (or index) based on a content type (eg, a'TV show' content type). Specifically, the content recognition engine 310a may have access to a first index related to a “TV show” content type and a second index related to a “movie” content type. The content recognition engine 310a appropriately selects the first index based on the'TV show' content type. Accordingly, by selecting the first index (and not selecting the second index), the content recognition engine 310a can more effectively identify the content item data (eg, the name of the TV show).

The disambiguation engine 304a receives content item data from the content recognition engine 310a. For example, the disambiguation engine 304a receives'TV show name' identification data from the content recognition engine 310a. The disambiguation engine 304a then provides identification data to a third party (eg, mobile computing device 102 of FIG. 1 ), during operation (C). For example, the disambiguation engine 304a provides the data'TV show name' identification data to a third party.

3B is a diagram illustrating a portion 300b including the content recognition engine 310b. The content recognition engine 310b may identify content item data based on environment data. In other words, the content recognition engine 310b properly processes the environment data to identify the content item data based on the environment data, and provides the content item data to the disambiguation engine 304b. The content recognition engine 310b selects at least one of the identified content item data so that the selected content item data matches a specific content type.

Specifically, during operation (A), the disambiguation engine 304b provides environmental data to the content recognition engine 310b. In some embodiments, the disambiguation engine 304b provides some of the environmental data to the content recognition engine 310b.

The content recognition engine 310b receives environmental data from the disambiguation engine 304b. Thereafter, during operation (B), the content recognition engine 310b identifies the content item data based on the environmental data and provides the identified content item data to the disambiguation engine 304b. Specifically, the content recognition engine 310b identifies content item data associated with two or more content items (eg, a name of a TV show, a title of a song, etc.) based on the environmental data. The content recognition engine 310b sends two or more candidates representing the identified content item data (eg, via a network) to the disambiguation engine 304b.

In some examples, as described above with respect to FIG. 1, when the environment data includes at least soundtrack audio associated with the currently displayed TV program, the content recognition engine 310b may be configured with a soundtrack associated with the currently displayed TV program. Identify content item data associated with two or more content items based on the audio. For example, the content recognition engine 310b identifies'theme song name' and'TV show name' associated with the soundtrack audio, and disambiguates the'theme song name' and'TV show name' identification data. To send.

Disambiguation engine 304b receives two or more candidates from content recognition engine 310b. For example, the disambiguation engine 304b receives'theme song name' and'TV show name' candidates from the content recognition engine 310b. Thereafter, during operation (C), the disambiguation engine 304b selects one of the two or more candidates based on the particular content type and selects the selected candidate to a third party (e.g., mobile computing device 102 in FIG. 1). To provide. Specifically, as described above with respect to FIG. 1, the disambiguation engine 304b may previously receive a particular content type (eg, associated with a utterance). The disambiguation engine 304b selects a particular candidate from among two or more candidates based on the particular content type. Specifically, the disambiguation engine 304b selects a specific candidate matching a specific content type from among two or more candidates. For example, the disambiguation engine 304b selects the'TV show name' candidate because the'TV show name' candidate matches the'TV show' content type.

In some embodiments, two or more candidates from the content recognition engine 310b are associated with a ranking score. The ranking score may be associated with any score metric determined by the disambiguation engine 304b. The disambiguation engine 304b may also adjust the ranking scores of two or more candidates based on a particular content type. Specifically, the disambiguation engine 304b may increase the ranking score of one or more candidates when each of the candidates matches a particular content type. For example, the ranking score of the candidate'TV Show Name' may increase as it matches the'TV Show' content type. In addition, the disambiguation engine 304b may reduce the ranking score of one or more candidates if each of the candidates does not match a particular content type. For example, the ranking score of the candidate'Theme Song Name' may decrease as it does not match the'TV Show' content type.

In some embodiments, two or more candidates may be ranked based on ranking scores respectively adjusted by the disambiguation engine 304b. For example, the disambiguation engine 304b is compared with the adjusted ranking score of the'theme song name' candidate, as the'TV show name' candidate has a higher adjusted ranking score, the'TV show name' candidate The ranking of can be given higher than the “Theme Song Name” candidate. In some examples, disambiguation engine 304b selects the highest ranked candidate (eg, with the highest adjusted ranking score).

4 shows a system 400 for identifying content item data based on environmental image data and spoken natural language queries. In short, system 400 may identify content item data that matches a particular content type associated with a spoken natural language query based on the environmental data. The system 400 includes a mobile computing device 102, a disambiguation engine 104, a speech recognition engine 106, a keyword mapping engine 108, and a content recognition engine 110 of the system 100 shown in FIG. 1. Each similar, includes a mobile computing device 402, a speech recognition engine 406, a keyword mapping engine 408, and a content recognition engine 410.

In some examples, user 112 is viewing the CD album cover of the movie's soundtrack. In the illustrated example, the user 112 wants to know what songs are included in the soundtrack. In some examples, user 112 may not know the name of the movie soundtrack, and thus may ask the question “What song is here?” or “What song is played in this movie?” . The mobile computing device 402 detects such utterances as well as environmental image data associated with the environment of the user 112.

In some examples, environmental image data associated with the environment of the user 112 may include image data of the environment of the user 112. For example, the environmental image data includes an image of a CD album cover depicting images related to a movie (eg, an image of a movie poster of an associated movie). In some examples, mobile computing device 402 detects environmental image data utilizing a camera of mobile computing device 402 that captures an image (or video) of a CD album cover.

During operation (A), the mobile computing device 402 processes the detected utterances to generate waveform data 414 representing the detected utterances, and transmits the waveform data 414 and environmental image data (e.g., via a network). ) To the disambiguation engine 404.

During operation (B), disambiguation engine 404 receives waveform data 414 and environmental image data from mobile computing device 402. The disambiguation engine 404 processes the waveform data 414 and transmits the speech to the speech recognition engine 406 (eg, via a network). In some examples, the utterance is related to a query (eg, a query related to a movie soundtrack).

Speech recognition engine 406 receives speech from disambiguation engine 404. During operation (C), the speech recognition engine 406 obtains a transcription of the utterance and provides the transcription to the keyword mapping engine 408. Specifically, the speech recognition engine 406 processes the speech received from the speech recognition engine 406 by generating a transcription of the speech.

For example, the speech recognition engine 406 transcribes the utterance to create a transcription "What's the song here?" In some embodiments, the speech recognition engine 406 provides two or more transcriptions of the utterance. For example, the speech recognition engine 406 transcribes the utterances so that “What songs are on this?” and “What sinks are on this?” )'warriors are created.

The keyword mapping engine 408 receives the transcription from the speech recognition engine 406. During operation (D), the keyword mapping engine 408 identifies one or more keywords in the transcription associated with the particular content type and provides the particular content type to the disambiguation engine 404.

For example, the keyword mapping engine 408 identifies the keyword “song” from the transcription of “What is the song here?”. The keyword “song” is associated with the “music” content type. In some embodiments, the keyword of the transcription identified by the keyword mapping engine 408 is associated with two or more types of content. For example, the keyword “song” is associated with “music” and “singer” content types. The keyword mapping engine 408 transmits (eg, over a network) a particular content type to the disambiguation engine 408.

In some embodiments, similar to the above, the keyword mapping engine 408 maps at least one keyword and at least one of a plurality of content types for each of the plurality of content types. Databases are used to identify one or more keywords associated with a particular content type within a transcription. For example, the keyword mapping engine 408 uses one or more databases that map “song” to “music” and “singer” content types.

The disambiguation engine 404 receives from the keyword mapping engine 408 a particular content type associated with the transcription of the utterance. In addition, as described above, the disambiguation engine 404 receives environmental image data associated with the speech. During operation (E), the disambiguation engine 404 provides the environmental image data and the specific content type to the content recognition engine 410.

For example, the disambiguation engine 404 identifies environmental image data associated with a movie soundtrack (eg, an image of a movie poster CD album cover) and a specific content type of the utterance transcription (eg, “music” content type). It is transmitted to the content recognition engine 410.

Content recognition engine 410 receives environmental image data and specific content types from disambiguation engine 404. Then, during operation (F), the content recognition engine 410 identifies content item data that is based on the environmental image data and matches a particular content type and provides the identified content item data to the disambiguation engine 404. Specifically, the content recognition engine 410 properly processes environmental image data to identify content item data (eg, the name of a content item). In addition, the content recognition engine 410 matches the identified content item with a specific content type (eg, the content type of the utterance transcription). The content recognition engine 408 transmits the identified content item data (eg, via a network) to the disambiguation engine 408.

For example, the content recognition engine 410 is based on environmental image data related to an image of a movie poster CD album cover, and further identifies data matching the'music' content type.

In some examples, if the environmental image data includes at least a movie poster associated with the CD album cover, the content recognition engine 410 is based on the movie poster associated with the CD album cover and also matches the'music' content type. Identify item data. Thus, in some examples, the content recognition engine 410 identifies content item data associated with the title of the movie soundtrack. For example, the content recognition engine 410 may associate a particular content item (e.g., a particular movie soundtrack) with a movie poster, and a particular content item (e.g., a particular movie soundtrack) is For example, it can be determined that it matches the'music' content type). Accordingly, the content recognition engine 410 is based on environmental image data (e.g., an image of a CD album cover) and further matches a particular content type (e.g.,'music' content type), a specific content item (e.g. For example, it is possible to identify data related to a specific movie soundtrack (eg, the title of a specific movie soundtrack).

The disambiguation engine 404 receives the identified content item data from the content recognition engine 410. Then, in operation (G), the disambiguation engine 404 provides the identified content item data to the mobile computing device 402. For example, the disambiguation engine 404 transmits the identified content item data associated with the movie soundtrack (eg, the title of the movie soundtrack) to the mobile computing device 402.

As mentioned above, FIGS. 1-4 illustrate various exemplary processes within a computing environment capable of identifying media content (or other content) based on environmental information such as ambient noise. Other processes for identifying content may also be used. In general, Figures 5 and 6 show that, in order to provide a more satisfactory answer to the spoken natural language query, the computing environment may generate a spoken natural language query for a context derived from environmental information, such as data identifying media content. It shows an exemplary process that can be increased.

More specifically, FIG. 5 shows a system 500 for identifying one or more results based on environmental audio and speech. In some examples, the one or more results may represent one or more answers to a natural language query. The system 500 includes a mobile computing device 502, a coordination engine 504, a speech recognition engine 506, a content identification engine 508, and a natural language query processing engine 510. The mobile computing device 502 communicates with the coordination engine 504 through one or more networks. Mobile device 510 may include a microphone, camera, or other sensing mechanism to detect utterances from user 512 and/or environmental data associated with user 512.

Similar to system 100 of FIG. 1, user 512 is watching a television program. In the illustrated example, the user 512 wants to know who has supervised the television program currently playing (eg, an entity). In some examples, user 512 may not know the name of the television program currently playing, and thus may ask the question “Who directed this show?”. The mobile computing device 502 detects this utterance, as well as environmental data associated with the user 512 environment.

In some examples, environmental data associated with the environment of the user 512 may include background noise of the environment of the user 512. For example, environmental data includes television program (eg, entity) sound. In some examples, environmental data associated with a currently displayed television program may include audio of a currently displayed television program (e.g., a dialog of a currently displayed television program, soundtrack audio associated with a currently displayed television program, etc.). I can. In some examples, environmental data may include environmental audio data, environmental image data, or both. In some examples, mobile computing device 502 detects environmental audio data after sensing the utterance; Sensing the utterance while simultaneously sensing the environmental audio data; Or it detects both. During operation (A), the mobile computing device 502 generates the detected speech and environmental data to generate waveform data 514 representing the detected speech and the detected environmental audio data (e.g., sound of a television program). Processes and transmits the waveform data 514 to the coordination engine 504 (eg, over a network).

Coordination engine 504 receives waveform data 514 from mobile computing device 502. During operation (B), the coordination engine 504 processes the waveform data 514 and processes the waveform data 514 corresponding to the waveform, including separating (or extracting) the utterance from other portions of the waveform data 514 ( 514) to the speech recognition engine 506 (eg, over a network). For example, the coordination engine 504 separates the utterance ("Who directed this show?") from the background noise of the user 512 environment (eg, the audio of the currently displayed television program). In some examples, the coordination engine 504 uses a voice detector to facilitate separation of speech from background noise by identifying a portion of waveform data 514 that includes voice activity. I can. In some examples, the utterance is associated with a query (eg, a query related to the currently displayed television program).

The speech recognition engine 506 receives a part of the waveform data 514 corresponding to the speech from the coordination engine 504. During operation (C), the coordination engine 506 acquires a transfer of the ignition and provides the transfer to the coordination engine 504. In particular, the speech recognition engine 506 properly processes a part of the waveform data 514 corresponding to the speech received from the coordination engine 504. In some examples, processing a portion of the waveform data 514 corresponding to the utterance by the speech recognition engine 506 includes generating a transcription of the utterance. Generating the transcription of the utterance may include transcribing the utterance into text or text-related data. In other words, the speech recognition engine 506 may provide a linguistic representation of the in written form of the utterance.

For example, the speech recognition engine 506 transcribes the utterance to create a transcription of "Who directed this show?" In some embodiments, speech recognition engine 506 provides two or more transcriptions of the utterance. For example, the speech recognition engine 506 to generate transcriptions of "Who directed this show?" and "Who directed this shoe?" The ignition can be transferred.

The coordination engine 504 receives the transcription of the speech from the speech recognition engine 506. Further, as mentioned above, the coordination engine 504 receives waveform data 514 including environmental audio data associated with the speech from the mobile computing device 502. The coordination engine 504 now uses the environmental data to identify the entity. In particular, the coordination engine 504 obtains data identifying an entity from the content identification engine 508. To do so, during operation (D), the coordination engine 504 provides the environmental data and a portion of the waveform data 514 corresponding to the utterance to the content identification engine 508 (eg, via a network).

For example, the coordination engine 504 includes audio of a currently displayed television program (e.g., a dialog of a currently displayed television program, soundtrack audio associated with a currently displayed television program, etc.). Environmental data associated with the program (eg, an entity) and a portion of the waveform 514 corresponding to the utterance (“Who directed this show?”) are sent to the content identification engine 508.

In some embodiments, the coordination engine 504 provides a portion of the environmental data to the content identification engine 508. In some examples, some of the environmental data may include background noise detected by mobile computing device 502 after detection of the utterance. In some examples, some of the environmental data may include background noise detected by the mobile computing device 502 concurrently with the detection of the utterance.

The content identification engine 508 receives environmental data from the coordination engine 504 and a portion of the waveform 514 corresponding to the utterance. During operation (E), the content identification engine 508 identifies data identifying an entity (e.g., content item data) based on environmental data and utterance, and transfers the data identifying the entity to the coordination engine 504. Provides (for example, over a network). In particular, the content identification engine 508 includes environmental data and data identifying an entity (e.g., content item data) associated with the environmental data (e.g., name of a television show, name of a song, etc.) A part of the waveform 514 corresponding to the utterance is appropriately processed.

For example, the content identification engine 508 processes environmental audio data to identify content item data associated with the currently displayed television program. In some embodiments, the content identification engine 508 is the system 100 of FIG. 1.

The coordination engine 504 receives data identifying an entity (eg, content item data) from the content identification engine 508. Further, as described above, the coordination engine 504 receives a transcription from the speech recognition engine 506. During operation (F), the coordination engine 504 now provides (eg, via a network) to the natural language query processing engine 510 a query including transcription and data identifying the entity. For example, the coordination engine 504 provides query and content item data ('television show name') including the transcription of the utterance ("Who directed this show?") to the natural language query processing engine 510 do.

In some examples, the coordination engine 504 generates a query. In some examples, the coordination engine 504 obtains a query (eg, from a third server). For example, the coordination engine 504 submits the transcription of the utterance and data identifying the entity to the third server, and returns a query based on the data identifying the transcription and the entity.

In some embodiments, generating the query by the coordination engine 504 may include associating the transcription of the utterance with data identifying an entity (eg, content item data). In some examples, associating the transcription of the utterance with content item data may include tagging the transcription with data identifying an entity. For example, the coordination engine 504 may tag the transcription "Who supervised this show?" with a'television show name' or other identifying information (e.g., an identification (ID) number) associated with the content item data. have. In some examples, associating the transcription of the utterance with data identifying the entity includes replacing a portion of the transcription with data identifying the entity. For example, the coordination engine 504 may replace a portion of the warrior “Who directed this show?” with data identifying “television show name” or “television show name”. In some examples, replacing a portion of the transcription with data identifying the entity may include replacing one or more words of the transcription of the utterance with data identifying the entity. For example, the coordination engine 504 may replace data identifying'television show name' or'television show name' in the transcription "Who directed this show?". For example, the substitution would be a transcription containing "Who directed'television show name'" or "Who directed'ID number'". Can lead to.

The natural language query processing engine 510 receives from the coordination engine 504 a query including transcription and data identifying an entity (eg, content item data). During operation (G), the natural language query processing engine 510 properly processes the query and, based on the processing, provides one or more results to the coordination engine 504 (eg, via a network). In other words, the coordination engine 510 obtains one or more results for the query (eg, from the natural language query processing engine 510).

In particular, the natural language query processing engine 510 obtains information resources (transcription of speech and content item data) related to the query (from a collection of information resources). In some examples, the natural language query processing engine 510 matches the query against database information (e.g., text document, image, audio, video, etc.) and determines how well each entity in the database matches the query. The score for is calculated. The natural language query processing engine 510 identifies one or more results based on the matched entities (eg, entities with a score higher than a threshold score).

For example, the natural language query processing engine 510 receives a query containing the transcription of the utterance "Who directed this show?" and a'television show name' (or other identifying information). The natural language query processing engine 510 matches the query against database information and provides one or more results of matching the query. The natural language query processing engine 510 calculates a score for each matched entity.

Coordination engine 504 receives one or more results from natural language query processing engine 510. In operation (H), the coordination engine 504 now provides the one or more results to the mobile computing device 502 (eg, via a network). For example, the coordination engine 504 transmits one or more results (eg, the name of the director of the television show) to the mobile computing device 502.

In some examples, one or more of the mobile computing device 502, coordination engine 504, speech recognition engine 506, content identification engine 508, and natural language query processing engine 510 may include mobile computing device 502 , A coordination engine 504, a speech recognition engine 506, a content identification engine 508, and a subset (or each) of the natural language query processing engine 510. In some embodiments, one or more coordination engines 504, speech recognition engines 506, content identification engines 508, and natural language query processing engines 510 may include one or more computing servers, distributed computing systems, or server farms. Or may be implemented by one or more computing devices, such as a cluster.

6 shows a flow diagram of an example process 600 for identifying one or more results based on environmental data and utterance. The example process 600 may be executed using one or more computing devices. For example, mobile computing device 502, coordination engine 504, speech recognition engine 506, content identification engine 508, and/or natural language query processing engine 510 execute example process 600. Can be used to

Audio data encoding speech and environment data is received (602). For example, coordination engine 504 receives waveform data 514 from mobile computing device 502. Waveform data 514 includes the user's speech (eg, “Who directed this show?”) and environmental data (eg, audio of the currently displayed television program). In some examples, receiving environmental data may include receiving environmental audio data, environmental image data, or both. In some examples, receiving the environmental data includes receiving additional audio data including background noise.

The transcription of the ignition is obtained (604). For example, the coordination engine 504 uses the speech recognition engine 506 to obtain a transcription of the utterance. The speech recognition engine 506 transcribes the utterance to create a transcription of the utterance (eg, “Who directed this show?”).

The entity is identified using the environmental data (606). For example, the coordination engine 504 uses the content identification engine 508 to obtain data identifying an entity. The content identification engine 508 is configured to identify environmental data (e.g., content item data) associated with the environmental data (e.g., name of a television show, title of a song, etc.) For example, environmental audio data associated with the displayed television program) can be appropriately processed. In some examples, the content identification engine 508 may further process the waveform 514 corresponding to the utterance (simultaneously or subsequent to processing of environmental data) to identify the entity.

In some examples, the coordination engine 504 generates a query. In some examples, generating a query by the coordination engine 504 may include associating the transcription of the utterance with data identifying the entity. In some examples, associating the transcription of the utterance with the content item data may include replacing a portion of the transcription with data identifying the entity. In some examples, replacing a portion of the transcription with data identifying the entity may include replacing one or more words of the transcription of the utterance with data identifying the entity.

The query is submitted 608 to the natural language processing engine. For example, the coordination engine 504 submits the query to the natural language processing engine 510. The query may include data identifying at least a portion of the transcription and an entity (eg, content item data). For example, the coordination engine 504 provides query and content item data ('television show name') including the transcription of the utterance ("Who directed this show?") to the natural language query processing engine 510 do.

One or more results for the query are obtained (610). For example, the coordination engine obtains one or more results for the query from the natural language query processing engine 510 (eg, the name of the director of the television show). In some examples, coordination engine 504 now provides the one or more results to mobile computing device 502.

7 shows an example of a typical computing device 700 and a typical mobile computing device 750 in which the techniques described herein may be used. Computing device 700 is used to represent various types of digital computers, such as laptops, desktops, workstations, PDAs, servers, blade servers, mainframes, and other suitable computers. Mobile computing device 750 is used to represent various types of mobile devices, such as PDAs, cell phones, smart phones, and other similar computing devices. The components, their connections and relationships, and their functions shown herein are meant to be illustrative only, and are not meant to limit the embodiments of the technology described or claimed herein.

The computing device 700 includes a processor 702, a memory 704, a storage device 706, a high-speed interface 708 and a high-speed expansion port 710 connecting to the memory 704, and a low-speed bus 714 and storage. It includes a low speed interface 712 that connects to the device 706. Each of the components 702, 704, 706, 708, 710, and 512 are connected to each other using a variety of buses, and can be mounted on a common motherboard or in other ways as appropriate. The processor 702 can process instructions for execution in the computing device 700, and these instructions include graphic information for GUI on an external input/output device such as a display 716 connected to the high-speed interface 708. For display, instructions stored in memory 704 or storage device 706 are included. In other embodiments, multiple processors and/or multiple buses may be used with multiple memories and memory types where appropriate. Further, the multiple computing devices 700 may be connected in a form in which each device provides a portion of the required operation (eg, a server bank, a group of blade servers, or a multiprocessor system).

Memory 704 stores information within computing device 700. In one embodiment, memory 704 is a volatile memory unit or units. In yet another embodiment, memory 704 is a non-volatile memory unit or units. Further, the memory 704 may be another type of computer-readable medium such as a magnetic or optical disk.

Storage device 706 can provide mass storage for computing device 700. In one embodiment, the storage device 706 includes a floppy disk device, a hard disk device, an optical disk device, or a tape device, a flash memory or other similar solid state memory device, or a device residing in a storage area network or other configuration. It may be a device array. The computer program product can be tangibly embodied in an information medium. Further, the computer program product may include instructions that, when executed, perform one or more methods such as those described above. The information carrier is a computer or machine readable medium such as memory 704, storage device 706, and memory on processor 702.

The low speed control 712 manages the lower bandwidth-intensive operations, while the high speed control 708 manages the band-intensive operation for the computing device 700. The arrangement of these functions is merely exemplary. In one embodiment, the high-speed controller 708 is connected to the memory 704, the display 716 (for example, through a graphics processor or accelerator), and can accommodate a variety of expansion cards (not shown). It is connected to the expansion port 710. In some embodiments, the low speed control 712 is connected to the storage device 706 and the low speed expansion port 714. The slow expansion port, which may include a variety of communication ports (e.g., USB, Bluetooth, Ethernet, wireless Ethernet), connects to one or more input/output devices such as a keyboard, pointing device, scanner, or connects to, for example, a network adapter. Through this, it can be connected to a networking device such as a switch or router.

The computing device 700 may be implemented in a plurality of different forms, as shown in the figure. For example, the computing device 700 may be implemented as a standard server 720 or may be implemented multiple times in a group of such servers. Additionally, computing device 700 may be implemented as part of rack server system 724. In addition, computing device 700 may be implemented in a personal computer such as laptop computer 722. Optionally, components from computing device 700 may be combined with other components in a mobile device (not shown), such as device 750. Each of these devices includes one or more computing devices 700, 750, and the entire system may consist of multiple computing devices 700, 750 in communication with each other.

Computing device 750 includes a processor 752, a memory 764, an input/output device such as a display 754, a communication interface 766, and a transceiver 768, among other components. Further, in order to provide additional storage to the device 750, a storage device such as a micro drive or other device may be provided. Each of the components 750, 752, 764, 754, 766, and 568 are connected to each other using a variety of buses, and some of the components may be mounted on a conventional motherboard or may be mounted in other suitable ways.

Processor 752 executes instructions within computing device 750, including instructions stored in memory 764. The processor is separate and can be implemented as a chip set of chips including multiple analog and digital processors. The processor may provide coordination between other components of the device 750, such as, for example, control of the user interface, applications executed by the device 750, and wireless communication by the computing device 750. have.

The processor 752 may communicate with a user through a control interface 758 and a display interface 756 coupled to the display 754. Display 754 may be, for example, a Thin-Film-Tansistor Liquid Crystal Display (TFT LCD) or Organic Light Emitting Diode (OLED) display, or other suitable display technology. Display interface 756 may include suitable circuitry to drive display 754 to present graphics and other information to a user. The control interface 758 receives instructions from the user and converts the instructions for submission to the processor 752. Moreover, the extended interface 762 may be provided in communication with the processor 752 to enable short-range communication between the device 750 and other devices. The extended interface 762, for example, provides wired communication in some embodiments and wireless communication in other embodiments, and multiple interfaces may also be used.

Memory 764 stores information within computing device 750. The memory 764 may be implemented as one or more of a computer-readable medium or media, a volatile memory unit or units, or a nonvolatile memory unit or units. In addition, an extended memory 774 may be provided and connected to the device 750 through an extended interface 774 including, for example, a Single In Line Memory Module (SIMM) card interface. The extended memory 774 may provide an extra storage space for the device 750 and may also store an application or other information for the device 750. In particular, the extended memory 774 includes instructions for executing or assisting the above-described process, and may also include security information. Thus, for example, the extended memory 774 may be provided as a security module for the device 750 and may be programmed with instructions that enable secure use of the device 750. Moreover, security applications can be provided via the SIMM card with additional information, such as placing identification information on the SIMM card in a non-hackable manner.

The memory may include, for example, flash memory and/or NVRAM memory, as discussed below. In one embodiment, the computer program product is tangibly embodied in an information carrier. The computer program product contains instructions that, when executed, perform one or more methods such as those described above. The information carrier may be a computer-or machine such as memory 764, extended memory 774, memory on processor 752, or a transmitted signal that may be received via, for example, transceiver 768 or extended interface 762. -It is a readable medium.

Device 750 may communicate wirelessly through a communication interface 766 that includes digital signal processing circuitry as needed. Communication interface 766 may provide communication under various modes or protocols such as GSM voice calls, SMS, EMS, or MMS messaging, CDMA, TDMA, PDC, WCDMA, CDMA2000, or GPRS, and the like. Such communication may be performed via a radio-frequency transceiver 768, for example. Further, short range communication may be performed using, for example, Bluetooth, WiFi, or other such transceivers (not shown). In addition, the Global Position System (GPS) receiver module 770 may provide additional navigation- and location-related wireless data to the device 750. This wireless data can be suitably used by an application running on device 750.

In addition, the device 750 may receive speech information from a user and communicate audibly using an audio codec 760 that converts the speech information into usable digital information. In addition, the audio codec 760 generates a user-audible voice, such as through a speaker in the handset of the device 750, for example. These voices may include voices from voice telephone calls, recorded voices (e.g., voice messages, music files, etc.), and voices generated by applications running on device 750 It may include.

As shown in the drawing, the computing device 750 may be implemented in a plurality of different forms. For example, computing device 750 may be implemented as a cellular phone 780. Further, computing device 750 may be implemented as part of a smartphone 782, PDA, or other similar mobile device.

Various embodiments of the various systems and methods described herein may be realized in digital electronic circuits, integrated circuits, application specific integrated circuits (ASICs) designed for specific purposes, computer hardware, firmware, software, and/or combinations thereof. have. Examples include embodiments of one or more computer programs, which computer programs are executable and/or interpretable on a programmable system, the programmable system being connected to a storage system to transmit and receive data and instructions, either dedicated or general purpose. At least one programmable processor, at least one input device, and at least one output device.

Such computer programs (also referred to as programs, software, software applications, or code) contain machine instructions for programmable processors, and can be implemented in high-level procedural and/or object-oriented programming languages, and/or assembly/machine languages. have. As used herein, the terms “machine-readable medium” and “computer-readable medium” refer to apparatus and/or devices used to provide machine instructions and/or data to a programmable processor (e.g., Magnetic disk, optical disk, memory, programmable logic device (PLD)), and also includes machine-readable media that receive machine instructions as machine-readable signals. The term “machine-readable signal” refers to any signal used to provide instructions and/or data to a programmable processor.

In order to provide interaction with a user, the systems and methods described herein include a display device that displays information to a user (e.g., a cathode ray tube (CRT) or liquid crystal display (LCD) monitor) and a user It can be implemented on a computer with a keyboard and a pointing device (for example, a mouse or a trackball) used to input into the computer. Other categories of devices can also be used to provide interaction with the user. For example, the feedback provided to the user may be in any form of perceptual feedback (visual, auditory or tactile feedback), and the input from the user may be received in any form including acoustic, voice, or tactile input.

The various systems and methods described herein include a backend component (e.g., a data server), or a middleware component (e.g., an application server) or a frontend component (e.g., as described herein). A graphical user interface (GUI) or a client computer with a web browser that a user can use to interact with embodiments of the system and method) or a computing system that includes any combination of such backend, middleware, or frontend components. Can be implemented. The components of the system may be interconnected by any form or medium of digital data communication (eg, a communication network). Examples of communication networks include a local area network (“LAN”), a wide area network (“WAN”), and the Internet.

The computing system may include a client and a server. Clients and servers are generally separated from each other and interact through communication networks. The relationship between the client and the server arises by means of computer programs running on each computer and having a client-server relationship to each other.

While this specification includes several features, they should not be construed as limiting the scope of the disclosed or claimed subject matter, but as a description of features of exemplary embodiments of the disclosed subject matter. Certain features described in the context of separate embodiments herein may be provided in combination in one embodiment. Conversely, the various features described in the context of one embodiment may be provided in multiple embodiments, separate or in any suitable sub-combination. Furthermore, although features operate under certain conditions above or even have been raised as such, one or more features from the proposed combinations may in some cases be deleted from the combination, and the proposed combination may be a sub-combination or a modification of the sub-combination. Can be indicated as.

Similarly, although the actions are shown in a specific order in the figure, it is understood that such actions must be performed in the specific order shown, or in a time-series order, or require all shown actions to be performed to achieve the desired result. It shouldn't be. In certain situations, multitasking and parallel processing may be advantageous. Furthermore, the classification of the various system components in the above-described embodiments should not be interpreted as requiring such classification in all embodiments, and the described program components and systems are generally integrated into one software product or packaged as multiple software products. It should be understood as possible.

That is, specific embodiments of the present invention have been described. Other embodiments are also within the scope of the following claims. For example, the operations recited in the claims may be performed in a different order and still achieve the desired result. A number of embodiments have been described. Nevertheless, it will be understood that various modifications may be made without departing from the scope and spirit of the present invention. For example, the various types of flows shown above can be used with reordering, adding, and removing steps. Accordingly, other embodiments are also within the scope of the following claims.

104: disambiguation engine
106: speech recognition engine
108: keyword mapping engine
110: content recognition engine

Claims (15)

As a computer implemented method,
Receiving, by one or more processors, (i) a spoken natural language query and (ii) audio data encoding music;
Receiving, by one or more processors, environmental image data related to music;
Determining, by the one or more processors, that one or more keywords in the transcription of the spoken natural language query are associated with a movie content type; And
Identifying, by one or more processors, a recognized movie content item based on music and environmental image data associated with the music, based on a determination that one or more keywords in the transcription of the spoken natural language query are associated with a movie content type. Computer-implemented method comprising a. The method of claim 1,
Receiving the audio data,
And receiving audio data from a mobile computing device. The method of claim 2,
Receiving the audio data,
And receiving environmental audio data associated with the mobile computing device. The method of claim 1,
Audio data encoding the music,
A computer-implemented method, characterized in that it is generated within a predetermined period of time prior to receiving audio data encoding a spoken natural language query. The method of claim 1,
Determining that one or more keywords in the transcription of the spoken natural language query are associated with a movie content type,
And identifying one or more keywords that map at least one of the keywords to a movie content type using the one or more databases. A computer-readable medium storing software containing instructions executable by one or more computers that, when executed, cause one or more computers to perform operations, the operations comprising:
Receiving, by one or more processors, audio data encoding (i) an image or video and (ii) a spoken natural language query;
Receiving, by one or more processors, environmental image data related to music;
Determining, by the one or more processors, that one or more keywords in the transcription of the spoken natural language query are associated with a musical content type; And
Identifying, by one or more processors, an image or video, and a musical content item recognized based on environmental image data associated with the music, based on a determination that one or more keywords in the transcription of the spoken natural language query are associated with a music content type. Computer-readable medium comprising the step of. The method of claim 6,
Receiving the (i) image or video and (ii) audio data encoding the spoken natural language query,
The computer-readable medium further comprising receiving from the mobile computing device (i) image or video and (ii) audio data encoding the spoken natural language query. The method of claim 6,
The image or video,
A computer-readable medium generated within a predetermined period of time prior to receiving audio data encoding a spoken natural language query. The method of claim 6,
Determining that at least one keyword in the transcription of the spoken natural language query is associated with a music content type,
And identifying one or more keywords for mapping at least one of the keywords or the like to a music content type using one or more databases. The method of claim 7,
Receiving the image or video,
And receiving an environmental image or video associated with the mobile computing device. As a system,
One or more computers and one or more storage devices storing instructions operable to cause the one or more computers to perform operations when executed by the one or more computers, the operations comprising:
Receiving, by one or more processors, (i) a spoken natural language query and (ii) audio data encoding music;
Receiving, by one or more processors, environmental image data related to music;
Determining, by the one or more processors, that one or more keywords in the transcription of the spoken natural language query are associated with a movie content type; And
Identifying, by one or more processors, a recognized movie content item based on music and environmental image data associated with the music, based on a determination that one or more keywords in the transcription of the spoken natural language query are associated with a movie content type. A system comprising a. The method of claim 11,
Receiving the audio data,
And receiving audio data from a mobile computing device. The method of claim 11,
Audio data encoding the music,
The system according to claim 1, wherein the audio data encoding the spoken natural language query is generated within a predetermined time prior to receiving. The method of claim 11,
Determining that one or more keywords in the transcription of the spoken natural language query are associated with a movie content type,
And identifying one or more keywords that map at least one of the keywords to a movie content type using the one or more databases. The method of claim 12,
Receiving the audio data,
And receiving environmental audio data related to the mobile computing device.

Images