JP2003178078A - Additional indicator data to image and voice data, and its adding method - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To solve such problems that in some cases, it is difficult to represent the demand of the viewer side with a simple text with respect to the image and voice in an archive, and further it is very difficult to add a sensuous keyword such as 'interesting' and the like in the demand of a viewer, to a scene as the metadata. <P>SOLUTION: The indicator data obtained by automatically taking in and processing the viewer's biotic reaction to the image and voice, and correlating the same for every content such as the image and the voice, and every scene in the contents, is used besides the conventional metadata. Thereby, the viewer's biotic reaction to the image or the voice is automatically taken in and processed as the indicator data to be automatically correlated to every content and every scene in the contents. <P>COPYRIGHT: (C)2003,JPO

Description

Detailed Description of the Invention

[0001]

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to marker data associated with data including images and sounds and a method for processing the marker data.

[0002]

2. Description of the Related Art Various images and sounds are broadcast or distributed on media such as television broadcasts and radio broadcasts using terrestrial waves, satellites or cables, and homepages currently on the Internet these days. In television and radio, the “contents themselves such as video and audio to be broadcast” (this is called content) is determined in advance by the business operator side that transmits it. Of course, the operator may be sensitive to the season, the time, or the time, especially the things that have attracted people's attention. It is possible, and is actually being done, to send out such content. However, it estimates the tastes of an unspecified number of viewers and selects them to match, and responds to perfect individuals and individual tastes and requests. I'm not.

On the other hand, in response to the recent development of storage technology and communication technology, attempts have been made to send contents to perfect individuals in response to individual tastes. One of such attempts is called TV Anytime. The television anytime is described in, for example, Reference 1 “Nikkei Electronics 2001”.
April 23, 2003, page 173 "and the Asahi Shimbun homepage (URL:" HYPERLINK "http: // w
ww.asahi.com./science/waza/010521.html "http: // ww
w.asahi.com./science/waza/010521.html ”) and the like. The amount of content (called archives) accumulated in televisions, radio stations, studios, etc. has reached an enormous amount and will continue to grow and diversify. Also, from the viewer's side, even if it is called sports, it is popular with all generations such as domestic baseball and sumo,
The tastes of viewers are diversifying, such as in the US Major League and soccer. Under such circumstances, it is difficult for television stations and radio stations to accurately infer the viewer's preferences and requests and send out the most suitable content. Moreover, since the viewers have different tastes and requested contents, it is impossible to send out the contents that meet the demand, which is originally 100%. So this technology
This is a mechanism designed to allow viewers to freely call the optimum content according to their wishes at that time, and is currently being developed for practical use. In particular, this technology gives each content in the archived data a text-format ID that indicates what kind of scene is included in the content in addition to the video / audio information, and uses this to specify the content the viewer likes. Designed to be selected and called. (Data that gives information about the content, which is added separately from the original content data such as video or audio, is called "metadata".)
Regarding this, in the second line of the main text of Document 2, the following is disclosed as follows: "For example, watch a professional baseball recorded at home at midnight. First of all, I would like to check the battalion of Giants and Matsui. At that time, "It's a mechanism that allows you to collect only Matsui's at-bats and see them," explains Nippon Television's representative, Takeharu Urano of the New Technology Research and Planning Division. Terrestrial television waves contain only video and audio. However, with digital broadcasting, other information can be added to the radio waves. Terrestrial digital broadcasting will begin in 2003 in Japan. Therefore,
Information such as genres such as "movies,""sports," and "news", titles, performers, etc. are also sent with the program. For example, if the TV station attaches an ID to the at-bat image of player Matsui, the viewer can select and collect only those parts. Mr. Urano said, "Depending on how to attach the ID, it is easy to see the game as a digest." That is, if you send text that represents the content or scene you want to view as a keyword, you will select and send back images and audio with text data (metadata) corresponding to that keyword, so a huge amount It will be very easy, quick and wide-ranging to extract the contents you want to see and hear from the archive.

[0004]

However, it is not always possible to express a request for viewing or listening to video or audio in an archive with a simple text. In some cases, it is difficult to express the viewer's request with simple text, and in other cases it is difficult to express the characteristic scene in the content with simple text. For example, this week's professional baseball Giants Matsui's home run scene should also express the viewer's request by using keywords such as “date”, “professional baseball”, “series”, “giants match”, “Matsui”, “home run”, “home run”. If possible, the content itself can be represented as a characteristic scene in it. In addition, the goal scene of Nakata in the soccer J-League can also be represented by metadata such as “date”, “place”, “soccer”, “J-League”, “Nakada”, and “goal”. However, the highlight of a match that was 1-0 in soccer, for example, is not only that one-point goal scene. There will be many places that are considered "points of interest" in the scenes leading up to the goal, and there will be many "points of interest" where the goal is prevented. Although it is not related to any of them, it seems that there are quite a few scenes where cheers occur and the attention is given as a big show during the match. However, it is not easy to express the highlights other than the goal scene with simple text and add metadata. For example, "cheers"
Even if a keyword such as is added to a certain scene, it is very difficult to determine who assigns it, how and how much it is attached, and it is difficult to automate it. Also,
It is also very difficult to attach the sensory keywords such as “interesting”, “amazing”, “sad”, “painful”, “gentle”, and “chilling” to the scene as metadata in the viewer's request. It is difficult to judge by what kind of criteria such a keyword should be associated, and it takes a lot of human labor and time for the work of judging and assigning.
This is also difficult to automate.

Therefore, according to the present invention, in addition to the conventional metadata, a marker associated with contents such as images and sounds and scenes in the contents is automatically captured and processed to correlate the biological reaction of a person who views the contents. Considered using the data.

An object of the present invention is to automatically take in a biological reaction of a human being who has been viewed with respect to a target image or sound and process it and automatically associate it with the contents or scenes in the contents as marker data. Then, for the content or scenes in the content that cannot be simply keyworded, the viewer can search, extract, using keywords that are ambiguous or sensuous.
The purpose is to provide indicator data that can be called.

[0007]

[Means for Solving the Problems] It is well known that human mental activities appear in various biological reactions. In other words, even in a human who does not exercise at all and is stationary, various changes appear in the biological response due to the mental activity.
For example, the heart rate increases / decreases, and the blood pressure increases / decreases. In addition, the blood flow in the brain increases or decreases, and the distribution of blood circulation changes. An electric pulse is generated in a nerve in the brain or its waveform is changed. Also, sweating of the whole body or a specific part may be increased or decreased, or muscles may be expanded or contracted. The pupils of the eyes also expand and contract. In addition, changes may occur in breathing rate, breathing depth, etc. In addition, the facial expression and the posture depending on how the limbs are placed will also change.

Such human biological reactions can be detected and measured by various devices at present, and are utilized in fields such as medical and physiological research, ergonomics and sports. For example, the blood flow in the brain can be measured by a large-scale device such as MRI or SQUID,
Recently, it is possible to measure easily with a small headgear type device using an infrared laser array. This device has only a few to several tens of infrared semiconductor lasers and light receiving sensors (both of which are several millimeters square) arranged in the headgear. It is simple and lightweight, inexpensive and easy to use anywhere. You can Further, for example, the expansion and contraction of the pupil of the eye can be observed by a small camera, and the amount of expansion and contraction can be easily quantified by an image recognition system combined with the camera. The pupil / iris recognition system for security is a category of its application, but it is already on the market. Also, for example, heart rate can be continuously detected in real time with a small sensor of a few millimeters square, and blood pressure measurement is already frequently used by the general public for daily health check, and nowadays. Measurements can be made in tens of seconds by simply passing one finger through the ring.

[0009] In addition to the biological reactions of humans represented by the examples given above, changes are often brought about by movements and works that move the body. However, not only that, but also when no exercise or work is performed, the biological reaction changes. That is, mental activity causes a change in biological reaction. Some types of mental activity are conscious and some are unconscious. Some are active and some are passive. It is very common for a person to change or react to mental activity in response to information received from the outside through his five senses, even if he / she does not perform a large physical action. In addition, the mental change / reaction causes a biological reaction. Surprise and excitement, for example, can affect any biological response, increase heart rate, raise blood pressure,
It may also cause the pupil to momentarily contract or increase the sweating of the palm, and naturally cause changes in the blood flow, current, etc. at specific parts of the brain.

When a human is viewing content such as video or audio, the video or audio stimulates the viewer to cause a mental reaction or activity. Of course, since viewers watch the content in search of excitement, excitement, comfort, fun, and fun, it is natural that a mental reaction or activity occurs as a result.

[0011] In other words, how much a certain content gives the viewer excitement, excitement, comfort, humor, and fun,
Further, what part and which scene of the content gives the viewer impression, excitement, comfort, weirdness, and fun can be a kind of evaluation of the viewer for the content. Then, the evaluation can be collected as information as the biological reaction of the viewer, which reflects the mental reaction / activity of the viewer. Then, after processing the collected biological reactions into data of an appropriate shape, and storing it in association with the content or the scene in the content, by using this data,
It will be possible to accurately search and select content that meets the desires of viewers from an enormous amount of archives.

[0012] In this way, the viewer's evaluation of a certain content is collected as the biological reaction that appears as a result of the mental reaction / activity, and is processed into the data associated with the original scene and the content and retrieved. To use. At this time, this created data evaluates the viewer's reaction, and as a result, the evaluation of the scene and the content, and since it is associated with the original scene and the content as individual tags (tags), This is referred to as an AR tag for Appraised Response Tag (Appraised: Appraised means appraising <person / ability, etc .; recognizing <situation, etc.>).

The present invention derived from the above consideration will be described below.

The AR tag in the present invention is marker data for image and voice data, which is characterized in that the biological reaction of the viewed human being to the target image or voice is processed and is associated with the image or voice. .

Further, the AR tag in the present invention processes the biological reaction of the viewed person to the desired image or sound in association with the emotional reaction of the viewed person and associates it with the image or sound. It is the marker data for the image and audio data characterized by the above.

The AR tag according to the present invention is the biological response of the human being who has viewed the target image or sound, and the existing tag data already associated with the target image or sound data. Is the tag data for the image and audio data, which is characterized in that it is processed by using the above and is associated with the image or audio.

Further, the AR tag in the present invention is characterized in that the biological reaction of the viewed human to the target image or sound is associated with any continuous data fragment of the target image or sound. It is the tag data for the image and audio data.

The present invention also provides a method of processing marker data for image and sound data, which is characterized by processing the biological reaction of a human being who has viewed the target image or sound and relating it to the image or sound. is there.

Further, the present invention is characterized in that the biological reaction of the viewed human being with respect to a desired image or sound is processed in association with the emotional reaction of the viewed human, and is associated with the image or sound. It is a method of processing the sign data for the image and sound data.

Further, the present invention utilizes both the biological response of the viewed human to the desired image or sound and the existing sign data already associated with the desired image or sound data. The method is a method of processing marker data for image and audio data, which is characterized by being processed and then associated with the image or audio.

Further, the present invention is characterized in that the biological reaction of the viewed human to the desired image or sound is associated with any continuous fragment of the desired image or sound. It is a method of processing label data for data.

The AR tag in the present invention is tag data for image and voice data, which is characterized in that it can identify an individual who has caused a biological reaction used for association.

[0023]

BEST MODE FOR CARRYING OUT THE INVENTION (First Embodiment) Next, an embodiment relating to the production and use of an AR tag which is a marker associated with a desired image or sound by processing a human biological reaction according to the present invention Will be described with reference to the drawings.

FIG. 1 shows an A according to a first embodiment of the present invention.
FIG. 6 is a relational diagram showing a procedure for producing an R tag.
First, there is a desired video or audio 11. This video or audio 11 is any kind of event that can be recorded by shooting, recording, or copying, and it may be the event itself that is currently in progress, or it can be played back after it has already been filmed, recorded, or copied. It may be an event that is being reproduced. Such phenomena include various contents such as those usually shown on television or video and movies, such as natural phenomena, sports,
Dramas, varieties, case reports, lectures, interviews, concerts, live broadcasts, live coverage, and entertainment. It may also be audio only, eg radio, CD,
There are various contents typified by what you can listen to with cassette tapes, for example, natural phenomena, music songs and dramas, variety, incident coverage, lectures, conferences, concerts, live broadcasts, live coverage, performances etc. is there. When the reaction provider 12 sees or hears this, the reaction to those images or sounds appears in the biological information of the reaction provider.

The reaction provider 12 is an individual or a plurality of people who provide the data of the biological reaction to the viewing of the event 11 for producing the AR tag,
In particular, it may be a person prepared for that purpose, or a person who happens to watch the video or audio without any purpose. Alternatively, it may be a photographer or a sound recorder who shoots or records the image or sound of the event 11, or may be an operator who reproduces, edits, or processes the image or sound. Further, the person who is executing, configuring, or forming the event when the event of the video or audio is occurring may be the person. It may also be a specific or unspecified large number of viewers who are using the content, or a set of such viewers.

The data collector 13 is a device for collecting and recording data of one or a plurality of types of biological reactions generated by the reaction provider 12, and a part for sensing the biological reactions and the sensed data temporarily. It is composed of a part to be stored in the device, a part to be sent to the processor 14 in the next process, etc., but may be combined to form a single device, or individual devices may communicate by function. It may be connected by means. As such an example, for example, a device including a pulse rate monitor and a sphygmomanometer attached to a reaction provider, a brain current measuring device, a muscle ammeter, and the like,
Or a camera for pupil photography installed near the reaction provider,
It may be a device including a data collection device such as a facial expression camera that is not directly attached to the reaction provider.

Further, the data collector 13 may be a device specifically for collecting and transmitting the biological reaction data, or may be a device combined with other functions. For example, it may be a photographing device for recording the event 11 itself such as video or audio, a recording device, or a combined device of a recording device and a biological reaction collecting device. As such an example, for example, a cameraman shooting a baseball broadcast and devices around the cameraman can be considered. In this case, the video and audio 11 is a baseball game actually being held in front of the eyes. The reaction provider 12 is a cameraman. The data collector 13 is a camera, and a device having a function of detecting a biological reaction of a cameraman and a device for recording or relaying the collected biological reaction to another camera are added to the camera. Alternatively, the data collector 13 may be a camera that records the event 11 such as video or audio and a device that detects and collects and records the biological reaction of the cameraman by communication means.

The processor 14 in the next step of the data collector 13 is a device for processing one kind or plural kinds of biological reactions generated by one or a plurality of reaction providers collected into an appropriate form to obtain AR data. In addition, it is also a device for performing a work of associating with the scene or content corresponding to the video or audio 11 of the original event. There are various processing methods, and data of multiple biological reactions may be simply added and then standardized, or weighting may be performed according to the types of multiple data, and calculation may be performed based on a certain function. Good. The size of the data may be various, and is usually considered to be around several bits to several tens of bits.
In addition, a plurality of data having such a bit number are provided to form one A
It may be an R tag or a combination of AR tags may be a new AR tag. Also identify the reaction provider 12,
The information may be incorporated in the AR tag or may be metadata separately from the AR tag.

The processed and generated AR tag is associated with the video or audio 11 of the original event, which may be associated with the entire content, a certain scene in the content, or the scene. It may be a shorter unit or frame in. However, in each case, the position information of the AR tag and the position associated with the AR tag match. That is, the AR tag associated with the entire content needs to be processed and generated as an evaluation for the entire content, and the AR tag associated with each certain scene needs to be processed and generated as an evaluation for each scene. The situation is conceptually shown in FIG. That is, one or a plurality of AR tags can be associated with the entire data 30 of the content itself. The AR tag in this case is processed and generated as an evaluation for the entire content. Also, one or a plurality of AR tags can be associated with a certain scene 31, which is a fragment of the content, and the same is true for a fragment 32 having a shorter length.

In some cases, the generated AR data is added to the recorded data itself of the video or audio 11 of the original event. Thereby, the tagged data 15 which is the recorded data of the video or audio 11 of the original event to which the AR data is added is formed. This data is stored and stored in a certain storage medium 16. This storage medium 1
6 may be a database attached to the system on the computer, or a server or gateway including them. Further, it may be a video or audio reproducing device or a transmitting device having a storage medium corresponding to video or audio in home electric appliances in the home. Further, an easily transportable storage medium represented by an optical recording medium such as a DVD, a magnetic recording medium such as a VHS video tape, or a solid-state memory such as a flash memory may be used.

In some cases, the generated AR tag may be stored separately from the data 17 which is the content recording data itself such as the video or audio 11 of the original event. The AR tag 19 may be stored in the same storage medium 18, while the data 17 is recorded in 18 in one storage medium, or may be stored in a completely different storage medium 20. In this case, the fact that the data 17 and the AR tag 19 are separated is different from the above case. Even if they are separated and stored in different places, the AR tag 19 is related to the original data 17 and the storage place can be specified. Even if the AR tag itself cannot specify the storage location of the original data 17, the original data 17
The storage location of the associated AR tag 19 may be stored in a database 21 and stored in another location 21.
The storage medium 18 and the storage medium 20 may be a database attached to a system on a computer, a server or a gateway. Further, it may be a video or audio reproducing device or a transmitting device having a storage medium corresponding to video or audio in home electric appliances in the home. Also D
A storage medium that can be easily carried, such as a VD or VHS video tape, or a solid-state memory such as a flash memory may be used.

(Second Embodiment) Next, A in the present invention
Embodiments relating to the use of R tags will be described with reference to the drawings.

FIG. 2 is a relational diagram showing a method of using the AR tag in the present invention. General audience 22
Is an unspecified person who usually watches TV, radio, and the Internet, and can be anyone. This general viewer 2
2 is one or more keywords that represent the scene or content that you want to watch.
Send to 3. This processor 23 may be a database attached to the system on the computer, or a server or gateway including them. Further, it may be a video or audio reproducing device or a transmitting device having a storage medium corresponding to video or audio in home electric appliances in the home. Further, an easily transportable storage medium represented by an optical recording medium such as a DVD, a magnetic recording medium such as a VHS video tape, or a solid-state memory such as a flash memory may be used. If the keyword sent to the processor 23 is a concrete one represented by "place", "time", "person", "phenomenon", "event", etc., and can be expressed by simple text, first, The contents and scenes are searched and selected using the metadata classified and labeled by the existing technology. Further, if the keyword is an ambiguous word such as “interesting”, “fun”, “sad”, “impressed”, “impressed”, “funny”, “calm”, “nagomu”, or a sensory word, the processor 23 uses AR Use tags to find archived data that can meet your needs. At this time, the processor 23 stores the video and audio content files with the AR tag stored in the storage medium 24.
The AR tag attached to the data is searched for, selected, and delivered to the general viewer 22. At this time, the processor 23 may directly search the tagged data to which the AR tag is added. In addition, the processor 23 stores the AR stored in the storage medium 26.
The tag 27, which is a tag, may be searched, and from the result, the original video or audio data 29 on the storage medium 28 may be retrieved, retrieved, and sent to the general viewer 22.

Alternatively, the processor 23 itself is a server provided with a database relating to the tagged data 25 and the tags 27, and the tagged data 25 may be taken out from the database and sent to the general viewer 22. Retrieve the tag 27 from the database and add the tag 27
The data 29 may be selected and sent to the general audience 22.

The AR tag sent to the general viewer 22 may remain as it is, but now the general viewer 22
Incorporating a new history that corresponds to the search request of
It may be processed by the processor 23. That is, FIG.
The search and use by the AR tag represented by the above may be reused as the start of the AR tag generation described in FIG. Further, the general viewer 22 who has made the search request may be identified, and that information may be newly incorporated as AR data.

[0036]

EFFECT OF THE INVENTION By using the AR tag of the present invention, it is originally difficult to define a specific keyword from stored information such as video and audio. However, an impression or a sensory keyword cannot be tagged. Regarding the event that could not be searched accurately because it was difficult in practice,
It becomes possible to automatically generate and associate a tag tag that serves as a clue for the search, and as a result, it becomes possible to search easily and quickly.

[Brief description of drawings]

FIG. 1 is a relational diagram showing a procedure of manufacturing an AR tag according to a first embodiment of the present invention.

FIG. 2 is a relational diagram showing an implementation procedure regarding use of an AR tag according to the first embodiment of the present invention.

FIG. 3 is a schematic diagram showing a method of adding an AR tag in the second embodiment of the present invention.

[Explanation of symbols]

11 Events including video and audio 12 Reaction provider 13 Data collector 14 Processor 15 Content data with tags 16 Storage media 17 Content data 18 Storage media 19 AR tags 20 Storage media 21 Database or server 22 General audience 23 Processor 24 Storage media 25 Tagged content data 26 Storage media 27 AR Tag 28 Storage media 29 Content data 30 Content data as a whole 31 Content data fragment (scene) 32 Content data fragments (frames)

─────────────────────────────────────────────────── ─── Continuation of front page (51) Int.Cl. ⁷ Identification code FI theme code (reference) H04N 7/08 H04N 7/08 Z 7/081

Claims (9)

[Claims] 1. Marking data for addition to an image or voice data, characterized by processing a biological reaction of a human being who has viewed the target image or voice or the like and associating it with the image or voice. 2. The biological reaction of the viewed person to a target image or sound is processed in association with the emotional reaction of the viewed person, and the processed image is related to the image or sound. Label data for addition to the image and audio data according to Item 1. 3. The processing using both the biological reaction of the viewed person to the target image or sound and the existing marker data already associated with the target image or sound data. The marker data for addition to the image and voice data according to claim 1 or 2, which is newly associated with the image or voice. 4. The method according to claim 1, wherein the biological response of the viewed human to the target image or sound is associated with any continuous data fragment of the target image or sound. Addition marker data to the image and audio data according to any one of 3 above. 5. A method of adding tag data for addition to image or voice data, characterized by processing a biological reaction of a human being who has viewed the image or voice of interest and associating it with the image or voice. 6. The method according to claim 6, wherein the biological reaction of the viewed person to the target image or sound is processed in association with the emotional reaction of the viewed person, and the processed image is related to the image or sound. 5. The method of adding the tag data for addition to the image and audio data according to 5. 7. The processing using both the biological response of the viewed human to the target image or sound and the existing marker data already associated with the target image or sound data. 7. The method of adding tag data for addition to the image or voice data according to claim 5 or 6, which is newly associated with the image or voice. 8. The method according to any one of claims 5 to 7, wherein the biological response of the viewed human to the target image or sound is associated with any continuous fragment of the target image or sound. A method of adding tag data for addition to the image or audio data according to any one of the above. 9. The image according to any one of claims 1 to 4, wherein in the labeling data according to any one of claims 1, 2, 3 and 4, an individual who has caused a biological reaction used for association can be identified. , Marker data for addition to voice data.