JP2008072572A - Content photographing apparatus - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To provide a content photographing apparatus capable of converting voice-inputted event information into metadata through voice recognition without increasing costs. <P>SOLUTION: During a recording mode or a recording standby mode, an explanatory voice inputted from a voice input means is filtered by a filtering means, an output level is then detected and in a case where the output level is equal with or higher than a preset output level for a period of time as long as or longer than a preset output term, a voice tag containing information of time when the input of the explanatory voice is started, is generated, the voice tag containing the time information is associated with time information of the explanatory voice, and the voice tag is recorded on an information storage medium as metadata. <P>COPYRIGHT: (C)2008,JPO&INPIT

Description

  The present invention relates to a content creation device that creates audio metadata for event information inputted during shooting by voice recognition.

  Normally, scenarios are created based on storyboards in movies and dramas, but scenarios rarely exist for shooting professional news, documentaries, variety shows, live performances, concerts, and the like. This is the same for amateur sports events, entrance ceremonies, graduation ceremonies, music recitals, weddings, and other photography. In such a state, the photographer captures an event that occurs in real time within or beyond the prediction range of the photographer.

  In other words, it is necessary to respond flexibly according to events that occur sequentially when editing after shooting. However, events such as the above-mentioned live performance stages, concerts, sports events, entrance ceremonies, graduation ceremonies, music recitals, weddings, etc. occur according to the progress of the program, but there are scenarios for the events that occur. do not do. In other words, the event occurs in real time with contents exceeding the photographer's expectations. Therefore, it is necessary to create information necessary for editing a captured image as metadata and save it together with the captured image data.

  For that purpose, depending on the event that occurs at the time of shooting, for example, the photographer or another person puts on the memo paper the shooting date (date, morning, day, night, etc.), shooting method (lens, camera, shot, light source, etc.) Take note of information about points of interest such as event attendees (line of sight, movement, facial expressions, tension, makeup, costume status, etc.), words (keywords such as ad lib), and sounds (sounds). Then, it is necessary to create metadata based on the memo after shooting.

  However, at the time of shooting, it is not uncommon to record in real time handwritten notes about the status (contents) of the event that occurred and the facial expression of the person. For this reason, a dedicated assistant is required to create metadata based on the recording related to content shooting and assign it to the shot image.

  As a metadata input method and editing system that does not require real-time memo writing as described above and a dedicated assistant, the one described in Patent Document 1 is known. Specifically, when metadata related to content is created or tagged, a keyword extracted in advance from a scenario or the like of the produced content is input by voice. Based on the scenario, dictionary fields are set and keywords are prioritized, and metadata is created by the voice recognition means. According to this method, even when metadata is given at intervals of several seconds, which is difficult by key input, it is possible to assign metadata efficiently by using voice recognition. A scene search can also be performed as a keyword for searching metadata.

Japanese Patent No. 3781715

  However, in the above-described conventional configuration, it is necessary to mount a voice recognition engine in the camera that can recognize keywords input in real time in real time. In order to implement a speech recognition engine, it is necessary to change the basic design, such as adding a speech recognition microcomputer processing system to the camera, which increases costs. In addition, the real-time speech recognition engine, which is the main cause of the cost increase, is useless except when a keyword is input at the time of shooting, and is not cost effective.

  Therefore, an object of the present invention is to provide a content photographing apparatus capable of converting event information inputted by voice into metadata by voice recognition without increasing cost.

In order to achieve the above object, the content photographing apparatus of the present invention provides:
A content photographing apparatus that converts any content including video, audio, or data that can be accessed at an arbitrary position with time information into a stream, and records the information in an information storage medium in combination with metadata related to the content,
Commentary voice filtering means including additional information of the content input from the voice input means when in the recording mode or the recording standby mode;
Output level detection means of the filtering means;
Means for generating a voice tag including time information when the input of the commentary voice is started when the output level of the filtering means is equal to or higher than a preset output level over a period longer than a preset output period. When,
Means for associating the audio tag including time information with the time information of the commentary audio and recording the audio tag as metadata in the information storage medium.

In the content photographing apparatus of the present invention,
Voice recognition means for accessing the commentary voice using the voice tag, inputting the commentary voice and converting it into text data;
It is preferable to further comprise means for associating the text data converted by the voice recognition means with time information of the photographed content.

  In the content photographing apparatus of the present invention, it is preferable that the conversion of the commentary sound into text data is performed in non-real time when neither the recording mode nor the reproduction mode is used.

  The content photographing apparatus of the present invention further comprises means for reproducing content including the search keyword from the recording medium when the search keyword input from the search means matches at least a part of the metadata. Is preferred.

In the content photographing apparatus of the present invention,
Means for accessing the commentary voice using the voice tag and outputting the commentary voice outside the device;
Means for inputting the text data of the commentary speech converted outside the device;
It is preferable to further comprise means for associating the text data with time information of the captured content.

  In the content photographing apparatus of the present invention, the zoom state of the lens unit, the aperture value, the focal length, the shutter speed, the horizontal or vertical tilt angle of the lens unit, and the angular velocity of the lens unit that rotates in the horizontal or vertical direction Or any one of the lens unit operation data of the moving acceleration in the front / rear / right / left direction or the vertical direction of the lens unit, input data by the user, and data obtained by performing a predetermined calculation process on the operation data It is preferable to further comprise control means for receiving the received data from the camera control means and temporarily storing the received data in association with the corresponding video frame as metadata.

  The content photographing apparatus of the present invention preferably further includes means for setting a priority for each of the recorded metadata and recording the priority on the information recording medium as additional information of each metadata.

In the content photographing apparatus of the present invention,
Read priority setting means,
It is preferable to further comprise means for outputting metadata information having a higher priority than the priority set by the read priority setting means.

  In the configuration of the present invention, the CPU load for operating the speech recognition engine can be reduced, so that speech metadata is converted into text metadata even in a consumer movie equipped with an inexpensive CPU of about several tens of MIPS or more. This makes it possible to reduce the introduction cost.

  Before specifically describing the embodiments of the present invention with reference to the drawings, the basic technical features of the present invention will be described. In the method proposed in Patent Document 1 described above, since the load for creating metadata in real time based on the sound input at the time of shooting is enormous, the demands on the resources of the content imaging apparatus must be severe. Absent.

  In the present invention, the voice recognition processing for voice input is not performed as real-time processing at the time of shooting, but is processed batchwise after shooting or when resource requirements for shooting are relaxed. In other words, in the present invention, primary metadata that is position information indicating the position of the speech recognition target portion is created in the sound input in real time during shooting. As the position information, the time code of the input audio data or the byte / bit position information in the file is used.

  As a result, the speech recognition engine can access the recognition target portion of the input speech data when the resource request is not tight based on the primary metadata. Based on the voice recognition result, the keyword is detected as text, and secondary metadata necessary for editing the content after shooting is created.

  Therefore, the CPU can process voice recognition in non-real time with a small load, instead of processing in real time with a large load. In other words, since the CPU load for operating the voice recognition engine can be reduced, it is possible to convert voice metadata into text metadata even in a consumer movie equipped with an inexpensive CPU of about several tens of MIPS. Therefore, it has a great merit that the introduction cost can be lowered. Needless to say, if the CPU power is sufficient, voice recognition processing may be performed in real time.

  A content photographing apparatus according to an embodiment of the present invention will be described with reference to FIG. In FIG. 1, the content photographing device Acc is represented as an example of a system model that creates video data, audio data, and metadata on a recording medium (or buffer memory) in the camera 101.

  In FIG. 1, reference numeral 101 indicates a camera, reference numeral 102 indicates a lens unit of the camera 101, reference numeral 103 indicates a microphone of the camera 101, and reference numeral 104 indicates a photographing target of the camera 101. Note that the imaging target 104 is, for example, an object such as a landscape, an animal such as a person or a pet, a car, or a building.

  Reference numeral 114 indicates a metadata input button, reference numeral 105 indicates data captured by the camera 101, and reference numeral 106 indicates an AV stream data file 106 including metadata. Reference numeral 107 indicates metadata such as shooting scene information IS (scene number, cut number, take number, adoption of the recorded take, non-adoption, hold). Reference numeral 109 indicates a remote controller for the camera 101. The user operates the metadata input button 114 and the remote controller 109 to input the metadata 107 to the camera 101. The image sensor used in the camera 101 is preferably composed of a CCD, C-MOS, or the like.

  Reference numeral 108 indicates a data sequence photographed by the camera 101. In the data sequence 108, video data, audio data, and metadata 107 are arranged on the time axis. The metadata 107 is handled as text data in text format, but may be data in binary format. The data sequence 108 includes clips # 1 to # 5 in a specific scene.

  Reference numeral 110 indicates a data sequence in which clip # 1 to clip # 5 are connected by editing. Reference numeral 111 denotes a television that can be connected to the camera 101. Reference numeral 112 indicates a connection cable for transmitting a signal from the camera 101 to the television 111. Reference numeral 113 indicates a connection cable for transmitting a signal from the television 111 to the camera 101. The user can operate the remote controller 109 at a location away from the camera 101 and display a list of each scene on the television 111 in the order of the edited data sequence via the signal cable 112.

  Reference numeral 115 denotes a microphone that detects sound and inputs it to the camera 101 as an audio signal, similarly to the microphone 103. Reference numeral 117 indicates a microphone built in the camera 101. However, the microphone 115 is connected to the camera 101 with a cable or the like to record the sound far away from the camera 101 compared to the case where the microphone 103 and the microphone 117 are directly attached to the camera 101 and record the sound near the camera 101. Used for. As will be described later, the microphone 115 can use an optical sensor instead of the microphone.

  A list display by the television 111 will be briefly described. In the screen of the television 111, the horizontal axis represents the passage of time, and each clip is divided into an effective part and an invalid part. In the list display of the television 111, a state in which representative clips of each of the three effective portions are displayed on the screen as representative thumbnails is illustrated. This representative clip may be the first frame of each effective portion or a representative frame in the middle of the effective portion.

  The metadata input button 114 described above is preferably composed of three buttons. When the user operates the metadata input button 114 in an important scene while shooting with the camera, the important shooting scene (clip) can be marked (referred to as “marking function”). The mark indicating the important clip is also metadata 107, and by using this metadata 107, the clip (the video of the head or the representative frame of the clip, or the thumbnail video thereof) marked by the mark search after shooting is used. ) Can be called quickly. The three buttons of the metadata input button 114 are, for example, the first button for registering important clips, the second button for mode switching for enabling button operation or switching to the character input mode, and the like. The first button is used for canceling registration.

  It is also possible to switch to a mode in which the period during which the first button is pressed is registered as an important clip. Further, the mode can be switched to a mode in which 5 seconds before and after the first button is pressed, or a total of 15 seconds, 5 seconds before and 10 seconds after, are registered as important clips. If there are three buttons, many functions can be realized by combining the type of button to be pressed, timing, and pressing length.

  The shooting scene information IS input as the metadata 107 is associated with the time code of the clip. The metadata 107 associated with the time code is generated as new metadata 107 by electronically associating it with a clapperboard sound or recorded content within the main body of the camera 101. As a result, not only the immediate access to the time when the clapper is sounded but also the deletion of unnecessary recorded data before the time when the clapper is sounded and the rearrangement of scenes and cuts where the recording results are adopted can be easily performed. For example, when shooting an athletic meet, a frame image at the start timing of a long distance competition such as a game (short distance competition), a relay, a tug of war, and a ball insertion can be called immediately.

  The user can rearrange the clips by selecting the start position (time) and the end position (time) or length of each clip from a data sequence that is a shooting material shot by the camera. Also, when displaying each clip on a TV monitor, etc., the most characteristic frame of the clip such as a fixed frame image before or after panning or zooming, or the frame (or field) video at the beginning or the end of the clip, The clip can be represented as a representative image.

  Note that movie button operations such as recording, pause, and stop, and the photographer's voice can be detected by the microphone 115 and registered as metadata associated with (marked) a specific time code of the clip. As an example of a photographer's voice, there is meta-information related to a photographing target. Specific examples include the shooting date (date, morning, day, night, etc.), shooting method (lens, camera, shot, light source, etc.), event participants (line of sight, movement, facial expression, tension, makeup, costume status) Etc.), words (keywords such as ad lib), sound (sound), and other information related to the shooting of content such as attention points.

  If the audio signal input from the microphones 103, 115, or 117 is equal to or higher than a preset level, the camera 101 determines that there is an audio input there, and uses the input audio as the time code of the shooting clip (file). Associate. The camera 101 generates audio metadata (also referred to as an audio tag) by creating a list including the primary metadata described above and information associated with the time code. Further, the camera 101 forms a file of the audio metadata together with the input audio and records it in the camera body.

  As described above, audio input as audio of a predetermined level or higher set in advance is associated with the clip time code in audio metadata. The input audio is recorded together with the audio metadata in a channel dedicated to audio metadata that has a one-to-one correspondence with the time code.

  Therefore, if the audio metadata list is viewed, it is possible to access the audio portion recorded in the audio metadata dedicated channel. The voice recognition engine built in the camera accesses the voice part at a higher speed than the voice position information of the file and converts the voice into text. The converted text data is then added to the metadata list to generate the secondary metadata described above.

  As described above, since the metadata related to the shooting event can be recorded as text data, it is possible to improve the searchability of shooting contents in applications such as editing and viewing.

  Usually, the voice recognition engine is realized by executing software for voice recognition on a CPU. In the present embodiment, since the input voice is filed together with the voice metadata and recorded in the camera body, the voice recognition engine can manage the management information (for example, time code or byte / bit in the file) of the captured content. Using the location information etc., the above-mentioned audio metadata can be accessed whenever necessary. Therefore, the CPU can execute voice recognition in non-real time when the load is low when shooting or reproduction is not performed, instead of executing the voice recognition in real time when the load is high as in shooting.

  That is, according to the present embodiment, it is possible to reduce the CPU load when operating the speech recognition engine. As a result, it is possible to operate the speech recognition engine in non-real time and convert the input speech to text metadata, which is the secondary metadata, even for consumer movies equipped with an inexpensive CPU of about several tens of MIPS. Become. Note that the voice recognition engine is not necessarily built in the camera. A voice recognition engine may be built in the PC, and a file including input voice and voice metadata may be transferred to the PC and processed by the PC, or may be processed by connecting the camera to the PC with a cable.

  The internal configuration of the camera 101 will be described with reference to FIG. Inside the camera 101 are a zoom control unit 201, a focus control unit 202, an exposure control unit 203, an image sensor 204, a shutter speed control unit 205, a camera microcomputer 206, an absolute tilt sensor 207, an angular velocity sensor 208, front / rear / left / right / vertical. Acceleration sensor 209, user input system 210, camera signal processing unit 211, audio processing system 212, H.P. A H.264 encoder 213, a recording medium 214, and an output interface 215 are provided.

  The zoom control unit 201 controls the zoom operation of the lens unit 102. The focus control unit 202 controls the focus operation of the lens unit 102. The exposure control unit 203 controls the exposure adjustment operation of the lens unit 102. A shutter speed control unit 205 controls the shutter speed adjustment operation of the image sensor 204. The absolute tilt sensor 207 detects the absolute tilt of the camera 101 in the horizontal / vertical direction. An angular velocity sensor 208 detects the angular velocity of the camera 101 in the horizontal / vertical direction. The acceleration sensor 209 detects the front / rear / left / right / vertical acceleration of the camera 101.

  The user input system 210 receives a user operation with a button or the like and generates an instruction signal. The audio processing system 212 receives input from the built-in microphone 117, the external microphone 103, or the microphone 115. H. The H.264 encoder 213 detects clapperboard sound from the sound input to the sound processing system 212 and generates clapperboard sound detection metadata.

  The operation parameters of the image sensor 204 include chromaticity space information of three primary colors, white coordinates, and gain information of at least two of the three primary colors, color temperature information, Δuv (delta uv), and three primary colors or luminance signals. There is at least one imaged image element operation data of gamma information. In the present embodiment, as an example, chromaticity space information of three primary color points, gain information of R (red) and B (blue) of three primary colors, and gamma curve information of G (green) are handled as metadata. Shall. If the chromaticity space information of the three primary color points is known, the range in which color reproduction in the color space is possible is known. If the gain information of R (red) and B (blue) among the three primary colors is known, the color temperature can be determined. Further, if the G (green) gamma curve information is known, the gradation expression characteristic can be known.

  Lens zoom information, lens focus information, lens exposure information, image sensor shutter speed information, horizontal / vertical absolute tilt information, horizontal / vertical angular velocity information, front / rear / left / right / vertical acceleration information, user's information Input button information, scene number, cut number, take number, information on adoption, non-adoption, hold, etc. of the recorded take, chromaticity space information of the three primary colors, R (red) and B (blue) of the three primary colors Gain information and G (green) gamma curve information are handled as metadata 107 (referred to as camera meta) in the camera microcomputer 206.

  Information (image data) captured by the image sensor 204 is subjected to processing such as pixel defect correction and gamma correction in units of pixels by the camera signal processing unit 211, and the H.264 data. After being compressed by the H.264 encoder 213, it is stored in the recording medium 214 together with the camera meta described above. H. The AV output from the H.264 encoder 213 and the camera meta output from the camera microcomputer 206 are output from the output interface 215, respectively.

  Next, with reference to FIG. 3, the AVC method that is a video compression method and the AAC method that is an audio compression method employed in the present embodiment will be described. FIG. 3 shows a detailed configuration of the video and audio compression engine and its peripheral processing means in the AV signal compression / recording control unit included in the camera 101. In the figure, reference numeral 301 indicates a video encoding unit, reference numeral 302 indicates a VCL (Video Coding Layer) -NAL (Network Abstraction Layer) unit buffer, and reference numeral 303 indicates an AAC-based audio encoding unit. ing.

  Reference numeral 304 indicates a PS (Parameter Set) buffer, reference numeral 305 indicates a VUI (Video Usability Information) buffer, reference numeral 306 indicates an SEI (Supplemental Enhancement Information) buffer, and reference numeral 307 indicates a non-VCL-NAL. A unit buffer is shown. Reference numeral 308 denotes an MPEG-PES packet generator. Reference numeral 309 indicates an MPEG-TS (MPEG Transport Packet) generation unit, reference numeral 310 indicates an ATS (Arrival Time Stamp) packet generation unit, and reference numeral 311 indicates an EP-map generation unit.

  As shown in FIG. 3, the video signal is temporarily stored in the VCL-NAL unit buffer 302 after being converted into data in the VCL-NAL unit format by the video encoding unit 301. The voice signal, the external input PS data, and the external input VUI data are converted into data of the Non VCL-NAL unit format by the voice encoding unit 303, the PS buffer 304, and the VUI buffer 305, respectively, and then the Non VCL-NAL unit buffer. It is temporarily held at 307. Similarly, the real-time metadata is converted into data in the non-VCL-NAL unit format by the SEI buffer 306 and then temporarily stored in the non-VCL-NAL unit buffer 307.

  The MPEG-PES packet generation unit 308 is based on the VCL-NAL unit format data output from the VCL-NAL unit buffer 302 and the Non VCL-NAL unit format data output from the Non VCL-NAL unit buffer 307. MPEG-PES packet (indicated as “PES” in FIG. 3) is generated. Further, the MPEG-TS generator 309 generates an 188-byte MPEG-TS (indicated as “TS” in FIG. 3) based on the MPEG-PES packet output from the MPEG-PES packet generator 308.

  The ATS packet generation unit 310 adds a 4-byte header including a time stamp to each of the MPEG-TS (188 byte length) output from the MPEG-TS generation unit 309 to generate a 192-byte ATS packet (in FIG. 3). Display "ATS"). This time stamp indicates the time at which each MPEG-TS packet arrives at the ATS packet generator 310. The clock for the time stamp is 27 MHz. All 4 bytes may be time stamps. Further, 30 bits of 4 bytes can be used as a time stamp, and the remaining 2 bits can be used as a flag for content protection.

  In addition, the EP-map generation unit 311 makes a pair of the PTS (Presentation Time Stamp) of the first picture of each GOP (Group of Picture) included in the stream and the serial number of the first ATS in the first picture of each GOP. Output as MAP. Since PTS and DTS (Decode Time Stamp) are included in the header of the PES packet, extraction is easy. The serial number of the first ATS in the first picture of each GOP is a number obtained by sequentially counting the number of ATSs from the stream head, with the serial number of the ATS at the head of the stream being 1. The relationship between EP-MAP, which is a pair of PTS and ATS serial number of the first picture of each GOP, and stream editing and playlist will be described later.

  H. The H.264 / AVC format is described in detail, for example, in “H.264 / AVC Textbook”, supervised by Okubo Satoshi, published by Impress Corporation. MPEG-TS (Moving Picture Experts Group, Transport Stream) signals are defined in IEC 61883-4. MPEG-TS is a collection of a plurality of MPEG transport packets (abbreviated as “TS packets”). The TS packet is a 188-byte fixed-length packet whose length is consistent with the ATM cell length (of 53 bytes, the ATM payload is 47 bytes), and applicable when performing error correction coding such as Reed-Solomon codes. Has been determined in consideration of.

  The TS packet includes a 4-byte fixed-length packet header, a variable-length adaptation field (adaptation field), and a payload (payload). PID (packet identifier) and various flags are defined in the packet header. The type of TS packet is identified by this PID. Adaptation_field and payload can be either only one or both, and the presence / absence can be identified by a flag (adaptation_field_control) in the packet header. The adaptation_field has information transmission such as PCR (Program_Clock_Reference) and a stuffing function in the TS packet for making the TS packet have a fixed length of 188 bytes.

  In the case of MPEG-2, the PCR is a time stamp of 27 MHz, and the PCR value is referred to reproduce the reference time at the time of encoding by the STC (System Time Clock) of the decoder. The clock of the time stamp added to each TS packet is equal to the MPEG system clock frequency, for example. Further, the packet transmitting apparatus receives the TS packet, removes transmission jitter added to the PCR by MPEG-TS network transmission from the time stamp added to the received TS packet, and reproduces the MPEG system clock. Clock recovery means is provided.

  In MPEG-2 TS, the STC of the decoder has a PLL motive function by PCR. In order to stabilize the PLL synchronization operation, the PCR transmission interval is defined within 100 msec in the MPEG standard. An MPEG-PES packet containing individual streams such as video and audio is divided into a plurality of TS packet payloads having the same PID number and transmitted. The head of the PES packet is configured to start from the head of the TS packet.

  A transport stream can be transmitted by mixing a plurality of programs. In order to enable such mixed transmission, table information representing the relationship between a program included in a stream and program elements such as video and audio streams constituting the program is used. This table information is called PSI (Program Specific Information) and includes tables such as PAT (Program Association Table) and PMT (Program Map Table). PSI such as PAT and PMT is arranged and transmitted in a payload in a TS packet in units called sections.

  In PAT, the PID of the PMT corresponding to the program number is designated. Since PMT describes video, audio, additional data, and PID of PCR included in the corresponding program, TS packets constituting the target program can be extracted from the stream by referring to PAT and PMT. it can. References regarding TS include, for example, CQ Publisher, TECH I Vo. 4. “All of image & audio compression technology (essential technology in the Internet / digital television and mobile communication era)”, supervised by Hiroshi Fujiwara, Chapter 6, “MPEG system for multiplexing images and audio” It is explained.

  Regarding the logical hierarchical structure related to PSI and SI, examples of processing procedures, and examples of channel selection processing, “Channel selection technology in digital broadcast receivers”, Miyake et al., Sanyo Electric Technical Report, VOL. 36, JUNE 2004, No. 74, pages 31-44.

  The types of real-time metadata input to the SEI buffer 306 include video and audio format information, time codes indicating video frames, and other than the above-described metadata. Specifically, metadata obtained by converting general data into metadata, metadata obtained by receiving digital broadcasts from SI (Service Information; program arrangement information), and metadata such as EPG information obtained from an EPG provider. Data, metadata such as EPG obtained from the Internet, and metadata associated with AV content (moving images such as still images, audio, and clips) taken by individuals. Furthermore, lens zoom information, lens focus information, lens exposure information, image sensor shutter speed information, horizontal / vertical absolute tilt information, horizontal / vertical angular velocity information, front / rear / left / right / vertical acceleration information, Button information entered by the user, scene number, cut number, take number, information on adoption, non-adoption, hold, etc. of the recorded take, chromaticity space information of the three primary colors described above, white coordinates, at least of the three primary colors There are two types of camera information such as gain information, color temperature information, Δuv (delta uv), three primary colors, or gamma information of a luminance signal.

  The metadata format includes, for example, properties (attributes) and attributes (attributes) as standard specifications of UPnP and UPnP-AV, such as http: // upnp. org and can be expressed in a description language such as XML (Extensible Markup Language) or BML (Broadcast Markup Language).

  http: // upnp. In org, for example, “Media Architecture V 1.0”, “Content Directory: 1 Service Template Version 1.01”, “MediaServer V 1.0 and MediaRenderer V 1.0”, “Media“ V ” “MediaRenderer V 1.0”, “ConnectionManager V 1.0”, “ContentDirectory V 1.0”, “RenderingControl V 1.0”, “AVTransport V 1.0”, “UPnP-AV Architecture V.83”, etc. The specification is published. As for metadata standards, there are metadata formats determined by EBU P / Meta, SMPTE KLV, TV Anytime, MPEG7, etc. The standardization trend of metadata for

  In addition, in order that a photographer such as a movie, a content creator, or a copyright holder of content adds value to each metadata and collects usage fees according to the usage details and frequency of users who use the content, You can associate metadata that gives value. Metadata giving value to each metadata may be given as an attribute of the metadata or may be given as an independent property.

  Specific information about the recording device and the recording condition will be described as an example. If it is considered that the metadata to be created and registered by the photographer, the content creator, or the copyright holder of the content of the movie device ID, the movie creator, etc. is high and needs to be licensed, the use of the metadata is based on authentication. It is also possible to adopt a configuration in which a configuration for executing the use permission process is incorporated in the present invention.

  In this case, the photographer creates a file in which the captured video content is encrypted, and uploads the encrypted file to a server on the Internet. Then, the description of the encrypted file, a part of the image, etc. can be made public so that the person who likes it can purchase it. In addition, when a valuable news source can be recorded, it is possible to adopt a configuration for auctioning among news departments of a plurality of broadcasting stations.

  By utilizing these metadata, it is possible to efficiently use many AV contents. Specific examples include searching for a desired content from many contents, library classification, extending recording time, automatic display, and content sales. Increasing the recording time can be achieved by reducing the resolution of low-value video content, or by using only audio and still images (for example, MPEG I pictures or H.264 IDR pictures can be extracted) It is possible to take a configuration such as only a picture.

  Next, referring to FIG. The H.264 stream structure will be described. FIG. 4A shows a GOP structure of a video composed of I (including IDR), B, and P pictures.

  FIG. 4B shows that each picture is composed of VCL and Non-VCL NAL units. NAL (video) is a video NAL unit, NAL (Audio) is an audio NAL unit, and NAL (SEI) is a SEI NAL unit. Metadata generated in real time can be inserted into NAL (SEI).

  The metadata generated in real time includes a time code synchronized with a video frame and marking information added by pressing a button in an important scene. As time codes, SMPTE time code (SMPTE 12M), MTC (MIDI Time Code), LTC (Longitudinal Time Code), VITC (Vertical Interval time Code), DV (IEC 61834, IEC 61883) S There are time codes defined by these time codes, and time codes derived from these time codes can be used as metadata.

  FIG. 4C shows the structure of the PES packet. In the PES packet, a PES packet header is added to the picture data shown in FIG. The PES packet header can include MPEG PTS / DTS as a header option. H. From the viewpoint of H.264, a PES packet is handled as 1 AU (Access Unit).

  As shown in FIG. 4D, in this example, the PES packet is divided every 188 bytes to generate an MPEG-TS packet.

  As shown in FIG. 4E, in this example, an ATS packet is formed by adding a 4-byte header including a time code to an MPEG-TS packet.

  Next, the relationship between a playlist and a stream will be described with reference to FIG. As described above with reference to FIG. 3, the ATS packet is received from the ATS packet generation unit 310 by an example in EP-MAP (FIG. 5B), which is a pair of the PTS of the first picture of each GOP and the first ATS serial number. Output) and used for stream editing and playlist creation. FIG. 5A shows an example of a playlist, and the playlist object is “name_2005 athletic meet” having the name “2005 athletic meet”.

  The “name_2005 athletic meet” is made up of two play items (PlayItems) “act” and play item objects “iname_act” and “iname_post” having the name “kakekko”. The IN point and OUT point of “iname_act” and “iname_kakekko” are indicated by a pair of a PTS to which each picture belongs and an ATS serial number from the beginning of the stream (FIG. 5B). The play item specifies a stream, and specifies the position from the head of the stream specified by the ATS serial number in units of 192 bytes. In FIG. 5B and FIG. 5C, “iname_act” is given by (1) to (2) on the stream, and “name_game” is given by (3) to (4).

  Each shooting clip described above with reference to FIG. 1 can be handled in association with each play item. For “iname_act”, the shooting scene information IS (scene number, cut number, take number, adoption, non-adoption, hold, etc. of the recorded take) is linked, and the start time of the important part of the play item is, for example, a clapperboard Can be set to the time when is sounded.

  With respect to “iname_kakekko”, it is possible to access a portion that starts running with kakeko using the detection time of the pistol firing sound. In addition, the program configuration of the event such as “iname_act” and “iname_kakekko” is registered in the device in advance, and the information registered at the time of shooting is selected from the menu displayed on the camera viewfinder, which is a kind of shooting scenario It can also be registered as metadata 107 representing. Since the metadata 107 is electronic data, the program can be registered and the registered contents can be corrected even after the event is over.

  Next, an example of a recording directory structure of a moving image file, a still image file, and metadata in the information recording medium will be described with reference to FIG. In FIG. 6, “Movie”, “Still Picture”, and “Metadata” directories exist under the root.

  Under the “Movie” directory, there are a management file group, a “PLAYLIST” directory, a “CLIPINF” directory, and a “STREAM” directory. In addition, under the “PLAYLIST” directory, there are a “* .rpls” file group that is a real-time playlist (file) and a “* .vpla” file group that is a virtual time-play playlist (file). In the “CLIPINF” (clip information) directory, a group of “* .clpi” files that are clip information files exist. Under the “STREAM” directory, there is a “* .m2ts” file group that is a stream file composed of ATS (192 bytes). Under the “Still Picture” directory, there are “* .jpg” file groups that are still images.

  Under the “Metadata” directory, there are a “META_PLAYLIST” directory and a “USER_METADATA” directory. Also, under the “META_PLAYLIST” directory, there is a “* .mtdt” file group having selected metadata among the metadata existing in the playlist (file). In the “USER_METADATA” directory, there is a “MENU_INF” directory related to movie menu settings. Here, an EDL (Edit Decision List) obtained by simple editing in the movie menu can be stored. In addition, there exists a “USER_PRIVATE” directory that stores private metadata uniquely set by the user. Here, a representative thumbnail, a time code, and the like for CLIP identification can be recorded.

  In FIG. 6, each playlist file associates a clip information file with a metadata file. Each clip information file is associated with a stream file composed of ATS (192 bytes). A major feature not found in the past is that each playlist file associates not only a clip information file but also a metadata file. Thus, a playlist, play item, and stream associated with the metadata 107 can be found by a search using the metadata 107.

  The metadata 107 will be described with reference to FIG. Examples of real-time metadata input to the SEI buffer 306 include button information pressed by a user in an important scene, shooting data (operating color temperature of an image sensor, lens system zoom value, focus, elevation angle data, angular velocity, acceleration, etc. ), Time code, position data, etc. Examples of non (non) real-time metadata include menu information, scene number, cut number, take number, information on adoption, non-adoption, hold, etc. of the recorded take, title list, image recognition data, voice recognition data, imaging Shooting data such as chromaticity space information of the three primary color points of the element, external input file (text of scenario etc. is input from XML, binary data format file input from external interface), index information, format information, still image, thumbnail, etc. Of these, an arbitrary one is selected. For example, a thumbnail of a representative picture, a description of a scene, and a time code are selected.

  Other metadata includes budget data, shooting schedules, shooting location data, shooting costs, equipment and tool lists, vendor lists, and schedules for casts, extras, staff, etc. and employment costs (unit price per hour) There is. For each metadata, for example, by giving priority as an attribute that can be freely set by the user through the camera microcomputer for each category, if metadata conflict occurs in the application, processing from the highest priority it can.

  With reference to FIG. 8, a search operation for a specific shooting scene (picture) will be described. The purpose of the search is cueing, rough editing, creation of a playlist, creation and recreation of a metadata map for search, and the like. Based on the algorithm shown in FIG. 8, the target picture data and the like can be searched as a search result in both the keyword search and the event search. The search result also includes operation information of the image sensor, so that image information reflecting the photographer's intention can be transmitted to the display device to display an image.

  A registered picture changing method will be described with reference to FIG. Note that this method is effective for the purpose of re-registering a correct picture when the picture found by the search method described with reference to FIG. Specifically, a representative image group is displayed on the time axis with a coarse interval of about 1 second centering on the search result picture, and when the nearest picture is designated, the designated picture is centered on the designated picture at intervals of about 5 frames. The representative image group is displayed on the time axis at intervals. When a representative image is further designated at intervals of about 5 frames, a representative image group is displayed on the time axis at intervals of 1 frame around the designated picture. Here, the target frame image can be obtained, and can be re-registered as a representative image or thumbnail of a clip or playlist.

  With reference to FIG. 10, an application using the content photographing apparatus of the present invention will be systematically described. In FIG. 10, video data, audio data, and metadata are recorded in an SD card memory 1002 that is a recording medium of the movie camera 1001. The SD card memory 1002 is inserted into the personal computer 1003, and the recording data is moved. At this time, if the metadata 107 such as the aforementioned important scenes or chapters is already recorded on the SD card 1002, the data is moved to the personal computer 1003 and the playlist is confirmed. If it is OK, a complete package file can be generated by automatically executing rough editing or non-linear editing at that time. In addition, the edited file can be recorded and stored in a medium 1004 such as a DVD-R or DVD-RAM almost automatically. Then, by playing the media 1004 on the DVD player 1005, the edited file can be viewed on the TV 1006.

  With regard to the metadata transmission method, the content can be attached to the content as text data, or the metadata can be attached to the content as binary data. Also, metadata can be attached to the content as a watermark. Further, it is possible to concode the metadata as a watermark embedded in the image data, and to record / reproduce the obtained image data, or to decode and use after transmission / reception.

  In the above-described embodiment, the example in which the metadata 107 and the video data are recorded and stored in the same medium has been described. However, the metadata 107 and the video data are separately stored in two or more associated media. You may do it. In addition, as long as the medium is associated, it is possible to store only metadata, further store only video data, or store metadata and video data.

  Whether the display device has a high image quality function for optimizing display processing including color reproduction by using the shooting metadata when the video signal is output from the imaging device to the display device. Check. The signals handled by this checking means are MPEG-TS, NTSC / PAL baseband signals (digital and analog signals equivalent to SMPTE 259M) and HDTV baseband signals (SMPTE 292M equivalent) described in the above embodiments. Either digital or analog signal) may be used. That is, 480 / 60i, 480 · 30P, 480 · 24P, 720 / 30P, 720 · 24P, 1080 / 60i, 1080 / 60P video signals may be used.

  As described above, the content photographing apparatus of the present invention can be used for shooting news, documentaries, variety programs and amateur sports days, entrance ceremonies, graduation ceremonies, music recitals, weddings, etc. Means that a person or a shooting assistant inputs event information to the camera by means of a microphone, and detects the level of the input sound in the camera and generates audio metadata associated with management information such as time code of the shooting content Means and means for creating a file after listing the metadata. According to the content photographing apparatus of the present invention, it is possible to convert audio metadata into text metadata even in a consumer movie equipped with an inexpensive CPU of about several tens of MIPS.

  Also, the content photographing apparatus of the present invention accesses the audio portion at a higher speed than the audio position information of the file by a voice recognition engine built in or outside the camera, converts the sound into text, and adds it as additional information to the metadata list. Means to add are provided.

  Since the present invention can generate text metadata using audio metadata without increasing the cost, it is highly useful as a content photographing apparatus.

Model diagram of content photographing apparatus according to an embodiment of the present invention Explanatory drawing of the internal configuration of the camera shown in FIG. H. Explanatory drawing of handling of metadata in H.264 compression H. Explanatory drawing of picture structure of H.264 compression and conversion method to MPEG-TS Illustration of the relationship between playlist and stream object Illustration of directory structure for recording streams and metadata Diagram showing an example of metadata classification Model diagram of search algorithm using metadata Model diagram of picture setting method Illustration of application example

Explanation of symbols

DESCRIPTION OF SYMBOLS 101 Camera 102 Camera lens part 103 Camera microphone 104 Camera photographing target 105 Data 106 photographed by camera 106 AV stream file data 107 Metadata 108 Data sequence photographed by camera 109 Remote control 110 Scenes # 1 to # 5 by editing Data sequence 111 connecting TVs (TV)
112 signal connection cable 113 signal connection cable 114 metadata input button (important scene registration button, still image shooting button)
115 Clapper 116 Microphone 201 Zoom control unit 202 Focus control unit 203 Exposure control unit 204 Image sensor 205 Shutter speed control unit 206 Camera unit microcomputer 207 Absolute tilt sensor 208 Angular velocity sensor 209 Acceleration sensor 210 User input system 211 Camera signal processing unit 212 Audio processing System 213 H.I. H.264 encoder 214 Recording medium 215 Output interface 301 Video encoding unit 302 VCL-NAL unit buffer 303 Audio encoding unit 304 PS buffer 305 VUI buffer 306 SEI buffer 307 Non VCL-NAL unit buffer 308 MPEG-PES packet generation unit 309 MPEG TS generation unit 310 ATS packet generation unit 311 EP-map generation unit 1001 Movie camera 1002 SD card memory 1003 Personal computer 1004 Media 1005 DVD player 1006 TV

Claims (8)

A content photographing apparatus that converts any content including video, audio, or data that can be accessed at an arbitrary position with time information into a stream, and records the information in an information storage medium in combination with metadata related to the content,
Commentary voice filtering means including additional information of the content input from the voice input means when in the recording mode or the recording standby mode;
Output level detection means of the filtering means;
Means for generating a voice tag including time information when the input of the commentary voice is started when the output level of the filtering means is equal to or higher than a preset output level over a period longer than a preset output period. When,
A content photographing apparatus comprising: means for associating the audio tag including time information with time information of the commentary audio, and recording the audio tag as metadata in the information storage medium. Voice recognition means for accessing the commentary voice using the voice tag, inputting the commentary voice and converting it into text data;
The content photographing apparatus according to claim 1, further comprising means for associating the text data converted by the voice recognition means with time information of the photographed content.   The content photographing apparatus according to claim 2, wherein the conversion of the commentary sound into text data is executed in non-real time when neither the recording mode nor the reproduction mode is used.   3. The apparatus according to claim 2, further comprising means for reproducing content including the search keyword from the recording medium when the search keyword input from the search means matches at least a part of the metadata. The content photographing apparatus described. Means for accessing the commentary voice using the voice tag and outputting the commentary voice outside the device;
Means for inputting the text data of the commentary speech converted outside the device;
The content photographing apparatus according to claim 1, further comprising means for associating the text data with time information of the photographed content.   The zoom state, aperture value, focal length, shutter speed, horizontal or vertical tilt angle of the lens unit, angular velocity of the lens unit rotating in the horizontal or vertical direction, or the front, rear, left and right of the lens unit Alternatively, at least one of the lens unit motion data of vertical movement acceleration, input data by the user, and data obtained by performing a predetermined calculation process on the motion data is received from the camera control unit, 2. The content photographing apparatus according to claim 1, further comprising control means for temporarily storing the received data in association with the corresponding video frame as metadata.   7. The method according to claim 1, further comprising means for setting a priority for each of the recorded metadata and recording the priority on the information recording medium as additional information of each metadata. The content photographing device described in 1. Read priority setting means,
8. The content photographing apparatus according to claim 7, further comprising means for outputting metadata information having a higher priority than the priority set by the read priority setting means.

Images