JP4906075B2 - Imaging device - Google Patents

Description

  The present invention relates to an imaging apparatus, and more specifically, to an imaging apparatus that notifies a user of image characteristics in cooperation with a feature recognition server that generates data indicating the characteristics of the object from an image.

  In recent years, digital still cameras and digital video cameras having a function of transmitting captured moving images or still images to an external device wirelessly or by wire have been realized. For example, there is a mobile phone with a built-in digital camera and data communication function. In addition, access points for data communication by wireless LAN can be used in various places.

  It is known to recognize an object included in a moving image or a still image and generate character code data indicating its characteristics. By using this, it is possible to search for an image including a desired object, or to search for an image having similar contents using the image as a search key.

In many cases, extraction of an object included in an image requires a high level of data processing capability, and this is imposed on the server (feature recognition server). Patent Document 1 describes a system that transmits an image captured by a camera of a mobile terminal to a server and receives character data that describes an object included in the image, that is, object information from the server.
JP 2003-323620 A

  Patent Document 1 does not describe a method for effectively displaying object information received from a feature recognition server to a user when the user wants to display image data on the display panel of the mobile terminal body. Also, the display when the feature recognition server fails in the search is not considered.

  In view of the above problems, an object of the present invention is to present an imaging apparatus that effectively displays image data and object data acquired from a feature recognition server.

In order to achieve the above object, an imaging apparatus according to the present invention is an imaging apparatus that communicates with an external server capable of searching metadata for describing an object included in image data, and includes a plurality of captured images. Recording means for recording data, list display control means for displaying a plurality of thumbnail images respectively corresponding to the plurality of recorded photographed image data on a display unit, and a plurality of thumbnail images displayed in the list Selection means for selecting one thumbnail image, enlarged display control means for enlarging the selected thumbnail image and displaying it on the display unit, and transmission means for transmitting the recorded photographed image data to the external server Receiving means for receiving a reply to the transmitted captured image data from the external server; and the received reply is the transmission If the metadata corresponds to the captured image data, the name of the object included in the transmitted captured image data and the received metadata and the object included in the received metadata If the received reply indicates a failure in the metadata search, a predetermined value indicating the search failure in the transmitted captured image data is recorded. Recording control means for causing the recording means to record the data in association with each other, and the list display control means records the received image data in association with the received metadata when the list is displayed. About the corresponding thumbnail image, versus the photographic image data corresponding to the thumbnail images without overlay the details The name recorded with the overlay is displayed as an overlay, while the thumbnail image corresponding to the captured image data recorded in association with the predetermined data is overlaid with specific information indicating the search failure, and the enlarged display is displayed. When the selected thumbnail image is a thumbnail image in which the name is displayed as an overlay, the control unit records the name recorded in association with the captured image data corresponding to the thumbnail image and the association image . details, and characterized in that overlays the enlarged to be displayed on the display unit an image.

ADVANTAGE OF THE INVENTION According to this invention, a user can be notified briefly about the metadata acquisition condition from an external server.

  Hereinafter, embodiments of the present invention will be described in detail with reference to the drawings.

  FIG. 1 is a schematic configuration block diagram of a digital video camera 10 according to an embodiment of the present invention, and FIG. 2 is a schematic configuration block diagram of a feature recognition server 50 that cooperates with the digital video camera 10. However, the solid line indicates the flow of image data, and the broken line indicates a control signal.

  The configuration of the digital video camera 10 will be described. The digital video camera 10 includes an imaging unit 12, a compression device 14, a recording / reproducing device 16, a metadata management device 18, an HDD (Hard Disk Drive) 20, a display control device 22, a display device 24, a communication device 26, and a system control device 30. , An operation device 32 and a memory 34 are provided.

  The imaging unit 12 includes an optical lens, an imaging device such as a CCD (Charge Coupled Device) or a CMOS (Complementary Metal Oxide Semiconductor), an autofocus mechanism, a zoom mechanism, an A / D converter, and signal processing (for example, gamma correction, color balance). Adjustment, luminance / color separation), etc., and converts an optical image of a subject into an electrical signal and outputs a digital image signal.

  The recording / reproducing device 16 records and reproduces the image data from the imaging unit 12 on the HDD 20. The recording / reproducing apparatus 16 also generates a thumbnail image of the image data recorded on the HDD 20 and records it on the HDD 20. Note that the recording / reproducing apparatus 16 compresses and decompresses the image data as necessary. In this embodiment, both moving images and still images can be recorded on the HDD 20. MPEG (Moving Picture Experts Group phase) 2 and Motion JPEG are known as compression methods for moving images, and JPEG 2000 is known as a compression method for still images.

  The HDD 20 also records an object information database 20b that stores metadata indicating the characteristics of the recorded image 20a. When the image data is recorded in the HDD 20, the metadata management device 18 generates a metadata record corresponding to the image data in the object information database 20b.

  The object information database 20b corresponds to, as basic elements, unique identification information (ID), a reply flag (Flag) indicating that the server 50 has not been received, the server 50 has been received, and the server 50 has failed to search. It consists of a file path (FilePath) to image data, a name (Name) indicating the name of the found object, and detailed information (MoreInfo).

  FIG. 3 shows a configuration and example contents of the object information database 20b. In the reply flag, “0” indicates that the server 50 has not been received, “1” indicates that it has been received from the server 50, and “−1” indicates that the server 50 has failed to search. At the stage where the image data is recorded on the HDD 20, the reply flag is “0”. In the example illustrated in FIG. 3, the corresponding image is a still image, six pieces of image data are recorded in the HDD 20, and none of the feature data is received from the server 50. Although details will be described later, the metadata management device 18 of the recording / reproducing device 16 receives the metadata from the server 50, stores it in the database 20b of the HDD 20, and updates the reply flag according to the reception status.

  Although the HDD 20 is exemplified as the recording medium, it may be a removable medium such as a volatile semiconductor memory or a recordable optical disk.

  The display control device 22 causes the display device 24 to display the image according to the image data from the recording / reproducing device 16. The display control device 22 generates a menu image signal of a character-based interface, that is, a CUI (Character-based User Interface) or a graphic-based interface, that is, a GUI (Graphical User Interface), and applies the generated menu image signal to the display device 24. . In particular, in this embodiment, the display control device 22 applies metadata information to the display device 24. The display device 24 is, for example, an LCD (liquid crystal display panel).

  The communication device 26 can transmit and receive data to and from the server 50 in accordance with IEEE (Institute of Electrical and Electronic Engineers) 1394, IEEE 802.11, USB (Universal Serial Bus), etc. In this embodiment, HTTP (Hyper An Internet protocol such as Text Transfer Protocol (FTP) or File Transfer Protocol (FTP) is used.

  The system control device 30 controls the entire video camera 10 in accordance with the operation of the operation device 32. The operation device 32 includes a moving image shooting switch, a release switch (SW1, SW2), a playback switch, and a stop switch, and the user operates the system to control moving image shooting, still image shooting, playback, stop, and the like. 30. The operation device 32 includes a menu key, a cross key, a determination key, and an enlargement key, and the user uses these to instruct the system control device 30 to display, select, determine, and enlarge a thumbnail image. To do.

  The memory 34 includes a ROM (Read Only Memory) that stores programs and fixed parameters of the system control device 30, a RAM (Random Access Memory) that stores temporary data and serves as a work memory of the system control device 30, and the like. Yes.

  The configuration of the server 50 will be described with reference to FIG. The communication device 52 communicates with the camera 10 and receives image data (moving image data or still image data) from the camera 10. The recording / reproducing device 54 records and reproduces the image data received by the communication device 52 on a hard disk (HDD) 56. The recording / reproducing device 58 reproduces the image data requested for feature extraction by the camera 10 from the image data recorded in the HDD 56 and supplies the image data to the image analyzing device 58. If there is no need to permanently store the received image data, the HDD 56 is simply used as a temporary storage for image analysis.

  The image analysis device 58 analyzes luminance or color distribution or pattern from the image data reproduced from the HDD 56 and extracts features from the analysis result. In other words, the image analysis device 58 is a feature extraction device. The collation device 60 collates the feature obtained by the image analysis device 58 with the feature database 62, and acquires the name of the image subjected to image analysis and the detailed description text from the feature database 62. The feature database 62 stores features extracted by image analysis of the image analyzer 58, object names, and detailed descriptions for various images or objects.

  The collation device 60 supplies the collation result with the feature database 62 to the metadata generation device 64. The metadata generation device 64 generates metadata in a predetermined format that accommodates the verification result of the verification device 60. The generated metadata is transmitted by the communication device 52 to the camera 10 that is the request source. The metadata management device 18 of the camera 10 stores the metadata transmitted from the server 50 in the HDD 20 and updates the reply flag.

  A basic operation of imaging, recording and reproduction of the camera 10 will be described. In the camera mode, the imaging unit 12 outputs image data corresponding to the optical image of the subject. Output image data from the imaging unit 12 is written in a video memory (VRAM) (not shown). The display control device 22 drives the display device 24 according to the image data stored in the video memory. Thereby, the user can confirm the subject and the composition. In the recording mode, the recording / reproducing device 16 further records the image data from the imaging unit 12 on the HDD 20. In this way, the image data of the image to be shot is stored in the HDD 20.

  In the playback mode, the recording / playback apparatus 16 reads image data of a specified image from the HDD 20. With respect to the read image data, the display control device 22 drives the display device 24 in accordance with the image data read from the HDD 20 by the recording / reproducing device 16. As a result, the reproduced image is displayed on the screen of the display device 24.

  In this embodiment, it is possible to select a list display of thumbnail images of still images recorded in the HDD 20 and a full screen display of one designated image. FIG. 4 shows an example of a screen displaying a list of thumbnails of six images, and FIG. 5 shows an example of displaying one designated image on a screen for shaking. For example, when a desired one is selected in the thumbnail display as shown in FIG. 4 and the enlargement key of the operation device 32 is operated, one designated image is displayed in a full screen as shown in FIG. The

  Next, with reference to FIGS. 6 to 9, a characteristic operation of the present embodiment for a still image, that is, an operation for displaying metadata overlay will be described. 6 shows an operation flowchart, FIG. 7 shows an example of contents of the object information database 20b stored in the HDD 20, FIG. 8 shows an example of thumbnail display, and FIG. 9 shows an example of full screen display.

  The system control device 30 determines whether or not the photographing switch has been pressed (S10). When the photographing switch is pressed (S10), a still image is photographed and the photographed image is recorded on the HDD 20 by the recording / reproducing device 16 (S11). At the same time, as described above, the metadata management device 18 generates a new metadata record for the still image data in the database 20b (S12). At this time, the new record of metadata consists only of an ID, an update flag with a value “0”, and a file path to image data. The captured image data is also transmitted to the server 50 via the communication device 26 (S13).

  The server 50 determines whether image data has been received by the communication device 52 (S20). When the image data is received (S20), the recording / reproducing device 54 records the received image data in the HDD 56 (S21). Image data is read from the HDD 56 in the order of recording and sent to the image analyzer 58. The image analyzer 58 analyzes the luminance or color distribution or pattern of the image data and extracts the features of the image data (S22).

  The extracted features are sent from the image analysis device 58 to the collation device 60, and the collation device 60 collates the features of the image data with the feature database 62 and searches the feature database 62 for records that match the features (S23). The metadata generation device 64 generates metadata that matches the image data from the camera 10 from the search result (S24). The generated metadata is transmitted to the camera 10 via the communication device 52 (S25). If the image analysis (S22) or search (S23) has failed, the failure is transmitted to the camera 10 (S25).

  The communication device 26 of the camera 10 stands by for receiving metadata (S30). When the metadata is received from the server 50 (S30), the server management device 18 stores each piece of information of the received metadata in the corresponding field of the corresponding record in the object information database 20b of the HDD 20 (S31). At this time, the reply flag is updated to a value “1” indicating that it has been received.

  FIG. 7 shows an example of the contents of the object information database 20b updated in this way. In the example illustrated in FIG. 6, metadata is obtained for the image data of ID1, ID2, and ID4, the metadata search has failed for the image data of ID3, and the images of ID5 and ID6 are detected. For data, it is waiting to receive metadata. The contents of the name field (Name) and the detail field (MoreInfo) are supplemented with information from the server 50.

  When metadata is acquired as shown in FIG. 6 and images recorded in the HDD 20 are displayed as a list of thumbnails, the character data in the name field of the metadata are indicated by reference numerals 80, 82, 84, 86 in FIG. As shown in FIG. 6, the thumbnail images are displayed in a superimposed manner. For an image whose reply flag is “−1”, as shown by reference numeral 86, the character “???” indicating unknown is superimposed on the thumbnail image, ie, is overlaid. It is obvious that a specific symbol or figure may be used instead of the character “???”.

  When the user selects a specific image on the thumbnail screen shown in FIG. 8 and presses the enlarge key of the operation device 32, a full screen display as shown in FIG. 9 is displayed. In this full screen display, the name field and the detail field are displayed. As shown by reference numeral 90 in FIG. 9, the character data is superimposed on the reproduced image, that is, displayed as an overlay.

  As described above, in this embodiment, the display mode is changed according to the presence or absence of the search result of the object information by the server 50, so that the user can check on the screen whether the object information has been acquired or the search has failed. .

  This embodiment can be applied to moving images. FIG. 10 shows an operation flowchart of this embodiment for moving images.

  The system control device 30 determines whether or not the moving image shooting switch has been pressed (S110). When the moving image shooting switch is pressed (S110), shooting of a moving image is started, and each shot frame image is sequentially recorded on the HDD 20 by the recording / reproducing device 16 (S111). At the same time, as described above, the metadata management device 18 generates a new metadata record for the moving image data on the database 20b (S112). At this time, the new record of metadata consists only of an ID, a reply flag with a value “0”, and a file path to image data.

  The moving image data recorded in the HDD 20 is transmitted to the server 50 via the communication device 26 while the moving image is being shot or after the moving image is shot (S113).

  The server 50 determines whether image data has been received by the communication device 52 (S120). When the image data is received (S120), the recording / reproducing device 54 records the received image data in the HDD 56 (S121). Image data is read from the HDD 56 in the order of recording and sent to the image analyzer 58. The image analyzer 58 analyzes the luminance or color distribution or pattern of the image data and extracts the features of the image data (S122).

  The extracted features are sent from the image analysis device 58 to the collation device 60. The collation device 60 collates the features of the image data with the feature database 62, and searches the feature database 62 for records that match the features (S123). . The metadata generation device 64 generates metadata that matches the image data from the camera 10 from the search result (S124).

  Steps S121 to S124 are executed for all the frames of the moving image (S125). For frames that fail image analysis or fail to retrieve from the database 62, image analysis of the next frame is immediately executed. When the end of the moving image is reached (S125), the generated metadata is transmitted to the video camera 10 via the communication device 52 (S126). If finally valid metadata cannot be obtained, a failure is transmitted to the camera 10 (S126).

  If the user of the camera 10 designates a frame for requesting metadata in the moving image, the processes of steps S121 to S124 are executed for the designated frame.

  The communication device 26 of the camera 10 stands by for receiving metadata (S130). When the metadata is received from the server 50 (S130), the server management device 18 stores the received metadata in the corresponding record in the object information database 20b of the HDD 20 (S131). At this time, the reply flag is updated to a value “1” indicating that it has been received.

  FIG. 11 shows an example of the contents of the object information database 20b in the case of a moving image. A field (Time) indicating the frame position of the object information is added. The Time field 35:05:00 for ID1 indicates that the record is metadata for a frame of 35 minutes 05 seconds 00 from the start of imaging. A number may be indicated. With respect to this metadata, in the full screen display, as shown in FIG. 12, the data of the Time field is also displayed as an overlay on the screen.

It is a schematic block diagram of one Example of this invention. It is a schematic block diagram of a server according to the present embodiment. It is an example of the structure and content of the database 20b. It is a thumbnail display example. It is an example of a full screen display. It is an operation | movement flowchart of a present Example. It is an example of the content of the database 20b after metadata acquisition. This is an example in which metadata is overlaid and displayed in thumbnail display. This is an example in which metadata is displayed as an overlay in full screen display. It is an operation | movement flowchart of a present Example with respect to a moving image. It is the example of the content of the database 20b with respect to a moving image. It is an example of metadata display in a full screen display.

Explanation of symbols

DESCRIPTION OF SYMBOLS 10 Video camera 12 Imaging unit 16 Recording / reproducing apparatus 18 Metadata management apparatus 20 HDD
20a: Image data 20b Object information database 22 Display control device 24 Display device 26 Communication device 50 Server 52 Communication device 54 Recording / playback device 56 HDD
58 Image analysis device 60 Collation device 62 Feature database 64 Metadata generation device

Claims (1)

An imaging device that communicates with an external server capable of searching for metadata for describing an object included in image data,
Recording means for recording a plurality of photographed image data;
List display control means for displaying a plurality of thumbnail images respectively corresponding to the recorded plurality of captured image data on a display unit;
Selecting means for selecting one thumbnail image from the plurality of thumbnail images displayed in the list;
An enlarged display control means for enlarging the selected thumbnail image and displaying it on the display unit;
Transmitting means for transmitting the recorded captured image data to the external server;
Receiving means for receiving a reply to the transmitted captured image data from the external server;
When the received reply is the metadata corresponding to the transmitted captured image data, the transmitted captured image data, the name of the object included in the received metadata, and the received In the case where the detailed information of the object included in the metadata is associated and recorded in the recording means, and the received reply indicates the metadata search failure, the transmitted captured image is displayed. Recording control means for causing the recording means to record the data in association with predetermined data indicating the search failure,
When the list display control means displays the list, the thumbnail image corresponding to the captured image data recorded in association with the received metadata is displayed without overlaying the detailed information. The name recorded in association with the photographed image data corresponding to is displayed as an overlay, while the thumbnail image corresponding to the photographed image data recorded in association with the predetermined data is specified information indicating the search failure Is displayed as an overlay,
In the case where the selected thumbnail image is a thumbnail image in which the name is displayed as an overlay, the enlarged display control unit is configured to associate the name recorded in association with the captured image data corresponding to the thumbnail image and the associated image. An image pickup apparatus , wherein the recorded detailed information is displayed as an overlay on the enlarged image displayed on the display unit.

Images