JP6029616B2 - Article information providing apparatus, article information providing system, article information providing method, and article information providing program - Google Patents

Description

  The present invention relates to an article information providing apparatus, an article information providing system, an article information providing method, and an article information providing program suitable for providing article information.

  In recent years, so-called AR (Augmented Reality) technology for displaying an image of a virtual object on a display unit as if the virtual object is in real space has been applied in various fields.

  As one of them, for example, an article information providing system that provides article information about articles such as merchandise displayed in a store in real time to a person who wants to check article information such as a customer, as shown in Patent Document 1, for example. It has been known. In this technology, the real space is imaged by the imaging unit, the article information related to the article included in the captured image captured by the imaging unit is acquired by the article information acquisition unit, and the captured image captured by the imaging unit by the image generation unit is acquired. The augmented reality image generated by the image generation unit by the display unit is generated by generating an augmented reality image for displaying the item information acquired by the item information acquisition unit superimposed on the real space based on the position of the included item. Can be displayed.

JP2011-253324A

  In the article information providing system of Patent Document 1 described above, when article information regarding an article included in a captured image captured by the imaging unit is acquired by the article information acquisition unit, a captured image captured by the imaging unit by the image generation unit. Since the article information image indicating the article information acquired by the article information acquisition unit is superimposed on the real space and displayed by the display unit based on the position of the article included in the item, the article about the article displayed in the store Information can be provided to customers in real time.

  By the way, in such an article information providing system, when a customer or the like points or grabs the article, it is determined that the customer or the like wants to know the article information because the captured image includes fingers. An article information image indicating article information is displayed. However, there are cases where a customer or the like pointing to or grasping an article is not necessarily an act of wanting to know article information. In other words, there are many cases where the user simply touches the article while watching the various articles.

  Thus, even when a customer or the like simply touches an article, if an article information image indicating article information is displayed, not only an extra article information image is displayed but also extra article information. An image may be annoying because it enters the field of view.

  For this reason, it has been desired to develop an apparatus capable of displaying an article information image indicating article information that a customer who is a user wants to know at an appropriate timing.

  The present invention has been made in view of such circumstances, and an article information providing apparatus, an article information providing system, and article information capable of displaying an article information image indicating article information that a user wants to know at an appropriate timing. An object is to provide a providing method and an article information providing program.

The article information providing apparatus according to the present invention is an article information providing apparatus that recognizes a subject imaged by the imaging unit, a display unit that displays an article information image indicating article information, an imaging unit that captures a real space, and the like. An object recognizing unit that recognizes an article and a hand from the subject recognized by the subject recognizing unit, and displays the article information image on the display unit according to the determined state of the article, The article determination unit determines whether or not the determined article has been moved by the determined hand, identifies a change in dynamics due to the article being turned over or tilted , and the determined article If any case change of a dynamic state of no fixed time, wherein the display unit to display the article information image, wherein if no discriminating the article recognized object by the object recognition unit, before Characterized by erasing the article information image from the display unit.
An article information providing system of the present invention includes an article information providing apparatus and an information management server, and the article information providing apparatus is provided between a display unit that displays an article information image indicating article information and the information management server. A first information transmission / reception unit that transmits and receives information, an imaging unit that captures an image of a real space, a subject recognition unit that recognizes a subject captured by the imaging unit, and an article from the subject recognized by the subject recognition unit Discriminating the hand and transmitting the item identification information indicating the determined item to the information management server via the first information transmission / reception unit, receiving the item information of the item from the information management server, and determining An article discriminating unit that displays the article information image on the display unit according to the state of the article, and the information management server includes data in which the article information is registered. A second information transmission / reception unit that transmits / receives information to / from the article information providing device, an article identification unit that identifies an article based on the article identification information, and identification of an article by the article identification unit An article information acquisition unit that acquires corresponding article information from the database based on the result and transmits the article information to the article information providing device via the second information transmission / reception unit; Judging whether or not the article has been moved by the identified hand , and identifying a change in behavior due to the article being turned over or tilted , and when there is no change in behavior of the identified article for a certain period of time If any, to display the article information image on the display unit, when not judged the article recognized object by the object recognition unit, to erase the article information image from said display unit It is characterized in.
Further, an article image and an article name for specifying the article are registered in the database, and the article identification unit stores the article image in the database when the article identification information is the article image or the article name. The article is identified by collation with the registered article image or the article name, and the article information acquisition unit acquires the corresponding article information from the database based on the identification result by the article identification unit. Features.
The article identifying unit may recognize the article by reading the barcode when the article identification information is a barcode.
An article information providing method according to the present invention is an article information providing method by an article information providing apparatus, a step of displaying an article information image indicating article information by a display unit, a step of imaging a real space by an imaging unit, and a subject. A step of recognizing a subject imaged by the imaging unit by a recognizing unit, an article and a hand are discriminated from the subject recognized by the subject recognizing unit by an article discriminating unit, and the display unit according to the determined state of the article And displaying the article information image, wherein the article discriminating unit judges whether or not the discriminated article is moved by the discriminated hand, and the article is turned over or tilted. when identifying the dynamic change, and, if any of the cases change of a dynamic state of the discriminated article is not a fixed time, to display the article information image on the display unit, the object If it does not determine the article recognized object by the body recognition unit, characterized by erasing the article information image from the display unit.
The article information providing method of the present invention is an article information providing method by an article information providing system having an information management server and an article information providing apparatus, wherein an article information image indicating the article information is displayed on a display unit of the article information providing apparatus. A step of displaying, a step of transmitting / receiving information to / from the information management server by the first information transmitting / receiving unit of the article information providing device, and a step of imaging a real space by the imaging unit of the article information providing device; A step of recognizing a subject imaged by the imaging unit by a subject recognizing unit of the article information providing device, and a discrimination of an article and a hand from the subject recognized by the subject recognizing unit by an article discriminating unit of the article information providing device Then, article identification information indicating the determined article is transmitted to the information management server via the first information transmission / reception unit, and the information management Receiving the article information of the article from the bar and displaying the article information image on the display unit according to the determined state of the article, and the article information providing device by the second information transmitting / receiving unit of the information management server Sending and receiving information to and from,
A step of identifying an article based on the article identification information by the article identification unit of the information management server, and the article information is registered based on the identification result of the article by the article identification unit by the article information acquisition unit of the information management server. And acquiring the corresponding article information from the stored database, and transmitting the article information to the article information providing apparatus via the second information transmitting / receiving unit, wherein the article discriminating unit determines that the discriminated article is the discriminating item. Whether or not the movement of the article is reversed or tilted , and when the change of the movement of the determined article does not exist for a certain period of time , If the said display unit to display the item information image, not to determine the article recognized object by the object recognition unit, said the article information image display unit Characterized by al erased.
The article information providing program of the present invention recognizes the subject imaged by the imaging unit by the step of displaying an article information image indicating article information on the display unit, the step of imaging the real space by the imaging unit, and the subject recognition unit. Article information, and a step of discriminating an article and a hand from the subject recognized by the subject recognition unit by the article discriminating unit and displaying the article information image on the display unit according to the state of the discriminated article. A computer that controls the providing device executes the apparatus, and the article determination unit determines whether the determined article has been moved by the determined hand, and changes dynamics due to the article being turned over or tilted. If identified, and, if any of the cases change of a dynamic state of the discriminated article is not a fixed time, to display the article information image on the display unit, before If it does not determine the article recognized object by the object recognition unit, characterized by erasing the article information image from the display unit.
The article information providing program of the present invention includes a step of displaying an article information image indicating article information on the display unit, a step of transmitting and receiving information to and from the information management server by the first information transmitting and receiving unit, and an imaging unit. The step of imaging the real space, the step of recognizing the subject imaged by the imaging unit by the subject recognition unit, the item and the hand from the subject recognized by the subject recognition unit by the item discrimination unit, and the discrimination Article identification information indicating an article is transmitted to the information management server via the first information transmission / reception unit, the article information of the article is received from the information management server, and the display unit according to the determined state of the article Causing the computer for controlling the article information providing device to execute the step of displaying the article information image on the second information transmitting / receiving unit. A step of transmitting / receiving information to / from the product information providing device, a step of identifying the product based on the product identification information by the product identification unit, and a product identification unit based on the identification result of the product by the product identification unit A computer for controlling the information management server to acquire the corresponding article information from the database in which the article information is registered and to transmit the article information to the article information providing apparatus via the second information transmitting / receiving unit. The article discriminating unit determines whether the discriminated article has been moved by the discriminated hand , and identifies a change in dynamics due to the article being turned over or tilted ; and If any case change of a dynamic state of the discriminated article is not a fixed time, to display the article information image on the display unit, the photographic recognized by the object recognition unit If it does not determine the article, characterized by erasing the article information image from the display unit.
In the article information providing apparatus, article information providing system, article information providing method, and article information providing program of the present invention, the real space is imaged by the imaging unit, the subject imaged by the imaging unit is recognized by the subject recognition unit, and the article discrimination is performed. When the article and the hand are discriminated from the subject recognized by the subject recognizing unit, the article identification information indicating the discriminated article is transmitted to the information management server via the first information transmitting / receiving unit. Further, when the article information of the article is received from the information management server, the article discriminating unit displays an article information image indicating the article information on the display unit according to the determined state of the article.

According to the article information providing apparatus, the article information providing system, the article information providing method, and the article information providing program of the present invention, when the article and the hand are discriminated from the subject recognized by the subject recognizing unit by the article discriminating unit, it is discriminated. Since the article information image indicating the article information is displayed on the display unit according to the state of the selected article, the article information image indicating the article information that the user wants to know can be displayed at an appropriate timing.

It is a figure which shows one Embodiment of the goods information provision system of this invention. It is a perspective view which shows an example of HMD of FIG. FIG. 1 illustrates a subject imaged by the HMD camera in FIG. 1, and FIG. 1A is a diagram illustrating a state in which an article that is a subject is gripped by the hand of an HMD user. FIG. 4B is a diagram showing a state in which an article held by the user's hand is turned over, and FIG. 4C is a diagram showing a state in which the article held by the user's hand is tilted. It is a flowchart for demonstrating the process by the side of HMD of FIG. 3 is a flowchart for explaining processing on the information management server side of FIG. 1.

  Hereinafter, an embodiment of an article information providing system according to the present invention will be described with reference to FIGS. Note that, as an example of the article information providing device in the following description, for example, a head-mounted display device (HMD: Head Mounted Display) is assumed. In addition, the head-mounted display device described below is used in a store, for example.

  First, as shown in FIG. 1, the article information providing system includes a head-mounted display device (hereinafter referred to as “HDM”) 10 and an information management server 20 having a database 24. The HMD 10 and the information management server 20 are connected via a wireless communication line 30.

  The HMD 10 is a form of an article information providing apparatus that provides article information in real time to the user of the HMD 10 by merging and displaying an article information in real space and an article information image showing article information drawn by CG or the like. is there. Moreover, HMD10 is used in the state mounted | worn with a user's head, for example, and is made into spectacles type as shown, for example in FIG. As a result, the user does not need to hold the HMD 10 with his / her hand, so he / she can shop with both hands as before.

  That is, the HMD 10 includes left and right display units 11 that display an article information image indicating article information, and a vine part 12 that is worn on the ear. A camera 13 is disposed between the left and right display units 11. Since the camera 13 is disposed between the left and right display units 11, the imaging range of the camera 13 follows the movement of the user's head. In addition, an apparatus main body 14 is disposed on one of the vine portions 12.

  The camera 13 is configured to use, for example, a CCD (Charge Coupled Device) sensor or a CMOS (Complementary Metal Oxide Semiconductor) sensor as an image sensor. In addition, the camera 13 can be a video camera that can capture an image of an object with an imaging cycle per image of, for example, 1/30 seconds (that is, a frame rate of 30 fps).

  Thereby, the camera 13 can continuously image the user's hand 50 as the subject and the article 40 held by the user's hand 50. The camera 13 is not limited to a video camera, and may capture a still image. In this case, by setting the still image capturing interval to, for example, 0.5 seconds, it is possible to intermittently capture the user's hand 50 as the subject and the article 40 held by the user's hand 50. it can.

  The display unit 11 employs, for example, an optical see-through method. That is, the display unit 11 is configured to be able to transmit visible light. Thereby, the wearer of the HMD 10 can directly see the sight of the real space that has passed through the display unit 11 in front of the eyes and can also see the augmented reality image displayed on the display unit 11. The augmented reality image in this case is an article information image indicating article information drawn with CG or the like.

  In addition, the display part 11 may be comprised by the half mirror, for example. Further, the display unit 11 includes a projection device (not shown) for projecting an article information image indicating article information generated by an image processing control unit 16 described later on the display unit 11. Moreover, the display part 11 may be comprised with the liquid crystal display. In this case, an article information image generated by an image processing control unit 16 to be described later is displayed on the display unit 11 by a liquid crystal driving circuit (not shown).

  The apparatus main body 14 includes an image recognition unit 15, an image processing control unit 16, and a wireless transmission / reception unit 17. The image recognition unit 15 recognizes the article 40 that is the subject imaged by the camera 13 described above. In this case, when the user holds the article 40, the user's hand 50 is also recognized as a subject. For example, when the article 40 is captured in a state where it is displayed on a shelf, the image recognition unit 15 recognizes the article 40 and the shelf as subjects.

  The recognition of the subject by the image recognition unit 15 is continuously performed while the camera 13 is capturing an image. The subject recognition by the image recognition unit 15 may be performed intermittently, for example, at intervals of 0.5 seconds.

  The image processing control unit 16 controls the recognition operation performed by the image recognition unit 15. Further, the image processing control unit 16 discriminates the article 40 from the subject recognized by the image recognition unit 15, and article identification information indicating the article 40 is connected to the wireless communication line 30 via the wireless transmission / reception unit 17. It transmits to the information management server 20. The article identification information may be an article image of the article 40, or may be an article name, a barcode, a lot number, a manufacturing date, a manufacturer name, or the like attached to the article 40. Good. That is, the article identification information is information that enables identification of the article 40 singly or in combination.

  Further, as described later, the image processing control unit 16 determines whether or not the user wants to know the article information. If the user determines to know the article information, the article indicating the article information received from the information management server 20 is displayed. An information image is generated and displayed on the display unit 11.

  Here, the determination by the image processing control unit 16 as to whether or not the user wants to know the article information is made by determining whether or not the article 40 has been moved by the user's hand 50, for example. That is, for example, as shown in FIG. 3A, it is assumed that the article 40 is held by the user's hand 50. In this case, the image of the article 40 held by the user's hand 50 is captured in a state where the mark 41 of the article 40 is positioned on the near side by the camera 13. In the figure, reference numeral 42 denotes a cap.

  Further, for example, as shown in FIG. 3B, the user may turn over the article 40 gripped by the hand 50 to see the entire article 40, or to see a description on the article 40. In this case, the image of the article 40 held by the user's hand 50 is captured while the mark 41 of the article 40 is moved to the back side by the camera 13.

  In addition, as shown in FIG. 3C, the user may tilt the article 40 held by the hand 50 to see the whole article 40 or to see an explanation on the article 40. In this case, the image of the article 40 held by the user's hand 50 is captured while the camera 42 is moving the cap 42 of the article 40 downward.

  Such a change in the behavior of the article 40 can be determined by the image processing control unit 16 discriminating the article 40 from the subject recognized by the image recognition unit 15 and identifying the change of the article 40. Therefore, the image processing control unit 16 can determine whether the user wants to know the article information by identifying the dynamic change of the article 40 due to the article 40 being turned over or tilted.

  When the image processing control unit 16 determines that the user wants to know the item information, the image processing control unit 16 generates an item information image indicating the item information based on the item information from the information management server 20 and displays it on the display unit 11. Display. Thereby, the article information of the article 40 can be provided to the user at a timing when the user wants to know the article information of the article 40.

  Note that the determination by the image processing control unit 16 as to whether or not the user wants to know the article information is not limited to the presence or absence of a change in the behavior of the article 40. That is, for example, the article 40 in the state shown in FIG. This is because even if the user grasps the article 40 with the hand 50, the user may see the entire article 40 as it is or may see an explanation on the article 40.

  In this case, if a certain time (for example, about 5 seconds) has passed without a change in the dynamics of the article 40, the article 40 is stared, and the user determines that he wants to know the article information. The fixed time required for the determination is not limited to 5 seconds and can be changed as appropriate. Further, the user may stare as it is without grasping the article 40 displayed on the shelf. Also in this case, if a certain time (for example, about 5 seconds) has passed without a change in the dynamics of the article 40, it is determined that the user wants to know the article information.

  The wireless transmission / reception unit 17 transmits the item identification information from the image processing control unit 16 to the information management server 20 or receives the item information from the information management server 20.

  On the other hand, the information management server 20 includes a wireless transmission / reception unit 21, an item identification unit 22, and an item information acquisition unit 23. The wireless transmission / reception unit 21 receives the above-described article identification information from the HMD 10 and transmits article information corresponding to the article identification information.

  The article identification unit 22 identifies the article 40 based on the article identification information from the HMD 10. When the article identification information is an article image of the article 40, the article 40 is identified by collating with the article image registered in the database 24. Further, when the article identification information is an article name given on the article 40, it is performed by collating with the article name registered in the database 24. When the article identification information is a barcode, the article 40 is determined by reading as it is.

  The article information acquisition unit 23 acquires the corresponding article information from the database 24 based on the identification result of the article 40 by the article identification unit 22. In the database 24, article information associated with each article 40, an article image for identifying the article 40, and an article name are registered.

  Next, the processing on the HMD 10 side will be described with reference to FIG. Note that in the processing on the HMD 10 side, imaging by the camera 13 is started when the HMD 10 is mounted on the user's head.

  First, it is assumed that the subject is imaged by the camera 13 of the HMD 10 from the state of waiting for imaging of the subject by the camera 13 of the HMD 10 (step S1; NO) (step S1: YES). Here, for example, as shown in FIG. 3A, it is assumed that the article 40 is grasped by the user's hand 50. At this time, the camera 13 of the HMD 10 images the article 40 as the subject and the user's hand 50.

  When the subject is imaged, the image recognition unit 15 recognizes the article 40 and the user's hand 50 as the subject (step S2). As described above, it is assumed that the camera 13 is a video camera and the imaging cycle per image is, for example, 1/30 seconds (that is, the frame rate is 30 fps). In this case, the image recognizing unit 15 recognizes the article 40 and the user's hand 50 as the subject from arbitrary frame images among 30 frame images obtained in one second, for example.

  Next, the image processing control unit 16 determines the article 40 recognized by the image recognition unit 15 (step S3), and the article identification information indicating the determined article 40 is transmitted to the wireless communication line 30 via the wireless transmission / reception unit 17. It transmits to the connected information management server 20 (step S4).

  Note that the article identification information indicating the article 40 may be an article image of the article 40 as described above, or an article name or barcode attached to the article 40. Further, when the item identification information is transmitted from the HMD 10 to the information management server 20, the corresponding item information is immediately returned from the information management server 20. The processing will be described later. Accordingly, here, it is assumed that the corresponding article information is returned from the information management server 20.

  In addition, since the imaging of the subject by the camera 13 is continuously performed, the image processing control unit 16 determines the presence or absence of the dynamic change of the article 40 in accordance with the recognition of the subject by the image recognition unit 15 (step S5).

  Here, for example, as shown in FIG. 3B, it is assumed that the user turns over the article 40 grasped by the hand 50 and sees the entire article 40 or sees an explanatory note on the article 40. In this case, as described above, the article 40 that is being held by the user's hand 50 is imaged while the mark 41 of the article 40 is moved to the back side by the camera 13.

  Further, as shown in FIG. 3C, it is assumed that the user tilts the article 40 grasped by the hand 50 and sees the entire article 40 or sees an explanation on the article 40 or the like. In this case, as described above, the image of the article 40 held by the user's hand 50 is captured while the cap 42 of the article 40 is moved downward by the camera 13.

  Such a change in the behavior of the article 40 can be determined by the image processing control unit 16 discriminating the article 40 from the subject recognized by the image recognition unit 15 and identifying the change of the article 40. Then, if the image processing control unit 16 determines that the article 40 has a dynamic change (step S5: YES), the image processing control unit 16 determines that the user wants to know the article information and indicates the article information received from the information management server 20. An image is generated and displayed on the display unit 11 (step S6). Thus, the article information is provided at an appropriate timing when the user wants to know the article information.

  Next, it is determined whether the image processing control unit 16 can determine the article 40 that is the subject recognized by the image recognition unit 15 (step S7). That is, the image processing control unit 16 can discriminate the article 40 from the recognition result by the image recognition unit 15 while the article 40 held by the user's hand 50 is captured by the camera 13. On the other hand, when the article 40 is returned to the original shelf, the article 40 is out of the imaging range of the camera 13, so that the article 40 cannot be determined from the recognition result by the image recognition unit 15.

  Therefore, when the image processing control unit 16 can determine the item 40 (step S7: YES), the display of the item information image indicating the item information on the display unit 11 is continued. On the other hand, when the image processing control unit 16 cannot determine the article 40 (step S7: NO), the display of the article information image indicating the article information is deleted (step S8). This prevents an extra article information image from entering the field of view.

  On the other hand, in step S5, when there is no dynamic change in the article 40 (step S5: NO), the image processing control unit 16 determines whether or not a certain time (for example, about 5 seconds) has passed (step S9). That is, for example, the article 40 may be continuously imaged by the camera 13 in the state shown in FIG. This is because, as described above, the user may look at the entire article 40 without moving the article 40 held by the hand 50, or may see an explanation on the article 40. .

  In this case, when the image processing control unit 16 determines that a certain amount of time (for example, about 5 seconds) has passed without a change in the state of the article 40 (step S9: YES), the image processing control unit 16 determines that the user wants to know the article information, and the information An article information image indicating article information received from the management server 20 is generated and displayed on the display unit 11 (step S6). As described above, the user may stare at the article 40 displayed on the shelf without holding it. Also in this case, the image processing control unit 16 determines that the user wants to know the article information if a certain time (for example, about 5 seconds) has elapsed. Thus, as described above, the article information is provided at an appropriate timing when the user wants to know the article information.

  On the other hand, if the image processing control unit 16 determines that a certain time (for example, about 5 seconds) has not passed since the state of the article 40 has not changed (step S9: NO), the subject recognized by the image recognition unit 15 It is determined whether or not the article 40 can be determined (step S10). When the article 40 can be identified (step S10: YES), the image processing control unit 16 determines whether the article 40 has a dynamic change (step S5).

  On the other hand, when the image processing control unit 16 cannot discriminate the article 40 (step S10: NO), the display of the article information image indicating the article information by the display unit 11 is not performed. That is, as described above, the image processing control unit 16 can discriminate the article 40 from the recognition result by the image recognition unit 15 while the article 40 held by the user's hand 50 is captured by the camera 13. .

  On the other hand, although the user once grabs the article 40, the article 40 may be immediately returned to the original shelf. Further, as described above, the user may stare at the article 40 displayed on the shelf without grasping the article 40, but may be removed from the article 40 staring at the line of sight within a short time. Further, although the user once grabs the article 40, it may be put in the basket as it is.

  In this case, as described above, when the article 40 is out of the imaging range of the camera 13, the image processing control unit 16 cannot discriminate the article 40 from the recognition result by the image recognition unit 15. Therefore, the image processing control unit 16 does not determine whether the user wants to know the item information when the item 40 cannot be determined in a short time. Thereby, when the user does not want to know the article information, the article information image indicating the article information is not displayed on the display unit 11, so that it is possible to avoid the display of an extra article information image.

  Next, processing on the information management server 20 side will be described with reference to FIG. First, it waits for reception of the article identification information from the HMD 10 side (step S20: NO). When the wireless transmission / reception unit 21 receives the item identification information from the HMD 10 side (step S20: YES), the item identification unit 22 identifies the item 40 based on the item identification information (step S21).

  Here, when the article 40 is identified, as described above, when the article identification information is an article image of the article 40, the identification is performed with the article image registered in the database 24. Further, when the article identification information is an article name given on the article 40, it is performed by collating with the article name registered in the database 24. When the article identification information is a barcode, the article 40 is determined by reading as it is.

  Next, the article information acquisition unit 23 acquires the corresponding article information from the database 24 based on the identification result by the article identification unit 22 (step S22). And if article information is acquired by the article information acquisition part 23, the radio | wireless transmission / reception part 21 will transmit the acquired article information to the HMD10 side (step S23).

  As described above, in the present embodiment, the real space is imaged by the camera 13 that is a form of the imaging unit of the HMD 10 that is a form of the article information providing apparatus, and the camera is captured by the image recognition unit 15 that is a form of the subject recognition part. When the subject imaged by the image recognition unit 13 is recognized and an article is identified from the subject recognized by the image recognition unit 15 by the image processing control unit 16 which is a form of the item discrimination unit, the item identification information indicating the identified item Is transmitted to the information management server 20 via the wireless transmission / reception unit 17 which is a form of the first information transmission / reception unit. Further, upon receiving article information of an article from the wireless transmission / reception unit 21 which is a form of the first information transmission / reception unit of the information management server 20, the image processing control unit 16 displays the article 11 according to the determined state of the article. An article information image indicating article information is displayed on the screen.

  On the information management server 20 side, when an article is identified by the article identification unit 22 based on the article identification information, the article information is registered by the article information acquisition unit 23 based on the identification result of the article by the article identification unit 22. Corresponding article information is acquired from the database 24, and is transmitted to the HMD 10 via the wireless transmission / reception unit 21 which is one form of the second information transmission / reception unit.

  As described above, the article information image indicating the article information is displayed on the display unit 11 according to the state of the article determined by the image processing control unit 16 of the HMD 10 in cooperation with the information management server 20. It is possible to display an article information image indicating article information to be known at an appropriate timing.

  The display of the article information image indicating the article information on the display unit 11 by the image processing control unit 16 is when the determined change in the behavior of the article is identified, or when the determined change in the behavior of the article does not exist for a certain period of time. This is a condition for determining whether or not the user wants to know the article information. When these conditions are satisfied, the article information image indicating the article information is displayed on the display unit 11 so that extra article information is displayed. The image will not be displayed. Further, since the extra article information image does not enter the user's field of view, the user does not have to worry about it.

In the present embodiment, the HMD 10 has been described as being used in a store, for example. However, the present invention is not limited to this example, and public facilities such as hospitals, libraries, and civic centers, various leisure facilities, roads, railroads, and the like. It can also be used in transportation facilities such as harbors and airports.
Moreover, although this embodiment demonstrated taking the goods currently displayed on the shelf of a store as an example, this invention can be implemented similarly also to the goods placed in other places.

10 HMD
DESCRIPTION OF SYMBOLS 11 Display part 12 Vine part 13 Camera 14 Apparatus main body 15 Image recognition part 16 Image processing control part 17 Wireless transmission / reception part 20 Information management server 21 Wireless transmission / reception part 22 Article identification part 23 Article information acquisition part 24 Database 30 Wireless communication line 40 Article 41 Mark 42 Cap 50 Hand

Claims (8)

An article information providing device,
A display unit for displaying an article information image indicating article information;
An imaging unit for imaging a real space;
A subject recognition unit for recognizing a subject imaged by the imaging unit;
An article discriminating unit that discriminates an article and a hand from the subject recognized by the subject recognition unit, and displays the article information image on the display unit according to the state of the discriminated article;
The article discrimination unit
The discriminated article, it is judged whether driven by hand to the determination, when identifying the dynamic changes caused by the article is tilted or not inverted, and, dynamic change of the discriminated article certain time In any of the cases where there is no, the article information image is displayed on the display unit ,
The article information providing apparatus , wherein the article information image is deleted from the display unit when the article is not determined as the subject recognized by the subject recognition unit . An article information providing device;
With an information management server,
The article information providing device includes:
A display unit for displaying an article information image indicating article information;
A first information transmitting / receiving unit for transmitting / receiving information to / from the information management server;
An imaging unit for imaging a real space;
A subject recognition unit for recognizing a subject imaged by the imaging unit;
An article and a hand are discriminated from the subject recognized by the subject recognition unit, and article identification information indicating the discriminated article is transmitted to the information management server via the first information transmission / reception unit. An article determination unit that receives article information of the article and causes the display unit to display the article information image according to the determined state of the article;
The information management server
A database in which the article information is registered;
A second information transmission / reception unit for transmitting / receiving information to / from the article information providing device;
An article identification unit for identifying an article based on the article identification information;
An article information acquisition unit that acquires the corresponding article information from the database based on the identification result of the article by the article identification unit, and transmits the article information to the article information providing device via the second information transmission / reception unit;
The article discrimination unit
The discriminated article, it is judged whether driven by hand to the determination, when identifying the dynamic changes caused by the article is tilted or not inverted, and, dynamic change of the discriminated article certain time In any of the cases where there is no , display the article information image on the display unit ,
The article information providing system , wherein the article information image is deleted from the display unit when the article is not determined as the subject recognized by the subject recognition unit . In the database, an article image and an article name for specifying the article are registered,
The article identification unit, when the article identification information is the article image or the article name, identifies the article by collating with the article image or the article name registered in the database;
The article information providing system according to claim 2 , wherein the article information acquiring unit acquires corresponding article information from the database based on an identification result by the article identifying unit. 4. The article information providing system according to claim 2 , wherein when the article identification information is a barcode, the article identification unit recognizes the article by reading the barcode. 5. An article information providing method by an article information providing apparatus,
A step of displaying an article information image indicating article information on the display unit;
Imaging the real space by the imaging unit;
A step of recognizing a subject imaged by the imaging unit by a subject recognition unit;
Determining an article and a hand from the subject recognized by the subject recognition unit by an article discrimination unit, and displaying the article information image on the display unit according to the determined state of the article,
The article discrimination unit
The discriminated article, it is judged whether driven by hand to the determination, when identifying the dynamic changes caused by the article is tilted or not inverted, and, dynamic change of the discriminated article certain time In any of the cases where there is no , display the article information image on the display unit ,
An article information providing method comprising: erasing the article information image from the display unit when the article is not determined as a subject recognized by the subject recognition unit . An article information providing method by an article information providing system having an information management server and an article information providing apparatus,
Displaying an article information image indicating the article information on a display unit of the article information providing apparatus;
Transmitting and receiving information to and from the information management server by the first information transmitting and receiving unit of the article information providing device;
Imaging the real space by the imaging unit of the article information providing device;
Recognizing a subject imaged by the imaging unit by a subject recognition unit of the article information providing device;
The article discriminating unit of the article information providing apparatus discriminates an article and a hand from the subject recognized by the subject recognition unit, and the item management information indicating the discriminated article is transmitted to the information management unit via the first information transmitting / receiving unit. Transmitting to the server, receiving the article information of the article from the information management server, and displaying the article information image on the display unit according to the determined state of the article;
A step of transmitting / receiving information to / from the article information providing apparatus by a second information transmitting / receiving unit of the information management server;
Identifying an article based on the article identification information by the article identification unit of the information management server;
The article information acquisition unit of the information management server acquires the corresponding article information from the database in which the article information is registered based on the identification result of the article by the article identification unit, and passes through the second information transmission / reception unit. Transmitting to the article information providing device,
The article discrimination unit
The discriminated article, it is judged whether driven by hand to the determination, when identifying the dynamic changes caused by the article is tilted or not inverted, and, dynamic change of the discriminated article certain time In any of the cases where there is no, the article information image is displayed on the display unit ,
An article information providing method comprising: erasing the article information image from the display unit when the article is not determined as a subject recognized by the subject recognition unit . A step of displaying an article information image indicating article information on the display unit;
Imaging the real space by the imaging unit;
A step of recognizing a subject imaged by the imaging unit by a subject recognition unit;
Controlling the article information providing apparatus with a step of discriminating an article and a hand from the subject recognized by the subject recognition unit by the article discriminating unit, and displaying the article information image on the display unit according to the determined state of the article To run on a computer
The article discrimination unit
The discriminated article, it is judged whether driven by hand to the determination, when identifying the dynamic changes caused by the article is tilted or not inverted, and, dynamic change of the discriminated article certain time In any of the cases where there is no, the article information image is displayed on the display unit ,
An article information providing program for deleting the article information image from the display unit when the article is not discriminated from the subject recognized by the subject recognition unit . A step of displaying an article information image indicating article information on the display unit;
Transmitting and receiving information to and from the information management server by the first information transmitting and receiving unit;
Imaging the real space by the imaging unit;
A step of recognizing a subject imaged by the imaging unit by a subject recognition unit;
An article discriminating unit discriminates an article and a hand from the subject recognized by the subject recognition unit, transmits article identification information indicating the discriminated article to the information management server via the first information transmitting / receiving unit, and Receiving the article information of the article from the information management server, and causing the computer for controlling the article information providing apparatus to execute a step of displaying the article information image on the display unit according to the determined state of the article,
A step of transmitting / receiving information to / from the article information providing apparatus by a second information transmitting / receiving unit;
A step of identifying an article based on the article identification information by an article identification unit;
Based on the identification result of the article by the article identification unit by the article information acquisition unit, the corresponding article information is acquired from the database in which the article information is registered, and the article information providing device is provided via the second information transmitting / receiving unit. To the computer for controlling the information management server,
The article discrimination unit
The discriminated article, it is judged whether driven by hand to the determination, when identifying the dynamic changes caused by the article is tilted or not inverted, and, dynamic change of the discriminated article certain time In any of the cases where there is no, the article information image is displayed on the display unit ,
An article information providing program for deleting the article information image from the display unit when the article is not discriminated from the subject recognized by the subject recognition unit .

Images