JP5842634B2 - Display method and display device - Google Patents

Description

  The present invention relates to a display method and a display device.

  A display device that displays an augmented reality (AR) image in which a real-world image and a content image are superimposed has been proposed. A portable terminal such as a smart phone, a tablet, or a personal digital assistant (PDA) is an example of such a display device. The portable terminal captures a real-world image with its own camera function, and displays an AR image in which an image of content such as computer graphics (CG) is superimposed on the captured image.

  However, since the real-world image captured by the camera function of the mobile terminal has been reduced in scale and smaller than the actual image, it is difficult to give the AR image a high sense of presence. Further, if there is no consistency between the illumination state of the real-world image taken by the camera function and the shadow processing of the content, the displayed AR image becomes unnatural. However, in order to know the illumination state of the real-world image captured by the camera function, it is necessary to estimate the position of the light source based on the position of the mobile terminal at the time of capture, so it is consistent with the real-world illumination state. It is difficult to perform shading processing of content that has been removed.

  On the other hand, a display device having a transmissive display has been proposed. Such a display device is also called an information presentation device or a digital signage. The user can view the real-world video through the transmissive display, and by displaying the content image such as CG on the transmissive display, the user can view the AR image in which the content image is superimposed on the real-world background video. Can see. In the case of a display device having a transmissive display, a user can view an image of the real world through the transmissive display at a scale of an actual image, and thus an AR interface capable of displaying an AR image with a high sense of reality is provided. Can do.

  However, if an image of content that does not match the lighting condition of the installation environment of the display device is displayed on the transmissive display, consistency between the illumination state of the real-world image seen through the transmissive display and the shading process of the content cannot be achieved. . In such a case, since the shading process of the content is unnatural with respect to the real-world image, the AR image seen through the transmissive display becomes unnatural, and the user feels uncomfortable with the AR image.

JP 2008-33531 A JP-A-9-237355

  In a display device having a conventional transmissive display, it is difficult to always appropriately perform shadow processing of content on a background image in the real world. For this reason, it is difficult to present an AR image that is consistent between the illumination state of the background in the real world and the shading process of the content.

  SUMMARY An advantage of some aspects of the invention is that it provides a display method and a display device capable of presenting an AR image that is consistent between the illumination state of a real-world image and the shading process of content.

According to one aspect of the present invention, an illumination condition in an environment where a transmission screen having a first surface and a second surface opposite to the first surface is installed, a user position, and the transmission A shading process is performed on the image content in accordance with a relative position with respect to the mold screen, and the image content subjected to the shading process on the real-world video viewed through the first surface of the transmissive screen from the user position is transmitted. displayed superimposed by projecting the first surface or the second surface of the mold screens, the lighting conditions, the light source direction der Ru display method from the perspective of the transmission screen is provided.

According to an aspect of the present invention, a transmissive screen having a first surface and a second surface opposite to the first surface , a plurality of photosensors, and detection outputs of the plurality of photosensors. an estimation unit for estimating a light source direction as seen from the transmission screen on the basis of a detection device for detecting the relative position of the transmission screen and the user position, the image content in response to the light source direction and the relative position A shadow processing unit that performs a shadow process; and an image content obtained by performing the shadow process on a real-world image that is visible from the user position through the first surface of the transmissive screen, and the first surface of the transmissive screen. Alternatively, a display device is provided that includes a projector that projects and projects the second surface .

  According to the disclosed display method and display apparatus, it is possible to present an AR image that is consistent between the illumination state of the real-world image and the shadow processing of the content.

It is a perspective view which shows an example of the display apparatus in one Example. It is a perspective view which shows the display apparatus at the time of learning. It is a flowchart explaining a learning process. It is a figure explaining the learning process using a neural network. It is a figure which shows an example of a neural network. It is a figure explaining an example of an interpolation process. It is a block diagram which shows an example of a structure of the display apparatus in one Example. It is a figure explaining the pointing by a user. It is a flowchart explaining an example of a pointing position detection process. It is a figure which shows an example of the database which matches a coordinate and a content, and a content database. It is a flowchart explaining an example of a content selection process. It is a front view explaining pointing by a user. It is a side view explaining the pointing by a user. It is a figure explaining an example of the information display regarding the pointed goods. It is a flowchart explaining an example of a history recording process. It is a figure which shows an example of a history database.

  In the disclosed display method and display device, by performing shading processing of content displayed on the transmissive display according to the illumination conditions of the environment where the transmissive display is installed and the relative position between the user and the transmissive display, Consistency between the lighting conditions of the real world background seen through the transmissive display and the shading of the content.

  Embodiments of the disclosed display method and display device will be described below with reference to the drawings.

  FIG. 1 is a perspective view illustrating an example of a display device according to an embodiment. A display device 1 shown in FIG. 1 includes a base 11, a frame 12 provided on the upper surface of the base 11, a plurality of optical sensors 13 arranged on the frame 12 at regular intervals, a projector 14 provided on the upper surface of the base 11, A wheel 15 provided on the lower surface of the base 11 and a transmission screen 16 indicated by the frame 12 are provided. The transmissive screen 16 is an example of a transmissive display, and a front surface 16-1 that can be seen by a user (not shown) when facing the arrow direction, and a back surface 16-2 on which an image is projected from the lens 14-1 of the projector 14. Have When the user turns in the direction of the arrow, a real-world image behind the back surface 16-2 can be seen through the transmissive screen 16. Further, when a content image such as CG is projected from the projector 14 onto the back surface 16-2, an augmented reality (AR) image in which a real-world video and a content image are superimposed is displayed to the user. Looks like you are.

  The projector 14 may project a content image on the surface 16-1 of the transmissive screen 16. In this example, since the display device 1 is a movable type (or a movable type), wheels 15 are provided, and the wheels 15 are driven by a drive unit (not shown) such as a motor as needed. The wheel 15 can be omitted.

  Furthermore, the shape of the transmissive screen 16 is not limited to a vertically long rectangle, and can be formed in any shape including a horizontally long rectangle, a circle, an ellipse, and the like. The frame 12 may have a shape that is the same as the shape of the transmissive screen 16.

  Machine learning such as a neural network is used for light source direction estimation for appropriately shading the AR image projected onto the transparent screen in accordance with the background illumination state. In the following, the movement refers to machine learning unless otherwise specified.

  FIG. 2 is a perspective view showing the display device during learning. In FIG. 2, the same parts as those of FIG. The plurality of optical sensors 131 shown in FIG. 2 are sensors used only at the time of learning, and are removed at times other than learning. Both the optical sensor 12 and the optical sensor 131 are regularly arranged, and in this example, are arranged in a matrix.

  The optical sensor 131 only needs to be removable at times other than during learning, and is an adapter that is installed on the surface 16-1 of the transmissive screen 16 only during learning, even if it is directly detachably bonded to the transmissive screen 16. (Not shown) may be provided. In the case of a film-like adapter, the optical sensor 131 may be printed on a film. In this case, the film may be configured to be a roller blind, for example, so that it can be stored in a position in the display device 1 that does not interfere with the detection operation of the optical sensor 12 except during learning.

  FIG. 3 is a flowchart for explaining the learning process. When the learning process of FIG. 3 is started, step S1 moves to a place (hereinafter also referred to as an installation position) where the display device 1 in which the optical sensor 131 is disposed is used as shown in FIG. If the display unit 1 is provided with a drive unit and can autonomously travel like an autonomously traveling robot, the display unit 1 can be automatically moved to the installation position by controlling the drive unit. Step S2 performs a learning process of learning the light source direction information using the display device 1 at the installation position. Step S3 determines whether or not the learning process has ended. If the determination result is NO, the process returns to step S2, and if the determination result is YES, the process ends.

  FIG. 4 is a diagram illustrating learning processing using a neural network. In FIG. 4, an input 31 of a neural network (NN) 32 is an illuminance distribution detected by the optical sensor 13 provided only in the frame 12. An output 33 of the NN 32 is an illuminance distribution detected by all the optical sensors 13 and 131 including the optical sensor 131 used only at the time of learning. The learning process is performed by comparing the output 33, which is the illuminance distribution detected by all the optical sensors 13, 131, with the teacher signal and changing the coupling strength of the NN 32 according to the comparison result.

FIG. 5 is a diagram illustrating an example of a neural network. The NN 32 illustrated in FIG. 5 is a hierarchical type having an input layer 321 to which an input 31 formed by X _{0 to} X _i is input, an intermediate layer 322, and an output layer 323 to which an output 33 is output. By learning the connection weight of the NN 32 based on the back propagation algorithm, the connection strength between neurons indicated by a circle in FIG. 5 is changed.

Although the optical sensors 31 and 131 are arranged in a matrix, the illuminance at a portion between adjacent optical sensors is interpolated by interpolation processing based on the illuminance detected by the adjacent optical sensors. FIG. 6 is a diagram illustrating an example of interpolation processing. In FIG. 6, the vertical axis indicates the y coordinate value of the xy coordinate system, and the horizontal axis indicates the x coordinate value of the xy coordinate system. FIG. 6 shows the illuminance S () detected at the positions (x _i , y _{j + 1} ), (x _{i + 1} , y _{j + 1} ), (x _i , y _j ), (x _{i + 1} , y _j ) of four adjacent optical sensors. x _i , y _{j + 1} ), S (x _{i + 1} , y _{j + 1} ), S (x _i , y _j ), S (x _{i + 1} , y _j ) and V _i from the position (x _i , y _j ) in the y direction, position away _{H i} to the x direction (x, y) indicating the illuminance P (x, y) in the. When the interpolation process is, for example, linear interpolation, the illuminance interpolation in the x direction at the position (x, y) can be performed based on H (x) = S (x _i , y _j ) + a × H _i , a = {S (x _{i + 1} , y _j ) −S (x _i , y _j )} / (x _{i + 1} −x _i ). Further, the illuminance interpolation in the y direction at the position (x, y) can be performed based on V (y) = S (x _i , y _j ) + b × V _i , and b = {S (x _i , y _{j + 1} ) −S (x _i , y _j )} / (y _{j + 1} −y _j ). Accordingly, the illuminance P (x, y) at the position (x, y) can be obtained from H (x) + V (y). The interpolation process is not limited to linear interpolation.

  FIG. 7 is a block diagram illustrating an example of a configuration of a display device according to an embodiment. The display device 1 has a configuration capable of communicating with an external device. The server 500 is an example of an external device. In this example, the server 500 can operate as a host device of the display device 1. The display device 1 and the server 500 may be connected via a cable or a wired network, or may be connected via a wireless network.

  The display device 1 includes optical sensors 13-1 to 13 -N (N is a natural number of 2 or more), a light source direction estimation and learning unit 51, a shadow processing unit 52, a projector 14, and an imaging device 53 connected as illustrated in FIG. 7. , Positioning device 54, human detection device 55, touch panel 56, microphone 57, communication unit 58, image processing unit 59, control unit 61, movement control unit 62, drive unit 63, content selector 64, storage unit 65, and sound reproduction unit 66. The optical sensors 131-1 to 131-M (M is a natural number of 2 or more) are connected to the light source direction estimation and learning unit 51 only during learning. The optical sensors 13-1 to 13-N correspond to the optical sensor 13 shown in FIG. 2, and the optical sensors 131-1 to 131-M correspond to the optical sensor 131 shown in FIG.

  The optical sensors 13-1 to 13-N and 131-1 to 131-M can be formed of photodiodes that can detect the intensity (or light amount) of light. The light source direction estimation and learning unit 51 can be formed by, for example, a neural network (NN), and includes a light source direction estimation NN and a learning NN. The light source direction estimation NN is based on light amount information (that is, illuminance distribution) detected by the optical sensors 13-1 to 13-N or the optical sensors 13-1 to 13-N and 131-1 to 131-M. The direction of the light source with respect to the surface 16-1 of the transmissive screen 16 viewed from the user side (ie, the light source direction) is estimated, and the light source direction information indicating the estimated light source direction is supplied to the shadow processing unit 52. When estimating the light source direction, the interpolation processing as described above may be performed. On the other hand, the learning NN can be formed by, for example, the NN 32 illustrated in FIG. 5 and supplies the output 33 to the control unit 61.

  The positioning device 54 measures the position of the display device 1 by a known method and supplies device position information indicating the position of the display device 1 to the control unit 61. The positioning device 54 can be formed by, for example, GPS (Global Positioning System). The human detection device 55 detects a user on the front side of the surface 16-1 of the transmission screen 16 of the display device 1 by a known method and sees it from the display device 1 (that is, the position of the display device 1 is a reference). The human detection information indicating the relative position of the user is supplied to the control unit 61 and the shadow processing unit 52. For example, the human detection device 55 may include one or more cameras. The control unit 61 estimates the position of the user with respect to the surface 16-1 of the transmissive screen 16 based on the apparatus position information and the person detection information by a known method, and obtains estimated position information indicating the estimated position of the user by the imaging apparatus. Feedback to 53. The controller 61 is supplied with an output 33 (ie, estimated illuminance information) indicating the estimated illuminance distribution from the light source direction estimating and learning unit 51, so that the display device is based on the estimated illuminance information and the human detection information. By outputting a movement command that finely adjusts the direction of 1, the transmissive screen 16 may be in a direction that is easy for the user to see.

  The imaging device 53 can be formed by, for example, a swing camera, and images an image behind the back surface 16-2 of the screen 16 based on the estimated position information. Since the estimated position information indicates the estimated position of the user, the imaging device 53 captures an image that can be seen through the transmission screen 16 from the estimated position of the user. Image information indicating an image photographed by the photographing device 53 is supplied to the image processing unit 59. The image processing unit 59 supplies the image feature information obtained by extracting the features of the image information by a known method to the control unit 61.

  The touch panel 56 is provided integrally with the transmissive screen 16, for example. When the user operates the touch panel 56, operation information is supplied to the control unit 61. When the user touches an arbitrary position on the touch panel 56, that is, an arbitrary position on the surface 16-1 of the transmissive screen 16, operation information indicating the touched position is supplied to the control unit 61. Since the image feature information extracted from the image information indicating the image viewed by the user through the transmissive screen 16 is supplied to the control unit 61, operation information indicating the position on the transmissive screen 16 touched by the user. Is supplied to the control unit 61, the control unit 61 can recognize the point where the user is pointing in the image that the user is looking through the transmissive screen 16. Since the microphone 57 supplies sound information indicating the detected sound to the control unit 61, the control unit 61 can recognize a question or the like from the user as necessary. If a plurality of microphones 57 are provided, the control unit 61 can recognize the position of the user as the sound source from the sound information, and therefore the transmissive screen 16 of the display device 1 according to the recognized position of the user. It is also possible to output a movement command for moving to a position where it is easy to see.

  The communication unit 58 includes a transmitter and a receiver for communicating with the server 500, and receives an operation command, content information, and the like from the server 500. The operation command includes a movement command for moving the display device 1 to the installation position, a selection command for selecting content information to be output, and the like. The content information includes audio content and image content. The audio content includes various types of audio information such as audio guidance, music, sound effects, and any combination thereof. The image content includes various pieces of image information such as a moving image including CG, a still image, text, and any combination thereof. The operation command received by the communication unit 58 is supplied to the control unit 61, and the content information received by the communication unit 58 is stored in the storage unit 65. The content information is in a reference coordinate system of each part of a user's visual object (that is, an object that the user sees through the transmissive screen 16) such as a product shelf (not shown) at a fixed known position behind the display device 1. Managed according to the coordinates.

  Note that the storage unit 65 may be provided outside the display device 1 instead of inside the display device 1. When the storage unit 65 is provided outside the display device 1, the content selector 64 reads content information to be selectively output via the communication unit 58 from the external storage unit 65.

  When the operation command supplied from the communication unit 58 is a movement command, the control unit 61 issues a movement command for moving the display device 1 to the installation position and supplies the movement command to the movement control unit 62. The movement control unit 62 outputs a control command for driving the driving unit 63 to move the display device 1 to the installation position in response to the movement command. The drive part 63 can be formed with a motor, for example, and drives the wheel 15 shown in FIG. 1 in response to a control command.

  If the operation command from the communication unit 58 is a selection command for selecting content information, the control unit 61 supplies the content selector 64 with a content selection command for selecting content in accordance with the selection command. The content selector 64 reads out the content selected by the content selection command from the storage unit 65 and selectively outputs it. The audio content selected by the content selection command is supplied to the audio reproduction unit 66, and the audio content is reproduced. The audio reproduction unit 66 is formed by a speaker, for example. The image content selected by the content selection command is supplied to the shadow processing unit 52. The shadow processing unit 52 performs a shadow process on the image content based on the light source direction information and the human detection information. In other words, the shadow processing unit 52 performs a shadow process on the image content based on the light source direction and the relative position of the user so as to match the lighting condition of the space viewed from the user's viewpoint. The shadow processing unit 52 supplies the image content subjected to the shadow processing to the projector 14. The projector 14 projects the image content subjected to the shading process onto the back surface 16-2 of the transmissive screen 16.

  For example, the content information relates to a product placed on a product shelf behind the display device 1. Since the control unit 61 can recognize the point where the user is pointing in the image the user is looking through the transmissive screen 16 as described above, if the product shelf is installed at a fixed position, the control unit 61 Can recognize the product on the product shelf that the user is pointing to. For example, if the position of each product placed on the product shelf is managed in the reference coordinate system, content information such as a description of each product is stored in the storage unit 65 according to the coordinate value of the reference coordinate system. Thus, it is possible to output audio content such as a description about the commodity at the pointed portion from the audio reproduction unit 66, or to project the image content onto the transmissive screen 16 by the projector 14. In addition, since the audio information indicating the audio detected by the microphone 57 is supplied to the control unit 61, the control unit 61 outputs content information such as an explanation regarding the product at the pointed position when recognizing a question from the user. You may do it. In this way, the display device 1 can determine content information to be output according to user interaction, display image content, and output audio information as necessary.

  Note that the location pointed by the user can be recognized even when the user does not touch the transmissive screen 16, as will be described below with reference to FIGS.

FIG. 8 is a diagram for explaining pointing by the user. 8A is a front view of the transmission screen 16, and FIG. 8B is a side view of the display device 1. FIG. In this example, the human detection device 55 is formed of a stereo camera that can measure the position and orientation of the user 501 viewed from the reference coordinate system. In addition, the user 501 points the product in the product shelf (not shown) behind the display device 1 with the hand 501A along the arrow direction. An intersection 503 between a target space surface 601 (hereinafter referred to as a target space surface 601) formed on the front surface of the product shelf and a pointing direction 502 indicated by a broken line corresponds to the position of the product that the user 501 is pointing to. The pointing direction 502 is a straight line indicating the direction of the hand 501A viewed from the reference coordinate system. It is assumed that the position of the target space plane 601 viewed from the reference coordinate system is known.

  FIG. 9 is a flowchart for explaining an example of the pointing position detection process. The pointing position detection process is executed by a processor such as a CPU (Central Processing Unit) that forms the control unit 61. The program executed by the CPU may be stored in a computer-readable storage medium such as a storage unit that forms the control unit 61. 9, step S11 measures the position and orientation of the display device 1 viewed from the reference coordinate system based on the device position information supplied from the positioning device 54 and the human detection information supplied from the human detection device 55. . In step S <b> 12, the direction of the hand 501 </ b> A of the user 501 viewed from the display device 1 is measured based on the human detection information supplied from the human detection device 55. In step S13, the orientation of the measured hand 501A is converted from the coordinate system of the display device 1 to the reference coordinate system. In step S14, the direction of the hand 501A viewed from the reference coordinate system, that is, the intersection point 503 between the pointing direction 502 and the target space surface 601 is calculated, and the process ends.

  The human detection device 55 is not limited to a stereo camera. For example, a three-dimensional measurement device that can measure the position and orientation of the user 501 viewed from the reference coordinate system, such as Kinect Sensor (registered trademark) manufactured by Microsoft Corporation, is used. Needless to say, you can.

  FIG. 10 is a diagram illustrating an example of a database that associates coordinates with content and a content database. 10, (a) shows a database (DB: Data Base) 651 for associating coordinates and content, and (b) shows a content database (DB) 652. The DB 651 manages content IDs corresponding to spatial coordinate values viewed from the reference coordinate system. In this example, the DB 651 stores the target ID of the content associated with the coordinate values x, y, and z. On the other hand, the DB 652 manages content information (for example, the file name of the content) using the content target ID as an index. In this example, the DB 652 stores content information (for example, a content file name) associated with a content target ID. The DBs 651 and 652 are stored in the storage unit inside the display device 1 or outside the display device 1, and may be stored in the storage unit 65, for example.

  FIG. 11 is a flowchart illustrating an example of content selection processing. The content selection process is executed by the content selector 64. When the processing of the content selector 64 is performed by the control unit 61, the content selection processing is executed by the CPU that forms the control unit 61. In FIG. 11, step S21 acquires the spatial coordinate value seen from the reference coordinate system of the intersection 503 calculated by the pointing position detection process of FIG. In step S22, in order to absorb the measurement error, the target ID corresponding to the spatial coordinate value whose distance from the acquired spatial coordinate value is equal to or less than the threshold ε is read from the DB 651 and acquired. In step S23, the DB 652 is searched using the acquired target ID as an index, and the corresponding content is acquired.

  12 and 13 are a front view and a side view for explaining pointing by the user, and FIG. 14 is a view for explaining an example of information display regarding the pointed product.

  As shown in FIG. 12, when the user 501 passes in front of the product shelf 600, the display device 1 moves so as to be positioned between the user 501 and the product shelf 600. As shown in FIG. 13, when the user 501 points to the target product 610 on the product shelf 600 that can be seen through the transmissive screen 16 of the display device 1, the intersection point 503 is calculated by the pointing position detection process as described above, and the position of the intersection point 503 is obtained. The product 610 located in is recognized. Further, the content ID of the content related to the product 610 at the position of the intersection 503 is acquired from the DB 651 by the content selection process as described above, and the content of the corresponding product 610 is acquired by searching the DB 652 using the target ID as an index. Thereby, as shown in FIG. 14, the content information related to the product 610 is projected by the projector 14 onto the back surface 16-2 of the transmission screen 16 of the display device 1.

  In the example of FIG. 14, the image content related to the product 610 “This product is OO” is displayed on the transmissive screen 16, so that the user 501 can view the product 610 of the product shelf 600 viewed through the transmissive screen 16. By viewing the image content displayed in correspondence with the position, information regarding the product 610 can be obtained. In this example, as shown in FIG. 14, since the label 615 indicating the product name or price of the product 610 is provided on the product shelf 600, the image content related to the product 610 is displayed so as to point to the label 615. However, the image content may be displayed so as to point to the product 610. In short, it is only necessary that the image content is displayed so that the user 501 can easily understand that the image content is related to the product 610. Note that the audio playback unit 66 may output the audio content related to the product 610 “This product is OO”.

  By the way, a history of pointing by the user may be recorded. By analyzing such a history, it is possible to grasp the product information that the user wants to know, the tendency of the product that the user is interested in, and the like and reflect it in the marketing strategy and the like.

  FIG. 15 is a flowchart illustrating an example of the history recording process. The history recording process can be executed by the CPU forming the control unit 61. In the history recording process, a history can be recorded in a storage unit inside the display device 1 or outside the display device 1. The history may be recorded in the storage unit 65, for example.

  In FIG. 15, in step S31, a time stamp at the time when the user points to an object such as a product is recorded, and the user is photographed with the camera of the human detection device 55. In step S32, the user's face is detected by performing known image processing on the captured image. In step S33, the user's gender, age, and facial expression are extracted by analyzing the detected user's face image by known image processing. A step S34 calculates the stay time that the user has been in front of the display device 1 from the elapsed time until the time when the user is no longer detected by the human detection device 55 and the user disappears. In step S35, the information obtained in steps S33 and S34 is stored in, for example, the history database 653 shown in FIG. The history database 653 is stored in the storage unit 65, for example.

  The history database 653 illustrated in FIG. 16 stores a time stamp, a target ID such as a pointed product, a user's stay time, a user's gender, a user's age, a user's facial expression, and the like. The user's facial expression is, for example, the user's smile, and the degree of smile (smile level) may be stored in the history database 653. In this example, the higher the smile level, the more smile the user has. A known method used in a digital camera or the like can be used for extracting or detecting a user's smile.

  According to the above-described embodiment, it is possible to display the image content appropriately shaded on the real-world video that the user is viewing through the transmissive screen of the display device. For example, mobile digital signage as a display device can be used as an avatar for a user at a remote location. In this case, a significant difference from conventional signage is that in the above embodiment, an image (avatar) of a person at a remote place is projected on a transmissive screen after being appropriately shaded on the background visible from the user's viewpoint. It is a point that an effect that a sense of reality is remarkably increased can be obtained.

  Further, the object that the user sees through the transmissive screen is not limited to the product shelf, but may be a landmark assembled in a landscape, a participant in a TV conference system, or the like.

  Further, in the above embodiment, since the light source direction is estimated using the optical sensor provided in the display device, a sudden change in the brightness of the light source can be detected, and the light source direction can be known in real time. There is no need to estimate the three-dimensional position of the light source, only the direction of the light source viewed from the transmissive screen that is the image projection plane is estimated, and shading processing based on the estimated light source direction is performed. It is possible to present an AR image that is consistent between the illumination state of the image in the world and the shading process of the content.

The following additional notes are further disclosed with respect to the embodiment including the above examples.
(Appendix 1)
Depending on the lighting conditions of the environment where the transmissive display is installed and the relative position between the user position and the transmissive display, the image content displayed on the transmissive display is subjected to shading processing,
The image content that has been subjected to the shading process is superimposed and displayed on a real-world image that can be seen through the transmissive display from the user position,
The display method according to claim 1, wherein the illumination condition is a light source direction viewed from the transmissive display.
(Appendix 2)
Detecting a position where the user points to the transmissive display;
The display method according to claim 1, wherein the image content includes information on the pointing position.
(Appendix 3)
The display method according to claim 1 or 2, wherein the pointing position is detected by detecting a position touched by a user on a touch panel provided integrally with the transmissive display.
(Appendix 4)
The pointed position is
Measuring the pointing direction as seen from the transmissive display based on the position information of the transmissive display supplied from the positioning device and the detection information of the user supplied from the three-dimensional measuring device,
The measured pointing direction is converted from the coordinate system of the transmissive display to the reference coordinate system, and the intersection of the pointing direction viewed from the reference coordinate system and the target space plane of the object seen through the transmissive display is obtained. The display method according to appendix 1 or 2, wherein the display method is obtained by calculation.
(Appendix 5)
The transmissive display is formed of a transmissive screen,
The display method according to any one of appendices 1 to 4, wherein the image content subjected to the shading process is projected onto the transmissive screen by a projector.
(Appendix 6)
Estimating the light source direction based on the illuminance distribution detected by the plurality of first optical sensors;
In learning, the light source direction is learned on the basis of illuminance distributions detected by the plurality of second optical sensors and the plurality of first optical sensors provided only during the learning. 6. The display method according to any one of 5 above.
(Appendix 7)
The display method according to any one of appendices 1 to 6, wherein a history of the pointing is recorded in a database.
(Appendix 8)
A transmissive display;
A plurality of light sensors;
An estimation unit that estimates a light source direction viewed from the transmissive display based on detection outputs of the plurality of optical sensors;
A detection device for detecting a relative position between a user position and the transmissive display;
A shadow processing unit that performs a shadow process on the image content according to the light source direction and the relative position;
A display device comprising: a projector that projects and displays the image content that has undergone the shading process on a real-world image that can be seen from the user position through the transmissive display.
(Appendix 9)
Detecting means for detecting a position pointed to by the user with respect to the transmissive display;
The display device according to appendix 8, wherein the image content includes information regarding the pointed position.
(Appendix 10)
The display according to appendix 8 or 9, wherein the detection means includes a touch panel that is provided integrally with the transmissive display and detects the pointed position by detecting a position touched by a user. apparatus.
(Appendix 11)
The detection means includes
A positioning device that measures the position of the transmissive display and outputs position information;
A three-dimensional measurement device that detects the user and outputs detection information;
The pointing direction seen from the display device measured based on the position information and the detection information is converted from the coordinate system of the display device to the reference coordinate system, and the pointing direction seen from the reference coordinate system; The display device according to appendix 8 or 9, further comprising: a control unit that calculates an intersection with a target space plane of the target that can be seen through the transmissive display.
(Appendix 12)
The display device according to any one of appendices 8 to 11, wherein the transmissive display is formed of a transmissive screen.
(Appendix 13)
The display device according to any one of appendices 8 to 12, further comprising a control unit that records a history of the pointing in a database.

  Although the disclosed display method and display device have been described with reference to the embodiments, it is needless to say that the present invention is not limited to the above embodiments, and various modifications and improvements can be made within the scope of the present invention. .

DESCRIPTION OF SYMBOLS 1 Display apparatus 11 Base 12 Frame 13, 13-1 to 13-N, 131 Sensor 14 Projector 16 Transmission type screen 51 Light source direction estimation and learning part 52 Shading process part 53 Shooting apparatus 54 Positioning apparatus 55 Human detection apparatus 56 Touch panel 61 Control Part

Claims (5)

The first surface and the first surface in accordance with a relative position between opposite second illumination conditions of the environment in which the transmission screen is provided with a surface, and the transmission screen with a user position of the Apply shading to image content,
Projecting, on the first or second surface of the transmissive screen, the image content obtained by performing the shading process on the real-world image seen through the first surface of the transmissive screen from the user position. Overlapping display
The display method according to claim 1, wherein the illumination condition is a light source direction viewed from the transmission screen . A transmissive screen having a first surface and a second surface opposite to the first surface ;
A plurality of light sensors;
An estimation unit that estimates a light source direction viewed from the transmission screen based on detection outputs of the plurality of optical sensors;
A detection device for detecting a relative position between a user position and the transmission screen ;
A shadow processing unit that performs a shadow process on the image content according to the light source direction and the relative position;
Projecting, on the first or second surface of the transmissive screen, the image content obtained by performing the shading process on the real-world image seen through the first surface of the transmissive screen from the user position. A display device comprising a projector that displays images in a superimposed manner. Detecting means for detecting a position pointed to by the user with respect to the transmissive screen ;
The display device according to claim 2, wherein the image content includes information regarding the pointed position. The display device according to claim 3 , wherein the detection unit includes a touch panel that is provided integrally with the transmission screen and detects the pointed position by detecting a position touched by a user. . The detection means includes
A positioning device that measures the position of the transmission screen and outputs position information;
A three-dimensional measurement device that detects the user and outputs detection information;
The pointing direction seen from the display device measured based on the position information and the detection information is converted from the coordinate system of the display device to a reference coordinate system, the pointing direction seen from the reference coordinate system, and The display device according to claim 3 , further comprising: a control unit configured to calculate an intersection point with a target space surface of a target seen through the first surface of the transmission screen .

Images