WO2016068015A1

WO2016068015A1 - Coordinate acquisition device and display device

Info

Publication number: WO2016068015A1
Application number: PCT/JP2015/079840
Authority: WO
Inventors: 洋一久下
Original assignee: シャープ株式会社
Priority date: 2014-10-28
Filing date: 2015-10-22
Publication date: 2016-05-06

Abstract

The purpose of the present invention is to realize a coordinate acquisition device with which it is possible to cause, inter alia, coordinates on the display screen of a display device, to be appropriately specified without requiring a touch panel on the basis of, inter alia, the position of a user's hand in three-dimensional space, even from a remote position. An image-capturing unit acquires a captured image according to a prescribed angle of view. A sensor unit (sensor (DS1) and a sensor control unit (2)) acquire: a user proximity point, which is the position of, inter alia, the hand of the user; and a distance to the user proximity point. An arithmetic unit (3) calculates the distance from the device to a plane that includes both eyes of the user on the basis of the captured image, and calculates an overlapping range of visual fields, which is the range in which the user proximity point and the display face of the display device appear to be overlapping from the user's visual point, on the basis of: the user proximity point, which is acquired by the sensor unit; the distance to the user proximity point from the device; and the horizontal and vertical widths of the display face of the display device. When it is determined that the user proximity point is within the overlapping range of visual fields, the arithmetic unit (3) calculates the coordinate information of the user proximity point and outputs the result of the calculation.

Description

Coordinate acquisition device and display device

The present invention relates to a technique for specifying coordinates on a display screen of a display device based on the position of a user's hand in a three-dimensional space.

For example, in Patent Document 1 (Japanese Patent Application Laid-Open No. 2012-215964), coordinate information can be input via a touch panel by displaying a stereoscopic image and allowing a user to touch an imaging position of the stereoscopic image. There is a disclosure of a stereoscopic image display operation device that can be used.

However, the above-described stereoscopic image display operation device is premised on input using a touch panel, and a user's proximity region to the extent that can cause a capacitance change (electric field change) on the touch panel of the stereoscopic image display operation device. It is necessary to move a finger or the like.

That is, in the above-described stereoscopic image display operation device, it is difficult for the user to operate the display device having a large display screen by moving a finger or the like from a remote position.

In addition, since a display device having a large display screen is usually viewed from a distance, it is extremely complicated if it cannot be operated unless it is close to the display device.

Furthermore, when a touch panel is mounted on a display device having a large display screen, the size of the touch panel is increased, resulting in an increase in cost.

Therefore, in view of the above problems, the present invention does not require a touch panel, and appropriately sets the coordinates on the display screen of the display device based on the position of the user's hand in the three-dimensional space, even from a remote position. An object is to realize a coordinate acquisition device and a display device that can be specified.

In order to solve the above-described problem, the first configuration is a coordinate acquisition device for acquiring coordinate information on the display surface of the display device, and includes an imaging unit, a sensor unit, and a calculation unit.

The imaging unit acquires a captured image by imaging a subject with a predetermined angle of view.

The sensor unit acquires the user proximity point, which is the position of the user's hand or the object being held by the user, and the distance to the user proximity point.

The calculation unit
(1) In the captured image acquired by the imaging unit, the user's eyes are detected, and based on the detected position of both eyes on the captured image, the distance between both eyes, and the angle of view, Calculate the distance to the plane containing the user ’s eyes,
(2) Based on the user proximity point acquired by the sensor unit and the distance from the user's own device to the user proximity point, and the horizontal and vertical widths of the display surface of the display device, Calculate the overlapping view range, which is the range where the proximity point and the display surface of the display device appear to overlap,
(3) When it is determined that the user proximity point is within the overlapping view range, coordinate information of the user proximity point is calculated and output.

According to the present invention, the coordinates on the display screen of the display device can be appropriately specified based on the position of the user's hand in the three-dimensional space even from a remote position without the need for a touch panel. A coordinate acquisition device and a display device can be realized.

1 is a schematic configuration diagram of a display device 1000 according to a first embodiment. In the display apparatus 1000 (coordinate acquisition apparatus 100) according to the first embodiment, the position of the user's hand (position in the three-dimensional space) is acquired, and the coordinate information on the display surface Scn corresponding to the position of the hand is acquired. The flowchart of the process to (output). The figure for demonstrating the method to calculate the distance D from the imaging part 1 to a user's position using the captured image Img1 which the imaging part 1 acquired with the horizontal view angle (alpha). The figure which showed typically the display surface Scn of display panel LCD, sensor DS1, and the user's hand as a measuring object. The figure for demonstrating the identification method of the x coordinate position of a user's viewpoint. The figure for demonstrating the calculation method of the x coordinate Xmin of the lower limit position of the x-axis direction of an overlapping visual field range. The figure for demonstrating the calculation method of x coordinate Xmax of the upper limit position of the x-axis direction of an overlapping visual field range. The figure for demonstrating the identification method of the y coordinate position of a user's viewpoint. The figure for demonstrating the calculation method of y coordinate Ymin of the lower limit position of the y-axis direction of an overlapping visual field range. The figure for demonstrating the calculation method of y coordinate Ymax of the upper limit position of the y-axis direction of an overlapping view range. The figure for demonstrating the process which determines whether the x coordinate of the position P0 of a user's finger | toe (part closest to the display surface Scn) is in an overlapping view range. The figure for demonstrating the process which determines whether the y coordinate of the position P0 of a user's finger | toe (part closest to the display surface Scn) is in an overlapping view range.

[First Embodiment]
The first embodiment will be described below with reference to the drawings.

Hereinafter, a display device 1000 will be described as an example of a device using a coordinate acquisition device.

<1.1: Configuration of display device>
FIG. 1 is a schematic configuration diagram of the display device 1000.

As shown in FIG. 1, the display device 1000 includes a coordinate acquisition device 100, a display control unit 4, and a display panel (eg, a liquid crystal display, an organic EL display, etc.) LCD.

As shown in FIG. 1, the coordinate acquisition apparatus 100 includes an imaging unit 1, sensors DS 1 and DS 2, a sensor control unit 2, and a calculation unit 3.

The imaging unit 1 is installed in, for example, a part of the housing of the display panel LCD (for example, near the center of the upper peripheral portion (frame portion) of the display surface Scn) and is a subject that exists in the normal direction of the display surface Scn (Scene) is imaged. That is, the imaging unit 1 captures an image of a subject or the like facing the display surface Scn (for example, a user looking at the display surface Scn).

Sensor DS1 is, for example, a proximity sensor. The sensor DS1 irradiates an electromagnetic wave such as infrared light, and receives the electromagnetic wave (reflected wave) reflected and returned by the object. Then, the sensor DS1 outputs a signal corresponding to the received light amount of the reflected wave (a signal converted into a predetermined physical quantity) to the sensor control unit 2. The sensor DS1 and the sensor control unit 2 may capture reflected light such as infrared light emitted by the sensor DS1, and determine the distance based on the position and intensity.

In addition, although this embodiment demonstrates to an example the case where there is one sensor, the number of sensors is not limited to one and may be two or more. Furthermore, the position where the sensor is arranged may be determined according to the measurement object.

The sensor control unit 2 inputs a signal output from the sensor DS1 (a signal corresponding to the amount of reflected wave received), and acquires (specifies the position of the measurement object (for example, a user's hand) based on the signal. ) The sensor control unit 2 controls the intensity and irradiation angle of electromagnetic waves such as infrared light emitted from the sensor DS1, and information such as the received light intensity and angle of the reflected wave from the object is output from the sensor DS1. Based on the signal to be identified. Thereby, the sensor control part 2 can acquire the positional information on a measuring object (for example, a user's hand). Then, the sensor control unit 2 outputs the acquired position information of the measurement object to the calculation unit 3.

The calculation unit 3 inputs an image (video) captured by the imaging unit 1 and position information of a measurement object (for example, a user's hand) acquired by the sensor control unit 2. Based on the image (video) captured by the imaging unit 1 and the position information of the measurement object (for example, the user's hand) acquired by the sensor control unit 2, the calculation unit 3 determines the position of the user's hand as follows: It is determined whether or not it exists at a position where it can be seen to overlap with the display surface Scn of the display panel LCD as viewed from the user. Then, when the calculation unit 3 determines that the position of the user's hand is present at a position where the user's hand is seen to overlap the display surface Scn of the display panel LCD, based on the position of the user's hand, The coordinates on the display surface Scn corresponding to the position of the hand are calculated. Then, the calculation unit 3 outputs information regarding the calculated coordinates to the display control unit 4.

The display control unit 4 includes an input control unit 41, an application unit 42, and a display processing unit 43, as shown in FIG.

The input control unit 41 inputs information regarding coordinates output from the calculation unit 3. The input control unit 41 outputs a control signal to the application unit 42 and / or the display processing unit 43 based on the coordinate information input from the calculation unit 3. For example, when the coordinate information is input from the calculation unit 3 after a period in which the coordinate information is not input from the calculation unit 3 continues for a certain period, the input control unit 41 determines the display surface Scn according to the coordinate information. Data and control signals necessary for updating a predetermined display (for example, for executing update processing such as changing the color of an icon) are generated, and the generated data and control signals are output to the application unit 42. .

Further, the input control unit 41 outputs a control signal for controlling the display processing unit 43 to the display processing unit 43.

The application unit 42 inputs data and control signals output from the input control unit 41. The application unit 42 executes predetermined processing on the data output from the input control unit 41 in accordance with the control signal output from the input control unit 41. For example, the application unit 42 performs predetermined display update (for example, icon display update) on the coordinate position of the display surface Scn based on the coordinate information indicating the position of the user's hand output from the input control unit 41. Data for performing is generated, and the generated data is output to the display processing unit 43.

The display processing unit 43 inputs a control signal output from the input control unit 41 and data output from the application unit 42. The display processing unit 43 controls the display panel LCD according to the control signal output from the input control unit 41 so that the display based on the data output from the application unit 42 is performed on the display surface Scn. That is, the display processing unit 43 outputs display data to be displayed on the display surface Scn of the display panel LCD. Further, the display processing unit 43 outputs a display control signal for causing the display data to be displayed on the display surface Scn of the display panel LCD to the display panel LCD.

The display panel LCD is realized by, for example, a liquid crystal display, an organic EL display, or the like. The display panel LCD displays the display data output from the display processing unit 43 of the display control unit 4 on the display surface Scn based on the display control signal output from the display processing unit 43 of the display control unit 4.

<1.2: Operation of display device>
The operation of the display device 1000 configured as described above will be described below.

Hereinafter, in the display apparatus 1000 (coordinate acquisition apparatus 100), a process of acquiring the position of the user's hand (position in the three-dimensional space) and acquiring coordinate information on the display surface Scn corresponding to the position of the hand. The case where it is executed will be described.

FIG. 2 shows a display device 1000 (coordinate acquisition device 100) that acquires the position of the user's hand (position in the three-dimensional space) and acquires coordinate information on the display surface Scn that corresponds to the position of the hand (output). It is a flowchart of the process to do.

(S1):
In step S1, the imaging unit 1 images a subject (scene) that exists in the normal direction of the display surface Scn. That is, the imaging unit 1 captures a subject or the like facing the display surface Scn (for example, a user looking at the display surface Scn) and acquires a captured image. Note that the imaging unit 1 acquires a captured image at a horizontal angle of view α.

(S2, S3):
In step S 2, the calculation unit 3 detects the position of the user's eyes from the captured image acquired by the imaging unit 1. For example, the computing unit 3 detects the position of the user's eyes by specifying an image region having human eye characteristics from the captured image by matching processing or the like.

If the user's eyes are detected (“Yes” in step S3), the process proceeds to step S4. If the user's eyes are not detected (“No” in step S3), the process is terminated. .

(S4):
In step S4, the calculation unit 3 calculates a distance D from the imaging unit 1 to the user. This will be described with reference to FIG.

FIG. 3 is a diagram for explaining a method of calculating the distance D from the imaging unit 1 to the user position using the captured image Img1 acquired by the imaging unit 1 at the horizontal angle of view α. In FIG. 3, it is assumed that the center of both eyes of the user is on the optical axis of the optical system of the imaging unit 1 and the distance from the imaging unit 1 to the center point of both eyes of the user is D. ,explain. In addition, the captured image Img1 illustrated in FIG. 3 is an image acquired (captured) at the horizontal angle of view α by the imaging unit 1 in the situation illustrated in FIG.

As shown in FIG. 3, the horizontal width of the captured image Img1 is Wc, the distance between both eyes of the user is d0, and the horizontal width of the captured image Img1 is the distance d0 between the eyes of the user with respect to Wc. If the ratio is P,
Wc = 2 × D × tan (α / 2) (1)
P = d0 / Wc (2)
Therefore, the calculation unit 3 calculates the distance D from the imaging unit 1 to the user's position,
D = d0 / (P × 2 × tan (α / 2)) (3)
To calculate.

Note that the distance between the eyes of the user (the distance between the eyes of the person) is about 6 [cm].
d0 = 6 [cm]
Thus, the calculation unit 3 can calculate the distance D from the imaging unit 1 to the user's position.

That is, in the captured image Img1 acquired at the horizontal angle of view α, the calculation unit 3 calculates the ratio P of the distance d0 between the user's eyes to the horizontal width of the captured image Img1 with respect to Wc. By executing the processing corresponding to the equation, the distance D from the imaging unit 1 to the user's position is calculated.

(S5):
In step S5, the sensor DS1 emits an electromagnetic wave such as infrared light, and receives the electromagnetic wave (reflected wave) reflected and returned from the object. Then, the sensor DS1 outputs a signal corresponding to the received light amount of the reflected wave (a signal converted into a predetermined physical quantity) to the sensor control unit 2.

The sensor control unit 2 acquires (identifies) the position of the measurement object (for example, the user's hand) based on the signal input from the sensor DS1 (signal corresponding to the amount of reflected wave received). The sensor control unit 2 controls the intensity and irradiation angle of electromagnetic waves such as infrared light emitted from the sensor DS1, and information such as the received light intensity and angle of the reflected wave from the object is output from the sensor DS1. Based on the signal to be identified. Thereby, the sensor control part 2 acquires the positional information on a measuring object (for example, a user's hand). Then, the sensor control unit 2 outputs the acquired position information of the measurement object to the calculation unit 3.

FIG. 4 is a diagram schematically showing the display surface Scn of the display panel LCD, the sensor DS1, and the user's hand as a measurement object. FIG. 4 is a diagram of the sensor DS1 and the user's hand as a measurement object as viewed from above. As shown in FIG. 4, the x-axis, y-axis, and z-axis are set, and the position of the sensor DS1 The x coordinate of is set to “0” (origin).

The sensor DS1 receives the reflected wave from the point P0 that is the tip position of the user's hand. The sensor control unit 2 is based on the received light amount of the reflected wave from the point P0 that is the tip position of the user's hand.
(1) a distance Z0 in the z-axis direction from the sensor DS1 to the point P0 that is the tip position of the user's hand;
(2) x-coordinate X0 of point P0;
To get.

Note that the distance Z0 is the distance in the z-axis direction from the sensor DS1 to the point P0 that is the tip position of the user's hand, and is therefore equal to the distance from the display surface Scn of the display panel LCD to the point P0.

(S6):
In step S 6, the calculation unit 3 specifies the overlapping view range when the user's finger (portion closest to the display surface Scn) exists at a distance Z 0 from the display surface Scn.

The “overlapping field of view” refers to the user's finger (tip of the hand) (which may be the tip of a pointer, pen, or the like) and the display surface Scn when the user views the display surface Scn. It is the range that appears to overlap.

The method for specifying the overlapping field of view will be described below.

First, a method for specifying the x coordinate position of the user's viewpoint will be described.

FIG. 5 is a diagram for explaining a method of specifying the x coordinate position of the user's viewpoint.

FIG. 5 is a diagram seen from above, and schematically shows the imaging unit 1, the display surface Scn of the display panel LCD, and a plane including both eyes of the user. FIG. 5 schematically shows a captured image Img2 captured by the imaging unit 1 at a horizontal angle of view α. Note that the captured image Img2 is illustrated as an image (mirror image) in which left and right are reversed in order to match the positional relationship of the x axis.

Also, as shown in FIG. 5, the x-axis, y-axis, and z-axis are set, and the x-coordinate of the position of the imaging unit 1 (the optical axis position of the optical system of the imaging unit 1) is set to “0” (the origin ). Also, as shown in FIG. 5, the x coordinate of the center position of both eyes of the user is X1, and the length in the x-axis direction of the range imaged by the horizontal angle of view α on the plane at a distance D from the display surface Scn. Is Wc.

Further, as shown in FIG. 5, the horizontal width of the captured image Img2 is Xpic, and the length in the x-axis direction (horizontal direction) from the center point of the captured image to the center position of both eyes of the user is Xpc. .

When set as described above, the calculation unit 3 calculates the x coordinate X1 of the center point of the user's eyes,
X1 = (Wc / 2) × Xpc / (Wpic / 2)
Calculated by

The calculation unit 3
Wc = 2D × tan (α)
To calculate the length Wc.

Through the above process, the x coordinate position of the user's viewpoint (the center point of both eyes of the user) is specified.

Next, when the user's finger (the portion closest to the display surface Scn) is present at a distance Z0 from the display surface Scn, when the display surface Scn is viewed from the user's viewpoint (center point of both eyes of the user) P1 The lower limit position in the x-axis direction (the lower limit position in the x-axis direction of the overlapping view range) and the upper limit position (the upper limit position in the x-axis direction of the overlapping view range) in which the user's finger and the display surface Scn appear to overlap each other A calculation method will be described. Note that the x coordinate of the lower limit position in the x-axis direction of the overlapping view range is Xmin, and the x coordinate of the upper limit position of the overlapping view range in the x-axis direction is Xmax.

FIG. 6 is a diagram for explaining a method of calculating the x-coordinate Xmin of the lower limit position in the x-axis direction of the overlapping view range.

FIG. 7 is a diagram for explaining a method of calculating the x-coordinate Xmax of the upper limit position in the x-axis direction of the overlapping view range.

The length of the display surface Scn in the x-axis direction (horizontal width) is Wd, the x coordinate of the center position of the display surface Scn is “0”, the x coordinate of the user's viewpoint P1 is X1, and explain.

As shown in FIG.
D: (D−Z0) = (X1 − (− Wd / 2)): (X1−Xmin)
Therefore, the x-coordinate Xmin of the lower limit position in the x-axis direction of the overlapping view range is
Xmin = (Wd / 2 + X1) × Z0 / D−Wd / 2
Can be calculated.

In addition, as shown in FIG.
D: (D−Z0) = (Wd / 2−X1): (Xmax−X1)
Therefore, the x coordinate Xmax of the upper limit position in the x-axis direction of the overlapping view range is
Xmax = (X1-Wd / 2) × Z0 / D + Wd / 2
Can be calculated.

The calculation unit 3 calculates the x-coordinate Xmin of the lower limit position in the x-axis direction of the overlapping view range and the x-coordinate Xmax of the upper limit position of the overlapping view range in the x-axis direction by a process corresponding to the above formula.

As described above, the x coordinate Xmin of the lower limit position in the x-axis direction of the overlapping view range and the x coordinate Xmax of the upper limit position in the x-axis direction of the overlapping view range are determined.

Next, a method for specifying the y coordinate position of the user's viewpoint will be described.

FIG. 8 is a diagram for explaining a method of specifying the y coordinate position of the user's viewpoint.

FIG. 8 is a view seen from the side (x-axis direction), and schematically shows the imaging unit 1, the display surface Scn of the display panel LCD, and a plane including both eyes of the user. FIG. 8 schematically shows a captured image Img2 captured by the imaging unit 1 at a vertical angle of view β. Note that the captured image Img2 is shown with the positional relationship of the y-axis matched. Further, the captured image Img2 is illustrated as an image (mirror image) obtained by inverting the left and right as in FIG.

Also, as shown in FIG. 8, the x-axis, y-axis, and z-axis are set, and the y-coordinate of the position of the imaging unit 1 (the optical axis position of the optical system of the imaging unit 1) is set to “0” (the origin ). The y coordinate of the center point of the display surface Scn is also “0”.

Also, as shown in FIG. 8, the y coordinate of the center position of both eyes of the user is Y1, and the length in the y-axis direction of the range imaged at the vertical angle of view β on the plane at the distance D from the display surface Scn. Is Hc.

Also, as shown in FIG. 8, the vertical width of the captured image Img2 is Hpic, and the length in the y-axis direction (vertical direction) from the center point of the captured image to the center position of both eyes of the user is Ypc. .

When set as described above, the calculation unit 3 sets the y-coordinate Y1 of the center point of both eyes of the user as
Y1 = (Hc / 2) × Ypc / (Hpic / 2)
Calculated by

The calculation unit 3
Hc = 2D × tan (β)
To calculate the length Hc.

Through the above process, the y coordinate position of the user's viewpoint (the center point of both eyes of the user) is specified.

Next, when the user's finger (the portion closest to the display surface Scn) is present at a distance Z0 from the display surface Scn, when the display surface Scn is viewed from the user's viewpoint (center point of both eyes of the user) P1 Furthermore, the lower limit position in the y-axis direction (the lower limit position in the y-axis direction of the overlapping view range) and the upper limit position (the upper limit position in the y-axis direction of the overlapping view range) in which the user's finger and the display surface Scn appear to overlap each other A calculation method will be described. Note that the y coordinate of the lower limit position in the y-axis direction of the overlapping view range is Ymin, and the y coordinate of the upper limit position of the overlapping view range in the y-axis direction is Ymax.

FIG. 9 is a diagram for explaining a method for calculating the y-coordinate Ymin of the lower limit position in the y-axis direction of the overlapping view range.

FIG. 10 is a diagram for explaining a method of calculating the y-coordinate Ymax of the upper limit position in the y-axis direction of the overlapping view range.

The length of the display surface Scn in the y-axis direction (vertical length) is Ht, the y-coordinate of the center position of the display surface Scn is “0”, and the y-coordinate of the user's viewpoint P1 is Y1. ,explain.

As shown in FIG.
D: (D−Z0) = (Y1 − (− Ht / 2)): (Y1−Ymin)
Therefore, the y coordinate Ymin of the lower limit position in the y-axis direction of the overlapping view range is
Ymin = (Ht / 2 + Y1) × Z0 / D−Ht / 2
Can be calculated.

Also, as shown in FIG.
D: (D−Z0) = (Ht / 2−Y1): (Ymax−Y1)
Therefore, the y coordinate Ymax of the upper limit position in the y-axis direction of the overlapping view range is
Ymax = (Y1−Ht / 2) × Z0 / D + Ht / 2
Can be calculated.

The calculation unit 3 calculates the y-coordinate Ymin of the lower limit position in the y-axis direction of the overlapping view range and the y-coordinate Ymax of the upper limit position in the y-axis direction of the overlapping view range by a process corresponding to the above mathematical formula.

As described above, the y-coordinate Ymin of the lower limit position in the y-axis direction of the overlapping view range and the y-coordinate Ymax of the upper limit position of the overlapping view range in the y-axis direction are determined.

The overlapping view range is specified by the above processing. That is, the calculation unit 3 performs the above processing, and (1) the x-axis of the overlapping view range when the user's finger (portion closest to the display surface Scn) exists at a distance Z0 from the display surface Scn. X coordinate Xmin of the lower limit position in the direction, x coordinate Xmax of the upper limit position in the x-axis direction of the overlapping view range, (2) y coordinate Ymin of the lower limit position in the y-axis direction of the overlapping view range, and y of the overlapping view range The y coordinate Ymax of the upper limit position in the axial direction is specified (calculated).

(S7):
In step S 7, the calculation unit 3 determines whether or not the user's finger (the part closest to the display surface Scn) is within the overlapping view range. This will be described with reference to FIGS.

FIG. 11 is a diagram for explaining processing for determining whether or not the x-coordinate of the position P0 of the user's finger (portion closest to the display surface Scn) is within the overlapping view range.

FIG. 12 is a diagram for explaining processing for determining whether or not the y coordinate of the position P0 of the user's finger (portion closest to the display surface Scn) is within the overlapping view range.

The calculation unit 3 acquires the x coordinate X0 of the position P0 of the user's finger, and the x coordinate X0 is
Xmin ≦ X0 ≦ Xmax
It is determined whether or not the above is satisfied.

Further, the calculation unit 3 acquires the y coordinate Y0 of the position P0 of the user's finger, and the y coordinate Y0 is
Ymin ≦ Y0 ≦ Ymax
It is determined whether or not the above is satisfied.

If the above two inequality conditions are satisfied, the calculation unit 3 determines that the position P0 of the user's finger is within the overlapping field of view (“Yes” in step S7), and advances the process to step S8.

On the other hand, in cases other than the above, the calculation unit 3 determines that the position P0 of the user's finger is not within the overlapping view range ("No" in step S7), and ends the process.

(S8, S9):
In step S8, the calculation unit 3 calculates the coordinates (X coordinate, Y coordinate) of the corresponding point P2 on the display surface Scn corresponding to the position P0 of the user's finger (portion closest to the display surface Scn).

First, the calculation process of the X coordinate of the point P2 will be described with reference to FIG. As shown in FIG. 11, the X coordinate on the display surface Scn is “0” for the end of the display surface Scn in the negative x-axis direction and “Wd” for the end of the display surface Scn in the positive x-axis direction. (Wd: length of the display surface Scn in the x-axis direction). Further, the X coordinate of the point P2 on the display surface Scn is set as Xout and will be described below.

As shown in FIG.
D: (D−Z0) = Xout: (X0−Xmin)
Therefore, the X coordinate Xout of the corresponding point P2 on the display surface Scn corresponding to the position P0 of the user's finger (portion closest to the display surface Scn) is
Xout = (X0−Xmin) × D / (D−Z0)
It is.

That is, the calculation unit 3 calculates the X coordinate Xout of the corresponding point P2 on the display surface Scn corresponding to the position P0 of the user's finger (portion closest to the display surface Scn) by performing processing corresponding to the above formula. To do.

Next, the calculation process of the Y coordinate of the point P2 will be described with reference to FIG. As shown in FIG. 12, the Y coordinate on the display surface Scn is “0” at the end of the display surface Scn in the negative y-axis direction (the lower end of the display surface Scn), and is positive on the x-axis of the display surface Scn. An end portion in the direction (upper end portion of the display surface Scn) is defined as “Ht” (Ht: length of the display surface Scn in the y-axis direction). Further, the Y coordinate of the point P2 on the display surface Scn is set as Yout and will be described below.

As shown in FIG.
D: (D−Z0) = Yout: (Y0−Ymin)
Therefore, the Y coordinate Yout of the corresponding point P2 on the display surface Scn corresponding to the position P0 of the user's finger (portion closest to the display surface Scn) is
Yout = (Y0−Ymin) × D / (D−Z0)
It is.

That is, the calculation unit 3 performs a process corresponding to the above mathematical formula to calculate the Y coordinate Yout of the corresponding point P2 on the display surface Scn corresponding to the position P0 of the user's finger (portion closest to the display surface Scn). To do.

The calculation unit 3 uses the display control unit 4 to obtain the X coordinate Xout and the Y coordinate Yout of the corresponding point P2 on the display surface Scn corresponding to the position P0 of the user's finger (the portion closest to the display surface Scn) acquired by the above processing. It outputs to the input control part 41 (step S9).

As described above, the display apparatus 1000 (coordinate acquisition apparatus 100) acquires the position of the user's hand (position in the three-dimensional space), and acquires coordinate information on the display surface Scn corresponding to the position of the hand. (Output.

The input control unit 41 outputs a control signal to the application unit 42 and / or the display processing unit 43 based on the coordinate information input from the calculation unit 3. For example, when the coordinate information is input from the calculation unit 3 after a period in which the coordinate information is not input from the calculation unit 3 continues for a certain period, the input control unit 41 determines the display surface Scn according to the coordinate information. Data and control signals necessary for updating a predetermined display (for example, for executing update processing such as changing the color of an icon) are generated, and the generated data and control signals are output to the application unit 42. .

The application unit 42 executes predetermined processing on data output from the input control unit 41 in accordance with a control signal output from the input control unit 41. For example, the application unit 42 performs predetermined display update (for example, icon display update) on the coordinate position of the display surface Scn based on the coordinate information indicating the position of the user's hand output from the input control unit 41. Data for performing is generated, and the generated data is output to the display processing unit 43.

The display processing unit 43 controls the display panel LCD according to the control signal output from the input control unit 41 so that the display based on the data output from the application unit 42 is performed on the display surface Scn. That is, the display processing unit 43 outputs display data to be displayed on the display surface Scn of the display panel LCD. Further, the display processing unit 43 outputs a display control signal for causing the display data to be displayed on the display surface Scn of the display panel LCD to the display panel LCD.

The display panel LCD displays the display data output from the display processing unit 43 of the display control unit 4 on the display surface Scn based on the display control signal output from the display processing unit 43 of the display control unit 4.

As described above, in the display device 1000 (coordinate acquisition device 100), the position information of the user's viewpoint is acquired from the captured image acquired by the imaging unit 1, and further, the three-dimensional space is acquired by the sensor DS1 and the sensor control unit 2. The coordinate information of the position of the user's hand (the part closest to the display surface Scn) is acquired. In the display device 1000 (coordinate acquisition device 100), the calculation unit 3 allows the user to display the display surface Scn based on the position information of the user's viewpoint and the coordinate information of the hand position (the portion closest to the display surface Scn). When the user sees, the overlapping view range, which is the range in which the position of the user's hand (the tip of the hand) (or the tip of a pointer or pen, etc.) and the display surface Scn can be seen, is specified. Then, in the display device 1000 (coordinate acquisition device 100), when the position of the user's hand (the portion closest to the display surface Scn) is within the overlapping field of view, the calculation unit 3 calculates the position of the user's viewpoint and the position of the user's hand. The coordinates (Xout, Yout) of the point where the straight line connecting to the display surface Scn intersects are calculated and output. Thus, since the display device 1000 (coordinate acquisition device 100) does not require a touch panel, the user's hand can be determined based on the position of the user's hand in the three-dimensional space even from a position away from the display surface Scn. It is possible to appropriately specify (acquire) the coordinate information on the display surface Scn corresponding to the position and the like.

In the captured image acquired by the imaging unit 1, the position of the user's hand and face is detected, and when the distance between the hand and the face is equal to or less than a predetermined value, the calculation unit 3 outputs the coordinate information in step S9. You may make it not perform the process to perform. Thereby, in the display apparatus 1000, for example, it is possible to detect a movement of the user's hand as a gesture and suppress malfunctions when executing processing according to the detected gesture. That is, by performing the processing as described above, when the user unconsciously touches his / her face, the display device 1000 can appropriately prevent the display device 1000 from erroneously recognizing as a predetermined gesture and malfunctioning. Can do.

In addition, when the position of the user's face is detected in the captured image acquired by the imaging unit 1 and the distance from the display device 1000 to the user's face is greater than a certain value (where the user's face is farther than a predetermined distance) The calculation unit 3 may not execute the process of outputting the coordinate information in step S9. Accordingly, it is possible to appropriately prevent the display device 1000 from erroneously recognizing the motion of the user's hand in a position farther than the predetermined distance as the predetermined gesture and causing the display device 1000 to malfunction.

(1) When the ratio of the distance from the display surface Scn to the position of the user's hand (distance in the z-axis direction) and the distance from the position of the user's hand to the face (distance in the z-axis direction) changes (2) When the coordinates (Xout, Yout) of the point where the straight line connecting the user's viewpoint and the position of the user's hand intersects the display surface Scn does not change, the display control unit 4 performs a click operation. May be determined, a predetermined process may be executed, and control may be performed so that display according to the process is performed on the display surface Scn.

For example, as shown in FIGS. 11 and 12, when the position of the user's viewpoint does not change and the user's finger is moved (or reciprocated) from the point P0 to the point P3, the display control unit 4 performs the click operation. It may be determined that a predetermined process is executed, and display according to the process is performed on the display surface Scn.

In addition, when the calculation unit 3 detects the position of the user's face and determines that the position of the user's face is outside the overlapping view range, the calculation unit 3 does not execute the process of outputting the coordinate information in step S9. You may do it. In this case, the display control unit 4 may determine that the movement of the user's hand is a predetermined gesture based on the movement of the user's hand detected by the calculation unit 3.

In addition, when the movement of the user's hand is the same, the display control unit 4 may determine that the gesture is different when the position of the user's hand is different. That is, the display control unit 4 may execute different processes depending on the position of the user's hand even if the detected movement (gesture) of the user's hand is the same.

In addition, the function of the “sensor unit” is realized by the sensor DS1 and the sensor control unit 2, for example.

[Other Embodiments]
In the above embodiment, the imaging unit 1 is arranged outside the display panel LCD as shown in FIG. 1, but the present invention is not limited to this. For example, the imaging unit 1 includes the housing of the display panel LCD. It may be arranged inside the body.

In the above embodiment, the orientation of the optical axis and the angle of view of the optical system of the imaging unit 1 are described as an example for convenience of explanation. You may change a corner | angular etc. from the thing of the said embodiment. For example, when the direction of the optical axis of the optical system of the imaging unit 1 has a predetermined inclination with respect to the z-axis, the coordinate acquisition device 100 considers the inclination of the optical axis of the optical system of the imaging unit 1. The calculation process may be executed.

In addition, part or all of the display device 1000 and the coordinate acquisition device 100 of the above embodiment may be realized as an integrated circuit (for example, an LSI, a system LSI, or the like).

Part or all of the processing of each functional block in the above embodiment may be realized by a program. And a part or all of the processing of each functional block of the above embodiment may be executed by a central processing unit (CPU) in the computer. A program for performing each processing is stored in a storage device such as a hard disk or ROM, and a central processing unit (CPU) reads the program from the ROM or RAM and executes it. Also good.

Further, each process of the above embodiment may be realized by hardware, or may be realized by software (including a case where it is realized together with an OS (operating system), middleware, or a predetermined library). Further, it may be realized by mixed processing of software and hardware. Further, it may be realized by mixed processing of software and hardware.

Further, the execution order of the processing methods in the above embodiment is not necessarily limited to the description of the above embodiment, and the execution order can be changed without departing from the gist of the invention.

A computer program that causes a computer to execute the above-described method and a computer-readable recording medium that records the program are included in the scope of the present invention. Here, examples of the computer-readable recording medium include a flexible disk, a hard disk, a CD-ROM, an MO, a DVD, a large-capacity DVD, a next-generation DVD, and a semiconductor memory.

The computer program is not limited to the one recorded on the recording medium, but may be transmitted via a telecommunication line, a wireless or wired communication line, a network represented by the Internet, or the like.

Further, in the above embodiment, there is a part in which only the main members necessary for the above embodiment are simplified among the constituent members. Therefore, it is possible to provide any constituent member that is not explicitly described in the above embodiment. Moreover, in the said embodiment and drawing, the dimension of each member has a part which does not represent an actual dimension, a dimension ratio, etc. faithfully.

The specific configuration of the present invention is not limited to the above-described embodiment, and various changes and modifications can be made without departing from the scope of the invention.

[Appendix]
The present invention can also be expressed as follows.

1st invention is a coordinate acquisition apparatus for acquiring the coordinate information on the display surface of a display apparatus, Comprising: An imaging part, a sensor part, and a calculating part are provided.

In this coordinate acquisition device, the position information of the user's viewpoint (the center point of both eyes) is acquired from the captured image acquired by the imaging unit, and further, the sensor proximity point (for example, the part closest to the display surface) ) Coordinate information is acquired. In this coordinate acquisition device, when the user views the display surface based on the position information of the user's viewpoint and the coordinate information of the user proximity point (portion closest to the display surface), the user proximity An overlapping field of view that is a range in which a point (for example, the tip of a user's finger or a pointing stick or a pen held by the user) and a display surface can be seen is specified. In this coordinate acquisition device, when the user proximity point is within the overlapping view range, the calculation unit calculates the coordinates (Xout, Yout) of the point where the straight line connecting the user viewpoint and the user proximity point intersects the display surface. Calculate and output. As described above, since this coordinate acquisition apparatus does not require a touch panel, the user proximity point (the position of the user's hand) is determined based on the position of the user's hand in the three-dimensional space even from a position away from the display surface. Etc.) can be appropriately specified (acquired) on the display surface.

The second invention is a display device comprising the coordinate acquisition device according to the first invention, a display panel, and a display control unit.

The display control unit is a display control unit that controls the display panel, and inputs the coordinate information of the user proximity point calculated by the calculation unit.

The arithmetic unit detects the position of the user's hand and face in the captured image acquired by the imaging unit, and determines that the distance between the hand and the face is equal to or less than a predetermined value, the coordinate information of the user proximity point Is not output to the display controller.

Thereby, in this display device, for example, it is possible to detect a movement of the user's hand as a gesture and suppress malfunctions when executing processing according to the detected gesture. In other words, in this display device, the processing as described above can appropriately prevent the display device from erroneously recognizing it as a predetermined gesture when the user unconsciously touches the face and malfunctioning. Can do.

The third invention is a display device comprising the coordinate acquisition device according to the first invention, a display panel, and a display control unit.

The calculation unit detects the position of the user's face in the captured image acquired by the imaging unit, and if the distance from the own device to the user's face is greater than a certain value, the coordinate information of the user proximity point is displayed on the display control unit. Do not output.

Thereby, in this display device, it is possible to appropriately recognize the movement of the user's hand at a position farther than a predetermined distance as a predetermined gesture, and to prevent erroneous operation.

The fourth invention is a display device comprising the coordinate acquisition device according to the first invention, a display panel, and a display control unit.

The display control unit determines that (1) the ratio between the distance from the display surface to the user proximity point and the distance from the user proximity point to the plane on which the user's eyes are present has changed, (2) If it is determined that the coordinates of the point where the straight line connecting the viewpoint that is the center point of the user's eyes and the user's proximity point intersects the display surface have not changed, it is determined that the user has clicked, and the display surface Then, a predetermined process associated with the display at the coordinates of the point where the straight line connecting the viewpoint and the user proximity point intersects the display surface is executed.

As a result, in this display device, the position of the user's viewpoint does not change, the distance between the user proximity point (for example, the user's finger) and the display device changes, and on the display surface corresponding to the user proximity point. When the coordinate position does not change, the display control unit can determine that the operation is a click operation, execute a predetermined process, and control so that, for example, a display corresponding to the process is performed on the display surface.

According to the present invention, the coordinates on the display screen of the display device can be appropriately specified based on the position of the user's hand in the three-dimensional space even from a remote position without the need for a touch panel. Since the coordinate acquisition device and the display device can be realized, it is useful in the display device related industrial field and can be implemented in this field.

1000 Display device 100 Coordinate acquisition device DS1 Sensor 1 Imaging unit 2 Sensor control unit 3 Calculation unit 4 Display control unit LCD Display panel Scn Display surface

Claims

A coordinate acquisition device for acquiring coordinate information on a display surface of a display device,
An imaging unit that acquires a captured image by imaging a subject with a predetermined angle of view;
A sensor unit for obtaining a user proximity point which is a position of a user's hand or an object held by the user and a distance to the user proximity point;
(1) In the captured image acquired by the imaging unit, the user's eyes are detected, based on the position of the detected both eyes on the captured image, the distance between both eyes, and the angle of view. , Calculate the distance from your device to the plane containing the user's eyes,
(2) Based on the user proximity point acquired by the sensor unit and the distance from the user device to the user proximity point, and the horizontal width and the vertical width of the display surface of the display device, Calculating an overlapping view range that is a range in which the user proximity point and the display surface of the display device appear to overlap each other from the viewpoint of
(3) When it is determined that the user proximity point is within the overlapping view range, a calculation unit that calculates and outputs coordinate information of the user proximity point;
A coordinate acquisition device comprising:
A coordinate acquisition device according to claim 1;
A display panel;
A display control unit for controlling the display panel, wherein the display control unit inputs coordinate information of the user proximity point calculated by the calculation unit;
A display device comprising:
The computing unit is
In the captured image acquired by the imaging unit, when the position of the user's hand and face is detected and it is determined that the distance between the hand and the face is equal to or less than a predetermined value, the coordinate information of the user proximity point is displayed. Do not output to the control unit,
Display device.
A coordinate acquisition device according to claim 1;
A display panel;
A display control unit for controlling the display panel, wherein the display control unit inputs coordinate information of the user proximity point calculated by the calculation unit;
A display device comprising:
The computing unit is
In the captured image acquired by the imaging unit, the position of the user's face is detected, and when the distance from the own device to the user's face is larger than a certain value, the coordinate information of the user proximity point is sent to the display control unit. Do not output,
Display device.
A coordinate acquisition device according to claim 1;
A display panel;
A display control unit for controlling the display panel, wherein the display control unit inputs coordinate information of the user proximity point calculated by the calculation unit;
A display device comprising:
The display control unit
(1) When it is determined that the ratio between the distance from the display surface to the user proximity point and the distance from the user proximity point to the plane on which the user's eyes are located has changed, (2) When it is determined that the coordinates of the point where the straight line connecting the viewpoint that is the center point of both eyes and the user proximity point intersects the display surface has not changed, it is determined that the click operation is performed by the user, and the display surface A predetermined process associated with a display at the coordinates of a point where a straight line connecting the viewpoint and the user proximity point intersects the display surface,
Display device.