KR20220005677A - Display device including optical sensor and measurement method for location of optical sensor - Google Patents

Abstract

An objective of the present invention is to provide a display device including an optical sensor and a position measurement method of an optical sensor, which can measure the position change amount and change direction of an optical sensor in decimal units as wells as in integer units based on sensor pixels. According to the present invention, the display device including an optical sensor comprises: a display panel including display pixels; and an optical sensor including a sensor controller and sensor pixels, wherein the sensor pixels are positioned on a first area of the display panel. The display pixels display a pattern including a plurality of markers on the first area, and generate a first image by photographing the pattern. The sensor controller divides the first image into sub-images corresponding to the markers. The sensor controller calculates first representative coordinates resulting from summing integer parts and decimal parts for the sub-images. The unit of the integer parts is the number of the sensor pixels. Each of the integer parts and the decimal parts independently includes a first dimensional value in a first direction and a second dimensional value in a second direction perpendicular to the first direction.

Description

DISPLAY DEVICE INCLUDING OPTICAL SENSOR AND MEASUREMENT METHOD FOR LOCATION OF OPTICAL SENSOR

The present invention relates to a display device including an optical sensor and a method for measuring a position of the optical sensor.

With the development of information technology, the importance of a display device, which is a connection medium between a user and information, has been highlighted. In response to this, the use of display devices such as a liquid crystal display device, an organic light emitting display device, and a plasma display device is increasing.

According to consumer demand, the display device may include various sensors such as a touch sensor, a fingerprint sensor, an illuminance sensor, and a camera. In particular, the fingerprint sensor may be attached to the rear surface of the display panel to recognize the ridges and valleys of the fingerprint based on information of a carrier (eg, light or ultrasound) passing through the display panel.

In this case, since the carrier includes different background information according to the internal layout of the transparent display panel, initial calibration is required to remove the background information and leave only the fingerprint information.

However, if the position of the fingerprint sensor is unintentionally changed after the initial calibration is performed, the position of the fingerprint sensor must be updated to properly use the calibration data. In addition to the fingerprint sensor, other sensors attached to the rear surface of the display panel may have the same problem.

A technical problem to be solved is to provide a display device including an optical sensor capable of measuring a positional displacement amount and a displacement direction of an optical sensor in integer units as well as decimal units based on sensor pixels and a method of measuring the position of the optical sensor. have.

According to an embodiment of the present invention, a display device including an optical sensor includes: a display panel including display pixels; and an optical sensor comprising a sensor controller and sensor pixels, wherein the sensor pixels are positioned on a first area of the display panel, wherein the display pixels display a pattern including a plurality of markers in the first area; The sensor pixels generate a first image by capturing the pattern, the sensor controller divides the first image into sub-images corresponding to the markers, and the sensor controller includes an integer part and an integer part for the sub-images. First representative coordinates obtained by summing the fractional part are calculated, the unit of the integer part is the number of the sensor pixels, and each of the integer part and the fractional part is a first dimension value for a first direction and a second value orthogonal to the first direction independently contain a second dimension value for the direction.

The sensor controller determines, for each of the sub-images, a first dimension value corresponding to a maximum value among the average brightnesses of the first dimension values as the first dimension value of the integer part, and adds the second dimension values to the second dimension values. A second dimension value corresponding to the maximum value among the average brightnesses of the corresponding values may be determined as the second dimension value of the integer part.

The sensor controller may determine, for each of the sub-images, the first dimension value of the fractional part by comparing current brightnesses of the first observation areas with previous brightnesses.

Each of the first observation areas may be longer in the second direction than the first direction, and the first observation areas may be positioned adjacent to each other in the first direction.

Each of the sub-images may include a first boundary whose brightness is rapidly changed in the first direction, and at least one of the first observation areas may be positioned to overlap the first boundary.

The sensor controller is configured to, when the current brightnesses of the first observation areas are smaller than previous brightnesses, determine that the first dimension value of the fractional part has a first sign, and the sensor controller is configured to: When the values are greater than the previous brightnesses, it may be determined that the first dimension value of the fractional part has a second sign opposite to the first sign.

The sensor controller is configured to set the first dimension value of the fractional part to 0 when current brightnesses of the first observation areas are smaller than previous brightnesses and the current brightnesses of the first observation areas have uniform difference values from previous brightnesses. can be determined as

The sensor controller may determine, for each of the sub-images, a second dimension value of the fractional part by comparing current brightnesses of second observation areas with previous brightnesses.

Each of the second observation areas may have a shape longer in the first direction than in the second direction, and the second observation areas may be positioned adjacent to each other in the second direction.

Each of the sub-images may include a second boundary whose brightness is rapidly changed in the second direction, and at least one of the second observation areas may be positioned to overlap the second boundary.

The sensor controller is configured to determine that the second dimension value of the fractional part has the first sign when current brightnesses of the second viewing areas are smaller than previous brightnesses, and the sensor controller is configured to: When the brightnesses are greater than the previous brightnesses, it may be determined that the second dimension value of the fractional part has the second sign.

The sensor controller may calculate the inclination of the optical sensor using second representative coordinates and the first representative coordinates of a second image generated at a time different from the first image.

wherein the sensor controller calculates at least one first inclination with respect to the first direction and at least one second inclination with respect to the second direction, and the sensor controller is configured to: When all of the second inclinations are the same, a third inclination in the first direction and in a third direction orthogonal to the second direction is determined as 0, and at least one of the at least one first inclination and the at least one second inclination is determined. When one is different, it may be determined that the third slope is non-zero.

According to an exemplary embodiment, a method for measuring a position of an optical sensor includes a method for measuring a position of an optical sensor positioned on a first area of a display panel, wherein the method for measuring the position of the optical sensor includes: displaying a pattern including a plurality of markers; generating a first image by imaging the pattern by the sensor pixels of the optical sensor; dividing, by the sensor controller of the optical sensor, the first image into sub-images corresponding to the markers; and calculating, by the sensor controller, first representative coordinates obtained by summing an integer part and a decimal part with respect to the sub-images, wherein a unit of the integer part is the number of the sensor pixels, and each of the integer part and the decimal part is a first and independently includes a first dimension value for a direction and a second dimension value for a second direction orthogonal to the first direction.

In the position measuring method, the sensor controller determines, as the first dimension value of the integer part, a first dimension value corresponding to a maximum value among average brightnesses of first dimension values for each of the sub-images, , determining a second dimension value corresponding to a maximum value among average brightnesses of the second dimension values as the second dimension value of the integer part.

The position measuring method may further include, for each of the sub-images, determining, by the sensor controller, a first dimension value of the fractional part by comparing current brightnesses of first observation areas with previous brightnesses. have.

Each of the first observation areas has a shape longer in the second direction than the first direction, the first observation areas are located adjacent to each other in the first direction, and each of the sub-images is based on the first direction and a first boundary whose brightness is rapidly changed as follows, and at least one of the first observation areas may be positioned to overlap the first boundary.

In the position measuring method, the sensor controller determines that the first dimension value of the fractional part has a first sign when the current brightnesses of the first observation areas are smaller than previous brightnesses, and the current brightness of the first observation areas The method may further include determining that the first dimension value of the fractional part has a second sign opposite to the first sign when the values are greater than the previous brightnesses.

The method for measuring the position, wherein the sensor controller, when the current brightnesses of the first observation areas are smaller than previous brightnesses and the current brightnesses of the first observation areas have uniform difference values from the previous brightnesses, the The method may further include determining the one-dimensional value to be 0.

The position measuring method may further include, for each of the sub-images, determining, by the sensor controller, a second dimension value of the fractional part by comparing current brightnesses of second observation areas with previous brightnesses. have.

The display device including the optical sensor and the method for measuring the position of the optical sensor according to the present invention can measure the position displacement amount and the displacement direction of the optical sensor in integer units as well as decimal units based on sensor pixels.

1 to 3 are diagrams for explaining a display device according to an exemplary embodiment.
4 is a view for explaining an image captured by an optical sensor according to an embodiment of the present invention.
5 is a diagram for explaining a method of calculating an integer part of representative coordinates of a sub-image according to an embodiment of the present invention.
6 to 9 are diagrams for explaining a method of calculating a fractional part of representative coordinates of a sub-image according to an embodiment of the present invention.
10 is a view for explaining a method of calculating the inclination of the optical sensor according to an embodiment of the present invention.

Hereinafter, various embodiments of the present invention will be described in detail with reference to the accompanying drawings so that those of ordinary skill in the art to which the present invention pertains can easily implement them. The present invention may be embodied in several different forms and is not limited to the embodiments described herein.

In order to clearly explain the present invention, parts irrelevant to the description are omitted, and the same reference numerals are assigned to the same or similar components throughout the specification. Accordingly, the reference numerals described above may be used in other drawings as well.

In addition, since the size and thickness of each component shown in the drawings are arbitrarily indicated for convenience of description, the present invention is not necessarily limited to the illustrated bar. In order to clearly express various layers and regions in the drawings, the thickness may be exaggerated.

1 to 3 are diagrams for explaining a display device according to an exemplary embodiment. FIG. 2 is an exemplary cross-sectional view taken along line A-A' of FIG. 1 .

The first direction (X), the second direction (Y), and the third direction (Z) may be directions orthogonal to each other.

1 to 3 , the display device 9 may include a display panel 10 and an optical sensor 20 .

The display panel 10 may include display pixels DPX and a display controller 101 . The display pixels DPX may be arranged on a plane defined in the first direction (X) and the second direction (Y). The display pixels DPX are not necessarily arranged in a flat shape, but may also be arranged in a curved shape. Although not shown, for example, the side surface of the display panel 10 may be curved, and the display pixels DPX disposed on the side surface of the display panel 10 may be arranged in a curved shape. The display pixels DPX may display an image by emitting light in the third direction Z. When the user looks at the first surface 10a of the display panel 10 , the displayed image can be viewed. The display controller 101 may receive RGB data, a control signal, and the like from an external processor and supply necessary data voltages and control signals to the display pixels DPX. A detailed configuration of the display panel 10 may be in accordance with the prior art.

The optical sensor 20 may include sensor pixels SPX, a sensor controller 201 , and a sensor memory 202 . The optical sensor 20 may be positioned on the first area AR1 of the display panel 10 . For example, the optical sensor 20 may be attached to the second surface 10b of the display panel 10 . The optical sensor 20 is a sensor using light as a carrier, and may be a fingerprint sensor, a camera, an illuminance sensor, or a proximity sensor. However, for convenience of description, it is assumed that the optical sensor 20 is a fingerprint sensor hereinafter.

The sensor pixels SPX may be arranged on a plane defined in the first direction X and the second direction Y. The sensor pixels SPX may be configured according to a conventional optical sensor technology such as a CMOS sensor. The sensor pixels SPX may transmit an image through the first area AR1 of the display panel 10 to generate an image.

The sensor memory 202 may include position data of the sensor pixels SPX and calibration data based on the position data. Since the layout of the display panel 10 through which the carrier passes is different according to the positions of the sensor pixels SPX, it is necessary to generate the calibration data at least once before shipping the product. Additionally, the calibration data may reflect a characteristic difference (dispersion) of each of the sensor pixels SPX.

The sensor controller 201 may generate a corrected image by applying calibration data to the image generated by the sensor pixels SPX. For example, if the image generated by the sensor pixels SPX simultaneously includes background information (eg, the layout of the display panel 10 ) and a fingerprint shape, the corrected image only includes the fingerprint shape from which the background information is removed. may include Accordingly, user authentication may be performed using the obtained fingerprint shape.

4 is a view for explaining an image captured by an optical sensor according to an embodiment of the present invention.

In order to check whether the position of the optical sensor 20 has changed, the display device 9 may periodically measure the position of the optical sensor 20 . In addition, the position measurement of the optical sensor 20 may be performed aperiodically by the user.

First, the display pixels DPX may display a pattern including a plurality of markers MK1 , MK2 , MK3 , and MK4 in the first area AR1 . The sensor pixels SPX may generate an image ARi1 by capturing a pattern.

Each of the markers MK1 , MK2 , MK3 , and MK4 may have a higher brightness than the background of the image ARi1 . For example, each of the markers MK1 , MK2 , MK3 , and MK4 may include a cross shape. For example, each of the markers MK1 , MK2 , MK3 , and MK4 may include box figures of different grayscales around the cross figure. However, the shapes of the markers MK1, MK2, MK3, and MK4 may vary according to embodiments, and the present embodiment is not limited by the shapes of the markers MK1, MK2, MK3, and MK4 of FIG. 4 . . The markers MK1 , MK2 , MK3 , and MK4 may be disposed at points of interest whose coordinates are to be known.

The sensor controller 201 may divide the image ARi1 into sub-images Q1 , Q2 , Q3 , and Q4 corresponding to the markers MK1 , MK2 , MK3 , and MK4 . For example, the image ARi1 may be a rectangle, and the markers MK1 , MK2 , MK3 , and MK4 may be located near each vertex of the rectangle. For example, the sub-images Q1 , Q2 , Q3 , and Q4 may be quadrants of the image ARi1 . The sensor controller 201 may calculate representative coordinates obtained by summing the integer part and the decimal part with respect to the sub images Q1 , Q2 , Q3 , and Q4 . In this case, the unit of the integer part may be the number of sensor pixels SPX. For example, when the difference between the previous representative coordinates and the current representative coordinates with respect to the first direction X is 3, the optical sensor 20 moves in the first direction by a distance corresponding to the width of three sensor pixels SPX. (X) means it has moved.

In this case, each of the integer part and the fractional part may independently include a first dimension value for the first direction (X) and a second dimension value for a second direction (Y) orthogonal to the first direction (X). That is, after calculating the integer part and the fractional part separately, the sensor controller 201 calculates the first dimension value of the representative coordinate by summing the first dimension value of the integer part and the first dimension value of the decimal part, and the second dimension value of the integer part and the second dimension value of the fractional part may be summed to calculate a second dimension value of the representative coordinates. The first dimension value may be a coordinate value of the x-axis, and the second dimension value may be a coordinate value of the y-axis.

According to the present embodiment, not only the integer part but also the transition direction of the decimal part can be measured. Accordingly, even when the optical sensor 20 moves minutely in units of decimals instead of integers, the displacement direction and the amount of displacement of the optical sensor 20 can be accurately measured.

Hereinafter, a method of calculating the representative coordinates of the sub-image Q1 will be described. Representative coordinates of the other sub-images Q2, Q3, and Q4 may also be calculated in the same way, and thus a redundant description will be omitted.

5 is a diagram for explaining a method of calculating an integer part of representative coordinates of a sub-image according to an embodiment of the present invention.

The sensor controller 201 sets the first dimension value corresponding to the maximum value among the average brightness values AVGX of the first dimension values for each of the sub-images Q1, Q2, Q3, and Q4 as the first integer part. It can be determined by the dimension value (xi1). Also, for each of the sub-images Q1, Q2, Q3, and Q4, the sensor controller 201 sets the second dimension value corresponding to the maximum value among the average brightness values AVGY of the second dimension values to the integer part. It may be determined as the second dimension value yi1.

For example, the average brightness AVGX of sensing values (brightnesses) of the sensor pixels SPX at the same position with respect to the first direction X may correspond to the height of the graph. For example, since sensing values of the first sensor pixels SPX in the first direction X correspond to the background of the image ARi1 , the average brightness AVGX may be the smallest. Meanwhile, since the sensed values of the sensor pixels SPX located at the xi1 th position with respect to the first direction X include a cross-shaped portion extending in the second direction Y, the average brightness AVGX is the maximum. can

For example, the average brightness AVGY of the sensed values (brightnesses) of the sensor pixels SPX at the same position in the second direction Y may correspond to the height of the graph. For example, the sensed values of the first sensor pixels SPX with respect to the second direction Y correspond to the background of the image ARi1 , so that the average brightness AVGY may be the smallest. Meanwhile, since the sensed values of the sensor pixels SPX located at the yi1 th position with respect to the second direction Y include a cross-shaped portion extending in the first direction X, the average brightness AVGY is the maximum. can

6 to 9 are diagrams for explaining a method of calculating a fractional part of representative coordinates of a sub-image according to an embodiment of the present invention.

Referring to FIG. 6 , the sensor controller 201 compares the current brightnesses of the first observation areas PVB, PVR, and PVD with previous brightnesses for each of the sub-images Q1, Q2, Q3, and Q4. The comparison may determine a first dimension value of the fractional part. The current brightness means brightnesses of the first observation areas PVB, PVR, and PVD in the image ARi1 captured at the current time point. The previous brightness refers to brightnesses of the first observation areas PVB, PVR, and PVD in an image captured at a past time point.

Each of the first observation areas PVB, PVR, and PVD may have a shape longer in the second direction Y than in the first direction X. For example, each of the first viewing areas PVB, PVR, and PVD may be a rectangle longer in the second direction (Y) than in the first direction (X). The sizes and shapes of the first observation areas PVB, PVR, and PVD may be the same.

The first observation areas PVB, PVR, and PVD may be located adjacent to each other in the first direction X. For example, the first observation areas PVB, PVR, and PVD may contact each other in the first direction X.

Each of the sub-images Q1 , Q2 , Q3 , and Q4 may include a first boundary EDG1 whose brightness is rapidly changed with respect to the first direction X. For example, the sub-image Q1 may include a first boundary EDG1 whose brightness is rapidly changed with respect to the first direction X. In the embodiment of FIG. 6 , the first boundary EDG1 of the white box figure positioned in the first direction (X) and the second direction (Y) from the cross figure is used.

At least one of the first observation areas PVB, PVR, and PVD may be positioned to overlap the first boundary EDG1 . For example, the first viewing area PVR may be positioned to overlap the first boundary EDG1 . Based on the second direction Y, the length of the first viewing area PVR may be the same as the length of the first boundary EDG1 . The first viewing area PVB may be positioned to overlap the white box. The first observation area PVD may be positioned to overlap the background.

Referring to FIG. 7 , a graph XGR for previous brightnesses of the first viewing areas PVB, PVR, and PVD and current brightnesses of the first viewing areas PVB, PVR, and PVD are different The graphs (XGP, XGM) of are shown by way of example. Each of the graphs XGR, XGP, and XGM is a straight line connecting the brightnesses of the first observation areas PVB, PVR, and PVD.

When the current brightnesses of the first observation areas PVB, PVR, and PVD are smaller than previous brightnesses, the sensor controller 201 may determine that the first dimension value of the fractional part has the first sign. For example, when current brightnesses of the first observation areas PVB, PVR, and PVD correspond to the graph XGP, the current brightnesses are smaller than previous brightnesses of the graph XGR. In this case, since the sensor pixels SPX move in the first direction X with respect to the display panel 10 , the first dimension value of the fractional part may have a (+) sign. The amount of displacement in the first direction X may be calculated by various algorithms according to positions, sizes, and shapes of the first observation areas PVB, PVR, and PVD. For example, in the embodiment of FIG. 7 , the first dimension value of the decimal part may be (+)0.2. This is a shift amount corresponding to 1/5 the width of the sensor pixel. Accordingly, the first dimension value of the representative coordinates of the sub-image Q1 may be xi1+0.2.

If the current brightnesses of the first observation areas PVB, PVR, and PVD are greater than previous brightnesses, the sensor controller 201 may determine that the first dimension value of the fractional part has a second sign opposite to the first sign. have. For example, when current brightnesses of the first observation areas PVB, PVR, and PVD correspond to the graph XGM, the current brightnesses are greater than previous brightnesses of the graph XGR. In this case, since the sensor pixels SPX move in a direction opposite to the first direction X with respect to the display panel 10 , the first dimension value of the fractional part may have a negative sign. The amount of displacement in the first direction X may be calculated by various algorithms according to positions, sizes, and shapes of the first observation areas PVB, PVR, and PVD. For example, in the embodiment of FIG. 7 , the first dimension value of the fractional part may be (−)0.2. This is a shift amount corresponding to 1/5 the width of the sensor pixel. Accordingly, the first dimension value of the representative coordinates of the sub-image Q1 may be xi1-0.2.

According to this embodiment, not only the integer part but also the mutation direction and the mutation amount of the decimal part can be known. Accordingly, when the displacement amount of the integer part of the optical sensor 20 is 0 and the displacement amount of the decimal part is not 0, that is, even when the optical sensor 20 moves minutely, the exact displacement direction can be known.

Representative coordinates of the sub-images Q1 , Q2 , Q3 , and Q4 calculated according to the present embodiment may be stored in the sensor memory 202 . Accordingly, the calibration data may be used for image correction based on the corrected representative coordinates.

Referring to FIG. 8 , a graph (XGR) of previous brightnesses of the first viewing areas (PVB, PVR, PVD) and a graph (XGD) of current brightnesses of the first viewing areas (PVB, PVR, PVD) ) is shown as an example. Each of the graphs XGR and XGD is a straight line connecting the brightnesses of the first observation areas PVB, PVR, and PVD.

The sensor controller 201 determines that current brightnesses of the first viewing areas PVB, PVR, and PVD are smaller than previous brightnesses, and the current brightnesses of the first viewing areas PVB, PVR, and PVD are uniform with previous brightnesses. In the case of having one difference value, the first dimension value of the fractional part may be determined to be 0. The range of the uniform difference values may be variously determined according to the location, size, and shape of the first observation areas PVB, PVR, and PVD.

For example, the current brightnesses of the first viewing areas PVB, PVR, and PVD are smaller than previous brightnesses, and the current brightnesses of the first viewing areas PVB, PVR, and PVD are uniformly different from previous brightnesses. , the graph XGD of current brightnesses may have substantially the same shape as the graph XGR, but may have a shape shifted downward based on the brightness axis. In this case, it may be a case that the display pixels DPX are deteriorated and the brightness is decreased, rather than that the sensor pixels SPX are not moved. That is, in both the graph XGP of FIG. 7 and the graph XGD of FIG. 8 , the current brightness of the first observation area PVR decreased compared to the previous brightness, but in the case of the graph XGP of FIG. 7 , it is determined that a variation has occurred and, in the case of the graph XGD of FIG. 8 , it may be determined that no mutation has occurred.

Therefore, according to the present embodiment, by observing the plurality of first observation areas PVB, PVR, and PVD to measure the relative variance, when only one first observation area PVR is observed and it is erroneously determined that the variance has occurred can prevent

Referring to FIG. 9 , the sensor controller 201 compares the current brightnesses of the second observation areas PHB, PHR, and PHD with previous brightnesses for each of the sub-images Q1, Q2, Q3, and Q4. The second dimension value of the fractional part may be determined by comparison. The current brightness means brightnesses of the second observation areas PHB, PHR, and PHD in the image ARi1 captured at the current time point. The previous brightness refers to brightnesses of the second observation areas PHB, PHR, and PHD in an image captured at a past time point.

Each of the second viewing areas PHB, PHR, and PHD may be longer in the first direction (X) than the second direction (Y). For example, each of the second viewing areas PHB, PHR, and PHD may be a rectangle longer in the first direction (X) than in the second direction (Y). The sizes and shapes of the second viewing areas PHB, PHR, and PHD may be the same.

The second viewing areas PHB, PHR, and PHD may be located adjacent to each other in the second direction Y. For example, the second viewing areas PHB, PHR, and PHD may contact each other in the second direction Y.

Each of the sub-images Q1 , Q2 , Q3 , and Q4 may include a second boundary EDG2 whose brightness is rapidly changed in the second direction Y. For example, the sub-image Q1 may include a second boundary EDG2 whose brightness is rapidly changed in the second direction Y. In the embodiment of FIG. 9 , the second boundary EDG2 of the white box figure positioned in the first direction (X) and the second direction (Y) from the cross figure is used.

At least one of the second observation areas PHB, PHR, and PHD may be positioned to overlap the second boundary EDG2 . For example, the second observation area PHR may be positioned to overlap the second boundary EDG2 . Based on the first direction X, the length of the second observation area PHR may be the same as the length of the second boundary EDG2 . The second viewing area PHB may be positioned to overlap the white box. The second viewing area PHD may be positioned to overlap the background.

When the current brightnesses of the second observation areas PHB, PHR, and PHD are smaller than previous brightnesses, the sensor controller 201 may determine that the second dimension value of the fractional part has the first sign. Also, when the current brightnesses of the second observation areas PHB, PHR, and PHD are greater than previous brightnesses, the sensor controller 201 may determine that the second dimension value of the fractional part has the second sign. Since this is substantially the same as the description of FIG. 7 , a redundant description thereof will be omitted.

The sensor controller 201 determines that current brightnesses of the second viewing areas PHB, PHR, and PHD are smaller than previous brightnesses, and the current brightnesses of the second viewing areas PHB, PHR, and PHD are uniform with previous brightnesses. In the case of having one difference value, the second dimension value of the fractional part may be determined to be 0. Since this is substantially the same as that of FIG. 8 , a duplicate description will be omitted.

10 is a view for explaining a method of calculating the inclination of the optical sensor according to an embodiment of the present invention.

Referring to FIG. 10 , a first image ARi1 generated at a first viewpoint and a second image ARi1 ′ generated at a second viewpoint other than the first viewpoint are illustrated. For example, the second time point may be a measurement time point following the first time point. Based on the sub-images Q1, Q2, Q3, Q4 of the first image ARi1, first representative coordinates ((x1, y1), (x2, y2), (x3, y3), (x4, y4) )) is calculated, and the second representative coordinates ((x1', y1'), (x2', y2'), (x3', y3'), (x4', y4')) is assumed to be calculated. For convenience of description, it is assumed that the first representative coordinates ((x1, y1)) and the second representative coordinates ((x1', y1')) are the same. For convenience of description, the first image ARi1 is a rectangle connecting the four first representative coordinates ((x1, y1), (x2, y2), (x3, y3), (x4, y4)). assume that For convenience of description, the second image ARi1' has four second representative coordinates ((x1', y1'), (x2', y2'), (x3', y3'), (x4', Assume that it is a rectangle connecting y4')).

The sensor controller 201 determines the second representative coordinates ((x1', y1'), (x2', y2'), (x3) of the second image ARi1' generated at a different time point than the first image ARi1. ', y3'), (x4', y4')) and the first representative coordinates ((x1, y1), (x2, y2), (x3, y3), (x4, y4)) of the optical sensor The slope (ag12, ag34, ag13, ag24) of (20) can be calculated. For example, the sensor controller 201 may generate at least one first inclination ag12 and ag34 in the first direction X and at least one second inclination ag13 and ag24 in the second direction Y. can be calculated.

For example, the gradient ag12 may be calculated as in Equation 1 below using the first representative coordinates ((x1, y1)) and the second representative coordinates ((x2', y2')).

[Equation 1]

Since the other gradients ag34, ag13, and ag24 can be calculated in substantially the same way, a redundant description will be omitted. Accordingly, according to the present embodiment, the amount of inclination shift and the shift direction of the optical sensor 20 in the plane can be calculated.

The sensor controller 201 is orthogonal to the first direction (X) and the second direction (Y) when the at least one first inclination (ag12, ag34) and the at least one second inclination (ag13, ag24) are all the same A third inclination in the third direction Z may be determined to be 0. For example, when the inclinations ag12, ag34, ag13, and ag24 are all the same, the optical sensor 20 is inclined only on the plane (the first direction (X) and the second direction (Y)), and the vertical direction ( In the third direction (Z)), it may be determined that there is no inclination.

When at least one of the first inclinations ag12 and ag34 and the at least one second inclination ag13 and ag24 is different, the sensor controller 201 may determine that the third inclination is not zero. For example, when at least two of the inclinations ag12, ag34, ag13, and ag24 are different from each other, the optical sensor 20 may determine that the inclination is in the vertical direction as well as in the planar direction.

Accordingly, according to the present embodiment, it is possible to determine whether or not the optical sensor 20 is inclined in the vertical direction as well as the inclination in the plane direction.

The drawings and detailed description of the described invention referenced so far are merely exemplary of the present invention, which are only used for the purpose of explaining the present invention, and are used to limit the meaning or limit the scope of the present invention described in the claims. it is not Therefore, those skilled in the art will understand that various modifications and equivalent other embodiments are possible therefrom. Accordingly, the true technical protection scope of the present invention should be defined by the technical spirit of the appended claims.

9: display device
10: display panel
101: display controller
DPX: display pixels
20: optical sensor
201: sensor controller
SPX: sensor pixels
202: sensor memory

Claims (20)

a display panel including display pixels; and
an optical sensor comprising a sensor controller and sensor pixels, wherein the sensor pixels are located on a first area of the display panel;
the display pixels display a pattern including a plurality of markers in the first area;
The sensor pixels generate a first image by imaging the pattern,
The sensor controller divides the first image into sub-images corresponding to the markers,
The sensor controller calculates first representative coordinates obtained by summing an integer part and a decimal part with respect to the sub-images,
The unit of the integer part is the number of the sensor pixels,
Each of the integer part and the fractional part independently includes a first dimension value for a first direction and a second dimension value for a second direction orthogonal to the first direction,
A display device including an optical sensor. According to claim 1,
The sensor controller, for each of the sub-images, determines a first dimension value corresponding to a maximum value among the average brightnesses of the first dimension values as the first dimension value of the integer part, and adds the second dimension values to the second dimension values. determining a second dimension value corresponding to the maximum value among the average brightness values for the integer part as the second dimension value of the integer part;
A display device including an optical sensor. According to claim 1,
the sensor controller, for each of the sub-images, compares current brightnesses of first viewing areas with previous brightnesses to determine a first dimension value of the fractional part;
A display device including an optical sensor. 4. The method of claim 3,
Each of the first observation areas is longer in the second direction than in the first direction,
the first observation areas are located adjacent to each other in the first direction;
A display device including an optical sensor. 5. The method of claim 4,
Each of the sub-images includes a first boundary whose brightness is rapidly changed with respect to the first direction,
at least one of the first viewing areas is located overlapping the first boundary;
A display device including an optical sensor. 6. The method of claim 5,
the sensor controller determines that the first dimensional value of the fractional part has a first sign when current brightnesses of the first observation areas are smaller than previous brightnesses;
the sensor controller is configured to determine that, when current brightnesses of the first observation areas are greater than previous brightnesses, the first dimension value of the fractional part has a second sign opposite to the first sign;
A display device including an optical sensor. 7. The method of claim 6,
The sensor controller is configured to set the first dimension value of the fractional part to 0 when current brightnesses of the first observation areas are smaller than previous brightnesses and the current brightnesses of the first observation areas have uniform difference values from previous brightnesses. to decide,
A display device including an optical sensor. 7. The method of claim 6,
the sensor controller, for each of the sub-images, compares current brightnesses of second viewing areas with previous brightnesses to determine a second dimension value of the fractional part;
A display device including an optical sensor. 9. The method of claim 8,
Each of the second observation areas is longer in the first direction than in the second direction,
the second observation areas are located adjacent to each other in the second direction;
A display device including an optical sensor. 10. The method of claim 9,
Each of the sub-images includes a second boundary whose brightness is rapidly changed with respect to the second direction,
at least one of the second viewing areas is located overlapping the second boundary;
A display device including an optical sensor. 11. The method of claim 10,
the sensor controller determines that the second-dimensional value of the fractional part has the first sign when current brightnesses of the second observation areas are smaller than previous brightnesses;
the sensor controller is configured to determine that the second dimension value of the fractional part has the second sign when current brightnesses of the second observation areas are greater than previous brightnesses;
A display device including an optical sensor. According to claim 1,
wherein the sensor controller calculates the inclination of the optical sensor using second representative coordinates and the first representative coordinates of a second image generated at a time different from the first image,
A display device including an optical sensor. 13. The method of claim 12,
wherein the sensor controller calculates at least one first tilt with respect to the first direction and at least one second tilt with respect to the second direction,
When the at least one first inclination and the at least one second inclination are all the same, the sensor controller determines a third inclination in a third direction orthogonal to the first direction and the second direction to be 0;
determining that the third slope is non-zero when at least one of the at least one first slope and the at least one second slope is different,
A display device including an optical sensor. A method for measuring a position of an optical sensor positioned on a first area of a display panel, the method comprising:
displaying, by a display panel, a pattern including a plurality of markers on the first area;
generating a first image by imaging the pattern by the sensor pixels of the optical sensor;
dividing, by the sensor controller of the optical sensor, the first image into sub-images corresponding to the markers; and
calculating, by the sensor controller, first representative coordinates obtained by summing an integer part and a decimal part for the sub-images;
The unit of the integer part is the number of the sensor pixels,
Each of the integer part and the fractional part independently includes a first dimension value for a first direction and a second dimension value for a second direction orthogonal to the first direction,
How to measure the position of an optical sensor. 15. The method of claim 14,
The sensor controller determines, as the first dimension value of the integer part, a first dimension value corresponding to a maximum value among average brightnesses of first dimension values for each of the sub-images, The method further comprising the step of determining a second dimension value corresponding to the maximum value among the average brightness values of the integer part as the second dimension value of the integer part;
How to measure the position of an optical sensor. 15. The method of claim 14,
determining, by the sensor controller, for each of the sub-images, a first dimension value of the fractional part by comparing current brightnesses of first viewing areas with previous brightnesses,
How to measure the position of an optical sensor. 17. The method of claim 16,
Each of the first observation areas is longer in the second direction than in the first direction,
The first observation areas are located adjacent to each other in the first direction,
Each of the sub-images includes a first boundary whose brightness is rapidly changed with respect to the first direction,
at least one of the first viewing areas is located overlapping the first boundary;
How to measure the position of an optical sensor. 18. The method of claim 17,
the sensor controller determines that the first dimensional value of the fractional part has a first sign when current brightnesses of the first observation areas are smaller than previous brightnesses, and the current brightnesses of the first observation areas are greater than previous brightnesses determining that the first dimension value of the fractional part has a second sign opposite to the first sign if
How to measure the position of an optical sensor. 19. The method of claim 18,
The sensor controller sets the first dimension value of the fractional part to 0 when current brightnesses of the first observation areas are smaller than previous brightnesses and the current brightnesses of the first observation areas have uniform difference values from previous brightnesses. further comprising the step of determining
How to measure the position of an optical sensor. 19. The method of claim 18,
determining, by the sensor controller, for each of the sub-images, a second dimension value of the fractional part by comparing current brightnesses of second viewing areas with previous brightnesses,
How to measure the position of an optical sensor.

Images