KR101407290B1 - Display apparatus having multi-touch recognizing function and driving method thereof - Google Patents

Abstract

The present invention relates to a display device having a multi-touch recognition function for preventing a malfunction by removing shadows and a driving method thereof.

This display device comprises a display element; A plurality of cameras provided in the vicinity of the edge of the display element; A reference value that is a function of an average value and a standard deviation value of a background image captured by the cameras and stored in advance is compared with a currently sensed touch image by the cameras to obtain a difference image, A touch control circuit for calculating the position of the touch object; And a display driving circuit for displaying an image of the touch object on the display device.

Description

DISPLAY APPARATUS HAVING MULTI-TOUCH RECOGNIZING FUNCTION AND DRIVING METHOD THEREOF BACKGROUND OF THE INVENTION 1. Field of the Invention [0001]

BRIEF DESCRIPTION OF THE DRAWINGS FIG. 1 schematically shows a display device having a multi-touch recognition function according to an embodiment of the present invention; FIG.

Fig. 2 is a detailed view of the display device shown in Fig. 1. Fig.

Fig. 3 is a circuit diagram equivalently showing a part of the pixel array shown in Fig. 2. Fig.

4 is a flow chart showing step-by-step control procedures of a multi-touch signal processing program according to an embodiment of the present invention;

5 is a diagram for explaining window processing;

6 is a view showing a shadow of a touch object;

FIG. 7 is a view showing an example of a malfunction which recognizes a shadow as shown in FIG. 6 as a touch object; FIG.

8 is a graph showing a histogram of a background image and a mean value and a standard deviation value in the histogram.

9 is a graph showing histograms of a background image, a touch object, a shadow, and average values in the histograms.

10 is a diagram illustrating labeling processing of each of the touch positions;

11 is a diagram for explaining a triangulation method;

12 is a diagram illustrating a process of removing a shadow from a touch image;

Description of the Related Art

10: liquid crystal display panel 11: source driver

12: gate driver 20: touch & display module

30: control board 31: timing controller

32: Multi-touch processor 40: System

21A to 21D: camera 22: blackened inner wall

The present invention relates to a display device, and more particularly, to a display device having a multi-touch recognition function for preventing a malfunction by removing shadows and a driving method thereof.

BACKGROUND ART [0002] A touch panel is generally mounted on a display device and is one of a user interface for detecting the touch position by changing its electrical characteristics at a touch position where the touch panel is in contact with a hand or a pen, and its application range is expanded to a small portable terminal, . Such a touch panel may malfunction when two or more multi-touches are generated at the same time, or can select any one by a preset program.

In order to overcome the limitation of multi-touch recognition in existing touch panels, a multi-touch recognition device that simultaneously recognizes a plurality of touches has been developed.

When a camera having a lens is used, such a multi-touch recognition device can misread the touch position due to a shadow of the touch object or aberration of the lens.

Accordingly, it is an object of the present invention to provide a display device having a multi-touch recognition function for preventing a malfunction by eliminating shadows and a driving method thereof.

In order to achieve the above object, a display device having a multi-touch recognition function according to an embodiment of the present invention includes a display device; A plurality of cameras provided in the vicinity of the edge of the display element; A reference value that is a function of an average value and a standard deviation value of a background image captured by the cameras and stored in advance is compared with a currently sensed touch image by the cameras to obtain a difference image, A touch control circuit for calculating the position of the touch object; And a display driving circuit for displaying an image of the touch object on the display device.

Wherein the background image comprises an image of at least a 100 frame period by each of the cameras with an image of a blackened sidewall of the display element.

The average value and the standard deviation value of the background image are analyzed as a histogram for each pixel of the background image, and the average values and variance values of the histogram for each pixel are calculated.

The touch control circuit stores the pixel value of the touch image as a pixel value of the touch image if the pixel value of the touch image is larger than the reference value and if the pixel value of the touch image is less than the reference value, And stores it as a pixel value of the touch image.

The touch control circuit calculates the difference between the pixel value of the background image and the pixel value of the touch image on a pixel-by-pixel basis to obtain the difference image.

The method of driving a display device according to an embodiment of the present invention compares a reference value that is a function of an average value and a standard deviation value of a background image captured by the cameras and stored in advance and a touch image currently captured by the cameras, Obtaining an image; Calculating a position of a touch object to be touched on the display element from the difference image; And displaying an image of the touch object on the display element.

Hereinafter, preferred embodiments of the present invention will be described with reference to FIGS. 1 to 12. FIG.

Referring to FIGS. 1 to 3, a display device having a multi-touch recognition function according to an embodiment of the present invention includes a touch-and-hold device having cameras 21A to 21D disposed at four corners of a pixel array 10A, A display module (20); A control board 30 for controlling the touch & display module 20 and performing an algorithm for recognizing the touch position; And a system 40 for supplying digital video data (RGB) to be displayed on a display element of the touch & display module 20 together with a timing signal to the control board 30.

The touch & display module 20 includes a liquid crystal display panel 10 in which a pixel array 10A on which an image is displayed is formed, a source for supplying a data voltage to the data lines D1 to Dm of the liquid crystal display panel 10, A gate driver 12 for supplying a scan pulse to the gate lines G1 to Gn of the liquid crystal display panel 10 and a plurality of drivers 12a to 13n arranged in the vicinity of the four corners of the pixel array 10A, (21A to 21D).

The liquid crystal display panel 10 includes a thin film transistor (hereinafter referred to as "TFT ") substrate and a color filter substrate. A liquid crystal layer is formed between the TFT substrate and the color filter substrate. On the TFT substrate, the data lines D1 to Dm and the gate lines G1 to Gn are formed to be orthogonal to each other on the lower glass substrate. The liquid crystal cells Clc are arranged in a matrix form in the cell regions defined by the data lines D1 to Dm and the gate lines G1 to Gn. The TFT formed at the intersection of the data lines D1 to Dm and the gate lines G1 to Gn is supplied via the data lines D1 to Dm in response to the scan pulse from the gate lines G1 to Gn To the pixel electrode of the liquid crystal cell Clc. To this end, the gate electrode of the TFT is connected to the gate lines G1 to Gn, and the source electrode thereof is connected to the data lines D1 to Dm. The drain electrode of the TFT is connected to the pixel electrode of the liquid crystal cell Clc. A common voltage (Vcom) is supplied to the common electrode facing the pixel electrode. The color filter substrate includes a black matrix, color filter formed on the upper glass substrate.

The common electrode is formed on the upper glass substrate in a vertical electric field driving method such as a TN (Twisted Nematic) mode and a VA (Vertical Alignment) mode, and a horizontal electric field such as IPS (In Plane Switching) mode and FFS (Fringe Field Switching) Is formed on the lower glass substrate together with the pixel electrode in the driving method.

The reference numeral 'Cst' is a storage capacitor. The storage capacitor Cst may be formed by overlapping the gate line and the pixel electrode of the liquid crystal cell Clc, or may be formed by overlapping another common line and the pixel electrode.

The source driver 11 includes a plurality of data integration circuits and converts the digital video data RGB input from the control board 30 into a positive or negative analog gamma compensation voltage under the control of the control board 30, And supplies the analog gamma compensation voltage to the data lines D1 to Dm as analog data voltages.

The gate driver 12 includes a plurality of gate integrated circuits and sequentially supplies scan pulses to the gate lines G1 to Gn under the control of the control board 30. [

The data integrated circuits of the source driver 11 and the gate integrated circuits of the gate driver 12 are connected to each other by tape automated bonding (TAB) using a tape carrier package (TCP) on glass (COG) method. The gate integrated circuits of the gate drive 12 may be formed directly on the lower glass substrate simultaneously with the TFTs of the pixel array 10A and in the same process as the TFT process.

The cameras 21A to 21D are cameras using CMOS (Complementary Metal Oxide Semiconductor) sensors and are arranged at four corners of the pixel array 10A in which an image is displayed on the liquid crystal display panel 10. [ Each of the cameras 21A to 21D captures an image of a display surface of the pixel array 10A and an image in the vicinity thereof. A touch image picked up from each of the cameras 21A to 21D is supplied to the control board 30. [

It is preferable that the lens angle of view of each of the cameras 21A to 21D is 90 [deg.]. The lens angle of view of the cameras 21A to 21D may vary depending on the distance between the liquid crystal display panel 10 and the cameras 21A to 21D and the size of the liquid crystal display panel 10, ° to 90 °. This is because if the lens angle of view of the cameras 21A to 21D is narrowed to less than 80 degrees, the number of shaded areas that the cameras 21A to 21D can not capture increases, If the angle is larger than 90 °, the multi-touch can not be accurately recognized including the portion that is out of the liquid crystal display panel 10.

The control board 30 is connected to the source driver 11 and the gate driver 12 through a flexible printed circuit (FPC) and a connector. The control board 30 includes a timing controller 31 and a multi-touch processor 32.

The timing controller 31 generates a gate control signal for controlling the operation timing of the gate driver 12 and an operation timing of the source driver 11 using the vertical and horizontal synchronous signals V and H and the clock CLK And generates a data control signal for controlling. The timing controller 31 supplies digital video data (RGB) input from the system 40 to the source driver 11.

The touch control circuit of the multi-touch processor 32 stores an average value of the background image, a standard deviation value of the background image, and a reference value to be compared for each pixel of the touch image, and executes a multi-touch signal processing program, . Such an algorithm includes a process of comparing a background image and a touch image, and a process of removing a shadow from a touch image to prevent a malfunction of recognizing a shadow of the touch object as a touch object during the comparison process. The multi-touch processor 32 supplies the system 32 with the touch position coordinate information Txy obtained by performing an algorithm for recognizing each touch position. The multi-touch processor 32 operates in synchronism with the timing controller 31 because it shares the timing signals such as the vertical / horizontal synchronizing signals V and H and the clock CLK with the timing controller 31.

The inner walls 22 of the touch & display module 20 exposed to the lenses of the cameras 21A to 21D are colored in black. In order to remove shadows of a touch object to be described later, the background image is picked up by the cameras 21A to 21D for a period of more than 100 frame periods and obtained as an average value for each pixel. This background image is stored in the memory of the multi-touch processor 30 and compared with the touch image during the touch recognition process.

The system 40 includes a memory in which an application program is embedded, a central processing unit for executing an application program, and a touch image synthesized with the image to be displayed on the liquid crystal display panel 10, Processing and resolution conversion, and the like. The system 40 receives touch position information Txy from the multi-touch processor 32 and executes an application program associated with the touch position information Txy. For example, if there is an icon for a particular program in the coordinates of the touch location, the system 40 loads and executes the program in memory. In addition, the system 40 generates digital video data (RGB) by synthesizing a touch image and an image to be displayed on the liquid crystal display panel 10. The system 40 may be implemented as a personal computer (PC) and may exchange data with the multi-touch processor 32 through a universal serial bus (USB) interface.

Fig. 4 is a flowchart showing the control procedure of the multi-touch signal processing program executed in the multi-touch processor 32 step by step.

Referring to FIG. 4, the multi-touch signal processing program receives a touch image captured by the cameras 21A to 21D, removes an unnecessary space image on the valid touch area through a window process as shown in FIG. 5 (S1 and S2) Window processing uses an image extraction technique for extracting only necessary images through a sub-matrix operation on an input image signal.

Then, the multi-touch signal processing program converts R, G, B included in the image of the valid touch area extracted by the window processing into gray information. (S3) The effective image extracted through the window processing is also converted to the color of red (R), green (G), and blue (B) Information. The background image and the effective touch image of the blackened side wall 22 are not compared with the color information but are compared based on the gray including the black and white and the middle tone therebetween. Therefore, in step S3, the color information of the valid touch image is converted into grayscale information by the following equation (1).

Gray scale intensity = pR + qG + sB

Here, 'p', 'q', and 's' are constants with different values.

In order to recognize the touch, it is necessary to compare only the touch image with the background image and the touch image having the actual touch. To this end, the multi-touch signal processing program compares a pre-stored background image and a touch image currently photographed by the cameras 21A to 21D with an average value for each pixel and extracts an image of an actual touch object corresponding to the difference between the two images (S4)

If the shadow of the touch object is seen on the blackened side wall 22 as shown in FIG. 6, not only the touch object but also the shadow of the touch object are extracted as the image of the touch object extracted by the difference in the process of comparing the background image and the touch image. In this case, as shown in Fig. 7, a shadow of the touch object is recognized as a touch object and a malfunction occurs. FIG. 7 shows an example of a difference image between a background image and a touch image when a background image, a touch image, and a shadow of a touch object are included in a touch image.

In order to prevent such a malfunction, the multi-touch signal processing program of the present invention is characterized in that the background image obtained by photographing by the cameras 21A to 21D for a period of 100 frames or more has a cumulative function of each pixel, i.e., a histogram And histograms are analyzed to calculate average values and standard deviation values for each pixel in the background image and store them in the memory.

The process of obtaining the background image will be described in detail as follows. 8 is an example of a histogram for the experimental pixels 140, 2 of the background image. This histogram is the experimental result obtained from the experimental pixels of the background image for 10,000 frame periods. Since the background image is the image of the blackened inner wall 22, the histogram of pixels other than the experimental pixels in the background image is also obtained with a histogram similar to FIG. The present invention produces a histogram for all pixels of the background image. The horizontal axis in FIG. 8 is the grayscale, and the vertical axis is the cumulative value. Then, an average value and a standard deviation value for each pixel of the background image are calculated for a period of 100 frames or more in the histograms for each pixel. The mean value mu of each pixel in the histogram is calculated by Equation (2), and the standard deviation value [sigma] is calculated by Equation (3).

The multi-touch signal processing program of the present invention compares a background image and a touch image on a pixel-by-pixel basis. 9 shows an example of a background image and a histogram of a touch image for one pixel. As shown in FIG. 9, the average value μ _S of the shadow of the touch object has a value similar to the average value μ _B of the background image, and the average value μ _O of the touch object has a larger difference than the shadow or the background. According to the experimental results, the average value of the background image and the touch image in all the pixels is as shown in FIG.

The multi-touch signal processing program of the present invention determines a reference value between a mean value μ _S of a shadow of a touch object and an average value μ _O of a touch object to remove a shadow of the touch object. The multi-touch signal processing program of the present invention compares the reference value with each pixel value of the currently captured touch image. As a result of the comparison, if the pixel value of the currently captured touch image is larger than the reference value, the multi-touch signal processing program stores the pixel value as the pixel value of the touch image. If the pixel value of the currently captured touch image is less than the reference value The pixel value is replaced with the average value of the previously stored background image pixels and is stored as the pixel value of the touch image.

Then, the multi-touch signal processing program of the present invention compares the background image with the touch image in each pixel unit, and obtains the difference image.

Equation (4) below defines the pixel value B _p of the background image, and Equation (5) defines the pixel value O _P of the touch image. Equation (6) is a formula for defining a difference image pixel value D _P between the background image and the touch image.

In Equations 4 to 6,

μ _B : histogram average value of the background image calculated over a period of 100 frame periods or more

σ _B : Histogram standard deviation value of the background image calculated over a period of 100 frame periods or more

alpha: adjustment coefficient (0?? 1), the reference value can be adjusted by this adjustment coefficient.

μ _B + α · σ _B : Reference value for distinguishing between shadow and touch object

B _P : pixel value of background image

I _P : pixel value of the touch image captured by the current camera

O _P : pixel value of the touch image

D _P : Difference between background image and touch image Image pixel value

Then, the multi-touch signal processing program compares the gray scale intensity of the difference image calculated in step S4 with a preset threshold value. The multi-touch signal processing program converts the gradation information of the touch image into white data only for data exceeding the threshold value, and converts the data of less than the threshold value into black data (S5). Touch position, and black data means invalid data that is not actually touched in the touch image. The threshold value is determined experimentally.

In step S6, the multi-touch signal processing program performs a labeling process. In order to distinguish each touch position, the labeling process assigns unique identification codes (? To?) To white data, that is, effective touch position data, as shown in FIG.

Then, the multi-touch signal processing program uses the angle calculation algorithm to find the position on the two-dimensional plane for each of the valid touch positions converted into the white data, so that the angle of each of the cameras 21A to 21D (S7)

Each of the measured positions between the valid touch positions and the camera measured in the step S7 includes a distortion amount corresponding to the lens aberration of the cameras 21A to 21D. Accordingly, the multi-touch signal processing program registers a compensation value for each of the angles between the effective touch position and the camera in a look-up table, adds a compensation angle from the look-up table to the measurement angle, (S8) The calculation method of the compensation angle and the compensation circuit using the compensation angle are described in detail in P07-024180, which is hereby incorporated by reference.

In step S9, the multi-touch signal processing program calculates the position of each of the valid touch positions using the triangulation method as shown in equation (6) from the angles in which the lens distortion is compensated. Equation (7) is a calculation formula for calculating one touch position as a two-dimensional xy coordinate value. As shown in FIG. 11, the measurement angles A and B of two cameras, the angle C between the touch positions and the cameras, And a distance (a, b, c) between the camera and the touch position. The angle C between the touch position and the cameras is calculated as "C = 180-angle A-angle B ". This triangulation method is applied to each of a plurality of touch positions.

When the touch position information Txy including the x and y coordinate data for each touch position is calculated from the multi-touch signal processing program, the system 40 displays the touch position information Txy on the liquid crystal display panel 10 Lt; / RTI > digital video data. The touch position information Txy synthesized on the background image is transmitted to the ICs of the source driver through the timing controller 31 and displayed on the liquid crystal display panel 10. (S10) . ≪ / RTI >

The liquid crystal display panel 10 and the cameras may be separated from each other by the touch & display module 20. In this case, the liquid crystal display panel 10 10 and the camera module. The liquid crystal display panel 10 may include other flat panel display panels, for example, a display panel of an organic light emitting diode (OLED), a plasma display panel (PDP), a field emission display FED) or a display panel of a stereoscopic image display device including a flat panel display panel.

FIG. 12 is a diagram illustrating an example in which shadows are removed from a touch image obtained through comparison with a reference value.

As described above, in the display device having the multi-touch recognition function and the driving method thereof, the shadow is removed from the touch image using the reference value between the average value of the actual touch object and the shadow average value of the touch object Malfunctions can be prevented.

It will be apparent to those skilled in the art that various modifications and variations can be made in the present invention without departing from the spirit or scope of the invention. Therefore, the technical scope of the present invention should not be limited to the contents described in the detailed description of the specification, but should be defined by the claims.

Claims (10)

Display element; A plurality of cameras provided in the vicinity of the edge of the display element; A reference value that is a function of an average value and a standard deviation value of a background image captured by the cameras and stored in advance is compared with a currently sensed touch image by the cameras to obtain a difference image, A touch control circuit for calculating the position of the touch object; And And a display driving circuit for displaying an image of the touch object on the display element; Wherein the background image is an image of a blackened sidewall of the display element, the image being captured by each of the cameras over a 100-frame period. delete The method according to claim 1, The average value and the standard deviation value of the background image are Wherein the average value and the variance values of the histogram for each pixel are analyzed as a histogram for each pixel of the background image and calculated as a result of the analysis. The method of claim 3, The touch control circuit includes: If the pixel value of the touch image is larger than the reference value, the pixel value is stored as a pixel value of the touch image, and if the pixel value of the touch image is less than the reference value, the pixel value is replaced with an average value of the stored background image pixel, And stores the pixel value as a pixel value of the image. 5. The method of claim 4, The touch control circuit includes: Wherein the difference image is obtained by calculating a difference between a pixel value of the background image and a pixel value of the touch image in each pixel unit. A driving method of a display device comprising a display element and a plurality of cameras provided in the vicinity of the edge of the display element, Obtaining a difference image by comparing a reference value that is a function of an average value and a standard deviation value of a background image captured by the cameras and stored in advance and a touch image currently captured by the cameras; Calculating a position of a touch object to be touched on the display element from the difference image; And And displaying an image of the touch object on the display element; Wherein the background image is an image of a blackened sidewall of the display element, the image being captured by each of the cameras for at least 100 frame periods. delete The method according to claim 6, The average value and the standard deviation value of the background image are Wherein the histogram includes a mean value and a variance value of a histogram for each pixel calculated as a histogram for each pixel of the background image and calculated as a result of the analysis. 9. The method of claim 8, Storing a pixel value of the touch image as a pixel value of the touch image if the pixel value of the touch image is larger than the reference value; And And storing the pixel value of the touch image as a pixel value of the touch image if the pixel value of the touch image is less than a reference value, . 10. The method of claim 9, The step of obtaining the difference image comprises: Wherein the difference image is obtained by calculating a difference between a pixel value of the background image and a pixel value of the touch image in each pixel unit.

Images