JP2010119064A - Color detection device, color detection program, computer readable recording medium, and color detection method - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To miniaturize a device for detecting the color of a contact object in contact and to prevent lowering of sensitivity and deterioration of resolution while ensuring the size of a sensor area. <P>SOLUTION: The device has a plurality of picture element circuits 31, a liquid crystal panel 301 with a built-in sensor including a plurality of optical sensors 6, at least one of which is provided to each of the picture element circuits 31, an image display processing section 18 for coloring the picture element circuits 31 contained in a region PP one by one by an R picture element circuit 31r to a B picture element circuit 31b, a scan processing section 12 for scanning an image of a pen P which comes into contact with the liquid crystal panel 301 with a built-in sensor by using each optical sensor 6 every time the image display processing section 18 colors the picture element circuits 31 contained in a region PP in the R picture element circuit 31r to the B picture element circuit 31b and a color decision processing section 14A for detecting the color of a contact part on the liquid crystal panel 301 with a built-in sensor of the pen P based on each image of the pen P, which is scanned by the scan processing section 12 every color development in the R picture element circuit 31r to the B picture element circuit 31b. <P>COPYRIGHT: (C)2010,JPO&INPIT

Description

  The present invention relates to a color detection device, a color detection program, a computer-readable recording medium, and a color detection method, and more specifically, detects the color of a contact portion of a contact object that has contacted a planar member. The present invention relates to a color detection apparatus, a color detection program, a computer-readable recording medium on which the color detection program is recorded, and a color detection method color.

  As is well known, image display devices having a liquid crystal display as an image display unit are widely used in various devices such as mobile phones and PDAs (Personal Digital Assistants).

  In particular, a PDA has been provided with a touch sensor (analog resistance film type, capacitance type) for a long time, enabling touch input to input information by directly bringing a finger or the like into contact with a liquid crystal display. In addition, in a mobile phone and other devices, a sensor built-in type liquid crystal display device including a touch sensor is expected to spread.

  On the other hand, for example, an electronic blackboard having a drawing display function recognizes an instruction position such as a pen touching the display screen, and displays a predetermined color in a predetermined area including the instruction position on the display screen. You can draw on the screen.

  Combining such an electronic blackboard function with an image display function such as a liquid crystal display can further improve user convenience.

  As a conventional example of a drawing display device that electronically draws on a display screen using a pen or the like, for example, Patent Literature 1 discloses a technique related to an input color determination method on an electronic blackboard.

  In this input color determination method for an electronic blackboard, the color detector in the pen holder determines which of the red, blue and black felt pens remains from the pen holder provided on the electronic blackboard. Thus, the color of the felt pen currently in use that does not exist in the pen holder is determined.

  As another example, Patent Document 2 discloses a technique related to a display method in CAD (computer-aided design).

  In this CAD display method, a pen number corresponding to a specific color can be input, and based on the input pen number, a color corresponding to the pen number can be drawn on the display screen. ing.

  As yet another example, Patent Document 3 discloses a technique related to a message print method.

In this message printing method, a printed photograph is printed out with a handwritten message written in the photo data by writing a message using a light pen on the display screen on which the photo data is displayed. . Note that the message color is selected by selecting an icon or the like displayed on the display screen with a light pen.
Japanese Patent Laid-Open No. 61-213918 (published September 22, 1986) Japanese Unexamined Patent Publication No. 02-121072 (published May 8, 1990) Japanese Patent Laid-Open No. 10-268440 (released on October 9, 1998) Japanese Patent Laid-Open No. 4-282609 (released on October 7, 1992) Japanese Patent Laid-Open No. 1-188069 (published July 27, 1989) JP-A-4-313722 (published November 5, 1992)

  However, since the conventional sensor-equipped liquid crystal display device cannot directly detect the color of the contacted object on the liquid crystal display, for example, drawing is performed on the liquid crystal display using an input device such as a pen. It was not possible to detect the color of the pen tip.

  Also, none of Patent Documents 1 to 3 discloses a method for directly detecting the color of a pen tip that is electronically drawing on a display screen.

  By the way, as a method for detecting the color of the pen tip such as a pen that contacts the liquid crystal display, the three primary colors are used in front of the image sensor in the same manner as in the technical field of a conventional CCD (Charge-coupled device) camera. It is conceivable to detect the color by providing a filter.

  As an example of a technique relating to such a conventional color detection method, Patent Document 4 discloses a technique relating to an ultra-thin input / output integrated information processing apparatus including a light source, an image sensor, a color TFT liquid crystal, and a color filter. Yes.

  In this ultra-thin input / output integrated information processing apparatus, by arranging image sensors corresponding to R (Red), G (Green), and B (Blue) pixels, respectively, The color image of the image reading object placed on the display screen can be read.

  Note that this ultra-thin input / output integrated information processing apparatus only discloses that when reading a color image of an image reading object, light is transmitted through all pixels of the color TFT liquid crystal. .

  Also, in this ultra-thin input / output integrated information processing apparatus, an image sensor is provided for each sub-pixel corresponding to each of R, G, and B pixels.

  Here, problems of the ultra-thin input / output integrated information processing apparatus described in Patent Document 4 will be described with reference to FIGS.

  FIGS. 17A and 17B are conceptual diagrams for explaining the relationship in size between the image sensor and the color filter when a color filter is provided between the light source and the image sensor. The sensor is placed under the color filter).

  Hereinafter, a case where three subpixels exist in one pixel pixel will be described.

  FIG. 17A conceptually shows the size relationship between the image sensor and the color filter in the ultra-thin input / output integrated information processing apparatus disclosed in Patent Document 4, and includes three image sensors. A case where three primary color filters of R, G, and B are provided in each subpixel is shown.

  As described above, in the method of providing an image sensor for each sub-pixel, fine processing is required and there is a problem in cost, and the area of the image sensor for each color of R, G, and B is reduced and sensitivity is lowered. There is a problem that it will.

  On the other hand, for example, FIG. 17B conceptually shows a case where the sizes of the three primary color filters of R, G, and B are the same as the size of one pixel, and an image sensor is provided for each pixel. .

  In this case, light for three subpixels can be used and the size of the image sensor can be made relatively large, so that the sensitivity can be improved by about three times, but the resolution is reduced to 1/3. There is a problem.

  The present invention has been made in view of the above-described conventional problems, and an object of the present invention is to provide a contact object in contact with the display surface of an input / output integrated display (planar member) useful for downsizing of the apparatus. Color detection device, color detection program, computer-readable recording medium, and color capable of detecting color while preventing the decrease in sensitivity and resolution while ensuring the size of the sensor area necessary for detection It is to provide a detection method.

  In order to solve the above problems, the color detection device of the present invention is included in a predetermined range and a planar member including a plurality of unit pixels and a plurality of imaging sensors provided for each unit pixel. Color development control means for sequentially developing the unit pixels in a first color, a second color, and a third color that satisfy the principle of equal colors, and the color development control means, the unit pixels included in the predetermined range are Image pickup control means for picking up an image of a contact object in contact with the planar member using the image pickup sensor, and the first color and the second color each time the color is developed with one color, the second color, and the third color. And color detecting means for detecting the color of the contact portion of the contact object on the planar member based on each of the images of the contact object imaged by the imaging control means each time the color is developed with the third color. It is characterized by having.

  In order to solve the above-described problem, the color detection method of the present invention is executed using a planar member including a plurality of unit pixels and a plurality of image sensors provided at least one for each unit pixel. A color control method in which the unit pixels included in a predetermined range are colored with a first color, a second color, and a third color that satisfy the principle of equal colors, and the color control step, Each time a unit pixel included in a predetermined range is colored in the first color, the second color, and the third color, an image of a contact object that contacts the planar member is captured using each of the imaging sensors. The surface of the contact object based on each of the images of the contact object imaged in the imaging control step each time the color is generated with the first color, the second color, and the third color. Detection to detect the color of the contact part on the member It is characterized in that it contains step.

  According to the configuration or the method, the planar member includes a plurality of unit pixels and a plurality of imaging sensors provided at least one for each unit pixel.

  Therefore, a case where an image is displayed on a planar member (for example, on a predetermined display surface by operating a plurality of unit pixels of the planar member is considered. It can be displayed on the “surface”.

  In addition, the light emitted from the unit pixels of the planar member is irradiated with light on the display surface, and the reflected light is detected by the imaging sensors of the planar member. By doing so, it is possible to capture an image of the contact object on the display surface.

  According to the above configuration, since it is not necessary to configure the image display unit and the image capturing unit as separate devices, the input / output integrated device can be downsized.

  Further, according to the configuration or the method, the color development control means (in the color development control step) applies the first color, the second color, and the third color that satisfy the principle of color matching for each of the unit pixels included in the predetermined range. In this way, color is developed sequentially.

  Further, the imaging control means (in the imaging control step) is configured so that the color development control means (in the color development control step) develops the unit pixels included in the predetermined range with the first color, the second color, and the third color. Each time an image of a contact object in contact with the planar member is picked up, each image sensor is used.

  Further, according to the configuration or the method, the color detection unit (in the color detection step) is performed by the imaging control unit (in the imaging control step) every time the first color, the second color, and the third color are developed. Based on each of the captured images of the contact object, the color of the contact portion of the contact object on the planar member is detected.

  Here, the “equal color principle” is a scientific term in chromatics, and is the principle that all colors can be expressed in three colors.

  Further, in order to detect the color of the object, it is only necessary to irradiate the object with light having three different colors and detect the reflected light based on the principle of color matching.

  As described above, if the color of the contact part of the contact object is detected by generating the first color, the second color, and the third color, the color of the contact part of the contact object is assumed to be the first color. Even in the case of the same color, the color can be detected based on imaging data obtained by imaging the remaining second color and third color.

  In addition, the “predetermined range” is a wide concept including, for example, the entire display surface on the planar member to a partial specific range of the display surface.

  Further, “coloring with the first color, the second color, and the third color” means that the first color is developed first, then the second color is developed, and then the third color is developed. .

  Here, from the viewpoint of improving the accuracy of color detection, the first color, the second color, and the third color are developed at a time, and one imaging data obtained by imaging the state in which these three colors are mixed is analyzed. Rather than performing color detection, the first color, the second color, and the third color are developed for the three colors that satisfy the principle of equal colors, and the first color, the second color, and the third color are developed. It is preferable to prepare three pieces of image data taken using each image sensor and analyze these three image data.

  This is because the first color, the second color, and the second color are compared with the case where the color detection is performed by analyzing one piece of imaging data obtained by imaging the state in which the first color, the second color, and the third color satisfying the principle of equal colors are mixed. This is because it is much easier to analyze and the detection accuracy is higher when color detection is performed by analyzing three imaging data in a state where the three colors are separated from each other.

  According to the above configuration or method, for example, even when an image sensor is provided for each unit pixel, each unit pixel is composed of at least one image sensor for three colors of the first color, the second color, and the third color. Image data can be obtained separately.

  From the viewpoint of detection accuracy of color detection, it is preferable to arrange an image sensor for each color pixel. However, according to the configuration or method, the detection accuracy of color detection is relatively high even if the number of image sensors is small. The current sensor area (for example, the number of unit pixels: the number of imaging sensors = 1: 1) and the detection accuracy of color detection can be ensured, and a decrease in detection sensitivity can be prevented.

  Further, according to the configuration or method, in order to increase the detection sensitivity, it is not necessary to develop one color using all the light emitted from the three color pixels in the unit pixel. It is also possible to prevent the deterioration.

  As described above, it is possible to detect the color of a contact object that is in contact with the display surface of the input / output integrated display (planar member), which is useful for downsizing of the apparatus, while preventing a decrease in sensitivity and a decrease in resolution. A color detection apparatus and a color detection method that can be provided can be provided.

  The color detection apparatus of the present invention further includes contact determination means for determining whether or not the contact object is in contact with the planar member, and the color development control means is configured to detect the contact object by the contact determination means. When it is determined that is in contact with the planar member, the unit pixel is preferably colored.

  According to the above configuration, the unit pixel is colored with the first color, the second color, and the third color only when the contact determination unit determines that the contact object is in contact with the planar member. When the color cannot be detected, it is possible to prevent a useless detection operation such as performing color detection.

  Here, based on FIG. 17C, another problem that the configuration for developing the first color, the second color, and the third color has will be described.

  FIG. 17C illustrates a conventional color scanner that sequentially irradiates light of R (Red), G (green), and B (blue) to a color image and captures the image with an image sensor. It is a conceptual diagram which shows a thing.

  For example, Patent Documents 5 and 6 disclose scanners in which a liquid crystal is provided between a light source and an original as a modification of the conventional color scanner, and the color of the light source is changed by the liquid crystal to capture the color.

  Note that the techniques disclosed in Patent Documents 5 and 6 neither describe nor suggest the viewpoint of performing color detection using pixels for image display. Further, another optical system element such as a lens is used, and the viewpoint of miniaturization of the apparatus is not taken into consideration.

  Here, in the case of a color scanner, image data of a document is taken in a state where the document is placed on the scan screen, so even if the R, G, and B lights are sequentially irradiated on the scan screen, The light is hidden from the manuscript or the like and is hardly visible to the user.

  However, in the case of detecting the color of the contact object that has come into contact with the display surface of the input / output integrated display (planar member), the user is viewing the state in which the image is displayed on the display surface. In view of detecting the color of the contact object, when the first color, the second color, and the third color are sequentially developed, another problem that the display flickers occurs.

  Therefore, in order to solve such a problem, in addition to the above-described configuration, the color detection device of the present invention is configured such that the contact on the planar member is based on an image of the contact object imaged by the imaging control unit. Contact part inclusion region specifying means for specifying a contact part inclusion region including a contact part of an object, wherein the color development control means sets the contact part inclusion region specified by the contact part inclusion region specifying means as the predetermined range, Each of the unit pixels is preferably colored.

  According to the above-described configuration, the coloring control unit causes the unit pixels included in the contact portion inclusion region (contact portion inclusion region = predetermined range) on the planar member to color.

  Here, the “contact portion inclusion region” may be a contact portion region of a contact object, or may be a closed region of any size including the contact portion and its vicinity. From the viewpoint of reducing the influence of flickering on the user, it is preferable to make it as small as possible.

  Thereby, the influence on the user by the flickering of the display can be reduced by making the size of the contact portion inclusion area as small as possible within the range where color detection is possible.

  That is, since the contact object is in contact with the contact part inclusion region, the display flicker in the contact part inclusion region can be hidden by the contact object and hardly visible to the user.

  As described above, the color of the contact object that is in contact with the display surface of the input / output integrated display (planar member), which is useful for downsizing of the device, is caused by display flickering while preventing deterioration in sensitivity and resolution. It is possible to provide a color detection device and a color detection method that can be detected with less influence on the user.

Further, in the color detection device of the present invention, in addition to the above configuration, the first color is one of four colors consisting of red (R), green (G), blue (B), and white,
The second color is one of the remaining three colors that are not the first color among the four colors.
The third color is preferably one of the remaining two colors that are not the second color among the three colors.

  According to the above configuration, the present invention can be applied to an image display device that uses a group of R, G, and B pixels generally used in an image display device as a unit pixel.

  In order to color the unit pixel in white, all of the R pixel, G pixel, and B pixel may be colored.

  Further, in the color detection apparatus of the present invention, in addition to the above configuration, the color development control unit may convert the unit pixel into the first color and the second color when the color of the contact portion is detected by the color detection unit. It is preferable to end the operation of generating the color and the third color.

  According to the above-described configuration, the operation of developing the unit pixel is continued until the color of the contact portion of the contact object is detected by the color detection means, so that the color of the contact portion of the contact object can be reliably detected. .

  In addition, once the color is detected, the coloring operation is terminated, so that it is possible to prevent a useless operation such as detecting the color in an overlapping manner.

  In addition to the above configuration, the color detection apparatus of the present invention preferably includes a storage unit that holds color information indicating the color of the contact portion of the contact object detected by the color detection unit.

  According to the configuration, the color information indicating the color of the contact portion of the contact object detected by the color detection unit is held, so that it is not necessary to re-detect the color of the same contact object.

  Note that the timing for holding the color information is preferably immediately after the color detecting means detects the color of the contact object.

  Further, as a period for holding the color information, the contact object comes into contact (for example, the contact determination unit determines that the contact object is in contact), and the color detection is performed (for example, the contact determination unit is The period until it is determined that the contact object is not in contact can be exemplified.

  Further, the retained color can be appropriately used for necessary processing. For example, if the detected pen color can be used to draw on the display surface, electronic drawing can be performed with the same feeling as drawing with color pencils on drawing paper.

  Each means, each function, each step, and each process in the color detection device and the color detection method may be realized by a computer. In this case, the computer is operated as each means, and the computer There is also provided a color detection program for realizing the above-described color detection device and color detection method by causing a computer to execute each step and each process, and a computer-readable recording medium recording the same. Falls within the scope of the present invention.

  As described above, the sensitivity of the contact object that touches the display surface of the input / output integrated display (planar member), which is useful for downsizing the device, decreases in sensitivity while ensuring the size of the sensor area required for detection. In addition, it is possible to provide a color detection device, a color detection program, a computer-readable recording medium, and a color detection method that can be detected while preventing deterioration in resolution.

  As described above, the color detection device of the present invention includes a planar member including a plurality of unit pixels and a plurality of imaging sensors provided for each unit pixel, and the unit pixels included in a predetermined range. A first color, a second color, and a third color that satisfy the principle of equal color, and a color control unit that sequentially develops the unit pixels included in the predetermined range with the first color, Imaging control means for imaging an image of a contact object that contacts the planar member using the imaging sensor each time the second color and the third color are developed, the first color, the second color, and the third color Color detecting means for detecting the color of the contact portion of the contact object on the planar member based on each of the images of the contact object imaged by the imaging control means each time color is developed. Is.

  In addition, as described above, the color detection method of the present invention performs color detection performed using a planar member including a plurality of unit pixels and a plurality of image sensors provided at least one for each unit pixel. In the method, the unit pixels included in a predetermined range are sequentially developed with a first color, a second color, and a third color that satisfy the principle of equal colors, and the predetermined color control step includes the predetermined color control step, and the predetermined color control step. Image pickup control for picking up an image of a contact object in contact with the planar member using each of the image sensors each time a unit pixel included in the range is colored with the first color, the second color, and the third color. Step, and each time the first color, the second color, and the third color are developed, based on each of the images of the contact object imaged in the imaging control step, on the planar member of the contact object A color detection step that detects the color of the contact area A Dale method.

  Therefore, it is possible to detect the color of a contact object that is in contact with the display surface of an input / output integrated display (planar member), which is useful for downsizing of the apparatus, while preventing a decrease in sensitivity and a decrease in resolution. There is an effect of providing a color detection device and a color detection method that can be performed.

  An embodiment of the present invention will be described below with reference to FIGS.

[Embodiment 1]
(Overview of data display / sensor device and overview of its functions)
First, based on FIG. 1, an outline configuration and functions of a data display / sensor device (color detection device) 100 according to an embodiment of the present invention will be described.

  FIG. 1 is a schematic diagram showing an overview configuration and a summary of functions of a data display / sensor device 100 according to an embodiment of the present invention.

  As shown in FIG. 1, the general configuration of the data display / sensor device 100 according to the present embodiment includes a housing 300A and a housing 300B that can be folded in two and coupled by a hinge H. A sensor built-in liquid crystal panel (planar member) 301A and a sensor built-in liquid crystal panel (planar member) 301B (hereinafter collectively referred to as a sensor built-in liquid crystal panel (planar shape) are respectively formed on two inner surfaces facing each other in a folded state of the housing 300B. Member) 301) are arranged so as to face each other.

  Also, as an overview of the function, a plurality of pixel circuits (unit pixels) 31 (see FIG. 8) included in each sensor built-in liquid crystal panel 301 and at least one photosensor (imaging sensor) provided for each pixel circuit 31 ) 6 (see FIGS. 2A and 2B), the pen tip of the pen (contacting object) P that is in contact with one of the two sensor-embedded liquid crystal panels 301A and 301B It is possible to detect the color of the (contact portion).

  Here, the unit pixel is described in parentheses in the parenthesis of the pixel circuit 31. However, as shown in FIG. 2A, the “unit pixel” is strictly in the case of a liquid crystal display, for example. This is a set including at least the color filter 53r, the color filter 53g, the color filter 53b, the liquid crystal layer 52, and the backlight 307. In the following description, the pixel circuit 31 is referred to as a “unit pixel” for the sake of convenience. Will be described. Note that the part describing the pixel circuit 31 itself means the pixel circuit 31 itself. The same applies to the notation in parentheses for the R pixel circuit (color pixel, R pixel) 31r, the G pixel circuit (color pixel, G pixel) 31g, and the B pixel circuit (color pixel, B pixel) 31b.

  Although details will be described later, as shown in FIG. 1, in the data display / sensor device 100, the pen tip of the pen P is in contact with the sensor built-in liquid crystal panel 301 (in FIG. 1, the sensor built-in liquid crystal panel 301B). ) Is detected (determined) to specify a region PP having a predetermined size (contact portion inclusion region: the size of the region PP is exaggerated in FIG. 1). The light of a specific color when only the pixel circuit 31 included in the region PP is colored is applied to the pen P, and the reflected light is detected by the plurality of photosensors 6, whereby the pen tip color of the pen P is detected. Is supposed to be detected.

  In the following description, for the sake of convenience, there is a portion where the area PP is described as a predetermined limited area. However, the area PP is not limited to such a limited area. It may be a region of a portion where the P pen tip contacts, or may be a closed region of any size including the contacting portion and its vicinity. Moreover, the area | region of the whole display surface of the liquid crystal panel 301 with a built-in sensor may be sufficient.

  Further, the data display / sensor device 100 holds the detected color at least while the pen P is in contact with the sensor built-in liquid crystal panel 301, and can draw a locus L with the color, for example. Yes.

  As described above, the sensor built-in liquid crystal panel 301 includes a plurality of pixel circuits 31 and a plurality of photosensors 6 provided at least one for each pixel circuit 31.

  Therefore, it is possible to display an image on a predetermined display surface on the sensor built-in liquid crystal panel 301 by operating the plurality of pixel circuits 31 of the sensor built-in liquid crystal panel 301. In addition, the plurality of pixel circuits 31 of the sensor built-in liquid crystal panel 301 illuminate the pen P on the display surface using light when color is generated, and the reflected light is applied to the plurality of optical sensors of the sensor built-in liquid crystal panel 301. By detecting at 6, it is possible to take an image of the pen P on the display surface.

  The number of each of the plurality of photosensors 6 and the plurality of pixel circuits 31 may be arbitrary as long as it can secure an image display function and a color detection function described below. From the viewpoint of the sensor area of the sensor 6, it is preferable that the number is one to one. As a result, the sensor built-in liquid crystal panel 301 can be easily created by a conventional fine processing technique, and the conventional sensor area can be maintained.

  However, the advancement of technology is expected to improve the sensitivity of the optical sensor 6 per sensor area in the future and advance the microfabrication technology. Therefore, it is possible to secure the size of the sensor area necessary for color detection. If so, the number of each of the plurality of photosensors 6 and the plurality of pixel circuits 31 may be 1: 3, and the latter may be three times as many as the former. By adopting such a configuration, in the case where the pixel circuit 31 has, for example, three color pixels, it is possible to secure independence for each color detected by the optical sensor 6 of light generated by each color pixel.

  Note that the number of each of the plurality of photosensors 6 and the plurality of pixel circuits 31 may be 1: 2, and the latter may be twice as many as the former.

  That is, the optical sensor 6 may include at least one for each of the plurality of pixel circuits 31.

  Each of the plurality of photosensors 6 and the plurality of pixel circuits 31 is preferably arranged in a plane in a predetermined range in the surface direction (in-plane direction of the display screen) of the sensor built-in liquid crystal panel 301. For example, a matrix-like arrangement can be exemplified.

  The positional relationship between the plurality of photosensors 6 and the plurality of pixel circuits 31 is, for example, the case where the numbers are one-to-one and the same number, and the pixel circuit 31 includes three color pixels as described below. Is preferably close to any of the three color pixels (see, for example, FIG. 2B).

  According to the above configuration, since the image display unit and the image capturing unit do not need to be configured as separate devices, the data display / sensor device 100 can be downsized.

  The planar member such as the sensor built-in liquid crystal panel 301 is not limited to the sensor built-in liquid crystal panel of the present embodiment. For example, as a display, in addition to a liquid crystal display, an electrophoretic display, a twist ball display, In addition to reflective displays using fine prism films, displays using light modulation elements such as digital mirror devices, organic EL (Light Emitting) light emitting elements, inorganic EL light emitting elements, and LEDs (Light Emitting Diodes) For example, a display using a variable emission luminance element, a field emission display (FED), a plasma display, etc., but these displays and an image sensor (image sensor, optical sensor) are integrated into an input / output integrated type. Even if the display is developed in the future It will be apparent from the following description that the present invention can be applied to these input / output integrated displays.

  First, an outline of the sensor built-in liquid crystal panel 301 included in the data display / sensor device 100 will be described below.

(Outline of LCD panel with built-in sensor)
The sensor built-in liquid crystal panel 301 provided in the data display / sensor device 100 is a liquid crystal panel capable of detecting an image of an object in addition to displaying data. Here, the image detection of the object is, for example, detection of a position pointed (touched) by the user with a finger or a pen, or reading (scanning) of an image of a printed material. The device used for display is not limited to a liquid crystal panel, and may be an organic EL (Electro Luminescence) panel or the like.

  The structure of the sensor built-in liquid crystal panel 301 will be described with reference to FIGS. 2 (a) and 2 (b). FIG. 2A is a diagram schematically showing a cross section of the sensor built-in liquid crystal panel 301. The sensor built-in liquid crystal panel 301 described here is an example, and any structure can be used as long as the display surface and the reading surface are shared.

  As shown in the figure, the sensor built-in liquid crystal panel 301 includes an active matrix substrate 51A disposed on the back surface side and a counter substrate 51B disposed on the front surface side, and has a structure in which a liquid crystal layer 52 is sandwiched between these substrates. is doing. The active matrix substrate 51A is provided with a pixel electrode 56, a data signal line 57, an optical sensor circuit 32 (not shown), an alignment film 58, a polarizing plate 59, and the like. The counter substrate 51B is provided with color filters 53r (red), 53g (green), 53b (blue), a light shielding film 54, a counter electrode 55, an alignment film 58, a polarizing plate 59, and the like. In addition, a backlight 307 is provided on the back surface of the sensor built-in liquid crystal panel 301.

  The photosensor 6 included in the photosensor circuit 32 is provided in the vicinity of the pixel electrode 56 provided with the blue color filter 53b, but is not limited to this configuration. It may be provided in the vicinity of the pixel electrode 56 provided with the red color filter 53r, or may be provided in the vicinity of the pixel electrode 56 provided with the green color filter 53g.

  Next, another example of the arrangement method of the optical sensor 6 will be described with reference to FIG.

  FIG. 2B is a schematic diagram showing one pixel of another example of the sensor built-in liquid crystal panel 301 provided in the data display / sensor device 100.

  As shown in the figure, in this example, the optical sensor 6 is arranged in a straight line so that the optical sensor 6 is close to any of the color filter 53r, the color filter 53g, and the color filter 53b, and the color filter 53g and the color filter. It is arranged on the upper side with respect to the paper surface of 53b. By adopting such an arrangement, the size of the optical sensor 6 can be increased, and light can be detected with high sensitivity from any of the color filter 53r, the color filter 53g, and the color filter 53b.

  Next, with reference to FIGS. 3A and 3B, two types of methods for detecting the position where the user touches the sensor built-in liquid crystal panel 301 with a finger or a pen will be described.

  FIG. 3A is a schematic diagram showing how the position touched by the user is detected by detecting the reflected image. When the light 63 is emitted from the backlight 307, the optical sensor circuit 32 including the optical sensor 6 detects the light 63 reflected by the object 64 such as a finger. Thereby, the reflected image of the target object 64 can be detected. Thus, the sensor built-in liquid crystal panel 301 can detect the touched position by detecting the reflected image.

  FIG. 3B is a schematic diagram showing how the position touched by the user is detected by detecting a shadow image. As shown in FIG. 3B, the optical sensor circuit 32 including the optical sensor 6 detects external light 61 transmitted through the counter substrate 51B and the like. However, when there is an object 62 such as a pen, the incident of the external light 61 is hindered, so that the amount of light detected by the optical sensor circuit 32 is reduced. Thereby, a shadow image of the object 62 can be detected. As described above, the sensor built-in liquid crystal panel 301 can also detect the touched position by detecting the shadow image.

  As described above, the optical sensor 6 may detect reflected light (shadow image) of the light emitted from the backlight 307 or may detect a shadow image caused by external light. Further, both the two types of detection methods may be used together to detect both a shadow image and a reflection image.

(Data display / sensor configuration)
Next, the configuration of the main part of the data display / sensor device 100 will be described with reference to FIG. FIG. 4 is a block diagram showing a main configuration of the data display / sensor device 100. As shown in FIG. As illustrated, the data display / sensor device 100 includes one or more display / light sensor units 300, a circuit control unit 600, a data processing unit 700, a main control unit 800, a storage unit 901, a primary storage unit 902, and an operation unit. 903, an external communication unit 907, an audio output unit 908, and an audio input unit 909. Here, although the data display / sensor device 100 is described as including a single display / light sensor unit 300, a plurality of data display / sensor devices 100 may be included.

  The display / light sensor unit 300 is a so-called liquid crystal display device with a built-in light sensor. The display / light sensor unit 300 includes a sensor built-in liquid crystal panel 301, a backlight 307, and a peripheral circuit 309 for driving them.

  The sensor built-in liquid crystal panel 301 includes a plurality of pixel circuits 31 and photosensor circuits 32 arranged in a matrix. The detailed configuration of the sensor built-in liquid crystal panel 301 will be described later.

  The peripheral circuit 309 includes a liquid crystal panel drive circuit 304, an optical sensor drive circuit 305, a signal conversion circuit 306, and a backlight drive circuit 308.

  The liquid crystal panel driving circuit 304 outputs a control signal (G) and a data signal (S) in accordance with the timing control signal (TC1) and the data signal (D) from the display control unit 601 of the circuit control unit 600, and the pixel circuit 31. It is a circuit which drives. Details of the driving method of the pixel circuit 31 will be described later.

  The optical sensor driving circuit 305 is a circuit that drives the optical sensor circuit 32 by applying a voltage to the signal line (R) in accordance with a timing control signal (TC2) from the sensor control unit 602 of the circuit control unit 600. Details of the driving method of the optical sensor circuit 32 will be described later.

  The signal conversion circuit 306 is a circuit that converts the sensor output signal (SS) output from the optical sensor circuit 32 into a digital signal (DS) and transmits the converted signal to the sensor control unit 602.

  The backlight 307 includes a plurality of white LEDs (Light Emitting Diodes) and is disposed on the back surface of the sensor built-in liquid crystal panel 301. When a power supply voltage is applied from the backlight drive circuit 308, the backlight 307 is turned on and irradiates the sensor built-in liquid crystal panel 301 with light. Note that the backlight 307 is not limited to white LEDs, and may include LEDs of other colors. The backlight 307 may include, for example, a cold cathode fluorescent lamp (CCFL) instead of the LED.

  When the control signal (BK) from the backlight control unit 603 of the circuit control unit 600 is at a high level, the backlight drive circuit 308 applies a power supply voltage to the backlight 307, and conversely, the backlight control unit When the control signal from 603 is at a low level, no power supply voltage is applied to the backlight 307.

  Next, the circuit control unit 600 will be described. The circuit control unit 600 has a function as a device driver that controls the peripheral circuit 309 of the display / light sensor unit 300. The circuit control unit 600 includes a display control unit 601, a sensor control unit 602, a backlight control unit 603, and a display data storage unit 604.

  The display control unit 601 receives display data from the display data processing unit 701 of the data processing unit 700, and performs timing control on the liquid crystal panel driving circuit 304 of the display / light sensor unit 300 in accordance with an instruction from the display data processing unit 701. A signal (TC1) and a data signal (D) are transmitted, and the received display data is displayed on the sensor built-in liquid crystal panel 301.

  The display control unit 601 temporarily stores the display data received from the display data processing unit 701 in the display data storage unit 604. Then, a data signal (D) is generated based on the primary stored display data. The display data storage unit 604 is, for example, a video random access memory (VRAM).

  The sensor control unit 602 transmits a timing control signal (TC2) to the optical sensor driving circuit 305 of the display / optical sensor unit 300 in accordance with an instruction from the sensor data processing unit 703 of the data processing unit 700, and the sensor built-in liquid crystal panel 301. Run the scan with.

  In addition, the sensor control unit 602 receives a digital signal (DS) from the signal conversion circuit 306. Then, image data is generated based on the digital signal (DS) corresponding to the sensor output signal (SS) output from all the optical sensor circuits 32 included in the sensor built-in liquid crystal panel 301. That is, the image data read in the entire reading area of the sensor built-in liquid crystal panel 301 is generated. Then, the generated image data is transmitted to the sensor data processing unit 703.

  The backlight control unit 603 transmits a control signal (BK) to the backlight drive circuit 308 of the display / light sensor unit 300 in accordance with instructions from the display data processing unit 701 and the sensor data processing unit 703 to drive the backlight 307. Let

  When the data display / sensor device 100 includes a plurality of display / light sensor units 300, the display control unit 601 determines which display / light sensor unit 300 displays the display data from the data processing unit 700. When an instruction is received, the liquid crystal panel drive circuit 304 of the display / light sensor unit 300 is controlled according to the instruction. When the sensor control unit 602 receives an instruction from the data processing unit 700 as to which display / light sensor unit 300 is to scan the object, the sensor control unit 602 responds to the light of the display / light sensor unit 300 according to the instruction. The sensor drive circuit 305 is controlled and a digital signal (DS) is received from the signal conversion circuit 306 of the display / light sensor unit 300 according to the instruction.

  Next, the data processing unit 700 will be described. The data processing unit 700 has a function as middleware that gives an instruction to the circuit control unit 600 based on a “command” received from the main control unit 800. Details of the command will be described later.

  The data processing unit 700 includes a display data processing unit 701 and a sensor data processing unit 703. When the data processing unit 700 receives a command from the main control unit 800, at least one of the display data processing unit 701 and the sensor data processing unit 703 depends on the value of each field (described later) included in the received command. One works.

  The display data processing unit 701 receives display data from the main control unit 800, and gives instructions to the display control unit 601 and the backlight control unit 603 according to the command received by the data processing unit 700, and displays the received display data. The image is displayed on the sensor built-in liquid crystal panel 301. The operation of the display data processing unit 701 according to the command will be described later.

  The sensor data processing unit 703 gives an instruction to the sensor control unit 602 and the backlight control unit 603 according to the command received by the data processing unit 700.

  The sensor data processing unit 703 receives image data from the sensor control unit 602 and stores the image data in the image data buffer 704 as it is. Then, in accordance with the command received by the data processing unit 700, the sensor data processing unit 703 performs “whole image data”, “partial image data (partial image coordinate data) based on the image data stored in the image data buffer 704. At least one of “including coordinate data” and “coordinate data” is transmitted to the main control unit 800. The whole image data, partial image data, and coordinate data will be described later. The operation of the sensor data processing unit 703 according to the command will be described later.

  Next, the main control unit 800 executes an application program. The main control unit 800 reads the program stored in the storage unit 901 into a primary storage unit 902 configured by, for example, a RAM (Random Access Memory) and executes the program.

  An application program executed by the main control unit 800 causes the data processing unit 700 to display display data on the sensor built-in liquid crystal panel 301 and to scan an object on the sensor built-in liquid crystal panel 301. Command and display data. When “data type” is designated as a command, at least one of whole image data, partial image data, and coordinate data is received from the data processing unit 700 as a response to the command.

  The circuit control unit 600, the data processing unit 700, and the main control unit 800 can be configured by a CPU (Central Processing Unit), a memory, and the like, respectively. The data processing unit 700 may be configured by a circuit such as an ASIC (application specific integrate circuit).

  Next, the storage unit 901 stores programs and data executed by the main control unit 800 as shown in the figure. The program executed by the main control unit 800 may be separated into an application-specific program and a general-purpose program that can be shared by each application.

  Next, the operation unit 903 receives an input operation of the user of the data display / sensor device 100. The operation unit 903 includes input devices such as a switch, a remote controller, a mouse, and a keyboard, for example. Then, the operation unit 903 generates a control signal corresponding to the user's input operation of the data display / sensor device 100, and transmits the generated control signal to the main control unit 800.

  As an example of the switch, a hardware switch such as a power switch 905 that switches power on and off and a user switch 906 to which a predetermined function is assigned in advance is assumed.

  In addition, the data display / sensor device 100 includes an external communication unit 907 for communicating with an external device by wireless / wired communication, an audio output unit 908 such as a speaker for outputting audio, and a microphone for inputting an audio signal. A voice input unit 909 such as the above may be provided as appropriate.

(Command details)
Next, details of commands transmitted from the main control unit 800 to the data processing unit 700 will be described with reference to FIGS. 5 and 6. FIG. 5 is a diagram schematically illustrating an example of a command frame structure. FIG. 6 is a diagram for explaining an example of values that can be specified for each field included in the command and an outline thereof.

  As shown in FIG. 5, the commands are “header”, “data acquisition timing”, “data type”, “scan method”, “scan image gradation”, “scan resolution”, “scan panel”, “display panel”. "And" Reserve "fields. In each field, for example, values shown in FIG. 6 can be designated.

  The “header” field is a field indicating the start of a frame. As long as it is possible to identify the “header” field, the value of the “header” field may be any value.

  Next, the “data acquisition timing” field is a field for designating a timing at which data should be transmitted to the main control unit 800. In the “data acquisition timing” field, for example, values “00” (sense), “01” (event), and “10” (all) can be specified.

  Here, “sense” specifies that the latest data is to be transmitted immediately. Therefore, when the sensor data processing unit 703 receives a command whose value in the “data acquisition timing” field is “sense”, the latest data specified in the “data type” field is immediately updated to the main control unit 800. Send to.

  The “event” designates transmission at a timing when a change occurs in the image data received from the sensor control unit 602. Therefore, when the sensor data processing unit 703 receives a command whose value in the “data acquisition timing” field is “event”, the image that receives the data specified in the “data type” field from the sensor control unit 602. The data is transmitted to the main control unit 800 at a timing when a change larger than a predetermined threshold occurs.

  “All” designates data transmission at a predetermined cycle. Therefore, when the sensor data processing unit 703 receives a command whose value in the “data acquisition timing” field is “all”, the data designated in the “data type” field is transferred to the main control unit 800 at a predetermined cycle. Send to. The predetermined period coincides with the period in which the optical sensor circuit 32 performs scanning.

  Next, the “data type” field is a field for designating the type of data acquired from the sensor data processing unit 703. In the “data type” field, for example, values of “001” (coordinates), “010” (partial image), and “100” (entire image) can be specified. Furthermore, by adding these values, “coordinates” and “partial image” / “whole image” can be specified simultaneously. For example, when “coordinate” and “partial image” are specified at the same time, “011” can be specified.

  When the sensor data processing unit 703 receives a command whose value of the “data type” field is “whole image”, the sensor data processing unit 703 transmits the image data itself stored in the image data buffer 704 to the main control unit 800. The image data itself stored in the image data buffer 704 is referred to as “whole image data”.

  In addition, when the sensor data processing unit 703 receives a command whose value of the “data type” field is “partial image”, the sensor data processing unit 703 selects a portion where a change larger than a predetermined threshold has occurred from the image data received from the sensor control unit 602. A region to be included is extracted, and image data of the extracted region is transmitted to the main control unit 800. Here, the image data is referred to as “partial image data”. When a plurality of partial image data are extracted, the sensor data processing unit 703 transmits the extracted partial image data to the main control unit 800.

  Further, when the sensor data processing unit 703 receives a command whose value of the “data type” field is “partial image”, the sensor data processing unit 703 detects representative coordinates in the partial image data as the coordinates of the point to which the object approaches. Then, coordinate data indicating the position of the representative coordinate in the partial image data is transmitted to the main control unit 800. The representative coordinates include, for example, the coordinates of the center of the partial image data, the coordinates of the center of gravity of the partial image data, and the like.

  Next, when receiving a command whose value of the “data type” field is “coordinate”, the sensor data processing unit 703 transmits coordinate data indicating the position of the representative coordinate in the entire image data to the main control unit 800. When a plurality of partial image data are extracted, the sensor data processing unit 703 detects representative coordinates in the entire image data of the extracted partial image data, and the coordinate data indicating the representative coordinates is detected. Each is transmitted to the main control unit 800 (multi-point detection).

  Specific examples of the whole image data, the partial image data, and the coordinate data will be described later with reference to schematic diagrams.

  Next, the “scan method” field is a field for designating whether or not the backlight 307 is turned on at the time of executing the scan. In the “scan method” field, for example, values of “00” (reflection), “01” (transmission), and “10” (reflection / transmission) can be designated.

  “Reflection” specifies that scanning is performed with the backlight 307 turned on. Therefore, when the sensor data processing unit 703 receives a command whose “scan method” field value is “reflection”, the sensor data processing unit 703 performs sensor control so that the optical sensor driving circuit 305 and the backlight driving circuit 308 operate in synchronization. An instruction is given to the unit 602 and the backlight control unit 603.

  “Transmission” specifies that scanning is performed with the backlight 307 turned off. Therefore, when the sensor data processing unit 703 receives a command whose “scan method” field value is “transparent”, the sensor control unit 602 operates the optical sensor driving circuit 305 and does not operate the backlight driving circuit 308. Instructions to the backlight control unit 603. Note that “reflection / transmission” specifies that scanning is performed using both “reflection” and “transmission”.

  Next, the “scanned image gradation” field is a field for designating gradations of the partial image data and the entire image data. In the “scanned image gradation” field, for example, values of “00” (binary) and “01” (multivalue) can be designated.

  When the sensor data processing unit 703 receives a command whose “scan image gradation” field value is “binary”, the sensor data processing unit 703 transmits the partial image data and the entire image data to the main control unit 800 as monochrome data. .

  When the sensor data processing unit 703 receives a command whose “scanned image gradation” field value is “multivalued”, the sensor data processing unit 703 transmits the partial image data and the entire image data to the main control unit 800 as multitone data. To do.

  Next, the “scan resolution” field is a field for designating the resolution of the partial image data and the entire image data. In the “resolution” field, for example, values of “0” (high) and “1” (low) can be designated.

  Here, “high” designates a high resolution. Therefore, when the sensor data processing unit 703 receives a command whose “scan resolution” field value is “high”, the sensor data processing unit 703 transmits the partial image data and the entire image data to the main control unit 800 with high resolution. For example, it is desirable to designate “high” for image data (image data such as a fingerprint) to be subjected to image processing such as image recognition.

  “Low” designates a low resolution. Therefore, when the sensor data processing unit 703 receives a command whose “scan resolution” field value is “low”, the sensor data processing unit 703 transmits the partial image data and the entire image data to the main control unit 800 at a low resolution. For example, it is desirable to designate “low” for image data (such as touched finger or hand image data) that only needs to be recognized.

  Next, in the “scan panel” field, when the data display / sensor device 100 includes a plurality of display / light sensor units 300, which display / light sensor unit 300 performs scanning of the object. This field is specified. In the “scan panel” field, for example, values “001” (first display / light sensor unit 300) and “010” (second display / light sensor unit 300) can be designated. By adding these values, a plurality of display / light sensor units 300 can be specified at the same time. For example, when both the first and second display / light sensor units 300 are specified at the same time, “011” can be specified.

  Here, the sensor data processing unit 703 controls the photo sensor driving circuit 305 and the backlight driving circuit 308 of the designated display / light sensor unit 300 according to the value of the “scan panel” field of the received command. An instruction is given to the sensor control unit 602 and the backlight control unit 603.

  Next, the “display panel” field specifies which display / light sensor unit 300 displays the display data when the data display / sensor device 100 includes a plurality of display / light sensor units 300. It is a field to be. In the “scanned image gradation” field, for example, values of “001” (first display / light sensor unit 300) and “010” (second display / light sensor unit 300) can be designated. By adding these values, a plurality of display / light sensor units 300 can be specified at the same time. For example, when both the first and second display / light sensor units 300 are specified at the same time, “011” can be specified.

  Here, for example, when the display data processing unit 701 receives a command whose value of the “display panel” field is the display / light sensor unit 300, the display / light sensor unit 300 displays the display / light sensor unit 300 to display the display data. An instruction is given to the display control unit 601 and the backlight control unit 603 so as to control the liquid crystal panel driving circuit 304 and the backlight driving circuit 308 of the optical sensor unit 300.

  Next, the “reserved” field is a field that is appropriately specified when it is necessary to further specify information other than information that can be specified in the above-described fields.

  Note that an application executed by the main control unit 800 does not need to use all the above-described fields when transmitting a command, and an invalid value (such as a NULL value) may be set for a field that is not used. .

  When the user wants to acquire coordinate data of a position touched with a finger or pen, a command specifying “coordinate” in the “data type” field is transmitted to the data processing unit 700. Therefore, it is desirable to specify “all” in the “data acquisition timing” field of the command to acquire coordinate data. Further, since it is only necessary to acquire coordinate data of the touched position, the scanning accuracy may not be high. Therefore, “low” may be specified as the value of the “resolution” field of the command.

  Further, when “coordinate” is specified in the “data type” field of the command, for example, when the user touches the sensor built-in liquid crystal panel 301 with a plurality of fingers or pens at the same time, the coordinate data of the touched position is used. Can be acquired (multi-point detection).

  When acquiring image data of an object such as a document, a command specifying “whole image” in the “data type” field is transmitted to the data processing unit 700. Since it is common to perform a scan in a stationary state, it is not necessary to periodically perform the scan. Therefore, in this case, it is desirable to designate “sense” or “event” in the “data acquisition timing” field. When scanning an object such as a document, it is desirable that the scanning accuracy is high so that the user can easily read the characters. Therefore, it is desirable to designate “high” in the “resolution” field.

(Whole image data / Partial image data / Coordinate data)
Next, the whole image data, the partial image data, and the coordinate data will be described with reference to FIGS. 7A and 7B. The image data shown in FIG. 7A is image data obtained as a result of scanning the entire sensor built-in liquid crystal panel 301 when the object is not placed on the sensor built-in liquid crystal panel 301. The image data shown in FIG. 7B is image data obtained as a result of scanning the entire sensor-equipped liquid crystal panel 301 when the user touches the sensor-equipped liquid crystal panel 301 with a finger.

  When the user touches the sensor built-in liquid crystal panel 301 with a finger, the amount of light received by the photosensor circuit 32 in the vicinity of the touch changes, so that the voltage output from the photosensor circuit 32 changes, and as a result, In the image data generated by the sensor control unit 602, the brightness of the pixel value of the portion touched by the user changes.

  In the image data shown in FIG. 7B, the brightness of the pixel value of the portion corresponding to the user's finger is higher than that in the image data shown in FIG. In the image data shown in FIG. 7B, the smallest rectangular area (area PP) that includes all pixel values whose lightness changes more than a predetermined threshold is “partial image data”.

  The image data indicated by the area AP is “whole image data”.

  Also, the coordinate data in the whole image data (area AP) of the representative coordinates Z of the partial image data (area PP) is (Xa, Ya), and the coordinate data in the partial image data (area PP) is (Xp, Yp). It is.

(Configuration of sensor built-in liquid crystal panel)
Next, the configuration of the sensor built-in liquid crystal panel 301 and the configuration of the peripheral circuit 309 of the sensor built-in liquid crystal panel 301 will be described with reference to FIG. FIG. 8 is a block diagram showing the main part of the display / light sensor unit 300, particularly the configuration of the sensor built-in liquid crystal panel 301 and the configuration of the peripheral circuit 309.

  The sensor built-in liquid crystal panel 301 includes a pixel circuit 31 for setting light transmittance (brightness) and an optical sensor circuit 32 that outputs a voltage corresponding to the intensity of light received by the sensor. The pixel circuit 31 is used as a general term for the R pixel circuit 31r, the G pixel circuit 31g, and the B pixel circuit 31b corresponding to the red, green, and blue color filters, respectively.

  The pixel circuits 31 are arranged on the sensor built-in liquid crystal panel 301 in the column direction (vertical direction) and 3n in the row direction (horizontal direction). A set of the R pixel circuit 31r, the G pixel circuit 31g, and the B pixel circuit 31b is continuously arranged in the row direction (lateral direction). This set forms one pixel.

  In order to set the light transmittance of the pixel circuit 31, first, the high level voltage (TFT 33 is turned on) to the scanning signal line Gi connected to the gate terminal of the TFT (Thin Film Transistor) 33 included in the pixel circuit 31. Voltage). Thereafter, a predetermined voltage is applied to the data signal line SRj connected to the source terminal of the TFT 33 of the R pixel circuit 31r. Similarly, the light transmittance is also set for the G pixel circuit 31g and the B pixel circuit 31b. Then, by setting these light transmittances, an image is displayed on the sensor built-in liquid crystal panel 301.

  Next, the photosensor circuit 32 is arranged for each pixel. One pixel may be arranged in the vicinity of each of the R pixel circuit 31r, the G pixel circuit 31g, and the B pixel circuit 31b.

  In order for the optical sensor circuit 32 to output a voltage corresponding to the light intensity, first, the sensor readout line RWi connected to one electrode of the capacitor 35 and the anode terminal of the photodiode 36 are connected. A predetermined voltage is applied to the sensor reset line RSi. In this state, when light is incident on the photodiode 36, a current corresponding to the amount of incident light flows through the photodiode 36. Then, according to the current, the voltage at the connection point (hereinafter referred to as connection node V) between the other electrode of the capacitor 35 and the cathode terminal of the photodiode 36 decreases. When the power supply voltage VDD is applied to the voltage application line SDj connected to the drain terminal of the sensor preamplifier 37, the voltage at the connection node V is amplified and output from the source terminal of the sensor preamplifier 37 to the sensing data output line SPj. Based on the output voltage, the amount of light received by the optical sensor circuit 32 can be calculated.

  Next, the liquid crystal panel drive circuit 304, the optical sensor drive circuit 305, and the sensor output amplifier 44, which are peripheral circuits of the sensor built-in liquid crystal panel 301, will be described.

  The liquid crystal panel drive circuit 304 is a circuit for driving the pixel circuit 31, and includes a scanning signal line drive circuit 3041 and a data signal line drive circuit 3042.

  The scanning signal line driving circuit 3041 sequentially selects one scanning signal line from the scanning signal lines G1 to Gm for each line time based on the timing control signal TC1 received from the display control unit 601, and A high level voltage is applied to the selected scanning signal line, and a low level voltage is applied to the other scanning signal lines.

  Based on the display data D (DR, DG, and DB) received from the display controller 601, the data signal line driver circuit 3042 generates a predetermined voltage corresponding to the display data for one row for each line time. The data signal lines SR1 to SRn, SG1 to SGn, and SB1 to SBn are applied (line sequential method). Note that the data signal line driver circuit 3042 may be driven by a dot sequential method.

  The optical sensor driving circuit 305 is a circuit for driving the optical sensor circuit 32. Based on the timing control signal TC2 received from the sensor control unit 602, the optical sensor driving circuit 305 selects a predetermined sensor readout signal line from the sensor readout signal lines RW1 to RWm for each line time. A read voltage is applied, and a voltage other than a predetermined read voltage is applied to the other sensor read signal lines. Similarly, based on the timing control signal TC2, a predetermined reset voltage is applied to the sensor reset signal line selected from the sensor reset signal lines RS1 to RSm for each line time, and the others. A voltage other than a predetermined reset voltage is applied to the sensor reset signal line.

  The sensing data output signal lines SP1 to SPn are grouped into p groups (p is an integer of 1 to n), and the sensing data output signal lines belonging to each group are connected via a switch 47 that is sequentially turned on in time division. And connected to the sensor output amplifier 44. The sensor output amplifier 44 amplifies the voltage from the group of sensing data output signal lines connected by the switch 47 and outputs the amplified voltage to the signal conversion circuit 306 as sensor output signals SS (SS1 to SSp).

(Configuration of main control unit and data stored in storage unit)
Next, the configuration of the main control unit 800 and various data stored in the storage unit 901 will be described with reference to FIG.

  FIG. 9 is a block diagram showing details of the main control unit 800 and the storage unit 901 in the main configuration of the data display / sensor device 100.

  As illustrated, the main control unit 800 includes a scan processing unit (imaging control unit) 12, a color determination processing unit (color detection unit) 14 A, and an image display processing unit (color generation control unit) 18.

  The storage unit 901 includes three storage areas: a scan attribute information storage unit 22, an image data storage unit 23, and a primary color image data storage unit 24. The scan attribute information storage unit 22 includes two storage areas, that is, a contact part inclusion area determination command storage part 222 and a color detection command storage part 223. The image data storage part 23 is used for specifying a contact part inclusion area. The primary color image data storage unit 24 includes a data storage unit 241 and a color determination result storage unit (storage unit) 242. The two recording areas are included.

  The image display processing unit 18 includes a primary color image creation unit (color generation control unit) 181, and primary color image data created by the primary color image creation unit 181 (hereinafter, for example, an R pixel circuit 31 r, a G pixel circuit 31 g, and a B pixel). When the circuit 31b is sequentially colored, the R image display data, G image display data, and B image display data are displayed / displayed via the data processing unit 700 (and the circuit control unit 600). Each of the pixel circuits 31 included in the sensor built-in liquid crystal panel 301 is transmitted to the optical sensor unit 300 and sequentially developed with a first color, a second color, and a third color that satisfy the principle of color matching.

  Specifically, for example, when the first color is developed, the entire primary color image data created by the primary color image creation unit 181 may be data for only the first color. Further, when the second color is developed, the entire primary color image data may be data of only the second color, and when the third color is developed, the entire primary color image data may be the data of only the third color.

  Here, the “equal color principle” is a scientific term in chromatics, and is the principle that all colors can be expressed in three colors. Specifically, the three primary colors of colors, the three primary colors of light, and the three colors (secondary colors, etc.) selected every four along the ring on the 12-color ring are preferable.

  Further, in order to detect the color of the pen tip of the pen P, it is only necessary to irradiate the pen P with light having three different colors and detect the reflected light based on the principle of color matching.

  For example, when the color of the pen tip of the pen P is red, the pen tip absorbs red light. Therefore, when red, green, and blue light is irradiated to the pen tip, reflected light of green and blue light is detected, but reflected light of red light is not detected at all, or detected light. Is less than the other two colors. Thereby, it can be determined that the pen tip is red.

  In addition, the “predetermined range” is a broad concept including, for example, the entire predetermined display surface of the liquid crystal panel 301 with a built-in sensor and a partial specific range (for example, the region PP) of the display surface. In the embodiment, for the sake of convenience, the description will be made assuming that the predetermined range = the region PP.

  Further, “coloring with the first color, the second color, and the third color” means that the first color is developed first, then the second color is developed, and then the third color is developed. . In addition, it is preferable that the switching speed from the specific color of the first color, the second color, and the third color to another color is as high as possible from the viewpoint of detection sensitivity.

  The pixel circuit 31 may be configured to develop the first color, the second color, and the third color by, for example, collecting a collection of color pixels that can develop each of the first color, the second color, and the third color. The circuit 31 may be used.

  Further, as the color pixels, as in the R pixel circuit 31r, the G pixel circuit 31g, and the B pixel circuit 31b of this embodiment, a combination of three primary color filters and liquid crystal is used, and light from the backlight 307 is transmitted to the three primary color filters. For example, a color pixel that does not use a backlight and emits (colors) that color without using a backlight, such as a color pixel that uses an LED element. It may be.

  As described below, the primary color image creation unit 181 sets the pixel circuit 31 to the first color when the color determination unit (color detection unit) 141 determines (detects) the pen tip color of the pen P. The series of operations for generating the second color and the third color is completed.

  According to this, until the color determination unit 141 detects the pen tip color of the pen P, the primary color image creation unit 181 performs an operation of causing the pixel circuit 31 to develop the first color, the second color, and the third color. Since it continues, the color of the pen point of the pen P can be detected reliably.

  In addition, once the color is detected, the primary color image creating unit 181 ends the coloring operation, and thus it is possible to prevent a useless operation such as detecting the color in an overlapping manner.

  Further, the image display processing unit 18 includes a color determination result display image creation unit 182, which is detected by a color determination processing unit (color detection unit) 14 </ b> A described below in a region PP where the pen tip of the pen P contacts. A color determination result display image for displaying in the same color as the pen tip color of the pen P (for example, displaying the locus L shown in FIG. 1) is also created.

  As an example of the first color, the second color, and the third color that satisfy the principle of color matching, the first color may be any one of the three colors consisting of red (R), green (G), and blue (B). The second color is one of the two colors that are not the first of the three colors, and the third color is the remaining one of the two colors that is not the second color. it can.

  As another example of the first color, the second color, and the third color that satisfy the principle of equal colors, the first color is any one of red, green, and blue, and three colors that are white. The second color is any one of the three colors that are not the first color, and the third color is the remaining one of the two colors that is not the second color. The case can be illustrated.

  As a result, a group of R pixels (R pixel circuit 31r), G pixels (G pixel circuit 31g), and B pixels (B pixel circuit 31b) generally used in the image display device is converted into a unit pixel (pixel circuit 31). The present invention can be applied to an image display device.

  In order to color the pixel circuit 31 in white, all of the R pixel circuit 31r, the G pixel circuit 31g, and the B pixel circuit 31b may be colored. Specifically, the entire primary color image data created by the primary color image creation unit 181 may be white data.

  The scan processing unit 12 includes a scan execution unit (imaging control unit) 131, and the scan execution unit 131 passes through the data processing unit 700 (and the circuit control unit 600), and the optical sensor circuit 32 of the display / photosensor unit 300. When the image display processing unit 18 causes the pixel circuit 31 included in the region PP to be colored with the first color, the second color, and the third color, the pen P that contacts the sensor built-in liquid crystal panel 301 is operated. The image is also scanned (captured) using each optical sensor 6.

  In addition, the scan processing unit 12 includes a contact part inclusion region specifying unit (contact determination unit, contact part inclusion region specifying unit) 136A, and the pen tip of the pen P is a sensor built-in liquid crystal panel 301 (sensor built-in liquid crystal panel 301A and It is also determined whether or not the sensor-incorporated liquid crystal panel 301B is touching.

  Here, whether or not the pen tip of the pen P is in contact with the sensor built-in liquid crystal panel 301 is determined, for example, by the contact part inclusion region specifying unit 136A in which the “data acquisition timing” of the command described above is “event”. If it is set, it is determined that the pen P has touched the sensor built-in liquid crystal panel 301, triggered by the transmission of “partial image data” or the like from the data processing unit 700 to the main control unit 800. It ’s fine.

  Conversely, when the “data acquisition timing” of the command is set to “event”, the contact part inclusion region specifying unit 136A displays the “partial image data” or the like for a predetermined time at all. It is only necessary to determine that the pen P has moved away from the sensor built-in liquid crystal panel 301 by using the fact that the pen P is not transmitted to the main control unit 800 as a trigger.

  According to this, when the contact portion inclusion area specifying unit 136A determines that the pen P is in contact with the sensor built-in liquid crystal panel 301, the pixel circuit 31 is caused to perform the color developing operation, and therefore, when the color cannot be detected. In addition, useless operations such as performing an operation for color detection can be prevented.

  In the present embodiment, the case where the contact part inclusion region specifying unit 136A has both the contact determination function described above and the contact part inclusion region specifying function described below will be described. You may comprise by a separate functional block.

  The scan data storage unit 133 scans (captures) captured image data including at least partial image data and representative coordinates (hereinafter simply referred to as “scan data”), as a scan data / contact part inclusion region specifying result holding unit 134, Control is performed to store the data in the contact part inclusion region specifying data storage unit 231 of the image data storage unit 23 and the color determination data storage unit 241 of the primary color image data storage unit 24.

  The scan processing unit 12 includes a scan attribute information setting unit 135. The scan attribute information setting unit 135 reads out various command data recorded in the scan attribute information storage unit 22, and displays the data display / sensor device. 100 scan attributes are set.

  Here, based on FIG. 17C, another problem that the configuration for developing the first color, the second color, and the third color has will be described.

  FIG. 17C is a conceptual diagram showing a conventional color scanner in which R, G, and B lights are sequentially irradiated onto a color image and an image is captured by an image sensor.

  For example, Patent Documents 5 and 6 disclose scanners in which a liquid crystal is provided between a light source and an original as a modification of the conventional color scanner, and the color of the light source is changed by the liquid crystal to capture the color.

  Note that the techniques disclosed in Patent Documents 5 and 6 neither describe nor suggest the viewpoint of performing color detection using pixels for image display. Further, another optical system element such as a lens is used, and the viewpoint of miniaturization of the apparatus is not taken into consideration.

  Here, in the case of a color scanner, image data of a document is taken in a state where the document is placed on the scan screen, so even if the R, G, and B lights are sequentially irradiated on the scan screen, The light is hidden from the manuscript or the like and is hardly visible to the user.

  However, in the case of detecting the color of the contact object touching the display surface of the input / output integrated display (planar member), the state where the user is viewing the state where the image is displayed on the display surface, In view of detecting the color of the contact object, when the first color, the second color, and the third color are sequentially developed, another problem of flickering in the display occurs.

  Therefore, in order to solve such a problem, the contact part inclusion region specifying unit (contact determination unit, contact part inclusion region specifying unit) 136A of the scan processing unit 12 has a built-in sensor based on the scanned image of the pen P. A predetermined region PP including the pen tip of the pen P on the liquid crystal panel 301 is specified.

  Further, when the contact part inclusion region specifying unit 136A specifies the region PP, the result is stored in the scan data / contact part inclusion region specifying result holding unit 134 or the contact part inclusion region specifying result storage unit 232 of the image data storage unit 23. It comes to memorize.

  The scan data / contact part inclusion area specifying result holding unit 134 is a part for temporarily storing each data. On the other hand, the contact part inclusion region specifying result storage unit 232 of the image data storage unit 23 may temporarily store each data or may be configured to always store the data.

  Further, the image display processing unit 18 causes only the pixel circuits 31 included in the region PP specified by the contact part inclusion region specifying unit 136A to develop color.

  Specifically, for example, when the region PP is colored with the first color, the portion corresponding to the region PP of the primary color image data created by the primary color image creation unit 181 is set as the data of only the first color, and the rest Colorless data may be used (however, when application data such as a moving image is displayed, the image display processing unit 18 sets an application image in which a portion corresponding to the area PP of the application image data is colorless data. It is preferable to create data and send it superimposed on the primary color image data (the same applies hereinafter).

  In addition, when the region PP is colored with the second color, the portion corresponding to the region PP of the primary color image data may be the data for the second color only, and the other portion may be the colorless data. In addition, when the region PP is colored with the third color, the portion corresponding to the region PP of the primary color image data may be the data for the third color only, and the other portion may be the colorless data.

  According to this, the image display processing unit 18 causes the unit pixels included in the region PP (contact portion inclusion region = predetermined range) on the sensor built-in liquid crystal panel 301 to color.

  Here, the region PP may be the region of the contact portion of the pen tip of the pen P, or may be a closed region of any size including the contact portion and the vicinity thereof. From the viewpoint of reducing the influence on the user by flickering, it is preferable to make it as small as possible.

  Thus, by reducing the size of the region PP as much as possible within the range where color detection is possible, the influence on the user due to the flickering of the display can be reduced.

  That is, since the pen P is in contact with the area PP, the display flicker in the area PP can be hidden from the pen P and hardly seen by the user.

  As described above, the color of the pen tip of the pen P in contact with the display surface of the sensor-embedded liquid crystal panel 301, which is useful for downsizing the device, can be displayed to the user due to display flickering while preventing a decrease in sensitivity and resolution. Thus, it is possible to provide the data display / sensor device 100 that can be detected with less influence.

  Next, the color determination processing unit (color detection unit) 14 </ b> A includes a color determination unit 141 and a color determination result holding unit (color detection unit) 145, and the color determination unit 141 is a color of the primary color image data storage unit 24. Each time the first color, the second color, and the third color stored in the determination data storage unit 241 are developed, the scan data scanned (imaged) using each photosensor 6 is read out, and this is read out. Analysis (analysis) is performed to determine (detect) the color of the pen tip of the pen P.

  The color determination processing unit 14A includes a color determination unit 141. The color determination data analysis unit (color detection unit) 142 of the color determination unit 141 includes the image display processing unit 18 that includes pixels included in the region PP. Each time the circuit 31 is developed with the first color, the second color, and the third color, the scan data of the image of the pen P scanned by the scan processing unit 12 is analyzed. The color of the pen tip of the pen P on the sensor built-in liquid crystal panel 301 is specified (detected).

  As described above, if the first color, the second color, and the third color are sequentially developed and the pen tip color of the pen P is detected, the pen tip color of the pen P is temporarily the first color. Even when the colors are the same, the color can be detected based on the scan data obtained by scanning the remaining second and third colors.

  Here, from the viewpoint of improving the accuracy of color detection, the first color, the second color, and the third color are developed at once, and one scan data obtained by scanning the mixed state of these three colors is analyzed. Rather than performing color detection, the first color, the second color, and the third color are sequentially developed for the three colors that satisfy the same color principle, and the first color, the second color, and the third color are developed. It is preferable to prepare three pieces of scan data scanned using each optical sensor 6 and analyze these three pieces of scan data.

  This is because, rather than performing color detection by analyzing one scan data obtained by scanning a state in which the first color, the second color, and the third color that satisfy the principle of equal colors are mixed, the first color, the second color, and the second color This is because it is much easier to perform color detection by analyzing the three scan data in a state where each of the three colors is separated, and the detection accuracy is high.

  According to the above, for example, even when the photosensor 6 is provided for each pixel circuit 31, each pixel circuit 31 is equivalent to three colors of the first color, the second color, and the third color by one photosensor 6. Scan data can be obtained separately.

  From the viewpoint of detection accuracy of color detection, it is preferable to arrange the photosensor 6 for each color pixel of the R pixel circuit 31r, the G pixel circuit 31g, and the B pixel circuit 31b, but if the above configuration is adopted, the photosensor Since the detection accuracy of color detection is relatively high even if the number of 6 is small, the current sensor area (for example, the number of pixel circuits 31 of the present embodiment: the number of photosensors 6 = 1) and the detection of color detection A decrease in detection sensitivity can be prevented while ensuring accuracy.

  Further, if the above configuration is adopted, in order to increase the detection sensitivity, it is not necessary to develop one color using all the light emitted from the three color pixels in the pixel circuit 31. Degradation of resolution can also be prevented.

  The color determination result storage unit 143 stores the color determination result of the color determination unit 141 in the color determination result storage unit (storage unit) 145 or the color determination result storage unit (storage unit) 242 of the primary color image data storage unit 24. It is something to be made.

  The color determination result holding unit 145 temporarily stores the color determination result of the color determination unit 141. Alternatively, the color determination result storage unit 242 of the primary color image data storage unit 24 may temporarily store the color determination result of the color determination unit 141 or may store it constantly.

  According to this, since the color information indicating the color of the pen tip of the pen P detected by the color determination unit 141 is held, it is not necessary to redetect the color of the same contact object.

  In addition, about the timing which hold | maintains color information, it is preferable immediately after the color determination part 141 detects the color of a contact thing.

  Further, as a period for holding the color information, the pen P comes into contact (for example, the contact part inclusion region specifying unit 136A determines that the pen P is in contact), and moves away after the color is detected (for example, contact). The period until the partial inclusion region specifying unit 136A determines that the pen P is not in contact can be exemplified.

  Further, the retained color can be appropriately used for necessary processing. For example, if it is configured to be able to draw on the display surface with the color of the detected pen tip of the pen P, electronic drawing can be performed with the same feeling as drawing with color pencils on drawing paper.

  Here, based on FIG. 11, the detail of the structure of the color determination result display image preparation part 182 for performing the electronic drawing mentioned above is demonstrated.

  FIG. 11 is a block diagram illustrating each configuration of the color determination result display image creation unit 182. As shown in FIG. 11, the color determination result display image creation unit 182 mainly includes a contact position specification unit (contact position specification unit) 1821, an interpolation line specification unit (interpolation line specification unit) 1822, and a display image generation unit (detection color). Display control means) 1823.

  The contact position specifying unit 1821 specifies the position (contact position) where the pen tip of the pen P is detected on the sensor built-in liquid crystal panel 301 of the pen P based on each image of the pen P scanned by the optical sensor 6. (To detect).

  The interpolation line specifying unit 1822 specifies the first position specified by the contact position specifying unit 1821 when the second contact position is specified after the contact position specifying unit 1821 specifies the first contact position. An interpolation line for interpolating between the first contact position and the second contact position is specified.

  The display image creation unit 1823 creates a color detection result display image for displaying the first contact position, the second contact position, and the interpolation line.

  Accordingly, the first contact position and the second contact position can be smoothly connected and displayed on the display surface.

  Although details are not described in the present embodiment, as interpolation methods applicable to the present invention, zero-order interpolation (nearest neighbor interpolation, nearest neighbor interpolation), linear interpolation (linear interpolation, first-order interpolation), and cubic interpolation ( For example, cubic interpolation).

  Further, when three or more contact positions are specified, an appropriate interpolation function may be interpolated. For example, as an example of an interpolation method using an interpolation function, Lagrange interpolation, spline interpolation, interpolation using a Sinc function, Lanczos interpolation (Lanzosh interpolation) and the like can be exemplified.

  As described above, data that can detect the color of the pen tip of the pen P that is in contact with the display surface of the sensor-equipped liquid crystal panel 301 while preventing reduction in sensitivity and deterioration in resolution, which is useful for downsizing the device. A display / sensor device 100 can be provided.

(Relationship between various commands and data display / sensor device operation)
Here, the relationship between various commands in the data display / sensor device 100 and the operation of the data display / sensor device 100 will be described.

  In the present embodiment, the name of the command related to the contact determination and the contact part inclusion area specifying process of the data display / sensor device 100 is referred to as a “contact part inclusion area determination command” and is stored in the contact part inclusion area determination command storage unit 222 in advance. Explain that it is.

  The name of the command related to the color determination process and the color determination result display process of the data display / sensor device 100 is referred to as a “color detection command” and is stored in the color detection command storage unit 223 in advance.

  The “contact part inclusion area determination command” is a command for mainly setting the scan attribute of the scan processing unit 12 for the contact determination process and the contact part inclusion area specifying process.

  The “color detection command” is a command for mainly setting scan attributes of the scan processing unit 12 for color determination processing and color determination result display processing.

  First, in the “contact part inclusion region determination command”, the value of the “data acquisition timing” field shown in FIG. 6 is designated as “01” (event).

  This is because, as described above, the “event” designates transmission of image data or the like from the data processing unit 700 to the main control unit 800 at the timing when the image data received from the sensor control unit 602 has changed. Therefore, by using this, it is possible to perform a contact determination process and a contact part inclusion region specifying process.

  Specifically, if the value of the “data acquisition timing” field is “event”, the image data received from the sensor control unit 602 is changed by a touch of the pen P at a timing when a change larger than a predetermined threshold value occurs. It is transmitted to the main control unit 800.

  Next, “010” (partial image) is preferably designated as the value of the “data type” field of the “contact determination command”.

  This is because, as described above, when the sensor data processing unit 703 receives a command whose value of the “data type” field is “partial image”, the change from the image data received from the sensor control unit 602 is greater than a predetermined threshold value. This is because it is possible to extract a region (for example, the region PP) including the portion where the occurrence occurs and to transmit image data (partial image data and representative coordinates) of the extracted region to the main control unit 800.

  In the following, for convenience, the case where the area occupied by the “partial image” is directly used as the area PP will be described. An area having a size different from that of the “partial image” area around the representative coordinates stored in the storage unit 231 may be used as the area PP.

  When a plurality of partial image data are extracted, the sensor data processing unit 703 transmits the extracted partial image data to the main control unit 800.

  Furthermore, when the sensor data processing unit 703 receives a command whose value of the “data type” field is “partial image”, the sensor data processing unit 703 represents the representative coordinates in the partial image data as the coordinates of the point where the pen tip of the pen P contacts. Is detected, and coordinate data indicating the position of the representative coordinate in the partial image data is transmitted to the main control unit 800.

  The scan data storage unit 133 stores “partial image data” and “representative coordinates” in the contact part inclusion region specifying data storage unit 231 of the image data storage unit 23.

  The representative coordinates include, for example, the coordinates of the center of the partial image data, the coordinates of the center of gravity of the partial image data, and the like.

  Next, “00” (reflection) is designated as the value of the “scan method” field of the “contact determination command”. This is for irradiating the pen tip of the pen P with light that the pixel circuit 31 develops and detecting the reflected light.

  Next, either “00” (binary) or “01” (multivalue) may be designated as the value of the “scanned image gradation” field of the “contact determination command”. In addition, the value of the “scan resolution” field of the “contact determination command” may specify either “0” (high) or “1” (low). This is because it is sufficient if the region PP can be specified, and it is sufficient to scan with binary gradation. Note that the value of the “scan resolution” field is preferably set to “0” (high) from the viewpoint of the specific accuracy of the region PP.

  On the other hand, in the color detection process of the “color detection command”, “00” (sense) is designated as the value of the “data acquisition timing” field shown in FIG. In the color detection process of the “color detection command”, the value of the “data acquisition timing” field shown in FIG. 6 is designated as “01” (event) in the detection color display process.

  This is because “sense” designates that the latest data is to be immediately transmitted from the data processing unit 700 to the main control unit 800, whereby the image display processing unit 18 causes the pixel circuit included in the region PP to be transmitted. Since the scan data scanned by the scan processing unit 12 can be transmitted to the main control unit 800 every time the color 31 is sequentially developed with the first color, the second color, and the third color, color detection is thereby performed. This is because processing is possible.

  On the other hand, “event” designates transmission of image data or the like from the data processing unit 700 to the main control unit 800 at a timing when the image data received from the sensor control unit 602 changes. This is because the position of the current pen tip can be specified by using and detection color display processing can be performed according to the specified position.

  Specifically, if the value of the “data acquisition timing” field is “event”, the image data received from the sensor control unit 602 is transmitted to the main control unit 800 following the movement of the pen P.

  Next, it is preferable to specify “010” (partial image) as the value of the “data type” field of the “color detection command” in both the color detection process and the detected color display process.

  This is because, in the color detection processing, flickering of display can be prevented by causing only a portion included in a region (or region PP) occupied by partial image data on all image data to be colored with a specific color.

  Further, in the detection color display process, it is sufficient if the contact position of the pen P can be specified.

  When a plurality of partial image data are extracted, the sensor data processing unit 703 transmits the extracted partial image data to the main control unit 800.

  Furthermore, when the sensor data processing unit 703 receives a command whose value of the “data type” field is “partial image”, the sensor data processing unit 703 represents the representative coordinates in the partial image data as the coordinates of the point where the pen tip of the pen P contacts. Is detected, and coordinate data indicating the position of the representative coordinate in the partial image data is transmitted to the main control unit 800.

  The representative coordinates include, for example, the coordinates of the center of the partial image data, the coordinates of the center of gravity of the partial image data, and the like.

  The scan data storage unit (imaging control unit) 133 touches each of the “partial image data” and “representative coordinates” with the color determination result storage unit 145 or the color determination data storage unit 241 of the primary color image data storage unit 24. The data is stored in the partial inclusion area specifying data.

  Next, the value of the “scan method” field of the “color detection command” designates “00” (reflection) in both the color detection process and the detected color display process. This is for irradiating the pen P with light that the pixel circuit 31 develops and detecting the reflected light.

  Next, “01” (multivalue) is designated as the value of the “scanned image gradation” field of the “color detection command” in the color detection process. This is because in order to enable color detection, it is necessary to scan with multi-value gradation.

  In the detected color display process, “00” (binary) may be used. This is because it is sufficient if the contact position of the pen P can be specified.

  In addition, the value of the “scan resolution” field of the “color detection command” may specify either “0” (high) or “1” (low), but from the viewpoint of color detection accuracy, It is preferably set to 0 "(high).

  Next, details of the relationship between the operation of the data display / sensor device 100 and the “contact determination command” and “color detection command” will be described.

  First, in a state where the pen tip of the pen P is not in contact with the sensor built-in liquid crystal panel 301, the scan attribute information setting unit 135 displays the “contact part inclusion region” stored in advance from the contact part inclusion region determination command storage unit 222. It is assumed that the “determination command” is read and the field value of “data acquisition timing” is set to “event”.

  Here, when the pen tip of the pen P comes into contact with the sensor built-in liquid crystal panel 301, as described above, the “data acquisition timing” is set to “event”. The sensor data processing unit 703 recognizes that the image data received from the sensor control unit 602 has changed, and transmits the scan data obtained to the main control unit 800.

  The contact portion inclusion area specifying unit (contact determination means, contact part inclusion area specifying means) 136A is triggered by the fact that the scan data is transmitted to the main control unit 800, and the pen tip of the pen P is the sensor built-in liquid crystal panel 301. It is determined that the contact has been made.

  Since the field value of the “data acquisition timing” of the command is set to “event”, the scan data storage unit 133 stores the obtained scan data as contact portion inclusion region specifying data as contact portion inclusion region specifying data storage Store in the unit 231.

  After that, the scan data storage unit 133 notifies the contact part inclusion region specifying unit 136A that the scan data is stored in the contact part inclusion region specifying data storage unit 231. The contact part inclusion region specifying unit 136A The area PP is specified based on the scan data recorded in the contact part inclusion area specifying data storage unit 231. In this case, the area occupied on the whole image data of the partial image data included in the scan data may be used as the area PP as it is, or an area having a predetermined size centered on the representative coordinates included in the scan data is set as the area PP. Also good.

  Thereafter, the contact part inclusion region specifying unit 136A notifies the primary color image creating unit 181 of the image display processing unit 18 that the region PP has been specified.

  Upon receiving the above notification, the primary color image creation unit 181 drives the scan attribute information setting unit 135, and the scan attribute information setting unit 135 reads a “color detection command” stored in advance from the color detection command storage unit 223. The field value of “data acquisition timing” is switched to “sense”.

  The primary color image creation unit 181 creates primary color image data from the first color to the third color, and sequentially transmits the primary color image data to the display / light sensor unit 300 via the data processing unit 700 (and the circuit control unit 600). Send.

  Thereafter, since the field value of “data acquisition timing” is set to “sense”, the scan data storage unit 133 has the sensor built-in liquid crystal panel 301 every time the first color, the second color, and the third color are developed. Scan data obtained by scanning the image of the pen P in contact with the optical sensor 6 by using each optical sensor 6 is received from the data processing unit 700.

  Since the field value of “data acquisition timing” of the command is set to “sense”, the scan data storage unit 133 stores the data as color determination data in the color determination data storage unit 241.

  When the color determination data is stored in the color determination data storage unit 241, the scan data storage unit 133 notifies the color determination data analysis unit 142 to that effect.

  In response to the notification, the color determination data analysis unit 142 reads the color determination data from the color determination data storage unit 241, analyzes the color determination data, and specifies the pen tip color of the pen P ( The color determination result is stored in the color determination result storage unit 242 as color information.

  Further, when the color determination data analysis unit 142 stores the color information in the color determination result storage unit 242, the color determination data analysis unit 142 notifies the scan attribute information setting unit 135 to that effect.

  Next, upon receiving the above notification, the scan attribute information setting unit 135 reads the “color detection command” stored in advance from the contact part inclusion region determination command storage unit 222 and sets the field value of “data acquisition timing” to “ Switch to “Event”.

  Here, since “data acquisition timing” is set to “event” as described above, when the pen tip of the pen P moves, the sensor data processing unit 703 converts the image data received from the sensor control unit 602 into image data. Recognizing that a change has occurred, the scan data obtained is transmitted to the main control unit 800.

  The scan data storage unit 133 stores the obtained scan data in the image data storage unit 23 as contact position specifying data.

  Thereafter, the scan data storage unit 133 notifies the contact position specifying unit 1821 that the scan data has been stored in the image data storage unit 23, and the contact position specifying unit 1821 is recorded in the image data storage unit 23. The contact position of the contact object is specified based on the scanned data. In the present embodiment, the coordinate data (Xa, Ya) in the whole image data (area AP) of the representative coordinate Z of the partial image data (area PP) described above is specified.

  Thereafter, the contact position specifying unit 1821 notifies the interpolation line specifying unit 1822 that the contact position of the contact object has been specified.

  When the interpolation line specifying unit 1822 receives the above notification and there is one contact position specified by the contact position specifying unit 1821, the interpolation line specifying unit 1822 stores the contact position in the image data storage unit 23, and displays that fact. The image creation unit 1823 is notified.

  On the other hand, the interpolation line specifying unit 1822 specifies the first position specified by the contact position specifying unit 1821 when the second contact position is specified after the contact position specifying unit 1821 specifies the first contact position. An interpolation line for interpolating between one contact position and the second contact position is specified, and all data relating to the first contact position, the second contact position, and the interpolation line are stored in the image data storage unit 23. This is notified to the display image creation unit 1823.

  The display image creation unit 1823 receives the notification and displays the contact position or the contact position specifying result display image for displaying the first contact position, the second contact position, and the interpolation line. Is supposed to create.

  The color used for displaying the contact position or the first contact position, the second contact position, and the interpolation line is a detected (determined) color.

  Accordingly, the first contact position and the second contact position can be smoothly connected and displayed on the display surface.

(Data display / sensor operation)
Next, based on FIGS. 10-13, the detail of operation | movement of the data display / sensor apparatus 100 is demonstrated.

  FIG. 10 is a schematic diagram conceptually showing the color detection operation of the data display / sensor device 100. FIG. 10 (a) shows the time passage of the color detection operation, and FIG. 10 (b) shows the color detection operation. The state of the screen display during the detection operation is shown.

  First, as shown in FIG. 10A, when it is determined that the pen tip of the pen P is in contact (pen touch) with the sensor built-in liquid crystal panel 301, the primary color of the primary color image creating unit 181 is determined from the position indicated by the white arrow. Image data creation starts.

Thereafter, R image display data, G image display data, and B image display data are sequentially generated. Here, R pixel lighting period T _R is, there is pen-touched, the time from the display of the R image display data to the display is completed, G pixel lighting period T _G is displayed G image display data time until the display is completed after being, B pixel lighting period T _B is the time from the B image display data is displayed until the display is completed.

Here, time T = R pixel lighting period T _{R +} G pixel lighting period T _{G +} B pixel lighting period T _B of the coloring operation, in terms of display flicker prevention, as short as possible is preferable.

  Further, as shown in FIG. 10B, the acquisition of the partial image data (region PP) and the representative coordinates is regularly performed in the scan processing unit 12 (scan data for determining the contact portion inclusion region). However, since partial image data and representative coordinates can be acquired even with scan data for color detection, coordinate detection is possible), but color detection processing for detecting the color of the pen tip of the pen P (with the color of the pen tip) The color development operation of the R image display data, the G image display data, and the B image display data in (detect coordinates) immediately ends when the color of the pen P of the pen P is determined by the color determination unit 141. Return to the contact determination process (coordinate detection).

  In FIG. 10, the process immediately returns to the contact determination process after the end of one color detection process. However, if the color cannot be detected for some reason, the color detection process is performed again.

  Next, the flow of the contact part inclusion region specifying step will be described based on the flowchart shown in FIG. FIG. 12 is a flowchart showing the operation of the data display / sensor device 100 (contact part inclusion region specifying step).

  As shown in FIG. 12, when the power switch of the data display / sensor device 100 is turned on and the display of the sensor built-in liquid crystal panel 301 is turned on, the scan attribute information setting unit 135 is stored in the scan attribute information storage unit 22 in advance. The value of each field of the “contact part inclusion region determination command” is read and the scan attribute information is set, and the process starts.

  In step S <b> 1 (hereinafter, “step” is omitted), the sensor control unit 602 causes the optical sensor 6 to pass through the optical sensor driving circuit 305 according to the “contact portion inclusion region determination command” sent from the scan execution unit 131. , And the operation of accumulating (holding) the scan data (including the entire image data here) in the image data buffer 704 of the data processing unit 700 is repeated at a predetermined timing TC2.

  Next, when the user brings the pen P, which is a specific color pencil, into contact with the sensor built-in liquid crystal panel 301B in S2, the process proceeds to S3.

  In S3, the sensor data processing unit 703 sets the field value of “data acquisition timing” of the “contact part inclusion region determination command” sent from the scan execution unit 131 to “01” (event), so that the sensor When an area including a portion having a change larger than a predetermined threshold is extracted from the image data received from the control unit 602 from the image data received from the sensor control unit 602 by the touch of the pen P, the extraction is performed. The image data (partial image data and representative coordinates) of the selected area is transmitted to the main control unit 800, "YES" is determined, and the process proceeds to S4.

  On the other hand, if a region including a portion having a change larger than the predetermined threshold value is not extracted from the image data received from the sensor control unit 602 due to the touch of the pen P in S3, “NO” is obtained, and the sensor data The processing unit 703 returns to the operation of S1.

  In S4, since the field value of “data acquisition timing” of the “contact part inclusion region determination command” is set to “01” (event), the scan data storage unit 133 performs main control from the sensor data processing unit 703. The scan data (partial image data and representative coordinates) placed in the part 800 is stored (held) in the contact part inclusion region specifying data storage part 231 and the process proceeds to S5.

  In S5, since the field value of “data acquisition timing” of the “contact part inclusion region determination command” is set to “01” (event), the contact part inclusion region specifying unit 136A determines from the sensor data processing unit 703. Triggered by sending scan data (partial image data and representative coordinates) to the main control unit 800, it is determined that “the pen P has touched the sensor built-in liquid crystal panel 301B”, and the region PP is specified (contact portion inclusion region) Specific processing) and proceed to “connector A”.

  In the contact part inclusion region specifying process, the contact part inclusion region specifying unit 136A may specify the region occupied by all the image data of the partial image data included in the scan data as the region PP, or the representative part included in the scan data. Based on the coordinates, an area having a predetermined size centered on the representative coordinates may be specified as the area PP.

  Hereinafter, a case will be described in which the contact part inclusion region specifying unit 136A specifies a region occupied by all the image data of the partial image data included in the scan data as the region PP.

  Next, FIG. 13 is a flowchart showing the operation (color detection step) of the data display / sensor device 100.

  From “connector A”, the process proceeds to S6. In S6, if the color determination data analysis unit 142 has already performed color determination by the contact portion inclusion area specifying unit 136A, “YES” is determined, and the process proceeds to S10. .

  On the other hand, if the color determination data analysis unit 142 has not performed color determination in S6, “NO” is determined, and the process proceeds to S7.

  In S7, the contact part inclusion region specifying unit 136A notifies the image display processing unit 18 that the region PP has been specified.

  Upon receiving the above notification, the image display processing unit 18 drives the primary color image creation unit 181. The primary color image creation unit 181 reads the scan data from the contact part inclusion region specifying data storage unit 231 and reads all the images in the region PP. Creates R image display data, G image display data, and B image display data in which only the area on the data satisfies R, G, and B display satisfying the principle of color matching, and sequentially transmits the data to the data processing unit 700 To do. Thereby, each of the pixel circuits 31 included in the region PP of the sensor built-in liquid crystal panel 301 is sequentially colored (displayed) with R, G, and B satisfying the color matching principle, and the process proceeds to S8.

  In S <b> 8, the primary color image creation unit 181 drives the scan attribute information setting unit 135, and the scan attribute information setting unit 135 reads the “color detection command” stored in advance from the color detection command storage unit 223, Switch the field value of “Acquisition Timing” to “All”.

  Thereafter, since the field value of “data acquisition timing” is set to “all”, the scan data storage unit 133 sets the color of the pen P in contact with the sensor built-in liquid crystal panel 301 every time R, G, and B are developed. Scan data obtained by scanning an image using each optical sensor 6 is received from the data processing unit 700.

  When the “data acquisition timing” field value of the command is set to “all”, the scan data storage unit 133 stores the data as color determination data in the color determination data storage unit 241. At this time, the scan data storage unit 133 uses the light sensor 6 to display the image of the pen P in contact with the sensor built-in liquid crystal panel 301 every time the color determination data storage unit 241 makes color development with R, G, and B. When all the three scan data scanned in this manner are stored, the color determination processing unit 14A is notified of this and the process proceeds to S9.

  In S <b> 9, the color determination processing unit 14 </ b> A receives the notification and drives the color determination unit 141, and the color determination unit 141 performs R based on the three scan data stored in the color determination data storage unit 241. , G, and B are analyzed for the intensity of reflected light to determine the color of the pen tip of the pen P in contact with the sensor built-in liquid crystal panel 301B, and the color determination result (color information) is output. Thereafter, the color determination result storage unit 143 stores the output color determination result in the color determination result holding unit 145 or the color determination result storage unit 242 and proceeds to S10.

  In S10, the color determination unit 141 of the color determination processing unit 14A notifies the contact position specifying unit 1821 of the color determination result display image creation unit 182 that the color determination result has been output.

  Upon receiving the above notification, the contact position specifying unit 1821 receives the above notification, and scan data (partial image data and representative coordinates) stored in the contact part inclusion region determination command storage unit 222 or the color determination data storage unit 241 in S4. Based on the representative coordinates, the contact position of the pen P is specified, the contact position specification result is transmitted to the interpolation line specifying unit 1822, and the process proceeds to S11.

  In S11, when the contact position specifying result specifies two or more contact positions, “YES” is determined, and the process proceeds to S12.

  On the other hand, when the contact position specifying result specifies only one contact position in S11, “NO” is determined, and the process proceeds to S14.

  In S12, the interpolation line specifying unit 1822 specifies each interpolation line between the sequentially specified contact positions, and the process proceeds to S13.

  In S13, the display image creation unit 1823 creates display image data in which the two or more specified contact positions and all the interpolation lines between the sequentially specified contact positions are displayed in the detected color. Proceed to S15.

  In S14, display image data displayed in the color in which one specified contact position is detected is created, and the process proceeds to S15.

  In S15, the display image creation unit 1823 transmits the created display image data to the display / light sensor unit 300 via the data processing unit 700, and the drawn image of the detected color is displayed on the sensor built-in liquid crystal panel 301. Proceed to S16.

  In S16, the display image creation unit 1823 notifies the contact part inclusion region specifying unit 136A that the color drawing image has been displayed.

  Upon receiving the above notification, the contact part inclusion area specifying unit 136A determines whether or not the contact state of the pen P has been released. If the contact state of the pen P has been released, the data becomes “END”. The display / sensor device 100 ends the operation (or returns to “START” in FIG. 12).

  On the other hand, it is determined whether or not the contact state of the pen P has been released. If the contact state of the pen P has not been released, “YES” is determined, and the process returns to S4.

  As described above, the color of the pen tip of the pen P in contact with the display surface of the input / output integrated display (sensor-embedded liquid crystal panel 301), which is useful for downsizing of the apparatus, is prevented while lowering sensitivity and resolution. The data display / sensor device 100 that can be detected can be provided.

[Embodiment 2]
The following will describe another embodiment of the present invention with reference to FIGS. Configurations other than those described in the present embodiment are the same as those in the first embodiment. For convenience of explanation, members having the same functions as those shown in the drawings of the first embodiment are given the same reference numerals, and explanation thereof is omitted.

  In the present embodiment, a case will be described in which the functions of the display control unit 601 and the sensor control unit 602 in FIG. 4 are configured by hardware logic.

  FIG. 14 is a block diagram showing an example when the main configuration of the data display / sensor device 100 is implemented by hardware.

  As shown in FIG. 14, the main configuration of the present embodiment (referred to as data display / sensor device A) includes an optical sensor 6, a shift register (color development control means) 401, an inverter (color development control means) 402r, and an XNOR circuit (color development). Control means) 402g, inverter (coloring control means) 402b, AND circuit (imaging control means) 403, shift clock generator (coloring control means) 404, transistor (imaging control means) 405r, transistor (imaging control means) 405g and transistor (Imaging control means) 405b, transistor with an inverter (imaging control means) 406r, transistor with an inverter (imaging control means) 406g, transistor with an inverter (imaging control means) 406b, R pixel circuit 31r, G pixel circuit 31g and B pixel circuit 31b, coordinate calculation unit 136B, Calculation unit 14B, and the color memory (storage unit) 407R, a color memory (storage unit) 407 g and a color memory (storage unit) consisting 407b configuration.

  FIG. 14 shows a case where all of the R pixel circuit 31r, the G pixel circuit 31g, and the B pixel circuit 31b are lit (white display).

  The shift register (color generation control means) 401 sequentially outputs the information “11000000000000” one by one by driving the shift clock generator 404.

  In FIG. 14, “0” (LOW level) is output to the inverter 402r and the XNOR circuit 402g, and “1” (HIGH level) is output to the XNOR circuit 402g and the inverter 402b. .

  Since “0” is input, the inverter 402r outputs “1”. As a result, the R pixel circuit 31r is turned on.

  The XNOR circuit 402g outputs “1” because “00” is input. Thereby, the G pixel circuit 31g is turned on.

  The inverter 402b outputs “1” because “0” is input. Thereby, the B pixel circuit 31b is turned on.

  Thereby, the optical sensor 6 receives reflected light generated by irradiating the pen P with light generated by the color (light emission) of the R pixel circuit 31r, the G pixel circuit 31g, and the B pixel circuit 31b.

  Here, the AND circuit 403 outputs “1” because “111” is input. However, since “1” is input to all the gates of the transistors 405 r to 405 b, the transistors 405 r to 405 The gate at 405b is opened.

  However, in the transistor with inverter 406r to the transistor with inverter 406b, “1” from the AND circuit 403 is inverted by the inverter and “0” is input to the gate of the transistor. It does not flow to 407r to color memory 407b.

  That is, the state of FIG. 14 shows a state where the color detection process is not performed.

  Here, when the pen tip of the pen P comes into contact, the coordinate calculation unit 136B performs coordinate calculation based on the detection signal of the optical sensor 6 and outputs "coordinate information".

  At this time, the coordinate calculation unit 136B drives the shift clock generation unit 404. Then, the shift clock generation unit 404 drives the shift register 401 to realize the state shown in FIG.

  FIG. 15A is a diagram showing a state in which the R pixel is lit when the main configuration of the data display / sensor device A is implemented by hardware, and FIG. 15B is a diagram in which the G pixel is lit. FIG. 15C is a diagram illustrating a state in which the B pixel is lit.

  First, as shown in FIG. 15A, the shift register 401 is driven, and “0” is input to the inverter 402r. Therefore, the inverter 402r outputs “1”. As a result, the R pixel circuit 31r is turned on.

  Since “01” is input to the XNOR circuit 402g, the XNOR circuit 402g outputs “0”. Thereby, the G pixel circuit 31g is not lit.

  Further, since “1” is input to the inverter 402b, the inverter 402b outputs “0”. Thereby, the B pixel circuit 31b is not lit.

  At this time, the optical sensor 6 receives reflected light generated by irradiating the pen P with light generated by the R pixel circuit 31r to develop color (emit light).

  Here, since “100” is input to the AND circuit 403, the AND circuit 403 outputs “0”.

  Here, since “0” is input from the AND circuit 403 to the inverters 402r to 402b, all “1” s are input to the gates of the transistors 406r to 406b with inverters. The gate of the transistor 406b with the inverter 406b is opened.

  On the other hand, in the transistors 405r to 405b, since only “1” is input to the gates of the transistors 405r, only the gates of the transistors 405r are opened.

  Thereby, the detection information based on the detection signal of the optical sensor 6 is stored in the color memory 407r. Thereafter, the shift clock generation unit 404 drives the shift register 401 and proceeds to the state of FIG.

  In FIG. 15B, since the shift register 401 is driven and “1” is input to the inverter 402r, the inverter 402r outputs “0”. As a result, the R pixel circuit 31r is not lit.

  In addition, since “11” is input to the XNOR circuit 402g, the XNOR circuit 402g outputs “1”. As a result, the G pixel circuit 31g is turned on.

  Further, since “1” is input to the inverter 402b, the inverter 402b outputs “0”. Thereby, the B pixel circuit 31b is not lit.

  At this time, the optical sensor 6 receives reflected light generated by irradiating the pen P with light generated by the G pixel circuit 31g to develop color (emit light).

  Here, since “010” is input to the AND circuit 403, the AND circuit 403 outputs “0”.

  Here, since “0” is input from the AND circuit 403 to the inverters 402r to 402b, all “1” s are input to the gates of the transistors 406r to 406b with inverters. The gate of the transistor 406b with the inverter 406b is opened.

  On the other hand, in the transistors 405r to 405b, since only “1” is input to the gate of only the transistor 405g, only the transistor 405g is opened.

  Thereby, the detection information based on the detection signal of the optical sensor 6 is stored in the color memory 407g.

  Similarly, in FIG. 15C, since the shift register 401 is driven and “1” is input to the inverter 402r, the inverter 402r outputs “0”. As a result, the R pixel circuit 31r is not lit.

  Further, since “10” is input to the XNOR circuit 402g, the XNOR circuit 402g outputs “0”. Thereby, the G pixel circuit 31g is not lit.

  Since “0” is input to the inverter 402b, the inverter 402b outputs “1”. Thereby, the B pixel circuit 31b is turned on.

  At this time, the optical sensor 6 receives reflected light generated by irradiating the pen P with light generated by the color (light emission) of the B pixel circuit 31b.

  Here, since “001” is input to the AND circuit 403, the AND circuit 403 outputs “0”.

  Here, since “0” is input from the AND circuit 403 to the inverters 402r to 402b, all “1” s are input to the gates of the transistors 406r to 406b with inverters. The gate of the transistor 406b with the inverter 406b is opened.

  On the other hand, in the transistors 405r to 405b, since only “1” is input to the gate of only the transistor 405b, only the gate of the transistor 405b is opened.

  Thereby, the detection information based on the detection signal of the optical sensor 6 is stored in the color memory 407b.

  With the above-described series of operations from FIG. 15 (a) to FIG. 15 (c), the color calculation unit 14B is triggered by the detection information being recorded in all of the color memories 407r to 407b. Analyzes the pen tip color and outputs color information.

  As described above, the color of the pen tip of the pen P in contact with the display surface of the input / output integrated display (sensor-embedded liquid crystal panel 301), which is useful for downsizing of the apparatus, is prevented while lowering sensitivity and resolution. A data display / sensor device A that can be detected can be provided.

[Embodiment 3]
The following will describe still another embodiment of the present invention with reference to FIG. Configurations other than those described in the present embodiment are the same as those in the first and second embodiments. For convenience of explanation, members having the same functions as those shown in the drawings of Embodiments 1 and 2 are given the same reference numerals, and explanation thereof is omitted.

  FIG. 16 is a block diagram showing another example in which the main configuration of the data display / sensor device (referred to as data display / sensor device B) of the present embodiment is implemented by hardware.

  This embodiment is different from the second embodiment only in that the transistor 405b is not provided and a transistor (imaging control means) 406w is provided instead of the transistor with an inverter 406b. The configuration is the same as in the second embodiment. Since the operation is the same as that of the second embodiment, the description is omitted here, and the difference in operation between the second embodiment and the present embodiment will be described.

  In the data display / sensor device A according to the second embodiment, each light of R, G, and B sequentially emits light, and each light of R, G, and B is color as detection information detected by the optical sensor 6. It is recorded in the memory 407r to the color memory 407b.

  On the other hand, in the third embodiment, each of R, G, and white (all of R to G are lit) emits light sequentially, and the R, G, and white lights are detected by the optical sensor 6. Information is recorded in the color memory 407r to the color memory (storage unit) 407w.

  As described above, the color of the pen tip of the pen P in contact with the display surface of the input / output integrated display (sensor-embedded liquid crystal panel 301), which is useful for downsizing of the apparatus, is prevented while lowering sensitivity and resolution. The data display / sensor device B that can be detected can be provided.

  The present invention is not limited to the above-described embodiments, and various modifications can be made within the scope of the claims, and the technical means disclosed in different embodiments can be appropriately combined. Such embodiments are also included in the technical scope of the present invention.

  Finally, each block of the data display / sensor device 100, particularly the main control unit 800, may be configured by hardware logic, or may be realized by software using a CPU as follows.

  That is, the data display / sensor device 100 includes a CPU (central processing unit) that executes instructions of a control program that realizes each function, a ROM (read only memory) that stores the program, and a RAM (random access) that expands the program. memory), a storage device (recording medium) such as a memory for storing the program and various data. An object of the present invention is a record in which a program code (execution format program, intermediate code program, source program) of a control program for the data display / sensor device 100, which is software for realizing the functions described above, is recorded so as to be readable by a computer. This can also be achieved by supplying a medium to the data display / sensor device 100 and reading and executing the program code recorded on the recording medium by the computer (or CPU or MPU).

  Examples of the recording medium include a tape system such as a magnetic tape and a cassette tape, a magnetic disk such as a floppy (registered trademark) disk / hard disk, and a compact disk-ROM / MO / MD / digital video disk / compact disk-R. A disk system including an optical disk, a card system such as an IC card (including a memory card) / optical card, or a semiconductor memory system such as a mask ROM / EPROM / EEPROM / flash ROM can be used.

  The data display / sensor device 100 may be configured to be connectable to a communication network, and the program code may be supplied via the communication network. The communication network is not particularly limited. For example, the Internet, intranet, extranet, LAN, ISDN, VAN, CATV communication network, virtual private network, telephone line network, mobile communication network, satellite communication. A net or the like is available. Also, the transmission medium constituting the communication network is not particularly limited. For example, even in the case of wired such as IEEE 1394, USB, power line carrier, cable TV line, telephone line, ADSL line, etc., infrared rays such as IrDA and remote control, Bluetooth ( (Registered trademark), 802.11 wireless, HDR, mobile phone network, satellite line, terrestrial digital network, and the like can also be used. The present invention can also be realized in the form of a computer data signal embedded in a carrier wave in which the program code is embodied by electronic transmission.

  The present invention can be applied to a display device including a plurality of unit pixels and a plurality of imaging sensors. Specifically, as a display device, for example, it can be used for an active matrix liquid crystal display device, and an electrophoretic display, a twist ball display, a reflective display using a fine prism film, a digital mirror device, etc. In addition to displays using light modulation elements, organic EL light-emitting elements, inorganic EL light-emitting elements, displays using variable-light-emitting elements such as LEDs (Light Emitting Diodes), and field emission displays (FED) It can also be used for plasma displays.

It is a schematic diagram which shows the outline structure of the data display / sensor apparatus which concerns on one Embodiment of this invention, and the outline | summary of the function. (A) is a schematic diagram which shows an example of the cross section of the liquid crystal panel with a sensor with which the said data display / sensor apparatus is equipped, (b) is another example of the liquid crystal panel with a sensor with which the said data display / sensor apparatus is equipped. It is a schematic diagram which shows 1 pixel. (A) is a schematic diagram which shows a mode that the position which the user touched is detected by detecting a reflected image with the sensor built-in liquid crystal panel with which the said data display / sensor apparatus is equipped, (b) is the said data It is a schematic diagram which shows a mode that the position which the user touched is detected by detecting a shadow image with the sensor built-in liquid crystal panel with which a display / sensor apparatus is provided. It is a block diagram which shows the principal part structure of the said data display / sensor apparatus. It is a figure which shows typically an example of the frame structure of the command used with the said data display / sensor apparatus. It is a figure explaining an example of the value which can be specified to each field contained in the command shown in FIG. 5, and its outline. (A) is image data obtained as a result of scanning the entire sensor built-in liquid crystal panel when the object is not placed on the sensor built-in liquid crystal panel in the data display / sensor device, and (b) Is image data obtained as a result of scanning when the user touches the sensor built-in liquid crystal panel with a finger in the data display / sensor device. It is a block diagram which shows the structure of the liquid crystal panel with a sensor with which the said data display / sensor apparatus is provided, and the structure of its peripheral circuit. It is a block diagram which shows the detail of a main control part and a memory | storage part among the principal part structures of the said data display / sensor apparatus. FIG. 4 is a schematic diagram conceptually showing a color detection operation of the data display / sensor device, where (a) shows a lapse of time of the color detection operation, and (b) shows a screen display during the color detection operation. Show the state. It is a block diagram which shows the detail of a color determination result display image preparation part among the principal part structures of the said main-control part. It is a flowchart which shows the operation | movement (contact part inclusion area specific step) of the said data display / sensor apparatus. It is a flowchart which shows operation | movement (color detection step) of the said data display / sensor apparatus. It is a block diagram which shows an example at the time of mounting the principal part structure of the said data display / sensor apparatus by hardware. (A) is a figure which shows a mode that R pixel is lighting when the principal part structure of the said data display / sensor apparatus is mounted by hardware, (b) is a mode that G pixel is lighting. (C) is a figure which shows a mode that B pixel is lighted. It is a block diagram which shows the other example at the time of mounting the principal part structure of the said data display / sensor apparatus by hardware. (A) And (b) is a conceptual diagram for demonstrating the relationship of the magnitude | size of an image sensor and a color filter in the case of providing a color filter between a light source and an image sensor in the conventional color scanner. (C) is a figure which shows a mode that red, blue, and green LED (light-emitting diode) is lighted one by one in the conventional color scanner, and each light is scanned.

Explanation of symbols

6 Optical sensor (imaging sensor)
12 Scan processing unit (imaging control means)
14A Color determination processing unit (color detection means)
14B Color calculation unit (color detection means)
18 Image display processing unit (coloring control means)
22 Scan attribute information storage unit 23 Image data storage unit 24 Primary color image data storage unit 31 Pixel circuit (unit pixel)
31r R pixel circuit (color pixel, R pixel)
31g G pixel circuit (color pixel, G pixel)
31b B pixel circuit (color pixel, B pixel)
100 Data display / sensor device (color detection device)
131 Scan execution unit (imaging control means)
132 Scan execution instruction section (imaging control means)
133 Scan data storage (imaging control means)
134 Scan data / contact part inclusion area specifying result holding part 135 Scan attribute information setting part 136A Contact part inclusion area specifying part (contact determination means, contact part inclusion area specifying means)
136B Coordinate calculation part (contact determination means, contact part inclusion area specifying means)
141 Color determination unit (color detection means)
142 Color determination data analysis unit (color detection means)
143 Color determination result storage unit (color detection means)
145 Color determination result holding unit (storage unit)
181 Primary color image creation unit 182 Color determination result display image creation unit 222 Contact part inclusion region determination command 223 Color detection command 231 Contact part inclusion region specification data storage unit 232 Contact part inclusion region specification result storage unit 241 Color determination data storage unit 242 Color determination result storage unit (storage unit)
300 Display / light sensor unit 301 Liquid crystal panel with built-in sensor (planar member)
301A Sensor built-in liquid crystal panel (planar member)
301B Liquid crystal panel with built-in sensor (planar member)
401 shift register (color development control means)
402r inverter (coloring control means)
402g XNOR circuit (color control means)
402b Inverter (color development control means)
403 AND circuit (imaging control means)
404 Shift clock generator (color control means)
405r, 405g, 405b Transistor (imaging control means)
406r, 406g, 406b Transistor with inverter (imaging control means)
406w transistor (imaging control means)
407r, 407g, 407b Color storage (recording unit)
407w Color memory (recording part)
600 circuit control unit 601 display control unit 602 sensor control unit 603 backlight control unit 700 data processing unit 701 display data processing unit 703 sensor data processing unit 800 main control unit 901 storage unit 903 operation unit 1821 contact position specifying unit (contact position specifying unit) means)
1822 Interpolation line identification unit (interpolation line identification means)
1823 Display image creation unit (detected color display control means)
P Pen (Contact)
PP region (contact part inclusion region)
L locus H hinge T _R R pixel lighting period T _G G pixel lighting period T _B B pixel lighting period

Claims (9)

A planar member including a plurality of unit pixels and a plurality of imaging sensors provided at least one for each unit pixel;
Color development control means for sequentially developing the unit pixels included in the predetermined range in the first color, the second color, and the third color that satisfy the principle of equal colors,
Each time the color development control unit develops the unit pixels included in the predetermined range with the first color, the second color, and the third color, the image of the contact object that contacts the planar member is displayed on the imaging sensor. Imaging control means for imaging using
Each time the first color, the second color, and the third color are developed, based on each of the images of the contact object imaged by the imaging control unit, the contact portion of the contact object on the planar member is displayed. A color detection apparatus comprising color detection means for detecting a color. Contact determining means for determining whether or not the contact object is in contact with the planar member;
2. The color detection according to claim 1, wherein when the contact determination unit determines that the contact object is in contact with the planar member, the color generation control unit causes the unit pixel to color. 3. apparatus. A contact part inclusion region specifying unit that specifies a contact part inclusion region including a contact part of the contact object on the planar member based on an image of the contact object imaged by the imaging control unit;
3. The color according to claim 1, wherein the color development control unit causes each of the unit pixels to develop color using the contact part inclusion region specified by the contact part inclusion region specification unit as the predetermined range. Detection device. The first color is one of four colors consisting of red, green, blue and white,
The second color is one of the remaining three colors that are not the first color among the four colors.
4. The color detection device according to claim 1, wherein the third color is one of the remaining two colors that are not the second color among the three colors. 5. .   The color development control unit terminates the operation of coloring the unit pixel in the first color, the second color, and the third color when the color of the contact portion is detected by the color detection unit. The color detection apparatus according to any one of claims 1 to 4.   5. The color detection device according to claim 1, further comprising a storage unit that holds color information indicating a color of a contact portion of the contact object detected by the color detection unit. .   The color detection program for operating a computer as each means in the color detection apparatus of any one of Claim 1-6.   A computer-readable recording medium on which the color detection program according to claim 7 is recorded. A color detection method that is performed using a planar member that includes a plurality of unit pixels and a plurality of imaging sensors provided at least one for each unit pixel,
A coloring control step of sequentially coloring the unit pixels included in the predetermined range in the first color, the second color, and the third color that satisfy the principle of equal colors;
In the color development control step, each time the unit pixels included in the predetermined range are colored with the first color, the second color, and the third color, an image of a contact object that contacts the planar member is captured. An imaging control step of imaging using a sensor;
A contact portion on the planar member of the contact object based on each of the images of the contact object imaged in the imaging control step each time the first color, the second color, and the third color are developed. A color detection method comprising a color detection step of detecting the color of the color.

Images