WO2005122553A1 - Image i/o device - Google Patents

Abstract

There is provided an image I/O device including light source means for emitting light, space modulation means for subjecting the light emitted from the light source means to space modulation and outputting image signal light, projection means for projecting image signal light outputted by the space modulation means toward the projection direction, and imaging means capable of imaging an object existing at least in the projection direction and acquiring the imaged data. In the image I/O device, the imaging means and the projection means are arranged in such a manner that imaging direction of the imaging means can be modified with respect to the projection direction of the projection means so that an object existing in a direction different from the projection direction can be imaged by the imaging means.

Description

 Specification

 Image input / output device

 Technical field

 The present invention can image a subject existing in a direction in which image signal light is projected by a projection unit and a subject existing in a direction different from the direction in which image signal light is projected by a projection unit. The present invention relates to an image input / output device capable of displaying and outputting arbitrary information in directions.

 Background art

 [0002] Conventionally, in an image input / output device capable of capturing an image of a subject placed on a surface and acquiring data of the captured image, an image capturing device capable of capturing an image on the surface to visually display an imageable area. There is known a device provided with a projecting unit for projecting light onto a specific area. The projection means provided in such an image input / output device includes a liquid crystal panel that performs spatial modulation on light emitted from the light source. The projection unit controls the transmission of light emitted from the light source for each pixel by using the liquid crystal panel, and projects a spot light indicating the imaging area. One such type of image input / output device is described in JP-A-8-32848 (hereinafter referred to as Document 1).

 Disclosure of the invention

 [0003] However, in the image input / output device as described in Document 1, the relative positional relationship between the projection unit and the imaging unit is fixed, and the direction in which the image signal light is projected by the projection unit. Is configured to be able to take an image of only the object existing in the camera. Therefore, such an image input / output device has a problem that the use of the image input / output device is limited.

 [0004] For example, it is conceivable that the user points the imaging unit at himself and tries to image the user himself. In this case, the projection unit is used to set the direction not including the imaging direction of the subject, For example, the information cannot be displayed and output on a desk with the projection direction of the projection means.

[0005] The present invention has been made to solve the above problems. That is, The light can capture an image of a subject existing in the direction in which the image signal light is projected by the projection means and an object present in a direction different from the direction in which the image signal light is projected by the projection means. The purpose of the present invention is to provide an image input / output device capable of displaying and outputting arbitrary information.

 According to one aspect of the present invention, there is provided a light source unit that emits light, and a spatial modulation unit that spatially modulates light emitted from the light source unit and outputs image signal light. An image input device comprising: a projection unit for projecting the image signal light output from the spatial modulation unit in the projection direction; and an imaging unit capable of capturing at least a subject existing in the projection direction and acquiring data of the captured image. An output device is provided. In this image input / output device, the imaging unit and the projection unit may be configured so that the imaging direction of the imaging unit is different from the projection direction so that an object existing in a direction different from the projection direction can be imaged by the imaging unit. It is provided so as to be relatively changeable with respect to the projection direction of the means.

 [0007] According to such a configuration, the imaging direction of the imaging unit and the projection direction of the projection unit can be relatively changed to various directions desired by the user, and the desired image signal can be changed by the spatial modulation unit. Light can be output, and the image signal light is projected by the projection means in the projection direction. Therefore, desired information can be displayed and output by the projection means in a direction different from the imaging direction.

 [0008] Further, according to such a configuration, when light emitted from the light source means is spatially modulated by the spatial modulation means and output as desired image signal light, the desired image signal light is output. Light is projected by the projection means in the projection direction. Here, the imaging direction of the imaging means and the projection direction of the projection means can be relatively changed, and an arbitrary image signal light is projected in the projection direction by the projection means. It is possible to image a subject existing in the direction in which the image signal light is projected, and a subject existing in a direction different from the direction in which the image signal light is projected by the projection means, and to display and output desired information in the projection direction. There is an effect that can be.

 Brief Description of Drawings

[0009] [Fig. 1] Figs. 1 (a) and 1 (b) are external perspective views of an image input / output device, and Fig. 1 (a) is a diagram viewed from the top side. () Is an enlarged view of the imaging head. FIG. 2 (a) to FIG. 2 (e) are views showing the internal configuration of an imaging head.

Garden 3] FIG. 3 is a diagram schematically showing an electric configuration inside the image input / output device.

Garden 4] FIG. 4 (a) is an enlarged view of an image projection unit, FIG. 4 (b) is a plan view of a light source lens, and FIG. 4 (c) is a front view of a projection LCD.

 [FIG. 5] FIGS. 5 (a) to 5 (c) are views for explaining the arrangement of LED arrays. Garden 6] FIG. 6 is an electrical block diagram of the image input / output device.

FIG. 7 is a flowchart of a main process.

 8 (a) and 8 (b) are diagrams showing a relative positional relationship between an image projection unit and an image pickup unit.

 FIG. 9 is a flowchart of digital camera processing.

Garden 10] FIG. 10 is a diagram showing a state in which rectangular image light indicating an imageable area is projected on a surface.

 FIG. 11 is a flowchart of webcam processing.

 FIG. 12 is a diagram showing a state in which image output light is projected by an image projection unit in a projection process of webcam processing.

 FIG. 13 is a diagram showing a state in which image output light is projected by an image projection unit in a projection process of webcam processing.

 FIG. 14 is a flowchart of a projection process.

 FIG. 15 is a flowchart of stereoscopic image processing.

 [FIG. 16] FIG. 16 (a) is a diagram for explaining the principle of the spatial code method, and FIG. 16 (b) is a diagram showing a mask pattern (gray code) different from FIG. 16 (a). is there.

 FIG. 17 (a) is a flowchart of a three-dimensional shape detection process, FIG. 17 (b) is a flowchart of an imaging process, and FIG. 17 (c) is a flowchart of a three-dimensional measurement process.

 FIG. 18 is a diagram showing a state in which pattern light is projected when the relative positional relationship between the image projection unit and the image pickup unit is C mode.

Garden 19] FIG. 19 is a diagram for explaining the outline of the code boundary coordinate detection process.

Garden 20] is a flowchart of a code boundary coordinate detection process.

[FIG. 21] FIG. 21 is a flowchart of a process for obtaining code boundary coordinates with subpixel accuracy. is there.

 [FIG. 22] FIG. 22 is a flowchart of a process for obtaining a boundary CCDY value for a luminance image having a mask pattern number of PatID [i].

 FIG. 23 (a) to FIG. 23 (c) are diagrams for explaining a lens aberration correction process.

 FIGS. 24 (a) and 24 (b) are diagrams for explaining a method of calculating three-dimensional coordinates in a three-dimensional space from coordinates in a CCD space.

 FIG. 25 is a flowchart of flattening image processing.

 FIGS. 26 (a) to 26 (c) are diagrams for explaining a document orientation calculation process.

 FIG. 27 is a flowchart of a plane conversion process.

 FIG. 28 (a) is a diagram for explaining the outline of the curvature calculation processing.

(b) is a diagram showing a flattened image flattened by the plane conversion process.

 FIG. 29 is a flowchart of an image conversion process for distortion-free projection.

 FIG. 30 (a) is a side view showing a light source lens 50 of a second embodiment, and FIG. 30 (b) is a plan view showing a light source lens 60 of the second embodiment.

 FIG. 31 (a) is a perspective view showing a state in which the light source lens 50 is fixed, and FIG. 31 (b) is a partial sectional view thereof.

 FIG. 32 is a diagram showing another example of the pattern light projected on the subject.

 [FIG. 33] FIGS. 33 (a) to 33 (c) are views showing a modification of the image projection unit, and FIG. 33 (a) is a plan view of the imaging head, and FIG. 33 (b) and FIG. 33 (c) is a sectional view taken along line AA of FIG. 33 (a).

 BEST MODE FOR CARRYING OUT THE INVENTION

Hereinafter, preferred embodiments of the present invention will be described with reference to the accompanying drawings. FIG. 1A and FIG. 1B are external perspective views of an image input / output device 1 as an embodiment of the present invention. FIG. 1A is a view of the image input / output device 1 as viewed from above, and FIG. 1B is an enlarged view of the imaging head 2.

[0011] The image input / output device 1 includes a digital camera mode functioning as a digital camera, a webcam mode functioning as a web camera, a stereoscopic image mode for detecting a three-dimensional shape to obtain a three-dimensional image, a curved document. Flattening to obtain a flattened image obtained by flattening etc. Various modes such as an image mode are provided. The image input / output device 1 is a device capable of projecting an arbitrary image. Further, the image input / output device 1 is configured so as to be able to capture not only a subject located in a direction in which an image is projected but also a subject located in a direction different from a direction in which an image is projected.

 In FIG. 1A, in particular, in the stereoscopic image mode or the flattened image mode, in order to detect the three-dimensional shape of the subject P, a stripe pattern formed by alternately arranging light and dark from an image projection unit 13 described later is used. The pattern light (image signal light having a predetermined pattern) is projected and projected.

 The image input / output device 1 includes an imaging head 2 and an arm member 3 having one end detachably connected to the imaging head 2. Further, at the other end of the arm member 3, a laptop personal computer 49 (hereinafter simply referred to as “PC49”) is provided with a mounting portion 4 to which the imaging head 2 and the arm member 3 can be mounted. RU

 The imaging head 2 has a case that internally includes an image projection unit 13 described later, and that holds a cylindrical imaging case 11 rotatably around an axis outside thereof. On one surface of the imaging head 2, a cylindrical lens barrel 5 is disposed at the center thereof. In the following description, the surface on which the lens barrel 5 is provided is referred to as the front surface of the imaging head 2.

 The lens barrel 5 is a member that protrudes from the front of the imaging head 2 and includes a projection optical system 20 that is a part of the image projection unit 13 therein. The projection optical system 20 is held by the lens barrel 5. The lens barrel 5 allows the entire projection optical system 20 to move for focus adjustment. A part of the lens of the projection optical system 20, which is a part of the image projection unit 13, is exposed to the outside from the end surface of the lens barrel 5, and the image signal light is projected from this exposed part toward the projection surface.

 The imaging case 11 is provided with a white balance sensor 6 and a flash 7.

 Further, between the white balance sensor 6 and the flash 7, a part of the lens of the imaging optical system 21 which is a part of the image imaging unit 14 described later is exposed on the outer surface. A subject facing the imaging optical system 21 is imaged.

The flash 7 is a light source for supplementing necessary subject illuminance in the digital camera mode. The flash 7 is composed of, for example, a discharge tube filled with xenon gas. Flat The shoe 7 can be used repeatedly by discharging from a capacitor (not shown) built in the imaging head 2.

 A monitor LCD 10 is arranged on the back of the imaging head 2. The monitor LCD 10 is configured by a liquid crystal display (Liquid Crystal Display), and receives an image signal from a processor 15 described later and displays an image to a user. For example, the monitor LCD 10 displays a captured image in the digital camera mode or the webcam mode, a three-dimensional shape detection result image in the stereoscopic image mode, a flattened image in the flattened image mode, and the like.

 Further, on a side surface of the imaging head 2, a connecting member 12 for detachably connecting the imaging head 2 and the arm member 3 is arranged. The imaging head 2 of the image input / output device 1 can be used as a single mopile digital camera.

 The connecting member 12 is formed in a ring shape and is fixed to a side surface of the imaging head 2. Further, a portion to be fitted to the connecting member 12 is formed on one end side of the arm member 3. By this fitting, the imaging head 2 and the arm member 3 can be connected, and the imaging head 2 can be fixed to the PC 49 at an arbitrary angle. By releasing the fitting, the arm member 3 and the imaging head 2 can be separated.

 The arm member 3 can hold the imaging head 2 with respect to the PC 49 and can hold the imaging head with the orientation of the imaging head 2 in a desired direction with respect to the PC 49. The arm member 3 is formed of a bellows-shaped pipe that can be bent to a desired shape. Since the user can turn the imaging head 2 to a desired position, even if the imaging head 2 is attached to the PC 49, the user can determine the direction in which the image can be projected by the image input / output device 1 and the direction in which the image can be captured. Can be set as follows.

 The arm member 3 has one end connected to the imaging head 2 via the connecting member 12 as described above, and the other end provided for detachably attaching the arm member 3 to the PC 49. A mounting part 4 is provided.

[0023] The mounting portion 4 includes a battery 26, an electronic circuit board, and the like, which will be described later. Further, a release button 8, an on-Z off switch 9a, a mode switching switch 9b, and a connector of the personal computer interface 24 are provided on the front side of the mounting portion 4. The on / off switch 9a is located above the release button 8, and the mode switch 9b is It is provided below the close button 8. The image input / output device 1 is connected to a PC 49 via a cable (not shown) connected to the personal computer interface 24.

 The release button 8 is constituted by a two-stage push button switch that can be set to two states, a “half-pressed state” and a “fully-pressed state”. The state of the release button 8 is managed by a processor 15 described later. When the release button 8 is pressed halfway, the well-known auto focus (AF) and auto exposure (AE) functions are activated, and the focus, aperture, and shutter speed are adjusted. An image is taken with the release button 8 “fully pressed”.

 When the on-Z off switch 9a is pressed, the main power supply of the image input / output device 1 is switched on and off.

 The mode switching switch 9b is a switch that can be set to various modes such as a digital camera mode, a webcam mode, a stereoscopic image mode, a flattened image mode, and an off mode. The state of the mode switching switch 9b is managed by the processor 15. When the state of the mode switching switch 9b is detected by the processor 15, the processing of each mode is executed.

 FIGS. 2A to 2E show the internal configuration of the imaging head 2. FIG. As shown in FIG. 2A, the imaging head 2 has a lid 2a and a bottom 2b. The imaging case 11 is a cylindrical case that includes an image imaging unit 14 described below. On the outer peripheral surface of the imaging case 11, a part of the lens of the imaging optical system 21 which is a part of the image imaging unit 14 is exposed, and a plurality of ribs 11a are provided upright.

 FIG. 2B is a diagram showing a state in which the lid 2a of the imaging head 2 has been removed, and the image projection unit 13 included in the imaging head 2 has been removed. As shown in FIG. 2 (b), the bottom 2b is provided with an imaging head position detection sensor S1, an imaging unit position detection sensor S2, and a hole 2c.

 The imaging head position detection sensor S 1 is a two-axis acceleration sensor for detecting the inclination of the imaging head 2. The imaging head position detection sensor S1 uses two axes to determine how many times the projection direction of the image projection unit 13 fixed to the imaging head 2 (the optical axis direction of the projection optical system 20 described later) is inclined with respect to the vertical direction. Detect and output as electrical signal.

The imaging unit position detection sensor S2 is a sensor for detecting the position of the imaging unit 14 with respect to the imaging head 2. The hole 2c is a substantially circular hole formed on the surface constituting the front surface of the imaging head 2, and the hole 2c is threaded. The lens barrel 5 (see FIG. 1) is screwed into the hole 2c. As will be described later, the image projection unit 13 included in the imaging head 2 projects an image from the lens barrel 5 movably screwed into the hole 2c.

 [0032] The imaging case 11 has small-diameter portions lib which are cylindrical members smaller in diameter than the imaging case 11 on both end surfaces. One of the small-diameter portions l ib is provided with an imaging portion position detecting rib 11c.

 [0033] The small diameter portion l ib is rotatably supported around an axis by a notch provided in the bottom portion 2b. Thus, the imaging case 11 is held so as to be rotatable relative to the imaging head 2.

 The imaging unit position detecting rib 11c is fixed to one of the small diameter parts l ib, and rotates around the axis of the imaging case 11 with the rotation of the imaging case 11. The above-described imaging unit position detection sensor S2 fixed to the bottom 2b is located near the imaging unit position detection rib 1lc. By detecting the position of the rib 11c for detecting the position of the imaging unit by the imaging unit position detection sensor S2, the direction of the imaging direction of the image imaging unit 14 with respect to the imaging head 2 is detected. An electric signal for determining whether or not the projection direction and the imaging direction have a positional relationship within a predetermined range is output by the imaging unit position detection sensor S2.

 FIG. 2 (c) is a diagram showing a state where the imaging case 11 has been removed from the bottom 2b, and FIG. 2 (d) is a diagram showing the imaging case 11. As shown in FIG. 2 (c), the bottom portion 2b includes a predetermined position fixing rib 2d and a rotation stopper 2e.

 The predetermined position fixing rib 2d is biasedly engaged with the small diameter portion l ib of the imaging case 11, so that the relative positional relationship between the image projecting unit 13 and the image capturing unit 14 becomes a predetermined positional relationship. In a certain state, the imaging case 11 is fixed at a predetermined position.

 [0037] The rotation stopper 2e is provided so as to engage with a plurality of ribs 11a provided upright on the outer peripheral surface of the imaging case 11. Therefore, by urging the imaging case 11 with the plurality of ribs 11a, the imaging case 11 can be intermittently rotated, and can be fixed at a desired position.

FIG. 2E is a diagram showing an internal configuration of the imaging case 11. As shown in FIG. 2 (e), the imaging case 11 includes an imaging optical system 21 and light incident through the imaging optical system 21 in an electric signal. The CCD22 that converts to the signal is fixed. The image pickup unit 14 composed of the image pickup optical system 21 and the CCD 22 can pick up an image of a subject that faces the image pickup unit 14 and obtain the image pickup data. Further, by rotating the imaging case 11 around the axis with respect to the imaging head 2, the imaging direction of the image imaging unit 14 fixed to the imaging case 11 is shifted by the image projection unit 13 fixed to the imaging head 2. It is changed relative to the projection direction. Therefore, since the image capturing section 14 can be made to face a subject existing in a direction different from the projection direction, the image projecting section 13 can only use the subject existing in the direction in which the image signal light is projected by the image projecting section 13. By means of 13, it is possible to capture an image of a subject existing in a direction different from the direction in which the image signal light is projected.

 Further, as shown in FIG. 2 (e), one of the small-diameter portions l ib is formed in a tubular shape, and a signal cable is passed therethrough. With this signal cable, the electrical components in the imaging case 11 and the electrical components in the imaging head 2 are electrically connected.

 FIG. 3 is a block diagram showing an electrical configuration inside the image input / output device 1.

 The imaging head 2 includes an image projection unit 13 and an image imaging unit 14 as main components. The mounting part 4 contains a processor 15 and a battery 26 as main components.

 A signal cable is inserted inside the arm member 3. The signal cable extends to the inside of the mounting portion 4 and connects the electrical components in the mounting portion 4 to the electrical components in the imaging head 2.

 [0042] The image projection unit 13 is a unit for projecting a desired projection image on a projection surface. The image projection unit 13 includes a substrate 16, a plurality of LEDs 17 (hereinafter collectively referred to as “LED array 17 A”), a light source lens 18, a projection LCD 19, and a projection optical system 20. They are arranged in a straight line along the optical axis 13a of the projection optical system 20). The image projection unit 13 will be described later in detail with reference to FIGS. 4 (a) to 4 (c).

 The image capturing section 14 is a unit for capturing an image of the subject P. In the image pickup section 14, an image pickup optical system 21 and a CCD 22 are arranged along a light input direction (optical axis of the image pickup optical system 21) 14a.

The imaging optical system 21 includes a plurality of lenses and has a well-known autofocus function. To do. With the autofocus function, the imaging optical system 21 is configured to automatically adjust the focal length and aperture and form an image of external light on the CCD 22.

 The CCD 22 has photoelectric conversion elements such as CCD (Charge Coupled Device) elements arranged in a matrix. The CCD 22 generates a signal corresponding to the color and intensity of the light of the image formed on the surface via the imaging optical system 21, converts the signal into digital data, and outputs the digital data to the port processor 15.

 [0046] The processor 15 includes a flash 7, a release button 8, an on-Z off switch 9a, a mode switching switch 9b, a personal computer interface 24, a power supply interface 25, an external memory 27, a cache memory 28, a light source driver 29, and a light source driver 29. They are electrically connected via an LCD driver 30 and a CCD interface 31. A battery 26 is connected to the processor 15 via a power supply interface 25, an LED array 17A is connected via a light source driver 29, a projection LCD 19 is connected via a projection LCD driver 30, and a CCD interface is connected to the processor 15. The CCD 22 is connected via the face 31. These components connected to the processor 15 are managed by the processor 15.

 The external memory 27 is a removable flash ROM. The external memory 27 stores captured images and three-dimensional information in digital camera mode, webcam mode, and stereoscopic image mode. Specifically, an SD card, an XD card (both are registered trademarks) or the like can be used as the external memory 27.

 [0048] The cache memory 28 is a high-speed storage device. The cache memory 28 is used, for example, as follows. Under the control of the processor 15, the captured image is transferred to the cache memory 28 at high speed, subjected to image processing, and stored in the external memory 27. As the cache memory 28, SDRAM, DDRRAM, or the like can be used.

 [0049] The power supply interface 25, the light source driver 29, the projection LCD driver 30, and the CCD interface 31 are configured by an IC (Integrated Circuit). A power supply interface 25, a light source driver 29, a projection LCD driver 30, and a CCD interface 31 control the notebook 26, the LED array 17A, the projection LCD 19, and the CCD 22, respectively.

As shown in FIG. 3, the main battery 26 for supplying drive power to the imaging head 2 is provided in the mounting section 4, so that the weight of the imaging head 2 can be reduced. Therefore, PC49 In contrast, the operation of changing the direction of the imaging head 2 becomes easy. Further, since the imaging head 2 is reduced in weight, the strength of the arm member 3 is not so required as compared with a case where the main battery 26 is provided in the imaging head 2.

 FIG. 4A is an enlarged view of the image projection unit 13, FIG. 4B is a plan view of the light source lens 18, and FIG. 4C shows an arrangement relationship between the projection LCD 19 and the CCD 22. FIG.

 As described above, the image projection unit 13 includes the substrate 16, the LED array 17A, the light source lens 18, the projection LCD 19, and the projection optical system 20 along the projection direction.

 [0053] The substrate 16 is for mounting the LED array 17A and for performing electrical wiring with the LED array 17A. Specifically, an insulating resin is applied to an aluminum substrate provided with through holes, and then a copper pattern is formed by electroless plating. ゃ A single-layer or multilayer substrate with a glass epoxy substrate as the core Can be used as 16.

 [0054] The LED array 17A is a light source that emits radial light. On the substrate 16, a plurality of LEDs 17 (light emitting diodes) are arranged in a staggered manner. These LEDs 17 are bonded and electrically connected to the substrate 16 via a silver paste. The LED 17 is also electrically connected to the substrate 16 via a bonding wire.

 [0055] By using a plurality of LEDs 17 as the light source in this way, the efficiency of converting electricity into light (electro-optical conversion efficiency) is increased as compared with the case where an incandescent light bulb, a halogen lamp, or the like is used as the light source. At the same time, generation of infrared rays and ultraviolet rays can be suppressed. Therefore, according to the present embodiment, the light source can be driven with low power consumption, and power saving and long life can be achieved. Further, according to the present embodiment, it is possible to reduce the temperature rise of the device.

 As described above, since the LED 17 generates extremely low heat rays and has a low total heat generation of the light source itself as compared with a halogen lamp or the like, the light source lens 18 and the projection optical system 20 described later are made of resin. A lens can be employed. Therefore, the light source lens 18 and the projection optical system 20 can be configured inexpensively and lightly as compared with the case where a glass lens is employed.

[0057] Each of the LEDs 17 constituting the LED array 17A emits the same emission color. Each of the LEDs 17 uses four elements of Al, In, Ga, and P as materials, and emits an amber color. Therefore, it is not necessary to consider the correction of chromatic aberration that occurs when emitting a plurality of emission colors. It is not necessary to use an achromatic lens as the projection optical system 20 to correct chromatic aberration. Thus, there is an effect of providing a simple surface configuration and an inexpensive material projection unit.

 [0058] In addition, by adopting an amber LED of a four-element material having an electric-to-optical conversion rate as high as about 801umen / W as compared with other emission colors, higher luminance, power saving, and longer life can be achieved. it can. The effect of arranging the LEDs 17 in a staggered manner will be described with reference to FIGS. 4 (a) to 4 (c).

 [0059] Specifically, the LED array 17A includes 59 LEDs 17, and each LED 17 is driven at 50mW (20 mA, 2.5V). Therefore, all 59 LEDs 17 are driven with approximately 3W of power consumption. The luminous power emitted from each LED 17 The brightness as the luminous flux when illuminated from the projection optical system 20 through the light source lens 18 and the projection LCD 19 is set to about 25 ANSI lumens even in the case of full illumination. ing.

 [0060] By adopting this brightness, for example, when detecting the three-dimensional shape of a subject such as the face of a person or an animal in the stereoscopic image mode, it does not give glare to the person or the animal, and Can detect 3D shapes in which the eyes are not closed.

 [0061] The light source lens 18 is a lens as a light-collecting optical system that collects light emitted radially from the LED array 17A, and is made of an optical resin represented by acrylic.

[0062] Specifically, the light source lens 18 projects at a position facing each LED 17 of the LED array 17A, and supports a convex lens portion 18a protruding toward the LCD 19 side, and the lens portion 18a. Epoxy or silicon for sealing the LED 17 and bonding the substrate 16 and the light source lens 18 to the base portion 18b and the space inside the base portion 18b, which fills the opening containing the LED array 17A. It includes a resin sealing material 18c and positioning pins 18d projecting from the base portion 18b toward the substrate 16 and connecting the light source lens 18 and the substrate 16.

 The light source lens 18 is fixed on the substrate 16 by inserting the positioning pins 18 d into the long holes 16 formed in the substrate 16 while enclosing the LED array 17 A inside the opening.

Accordingly, the light source lens 18 can be arranged in a small space. Also, by providing a function of supporting the light source lens 18 in addition to the function of mounting the LED array 17A on the substrate 16, it is not necessary to separately provide a component for supporting the light source lens 18, and the part is not required. The number of articles can be reduced.

[0065] Further, each lens portion 18a is arranged at a position facing each LED 17 of the LED array 17A in a one-to-one relationship.

 Therefore, the radial light emitted from each LED 17 is transmitted to each lens unit facing each LED 17.

 The light is efficiently condensed by 18 and radiated to the projection LCD 19 as radiation having high directivity as shown in the figure. The reason why the directivity is increased in this way is that by injecting light substantially perpendicularly to the projection LCD 19, the in-plane transmittance unevenness due to the optical rotation of the liquid crystal can be suppressed. At the same time, since the projection optical system 20 has telecentric characteristics and its incident NA is about 0.1, it is regulated so that only light within vertical ± 5 ° can pass through the internal aperture. is there. Therefore, it is essential to improve the image quality that the emission angle of the light from the LED 17 is set to be perpendicular to the projection LCD 19 and that almost all of the luminous flux is within ± 5 °. This is because, if light that is out of the vertical direction enters the projection LCD 19, the transmittance changes depending on the incident angle due to the optical rotation of the liquid crystal, resulting in transmittance unevenness.

 The projection LCD 19 is a spatial modulation element that spatially modulates light condensed through the light source lens 18 and outputs a desired image signal light to the projection optical system 20. Specifically, the projection LCD 19 is composed of a plate-like liquid crystal display (Liquid Crystal Display) having a different aspect ratio.

 As shown in FIG. 4 (c), each pixel constituting the projection LCD 19 is composed of one pixel row arranged in a straight line along the longitudinal direction of the projection LCD 19, and one pixel row. The projection LCD 19 is arranged so that other pixel rows shifted by a predetermined distance in the longitudinal direction are alternately arranged in parallel.

 In FIG. 4 (c), the front of the imaging head 2 is directed to the front side of the paper, light is irradiated toward the projection LCD 19 from the back side of the paper, and a subject image is formed on the CCD 22 from the front side of the paper. It is assumed to be in the state.

As described above, by arranging the pixels constituting the projection LCD 19 in a staggered manner in the longitudinal direction, the light that is spatially modulated by the projection LCD 19 in one direction perpendicular to the longitudinal direction (transverse direction) is reduced by one. / 2 pitch can be controlled. Therefore, the projection pattern can be controlled at a fine pitch, and the three-dimensional shape can be detected with high accuracy by increasing the resolution. In particular, in a stereoscopic image mode to be described later, when projecting a striped pattern light in which light and dark are alternately arranged toward an object to detect the three-dimensional shape of the object, the direction of the stripe is determined. By aligning the image with the short side of the LCD 19, it is possible to control the boundary between light and dark at a half pitch, so that the three-dimensional shape can be detected with high resolution and high accuracy as well. .

 [0072] Inside the imaging head 2, the projection LCD 19 and the CCD 22 are arranged in a relationship as shown in Fig. 4 (c). More specifically, since the wide surface of the projection LCD 19 and the wide surface of the CCD 22 are arranged and directed in substantially the same direction, an image projected from the projection LCD 19 onto the projection surface is formed on the CCD 22. When forming an image, the projected image can be formed as it is without being bent by a half mirror or the like.

 The CCD 22 is arranged on the longitudinal direction side of the projection LCD 19 (in the direction in which the pixel columns extend). Therefore, in particular, when detecting the three-dimensional shape of the subject using the principle of triangulation in the three-dimensional image mode, the inclination S between the CCD 22 and the subject can be controlled at a half pitch, so that the force S can be obtained. Similarly, three-dimensional shapes can be detected with high accuracy.

 [0074] The projection optical system 20 is a plurality of lenses that project the image signal light that has passed through the projection LCD 19 toward the projection surface, and is formed of a telecentric lens made of a combination of glass and resin. Telecentric refers to a configuration in which the principal ray passing through the projection optical system 20 is parallel to the optical axis in the space on the incident side, and the position of the exit pupil is infinite. By being telecentric in this manner, as described above, only light that passes through the projection LCD 19 at a vertical angle of ± 5 ° can be projected, so that image quality can be improved.

 FIGS. 5A to 5C are views for explaining the arrangement of the LED array 17A. FIG. 5 (a) is a diagram showing an illuminance distribution of light passing through the light source lens 18, FIG. 5 (b) is a plan view showing an arrangement state of the LED array 17A, and FIG. FIG. 5 is a diagram showing a composite illuminance distribution in the example.

 As shown in FIG. 5 (a), the light that has passed through the light source lens 18 is a light having a half-value half-width 略 (= approximately 5 °) and an illuminance distribution as shown on the left side of FIG. 5 (a). The projection is designed to reach the surface of the LCD 19 as a level.

As shown in FIG. 5B, the plurality of LEDs 17 are arranged in a staggered pattern on the substrate 16. Ingredient Specifically, a plurality of LEDs 17 in which a plurality of LEDs 17 are arranged in series at a d pitch are arranged in parallel at a 3 / 2d pitch, and furthermore, a plurality of LED columns arranged in such a manner are arranged in parallel. In this case, every other row is moved by l / 2d in the same direction with respect to the next row.

 In other words, the distance between the one LED 17 and the LCD 17 around the one LED 17 is set to be d (triangular lattice arrangement).

[0079] The length of d is determined so as to be equal to or less than the full width half maximum (FWHM) of the illuminance distribution formed on the projection LCD 19 by the light emitted from the LED 17 even with one power. .

 Thus, the combined illuminance distribution of the light passing through the light source lens 18 and reaching the surface of the projection LCD 19 becomes a substantially linear shape including a small lip gap as shown in FIG. Light can be applied to the surface substantially uniformly. Therefore, illuminance unevenness in the projection LCD 19 can be suppressed, and as a result, a high-quality image can be projected.

 FIG. 6 is an electrical block diagram of the image input / output device 1. The description of the configuration already described above is omitted. The processor 15 includes a CPU 35, a ROM 36, and a RAM 37.

 The CPU 35 performs various kinds of processing using the RAM 37, according to the program stored in the ROM 36. The processing performed under the control of the CPU 35 includes detection of a pressing operation of the release button 8, capture of image data from the CCD 22, transfer and storage of the image data, detection of the state of the mode switching switch 9b, and the like.

 [0083] The ROM 36 includes a camera control program 36a, a pattern light photographing program 36b, a luminance image generation program 36c, a code image generation program 36d, a code boundary extraction program 36e, a lens aberration correction program 36f, and a triangle. A survey calculation program 36g, a document attitude calculation program 36h, and a plane conversion program 36i are stored.

 [0084] The camera control program 36a is a program relating to control of the entire image input / output device 1 including the main processing shown in FIG.

[0085] The pattern light photographing program 36b is a program for imaging a state in which pattern light is projected on a subject in order to detect a three-dimensional shape of the subject P, and a state in which the projected pattern light is projected. [0086] The luminance image generation program 36c is provided for each of the pattern light existence image obtained by imaging the state where the pattern light is projected by the pattern light imaging program 36b and the pattern light absence image obtained by imaging the state where the pattern light is not projected. This is a program that calculates the Y value in the YCbCr space from the RGB value of the image.

[0087] In addition, a plurality of types of pattern light are projected in time series and imaged for each pattern light to generate a plurality of types of luminance images.

[0088] The code image generation program 36d binarizes the plurality of luminance images generated by the luminance image generation program 36c with reference to a preset luminance threshold or a luminance image of an image without pattern light, and based on the result, This is a program for generating a code image in which a predetermined code is assigned to each pixel.

[0089] The code boundary extraction program 36e uses the code image generated by the code image generation program 36d and the luminance image generated by the luminance image generation program 36c to determine the code boundary coordinates with sub-pixel accuracy. It is a program.

The lens aberration correction program 36f is a program for correcting the aberration of the imaging optical system 20 with respect to the code boundary coordinates obtained by the sub-pixel accuracy by the code boundary extraction program 36e.

[0091] The triangulation calculation program 36g is a program that calculates three-dimensional coordinates in the real space related to the boundary coordinates from the boundary coordinates of the code whose aberration has been corrected by the lens aberration correction program 36f.

[0092] The document attitude calculation program 36h is a program for estimating and obtaining the three-dimensional shape of the subject P such as a book from the three-dimensional coordinates calculated by the triangulation calculation program 36g.

[0093] The plane conversion program 36i is a program that generates a flattened image as if the subject P such as a book is imaged from the front, based on the three-dimensional shape of the subject P such as a manuscript calculated by the manuscript attitude calculation program 36h. It is.

[0094] The RAM 37 includes a pattern light image storage unit 37a, a pattern light non-image storage unit 37b, a luminance image storage unit 37c, a code image storage unit 37d, a code boundary coordinate storage unit 37e, and an ID storage unit. 37f, an aberration correction coordinate storage unit 37g, a three-dimensional coordinate storage unit 37h, The calculation result storage unit 37i, the plane conversion result storage unit 37j, the projection image storage unit 37k, and the marking area 371 are allocated as storage areas.

 [0095] The pattern light existence image storage unit 37a stores a pattern light existence image obtained by imaging a state where the pattern light is projected on the subject P by the pattern light photographing program 36b. The pattern light non-image storage unit 37b stores the pattern light non-image obtained by projecting the pattern light on the subject P by the pattern light photographing program 36b and capturing the image, state, and state.

 [0096] The luminance image storage unit 37c stores the luminance image generated by the luminance image generation program 36c. The code image storage unit 37d stores a code image generated by the code image generation program 36d. The code boundary coordinate storage unit 37e stores the boundary coordinates of each code obtained with the sub-pixel accuracy to be extracted by the code boundary extraction program 36e. The ID storage unit 37f stores an ID or the like assigned to a luminance image having a change in brightness at a pixel position having a boundary. The aberration correction coordinate storage unit 37g stores the boundary coordinates of the code whose aberration has been corrected by the lens aberration correction program 36f. The three-dimensional shape coordinate storage unit 37h stores the three-dimensional coordinates of the real space calculated by the triangulation calculation program 36g.

 The document attitude calculation result storage unit 37i stores parameters related to the three-dimensional shape of the subject P such as a document calculated by the document attitude calculation program 36h. The plane conversion result storage unit 37j stores the plane conversion result generated by the plane conversion program 36. The projection image storage unit 37k stores image information projected from the image projection unit 13. The working area 371 stores data temporarily used for the operation in the CPU 15.

 [0098] An amplifier 32 is connected to the processor 15 via a bus line in addition to the configuration described above. The amplifier 32 sounds a speaker 33 connected to the amplifier 32 to output a warning sound or the like.

 [0099] The PC 49 has a CPU 50. In the PC 49, a ROM 51 and a RAM 52 are connected to the CPU 50 via an internal bus. In the PC 49, an interface 55 connectable to the PC interface 24, a CRT display 58, and a keyboard 59 are electrically connected to an internal bus to which the CPU is connected via an input / output port 54. .

FIG. 7 is a flowchart of a main process executed under the control of the CPU 35. still Details of the digital camera processing (S605), webcam processing (S607), stereoscopic image processing (S609), and flattened image processing (S611) in the main processing will be described later.

 In the main processing, first, when the power is turned on (S601), the processor 15 and other interfaces are initialized (S602).

 [0102] Then, a key scan is performed to determine the state of the mode switching switch 9 (S603), and it is determined whether the setting of the mode switching switch 9 is the digital camera mode (S604). If the setting of the mode switching switch 9 is the digital camera mode (S604: Yes), the process shifts to a digital camera process described later (S605).

 On the other hand, if the setting of the mode switching switch 9 is not the digital camera mode (S604: No), it is determined whether or not the setting of the mode switching switch 9 is the webcam mode (S606). If the setting of the mode switching switch 9 is the webcam mode (S606: Yes), the process shifts to a webcam process described later (S607).

 On the other hand, if the setting of the mode switching switch 9 is not the webcam mode (S606: No), it is determined whether the setting of the mode switching switch 9 is the stereoscopic image mode (S608). If the setting of the mode switching switch 9 is the stereoscopic image mode (S608: Yes), the processing shifts to the stereoscopic image processing described later (S609).

 On the other hand, if the setting of the mode switching switch 9 is not the stereoscopic image mode (S608: No), it is determined whether or not the setting of the mode switching switch 9 is the flattened image mode (S610). If the setting of the mode switching switch 9 is the flattened image mode (S610: Yes), the processing shifts to flattened image processing described later (S611).

 On the other hand, if the setting of the mode switching switch 9 is not the flat image mode (S610: No), it is determined whether the setting of the mode switching switch 9 is the off mode (S612). If the setting of the mode switching switch 9 is not the off mode (S612: No), the processing from S603 is repeated. If the setting of the mode switching switch 9 is the off mode (S612: Yes), the process ends.

[0107] Hereinafter, digital camera processing, webcam processing, and stereoscopic image processing will be described in detail. In these processes, since the condition determination is performed based on the relative positional relationship between the image projection unit 13 and the image imaging unit 14, first, the relative position between the image projection unit 13 and the image imaging unit 14 is determined. Rank The relationship will be described.

 FIGS. 8 (a) and 8 (b) are diagrams showing the positional relationship between the image projection unit 13 and the image pickup unit 14. FIG. First, based on signals output from the imaging head position detection sensor S1 and the imaging unit position detection sensor S2, the projection direction of the image projection unit 13 (the optical axis direction of the projection optical system 20) is 0 ° ± 30 ° (approximately 30 ° to approximately 30 °), and the image pickup direction of the image pickup unit 14 (the optical axis direction of the image pickup optical system 21) is set in front of the image pickup head 2 (the surface on which the lens barrel 5 is provided). Against 0. In the following, the case where it is detected that it is other than A is referred to as A mode (the state shown in FIG. 8 (b) A).

 [0109] Based on the signals output by the imaging head position detection sensor S1 and the imaging unit position detection sensor S2, the projection direction of the image projection unit 13 is 0 ° ± 30 ° with respect to the vertical direction, and the image imaging unit The case where the imaging direction of 14 is detected to be substantially 0 ° with respect to the front of the imaging head 2 is hereinafter referred to as B mode (the state shown in FIG. 8B).

 [0110] In the B mode, the direction in which the subject captured by the image capturing unit 14 is present and the direction in which the image signal light is projected by the image projecting unit 13 are large angles that are close to perpendicular to each other. The image signal light is not projected on the top, and even if the image signal is projected by the image projection unit 13 and the image is captured by the image capturing unit 14, the projected image can be prevented from being captured. Therefore, when the positional relationship between the image projection unit 13 and the image capturing unit 14 is the B mode, for example, a character string or the like indicating the operation method or the operation procedure (related information related to the captured image acquired by the imaging unit) is projected. In this state, the user can appropriately perform imaging while checking the projected character strings and the like. Also, for example, by projecting the image signal light in a direction different from the direction in which the user himself is present, and by taking the image with the image capturing unit 14 facing the user himself, the user can project the image with the image projecting unit 13. You can capture yourself while viewing the shadowed image.

 [0111] Based on the signals output by the imaging head position detection sensor S1 and the imaging unit position detection sensor S2, the projection direction of the image projection unit 13 is other than 0 ° ± 30 ° with respect to the vertical direction, and The case where it is detected that the imaging direction of 14 is not substantially 0 ° with respect to the front of the imaging head 2 is hereinafter referred to as C mode (the state shown in FIG. 8 (b) C).

[0112] Based on the signals output by the imaging head position detection sensor S1 and the imaging unit position detection sensor S2, the projection direction of the image projection unit 13 is other than 0 ° ± 30 ° with respect to the vertical direction. The case where the imaging direction of the image imaging unit 14 is detected to be substantially 0 ° with respect to the front of the imaging head 2 is hereinafter referred to as a D mode (the state shown in FIG. 8 (b) D).

[0113] In the D mode, the direction in which the subject captured by the image capturing unit 14 is present and the direction in which the image signal light is projected by the image projecting unit 13 are large angles that are close to perpendicular to each other. The image signal light is not projected on the top, and even if the image signal is projected by the image projection unit 13 and the image is captured by the image capturing unit 14, the projected image can be prevented from being captured.

 FIG. 9 is a flowchart of the digital camera process (S605 in FIG. 6). The digital camera process is a process of acquiring an image captured by the image capturing unit 14.

 In this process, first, a high resolution setting signal is transmitted to the CCD 22 (S701). As a result, a high quality captured image can be provided to the user.

 Next, based on the signal output from the imaging head position detection sensor S1, the image projection unit

 The position of the image pickup unit 13 is obtained, and the position of the image pickup unit 14 is obtained based on the signal output from the image pickup unit position detection sensor S2 (S702b).

 Subsequently, it is checked whether or not the positional relationship between the image projecting unit 13 and the image capturing unit 14 is the above-described A mode (see FIGS. 8A and 8B) (S702c). When the positional relationship between the image projecting unit 13 and the image capturing unit 14 is the A mode (S702c: Yes), the image projecting unit 13 projects rectangular image light, which is an image indicating an imageable range, onto an imageable area. (S702d). FIG. 10 is a diagram illustrating a state in which rectangular image light indicating an imageable area is projected on a surface. As described above, the user can visually recognize the area that can be imaged by the rectangular image light before actually imaging by pressing the release button 8.

On the other hand, when the positional relationship between the image projecting unit 13 and the image capturing unit 14 is not the A mode described above (S702c: No), it is checked whether the mode is the B mode (see FIG. 8) (S702e). When the positional relationship between the image projecting unit 13 and the image capturing unit 14 is the B mode (S702e: Yes), the image is projected by the image projecting unit 13 such as a character for notifying that the image can be captured (S70). 2f) ₀

On the other hand, when the positional relationship between the image projecting unit 13 and the image capturing unit 14 is not the above-described B mode (S702e: No), the speaker 33 (see FIG. 6) sounds that the image can be captured. Be informed (S702g) _{o In} other words, when the projection direction of the image signal light by the image projection unit 13 is other than 0 ° ± 30 ° (C mode or D mode) with respect to the vertical direction, the projection direction of the image projection unit 13 Since there is a high possibility that there is no surface that can be projected, when the positional relationship between the image projecting unit 13 and the image capturing unit 14 is not in either the A mode or the B mode described above, it is determined that imaging is possible. It is notified by sound rather than by image. It should be noted that, instead of the voice notification, a pilot lamp (not shown) or the like may be lit or flashed to notify that the image can be captured.

 Next, the release button 8 is scanned (S703a), and it is determined whether or not the release button 8 is half-pressed (S703b). If the release button 8 is half-pressed (S703b: Yes), the auto focus (AF) and automatic exposure (AE) functions are activated, and the focus, aperture, and shirt speed are adjusted (S703c). If the release button 8 has not been half-pressed (S703b: No), the processing from S703a is repeated.

 Next, the release button 8 is scanned again (S703d), and it is determined whether or not the release button 8 is fully pressed (S703e). If the release button 8 is fully pressed (S703e: Yes), it is determined whether or not the flash mode is set (S704).

 As a result, if the mode is the flash mode (S704: Yes), the flash 7 is emitted (S705), and shooting is performed (S706). If not in the flash mode (S704: No), shooting is performed without emitting the flash 7 (S706). If it is determined in S703e that the button is not fully pressed (S703e: No), the processing from S703a is repeated.

[0123] Here, in the process of S702e, when it is determined that the mode is the B mode, and in the process of S702f, an image indicating that imaging is possible is being projected, in the process of S706, the projection of that image is continued. , The subject P is imaged. That is, in the B mode, the imaging direction is substantially 0 ° with respect to the front of the imaging head 2, and the direction in which the image is captured by the image capturing unit 14 and the direction in which the image signal light is projected by the image projecting unit 13 are substantially perpendicular to each other. A large angle close to. Therefore, no image signal light is projected by the image projection unit 13 on the subject to be imaged by the image imaging unit 14. Therefore, even if the image is captured while projecting the image by the image projection unit 13, the force S for imaging the subject P without including the projected image can be obtained. Next, the captured image is transferred from the CCD 22 to the cache memory 28 (S707). The captured image stored in the cache memory 28 is displayed on the monitor LCD 10 (S708). As described above, by transferring the captured image to the cache memory 28, the captured image can be displayed on the monitor LCD 10 at a higher speed than when the captured image is transferred to the main memory. Next, the captured image is stored in the external memory 27 (S709).

 [0125] Finally, it is determined whether or not there is a change in the mode switching switch 9 (S710). If there is no change (S710: Yes), the processing from S702b is repeated. If there is a change in the mode switching switch 9 (S710: No), the process ends.

 FIG. 11 is a flowchart of the webcam process (S607 in FIG. 7). The webcam process is a process of transmitting a captured image (including a still image and a moving image) captured by the image capturing unit 14 to an external network. In this embodiment, it is assumed that a moving image is transmitted to an external network as a captured image.

 In this process, first, a low-resolution setting signal is transmitted to the CCD 22 (S801), and the well-known auto focus (AF) and automatic exposure (AE) functions are activated, and the focus, aperture, and shutter speed are adjusted. Is performed (S802a).

 Next, based on the signal output from the imaging head position detection sensor S1, the image projection unit

 The position of the image pickup unit 13 is obtained, and the position of the image pickup unit 14 is obtained based on the signal output from the image pickup unit position detection sensor S2 (S802b).

 Subsequently, it is checked whether or not the positional relationship between the image projecting unit 13 and the image capturing unit 14 is the above-described A mode (see FIG. 8) (S802c). When the positional relationship between the image projecting unit 13 and the image capturing unit 14 is the A mode (S802c: Yes), the image capturing range is indicated by the image projecting unit 13 as in the process of S702d in the digital camera process (see FIG. 9). A rectangular image light, which is an image, is projected (S802d).

On the other hand, when the positional relationship between the image projection unit 13 and the image pickup unit 14 is not the A mode described above (S802c: No), it is checked whether or not the B mode (see FIG. 8) is used (S802e). When the mode is the B mode (S802e: Yes), a projection process described later is performed (S802f), and then the image is captured by the image capturing unit 14. When the positional relationship between the image projecting unit 13 and the image capturing unit 14 is the B mode, the image capturing direction intersects the projection direction at a large angle that is almost perpendicular to the projection direction. The image signal light is not projected by the image projection unit 13 on the subject P to be imaged by the image unit 14. Therefore, similar to the process of S702f in the digital camera process (see FIG. 9), even if the image is projected while the image projection unit 13 projects the image, the subject P can be imaged without including the projected image. .

 FIG. 12 is a diagram showing a state in which the image output light is projected by the image projection unit 13 in the projection processing (S802f) of the webcam processing. As shown in FIG. 12, in the webcam processing, for example, a message image M composed of a character string indicating an imaging mode such as “WEBCAM mode” and a captured image f are synthesized and projected. At the start of the webcam process, the captured image f is still obtained, and only the message M is projected.

 On the other hand, when the positional relationship between the image projecting unit 13 and the image capturing unit 14 is not the B mode described above (S802e: No), the speaker 33 (see FIG. 6) sounds to indicate that the image can be captured. The user is notified (S802g), and shooting is started (S803). That is, for the same reason as the digital camera processing (see FIG. 9), when the positional relationship between the image projecting unit 13 and the image capturing unit 14 is neither the A mode nor the B mode, the image projection by the image projecting unit 13 is not performed. The notification is not given to the user only by voice.

 [0133] Next, the captured image is displayed on the monitor LCD 10 (S804). Next, the message image M composed of a character string such as “imaging is in progress” and the captured image f are combined and stored in the projection image storage unit 37k (S805). That is, in the process of S804, the projection image stored in the projection image storage unit 37k is updated. Therefore, in the projection process of S802f, the captured image f and the message image M are updated by the updated image output light. The projected image allows the user to visually recognize the change in the image capturing state by the image capturing unit 14 using the projected image.

 [0134] Next, the captured image is transferred from the CCD 22 to the cache memory 28 (S807), and the captured image transferred to the cache memory 28 is transmitted to an external network via the RF driver 24 and the antenna 11 as an RF interface. Is performed (S808).

 [0135] Finally, it is determined whether or not the mode switching switch 9 has changed (S809). If there is no change in the mode switching switch 9 (S809: Yes), the processing from S802 is repeated. If there is a change in the mode switching switch 9 (S809: No), the process ends.

[0136] In the above-described webcam processing, as shown in FIG. Force with which f was projected As shown in FIG. 13, only the captured image f may be projected by the image projection unit 13.

 Further, as shown in FIG. 13, when the relative positional relationship between the image projecting unit 13 and the image capturing unit 14 is the B mode, for example, the user captures the image while himself or herself. The captured image f obtained by can be checked at any time.

 FIG. 14 is a flowchart of the projection process (S802f in FIG. 11). This process is a process of projecting an image stored in the projection image storage unit 37k from the image projection unit 13 onto a projection plane. In this processing, first, it is confirmed whether or not an image is stored in the projection image storage unit 37k (S901). If it is stored (S901: Yes), the image stored in the projection image storage unit 37k is transferred to the projection LCD driver 30 (S902). Next, an image signal corresponding to the image is sent from the projection LCD driver 30 to the projection LCD 19, and an image is displayed on the projection LCD 19 (S903).

 Next, the light source driver 29 is driven (S904), and the LED array 17A is turned on by an electric signal from the light source driver 29 (S905). After that, the process ends.

 [0140] Thus, when the LED array 17A is turned on, the light emitted from the LED array 17A reaches the projection LCD 19 via the light source lens 18, and in the projection LCD 19, an image signal transmitted from the projection LCD driver 30 is transmitted. A corresponding spatial modulation is applied to the light. The light subjected to the spatial modulation is output from the projection LCD 19 as image signal light. Then, the image signal light output from the projection LCD 19 is projected as a projection image on a projection surface via the projection optical system 20.

 FIG. 15 is a flowchart of the stereoscopic image processing (S609 in FIG. 7). The stereoscopic image processing is a process of detecting a three-dimensional shape of a subject, obtaining a three-dimensional shape detection result image as the stereoscopic image, and displaying the image.

 In this processing, first, a high resolution setting signal is transmitted to the CCD 22 (S1001). Then, the position of the image projection unit 13 is obtained based on the signal output from the imaging head position detection sensor S1, and the position of the image imaging unit 14 is determined based on the signal output from the imaging unit position detection sensor S2. Is obtained (S1002b).

Next, the positional relationship between the image projecting unit 13 and the image capturing unit 14 is described in the C mode (see FIG. 8). Is checked (S1002c). When the positional relationship between the image projecting unit 13 and the image capturing unit 14 is the C mode (S1002c: Yes), the fact that the image can be captured is sounded by the speaker 33 and notified by voice (S1002d).

On the other hand, if the relative positional relationship between the image projecting unit 13 and the image capturing unit 14 is not the above-described C mode (S1002c: No), it is determined whether the mode is the A mode (see FIG. 8). (1002e). When the mode is the A mode (S1002e: Yes), the fact that the image can be captured is sounded by the speaker 33 and notified by voice (S1002d). Note that in the case of the A mode, a message may be projected or a rectangular image light indicating an imageable range may be projected instead of the notification by voice.

 On the other hand, when the relative positional relationship between the image projecting unit 13 and the image capturing unit 14 is not the A mode which is not the same as the C mode described above (S1002e: No), the state is set to the A mode or the C mode. The speaker 33 is sounded to give a warning by voice (S1002g), and the process returns to the process of S1002b. That is, until the user adjusts the imaging head 2 and the imaging case 11 to switch to the A mode or the C mode, the detection of the three-dimensional shape by the three-dimensional shape detection process (S1006) described later is prohibited.

 As will be described in detail later, in the three-dimensional shape detection process (S1006), the image capturing unit 14 captures an image of a subject in a state where a predetermined pattern light is projected by the image projection unit 13, and the captured image is captured. The three-dimensional shape of the subject P is detected based on the image data acquired by the above. Therefore, when the relative positional relationship between the image projecting unit 13 and the image capturing unit 14 is not a predetermined relationship, the accuracy of measurement of the three-dimensional shape is significantly reduced. Only when the image capturing unit 14 and the image capturing unit 14 have a predetermined relationship, execution of the three-dimensional shape detection process (S1006) is permitted.

 Next, the release button 8 is scanned (S1003a), and it is determined whether or not the release button 8 is half-pressed (S1003b). If the release button 8 is pressed halfway down (S1003b: Yes), the auto focus (AF) and auto exposure (AE) functions are activated, and the focus, aperture, and shirt speed are adjusted (S1003c). If the release button 8 is not half-pressed (S1003b: No), the processing from S1003a is repeated.

[0148] Next, the release button 8 is scanned again (S1003d), and the release button 8 is fully pressed. It is determined whether it has been performed (S1003e). If the release button 8 is fully pressed (S1003e: Yes), it is determined whether or not the flash mode is set (S1003f).

As a result, in the flash mode (S1003f: Yes), the flash 7 is emitted (S1003g), and shooting is performed (S1003h). If not in the flash mode (S1003f: No), shooting is performed without emitting the flash 7 (S1003h). If it is determined in S1003e that the button is not fully pressed (S1003e: No), the processing from S1003a is repeated.

 Next, a three-dimensional shape detection process described later is executed to detect a three-dimensional shape of the subject (S1006).

 Next, the three-dimensional shape detection result in the three-dimensional shape detection processing (S1006) is stored in the external memory 27 (S1007), and the three-dimensional shape detection result is displayed on the monitor LCD 10 (S1008a). The three-dimensional shape detection result is displayed as a set of three-dimensional coordinates (X, Υ, Z) in the real space of each measurement vertex. That is, a three-dimensional shape detection result image is displayed on the monitor LCD 10 as a three-dimensional image (3D CG image) displaying the surface by connecting measurement vertices as a three-dimensional shape detection result with a polygon.

 Next, it is determined whether there is no change in the mode switching switch 9 (S1011), and if there is no change (S1011: Yes), the processing from S702 is repeated. If there is a change in the mode switching switch 9 (S1011: No), the process ends.

 [0153] FIG. 16 (a) is a diagram for explaining the principle of the spatial code method used to detect a three-dimensional shape in the above-described three-dimensional shape detection processing (S1006 in FIG. 15). 16 (b) is a view showing pattern light different from 16 (a). Either of FIG. 16 (a) or FIG. 16 (b) may be used for the pattern light, and further, a gray level code which is a multi-tone code may be used.

 [0154] Details of this spatial coding method are described in Kosuke Sato and one other, "Distance Image Input by Spatial Coding", Transactions of the Institute of Electronics, Information and Communication Engineers, 85Z3Vol. J 68—D No3 p369-375. It is disclosed in detail.

[0155] The spatial code method is one type of a method for detecting the three-dimensional shape of a subject based on triangulation between the projected light and the observed image. As shown in FIG. Observer O It is characterized by being installed at a distance D, dividing the space into elongated fan-shaped areas, and coding them.

 When the three mask patterns A, B, and C in the figure are projected in order from the MSB, each fan-shaped area is coded by the mask into bright “1” and 喑 “0”. For example, the area containing point P

A and B are not illuminated and mask C is bright, so it is coded as 001 (A = 0, B = 0, C = l).

 [0157] Each fan-shaped area is assigned a code corresponding to the direction φ, and the boundary of each code can be regarded as one slit light beam. Therefore, the scene is photographed for each mask with a camera as an observation device, and the light and dark patterns are binarized to configure each bit plane of the memory.

 [0158] The horizontal position (address) of the obtained multi-bit plane image is determined in the observation direction.

 And the contents of the memory at this address give the projected light code, ie, φ. The coordinates of the point of interest are determined from Θ and Φ.

 [0159] The mask pattern used in this method includes mask pattern A,

When pure binary codes such as B and C are used, there is a risk that a large error will occur at the boundary of the area if the force S and the mask are misaligned as shown.

[0160] For example, the point Q in Fig. 16 (a) is a force indicating the boundary between the area 3 (011) and the area 4 (100). If the mask A shifts by 1, the code of the area 7 (111) may be generated. There is. In other words, where the Hamming distance is 2 or greater between adjacent regions, a large error is likely force ^s occur.

 [0161] Therefore, as shown in Fig. 16 (b), as a mask pattern used in this method, a code in which the Hamming distance is always 1 between adjacent regions is used. It is said that you can avoid.

FIG. 17A is a flowchart of the three-dimensional shape detection processing (S1006 in FIG. 15). In this process, first, an imaging process is performed (S1210). This imaging processing is performed by using the mask patterns of a plurality of pure binary codes shown in FIG. 16 (a) and from the image projection unit 13 to obtain a striped pattern light (FIGS. 1 and 18) in which light and dark are alternately arranged. ) Is projected onto the subject in chronological order, and the pattern light with the pattern light projected image and the pattern light This is a process of acquiring a pattern light non-image obtained by capturing an image that is not projected. FIG. 1 is a diagram showing a state in which the pattern light is projected when the relative positional relationship between the image projecting unit 13 and the image capturing unit 14 is A mode, and FIG. FIG. 9 is a diagram showing a state where the pattern light is projected and projected when the relative positional relationship between the projection unit 13 and the image pickup unit 14 is a C mode.

 When the imaging process ends (S1210), a three-dimensional measurement process is performed (S1220). The three-dimensional measurement process is a process of actually measuring the three-dimensional shape of a subject by using the image with pattern light and the image without pattern light acquired by the imaging process. Thus, when the three-dimensional measurement processing ends (S1220), the processing ends.

 FIG. 17B is a flowchart of the imaging process (S1210 in FIG. 12A). This process is executed based on the pattern light photographing program 36a. First, an image of a subject is captured by the image photographing unit 14 which does not project the pattern light from the image projecting unit 13, thereby obtaining a pattern light non-image (S1211). ). The acquired pattern light non-image is stored in the pattern light non-image storage unit 37b.

 Next, the counter i is initialized (S1212), and it is determined whether or not the value of the counter i is the maximum value imax (S1213). The maximum value imax is determined by the number of mask patterns used. For example, when eight types of mask patterns are used, the maximum is imax (= 8).

If the result of the determination indicates that the value of the counter i is smaller than the maximum value imax (S1213: Yes), the i-th mask pattern among the mask patterns to be used is displayed on the projection LCD 19, and The i-th pattern light projected by the No. mask pattern is projected on the projection surface (S1214) ₀ Then, the state where the pattern light is projected is photographed by the image pickup section 14 (S1215). ).

 [0167] In this way, a pattern light existence image obtained by imaging the state where the i-th pattern light is projected on the subject is obtained. The acquired pattern light image is stored in the pattern light image storage section 37a.

[0168] When the photographing is completed, the projection of the i-th pattern light is terminated (S1216), "1" is added to the counter i for projecting the next pattern light (S1217), and the processing from S1213 is repeated. Returned It is.

 [0169] When it is determined that the value of the counter i is larger than the maximum value imax (S1213: No), the process ends. That is, in this imaging process, one image with no pattern light and the maximum value imax images with pattern light are acquired.

 FIG. 17 (c) is a flowchart of the three-dimensional measurement process (S1220 in FIG. 17 (a)). This processing is executed based on the luminance image generation program 36c, and first, a luminance image is generated (S1221). Here, the luminance is a Y value in the YCbCr space, and is a value calculated from Y = 0.2989−R + O.5866−G + 0.1145 · power from the RGB value of each pixel. By obtaining the Υ value for each pixel, a luminance image for each pattern light presence / absence image is generated. The generated luminance image is stored in the luminance image storage unit 37c. A number corresponding to the number of the pattern light is assigned to each luminance image.

 Next, the code image generation program 36d generates a code image coded for each pixel by combining the generated luminance images using the spatial coding method described above (S1222). .

 [0172] This code image is binarized by comparing each pixel of the brightness image related to the image with pattern light stored in the brightness image storage unit 37c with the brightness threshold set in advance or the image without pattern light. The result can be generated by assigning the LSB to MSB as described in FIGS. 16 (a) and 16 (b). The generated code image is stored in the code image storage unit 37d.

 Next, a code boundary coordinate detection process described later is performed by the code boundary extraction program 36e (S1223), and the boundary coordinates of the code assigned to each pixel are detected with sub-pixel accuracy.

 Next, lens aberration correction processing is performed by the lens aberration correction program 36f (S1224). By this processing, it is possible to correct the error of the code boundary coordinates detected in S1223, which includes the error due to the influence of the distortion of the imaging optical system 21, and the like.

Next, a real space conversion process based on the triangulation principle is performed by the triangulation calculation program 36g (S1225). The code boundary coordinates in the CCD space after the aberration correction has been performed by this process are converted into three-dimensional coordinates in the real space, and the three-dimensional shape detection result is obtained. Three-dimensional coordinates are determined.

 FIG. 19 is a diagram for explaining the outline of the code boundary coordinate detection process (S1223 in FIG. 17). In the upper diagram of FIG. 19, the boundary between the light and dark of the actual pattern light in the CCD space is indicated by a boundary line K, and the code 1 when the pattern light is coded by the space coding method described above. The boundary between the code and another code is indicated by a bold line in the figure.

 [0177] That is, since the coding in the spatial coding method described above is performed on a pixel-by-pixel basis, the sub-pixel accuracy is defined between the actual pattern light boundary K and the coded boundary (thick line in the figure). Error occurs. Therefore, the purpose of this code boundary coordinate detection processing is to detect code boundary coordinates with subpixel accuracy.

 In this process, first, at a certain detection position (hereinafter, referred to as “curCCDX”), a first pixel G that changes from a certain code of interest (hereinafter, “curCode”) to another code is detected (first pixel G). Detection step).

 For example, when each pixel is detected in order from the top in curCCDX, curCode is changed at the next pixel of the force boundary, that is, the first pixel G up to the boundary (thick line), which is a pixel having a curCode. This is detected as the first pixel G.

 Next, at the pixel position of the first pixel G, all of the luminance images having a change in brightness are extracted from among the luminance images stored in the luminance image storage unit 37c in S1221 in FIG. (Brightness image extraction step).

 Next, the detection position is moved to the left by “2” in order to specify a pixel region to be used for approximation, and at the position of the detection position curCCDX-2, the code image of interest is referenced by referring to the code image. (CurCode) force A pixel that changes to another code (boundary pixel (pixel H at the detection position of curCCDX-2)) is searched, and a predetermined range (one in the Y-axis direction in this embodiment) is set around that pixel. The pixel range of (pixels and +2 pixels) is specified (part of the pixel region specifying step).

 Next, within the predetermined range, as shown in the lower left graph in FIG. 19, an approximate expression (indicated by a solid line in the figure) regarding the pixel position and the luminance in the Y direction was obtained, The Y coordinate Y1 at the intersection with the luminance threshold bTh in the approximate expression is obtained (part of the boundary coordinate detection step).

Note that the luminance threshold bTh is calculated from a predetermined range (for example, the luminance threshold Even if it is a fixed value that is given in advance, it is OK. Thus, the boundary between light and dark can be detected with sub-pixel accuracy.

[0184] Next, the detection position is moved to the right side of "1" from curCCDX-2, and the same processing as described above is performed for curCCDX-1 to obtain a representative value in curCCDX_l (boundary boundary Part of the target detection process).

 [0185] As described above, a pixel area including a predetermined range in the Y-axis direction around the boundary pixel and a range from curCCDX-2 to curCCDX + 2 in the X-axis direction (see a hatched portion falling rightward in the figure) In), a representative value at each detection position is obtained.

 [0186] The above processing is performed on all luminance images having pixels that change to other codes, such as curCode force, and the weighted average value of the representative values for each luminance image is finally set as the boundary coordinates in curCode. Adopt (part of the boundary coordinate detection process).

 As a result, the boundary coordinates of the code can be detected with high accuracy and subpixel accuracy, and the real space conversion processing (S1225 in FIG. 17) based on the triangulation principle described above is performed using the boundary coordinates. This makes it possible to detect the three-dimensional shape of the subject with high accuracy.

 [0188] Further, since the boundary coordinates can be detected with sub-pixel accuracy using the approximate expression calculated based on the luminance image in this manner, the number of captured images can be increased as in the related art. It is not necessary to use a gray code, which is a special pattern light that can be used even if a pattern light that is lit and shaded by a pure binary code is used.

 In the present embodiment, at each detection position, a range from “−3” to “+2” in the Y-axis direction around the boundary pixel, and curCCDX—2 to curCCD as the detection position in the X-axis direction. Although the region constituted by the range of X + 2 has been described as a pixel region for obtaining an approximation, the range of the Y-axis and the X-axis of this pixel region is not limited thereto. For example, only a predetermined range in the Y-axis direction around the boundary pixel at the cur CCDX detection position may be set as the pixel region.

FIG. 20 is a flowchart of the code boundary coordinate detection process (S1223 in FIG. 17). This process is executed based on the code boundary extraction program 36e. First, each element of the code boundary coordinate sequence in the CCD space is initialized (S1401), and curCCDX is set as a start coordinate (S1402). Next, it is determined whether or not curCCDX is equal to or smaller than the end coordinate (S1403). If curCCDX is equal to or smaller than the end coordinate (S1403: Yes), curCode is set to “0” (S1404). That is, curCode is initially set to the minimum value.

 Next, it is determined whether or not curCode is smaller than the maximum code (S1405). If curCode is smaller than the maximum code (S1405: Yes), the code image is referred to in curCCDX to search for a pixel of curCode (S1406). Next, it is determined whether or not a pixel of curCode exists (S1407).

 As a result, if there is a pixel of curCode (S1407: Yes), a pixel of Code larger than the curCode is searched for in curCCDX by referring to the code image (S1408). Next, it is determined whether or not a pixel having a curCode larger than the curCode exists (S1409).

 [0194] As a result, if there is a pixel having a code larger than the curCode (S1409: Yes), a process of obtaining a boundary described later with sub-pixel accuracy is performed (S1410). Then, “1” is added to the curCode for obtaining the boundary coordinates for the next curCode (S1411), and the processing from S1405 is repeated.

 That is, since the boundary exists at the pixel position of the pixel having the curCode or the pixel position of the pixel of the Code larger than the curCode, in the present embodiment, the boundary is provisionally set to the curCode larger than the curCode. The processing proceeds assuming that the pixel is located at the pixel position of the pixel.

 [0196] Also, when curCode does not exist (S1407: No), or when there is a pixel having a Code larger than curCode, or when it does not exist (S1409: No), the next curCode First, “1” is added to the curCode for calculating the boundary coordinates (S1411), and the processing from S1405 is repeatedly exempted.

 [0197] In this way, the processing up to S1405 and S1411 is repeated for curCode up to the maximum code and zero power, and when curCode becomes larger than the maximum code (S1405: No), the detection position must be changed. Is added to “dCCDX” (S1412), and the process from S1403 is repeated at the new detection position in the same manner as described above.

[0198] The curCCDX is changed, and when the curCCDX finally becomes larger than the end coordinate (S14 03: N〇), that is, when the detection from the start coordinate to the end coordinate is completed, the process ends. To do.

 FIG. 21 is a flowchart of the process (S1410 in FIG. 20) for obtaining the code boundary coordinates with sub-pixel accuracy.

 [0200] In this processing, first, from among the brightness images stored in the brightness image storage unit 37c in S1221 in Fig. 17, at the pixel position of a pixel having a Code larger than the curCode detected in S1409 in Fig. 20. Then, all of the luminance images having the change in brightness are extracted (S1501).

 [0201] Next, the mask pattern number of the extracted luminance image is stored in array PatID [], and the number of images of the extracted luminance image is stored in noPatID (S1502). The arrays PatID [] and noPatID are stored in the ID storage unit 37f.

 Next, the counter i is initialized (S1503), and it is determined whether or not the value of the counter i is smaller than oPatID (S1504). As a result, if it is determined to be small (S1504: Yes), the CCD Y value of the boundary is obtained for the luminance image having the mask pattern number of PatID [i] corresponding to the counter i, and the value is calculated as f CCDY [i ] (S1505).

 When the process of S1505 ends, “1” force S calories are calculated in counter i (S1506), and the process from S1504 force is repeated. Then, in S1504, when it is determined that the value of the counter i is greater than 0 & «0 (S1504: No), that is, when the processing of S1505 is completed for all the luminance images extracted in S1501, the processing of S1505 The weighted average value of fCCDY [i] obtained in is calculated, and the result is used as the boundary value (S1507).

 [0204] Instead of the weighted average value, the median value of fCCDY [i] obtained in the process of S1505 is calculated, and the result is used as a boundary value, or the boundary value is calculated by statistical calculation. Can also

[0205] That is, the boundary coordinates are represented by the coordinates of curCCDX and the weighted average value obtained in S1507, the boundary coordinates are stored in the code boundary coordinate storage unit 37e, and the process ends.

 [0206] Fig. 22 shows the boundary CCD image for a luminance image with a mask pattern number of PatID [i]

22 is a flowchart of a process for obtaining a Y value (S1505 in FIG. 21).

[0207] In this process, first, a large value between "curCCDX—dx" and "0" is set as ccdx. Then, the process represented by “ccdx = MAX (curCCDX−dx, 0)” is performed, and the counter j is initialized (S1601).

 [0208] Specifically, "0" in S1601 means the minimum value of the CCDX value. For example, the curxc DX value power S "1" as the detection position is now set in advance to the dx value. If the force S is “2”, “curCCDX—dx” is “1”, which is smaller than the minimum CCDX value of “0”. ccdx = 0 ”is set.

 [0209] That is, for a position smaller than the minimum value of the CCDX value, processing for excluding subsequent processing is performed.

 [0210] The value of "dx" can be set to an appropriate integer including "0" in advance, and in the example described in Fig. 19, "dx" is set to "2". Therefore, according to the example of FIG. 19, this ccdx is set to “curCCDX-2j”.

 Next, it is determined whether or not ccdx <= MIN (curCCDX + dx, ccdW−1) (S1602). In other words, "MIN (curCCDX + dx, ccdW-1)" on the left side is the smaller of "curCCDX + dx" and "ccdW-l" obtained by subtracting "1" from the maximum CCDX value "ccdW". Since this means that the value is a value, the value is compared with the "ccdx" value.

 That is, for a position larger than the maximum value of the CCDX value, a process for excluding the subsequent processes is performed.

 [0213] Then, as a result of the determination, if ccdx is smaller than MIN (curCCDX + dx, ccdW-1) (S1602: Yes), the code image and the luminance image to which PatID [i] is assigned are referenced to determine the boundary. The eCCDY value at the pixel position of the pixel where the field exists is determined (S1603).

 For example, assuming that the detection position is curCCDX-1 shown in FIG. 19, pixel I is detected as a pixel candidate having a boundary, and an eCCDY value is obtained at the position of pixel I.

 [0215] Next, from the luminance image having the mask pattern number of PatID [i], in the range of MAX (eCCDY_dy, 0) <= ccdy <= MIN (eCCDY + dy-l, ccdH_l) in the ccdy direction. An approximate polynomial Bt = fb (ccdy) relating to the luminance is calculated (S1604).

Next, a ccdy value at which the approximate polynomial Bt intersects with the luminance threshold value bTh is determined, and the ccdy value is assigned to the value efCCDY [j] (S1605). By using S1604 and S1605, it is possible to detect boundary coordinates with sub-pixel accuracy. Next, “1” is added to each of ccdx and counter j (S1605), and the processing from S1602 is repeated. That is, a boundary of sub-pixel accuracy is detected at each detection position within a predetermined range on the left and right with respect to curCCDX.

 [0218] Then, in S1602, if the "ccdx" force S is larger than the S "MIN (curCCDX + dx, ccdW-1)" and the IJ is cut off (S1602: No), the curCCDX-dx For efCCDY [j] calculated within the range of curCCDX + dx, an approximate polynomial of ccdy = fy (ccdx) is obtained (S1606). Since each value detected in step S1605 is used in this process, it is possible to improve the detection accuracy of the boundary coordinates as compared with the case where the detection of the boundary coordinates is performed at one detection position.

 [0219] The intersection of the thus obtained approximate polynomial and curCCDX is set as the CCDY value of the boundary for the luminance image having the mask pattern number of PatID [i] (S1607), and the process ends. As shown in the flowchart in Fig. 15, the processing up to this point is performed on one of the extracted luminance images, and a weighted average value is calculated for the obtained boundary coordinates. (1507), the detection accuracy of the boundary coordinates can be further improved.

 [0220] Figs. 23 (a) to 23 (c) are diagrams for explaining the lens aberration correction processing (S1224 in Fig. 17). As shown in FIG. 23 (a), the lens aberration correction process is performed for the case where the incident light flux is shifted from the position where the image should be formed by the ideal lens due to the aberration of the imaging optical system 21. This is a process for correcting the position to the position where the image should be originally formed.

 This aberration correction is performed by, for example, calculating the aberration of the optical system in the imaging range of the imaging optical system 21 using the half angle of view hfa, which is the angle of the incident light, as shown in FIG. Correct based on the data obtained.

 [0222] This aberration correction processing is executed based on the lens aberration correction program 36f, and is performed on the code boundary coordinates stored in the code boundary coordinate storage unit 37e. Stored in part 37g.

[0223] Specifically, the camera calibration (approximation) of the following (a) to (c) for converting the arbitrary point coordinates (ccdx, ccdy) in the real image to the coordinates (ccdcx, ccdcy) in the ideal camera image The correction is performed using the following expression. [0224] In the present embodiment, the aberration amount dist (%) is described as dist = f (Ma) using the half angle of view hfa (deg). The focal length of the imaging optical system 21 is focal length (mm), the ccd pixel length is pixel length (mm), and the center coordinates of the lens in the CCD 22 are (Centx, Centy).

 [0225] (a) ccdcx = (ccdx- Centx) / (1 + dist / lOO) + Centx

 (b) ccdcy = (ccdy- Centy) / (l + dist / lOO) + Centy

(c) hfa = arctan [(((ccdx—Centx + (ccdy—Centy) ² )) X pixellength / focallength]

 FIGS. 24 (a) and 24 (b) are diagrams for explaining a method of calculating three-dimensional coordinates in a three-dimensional space from coordinates in a CCD space in a real space conversion process (S1225 in FIG. 17) based on the principle of triangulation. FIG.

 [0226] In the real space conversion processing based on the principle of triangulation, the three-dimensional coordinates in the three-dimensional space of the aberration-corrected code boundary coordinates stored in the aberration-corrected coordinate storage unit 37g are calculated by the triangulation operation program 36g. Is calculated. The three-dimensional coordinates calculated in this way are stored in the three-dimensional coordinate storage unit 37h.

 In the present embodiment, as the coordinate system of the image input / output device 1 with respect to the horizontally curved subject P to be imaged, the optical axis direction of the imaging optical system 21 is the Z axis, and the imaging optical system is along the Z axis. The origin of a point away from the principal point force VPZ of the system 21 is defined as the origin, the X axis is horizontal to the image input / output device 1, and the Y axis is vertical.

 [0228] Also, the projection angle θp from the image projection unit 13 to the three-dimensional space (X, Υ, Z), the distance between the optical axis of the imaging optical system 21 and the optical axis of the image projection unit 13 is D, Yf opto-force Yfbottom for the Y-direction field of view of the imaging optical system 21, Xfstart force Xfend for the X-direction field of view, Hc the length (height) in the Y-axis direction of the CCD22, and Wc the length (width) in the X-axis direction And The projection angle Θp is given based on a code assigned to each pixel.

 [0229] In this case, the three-dimensional space position (X, Y, Z) corresponding to the arbitrary coordinates (ccdx, ccdy) of the CCD 22 is a point on the imaging plane of the CCD 22, a projection point of the pattern light, It can be obtained by solving five equations for the triangle formed by and the point that intersects the Y plane. (l) Y =-(ta η θ ρ) Ζ + ΡΡΖ + tan θ ρ _ D + cmp (Xtarget)

(2) Y =-(Ytarget / VPZ) Z + Ytarget (3) X =-(Xtarget / VP) Z + Xtarget

 (4) Ytarget = Yf top-(ccdcy / Hc) X (Yf top-Yf bottom)

 (5) Xtarget = Xf start + (ccdcx / Wc) X (Xfend— Xfstart)

 Note that cmp (Xtarget) in the above equation (1) is a function for correcting the displacement between the imaging optical system 21 and the image projection unit 13. In an ideal case where there is no displacement, cmp (Xtarget) = 0. It can be regarded as S. This equation can be used for an object having an arbitrary three-dimensional shape, not limited to the curved object P.

On the other hand, as described above, arbitrary coordinates on the projection LCD 19 included in the image projection unit 13

 The relationship between (lcdcx, lcdcy) and the three-dimensional coordinates (X, Υ, Z) in the three-dimensional space can be expressed by the following equations (6) to (9).

 In the present embodiment, the principal point position (0, 0, PPZ) of the image projection unit 13, the field of view in the Y direction of the image projection unit 13 is Ypftop force, Ypfbottom, and the field of view in the X direction is Xpfstart to Xpfend. The length (height) of the LCD 19 in the Y-axis direction is Hp, and the length (width) in the X-axis direction is Wp.

 (6) Y =-(Yptarget / PPZ) Z + Yptarget

 (7) X =-(Xptarget / PPZ) Z + Xptarget

 (8) Yptarget = Ypf top-(lcdcy / Hp) X (Ypf top-Ypf bottom)

 (9) Xptarget = Xpf start + (lcdcx / Wp) X (Xpfend— Xpfstart)

 By using this relational expression, it is possible to calculate the LCD space coordinates (lcdcx, lcdcy) by giving the three-dimensional space coordinates (X, Υ, Z) to the above equations (6) to (9). it can. Therefore, for example, it is possible to calculate an LCD element pattern for projecting an arbitrary shape and character in a three-dimensional space.

 FIG. 25 is a flowchart of the flattened image processing (S 611 in FIG. 7). The flattened image processing is performed, for example, when a document (subject) P in a curved state such as a book is imaged or when a rectangular document P is imaged from an oblique direction (the imaged image becomes trapezoidal). Even so, this is a process of acquiring and displaying a flattened image corrected in a state where the original is not curved or in a state where the original is imaged in a direction perpendicular to the surface.

In this process, first, a high-resolution setting signal is transmitted to the CCD 22 (S1901).

[0234] Next, the release button 8 was scanned (S1903a), and the release button 8 was half-pressed. Is determined (SI 903b). If the release button 8 is half-pressed (SI 903b: Yes), the auto focus (AF) and automatic exposure (AE) functions are activated, and the focus, iris, and shutter speed are adjusted (S1903c). If the release button 8 is not half-pressed (SI 903b: No), the processing power of the SI 903a is returned.

 Next, the release button 8 is scanned again (S1903d), and it is determined whether or not the release button 8 is fully pressed (S1903e). If the release button 8 is fully pressed (S1903e: Yes), it is determined whether or not the flash mode is set (S1903f).

 As a result, in the flash mode (S1903f: Yes), the flash 7 is emitted (S1903g), and the photographing power is performed (S1903h). If not in the flash mode (SI 903f: No), shooting is performed without emitting the flash 7 (S1903h). If it is determined in S1903e that it has not been fully attacked (S1903e: No), the processing power such as SI 903a will be returned.

 Next, the same three-dimensional shape detection process as the above-described three-dimensional shape detection process (S1006 in FIG. 15) is performed, and the three-dimensional shape of the subject is detected (S1906).

 Next, based on the three-dimensional shape detection result obtained by the three-dimensional shape detection process (S1906), a document posture calculation process for calculating the posture of the document P is performed (S1907). Through this process, the position L, the angle Θ, and the curvature φ (X) of the document P with respect to the image input device 1 are calculated as the posture parameters of the document P.

 Next, a plane conversion process described later is performed based on the calculation result (S 1908). According to this plane conversion processing, even if the document P is curved, a flattened image that is flattened to a state where the original P is not curved is generated.

[0240] Next, the flattened image obtained by the plane change process (S1908) is stored in the external memory 27 (S1909), and the flattened image is displayed on the monitor LCD 10 (S1910).

[0241] Then, it is determined whether or not the mode switching switch 9 has changed (S1911).

If there is no change (S1911: Yes), the processing from S702 is repeated again. If there is a change in the mode switching switch 9 (S1911: No), the process ends.

FIG. 26 (a) Force FIG. 26 (c) is a diagram for explaining the document posture calculation process (S1907 in FIG. 25). It is assumed that the curvature of the document P is uniform in the y direction as a precondition for a document such as a book. To do. In this document attitude calculation process, first, as shown in FIG.

From the coordinate data on the code boundary stored in 37h

Two curves are obtained by regression curve approximation of the points in two columns.

[0243] For example, such a curve is represented by position information (a boundary between the boundary between the code 63 and the code 64 and the boundary between the code 191 and the code 192) of the upper and lower quarters of the range where the pattern light is projected. You can ask for it.

[0244] Assuming a straight line connecting the points where the positions of the two curves in the X-axis direction are "0", the point at which this linear force axis intersects, that is, the point at which the optical axis intersects with document P, is defined as document P. 3D space position (0

, 0, and the angle between the straight line and the XY plane is defined as the inclination Θ of the document P about the X axis.

Next, as shown in FIG. 26 (b), the document P is rotationally transformed in the opposite direction by the previously obtained inclination ま わ り around the X axis, that is, the document P is rotated with respect to the X_Y plane. Assume a parallel state.

Then, as shown in FIG. 26 (c), for the section of the original に お け る on the Χ_Ζ plane, the displacement in the Ζ-axis direction can be represented by the curvature φ (X) as a function of X. In this way, the position L, angle Θ, and curvature φ (X) of the document Ρ are calculated as the document orientation parameters, and the process ends.

FIG. 27 is a flowchart of the plane conversion process (S1908 in FIG. 25). In this processing, first, a processing area of the processing is allocated to the working area 371 of the RAM 37, and a variable of a counter b used for the processing is set to an initial value (b = 0) (S2101).

 Next, based on the position L of the document P, the inclination Θ, and the curvature Φ (X) based on the calculation result of the document posture calculation program 36h, the pattern light stored in the pattern lightless image storage unit 37b is stored. The four corner points of the image are moved by L in the Z direction, rotated by Θ in the X axis direction, and then rotated by Θ, and then inversely transformed into φ (X) (equivalent to the “bending process” described later) A rectangular area (that is, a rectangular area in which the surface of the original P on which characters and the like are written is an image as viewed from a substantially orthogonal direction) is set at the point determined by The number of pixels a included in is obtained (S2102).

[0249] Next, coordinates on the pattern light no-image corresponding to each pixel constituting the set rectangular area are obtained, and pixel information of each pixel of the flattened image is obtained from pixel information around the coordinates. Is set. That is, first, it is determined whether or not the counter b has reached the number of pixels a (S2103). If the counter b does not reach the number of pixels a (S2103: No), the curvature calculation processing for rotating one pixel constituting the rectangular area by the curvature φ (X) about the Y axis is performed ( S2 104), tilt around the X axis Θ Rotate and move (S2105), and shift by distance L in the Z axis direction (S2106).

 [0251] Next, the coordinates (ccdcx, ccdcy) on the CCD image captured by the ideal camera are obtained by the inverse function of the previous triangulation for the obtained three-dimensional space position (S2107), and are used. According to the aberration characteristic of the imaging optical system 20, the coordinates (ccdx, ccdy) on the CCD image captured by the actual camera are obtained by the inverse function of the previous camera calibration (S2108), and the position corresponding to this position is obtained. The state of the pixel of the pattern light non-image is obtained and stored in the working area 371 of the RAM 37 (S2109).

 Next, “1” is added to the force counter b for executing the processing from S2103 to S2109 for the next pixel (S2110).

 [0253] Thus, when the processing from S2104 to S2110 is repeated until the counter b reaches the number of pixels a (S2103: Yes), in S2101, the processing area is allocated to the working area 371 to execute the processing. The processing area is released (S2111), and the processing ends.

 [0254] Fig. 28 (a) is a diagram for explaining the outline of the bending process (S2104 in Fig. 27), and Fig. 28 (b) is flattened by the plane conversion process (S1908 in Fig. 25). The original P. The details of the curving process are disclosed in detail in IEICE Transactions DII Vol.J86-D2 No.3 p409 “Bending Document Photograph Using Eye Scanner”.

 [0255] The curvature Ζ = φ (x) is obtained by converting the three-dimensional shape composed of the calculated code boundary coordinate sequence (real space) into a plane-cut cross-sectional shape parallel to the XZ plane at an arbitrary Y value. It is represented by an equation approximated by a polynomial using the least squares method.

[0256] When planarizing a curved surface, as shown in Fig. 28 (a), the flattened point corresponding to the point on Ζ = φ (χ) is calculated from Ζ = φ (0) to Ζ = Corresponding to the length of the curve up to φ (X).

[0257] By the plane conversion process including such a bending process, for example, a document in a curved state Even when P is imaged, a flattened planar image can be obtained as shown in Fig. 22 (b), and using such a flattened image improves the accuracy of OCR processing. be able to. Therefore, the characters and figures written on the document can be clearly recognized from the flattened image.

In the digital camera processing shown in FIG. 9, the webcam processing shown in FIG. 11, and the stereoscopic image processing shown in FIG. 15, when projecting a distortion-free projection image without depending on the projection direction, FIG. In place of the projection processing (S702f, S802f, S1010) executed in the same manner as the projection processing in S806, a projection image conversion processing (S2900) described later is executed.

 FIG. 29 is a flowchart of the distortion-free projection image conversion process (S2900). The distortion-free projection image conversion process (S2900) is a process of converting an image displayed on the projection LCD 19 into an image that can be projected onto a subject without distortion according to image information stored in the projection image storage unit 37k. is there.

 In this processing, first, a processing area of the processing is allocated to the working area 371 of the RAM 37, and a variable of a counter q used for the processing is set to an initial value (q = 0) (S2901) .

 [0261] Next, as a rectangular area that is an image after being transformed into a distortion-free projection image (that is, an image that has no distortion on a curved subject), the rectangular area in the LCD space coordinates (lcdcx, lcdcy) is used. Corresponding memory is secured and set in the working area 371 of the RAM 37, and the number Qa of pixels included in this rectangular area is obtained (S2902).

 [0262] Next, each pixel value of image information stored in the projection image storage unit 37k, such as the message image M or the captured image f, is allocated to each pixel of the ideal camera image coordinate system (ccdcx, ccdcy). Is performed (S2903).

[0263] Next, for each pixel on the LCD space coordinates (kdcx, lcdcy) constituting the set rectangular area, the three-dimensional coordinates are stored in the three-dimensional coordinate storage unit 37h by the above equations (6) to (9). The three-dimensional coordinates (X, Υ, Z), which are the corresponding points on the surface of the subject, are obtained, and by solving equations (ccdcx, ccdcy) in equations (1) to (5), the distortion-free projection Pixel information of each pixel of the image is calculated and set. That is, first, it is determined whether or not the counter q has reached the number of pixels Qa (S2904). If the power counter q does not reach the number of pixels Qa (S2904: No), the LCD space coordinates (lcdcx, lcdcy) of the pixel corresponding to the value of the counter q are calculated by the equations (6) to (9). Is converted into coordinates (X, Υ, Z) on the subject stored in the working area 371 (S2905).

 Next, the coordinates (X, Υ, Z) on the object obtained by the conversion in the process of S2905 are obtained by using the equations solved for (ccdcx, ccdcy) in equations (1) to (5). Then, it is converted to coordinates (ccdcx, ccdcy) on the ideal camera image (S2906).

 [0266] Next, pixel information arranged at the coordinates (ccdcx, ccdcy) obtained by the conversion in the process of S2906 is obtained, and the pixel information is stored in the LCD spatial coordinates (LCD) corresponding to the value of the counter q. lcdcx, lcdcy) (S2907).

 [0267] Then, "1" is added to the counter q in order to execute the above-described processing from S2904 to S2907 for the next pixel (S2908).

[0268] Thus, the processing from S2904 to S2908; ^, when the counter q is repeated until the number of pixels becomes Qa (S2904: Yes), the LCD space coordinates (lcdcx, lcdcy, which constitute the set rectangular area) pixel information associated with the), is transferred to the projection LCD driver 30 (S2909) ₀

 [0269] Finally, in S2901, the processing area allocated to the working area 371 to execute the processing is released (S2910), and the processing ends.

 [0270] By the process of S2909, the pixel information on the LCD space coordinates (lcdcx, lcdcy) is transferred to the projection LCD driver 30, so that the projection LCD 19 outputs the projection image projected without distortion on the distorted surface. Is displayed. Therefore, an undistorted image is projected on the subject.

 [0271] Therefore, by performing the distortion-free projection image conversion process (S2900), even if the projection direction is oblique, even if the projection surface is curved, the projection image without distortion can be obtained. Projecting power S can. As a result, especially when projecting a character string such as the message image M, the user can accurately recognize the information.

FIGS. 30 (a) and 30 (b) are views for explaining a light source lens 50a as another example of the light source lens 18 in the above-described embodiment. FIG. 30A is a side view showing the light source lens 50a, and FIG. 30B is a plan view showing the light source lens 50a. In addition, The same members as those described above are denoted by the same reference numerals, and description thereof will be omitted.

 [0273] The light source lens 18 in the above-described embodiment is configured by arranging a lens portion 18a having a convex aspherical shape corresponding to each LED 17 on a base 18b and being arranged integrally. In the example shown in FIG. 30 (a) and FIG. 30 (b), a bullet-shaped resin lens containing each of the LEDs 17 is formed separately.

 As described above, by separately configuring the light source lens 50a including each LED 17, the position of each LED 17 and the corresponding optical lens 50a can be determined on a one-to-one basis. The relative positional accuracy can be improved, and there is an effect that the light emitting directions are aligned.

 [0275] On the other hand, when the lens array is aligned on the substrate 16 at a time, the light emission due to the positioning error when each LED 17 is die-bonded and the difference in the linear expansion coefficient between the lens array and the substrate. There is a risk that directions will be scattered.

 [0276] Therefore, the surface of the projection LCD 19 is irradiated with light whose incident direction from the LED 17 is aligned perpendicular to the surface of the projection LCD 19, and can uniformly pass through the stop of the projection optical system 20. Illuminance unevenness of the projected image can be suppressed. As a result, the ability to project high-quality images can be achieved. The LED 17 included in the light source lens 50a is mounted on the substrate 16 via an electrode 51a composed of a lead and a reflector.

 [0277] A frame-shaped elastic fixing member 52a that bundles the light source lenses 50a and regulates them in a predetermined direction is arranged on the outer peripheral surface of the group of light source lenses 50a. The fixing member 52a is made of a resin material such as rubber or plastic.

 [0278] Since the light source lens 50a is formed separately from each LED 17, the light source lens 50a faces the projection LCD 19 with the angle of the optical axis formed by the convex tip end of each light source lens 50a correctly aligned. It is difficult to install.

[0279] Therefore, in the example shown in Fig. 30 (a) and Fig. 30 (b), the fixing member 52a surrounds a group of light source lenses 50a, and the outer peripheral surfaces of the light source lenses 50a are brought into contact with each other. The position of each light source lens 50a is regulated so that the optical axis of the lens 50a faces the projection LCD 19 at a correct angle. According to such a configuration, light can be emitted from the light source lenses 50a to the projection LCD 19 almost vertically. Therefore, the uniform light is radiated on the surface of the projection LCD 19, and it can pass through the aperture of the projection lens uniformly. Degree unevenness can be suppressed. Therefore, a higher quality image can be projected.

[0280] Note that the fixing member 52a may have rigidity specified to a predetermined size in advance, and may be made of a material having elasticity, and the position of each light source lens 50 may be determined by the elasticity. May be restricted to a predetermined position.

FIGS. 31 (a) and 31 (b) show another example of the fixing member 52a for restricting the light source lens 50a to a predetermined position described in FIGS. 30 (a) and 30 (b). FIG. 9 is a view for explaining a fixing member 60. FIG. 31A is a perspective view showing a state in which the light source lens 50a is fixed.

FIG. 31 (b) is a partial sectional view thereof. The same members as those described above are denoted by the same reference numerals, and description thereof will be omitted.

[0282] The fixing member 60 is formed in a plate shape with a through-hole 60a having a cross section along the outer peripheral surface of each light source lens 50a and having a conical cross section in a sectional view. Each light source lens 50a is inserted and fixed in each of the through holes 60a.

[0283] An elastic urging plate 61 is interposed between the fixing member 60 and the substrate 16, and an electrode is provided between the urging plate 61 and the lower surface of each light source lens 50a. An annular O-ring 62 having elasticity is arranged so as to surround 51a.

[0284] The LED 17 included in the light source lens 50a is mounted on the substrate 16 via the biasing plate 61 and the electrode 51a penetrating through holes formed in the substrate 16.

According to the fixing member 60 described above, each light source lens 50a is fixed by penetrating through each through hole 60a having a cross section along the outer peripheral surface of the light source lens. In addition, the optical axis of the light source lens 50 can be more reliably fixed so as to face the projection LCD 19 at a correct angle.

[0286] At the time of assembly, the LED 17 can be urged to the correct position by the urging force of the O-ring 62 and fixed.

 [0287] Further, an impact force that may be generated when the present apparatus 1 is carried can be absorbed by the elastic force of the O-ring 62, and the position of the light source lens 50a is shifted by the influence of the impact. As a result, it is possible to prevent the inconvenience that light cannot be irradiated from the light source lens 50a vertically toward the projection LCD 19.

[0288] The processing of S706 in the digital camera processing of Fig. 9 and the processing of S803 in the webcam processing of Fig. 11 The processing is regarded as projection imaging control means. The processing of S805 in the webcam processing of FIG. 11 is positioned as a projection image updating means. The processing of S1006 in the stereoscopic image processing of FIG. 15 is positioned as three-dimensional information detection means.

[0289] The processing of S1002g in the stereoscopic image processing of Fig. 15 is positioned as a three-dimensional information detection prohibition unit. The processing of S801 in the webcam processing of FIG. 11 is positioned as a resolution lowering transmission means.

 [0290] Although the present invention has been described based on the embodiments, the present invention is not limited to the above embodiments, and various improvements and modifications can be made without departing from the gist of the present invention.

 For example, in the above embodiment, the process of acquiring and displaying a flattened image has been described as the flattened image mode. However, by mounting a well-known OCR function on the image input / output device 1, Even if the image input / output device 1 is configured so that the converted planar image can be read by this OCR function, it is acceptable. In the case of such a configuration, a great effect is obtained in that the text written on the document can be read with higher accuracy than when the document that is curved by the OCR function is read.

 In the web cam process of FIG. 11 in the above embodiment, the process of transmitting the low resolution setting signal to the CCD 22 (S801) is replaced with the process of transferring from the CCD 22 to the cache memory 28 (S807) or the captured image In the process of transmitting to the network (S808), the resolution of the captured image may be reduced before transmission. Also in this case, the subsequent processing speed can be improved by lowering the resolution of the captured image.

 Also, in S1501 of FIG. 21 in the above embodiment, all the luminance images having a change in brightness are extracted, and a case where a provisional CCDY value is obtained for all of the luminance images has been described. The number of luminance images is not limited to one if it is one or more, not necessarily the whole. The boundary coordinates can be obtained at high speed by reducing the number of sheets to be extracted.

[0294] In S1507 of Fig. 21 in the above embodiment, fCCDY [i] is weighted and averaged, and in S1607 of Fig. 22, each value is averaged using efCCDY [j] as an approximate polynomial. Although explained, the method of averaging each value is not limited to these. For example, a method of taking the simple average of each value, a method of using the median of each value, a method of calculating an approximate expression of each value, and using the detection position in the approximate expression as a boundary coordinate, and a statistical operation A method for obtaining the information may be adopted.

 [0295] For example, in the three-dimensional shape detection process in the flattened image mode in the above embodiment, in order to detect the three-dimensional shape of the document P, a striped pattern light in which a plurality of types of light and dark are alternately arranged. Although the description has been given of the case of projecting a three-dimensional shape, the light for detecting the three-dimensional shape is not limited to the power and the pattern light.

 For example, as shown in FIG. 32, in the case where the three-dimensional shape of a curved document is easily detected, even if two band-shaped slit lights 70, 71 are projected from the image projection unit 13, it is acceptable. ,. In this case, it is possible to detect a three-dimensional shape from two captured images at a higher speed than in the case of projecting eight pattern lights.

 [0297] In the image projection unit 13 in the above embodiment, the light source lens 18, the projection LCD 19, and the projection optical system 20 are provided in a straight line along the projection direction. The configuration of 13 is not necessarily limited to this.

 FIG. 33 (a) to FIG. 33 (c) are diagrams showing modified examples of the image projection unit 13. FIG. 33 (a) is a plan view of the imaging head 2, and FIGS. 33 (b) and 33 (c) are cross-sectional views taken along line AA of FIG. 33 (a). Note that, in FIG. 33 (a), the illustration of the cover 2a is omitted to show the inside of the imaging head 2. As shown in FIGS. 33 (b) and 33 (c), the image projection unit is provided with a reflection mirror 200 in a projection optical system 20 composed of a plurality of lenses, and a direction of light emitted from the light source lens 18. May be changed to a predetermined projection direction and projected.

 [0299] In this way, since the optical path can be changed by the reflection mirror 200, the light source lens 18, the LCD 19, and the projection optical system 20 need not always be provided in a straight line along the projection direction. It can be stored compactly according to the shape of the head 2.

In the above embodiment, the light source is provided with the LED array 17A and the light source lens 18. Instead of such a configuration, the light source may be configured as shown in FIG. 33 (c). good. The light source means shown in FIG. 33 (c) includes a light source 170 such as a halogen lamp that emits white light or a white LED (Light Emitting Diode), and a light source 170 that emits white light. And a reflecting plate 180 for efficiently entering the LCD 19.

[0301] In each of the digital camera processing (see Fig. 9), the webcam processing (see Fig. 11), and the stereoscopic image processing (see Fig. 15) in the above embodiment, the digital camera is fixed to the small diameter portion l ib of the imaging case 11. By detecting the position of the rib 11c for detecting the position of the imaging unit by the imaging unit position detection sensor S2, the position of the image imaging unit 14 with respect to the imaging head 2 has been detected, but the image projection unit 13 and the image imaging unit Means for detecting the relative positional relationship with 14 is not limited to this. For example, based on the fact that the predetermined position fixing rib 2d is engaged with a stove (not shown) provided on one of the small diameter portions l ib of the imaging case 11, and further rotation of the imaging case 11 is restricted, It is OK even if it is detected that the image capturing unit 14 has a predetermined positional relationship with the image projecting unit 13. With this configuration, the detection of the three-dimensional shape is permitted only when the positional relationship between the image projecting unit 13 and the image capturing unit 14 surely becomes a predetermined positional relationship, and the detection of the three-dimensional shape is more improved. Can be done with precision.

[0302] In one embodiment of the present invention, the image input / output device includes a projection housing for holding projection means,

 The image processing apparatus may further include an imaging housing that holds the imaging unit and that is disposed so as to be relatively movable with respect to the projection housing. In this case, the spatial modulation means is constituted by a liquid crystal panel, the projection means has a projection optical system for forming an image signal light output from the liquid crystal panel on a predetermined projection surface, and the imaging means converts the light into electricity. A light-receiving element that converts the light into a signal and an imaging optical system that forms incident light on the light-receiving element may be provided.

[0303] According to such a configuration, the projection unit has the projection optical system that forms an image signal light output from the liquid crystal panel on a predetermined projection surface, and the imaging unit transmits incident light to the light receiving element. It has an imaging optical system for forming an image. In general, the light receiving element provided in the imaging means is often smaller than the liquid crystal panel provided in the projection means. In such a case, the imaging optical system of the imaging means is configured to be smaller than the projection optical system of the projection means. Further, the projection means is provided with a light source means. Therefore, it is easy to make the imaging housing holding the imaging means smaller than the projection housing holding the projection means, and the imaging housing is arranged so as to be relatively movable with respect to the projection housing. Is established [0304] Further, according to such an image input / output device, it is easy to make the imaging housing smaller than the projection housing, and the imaging housing moves relatively to the projection housing. Since it is provided as possible, there is an effect that the imaging means held by the imaging housing and the projection means held by the projection housing can be relatively easily moved.

 [0305] In one embodiment of the present invention, the projection unit may include a projection direction determination unit for projecting the light emitted from the light source unit in a predetermined projection direction.

 According to such a configuration, since the projection means has the projection direction determination means for projecting the light emitted from the light source means in a predetermined projection direction, the light source means, the spatial modulation means, The projection means can be provided outside the comparator in accordance with the shape of the projection housing in which it is not necessary to arrange the projection optical system on a straight line, and the projection housing can be downsized.

 [0307] In one embodiment of the present invention, the image input / output device includes a base supporting the projection unit and the imaging unit, and movably supporting the position of at least one of the projection unit and the imaging unit with respect to the base. And a supporting member.

 According to such a configuration, at least one of the projection unit and the imaging unit can be moved with respect to the base, so that even when the projection unit and the imaging unit are supported by the base, There is an effect that flexibility can be provided in a direction in which the image signal light can be projected by the projection means and in a direction in which the image signal can be captured by the imaging means.

 [0309] In one embodiment of the present invention, the support member has one end connected to at least one of the projection means and the imaging means, and has, at the other end, an attachment portion for attaching the support member to the base. May be provided. Further, at least one of the base and the support member may be provided with a power supply unit for supplying power to at least one of the projection unit and the imaging unit.

[0310] According to such a configuration, the power supply unit for supplying power to at least one of the projection unit and the imaging unit is provided on at least one of the base and the support member. It is possible to reduce the weight and to easily move the positions of the projection unit and the imaging unit with respect to the base. Further, since the weights of the projecting means and the imaging means are reduced, there is an effect that the strength of the support member is not so required as compared with the case where a power supply unit is provided in the projecting means and the imaging means. [0311] In one embodiment of the present invention, the image input / output device may include a projection imaging control unit that executes imaging by the imaging unit while projecting image signal light by the projection unit.

 [0312] According to such a configuration, imaging by the imaging unit is performed while projecting the image signal light by the projection unit. Here, since the imaging means can capture an image of an object existing in a direction different from the projection direction, for example, while projecting the image signal light in a direction different from the direction in which the user himself is present, the imaging means can be used. By imaging the user himself by means, there is an effect that the user can image himself while visually recognizing the image projected by the projection means. Further, for example, when the imaging unit captures an image of a subject existing in a direction different from the projection direction, the image signal light is not projected on the subject, so that the subject is not included in the image projected by the projection unit. It is possible to take an image of the above, which has an effect.

 [0313] In one embodiment of the present invention, the projection imaging control means may be configured to control the projection means to project the relevant information related to the captured image acquired by the imaging means by the projection means. .

[0314] According to such a configuration, since the related information on the captured image acquired by the imaging unit is projected by the projection unit, the user visually recognizes the related information on the captured image acquired by the imaging unit. In addition, there is an effect that imaging can be performed. The relevant information related to the imaging unit corresponds to, for example, a character string or a graphic indicating an operation method and an operation procedure of the imaging unit, and an imaging mode of the imaging unit.

[0315] In one embodiment of the present invention, the projection imaging control means may be configured to control the projection means to project the captured image of the subject imaged by the imaging means by the projection means.

According to such a configuration, the captured image of the subject captured by the imaging unit is projected by the projection means, so that the user can visually recognize the captured image of the subject captured by the imaging unit, There is an effect that imaging can be performed.

[0317] In one embodiment of the present invention, the image input / output device includes a projection image updating unit that updates the image signal light projected by the projection unit. According to such a configuration, since the image signal light projected by the projecting unit is updated, the user can execute imaging while visually confirming the update of the image projected by the projecting unit. There is an effect that can be.

 [0319] In one embodiment of the present invention, when an image signal light having a predetermined pattern is projected on a subject by a projecting unit, the image input / output device outputs an image captured by the imaging unit. A three-dimensional information detecting means for detecting three-dimensional information of the subject based on the three-dimensional information may be provided.

 According to such a configuration, when the image signal light having the predetermined pattern is projected on the object by the projecting means, the object is reproduced based on the image taken by the image sensing means. In the image input / output device in which the three-dimensional information is detected, by moving the imaging means and the projection means relatively to each other, it is possible to image a subject existing in a direction different from the projection direction. Therefore, there is an effect that the use of the projection unit and the imaging unit provided for detecting three-dimensional information in the image input / output device is expanded.

 [0321] In one embodiment of the present invention, the image input / output device has a position determination unit that determines whether the relative positional relationship between the imaging unit and the projection unit is within a predetermined range. Is also good.

 [0322] According to such a configuration, there is an effect that the position determination unit can determine that the relative positional relationship between the imaging unit and the projection unit is not within the predetermined range.

[0323] In one embodiment of the present invention, the image input / output device fixes the positional relationship between the imaging unit and the projection unit in a state where the relative positional relationship between the imaging unit and the projection unit is in a predetermined positional relationship. May be provided.

[0324] According to such a configuration, the positional relationship between the imaging means and the projection means is fixed by the fixing means in a state where the relative positional relationship between the imaging means and the projection means is in the predetermined positional relation. If it can be done, it has the effect.

[0325] In one embodiment of the present invention, the image input / output device includes a position determination unit that determines a relative positional relationship between the imaging unit and the projection unit, and the imaging unit and the projection unit that use the position determination unit. If it is determined that the relative positional relationship of In this case, a three-dimensional information detection prohibiting means for prohibiting the detection of the three-dimensional information by the three-dimensional information detecting means is provided.

 According to such a configuration, when the position determining unit determines that the relative positional relationship between the imaging unit and the projecting unit is not within a predetermined range, the three-dimensional information detecting unit is used. Since the detection of three-dimensional information due to is prohibited, it is possible to reliably detect high-precision three-dimensional information. That is, the three-dimensional information detecting means detects the three-dimensional information based on the image taken by the imaging means when the image signal light having a predetermined pattern is projected on the subject by the projecting means. I do. Therefore, the image signal light having the predetermined pattern cannot be projected onto the subject by the projection means in which the relative positional relationship between the imaging means and the projection means is within the predetermined range, or the accuracy is high. When the three-dimensional information cannot be obtained, the detection of the three-dimensional information is prohibited, so that the detection of the three-dimensional information with low accuracy can be suppressed.

 [0327] In one embodiment of the present invention, the image input / output device may include a resolution reduction transmitting unit that reduces the resolution of a captured image obtained by the imaging unit and transmits the reduced image to the outside.

 According to such a configuration, the resolution of the captured image obtained by the imaging unit is reduced and transmitted to the outside, so that there is an effect that the processing speed can be improved.

Claims

The scope of the claims

 [1] Light source means for emitting light, and spatial modulation means for spatially modulating light emitted from the light source means and outputting image signal light, and an image output by the spatial modulation means Projecting means for projecting the signal light in the projection direction;

 An image input / output device comprising: an imaging unit capable of capturing at least a subject existing in the projection direction and acquiring the captured data,

 The imaging unit and the projection unit are arranged such that an imaging direction of the imaging unit is relatively to a projection direction of the projection unit so that a subject existing in a direction different from the projection direction can be imaged by the imaging unit. An image input / output device, which is provided so as to be changeable.

[2] a projection housing for holding the projection means,

 An imaging housing for holding the imaging unit, the imaging housing being arranged to be relatively movable with respect to the projection housing,

 The spatial modulation means is constituted by a liquid crystal panel,

 The projection means has a projection optical system for forming an image signal light output from the liquid crystal panel on a predetermined projection surface,

 2. The image input / output device according to claim 1, wherein the imaging unit includes a light receiving element that converts light into an electric signal, and an imaging optical system that forms an image of incident light on the light receiving element.

 3. The image input / output apparatus according to claim 2, wherein the projection unit has a projection direction determination unit for projecting the light emitted from the light source unit in a predetermined projection direction.

[4] a base supporting the projection unit and the imaging unit,

 2. The image input / output device according to claim 1, further comprising a support member that movably supports at least one of the projection unit and the imaging unit with respect to the base.

[5] The support member has one end connected to at least one of the projection unit and the imaging unit, and has the other end attached to the base member. Has an attachment part,

 The image input / output device according to claim 4, wherein a power supply unit for supplying power to at least one of the projection unit and the imaging unit is provided on at least one of the base and the support member. .

6. The image input / output apparatus according to claim 1, further comprising: a projection imaging control unit that executes imaging by the imaging unit while projecting image signal light by the projection unit.

7. The projection imaging control unit, wherein the projection unit controls the projection unit to project related information on the captured image acquired by the imaging unit by the projection unit. Image input / output device.

[8] The image according to claim 6, wherein the projection imaging control means controls the projection means so as to project the captured image of the subject imaged by the imaging means by the projection means. I / O device.

9. The image input / output device according to claim 6, further comprising a projection image updating unit that updates an image signal light projected by the projection unit.

[10] When image signal light having a predetermined pattern is projected onto the subject by the projection unit, three-dimensional information of the subject is detected based on a captured image captured by the imaging unit. The image input / output device according to claim 1, further comprising three-dimensional information detection means.

 11. The image input / output device according to claim 1, further comprising a position determining unit that determines whether a relative positional relationship between the imaging unit and the projecting unit is within a predetermined range.

 [12] A fixing means for fixing a positional relationship between the imaging means and the projection means in a state where a relative positional relation between the imaging means and the projection means is the predetermined positional relation. 12. The image input / output device according to claim 11, wherein:

[13] A position determining means for determining a relative positional relationship between the imaging means and the projecting means, and a position within a predetermined range of the relative positional relationship between the imaging means and the projecting means by the position determining means. And a three-dimensional information detection prohibiting unit that prohibits the detection of the three-dimensional information by the three-dimensional information detecting unit when it is determined that the three-dimensional information is not related. The image input / output device according to claim 10, wherein

 2. The image input / output apparatus according to claim 1, further comprising: a resolution lowering transmission unit configured to reduce the resolution of a captured image acquired by the imaging unit and transmit the reduced image to the outside.