JP6790766B2 - Projection system, image processing equipment and programs - Google Patents

Description

The present invention relates to a projection system for projecting an image on a surface to be projected by one or more projection means, an image processing device, and a program for realizing the image processing device.

In recent years, digital signage using a projector has been performed. There are many curved parts around us, such as the sides and ceilings of the interior of trains, passages and tunnels with arched ceilings, and arcade shopping streets, where the plane is bent only in the vertical direction. In addition, advertisements are posted in the space above the train window. From such a background, there is a desire to project an image on a curved surface as described above.

When a desired image is projected onto a projectile by a projector, calibration work is performed to project the desired image onto a projectile to be a screen. For example, Japanese Patent Application Laid-Open No. 2015-056834 (Patent Document 1) is known as a calibration process in multi-projection. In the prior art of Patent Document 1, calibration is performed by projecting a calibration pattern and imaging it in a plurality of times using an imaging device such as a camera. With the above calibration, it becomes possible for a plurality of projectors to project an image on a flat screen with appropriate correction.

However, when a screen such as an inner wall of a cylinder projects on a projection body having a curved surface bent in one direction (for example, in the vertical direction), the observer may feel uncomfortable. For example, when observing a projected image projected on a projection body having a curved surface, when observing from a viewpoint distant from the position of the viewpoint of the camera at the time of calibration, the entire corrected image seems to bulge at both ends. It sometimes looked like.

The present invention has been made in view of the above points, and the present invention is a projection system for projecting an image on a projected surface by one or more projection means, the projected surface having a bend in one direction. On the other hand, it is an object of the present invention to provide a projection system capable of correcting a projected image with a simple operation.

The present invention has been made in view of the above points, and provides a projection system having the following features. This projection system includes a grid point extraction means that extracts grid point data indicating distortion of a projected image for each of the one or more projection means from each of the prepared one or more calibration captured images, and a grid point extraction means of the calibration captured image. Correction for each of the one or more projection means based on the information input means that receives the input of the geometric information of the projected surface including the relative positional relationship with the image pickup means, the grid point data, and the geometric information. Includes a correction coefficient calculation means for calculating the coefficient.

With the above configuration, in a projection system for projecting an image on a projected surface by one or more projection means, it is possible to correct the projected image with a simple operation on a projected surface having a bend in one direction. It becomes.

The schematic diagram which shows the whole structure of the image projection system by this embodiment. The functional block diagram of the image projection system according to this embodiment. The figure which illustrates two kinds of calibration images used in the image projection system by this embodiment. The figure explaining (A) perspective projection model and (B) parallel projection model. The figure explaining the discomfort given to the observer when projected onto the inner wall of a cylinder. The flowchart which shows the overall processing flow of the grid point extraction integrated processing, the cylinder correction processing, the calculation processing of various correction coefficients, and the correction processing based on the correction coefficient by this embodiment. The figure explaining the proofreading projection scene which is sequentially projected in this embodiment, and the photographing method thereof. The figure explaining the distortion of the projection image due to the shape of the cylindrical screen in this embodiment. The flowchart which shows the flow of the adjustment work in the calibration process by this embodiment. A table explaining how to adjust the calibration parameters in the adjustment work according to the present embodiment. The figure explaining the method of giving the initial value of the calibration parameter in this embodiment, and the derivation parameter and the derivation function derived from the calibration parameter for use in the internal calculation. In the present embodiment, (A) the relationship between the coordinate value on the x'axis along the cylinder and the coordinate value on the x-axis, the coordinate value on the x'axis along the cylinder, and the coordinate value on the u-axis of the captured image. The figure explaining the relationship with. FIG. 5 is a diagram illustrating (A) correction in the y-axis direction and (B) plane projection transformation performed before and after correction in the present embodiment. The flowchart which shows the cylinder correction processing which the cylinder correction part by this embodiment executes. The figure which shows the hardware configuration of the general-purpose computer apparatus which comprises the image processing apparatus by one or a plurality of embodiments.

Hereinafter, the present embodiment will be described, but the present embodiment is not limited to the embodiments described below. In the embodiment described below, as an example of the projection system, a video projection system including one or more projectors as projection means, one camera as an imaging means, and an image processing device that performs overall control is used. I will explain.

(overall structure)
FIG. 1 is a schematic view showing the overall configuration of the image projection system 100 according to the present embodiment. The image projection system 100 shown in FIG. 1 includes an image processing device 110 that controls the entire system, one or more projectors 150, a camera 160, and an input device 170. In the embodiment described, the image projection system 100 synthesizes the projected images of the two upper and lower projectors 150a and 150b on the projection surface and projects the image on a region larger than that of a single projector, that is, a so-called large screen. It is said to be configured to support multi-projection.

However, the number of projectors 150 is not particularly limited. In another embodiment, a configuration corresponding to a normal projection using one projector 150 may be used. In still another embodiment, a configuration corresponding to so-called stack projection, in which projected images of a plurality of projectors 150 are superimposed on a projection surface and an image is projected onto an area of about a single projector, may be used.

The image processing device 110 is typically configured as a general purpose computer such as a personal computer or workstation. The image processing device 110 is not limited to a general-purpose computer, and may be mounted as a dedicated controller or incorporated in any of the projectors 150.

The projector 150 is a projection device that employs a liquid crystal system, a CRT (Cathode Ray Tube) system, a DLP (Digital Light Processing) system, an LCOS (Liquid Crystal On Silicon) system, or the like, respectively. The projector 150 may be an ultra-short throw projector. The camera 160 is an imaging device including an image sensor such as a CMOS (Complementary Metal Oxide Semiconductor) or a CCD (Charge Coupled Device) and an imaging optical system such as a lens for forming an image on a light receiving region of the image sensor. The camera 160 may be configured as a dedicated device such as a WEB camera, a digital still camera, a digital video camera, a spherical camera, or is incorporated in a general-purpose device such as a smartphone terminal or a tablet terminal. It may be configured as a device.

The input device 170 is an input device such as a mouse, keyboard, touch panel, and operation panel, and receives instructions from the user. The input device 170 is used when inputting the initial parameters of calibration and when making fine adjustments while observing the calibration results. The input device 170 may be configured as a device connected to the image processing device 110, the projector 150, or the camera 160, or may be configured as a device incorporated in these devices.

In the image projection system 100, a screen 102, which is a projection body that provides a projection surface, is installed. In the embodiment described, the screen 102 has a cylindrical (vertical cylindrical) shape having a bend in the horizontal direction, and a part or all of the inner wall of the cylindrical shape is a projection surface. The plurality of projectors 150 shown in FIG. 1 are installed so as to project a projected image on the screen 102 while shifting the position of the projection center in the vertical direction substantially perpendicular to the bending of the screen 102. In the embodiment described, the inner wall of the cylinder having a bend in the horizontal direction is the screen 102, but the screen 102 is not particularly limited, and any curved surface having a bend in one direction and a known geometric shape can be used. You may have.

The image processing device 110 generates a plurality of projected images to be projected on the plurality of projectors 150a and 150b, and outputs each of the projected images to each of the corresponding projectors 150. Each of the projectors 150 projects a projected image input from the image processing device 110 onto the screen 102. As shown in FIG. 1, a plurality of projected images 104a and 104b from each of the plurality of projectors 150a and 150b are projected on the screen 102. The plurality of projected images 104a and 104b are overlapped on the projection plane and combined into a single projected image 106.

The image projection system 100 projects a single projection image 106 using the plurality of projectors 150a and 150d as described above in the projection mode, but the calibration process is performed before the projection mode described above. The camera 160 shown in FIG. 1 is used in this calibration process. During the calibration mode, the image processing device 110 outputs a calibration image to each of the plurality of projectors 150, and projects a projection image for calibration on the screen 102. Then, the camera viewpoint and the field of view are set so that the projected image 104 on the screen 102 projected by the predetermined projector 150 is within the angle of view of the camera 160, and one or a plurality of imaging for calibration is performed. become.

The captured image captured by the camera 160 (hereinafter, the captured image in which the projected image for calibration is captured is referred to as the captured image for calibration) is a wireless LAN (Local Area Network), Bluetooth (registered trademark), or the like, respectively. It is transmitted to the image processing device 110 via a wireless connection or a wired connection such as a wired USB or a wired LAN. Alternatively, the calibration captured image captured by the camera 160 is read by the image processing device 110 via a removable medium such as an SD card or CompactFlash®.

The image processing device 110 calculates various correction coefficients for each of the projected images of the plurality of projectors 150a and 150b by using the plurality of input images for calibration. The image processing device 110 generates a corrected projected image for projection on each of the projectors 150a and 150b based on various calculated correction coefficients in the projection mode. Hereinafter, the cylinder correction process, the calculation process of various correction coefficients, and the correction process based on the correction coefficients will be described with reference to FIGS. 2 to 14.

(Functional configuration)
FIG. 2 is a functional block diagram of the image projection system 100 according to the present embodiment. The system 100 shown in FIG. 2 includes a plurality of functional blocks operating on the image processing device 110. The image processing device 110 includes a content storage unit 112, correction processing units 114a and 114b for each projector, projected image output units 116a and 116b for each projector, and switching units 122a and 122b for each projector. The image processing device 110 further includes a calibration image storage unit 124, a calibration image capture image input unit 126, a calibration parameter input unit 128, and a correction coefficient calculation unit 130.

The content storage unit 112 stores a content image to be projected as a single projected image 106. The content storage unit 112 is provided as a storage area for HDD (Hard Disk Drive), SSD (Solid State Drive), removable removable media, and the like. The content image to be projected is given as a display screen generated by an application such as a presentation or an operating system, a still image of a still image file, a frame at an arbitrary timing in a moving image file, or an externally input image. obtain. Hereinafter, for convenience of explanation, a case where a content image is given as a still image will be described as an example.

The correction processing units 114a and 114b and the projected image output units 116a and 116b are provided corresponding to the projectors 150a and 150b included in the system 100. Each of the correction processing units 114 reads a content image from the content storage unit 112, performs correction processing, and generates a projection image for the corresponding projector. Each of the projected image output units 116 includes a display output connected to the corresponding projector 150, and outputs an input image selected by the switching unit 122 to the connected projector as a video.

The switching units 122a and 122b switch the image flow according to the operation mode of the system 100. During the projection mode for projecting the content image, the switching unit 122 switches the input side to the output of the correction processing unit 114. Along with this, the projection image output unit 116 outputs a video of the processing result based on the content image by the corresponding correction processing unit 114, respectively. On the other hand, during the calibration mode, the switching unit 122 switches the input side to the output of the calibration image storage unit 124. Along with this, each of the projected image output units 116 outputs the calibration image read from the calibration image storage unit 124 as a video.

The calibration image storage unit 124 stores a calibration image to be projected from the projector 150 during the calibration mode. The calibration image storage unit 124 is provided as a storage area for HDDs, SSDs, and removable removable media. The proof image is not particularly limited, but is typically given as a pre-prepared still image.

Here, the calibration image includes both or one of a calibration pattern that defines the grid points of the projected image and an alignment pattern that defines the alignment points of the projected image. FIG. 3 is a diagram illustrating two types of calibration images used in the image projection system 100 according to the present embodiment. FIG. 3A illustrates a first calibration image 200 that includes both alignment patterns 202LT, 202RT, 202LB, 202RB and calibration pattern 206. FIG. 3B illustrates a second calibration image 210 containing only the alignment patterns 212LT, 212RT, 212LB, 212RB.

The calibration pattern 206 defines the coordinates on the projector memory, and is configured as a pattern formed by arranging arbitrary graphic elements 204 according to a predetermined rule. By imaging the calibration pattern 206 projected on the screen 102 and extracting the grid points thereof, trapezoidal distortion and local distortion of the projected image can be detected. The alignment patterns 202 and 212 define the reference position (alignment point) of the projected image between the images captured for calibration, and are configured by arranging a plurality of arbitrary graphic elements at predetermined positions. To. By accommodating the common alignment patterns 202 and 212 projected on the screen 102 and taking images a plurality of times, alignment can be performed between the plurality of captured images.

Although the specific pattern has been described with reference to FIG. 3, the calibration pattern 206 and the alignment patterns 202 and 212 are not particularly limited.

Now, refer to FIG. 2 again. In the calibration process according to the present embodiment, the calibration pattern for detecting the geometric distortion of the projected image of each projector 150 is imaged in one or a plurality of times, and the plurality of imaging results are integrated by the alignment pattern. ..

First, each calibration image is read out from the above-mentioned calibration image storage unit 124, and an appropriate calibration image is selected and output for each of the plurality of projectors 150a and 150b. Here, the calibration image is selected according to each stage of the calibration process so that the calibration result of each projector 150 can be obtained without excess or deficiency as a whole. At this time, the user uses the camera 160 to take a picture for each calibration projection scene so that the projected calibration projection image fits in the angle of view. The calibration captured image input unit 126 receives input of each captured image from the camera 160 via a wireless connection, a wired connection, or a removable medium, and prepares a plurality of calibration captured images.

The correction coefficient calculation unit 130 captures the projected calibration image, and based on one or more prepared calibration images, particularly cancels the distortion caused by the fluoroscopic conversion at the time of imaging. Calculate the correction factor for the projector.

Here, before explaining the specific calibration process, the case of projecting onto a projection body having a curved surface curved in one direction (for example, the horizontal direction) such as the inner wall of a cylinder, with reference to FIGS. 4 and 5 below. First, consider the discomfort given to the observer.

FIG. 4A is a diagram illustrating a perspective projection model. FIG. 4B is a diagram illustrating a parallel projection model. FIG. 5A is a diagram showing the relationship between the camera 160 and the screen 102 on the inner wall of the vertical cylinder. FIG. 5B is a diagram showing a screen 220 of the inner wall of the vertical cylinder imaged by the camera. FIG. 5C shows a projection target region 222 having a rectangular shape on the captured image coordinate system, which is set to project the content. FIG. 5D shows a corrected image 226 in which the corrected projected image corrected so as to fit in the projection target area is projected in parallel.

When projecting onto a screen 102 such as the inner wall of a cylinder, with the usual calibration method, as described above, the entire corrected image appears to bulge at both the upper and lower ends, giving the observer a sense of discomfort. It may end up. This is because when calibrating using a camera, the coordinates of the captured image are usually regarded as the same as the screen coordinates, and the corrected projected image is corrected to be rectangular in the captured image coordinate system. ..

Imaging with a camera is equivalent to converting a three-dimensional coordinate system (X, Y, Z) to a two-dimensional coordinate system (x, y), but perspective projection conversion (figure) as shown in the upper part of the following equation. 4 (A)) can be modeled. However, the perspective projection conversion is a normal projection (or parallel projection: FIG. 4B) that directly projects the (X, Y) coordinates of (X, Y, Z) in the three-dimensional coordinate system as shown in the lower row below. ) Gives different results.

When a rectangular flat screen is imaged from the front, the size of the perspective projection conversion changes in inverse proportion to the distance from the screen, but the perspective projection conversion is the same as in the case of normal projection. It will look like a rectangle. On the other hand, when a curved screen such as the inner wall of a vertical cylinder with depth is imaged from the front, it is converted into a rectangle in normal projection, but in perspective projection conversion, it is wound as shown in FIG. 5 (B). It will look like a shape. That is, as shown by the dotted line and the broken line in FIGS. 5A and 5B, the size differs depending on the depth of each position of the screen 102, so that the front part is large and the back part is large. Appears small, does not become rectangular, and appears distorted in the pincushion shape described above. That is, when the screen is a curved surface, the coordinates of the captured image are different from those represented by the xy coordinate system when the curved screen is viewed from the front (parallel projected x, y coordinate system). become.

Here, as shown in FIG. 5C, when the image is corrected so as to fit in the rectangular projection target area 222 in the captured image coordinate system distorted in the thread winding shape, as a result, the corrected image 224 is obtained. , It looks like a rectangle only when viewed from the camera's point of view, and it is not a correction like pasting a rectangular wallpaper on a curved surface. FIG. 5 (D) schematically shows an image obtained by parallel-projecting the corrected projected image, but in this way, the wallpaper bulging in a barrel shape is pasted on the curved surface. Become. For this reason, it looks like a rectangle when viewed from the camera's point of view, and there is no sense of discomfort when viewed from the camera's point of view, but compared to, for example, when viewed from the front of the upper and lower edges of the screen, a rectangular wallpaper is attached. Since it is projected in a barrel shape, it gives the observer a sense of discomfort.

In the above-mentioned normal correction, when there is only one viewer and the viewing position is fixed to the center of the front of the screen, there is no problem if the camera is placed at the viewer's position and calibrated. However, when a plurality of people view the projected projected image from various viewpoints, it is easy for the person to feel a sense of discomfort if the projected image appears to bulge into a barrel shape when observed toward the edge of the projected image.

Therefore, in the above cases, it is desirable to make a correction that makes the rectangular wallpaper look like it is attached to a curved surface. It is expected that human beings can imagine the original rectangular image as if it were affixed to a curved surface, such as a poster affixed to a curved surface or a label of a wine bottle, and recognize it without discomfort. There is from.

Therefore, in the present embodiment, the image processing device 110 extracts grid point data indicating distortion of the projected image for each of the projectors 150 from each of the prepared one or more calibration captured images. Further, the image processing device 110 accepts an input of geometric information of the screen 102 including a relative positional relationship with the camera 160 that has captured the image for calibration. Further, the image processing apparatus 110 is based on the extracted grid point data and the input geometric information, so that the distortion due to the fluoroscopic conversion at the time of imaging due to the shape of the screen 102 is offset by the one or more projectors 150. Calculate the correction factor for each. As a result, the projected image of the screen 102 having a bend in one direction is corrected so as not to give a sense of discomfort even when observed from a plurality of viewpoints.

With reference to FIG. 2 again, the functions of the calibration image input unit 126, the calibration parameter input unit 128, and the correction coefficient calculation unit 130 for realizing the above-mentioned correction will be described.

The calibration image capture image input unit 126 receives the input of the image captured by capturing the projection image projected in each of the above-mentioned calibration projection scenes, and prepares a plurality of calibration image images on the storage area.

The calibration parameter input unit 128 accepts input from the user of calibration parameters indicating the size of the screen 102 and the geometric relationship with the camera 160. The calibration parameter input unit 128 constitutes an information input means for receiving the input of the geometric information of the screen 102 in the present embodiment.

More specifically, the correction coefficient calculation unit 130 includes a feature point extraction integration unit 132, a cylindrical correction unit 134, a geometric correction coefficient calculation unit 136, and a blending coefficient calculation unit 138.

The feature point extraction integration unit 132 extracts each feature point from each of the prepared one or more calibration captured images. Here, it is assumed that each calibration image and the above-mentioned calibration projection scene are associated with each other and given to the calibration image input unit 126. In the embodiments described, the calibration image includes an array of calibration patterns for detecting distortion of the projected image of the projector that projects the calibration image. The feature point extraction integration unit 132 extracts a set of grid points defined by the arrangement of the calibration pattern and generates grid point data. The grid points extracted from each calibration image are extracted as coordinate positions on the coordinate system of the extraction source image, and are appropriately referred to as a single coordinate system (hereinafter referred to as an integrated coordinate system) by an alignment marker. .) Integrated on. The feature point extraction integration unit 132 constitutes the grid point extraction means and the grid point conversion means in the present embodiment.

The cylinder correction unit 134 corrects the integrated coordinate system (original coordinate system) of the above-mentioned calibration image to a coordinate system (corrected coordinate system) corresponding to the cylindrical shape of the screen 102. More specifically, the cylindrical correction unit 134 corrects the grid point data on the integrated coordinate system obtained by the feature point extraction integration unit 132 from the original coordinate system to the corrected coordinate system. The cylindrical correction unit 134 constitutes the correction means according to the present embodiment. The details of the cylinder correction will be described later.

The geometric correction coefficient calculation unit 136 calculates the geometric correction coefficient based on the grid point data for each projector 150 corrected by the cylindrical correction unit 134, and sets it for the correction processing units 114a and 114b. The geometric correction coefficient is a correction coefficient that incorporates geometric corrections such as alignment, scale adjustment, and distortion correction, and gives a projected image projected from each of the projectors 150.

Similarly, the blending coefficient calculation unit 138 calculates the correction coefficient for the blending of the projected image and sets it in the correction processing units 114a and 114b. The blending coefficient is a correction coefficient for adjusting the color and brightness when superimposing overlapping regions. The blending coefficient calculation unit 138, more specifically, for each of the plurality of projectors 150, detects an overlapping region between the projected image of the projector of interest and the projected image of each of the projectors adjacent to the projector of interest. The blending coefficient is calculated based on the detection result of the overlapping region. Due to the blending coefficient for each projector, the images are smoothly combined at the portion where the projected images of the plurality of projectors 150 overlap on the screen 102.

In this way, the correction coefficient calculation unit 130 cancels out the distortion due to the fluoroscopic conversion at the time of imaging due to the screen shape based on the extracted grid point data and the input geometric information. Calculate the correction factor for each. The correction coefficient calculation unit 130 constitutes the correction coefficient calculation means in the present embodiment.

The correction processing unit 114 generates a projection image for each projector from the content image based on various correction coefficients calculated by the correction coefficient calculation unit 130, respectively. The projected image output unit 116 outputs the corrected plurality of corrected images so as to be projected by the corresponding plurality of projectors 150. The projected image output unit 116 constitutes the image output means in the present embodiment.

A plurality of corrected images corrected based on the correction coefficients calculated as described above are projected by the corresponding plurality of projectors 150. As a result, the curvature distortion caused by the fluoroscopic transformation at the time of imaging of a deep side (one direction having a bend) when projected onto the cylindrical screen 102 is corrected.

In a preferred embodiment, in order to obtain a better result when the curvature distortion remains despite the initial correction by the input geometric information, the calibration parameter input unit 128 further makes a minute of the geometric information. The input of the adjustment amount can be accepted. In this case, the correction coefficient calculation unit 130 recalculates the correction coefficient in consideration of the input fine adjustment amount. While observing the projection results projected by the plurality of projectors 150 based on the recalculated correction correction coefficient, the user performs fine adjustment amount input and projection based on the correction correction coefficient until it is determined that the curvature distortion is sufficiently corrected. Can be repeated. This makes it possible to correct the curvature distortion caused by the fluoroscopic transformation at the time of imaging with higher accuracy and to adapt the projection region to the screen 102.

In the embodiment shown in FIG. 2, each functional unit 112 to 138 has been described as being realized on a single image processing device 110, but the embodiment of the image projection system 100 is the one shown in FIG. It is not limited. In another embodiment, each of the correction processing units 114 may be realized on each of the projectors 150 in order to reduce the load concentrated on the image processing apparatus due to the increase in the number of units. In still other embodiments, the functional units 112 to 138 may be distributed and mounted on a plurality of image processing devices, or all the functional units 112 to 138 may be mounted on any of the projectors 150. , The image processing apparatus 110 may be configured as a single apparatus having the functions of the image processing apparatus 110 and the functions of a plurality of projectors. Further, in another embodiment, the function of the correction coefficient calculation unit 130 may be implemented as a server that is provided as a service via a network.

(Overall processing flow)
Hereinafter, the overall flow of the calibration process according to the present embodiment will be described with reference to the flowchart shown in FIG. 6 and the calibration projection scene shown in FIG. 7. FIG. 6 is a flowchart showing the overall processing flow of the calibration process according to the present embodiment. The process shown in FIG. 6 is started from step S100 in response to an instruction from the user to start the calibration process.

In step S101, the image processing device 110 accepts input from the user of calibration parameters indicating the size of the screen 102 and the geometric relationship with the camera 160. This input is received by the input device 170 via the condition input screen by displaying the condition input screen on the display connected to the image processing device 110. Here, in the embodiment described, the projected surface is at least a part of a substantially cylindrical surface, and the calibration parameters constituting the above-mentioned geometric information are defined on the radius of the cylindrical shape of the screen 102, the substantially cylindrical surface. Size information such as a central angle that defines the range of the projected surface to be projected (hereinafter, may be referred to as a cursor area), distance information of the screen 102 from the camera 160, and a deviation of the center of the screen 102 from the camera 160. May contain information.

In step S1102, the image processing device 110 sequentially projects each of the generated calibration projection scenes using the plurality of projectors 150a and 150b, and each image is taken by the camera 160 corresponding to each calibration projection scene. Acquire an image for calibration.

FIG. 7 is a diagram illustrating a calibration projection scene projected by the image processing device 110 according to the present embodiment from the upper and lower two-stage projectors 150a and 150b. FIG. 7 shows two calibration projection scenes in which the calibration images shown in FIGS. 3 (A) and 3 (B) are alternately projected from each projector.

In the first calibration projection scene shown in FIG. 7A, the image processing device 110 first projects the first calibration image shown in FIG. 3A from the lower projector 150a, and then projects the first calibration image shown in FIG. 3A, and then the upper projector 150b. The second calibration image shown in FIG. 3 (B) is projected from. In the second scene of the second imaging shown in FIG. 7B, on the contrary, the second calibration image shown in FIG. 3B and the second calibration image shown in FIG. 3A are shown from the lower projector 150a and the upper projector 150b, respectively. The first calibration image is projected.

The user fixes the camera 160 with a tripod or the like so that the entire projected image of all the connected projectors 150a and 150b fits within the angle of view, or holds the camera 160 and performs a plurality of shootings at each of the above-mentioned stages. Each of the calibration captured images corresponding to each of the calibration projection scenes from the camera 160 is acquired by the image processing device 110 collectively or sequentially, and processing proceeds to step S103.

In the embodiment to be described, it is assumed that the entire projected images of the plurality of projectors 150a and 150b are imaged so as to fit in the viewing angle of the camera 160, and the positions of the cameras 160 are substantially the same. The camera 160 may or may not be physically fixed by a tripod or may not be hand-held. Even if the camera 160 is not fixed, if the entire projected image of the plurality of projectors 150a and 150b is imaged so as to fit in the viewing angle of the camera 160, the projection conversion is performed so that the above-mentioned alignment patterns 202 and 212 match. By doing so, the imaging results can be integrated. Hereinafter, the case where the camera is fixed and the image is taken will be described as an example.

In step S103, the image processing device 110 executes a feature point extraction integrated process for extracting feature points from each of the acquired one or more calibration captured images. In the feature point extraction integrated processing, the coordinate position of the reference point defined by the set of grid points of each projector and the alignment marker is extracted in the integrated coordinate system.

More specifically, the image processing device 110 first detects the calibration pattern (circular shape) of the projected image of each projector 150 in each captured image, and sets the coordinates of the center of gravity in the coordinate system of the captured image to the grid points. Extract as coordinates. The coordinates of the center of gravity of the circular shape can be calculated, for example, by binarizing the image, cutting out a mass of white pixels by pattern matching or the like, and obtaining the coordinates of the center of gravity. The same applies to the alignment marker. The image processing device 110 detects the coordinates of the center of gravity of the alignment marker in the coordinate system of each captured image from the plurality of captured images, and extracts the coordinates of the center of gravity as the coordinates of the reference point. To do.

Hereinafter, represent lower and upper 3 projector 150a, the coordinates of the lattice points 150b in _L 1, _{L 2,} representing the coordinates of the reference point of the alignment marker at _M 1, _{M 2.} In the embodiment described, since the camera 160 is fixedly imaged, the coordinate system of the captured images over a plurality of times does not substantially deviate. Therefore, the obtained grid point coordinates L ₁ , L ₂ and the reference point coordinates M _{1 are obtained.} , M ₂ are the coordinates of a single coordinate system (integrated coordinate system).

In step S104, the details will be described later, but based on the calibration parameters input in step S101, the calculated grid point coordinates L ₁ and L ₂ are subjected to cylindrical correction. In step S105, the corrected grid point coordinate L ₁ ', L _2' based on, to calculate the geometric correction factor for each projector. In step S106, the image processing device 110 calculates the blending coefficient of each projector. In step S107, the image processing apparatus 110 sets the geometric correction coefficient and the blending coefficient for each projector calculated in steps S105 and S106 for each correction processing unit 114.

In step S108, the image processing device 110 reads out the content image to be projected, and in step S109, the correction processing unit 114 for each projector executes correction processing on the content image. Upon starting the correction process, the switching unit 122 switches the input of the projected image output unit 116 to the output of the correction processing unit 114, and shifts to the projection mode. In step S110, the image processing device 110 outputs the corrected projected image for each projector from the projected image output unit 116 for each projector. As a result, the content image is projected on the screen 102 as a whole.

In the initial stage of the projection mode, there is a user interface for adjusting the four corners of the area where the content image is projected to match the screen 102 and adjusting the distortion of the projected image by increasing or decreasing the input calibration parameters. Is displayed. These user interfaces may be displayed on a part of the screen projected by the projector 150 or on a separately provided display screen.

Using this user interface, the user visually fine-tunes the four corners of the projection target area on which the content image is projected and the calibration parameters. Then, when it is determined that the content image fits perfectly on the screen 102 and the distortion is corrected, the user gives an instruction to end the adjustment. Details of the adjustment work will be described later with reference to FIGS. 9 and 10.

In step S111, the image processing device 110 determines whether or not the end instruction has been received from the user. If it is determined in step S111 that the adjustment end instruction is not accepted (NO), the process proceeds to step S112. In step S112, the image processing apparatus 110 updates the position coordinates and calibration parameters of the reference points that define the four corners of the projection target area based on the adjustment amount input via the user interface, and loops the process to step S104. Let me. As a result, the feature points are corrected and the geometric correction coefficient and blending coefficient of each projector are recalculated while the adjustment result is reflected.

On the other hand, if it is determined in step S111 that the adjustment end instruction has been accepted (YES), the process is branched to step S113, the adjustment is completed, and the normal projection mode is entered.

(Outline of cylinder correction)
Hereinafter, the outline of the process of correcting the expansion and contraction of the scale in the vertical direction and the horizontal direction due to the fluoroscopic conversion at the time of imaging will be described. Here, as shown in FIG. 8A, a case where two projectors 150a and 150b installed on the top and bottom perform vertical connection projection on the vertically cylindrical screen 102 will be described as an example.

In FIG. 8A, the central axis of the cylindrical screen is the y-axis, the z-y plane is taken facing the cylindrical screen surface, and the center position of the cylindrical screen is the origin (x, y) = (0,0). To do. FIG. 8B is a view of the cylindrical screen shown in FIG. 8A as viewed from directly above. FIG. 8C shows a case where a curved axis along the horizontal arc of the cylindrical surface is the x'axis, and equidistant scales on the x'axis on the cylindrical surface are imaged from infinity, and It schematically shows how each changes when a close-up image is taken.

When the image is taken from infinity, it is projected in parallel to the virtual projection plane as shown in the center of FIG. 8C, and this is represented by the x-axis. In the case of close-up imaging, as shown on the right side of FIG. 8C, perspective transformation projection is performed on the imaging surface, and this is represented by the u-axis.

If the central angle of the cylindrical surface is sufficiently small, the scales of the x-axis projected parallel to the virtual projective plane and the x'axis on the cylindrical surface do not expand or contract significantly. However, in the case of close-up imaging, as shown in FIG. 8C, the smaller the imaging distance, the greater the expansion and contraction of the scale on the u-axis on the imaging surface.

If the correspondence between the curved x'axis on the cylindrical surface and the u-axis on the imaging surface in close-up imaging can be analytically solved, the horizontal coordinates on the captured image can be changed to the horizontal coordinates on the cylindrical surface (cylindrical surface). It can be converted to the coordinate system of the captured image corresponding to the horizontal coordinates of the developed plane). Details of the process of converting the grid point coordinates of the close-up image to the plane coordinates of the expanded cylindrical surface, including the analysis of this correspondence, will be described later with reference to FIGS. 11 to 14.

If the initial values of the size information of the cylindrical surface and the position information relative to the cylindrical surface of the camera 160 cannot sufficiently correct the curvature distortion, the remaining curvature distortion of the upper and lower sides is further corrected by user input. Can be fine-tuned. In this case, the user assumes that the camera 160 faces the screen 102, and the deviation of the camera 160 from the center of the screen to the left, right, up and down (xoffset, yoffset) and the imaging distance (distance) with respect to the cylindrical surface of the camera 160. The correction result can be projected while making fine adjustments, and the curvature distortion can be visually adjusted to the screen.

(Work flow from initial correction to adjustment)
Hereinafter, the adjustment work by the user in the calibration process according to the present embodiment will be described with reference to the flowchart shown in FIG. 9 and the table shown in FIG. FIG. 9 is a flowchart showing a flow of adjustment work in the calibration process according to the present embodiment. FIG. 10 is a table for explaining how to adjust the calibration parameters in the adjustment work according to the present embodiment.

The process shown in FIG. 9 is started from step S200. In step S201, the image processing device 110 outputs an initial corrected image corrected based on the initial value of the calibration parameter. Here, it is assumed that the approximate design value, actual measurement value, or default value is input for the calibration parameter whose details will be described later.

In step S202, the image processing device 110 receives the fine adjustment amount of the reference points at the four corners that define the outer shape of the projection target area. Here, the user fine-tunes the reference points of the four corners so that the four corners of the corrected image match the target area on the screen 102 while observing the projected image of the initial corrected image and the corrected image after the fine adjustment. Enter the amount.

In step S203, the image processing device 110 accepts fine adjustment of the distance parameter (distance), which is one of the calibration parameters. Here, the user inputs the fine adjustment amount of the distance parameter (distance) so as to reduce the distortion of the barrel shape on the upper and lower sides while observing the projected image of the corrected image after the fine adjustment.

Here, referring to the upper two rows shown in FIG. 10, how to adjust the distance parameter is explained. As shown in the first row of the table of FIG. 10, when the tendency of the edge to bulge is observed, it is considered that the position of the camera is set farther than the correct position. Therefore, by reducing the distance parameter, it is possible to reduce the bulge on both the upper side and the lower side than the current state. On the other hand, as shown in the second row of the table in FIG. 10, when the tendency of the edge to be dented is observed, it is considered that it corresponds to the case where the position of the camera is set closer than the correct position. .. Therefore, by increasing the distance parameter, both the upper side and the lower side can be inflated from the current state. It should be noted that the correct position here means the value of the distance parameter in which the correction result is corrected in a rectangular pasted shape, and does not necessarily match the physical camera position.

In step S204, the image processing device 110 inquires whether or not the distortion is out of balance vertically and horizontally, and accepts input from the user. If the camera 160 is installed in the center of the screen 102 facing the screen 102, the balance will not be lost and distortion will occur symmetrically in the vertical and horizontal directions. When the response from the user that the balance is lost is received in step S204 (YES), the process is branched to step S205.

In step S205, the image processing apparatus 110 accepts fine adjustment of the left-right and up-down offset parameters that are part of the calibration parameters. The left-right and up-down offset parameters (xoffset, yoffset) are for adjusting the up-down balance and the left-right balance of the curved bulges on the upper and lower sides. Here, the user inputs the fine adjustment amount of the offset parameter so as to reduce the imbalance between the vertical and horizontal directions while observing the projected image of the corrected image after the fine adjustment.

Here, referring to the middle two rows shown in FIG. 10, how to adjust the left-right offset parameter (xoffset) is explained. With reference to the lower two rows shown in FIG. 10, how to adjust the vertical offset parameter (yoffset) is explained.

As shown in the third row of the table in FIG. 10, when a bulge biased to the left side of the edge is observed, it is considered that the position of the camera is shifted to the left from the center of the cursor area. Therefore, by setting the left-right offset (xoffset) to be negative, the left side can be dented from the current state and the right side can be bulged more than the current state. On the other hand, as shown in the fourth row of the table in FIG. 10, when a bulge biased to the right side of the edge is observed, it is considered that the position of the camera is shifted to the right from the center of the cursor area. Be done. Therefore, by setting the left-right offset (xoffset) to be positive, the left side can be inflated from the current state and the right side can be dented from the current state.

Further, as shown in the fifth row of the table in FIG. 10, when the tendency of the lower side to dent below the upper side is observed, it is considered that the position of the camera is shifted above the center of the cursor area. Be done. Therefore, by setting the vertical offset (yoffset) to be negative, the upper side can be inflated from the current state and the lower side can be dented from the current state. On the other hand, as shown in the sixth row of the table in FIG. 10, when the tendency of the upper side to rise above the lower side is observed, it is considered that the camera position is deviated below the center of the cursor area. Be done. Therefore, by setting the vertical offset (yoffset) to be positive, the upper side can be dented from the current state and the lower side can be inflated from the current state.

In this way, the user adjusts the amount of barrel distortion by the distance parameter (distance), and then fine-tunes it by the left-right offset parameter (xoffset) and the vertical offset parameter (yoffset). Then, the user corrects the vertical and horizontal balances of the barrel-shaped distortions on the upper and lower sides so as to be in a horizontal and vertical contrast state while looking at the projected corrected image after the fine adjustment. As a result, it is possible to approach the state in which the rectangle is attached as if the rectangle was stretched as it is on the cylinder.

When step S205 is completed, the main adjustment work is completed in step S206. When the response from the user that the balance is not lost is received in step S204 (NO), the process directly proceeds to step S206, and the main adjustment work is completed.

(Details of cylinder correction processing)
Hereinafter, with reference to FIGS. 11 to 14, a more detailed description will be made of the cylinder correction process for converting the grid point coordinates taken in close proximity to the plane coordinates in which the cylindrical surface is expanded. First, the correction in the x-axis direction will be described below with reference to FIGS. 11 and 12.

Hereinafter, with reference to FIGS. 11A and 11B, a method of giving initial values of calibration parameters, and derivation parameters and derivation functions derived from the calibration parameters for use in internal calculations will be described. .. 11 (A) and 11 (B) show the xz plane when the cylindrical screen is viewed from directly above. The range (cursor area) on which the projected image is projected on the cylindrical screen is determined by the left and right cursors.

FIG. 11A is a diagram illustrating calibration parameters given by the cylindrical surface model. As shown in FIG. 11A, two methods can be considered as a method of giving initial values of calibration parameters. In the first method, the cursor central angle θ and the radius of curvature R that define the range (cursor area) of the projected surface on the cylindrical surface can be given as the size information. In the second method, the distance between the left and right cursors A and the bending depth E at the midpoint between the left and right cursors, which defines the range (cursor area) of the projected surface on the cylindrical surface, can be given as the size information.

Here, considering the isosceles triangles ΔOLM and ΔMLL'using the point L on the cylindrical screen, the midpoint K of the cursor area, and the point symmetry L'of the point L with respect to the midpoint K, these two triangles. Have a similar relationship, the following relationship

Therefore, R can be expressed using E and A by the following formula. Therefore, the above-mentioned A and E and the above-mentioned θ and R can be exchanged with each other by using whichever is easier to measure.

FIG. 11B is a diagram illustrating a derivation parameter and a derivation function derived from the calibration parameters given by the cylindrical surface model. Here, in order to facilitate the calculation described later, the derivation parameters (a, s, e) shown below can be defined.

Further, the correction depth d (x) at a certain x can be defined as follows by applying the three-square theorem to the triangle shown by the dotted line in FIG. 11 (B). In the following equation, x is a value on the x-axis passing through the center of curvature O.

As shown above, even if θ is not given, it is not necessary to derive θ from A and E, and if the calibration parameters are given by the second method above, R is calculated in advance from A and E. It means that it should be done.

Subsequently, the relationship between the coordinate value on the x'axis and the coordinate value on the x-axis will be described with reference to FIG. 12 (A). Here, considering the x'axis along the cylindrical screen, the coordinate value of the point P on the cylinder on the x-axis and the coordinate value on the x'-axis are set, where Q is the point on the cylinder when x = 0. When x and x'are used respectively, the following formula

Is obtained, and φ can be eliminated from the above equation and expressed by the following equation (1). Up to this point, the items are determined from the cylindrical shape of the screen, which does not depend on the camera 160.

Since it is very complicated to proceed with all the progress of the calculation by the formula of x', the above formula (1) when it becomes necessary to obtain x'while proceeding with the explanation by the formula of x first. ) Shall be converted from x to x'.

Next, the relationship between the x'axis along the cylinder and the u-axis of the captured image will be described with reference to FIG. 12 (B). Here, a camera lens having a focal length F is placed at point J, which is separated from the cursor plane (the plane formed by the left and right cursors) by D in the depth direction and shifted horizontally by x0 from the center of curvature, and an image is formed on the imaging plane. Consider the case. Here, when the horizontal axis on the captured image is the u-axis and the point I on the cylinder whose horizontal coordinate value is x corresponds to the coordinate u (L point) on the captured image, the two triangles ΔIJK and Δ Since JLM has a similar relationship, the following equation (2) is derived.

Here, when examining the possible range of the coordinate value u, the coordinate value u [a] on the u axis corresponding to the right cursor position x = a has a depth of zero at the right cursor position (d (a) = 0). Considering that, it is expressed by the following equation.

By eliminating the focal length F from the above equation and the above equation (2), the following equation can be obtained.

Here, since u is proportional to u [a], the scale of the u axis has a constant multiple of degrees of freedom, so the equation is simplified by setting u [a] = a-x0 for convenience. It can be done by the following formula. At this time, the coordinate value on the u axis corresponding to the left cursor position is u [a] =-(a-x0).

By using the above equations (1) and (3), the coordinate value u on the horizontal axis on the captured image, the coordinate value x on the x-axis, and the coordinate value x on the cylinder (x'axis) are passed. Can be converted to'.

Hereinafter, the correction in the y-axis direction will be described with reference to FIG. 13 (A). FIG. 13A describes the distortion in the vertical direction (Y-axis direction) due to the perspective transformation in close-up imaging due to the depth difference (Z-axis direction) of the cylindrical surface when the cylindrical surface of the screen is viewed from the side. As described above, the correction depth d (x) is the bending depth at a certain x, and here, the description will be continued using the horizontal coordinates x instead of x'on the axis along the cylinder. As described above, D represents the distance from the camera lens to the cursor plane. As shown in FIG. 13A, the height of each image in the y-axis direction on the imaging surface can be obtained from the depth difference between the two points on the upper and lower sides of the same height of the cylindrical surface and the imaging distance D. Become.

Due to the curved depth, as shown in FIG. 13 (A), the partial image of the point P at the height h on the physical cylindrical screen is recorded as the pixel at the vertical position y on the captured image. .. If this is subjected to distortion correction on the premise of a conventional flat screen, the correction works so that the same image appears at the point P'. That is, a convex effect is applied such that the pixel at the vertical position y moves to y', which causes edge bending when the cylindrical screen is projected. In order to cancel out this extra convex effect, the position y of the pixel corresponding to P on the captured image may be corrected as if it were in y'from the beginning.

Focusing on similarity triangles using the focal length F of the camera lens, the following relational expression is derived.

Then, by rearranging the above equations, the following equations can be obtained.

According to the above, when y = 0, y = y'= 0 regardless of x. When the above equation is modified so that y = y'= y0 regardless of x when y = y0, the following equation (4) is obtained.

From the above equation, if the coordinates on the x'or x-axis and the size information of the cylindrical shape are known, the depth difference at the x'or x-coordinate can be obtained, so that the distortion in the y-axis direction can be corrected.

By the correction in the horizontal direction and the vertical direction described above with reference to FIGS. 11 to 13 (A), the grid point coordinates captured in close proximity can be converted into the plane coordinates in which the cylindrical surface is expanded.

Hereinafter, the flow of the cylinder correction process will be described in more detail with reference to FIG. FIG. 14 is a flowchart showing a cylinder correction process according to the present embodiment. The process shown in FIG. 14 is called by the process of step S104 shown in FIG. 6 and is started from step S300.

In step S301, the image processing device 110 acquires the initial value of the calibration parameter by the calibration parameter input unit 128, and acquires the coordinates of the four cursor points from the reference point coordinates M ₁ and M ₂ . In the arrangement shown in FIG. 8 (A), bract under two of the reference point M ₁ of the four corners of the lower projector 150a, 2 bract is obtained on one of the reference point M ₂ of the four corners of the upper projector 150. The initial values of the calibration parameters are the set of R, θ, D (Distance), x0 (xoffset) and y0 (yoffset), or A, E, D (Distance), x0 (xoffset) and y0 (yoffset). Pair is obtained.

In step S301, the image processing apparatus 110 calculates the derivation parameters (a, s, e) and the correction depth function (d (x)) based on the input calibration parameters.

In step S303, the image processing device 110 performs a planar projective transformation such that the quadrilateral formed by the cursor becomes a rectangular shape (standardized shape) based on a plurality of reference points of the cursor that define the outer shape of the entire projection area. Find H. At that time, as shown in FIG. 13B, the upper left and lower right coordinates of the cursor are set to be (-(a + x0), -1) and (a-x0,1), and normalization is performed. Can be done. By normalizing the coordinate values before the correction, it is possible to prevent the four corner positions of the projection target area from fluctuating after the correction when performing this correction after adjusting the four corner positions of the projection target area.

In step S304, the image processing device 110 is obtained above so that the quadrilateral formed by the alignment markers at the four corners becomes a rectangle at an assumed position according to the imaging position (x-axis direction) with respect to the grid point coordinates. The plane projection transformation H is applied to normalize the coordinate values.

In step S305, the image processing device 110 is based on the correspondence represented by the above equation (3), where u is obtained by multiplying the horizontal coordinate value X of the grid point coordinates by the plane projection transformation H. , The value x of the coordinates on the virtual projective plane is obtained as an unknown number from the value u of the original coordinate system. Here, the correspondence relationship represented by the above equation (3) is a correspondence relationship between the L point on the original coordinates based on the coordinate value x on the virtual projective plane and the I point on the coordinates on the projected plane. .. At this time, a corrected depth d (x) is also obtained as a by-product.

Here, a method of obtaining the unknown number x from the above equation (3) will be described. In the above formula (3), corrected depth d (x), since it contains x ² term in the square root is complex, solving for x above equation (3) directly is considered difficult. Therefore, first, as an intermediate step, d is obtained after transforming the above equation (3) into an "equation for unknown number d".

If the function d (x) is described as d for convenience of explanation, the following equation can be obtained from the modification of (3) above.

Further, by squaring both sides, the following equation (5) can be obtained.

On the other hand, from the definition of the correction depth d (x), the following equation (6) can be obtained.

On the other hand, the above equation (6) is substituted into the right side of the equation (5), further, to organize a polynomial d and ^{(R 2} -S ² = use the relation ^{a 2.),} The following equation (7 ) Is obtained.

D can be obtained from the formula of the solution of the quadratic equation of the above equation (7), and if d is obtained, x can be obtained from (6). Regarding the above quadratic equation and (6), details such as the existence of a real number solution, which of the two real number solutions should be adopted, and which of the positive and negative should be adopted as x when the square root of (6) is taken. The discussion will be described later.

Again, referring to FIG. 14, in step S306, the image processing apparatus 110 is based on the above equation (1), which defines the geometric relationship between the virtual projective plane and the projected plane according to the radius of curvature. The coordinate value x'on the projective plane is obtained from the coordinate value x on the virtual projective plane obtained in.

In step S307, the image processing apparatus 110 linearly transforms the obtained x'to obtain the'corrected coordinate system value u'from the coordinate value x'on the projected surface as the correction result for u. .. Here, the linear transformation is expressed by the following equation using the captured image coordinate values ul and ur at the left and right cursor positions.

The above linear transformation is for satisfying the condition that the captured image coordinate values ul and ur at the right and left cursor positions do not change before and after the correction. Note that e is the length of the arc between the left and right cursors. Substituting ul =-(a + x0) and ur = (a-x0) into the above equation gives the following equation (8).

In step S308, the image processing device 110 cancels the optical effect due to bending based on the above equation (4), where y is obtained by applying the above-mentioned planar projective transformation to the vertical coordinates Y of the grid points. As described above, the value y'to the corrected coordinate system is obtained from the value y of the original coordinate system. As a method of coordinate conversion, the captured image itself can be deformed, but for the purpose of this embodiment, it is not necessary to deform the captured image itself, and the conversion from y to y'with respect to the grid point coordinates. It is the same even if you do.

In step S309, the image processing apparatus 110 multiplies the obtained (u', y') by the inverse transformation H- ¹ of the above-mentioned planar projective transformation, and in step S310, the coordinate values (X', Y') are used. Rewrite the coordinates of the grid points.

In this way, the cylinder correction unit 134 performs the projective transformation H on the grid point data (X, Y) extracted by the feature point extraction integration unit 132. Then, the cylinder correction unit 134 corrects the grid point data after the projective transformation from the original coordinate system (u, y) to the corrected coordinate system (u', y'). Further, the cylindrical correction unit 134 further applies the inverse transformation H- ¹ of the projective transformation to the corrected grid point data to obtain the corrected grid point data (X', Y'). Then, the correction coefficient calculation unit 130 calculates the correction coefficient based on the grid point data (X', Y') after the projection conversion, the correction, and the inverse conversion.

(Detailed consideration of u → x conversion in cylinder correction)
In the above formula (7), the following cases will be examined separately.
[1] When u = a-x0 or u =-(a + x0):
In this case, since the coordinates correspond to the left and right cursor positions, the depth d = 0 should be, and in fact d = 0 is the solution of the quadratic equation (7). Therefore, in the case of u = a-x0 or u =-(a + x0), d = 0 may be set by a fixed decision.

[2]-(a + x0) <u <a-x0:
Since it is the coordinates between the left and right cursors, it should have a real solution such that d> 0. Since u + x0 + a> 0 and u + x0-a <0 actually, C <0 and always A> 0, the discriminant B ^2- AC> 0 has a real number solution. If C <0 due to the relationship between the solution and the coefficient, the two solutions have different signs. Therefore, in this case, the larger solution may be adopted, that is, it can be calculated by the following formula.

[3] When u <-(a + x0) or u> a-x0:
This case also needs to be considered because the grid points can take up to the u coordinate outside the left and right cursors.

[3-a] Discriminant B ^2- AC <0:
Here, considering the u → x conversion from a slightly different point of view, the position u on the captured image, the straight line passing through the center of the lens (= the straight line passing through L and J in FIG. 12B) and the arc are extended to 360 degrees. It comes down to the problem of finding the intersection with the whole circle. Here, the discriminant <0 means that this straight line and the whole circle do not have an intersection. It can realistically occur when the imaging distance exceeds the curvature diameter of 2R. In this case, d = impossible.

From here, it may be limited to the case where the discriminant ≧ 0 and the real number solution is obtained. Considering graphically as the above "problem of finding the intersection of a straight line and a whole circle", the intersection in this case is below the straight line connecting the left and right cursors, that is, the correction depth d (x) is negative. , The two solutions of the above (7) quadratic equation are both negative values. Then, it can be seen that the solution that is graphically on the upper side and that is closer to 0 should be adopted. Here, it is assumed that d is a value adopted in this way.

[3-B] When d> -s:
Corresponds to the case where the intersection corresponding to d is included in the "upper semicircle". In this case, as a natural extension of the above [2], there is no problem in processing both the X direction and the Y direction in the same manner as in [2]. In the X direction, a transformation that "approaches" toward the immovable cursor position is applied so as to maintain the scale. On the other hand, in the Y direction, since the above equation (4) is negative, a conversion such as "approaching" toward y0 is applied, and the effect of straightening the bend can be obtained. In terms of program implementation, it is preferable that the conditions are stricter than d <-s.

[3-c] When d ≤ -s:
Corresponds to the case where the intersection corresponding to d is included in the "lower semicircle". In this case, if the logic of the above [2] is applied as it is, the x coordinate of the intersection becomes | x | <R, and further | x | <a, which may overlap with the result of the case of the above [2]. There is. It is thought that overlapping can be avoided by expressing with x'coordinates along the cylinder, but since it is unavoidable to do too complicated things outside the cursor area where the content is not projected, it is easy to convert both X and Y. Can be dealt with. Since only the immovable cursor position and the conversion toward y0 are not performed, the magnitude relation of the coordinate values in all the columns and rows of the lattice points is not broken, and it is considered that there is no inconvenience in the subsequent processing.

There is also the idea that the case of [3-B] is not converted, but it is possible that the grid points on which the pixels at the edge of the projected content depend as the bilinear interpolation source are those of the case of [3-B]. Therefore, in order to maintain smoothness, it is better to perform conversion in [3-B].

From the above, the correction depth d (non-negative value on and inside the left and right cursor, negative value on the outside) or d = impossible can be obtained. If d = impossible, x cannot be obtained, so X and Y are treated as not being converted. If it is not impossible, x can be obtained from the above equation (3) by arranging as follows.

(Hardware configuration)
Hereinafter, the hardware configuration of the image processing apparatus 110 according to the above-described embodiment will be described with reference to FIG. The image processing device 110 is typically configured as a general-purpose computer device. FIG. 15 is a diagram showing a hardware configuration of a general-purpose computer device according to the present embodiment.

The general-purpose computer device 110 shown in FIG. 24 includes a CPU 12, a north bridge 14 for connecting the CPU 12 and a memory, and a south bridge 16. The south bridge 16 is connected to the north bridge 14 via a dedicated bus or a PCI bus, and is responsible for connecting to an I / O such as a PCI bus or USB.

A RAM 18 that provides a work area for the CPU 12 and a graphic board 20 that outputs a video signal are connected to the north bridge 14. The graphic board 20 has a video output interface such as analog RGB, HDMI (High-Definition Multimedia Interface; HDMI and High-Definition Multimedia Interface are registered trademarks or trademarks), DVI (Digital Visual Interface), DisplayPort (registered trademark), and the like. It is connected to the display 50 and the projector 150 via the above.

A PCI (Peripheral Component Interconnect) 22, a LAN port 24, an IEEE 1394, a USB port 28, an auxiliary storage device 30, an audio input / output 32, and a serial port 34 are connected to the south bridge 16. The auxiliary storage device 30 is an HDD, SSD, or the like, and stores an OS for controlling a computer device, a program for realizing the above-mentioned functional unit, various system information, and various setting information. The LAN port 24 is an interface device that connects the general-purpose computer device 110 to the network by wire and wirelessly.

An input device 170 such as a keyboard 52 and a mouse 54 may be connected to the USB port 28, and a user interface for receiving input of various instructions from the operator can be provided. The general-purpose computer device 110 according to the present embodiment reads a program from the auxiliary storage device 30 and deploys it in the work space provided by the RAM 18 to realize each of the above-mentioned functional units and each process under the control of the CPU 12. Although the projector 150 and the camera 160 will not be described in particular, they are similarly provided with hardware such as a CPU and RAM, and hardware suitable for a specific application.

As described above, according to the embodiment of the present invention, in the projection system for projecting an image on the projected surface by one or more projection means, it is easy for the projected surface having a bend in one direction. It is possible to provide a projection system capable of correcting a projected image by various operations, an image processing device constituting the projection system, and a program for realizing the image processing device.

The calibration process according to the above-described embodiment is particularly effective in the case of multi-projection having a cylindrical projected surface and a small number of connections so that the screens of all projectors fit in the angle of view of the camera. Compared to the case of divided imaging, it is not necessary to increase the number of images to be imaged and the imaging position, it is possible to appropriately correct the distortion of the barrel shape at both ends of the cylindrical screen, and the content is mapped to the plane where the cylindrical surface is developed. Such geometric correction can be performed.

In particular, in the case of a vertical cylindrical screen, it is necessary to take an image at the upper end position, and in the case of a cylindrical screen having a height of about 3 m, it is necessary to take an image at a height that cannot be reached by a normal tripod or the like. These are not required in the calibration process by. Further, according to the calibration process according to the above-described embodiment, the geometry can be suitably corrected for the entire projector projection area.

Furthermore, by accepting the fine adjustment amount, even if the barrel distortion cannot be completely corrected after the correction is made, the user can visually check the correction result through the user interface so that the remaining bending distortion is straightened. However, it is configured so that it can be fine-tuned. Further, preferably, by performing the plane projection transformation before and after the above-mentioned correction, once the four corner positions of the projection area are finely adjusted, the fine adjustment for reducing the distortion of the barrel shape is performed, and the projection area is preferably adjusted. The positions of the four corners can be fixed.

The above functional unit can be realized by a computer-executable program written in a legacy programming language such as an assembler, C, C ++, C #, Java (registered trademark) or an object-oriented programming language, and is ROM, EEPROM, EPROM. , Flash memory, flexible disk, CD-ROM, CD-RW, DVD-ROM, DVD-RAM, DVD-RW, Blu-ray disk, SD card, MO, etc. Stored in a device-readable recording medium, or through a telecommunication line Can be distributed.

Although the embodiments of the present invention have been described so far, the embodiments of the present invention are not limited to the above-described embodiments, and those skilled in the art may think of other embodiments, additions, changes, deletions, and the like. It can be changed within the range possible, and is included in the scope of the present invention as long as the action / effect of the present invention is exhibited in any of the embodiments.

100 ... Projection system, 102 ... Screen, 104 ... Projection image, 106 ... Single projection image, 110 ... Image processing device, 112 ... Content storage unit, 114 ... Correction processing unit, 116 ... Projection image output unit, 122 ... Switching Unit, 124 ... Calibration image storage unit, 126 ... Calibration captured image input unit, 128 ... Calibration parameter input unit, 130 ... Correction coefficient calculation unit, 132 ... Feature point extraction integration unit, 134 ... Cylindrical correction unit, 136 ... Geometry Correction coefficient calculation unit, 138 ... Blending coefficient calculation unit, 150 ... Projector, 160 ... Camera, 170 ... Input device, 200 ... Calibration image, 210, 212, 214, 216, 230 ... Calibration projection image, 12 ... CPU, 14 ... North Bridge, 16 ... South Bridge, 18 ... RAM, 20 ... Graphic Board, 22 ... PCI, 24 ... LAN Port, 26 ... IEEE1394 Port, 28 ... USB Port, 30 ... Auxiliary Storage, 32 ... Audio Input / Output, 34 ... serial port, 52 ... keyboard, 54 ... mouse

Japanese Unexamined Patent Publication No. 2015-056834

Claims (10)

A projection system for projecting an image onto a projected surface having a bend in at least one direction by one or more projection means.
A grid point extraction means for extracting grid point data indicating distortion of the projected image for each of the one or more projection means from each of the prepared one or more calibration images.
An information input means for receiving the input of geometric information of the projected surface including the relative positional relationship of the calibration image to the image pickup means.
A correction coefficient calculating means for calculating a correction coefficient for each of the one or more projection means based on the grid point data and the geometric information, and the subject is described from the original coordinate system of the one or more calibration captured images. With a correction coefficient calculation means including a correction means for correcting to a coordinate system after correction according to the shape of the projection surface
Including
The correction means
With respect to the at least one direction having the bend
Based on the correspondence between the point on the original coordinate based on the coordinate value on the virtual projective plane and the point on the coordinate on the projected plane, the value of the original coordinate system is used to determine the coordinate on the virtual projective plane. The means to find the value and
Coordinates on the projected plane from the values of the coordinates on the virtual projected plane based on the geometric relationship between the virtual projected plane and the projected plane according to the bending in at least one direction. And the means to find the value of
As a means for obtaining the value of the corrected coordinate system from the value of the coordinate on the projected surface based on the linear transformation.
Including
Further, with respect to a direction orthogonal to the at least one direction.
A projection system comprising a means of obtaining a value from a value in the original coordinate system to the corrected coordinate system so as to cancel the optical effect of the bend in at least one direction . The correction coefficient calculating means is based on a plurality of reference points that define the outer shape of the entire projection area by the one or more projection means, so that the outer shape defined by the plurality of reference points has a predetermined standardized shape. The projection system according to claim 1 , further comprising a means for obtaining a uniform projective transformation. A projection system for projecting an image onto a projected surface having a bend in at least one direction by one or more projection means.
A grid point extraction means for extracting grid point data indicating distortion of the projected image for each of the one or more projection means from each of the prepared one or more calibration images.
An information input means for receiving the input of geometric information of the projected surface including the relative positional relationship of the calibration image to the image pickup means.
A correction coefficient calculating means for calculating a correction coefficient for each of the one or more projection means based on the grid point data and the geometric information, and the subject is described from the original coordinate system of the one or more calibration captured images. With a correction coefficient calculation means including a correction means for correcting to a coordinate system after correction according to the shape of the projection surface
Including
The correction coefficient calculation means is
A means for obtaining a projective transformation such that the outer shape defined by the plurality of reference points has a predetermined standardized shape based on a plurality of reference points defining the outer shape of the entire projection area by the one or more projection means. Including
The projective transformation is performed on the lattice point data extracted by the lattice point extraction means, the lattice point data after the projection transformation is corrected from the original coordinate system to the corrected coordinate system, and the corrected lattice points are applied. The data is further subjected to the inverse transformation of the projective transformation, and the correction coefficient is calculated based on the grid point data after the projective transformation, after the correction, and after the inverse conversion.
Projection system. The projected surface is at least a part of a substantially cylindrical surface, and the geometric information includes size information of the projected surface, distance information of the projected surface from the imaging means, and the imaging of the projected surface. The projection system according to any one of claims 1 to 3, which includes information on the deviation of the center from the means. The information input means further receives an input of a fine adjustment amount of the geometric information, and the correction coefficient calculation means recalculates the correction coefficient by adding the fine adjustment amount to the geometric information. The projection system according to any one of claims 1 to 4 , wherein the projection system is characterized. A grid point conversion means for converting the grid point data of each projection means extracted from the plurality of calibration images to a common coordinate system, and
Claims 1 to 5 further include an image output means for projecting a plurality of corrected images corrected based on the correction coefficients for each of the plurality of projection means calculated by the correction coefficient calculation means by the corresponding plurality of projection means. The projection system according to any one of the above. An image processing device for projecting an image onto a projected surface having a bend in at least one direction by one or more projection means.
A grid point extraction means for extracting grid point data indicating distortion of the projected image for each of the one or more projection means from each of the prepared one or more calibration images.
An information input means for receiving the input of geometric information of the projected surface including the relative positional relationship of the calibration image to the image pickup means.
A correction coefficient calculating means for calculating a correction coefficient for each of the one or more projection means based on the grid point data and the geometric information, and the subject is described from the original coordinate system of the one or more calibration captured images. With a correction coefficient calculation means including a correction means for correcting to a coordinate system after correction according to the shape of the projection surface
Including
The correction means
With respect to the at least one direction having the bend
Based on the correspondence between the point on the original coordinate based on the coordinate value on the virtual projective plane and the point on the coordinate on the projected plane, the value of the original coordinate system is used to determine the coordinate on the virtual projective plane. The means to find the value and
Coordinates on the projected plane from the values of the coordinates on the virtual projected plane based on the geometric relationship between the virtual projected plane and the projected plane according to the bending in at least one direction. And the means to find the value of
As a means for obtaining the value of the corrected coordinate system from the value of the coordinate on the projected surface based on the linear transformation.
Including
Further, with respect to a direction orthogonal to the at least one direction.
An image processing apparatus comprising a means for obtaining a value from a value in the original coordinate system to the corrected coordinate system so as to cancel the optical effect of the bending in at least one direction . An image processing device for projecting an image onto a projected surface having a bend in at least one direction by one or more projection means.
A grid point extraction means for extracting grid point data indicating distortion of the projected image for each of the one or more projection means from each of the prepared one or more calibration images.
An information input means for receiving the input of geometric information of the projected surface including the relative positional relationship of the calibration image to the image pickup means.
A correction coefficient calculating means for calculating a correction coefficient for each of the one or more projection means based on the grid point data and the geometric information, and the subject is described from the original coordinate system of the one or more calibration captured images. With a correction coefficient calculation means including a correction means for correcting to a coordinate system after correction according to the shape of the projection surface
Including
The correction coefficient calculation means is
A means for obtaining a projective transformation such that the outer shape defined by the plurality of reference points has a predetermined standardized shape based on a plurality of reference points defining the outer shape of the entire projection area by the one or more projection means. Including
The projective transformation is performed on the lattice point data extracted by the lattice point extraction means, the lattice point data after the projection transformation is corrected from the original coordinate system to the corrected coordinate system, and the corrected lattice points are applied. The data is further subjected to the inverse transformation of the projective transformation, and the correction coefficient is calculated based on the grid point data after the projective transformation, after the correction, and after the inverse conversion.
Image processing device. A program for projecting an image onto a projected surface having a bend in at least one direction by one or more projection means, the computer.
A grid point extraction means for extracting grid point data indicating distortion of the projected image for each of the one or more projection means from each of the prepared one or more calibration images.
An information input means for receiving the input of geometric information of the projected surface including the relative positional relationship of the calibration image to the image pickup means.
A correction coefficient calculating means for calculating a correction coefficient for each of the one or more projection means based on the lattice point data and the geometric information, and the subject is described from the original coordinate system of the one or more calibration captured images. Correction coefficient calculation means including correction means for correcting to the coordinate system after correction according to the shape of the projection surface
To function as
The correction means
With respect to the at least one direction having the bend
Based on the correspondence between the point on the original coordinate based on the coordinate value on the virtual projective plane and the point on the coordinate on the projected plane, the value of the original coordinate system is used to determine the coordinate on the virtual projective plane. The means to find the value and
Coordinates on the projected plane from the values of the coordinates on the virtual projected plane based on the geometric relationship between the virtual projected plane and the projected plane according to the bending in at least one direction. And the means to find the value of
A means for obtaining the value of the corrected coordinate system from the value of the coordinate on the projected surface based on the linear transformation.
Acts as
Further, with respect to a direction orthogonal to the at least one direction.
A means for obtaining a value from a value in the original coordinate system to the corrected coordinate system so as to cancel the optical effect of the bending in at least one direction.
A program that acts as . A program for projecting an image onto a projected surface having a bend in at least one direction by one or more projection means, the computer.
A grid point extraction means for extracting grid point data indicating distortion of the projected image for each of the one or more projection means from each of the prepared one or more calibration images.
An information input means for receiving the input of geometric information of the projected surface including the relative positional relationship of the calibration image to the image pickup means.
A correction coefficient calculating means for calculating a correction coefficient for each of the one or more projection means based on the lattice point data and the geometric information, and the subject is described from the original coordinate system of the one or more calibration captured images. Correction coefficient calculation means including correction means for correcting to the coordinate system after correction according to the shape of the projection surface
To function as
The correction coefficient calculation means is
As a means for obtaining a projective transformation such that the outer shape defined by the plurality of reference points has a predetermined standardized shape based on a plurality of reference points defining the outer shape of the entire projection area by the one or more projection means. Function,
The projective transformation is performed on the lattice point data extracted by the lattice point extraction means, the lattice point data after the projection transformation is corrected from the original coordinate system to the corrected coordinate system, and the corrected lattice points are applied. The data is further subjected to the inverse transformation of the projective transformation, and the correction coefficient is calculated based on the grid point data after the projective transformation, after the correction, and after the inverse conversion.
program.