JP5779071B2 - Multiple projection brightness adjustment method, multiple projection brightness adjustment apparatus, computer program, and recording medium - Google Patents

Description

  The present invention automatically corrects the output luminance value from the projector so that the observed luminance value of each pixel becomes a predetermined value when observing the projected images from a plurality of projectors with a camera using a multi-projector system. The present invention relates to multiple projection brightness adjustment technology.

  Projector / camera systems are used in various applications. For example, in addition to using a projector for a conference or presentation at an academic conference presentation, it can also be used as a display system for human interaction or augmented reality. Furthermore, if a plurality of projectors are used, images from the projectors can be connected to generate a huge image (tiling), and in recent years, it is also expected as an application for displaying HDR (High Dynamic Range) images.

For example, in some applications such as a teleconferencing system, when projecting an image from a single projector and observing the projected image with a single camera, the brightness of the original image is observed so that the camera observes a predetermined brightness. Need to be adjusted. Conventionally, this technique is called luminance correction, and it is called a color mixing matrix by focusing on the correlation between RGB luminance values from a projector and RGB luminance values observed by a camera. Using the 3 × 3 matrix V, the RGB output values P = (P _R , P _G ,...) From the projector so that the camera observes a predetermined RGB luminance value C = (C _R , C _G , C _B ). P _B ) is corrected. In this system, the following expression (1) is established between the luminance value C of the observed image and the output luminance P from each projector.

V is a color mixing matrix of the projector, and F = (F _r , F _g , F _b ) is ambient light including illumination from a light source other than the projector (considering the reflectance of the object surface is constant). According to Non-Patent Document 1, if four sample images can be prepared, a color mixing matrix and ambient light can be obtained.

  When the above luminance correction is extended to a multi-projector system composed of N projectors and one camera, the relationship of the following equation (2) is obtained.

P _j is the luminance of the j-th projector, and V _j is a color mixing matrix that combines the luminance of the j-th projector and the luminance obtained by camera observation. When a black image is set on N-1 projectors other than the j-th projector, the multi-projector system becomes a conventional projector / camera system represented by Expression (1). Therefore, in this system, the color mixing matrix V _j can be obtained using the method of Non-Patent Document 1.

  Regarding the luminance correction of the multi-projector system, Non-Patent Document 2 is known. This is a method of determining the brightness of each projector by sharing the brightness correction parameters of all the projectors and distributing the original brightness equally among the N projectors. When the luminance equalization method is adopted for luminance correction using the color mixing matrix, the luminance of each projector can be corrected by the following equation (3).

However, N _j is the number of overlapping projectors, and if N projectors project the pixel in multiple, N _j = N, and if only one projector projects the pixel, N _j = Set to 1.

K. Fujii, MD Grossberg, and SK Nayar: “A Projector-Camera System with Real-Time Photometric Adaptation for Dynamic Environments”, Proc. Of IEEE Computer Vision and Pattern Recognition (CVPR), vol.1, pp.814-821 , 2005. O. Bimber, A. Emmerling, and T. Klemmer: “Embedded Entertainment with Smart Projec-tors”, IEEE Computer, vol. 38, no. 1, pp. 48-55, 2005.

  FIG. 11 is a conceptual diagram showing a typical example of a luminance output characteristic (tone curve) obtained by observing input luminance (a certain color channel) given to a projector with a camera. The luminance correction of the multi-projector system assumes a linear sum of the projector luminances as shown in Equation (2). However, in many cases, a tone curve as shown in FIG. 11 is obtained in the input / output characteristics of the projector input luminance and the camera observation luminance. The low gradation shows a gentle response characteristic with respect to the input luminance of the projector, the intermediate gradation becomes a substantially linear characteristic, and the high gradation becomes a gentle response characteristic again. Therefore, it deviates from the linear characteristic at a response level having a low luminance or a high response level as compared with the intermediate gradation. In addition, as shown by the low gradation in FIG. 11, even when the brightness value of the projector is 0, there is a black offset in camera observation (complete black is not observed).

  The tone curve between one projector and one camera also has such a non-linear characteristic, so that a more complicated tone curve is obtained between a plurality of projectors and one camera. Therefore, in the multi-projector system, there is a limit to the application range in correcting the luminance with the linear sum of the luminance from each projector. For example, in the luminance correction according to Equation (3), as the number of projectors increases, the distributed luminance value is inversely proportional, so the response characteristic of the camera approaches the low gradation characteristic of FIG. Therefore, in the multi-projector system, the non-linear response characteristic becomes more prominent, and there is no guarantee that sufficient correction can be made by simply equally distributing the luminance.

  In addition, in the brightness correction method known in Non-Patent Document 2, the correction parameters are shared by all projectors, and the equally distributed brightness is calculated using those parameters. That is, brightness correction for all projectors cannot be performed unless correction parameters for all projectors are used. As a result, there is a problem that this method cannot be easily applied when the multi-projector system has nonlinear characteristics as described above. Furthermore, it is necessary to implement correction parameter sharing in response to changing the number of projectors or changing the overall system configuration.

  In view of convenience in various applications, it is important to be able to flexibly cope with luminance correction even if such a system change occurs. Therefore, even in a system in which the linear sum of the projector luminances shown in Equation (2) does not hold in the multi-projector system and there is a non-linear response that cannot be modeled by the equations, the luminance projected from a plurality of projectors can be corrected. desired.

  The present invention has been made in view of such circumstances, and its purpose is to provide an arbitrary content image projected by a plurality of projectors, even if it has a non-linear response in a multiple projection system using a plurality of projectors. It is an object of the present invention to provide a technique capable of correcting the luminance of an arbitrary content image so that is substantially equal to the original image.

One aspect of the present invention is a multi-projection brightness adjustment method in a multi-projector system that captures images projected by N projectors using a single camera, and the same test image that has been input is input to the N projectors . An observation image acquisition step of acquiring a projection image superimposed and projected by a projector as an observation image observed by the camera; a luminance variation detection step of detecting a luminance variation representing a difference between the test image and the observation image; A luminance control step for controlling the luminance for each of the N projectors based on the detected luminance variation, and distribution of each pixel for distributing the luminance to the N projectors based on the result of the luminance control A calculation step of calculating a coefficient, reading a content image, and calculating a luminance value of each pixel on the content image Based on the distribution coefficient, the distribution step of distributing a luminance value assigned to the projector each of the N number, relative to the content image and the brightness value is distributed, so as to be projected onto a predetermined plane A projective conversion step of performing a projective transformation in accordance with the position of each of the N projectors; and a step of projecting each content image subjected to the projective transformation from the N projectors in accordance with the projective transformation. This is a characteristic feature of the multiple projection brightness adjustment method.

  One aspect of the present invention is the above-described multiple projection luminance adjustment method, wherein the calculating step includes calculating each pixel value of the test image obtained based on the luminance after the luminance control. It is a step of calculating the distribution coefficient based on a luminance distribution model associated with pixel values by linear combination.

One aspect of the present invention is the multiple projection luminance adjustment method described above, wherein the observation image acquisition step and the luminance variation detection step are repeatedly executed, and the luminance control step includes the luminance variation being a predetermined allowable value or more. In this case, after controlling the luminance state input to the N projectors, projecting an image again from the N projectors based on the updated luminance, and the luminance fluctuation is the predetermined allowable value. If it is smaller, a completion determination step for determining that the brightness control of the N projectors for the test image has been completed is included.

One aspect of the present invention is a multi-projection brightness adjustment apparatus for a multi-projector system that captures images projected by N projectors with a single camera, and the same test image input is input to the N projectors. An observation image acquisition unit that acquires a projection image superimposed and projected as an observation image observed by the camera, a luminance variation detection unit that detects a luminance variation representing a difference between the test image and the observation image, and A luminance control unit for controlling the luminance for each of the N projectors based on the detected luminance variation; and a distribution coefficient for each pixel for distributing the luminance to the N projectors based on the result of the luminance control And a calculation unit that calculates a luminance value of each pixel on the content image based on the distribution coefficient. Of a distribution unit for distributing a luminance value assigned to the projector each, relative to the content image and the brightness value is distributed to the N of projectors each position to be projected onto a predetermined plane A projection conversion unit that performs a corresponding projection conversion; and a projection unit that projects the content images subjected to the projection conversion from the N projectors according to the projection conversion. It is.

  One aspect of the present invention is a computer program that causes a computer to execute the above-described multiple projection luminance adjustment method.

  One aspect of the present invention is a recording medium on which a computer program for causing a computer to execute the above-described multiple projection luminance adjustment method is recorded.

  According to the present invention, in the multiple projection system using a plurality of projectors, the brightness of the content images projected by the plurality of projectors can be corrected so as to be almost the same as the original image even if it has a non-linear response. it can.

It is a block diagram which shows the structure of the multiple projection brightness adjustment apparatus 1 of 1st Embodiment which concerns on this invention. It is a flowchart for demonstrating the operation | movement of this 1st Embodiment. It is a flowchart for demonstrating the operation | movement of this 1st Embodiment. It is a block diagram which shows the structure of the multiple projection brightness adjustment apparatus 1 of 2nd Embodiment which concerns on this invention. It is a flowchart for demonstrating the operation | movement of this 2nd Embodiment. It is a flowchart for demonstrating the operation | movement of this 2nd Embodiment. It is a block diagram which shows the structure of the multiple projection brightness adjustment apparatus 1 of 3rd Embodiment which concerns on this invention. It is a flowchart for demonstrating the operation | movement of this 3rd Embodiment. It is a block diagram which shows the structure of the multiple projection brightness adjustment apparatus 1 of 4th Embodiment which concerns on this invention. It is a flowchart for demonstrating operation | movement of this 4th Embodiment. It is a conceptual diagram which shows the typical example of the characteristic (tone curve) of the luminance output which observed the input luminance (a certain color channel) given to the projector with the camera.

  Hereinafter, an embodiment of the present invention will be described with reference to the drawings.

A. First Embodiment A first embodiment of the present invention will be described.
FIG. 1 is a block diagram showing a configuration of a multiple projection brightness adjustment apparatus 1 according to the first embodiment of the present invention. In FIG. 1, a multiple projection brightness adjustment apparatus 1 realizes an automatic projection brightness adjustment method for a multi-projector system including N projectors 10 and one camera 20. The projection brightness automatic adjustment apparatus 1 includes an image input unit 2, an image fluctuation detection unit 3, a mode determination unit 4, brightness adjustment units 5-1 to 5-N, and brightness distribution units 6-1 to 6-N. . The image input unit 2 acquires image data from the camera 20. The image variation detection unit 3 acquires an image from the image input unit 1 and detects luminance variation using the image information. The mode determination unit 4 sends out a synchronization signal for controlling all projectors, and instructs the luminance adjustment units 5-1 to 5-N or the luminance distribution units 6-1 to 6-N to perform processing. The luminance adjustment units 5-1 to 5-N adjust the luminance output from the projectors 10-1 to 10-N according to the luminance variation obtained by the image variation detection unit 3, and are used for multiple projection luminance correction. Calculate the distribution coefficient. The luminance distribution units 6-1 to 6-N correct the luminance of each projector using the distribution coefficient.

  Unlike the prior art, the first embodiment does not work with other projectors. In the first embodiment, the luminance adjustment units 5-1 to 5-N and the luminance distribution units 6-1 to 6-N that control the projectors 10-1 to 10-N are transmitted from the mode determination unit 4. Each operates independently according to the synchronization signal. In the first embodiment, the distribution coefficient obtained by the brightness adjusting units 5-1 to 5-N so that any content image projected by the plurality of projectors 10-1 to 10-N is substantially equal to the original image. Is used to automatically correct the luminance of an arbitrary content image without interlocking with other projectors.

  In this configuration, the projectors 10-1 to 10-N and the camera 20 do not necessarily have to be connected as constituent elements, and may acquire data necessary for processing. The data flow from the image input unit 2, the image fluctuation detection unit 3, the luminance adjustment units 5-1 to 5-N, and the luminance distribution units 6-1 to 6-N to the respective arrows includes a hard disk, a RAID device, and a CD- Either a form using a recording medium such as a ROM or a form using a remote data resource via a network may be used.

  Details of each part will be described below.

(Mode determination unit 4)
When activated, mode determination unit 4 detects a state for controlling projectors 10-1 to 10-N. If there is no data for performing luminance correction in the initial state, it is determined that there is no correction parameter in the state detection, and “process of luminance adjustment unit” is selected in mode selection. When the mode is the adjustment mode, the mode determination unit 4 causes the luminance adjustment units 5-1 to 5-N to execute processing. When the mode is not the adjustment mode, the mode distribution unit 6-1 to 6-N Execute the process.

  Further, the mode determination unit 4 sends a synchronization signal to the luminance adjustment units 5-1 to 5-N and the luminance distribution units 6-1 to 6-N that control the projectors. When the luminance adjustment units 5-1 to 5-N and the luminance distribution units 6-1 to 6-N receive this synchronization signal, the corrected images are output to the projectors 10-1 to 10-N. Thereby, all the projectors 10-1 to 10-N can project an image without generating a time lag.

(Image input unit 2)
The image input unit 2 is a unit that acquires an image captured by the camera 20 or the like, and sends the acquired image data to the luminance adjustment units 5-1 to 5-N or the luminance distribution units 6-1 to 6-N. Forward. An example of an image acquired by the image input unit 2 is an image captured without hiding the entire screen, or an image obtained by photographing a predetermined area.

(Image fluctuation detection unit 3)
In the first embodiment, each projector 10-1 to 10-N projects an image on a predetermined plane, and the camera 20 operates (observes) the image. In this embodiment, the image fluctuation detection unit 3 performs projective transformation for associating pixels on the image observed from the image input unit 2 with pixels on the original image. The projective transformation method will be described below.

The two-dimensional coordinates of a point on a projector screen are (u, v), the three-dimensional coordinates (X, Y, 0) when the point is projected onto the XY plane with Z = 0, and the projected point is the camera 20. If the two-dimensional coordinates of the point when observed at (x, y) are (x, y), the plane projection transformation H _ps between a certain projector 10-i (i = 1 to N) and the plane, and the plane projection between the plane and the camera 20 With the conversion H _sc , each pixel can be associated with each other by calculation of the following equations (4) and (5).

  Further, the two-dimensional coordinates (u, v) of the image output from the projector 10-i and the two-dimensional coordinates (u ′, v ′) of the point on the original image are calculated by calculating the following equation (6). Can be associated.

H _ip is a planar projective transformation between the original image and the projector screen. Therefore, if these plane projective transformations are known, the two-dimensional coordinates (u ′, v ′) of the v image and the two-dimensional coordinates (x, y) observed by the camera 20 and on the projector screen are displayed. The two-dimensional coordinates (u, v) and the two-dimensional coordinates (x, y) observed by the camera 20 can be linked. Planar projection transformations H _ps , H _sc , and H _ip are obtained for each projector by a pre-calibration operation, and these data are stored in the projection transformation DB 8. In this calibration, a known planar projective transformation estimation method can be used.

  When obtaining the observation image from the image input unit 2, the image fluctuation detection unit 3 reads the plane projection transformation data from the projection transformation DB 8 and follows the two-dimensional coordinate values (u ′, v) of the pixels of the original image according to Equation (6). The two-dimensional coordinate value (x, y) corresponding to the pixel on the observation image is calculated from '). The image fluctuation detection unit 3 applies this coordinate calculation to all the target pixels, and generates an image obtained by projective transformation of the observed image.

  Since each pixel of the projective transformation image is associated with each corresponding pixel of the original image, the pixel value C ′ (t) of the observed image can be compared with the pixel value C on the original image. Since the images are sequentially observed from the camera 20 until the operation of the first embodiment is completed, the pixel value of the observed image at the state t or at the time t is represented as C ′ (t). The luminance fluctuation e (t) of each pixel is calculated by the following equation (7) while associating each pixel of the original image and the observed image using planar projective transformation.

  The image fluctuation detection unit 3 detects a luminance fluctuation with respect to the original image by using Equation (7) for the sequentially observed images.

(Brightness adjustment unit 5-1 to 5-N)
When the mode determination unit 4 selects luminance adjustment processing, the luminance adjustment units 5-1 to 5-N start processing, and first set a test image. As the test image, four types of images (gray image, red image, green image, and blue image) used in Non-Patent Document 1 are used. In the luminance adjustment units 5-1 to 5 -N, processing is started from the gray image, and when the luminance correction is completed to some extent, the red image, the green image, and the blue image are sequentially switched and processed. .

  In the following, the processing is described for a certain test image, but the other images are similarly performed. Since the present invention performs the same process for all projectors, the brightness adjustment unit 5-j for the j-th projector 10-j out of the N projectors 10-1 to 10-N is the process. explain. In the following, j is added to the subscript to emphasize the j-th.

  After the test image is set, the state equation for controlling the brightness is initialized at each time or in each state. In the initialization of the luminance state, each pixel value is expressed by the following equation (8).

  α is a constant for distributing the original luminance of the test image to each projector. When N projectors 10-1 to 10-N are used, α = 1 / N.

  The luminance adjustment unit 5-j takes out the data of the plane projective transformation from the projective transformation DB 8 so that the test image is projected onto a predetermined plane or region, and projects the test image from the test image using the data of the plane projective transformation. A converted image is generated. The plane projection transformation data is read once and temporarily stored in a memory (not shown). Since this planar projective transformation has been described in the above-described image fluctuation detection unit 3, details thereof will be omitted. Since the synchronization signal is transmitted from the mode determination unit 4 at the timing of observing the image, the luminance adjustment unit 5-j receives the synchronization signal, and after a certain period of time, receives the projection conversion image from the projector 10. Set to j and project the test image onto a plane.

  The luminance adjustment unit 5-j calculates an average value (luminance error) of errors for all pixels from the luminance variation e (t) obtained by the image variation detection unit 3, and the luminance error is a predetermined allowable value ε ( For example, if ε = 10), the processing is stopped at the present time, and the next test image is set. On the other hand, when the luminance error is greater than or equal to the predetermined allowable value ε, the luminance state of each pixel in the state t + 1 is controlled according to the following equation (9).

  In this state equation, K plays a role of a gain parameter in feedback control, and is given by the following equation (10) in the first embodiment.

  As for the set value of k, for example, k = 0.7 in an example using two projectors, and k = 0.5 in an example using three projectors. Next, in order to obtain the input luminance to the projector 10-j, the luminance is converted by the following equation (11).

V _j is the color mixing matrix of projector 10-j, and F is the ambient light vector. These parameters are obtained in advance using Non-Patent Document 1.

  The pixel whose luminance is updated using Expression (9) is set to the pixel on the image given as the next state to the projector 10-j. At this time, similarly to the projection conversion of the observation image, the luminance adjustment unit 5-j generates a projection conversion image using the plane projection conversion data extracted from the projection conversion DB 8. The pixel of the conversion image for projection is a pixel of the projector screen. Since an image buffer having the same size as the test image is prepared and each pixel corresponds to (u ′, v ′), the relationship between the planar projection transformations of the equations (4), (5), and (6) is used. The coordinates (u, v) on the projector screen are calculated. The pixel whose luminance has been updated is set at the point of the coordinates (u, v). After updating the luminance of all the pixels, the luminance adjusting unit 5-j sets the projection conversion image on the projector 10-j and projects the updated image on the plane again.

  The luminance adjustment according to Equation (9) is performed according to the luminance fluctuation in each state t, and is continued until the luminance error becomes smaller than the allowable error ε. For each test image, when the luminance error becomes smaller than the allowable error ε, it is determined that the luminance adjustment of each test image is completed, and is temporarily reserved in the working memory. The above processing is repeated by setting test images one after another.

After correcting the luminance of all test images, the distribution coefficient is calculated. The above processing is repeated to adjust the brightness of the gray-based image, red-based image, green-based image, and blue-based image, and when the final brightness correction image is obtained, the brightness value of each pixel on each image is C It is assumed that ^W _j, C ^R _j , C ^G _j , and C ^B _j and the luminance values of the respective original pixels on the test image are C _W , C _R , C _G , and C _B. These luminance values are related by the following equation (12).

D _j is a 3 × 3 matrix, and d _j is a three-dimensional vector, which are referred to as distribution coefficients in the present invention. In other words, the luminance of each pixel of an arbitrary content image is corrected by applying the relational expression (12). D _j and d _j are calculated by solving simultaneous equations of Equation (12). When the luminance adjustment unit 5-j calculates the distribution coefficients D _j and d _j of each pixel, the luminance adjustment unit 5-j stores them in the color distribution DB 7-j and ends the process.

(Luminance distribution units 6-1 to 6-N)
The luminance distribution units 6-1 to 6-N change the mode to “luminance distribution mode” when the mode determination unit 4 detects the state “luminance adjustment is completed”. When the mode determination unit 4 selects the processing of the luminance distribution units 6-1 to 6-N, the luminance distribution units 6-1 to 6-N execute setting of luminance distribution data. D _j and d _j of each pixel are extracted from the color distribution DBs 7-1 to 7-N and stored in a temporary memory (not shown). Next, when an arbitrary image is read from the content image DB 9, the luminance value C of each pixel on the image is calculated and the luminance value P _j to each projector 10- _j is calculated by the following equations (13) and (14). And distribute.

At this time, the projection conversion image is generated using the data of the plane projection conversion extracted from the projection conversion DB 8 in the same manner as the projection conversion image is generated in the luminance adjustment unit 5-j. The pixel of the conversion image for projection is a pixel of the projector screen. Since an image buffer having the same size as the test image is prepared and each pixel corresponds to (u ′, v ′), the relationship between the planar projection transformations of the equations (4), (5), and (6) is used. The coordinates (u, v) on the projector screen are calculated. The new luminance value P _j is set at the point of the coordinates (u, v). After setting all the pixels, the synchronization signal from the mode determination unit 4 is received, and the converted image is output to the projector 10-j after a certain time (a minute time of 1 second or less). Thereby, the brightness corrected image is projected onto a predetermined plane. If the synchronization signal is received and the signal is a stop command, the process is terminated.

  2 and 3 are flowcharts for explaining the operation of the first embodiment. When activated, mode determination unit 4 detects a state for controlling projectors 10-1 to 10-N (step S1). Next, mode selection is executed (step S2). If there is no data for performing luminance correction in the initial state, it is determined that “there is no correction parameter” in the state detection in step S1, and “luminance adjustment unit 5-1” is selected in the mode selection in step S2. Select "5-N Processing". Next, it is determined whether or not the selected mode is an adjustment mode (step S3). If the selected mode is the adjustment mode (YES in step S3), the processing of the luminance adjustment units 5-1 to 5-N is executed.

  When the mode determination unit 4 selects the luminance adjustment process, the luminance adjustment units 5-1 to 5-N set a test image (step S4) and determine whether or not the adjustment is completed (step S5). If the adjustment has not been completed (NO in step S5), the brightness state (state equation) of the image set on the projector 10 is initialized (step S6), and the test image is transferred to a predetermined plane or region. In order to perform projection, plane projection transformation data is extracted from the projection transformation DB 8, and a projection transformation image is generated from the test image using the plane projection transformation data (step S12).

  Next, when the brightness adjustment units 5-1 to 5-N identify that the synchronization signal has been received from the mode determination unit 4 at the timing of observing the image (step S13), after a certain time has elapsed. The conversion image for projection is set on the projectors 10-1 to 10-N, and the test image is projected onto a plane (step S14).

  The image input unit 2 acquires an image observed (captured) by the camera 20 and projected on the screen (plane) by the projectors 10-1 to 10-N, and transfers the acquired image to the luminance variation detection unit 3 (step S7). ). The luminance fluctuation detection unit 3 takes out the data of the planar projection conversion from the projection conversion DB 8, performs the projection conversion for associating the pixels on the image with the pixels on the original image, and generates a projection conversion image (step S8). Next, the luminance variation detection unit 3 detects the luminance variation e (t) with the original image for the sequentially observed images (step S9).

  The luminance adjustment units 5-1 to 5-N calculate an average value (luminance error) of errors for all pixels from the luminance variation e (t) obtained by the image variation detection unit 3, and the luminance error is a predetermined value. It is determined whether or not it is smaller than an allowable value ε (for example, ε = 10 is set) (step S10). If the luminance error is greater than or equal to the predetermined allowable value ε, the luminance state of each pixel in the state t + 1 is controlled according to Equation (9) (step S11).

  Next, the luminance adjustment units 5-1 to 5-N generate a projection conversion image using the data of the plane projection transformation extracted from the projection transformation DB 8 in the same manner as the projection transformation of the observation image (Step S1). S12) When it is identified that the synchronization signal has been received from the mode determination unit 4 (step S13), after updating the luminance of all the pixels, the luminance adjusting units 5-1 to 5-N display the projection conversion images. The projectors 10-1 to 10-N are set, and the updated image is projected again on the plane (step S14).

  The luminance adjustment in steps S7 to S14 described above is performed according to the luminance fluctuation in each state t, and is continued until the luminance error becomes smaller than the allowable error ε. Then, for each test image, when the luminance error becomes smaller than the allowable error ε (YES in step S10), the next test image is set and the above processing is repeated (steps S4 to S14).

On the other hand, when the correction of the luminances of all the test images is completed (YES in step S5), the luminance adjusting units 5-1 to 5-N perform distribution coefficient calculation, and the distribution coefficients D _j and d _{j of} each pixel are calculated. Are calculated (step S15), they are stored in the color distribution DBs 7-1 to 7-N, and the process returns to the state detection in step S1.

On the other hand, when the selected mode is not the adjustment mode (NO in step S3), that is, when the mode is the “luminance distribution mode”, the process of the luminance distribution unit 6-j is executed. The luminance distribution unit 6-j extracts D _j and d _j of each pixel from the color distribution DB 7-j, and sets luminance distribution data (step S16 in FIG. 3). Next, an arbitrary image is read from the content image DB 9 (step S17), and the luminance value C of each pixel on the image is distributed as the luminance value to each projector 10-j (step S18).

  Next, the luminance distribution unit 6-j generates a projection conversion image using the plane projection conversion data extracted from the projection conversion DB 8 (step S19), sets all the pixels, and then sets the mode determination unit 4. Is identified (step S20), and it is determined whether or not the synchronization signal is the end of processing (stop command) (step S21). If the synchronization signal is not the end of processing (stop command) (NO in step S21), the converted image is output to the projector 10-j after a certain time (a minute time of 1 second or less). Then, the luminance corrected image is projected onto a predetermined plane (step S22). Then, it returns to step S1 of FIG. On the other hand, when the synchronization signal is the process end (stop command) (YES in step S21), the process ends.

  According to the first embodiment described above, the parameters and the like are shared among all the projectors by holding the distribution coefficient for distributing the luminance of the image of an arbitrary content from the appropriately given test image to each projector. The brightness correction of the multi-projector system can be performed flexibly. Furthermore, since the distribution coefficient does not depend on the content image, an arbitrary content image can be corrected by using the distribution coefficient only by holding the distribution coefficient once in each projector.

B. Second Embodiment Next, a second embodiment of the present invention will be described.
FIG. 4 is a block diagram showing the configuration of the multiple projection luminance adjustment apparatus 1 according to the second embodiment of the present invention. It should be noted that portions corresponding to those in FIG. In the second embodiment, when a change occurs in an environment such as lighting while projecting an arbitrary content image, the brightness of the projectors 10-1 to 10-N is controlled according to the brightness change. It is characterized by. The second embodiment has the same configuration as that of the first embodiment described above, but the image fluctuation detection unit 3 in both the luminance adjustment units 5-1 to 5-N and the luminance distribution units 6a-1 to 6a-N. Use the luminance fluctuation obtained in.

(Luminance distribution units 6a-1 to 6a-N)
When the mode determination unit 4 selects the processing of the luminance distribution units 6a-1 to 6a-N, the luminance error e (t) in the state t obtained by the image fluctuation detection unit 3 is passed to the luminance distribution units 6a-j. The luminance distribution unit 6a-j sets the distribution coefficient data D _j and d _j of each pixel from the color distribution DB 7-j, and then reads an arbitrary image from the content image DB 9, and the luminance value of each pixel on the image The luminance value P _j is calculated and distributed to each projector 10-j by the following equation (15).

  K uses Equation (10) (the value of parameter k may be changed). When a luminance error occurs in each pixel, it is possible to correct the luminance by absorbing the luminance variation by feedback control according to Equation (15).

5 and 6 are flowcharts for explaining the operation of the second embodiment. In addition, the same step is given to the part corresponding to FIG. 2, FIG. 3 in 1st Embodiment, and description is simplified. If the selected mode is the adjustment mode (YES in step S3), steps S4 to S14 are repeatedly executed as in the first embodiment described above, and finally the luminance of all test images is corrected. When completed (YES in step S5), the luminance adjustment unit 5-j calculates the distribution coefficient, calculates the distribution coefficients D _j and d _j of each pixel (step S15), and stores them in the color distribution DB 7-j. And return to the state detection in step S1.

  On the other hand, when the selected mode is the luminance distribution mode (NO in step S3), the screen (plane) is observed by the projectors 10-1 to 10-N observed (captured) by the camera 20 as in the first embodiment described above. The image input unit 2 obtains the image projected above (step S7), and the luminance fluctuation detection unit 3 takes out the data of the plane projective transformation from the projective transformation DB 8, and extracts the pixels on the image and the pixels on the original image. Projective transformation for associating with each other to generate a projective transformation image (step S8). Next, the luminance variation detection unit 3 detects the luminance variation e (t) with the original image for the sequentially observed images (step S9). In this case, since the mode is the “luminance distribution mode” (NO in step S3), the above-described steps S16 to S22 are executed again.

Here, the luminance distribution section 6a-1~6a-N is a step S18, after setting the distribution coefficient data _{D j} and _{d j} of each pixel from the color distribution DB7-j, reads an arbitrary image from the content image DB9 And the luminance value C of each pixel on the image, using the luminance error e (t) from the image fluctuation detection unit 3, the luminance value P _j for each projector 10- _j is calculated according to Equation (15). And distribute. In step S22, the converted image is output to the projector 10-j, and the luminance-corrected image is projected onto a predetermined plane. Thereafter, the process returns to step S7 in FIG. 5, and the above-described processing is repeatedly executed until the synchronization signal reaches the end of processing (stop command). When the synchronization signal reaches the end of processing (stop command) (YES in step S21), the processing is performed. Exit.

  According to the second embodiment described above, even when a change occurs in the environment such as lighting while the content image is projected by the projectors 10-1 to 10-N, the image is taken by the camera 20, Since the luminance variation is detected in real time by the luminance variation detecting unit 3 and the luminance distributed to the projectors 10-1 to 10-N is controlled by the luminance distributing units 6a-1 to 6a-N, the content image is projected. The brightness of the projectors 10-1 to 10-N can be controlled in accordance with the brightness change.

  Furthermore, according to the second embodiment, since the brightness is corrected without sharing the correction parameter for all projectors, the brightness correction can be performed flexibly even if the number or arrangement of projectors is changed.

C. Third Embodiment Next, a third embodiment of the present invention will be described.
In a situation where there is no change in the system settings or the number and arrangement of projectors, it is possible to save time and effort for calculating the distribution coefficient in the luminance adjustment units 5-1 to 5-N of the first embodiment or the second embodiment. . In the third embodiment, distribution coefficients are stored in the color distribution DBs 7-1 to 7-N in the previous setup, and brightness correction is performed using the distribution coefficients.

  FIG. 7 is a block diagram showing the configuration of the multiple projection brightness adjustment apparatus 1 according to the third embodiment of the present invention. It should be noted that portions corresponding to those in FIG. As shown in the figure, in the third embodiment, the image input unit 1, the image variation detection unit 3, the mode determination unit 4, and the luminance adjustment units 5-1 to 5-N are omitted.

FIG. 8 is a flowchart for explaining the operation of the third embodiment. The flowchart shown in FIG. 8 is executed when the mode determination unit 4 selects the processing of the luminance distribution units 6-1 to 6-N in step S3 of FIG. 2 in the first embodiment described above. 3 is the same as the flowchart shown in FIG. The luminance distribution unit 6-j extracts D _j and d _j of each pixel from the color distribution DB 7-j, and sets the luminance distribution coefficient data (step S30). Next, an arbitrary image is read from the content image DB 9 (step S31), and the luminance value C of each pixel on the image is distributed as the luminance value to each projector 10-j (step S32).

  Next, the luminance distribution units 6-1 to 6-N generate a projection conversion image using the plane projection conversion data extracted from the projection conversion DB 8 (step S33), and after setting all the pixels, If the synchronization signal from the mode determination unit 4 is not the end of processing (stop command) (NO in step S34), the converted image is output to the projector 10-j, and the luminance-corrected image is projected onto a predetermined plane. (Step S35). Then, it returns to step S31 and repeats the process mentioned above with respect to the next content image. On the other hand, when the synchronization signal is the process end (stop command) (YES in step S34), the process ends.

  According to the third embodiment described above, when there is no change in the system settings or the number and arrangement of projectors, the distribution coefficients are stored in the color distribution DBs 7-1 to 7-N in the previous setup, Since the luminance correction is performed using the distribution coefficient, the image input unit 1, the image fluctuation detection unit 3, the mode determination unit 4, and the luminance adjustment units 5-1 to 5-N can be omitted. The trouble of calculating the coefficient can be saved.

D. Fourth Embodiment Next, a fourth embodiment of the present invention will be described.
In the above-described third embodiment, a situation is assumed in which there is no change in illumination other than the projector or ambient light. On the other hand, the fourth embodiment is characterized in that when a change occurs in an environment such as illumination while an arbitrary content image is projected, the luminance is controlled according to the luminance change.

FIG. 9 is a block diagram showing the configuration of the multiple projection luminance adjustment apparatus 1 according to the fourth embodiment of the present invention. It should be noted that portions corresponding to those in FIG. As shown in the figure, in the fourth embodiment, the mode discriminating unit 4 and the luminance adjusting units 5-1 to 5-N are omitted. The image fluctuation detection unit 3 calculates the luminance error e (t) in the state t calculated based on the image from the image input unit 2 when an arbitrary content image is projected, as luminance distribution units 6a-1 to 6a-1. Pass to 6a-N. The luminance distribution unit 6a-j sets the distribution coefficient data D _j and d _j of each pixel from the color distribution DB 7-j, and then reads an arbitrary image from the content image DB 9, and the luminance value of each pixel on the image The luminance value P _j is calculated and distributed to each projector 10-j according to Equation (15).

FIG. 10 is a flowchart for explaining the operation of the fourth embodiment. The luminance distribution unit 6-j extracts D _j and d _j of each pixel from the color distribution DB 7-j, and sets luminance distribution coefficient data (step S40). Next, an arbitrary image is read from the content image DB 9 (step S41), and the luminance value C of each pixel on the image is distributed as the luminance value to each projector 10-j (step S42).

  Next, the luminance distribution units 6-1 to 6-N generate a projection conversion image using the planar projection conversion data extracted from the projection conversion DB 8 (step S43), and after setting all the pixels, The synchronization signal from the mode determination unit 4 is identified (step S44). If the synchronization signal is not the end of processing (stop command) (NO in step S45), the converted image is output to the projector 10-j, and brightness correction is performed. The obtained image is projected onto a predetermined plane (step S46).

  The image input unit 2 acquires images projected (projected) on the screen (plane) by the projectors 10-1 to 10 -N, which are observed (captured) by the camera 20, and transfers them to the luminance variation detection unit 3 (step S 47). ). The luminance variation detection unit 3 takes out the data of the planar projective transformation from the projective transformation DB 8, performs the projective transformation for associating the pixels on the observed image with the pixels on the original image, and generates a projective transformed image (step S48). . Next, the luminance variation detection unit 3 detects a luminance variation e (t) with the original image for the observed image (step S49). The luminance distribution unit 6-j reads an arbitrary image from the content image DB 9 (step S41), and distributes the luminance value C of each pixel on the image as the luminance value to each projector 10-j (step S42). .

  As described above, the content image is captured by the camera 20 while being projected by the projectors 10-1 to 10-N, the luminance variation is detected in real time by the luminance variation detection unit 3, and the luminance distribution units 6-1 to 6-N are detected. The above-described processing is repeated to control the luminance distributed to the projectors 10-1 to 10-N. On the other hand, when the synchronization signal is the process end (stop command) (YES in step S45), the process ends.

  According to the fourth embodiment described above, even when a change occurs in an environment such as illumination while an arbitrary content image is projected, the luminance can be controlled according to the luminance change.

  Furthermore, according to the fourth embodiment, since the brightness is corrected without sharing the correction parameter for all projectors, the brightness correction can be flexibly performed even if the number or arrangement of projectors is changed.

  In the present invention, a storage medium storing software program codes for realizing the functions of the first to fourth embodiments described above is supplied to a system or apparatus, and the CPU (MPU) of the system or apparatus is provided. It can also be realized by reading and executing the program code stored in the storage medium. In this case, the program code itself read from the storage medium realizes the functions of the above-described embodiment, and a storage medium storing the program code, for example, a CD-ROM, DVD-ROM, CD-R , CD-RW, MO, HDD, etc. constitute the present invention.

DESCRIPTION OF SYMBOLS 1 Multiple projection brightness adjustment apparatus 2 Image input part 3 Brightness fluctuation | variation detection part 4 Mode discrimination | determination part 5-1 to 5-N Brightness adjustment part 6-1 to 6-N, 6a-1 to 6a-N Brightness distribution part 7-1 ~ 7-N Color distribution DB
6 Projective transformation DB
8 Brightness correction DB
9 Content image DB
10-1 to 10-N Projector 20 Camera

Claims (6)

A multi-projection brightness adjustment method in a multi-projector system that captures an image projected by N projectors with a single camera,
An observation image acquisition step of acquiring a projection image obtained by superimposing and projecting the same input test image by the N projectors as an observation image observed by the camera;
A luminance fluctuation detecting step for detecting a luminance fluctuation representing a difference between the test image and the observed image;
A luminance control step for controlling the luminance for each of the N projectors based on the detected luminance variation;
A calculation step of calculating a distribution coefficient of each pixel for distributing the luminance to the N projectors based on the result of the luminance control;
A distribution step of reading a content image and distributing a luminance value of each pixel on the content image as a luminance value assigned to each of the N projectors based on the distribution coefficient;
A projective transformation step of performing a projective transformation in accordance with the position of each of the N projectors so that the content image to which the luminance value is distributed is projected onto a predetermined plane;
Projecting each content image that has undergone the projective transformation from the N projectors according to the projective transformation . The calculating step includes:
Calculating the distribution coefficient based on a luminance distribution model in which each pixel value of the test image is linearly combined with each pixel value of the test image obtained based on the luminance after the luminance control. The multiple projection luminance adjustment method according to claim 1, wherein: Repeatedly executing the observation image acquisition step and the luminance variation detection step;
The luminance control step includes
Projecting an image again from the N projectors based on the updated luminance after controlling the luminance state input to the N projectors when the luminance variation is greater than or equal to a predetermined allowable value; ,
If the brightness variation is less than the predetermined tolerance, the N for the test image
A completion determination step for determining that the brightness control of one projector has been completed;
The multi-projection luminance adjustment method according to claim 1 or 2, characterized by comprising: A multi-projection brightness adjustment device for a multi-projector system that captures an image projected by N projectors with a single camera,
An observation image acquisition unit that acquires, as an observation image observed by the camera, a projection image obtained by superimposing and projecting the same input test image by the N projectors;
A luminance fluctuation detecting unit for detecting a luminance fluctuation representing a difference between the test image and the observed image;
A luminance control unit for controlling the luminance for each of the N projectors based on the detected luminance variation;
A calculation unit that calculates a distribution coefficient of each pixel for distributing luminance to the N projectors based on the result of the luminance control;
A distribution unit that reads a content image and distributes a luminance value of each pixel on the content image as a luminance value assigned to each of the N projectors based on the distribution coefficient;
A projective transformation unit that performs projective transformation in accordance with the position of each of the N projectors so that the content image to which the luminance value is distributed is projected onto a predetermined plane;
A projection unit for projecting each content image having undergone the projective transformation from the N projectors according to the projective transformation . The computer program for making a computer perform the multiple projection brightness | luminance adjustment method as described in any one of Claim 1 to 3. RECORDING MEDIUM computer program for executing the multi-projection luminance adjustment method according to the computer in any one of claims 1 3.

Images