JP2006270160A - Image data processing apparatus - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To provide an image data processing apparatus, capable of outputting image data having high image quality on a time axis, using a general-purpose camera capable of imaging moving images. <P>SOLUTION: The apparatus has an image data compositing processing means 12, having a means for inputting a plurality of image data having different frame rates, obtained by picking up the same object by a plurality of image pickup means within the same image pickup range, without taking synchronization between the plurality of imaging means into consideration; and a means of compositing the plurality of input image data so that the image data are temporally continuous to generate a plurality of composite image data. <P>COPYRIGHT: (C)2007,JPO&INPIT

Description

  The present invention relates to an image data processing device, an image data processing method, and an image data processing program suitable for imaging the same subject using a plurality of imaging means. Further, the present invention relates to an image data processing device and an output from the image data processing device. The present invention relates to an image display system having an image display device capable of displaying the image data.

  Digital cameras and digital video cameras that can capture images as digital data, and digital imaging devices such as mobile phones, videophones, and Web cameras are becoming widely used. As these digital imaging devices become more widespread, higher image quality is required.

  As for still images, the number of pixels of images that can be captured is increasing, and the improvement in image quality on the spatial axis is reaching a high level. On the other hand, the current situation is that the moving image is still less than the still image in terms of image quality. For example, a high-definition image is 2 million pixels at most in terms of the number of pixels, and is about the same as or lower than that of a digital camera one or two generations before. Furthermore, further improvement is demanded for improving the image quality on the most important time axis in moving images.

  As digital imaging devices that can be easily obtained, there are, for example, a mobile phone and a Web camera, but none of them has high quality of moving images. In general, if priority is given to the number of pixels (resolution), the frame rate is sacrificed. Conversely, if priority is given to the frame rate, the number of pixels (resolution) is sacrificed. Note that the frame rate indicates the height of image quality on the time axis. If the frame rate is high, it can be said that the image quality is high on the time axis.

Patent Document 1 is an example of a technique for realizing high image quality on a time axis in moving images.
The technique disclosed in Patent Document 1 is intended to realize high-speed imaging by linking a continuous shooting function of a digital camera or a film camera using a clock or a timer between a plurality of cameras.

JP 2001-166374 A

  However, the technology disclosed in Patent Document 1 requires (i) a plurality of cameras having a continuous shooting function, (ii) a complicated mechanism for coordinating a plurality of cameras, and (iii) a plurality of cameras. There is a problem that a mechanism for monitoring the state of the device is necessary.

  Regarding (i), a problem is that a camera having a continuous shooting function is generally expensive. There are some relatively inexpensive cameras that take several shots per second, but with the technology disclosed in Patent Document 1, a large number of cameras are required to achieve a general video frame rate. Necessary. That is, in order to realize 30 frames / second (hereinafter, frames / second is expressed as fps), 10 cameras are required for a 3 frames / second camera, and 5 frames for a 5 frames / second camera. A camera is required. Therefore, for example, more cameras are required to realize high-quality image data of 60 fps, which is not realistic.

  As for (ii), in the case of the technique disclosed in Patent Document 1, since a still image camera is used, control of the system becomes complicated, such as shutter control and consideration of synchronization between cameras corresponding thereto. In addition, high-precision control is required. As a result, there is also a problem that the system itself becomes expensive.

  Further, regarding (iii), this is also a problem due to the use of a still image camera similar to (ii). In general, the continuous shooting function takes time to record on a recording medium built in the camera although imaging is performed at high speed. For continuous imaging, it is necessary to monitor the status of each camera and always monitor whether or not the next continuous shooting is possible after the continuous shooting function ends. This also becomes a factor that complicates the system.

  Therefore, the present invention can generate high-quality image data on the time axis by using a plurality of general-purpose digital imaging devices capable of capturing moving images, and does not complicate the system. An image data processing device and an image display device capable of displaying image data output from the image data processing device are provided. An object of the present invention is to provide an image display system.

  (1) In the image data processing apparatus of the present invention, different frame rates obtained by a plurality of imaging units imaging the same subject in the same imaging range and without considering synchronization between the plurality of imaging units. And a means for synthesizing the plurality of input image data so as to be temporally continuous to generate combined image data.

As a result, a high frame rate can be realized without sacrificing resolution, and high-quality image data can be generated on the time axis. That is, the composite image data generated by the present invention is image data that can reproduce a moving image with smooth motion. In the present invention, it is also possible to generate composite image data for image data with different frame rates respectively input from a plurality of imaging means.
Note that the imaging means is not specially prepared for the present invention, and general-purpose imaging equipment (such as a digital video camera) capable of capturing moving images can be used, and each imaging means is different. It may have a frame rate.

(2) In the image data processing device according to (1), the means for generating the composite image data includes a frame rate of the imaging means having the highest frame rate and the highest frame among the plurality of imaging means. Based on the ratio with the frame rate of the other imaging means other than the imaging means having the rate, the insertion position of the image data of the other imaging means with respect to the image data of the imaging means having the highest frame rate is set, It is preferable to generate the composite image data by combining the frame images in the image data at the insertion position so as to be temporally continuous.
By performing such image data combining processing, it is possible to appropriately generate combined image data for image data of different frame rates respectively input from a plurality of imaging means. In each embodiment of the present invention, image data corresponding to each frame is referred to as a frame image.

(3) The image data processing apparatus according to (1) or (2) further includes an imaging unit control unit capable of setting an imaging start time for at least one of the plurality of imaging units. It is preferable.
By providing such an imaging means control means, the setting of the imaging start time can be arbitrarily performed within a predetermined range by the user or the like. For example, the imaging start time can be set individually for each imaging means, or the time difference with respect to the imaging means can be set by setting a reference imaging means.

(4) In the image data processing device according to any one of (1) to (3), at least one of misalignment correction, color correction, and luminance correction is performed on the plurality of input image data. It is preferred to have a means to do.
As a result, when image data obtained from a plurality of imaging means is combined temporally continuous composite image data, the composite image data can be high-quality image data with consistency between the frame images. .

(5) The image data processing device according to any one of (1) to (4), further including an image data recording device that stores each image data input from the plurality of imaging units, and the composite image data Preferably, the generating unit generates composite image data using each of the image data recorded in the image data recording device.
As described above, the image data recording device that records the image data captured by each imaging unit is provided, and the image data is temporarily recorded in the image data recording device, so that the composite image data performed by the image data composition processing unit Processing such as generation processing can be performed with sufficient time, and it is not necessary to increase the processing capability of the image data synthesis processing unit. In addition, the user can acquire image data at an arbitrary time from the image data recording apparatus.

(6) In the image data processing method according to the present invention, different frame rates obtained by a plurality of imaging units imaging the same subject in the same imaging range and without considering synchronization between the plurality of imaging units. And a step of generating a composite image data by synthesizing the input image data so as to be temporally continuous.
This image data processing method can provide the same effects as those of the image data processing apparatus described in (1). Note that this image data processing method preferably has the same characteristics as those of the image data processing apparatuses (2) to (5).

(7) The image data processing program according to the present invention is configured such that a plurality of imaging units captures the same subject in the same imaging range and images without considering synchronization between the plurality of imaging units. A step of inputting a plurality of image data having the above and a step of generating the combined image data by combining the plurality of input image data so as to be temporally continuous can be executed.
This image data processing method program can provide the same effects as those of the image data processing apparatus described in (1). Note that this image data processing method preferably has the same characteristics as those of the image data processing apparatuses (2) to (5).

  (8) In the image data display system of the present invention, different frame rates obtained by a plurality of imaging units imaging the same subject in the same imaging range and without considering synchronization between the plurality of imaging units. An image data processing apparatus having means for inputting a plurality of image data, and a means for generating composite image data by synthesizing the plurality of input image data so as to be temporally continuous, and from the image data processing apparatus And an image display device that displays the output composite image data.

According to such an image display system, a high frame rate can be realized without sacrificing resolution, and high-quality image data can be generated on the time axis. That is, the composite image data generated by the present invention is image data that can reproduce a moving image with smooth motion, and the image display device can display high-quality image data on the time axis.
The imaging means is not specially prepared for the present invention, and general-purpose imaging means (such as a digital video camera) capable of capturing moving images can be used. The system can be configured at low cost. In the present invention, each imaging means may have a different frame rate. Note that this image display system also preferably has the same characteristics as the image data processing apparatuses (2) to (5).

  (9) In the image data processing apparatus of the present invention, different frame rates obtained by a plurality of imaging units imaging the same subject in the same imaging range and without considering synchronization between the plurality of imaging units. Means for inputting a plurality of image data, means for performing content analysis using the plurality of input image data, means for setting a delay time based on the result of the content analysis, and based on the delay time Then, the intermediate image data is generated, and the generated intermediate image data and at least one of the input plurality of image data are combined so as to be temporally continuous to generate combined image data. Means.

  As a result, the same effect as that of the image data processing apparatus of (1) can be obtained, and in the image data processing apparatus of (9), intermediate image data based on a delay time set according to the content content And the generated intermediate image data and the image data input from the imaging means having a plurality of imaging means are temporally continuous, so that a composite image capable of reproducing a smoother moving image is obtained. Data can be generated.

(10) In the image data processing apparatus according to (9), the content analysis is performed by detecting a change in image motion and / or a change in brightness based on the plurality of input image data. It is preferable.
According to this, it is possible to set a delay time according to a change in motion and / or a change in brightness. For example, a delay time setting that shortens the delay time in scenes with a lot of movement or changes in brightness, and conversely increases the delay time in scenes with a slow movement or changes in brightness. I do. Accordingly, it is possible to set an optimal delay time following the change in the content content, and it is possible to generate intermediate image data corresponding to the content content by using the delay time.

(11) In the image data processing device according to (9) or (10), the means for performing the content analysis includes a frame rate of the imaging means having the highest frame rate among the plurality of imaging means and the highest Based on the ratio with the frame rate of other imaging means other than the imaging means having a high frame rate, the frame image of the imaging means having the highest frame rate and the frame of the other imaging means when performing the content analysis It is preferable to perform association with images and perform content analysis between the associated frame images.
Accordingly, it is possible to generate appropriate intermediate image data corresponding to the difference in imaging timing of each imaging unit, and the composite image data generated using such intermediate image data can reproduce a more natural moving image. Possible image data can be obtained.

(12) In the image data processing device according to (11), the means for generating the composite image data includes the intermediate image data generated by performing content analysis between the attached frame images, and the It is preferable that the image data input from the imaging means having the highest frame rate is combined so as to be temporally continuous in consideration of the temporal positional relationship of the associated frame images.
The composite image data generated in this way can be image data in accordance with the imaging order of each imaging means, and can be image data that can reproduce natural motion.

(13) In the image data processing device according to any one of (9) to (12), the composite image data has a data format in which information indicating a time length of each frame image in the composite image data is added. It is preferable to have means for outputting as data.
As a result, appropriate display can be performed when the composite image data is displayed on the image display device that displays the composite image data.

(14) In the image data processing device according to any one of (9) to (12), the composite image data includes information indicating a time length of each frame image in the composite image data. It is preferable to have a means for outputting as separate data.
Thereby, in the image display device corresponding to the present invention, it is possible to read both the data of the composite image data and the data of the information indicating the time length of each frame image, and perform the processing according to the contents. In addition, in an image display device that does not correspond to the present invention, it is possible to read only the composite image data and perform processing according to the contents.

(15) In the image data processing method of the present invention, a plurality of imaging units have different frame rates obtained by imaging the same subject in the same imaging range and without considering synchronization between the plurality of imaging units. A step of inputting a plurality of image data, a step of performing content analysis using the plurality of input image data, a step of setting a delay time based on a result of the content analysis, and a step of based on the delay time Generating intermediate image data, and generating the combined image data by combining the generated intermediate image data and at least one of the input plurality of image data so as to be temporally continuous. And a step.
This image data processing method can provide the same effects as those of the image data processing apparatus described in (9). Note that the image data processing method described in (15) also preferably has the same characteristics as the image data processing devices of (3) to (5) and (10) to (14).

(16) According to the image data processing program of the present invention, different frame rates obtained by a plurality of imaging units imaging the same subject in the same imaging range and without considering synchronization between the plurality of imaging units. A step of inputting a plurality of image data, a step of performing content analysis using the plurality of input image data, a step of setting a delay time based on a result of the content analysis, and a step of based on the delay time Generating intermediate image data, and generating the combined image data by combining the generated intermediate image data and at least one of the input plurality of image data so as to be temporally continuous. Steps can be executed.
This image data processing program can provide the same effects as those of the image data processing apparatus described in (9). Note that the image data processing method described in (15) also preferably has the same characteristics as the image data processing devices of (3) to (5) and (10) to (14).

  (17) In the image data display system of the present invention, different frame rates obtained by a plurality of imaging units imaging the same subject in the same imaging range and without considering synchronization between the plurality of imaging units. Means for inputting a plurality of image data, means for performing content analysis using the plurality of input image data, means for setting a delay time based on the result of the content analysis, an intermediate image based on the delay time An image having means for generating data and means for generating synthesized image data by synthesizing the generated intermediate image data and at least one of the input plurality of image data so as to be temporally continuous A data processing device, and an image display device for displaying the composite image data output from the image data processing device And it features.

  The image display system described in (17) can obtain the same effect as the image display system described in (8), and the image display system described in (17) can be set according to the contents. The intermediate image data based on the generated delay time is generated, and the generated intermediate image data and the image data input from the imaging unit having a plurality of imaging units are temporally continuous, so that the movement is smoother. It is possible to generate composite image data that can reproduce a moving image. Note that the image display system described in (17) preferably has the same characteristics as the image data processing devices described in (3) to (5) and (10) to (14).

Embodiments of the present invention will be described below.
[Embodiment 1]
FIG. 1 is a diagram illustrating a configuration of an image data processing apparatus according to the first embodiment. The image data processing apparatus 10A according to the first embodiment includes a plurality of imaging units (two in the first embodiment) that can output image data obtained by imaging the same subject with a predetermined time difference in the same imaging range at different frame rates. The image pickup means control unit 11 controls the image pickup of the cameras 1 and 2, the image data recording device 13 that stores the respective image data picked up by the cameras 1 and 2, and the image data recording device 13. Image having means for inputting image data of cameras 1 and 2, means for generating combined image data obtained by combining the input image data so as to be temporally continuous, and means for outputting the generated combined image data And a data composition processing unit 12. Although the image data recording device 13 is shared by the cameras 1 and 2 in FIG. 1, it may be provided for each of the cameras 1 and 2.

  The cameras 1 and 2 are cameras that can capture moving images as digital data. For example, digital video cameras can be used. These cameras 1 and 2 do not have to be dedicated cameras prepared for the present invention, and general-purpose cameras can be used. In terms of performance, it is not necessary to have a particularly high performance. For example, if the high performance is expressed by a frame rate, it may be a frame rate of about 30 fps.

  These cameras 1 and 2 can image the same subject 3 and can reduce the parallax of the cameras 1 and 2 (preferably without parallax), that is, imaging within the same imaging range. It is desirable to install so that it becomes. It is also desirable that the optical axis and the scanning line are installed in parallel. Thus, it is desirable that the cameras 1 and 2 be installed so that the same image can be obtained when the subject 3 is simultaneously captured as a still image. In the present invention, the cameras 1 and 2 are assumed to have different frame rates of image data output from the cameras.

  In the first embodiment, the camera 2 is set with respect to the camera 1 by the user or the like at the start of imaging, that is, at the first imaging (imaging for obtaining image data output as the first frame image). Assume that the subject 3 is imaged with a time difference corresponding to the delay time. The setting of the time difference in the imaging at the start of imaging (hereinafter referred to as the first imaging) can be set by the imaging means control unit 11. The time difference to be set can be arbitrarily set within a predetermined range by the user.

FIG. 2 is a diagram showing the configuration of the imaging means control unit 11 used in the image data processing apparatus 10A.
As shown in FIG. 2, the imaging means control unit 11 includes a user setting information input unit 111 for inputting a delay time and a control command (such as an imaging start / end command) set by the user, and a user setting information input unit 111. Based on the input delay time, the delay time is set and the delay time can be stored, and the delay time set by the delay time setting unit 112 and the user setting information input unit 111 are input. A transmission unit 113 that transmits a control command to the cameras 1 and 2 is provided. In the first embodiment, as described above, in the first imaging, the camera 2 starts imaging with a time difference corresponding to the delay time set by the user with respect to the camera 1.

  Further, the delay time and the control command can be input by, for example, a set value input button or the like, or can be input from a remote controller or a personal computer (referred to as a PC) via a network or USB. is there.

  In each embodiment of the present invention, the delay time is represented by the number of frames. That is, if the frame rate is 30 fps, if the delay time is 1/30 seconds, it is 1 frame in terms of the number of frames, and if the delay time is 1/60 seconds, it is framed. In terms of numbers, this is 0.5 frames.

  Therefore, if 1 frame is set as the delay time, 1/30 second is set as the delay time, and if 0.5 frame is set as the delay time, the delay time is 1/60 second is set.

  FIG. 3 is a diagram showing the configuration of the image data composition processing unit 12 used in the image data processing apparatus 10A. As shown in FIG. 3, the image data composition processing unit 12 is input to the input unit 121 that inputs each image data output from the cameras 1 and 2 recorded in the image data recording device 13, and is input to the input unit 121. Each image data, that is, a composite image data generating unit 123 that uses each frame image of each image data captured by the cameras 1 and 2 output from the image data recording device 13 as temporally continuous composite image data, An output unit 124 that outputs the composite image data generated by the data generation unit 123 is provided. The input unit 121 includes a buffer 121a.

  FIG. 4 is a flowchart for explaining an image data processing procedure in the image data processing apparatus 10A. Prior to the operation of the image data processing apparatus 10A, first, the user inputs a delay time and a control command. Then, the imaging means control unit 11 sends an imaging start command to each of the cameras 1 and 2 (step S1).

  The camera 1 starts imaging by an imaging start command (step S2), and sends the captured image data to the image data recording device 13 (step S3). On the other hand, the camera 2 first determines whether or not it is the first imaging (step S4), and if it is the first imaging, determines whether or not the delay time set by the delay time setting unit 112 has passed ( Step S5) When the delay time elapses, imaging is started (step S6). If it is determined in step S4 that the first imaging is not performed, the operation in step S6 is performed directly. Image data captured by the camera 2 is also sent to the image data recording device 13 (step S3). Note that the delay time functions as a synchronization signal for the cameras 1 and 2 only in the first imaging.

  Then, the presence / absence of an imaging end command is determined (step S7). If there is no imaging end command, the camera 1 returns to step S2, and the camera 2 returns to step S4. If it is determined in step S7 that an imaging end command has been input, the composite image data generation unit 123 of the image data composition processing unit 12 captures the image data captured by the cameras 1 and 2 recorded in the image data recording device 13. Receive (step S8). The composite image data generation unit 123 combines the image data captured by the cameras 1 and 2 received from the image data recording device 13 so as to be temporally continuous (step S9), and outputs the composite image data (step S10). ).

By the way, in the present invention, cameras 1 and 2 having different frame rates are used. For example, assuming that the frame rate of the camera 1 is 30 fps and the frame rate of the camera 2 is 10 fps, if the imaging timings of the cameras 1 and 2 are synchronized with each other and the imaging starts simultaneously, the camera 1 , 2 output timing of the image data is as shown in FIG. As can be seen from FIG. 5, when the frame rate of the camera 1 is 30 fps and the frame rate of the camera 2 is 10 fps, three frames of the camera 1 correspond to one frame of the camera 2.
In FIG. 5, frames F1, F2,... Represent frame images in the image data output from the camera 1, and frames F1 ′, F2 ′,. Represents a frame image. In the specification, frame images F1, F2,... Are written, but in FIG. 5, “image” is omitted and frames F1, F2,. . The same applies to other drawings used in the following description.

  FIG. 6 is a time chart showing the input timing of image data from the cameras 1 and 2 in the image data processing apparatus 10A. In the first embodiment, the camera 2 starts imaging with respect to the camera 1 with a time difference corresponding to the delay time set by the user (here, a time difference of 0.3 frames) in the first imaging. I am going to do it. In Embodiment 1 (the same applies to other embodiments described later), it is assumed that the frame rates of the cameras 1 and 2 are 30 fps for the camera 1 and 10 fps for the camera 2.

  Thus, in the cameras 1 and 2 having different frame rates, when the camera 2 starts imaging with a delay of 0.3 frames from the camera 1, as shown in FIGS. Image data obtained by the first imaging of the camera 2 is output with a time difference of 0.3 frames from the camera 1. It should be noted that the delay time in the first imaging is not limited to 0.3 frames and can be arbitrarily set within a predetermined range.

  Further, since the subsequent imaging, that is, the imaging after the second time (imaging for obtaining the image data output as the second frame image) is not synchronized between the cameras 1 and 2, respectively. The imaging timing of the image data is arbitrary under a predetermined condition. For this reason, the output timing of each frame image obtained by the second and subsequent imaging is also arbitrary. It is assumed that the predetermined condition described above is that the camera 2 starts imaging with a delay within a time difference of 0 to 1 frame from the camera 1.

FIG. 7 is a time chart for explaining the image data composition processing in the image data processing apparatus 10A. FIG. 7 shows a case where 0.3 frame is set as the delay time for the delay time setting unit 112. FIGS. 7A and 7B are diagrams showing input timings of image data captured by the cameras 1 and 2, which are the same as FIGS. 6A and 6B.
Image data as shown in FIGS. 7A and 7B captured by the cameras 1 and 2 is recorded in the image data recording device 13 as shown in FIG.

  FIG. 8 is a diagram illustrating an example of a recording state of image data on the image data recording device 13. As shown in FIG. 8, the respective frame images in the order of imaging from the top of the storage area (in this case, camera 1 and camera 2 in this order) are the frame image F1 of camera 1, the frame image F1 ′ of camera 2, and the camera 1 respectively. The frame image F2, the frame image F3 of the camera 1, and so on are recorded. When the image data recording device 13 is provided corresponding to the cameras 1 and 2, each frame of image data input from the camera is sent to each image data recording device corresponding to the cameras 1 and 2. Each image is recorded in the order of imaging.

  Then, using the image data input from the cameras 1 and 2 recorded in the image data recording device 13, the composite image data generation unit 123 performs an image data composition process. In this image data synthesis process, since the frame rate of the camera 2 is 1/3 of that of the camera 1, as shown in FIG. 7C, the three frame images F1, F2,. A synthesis process is performed in which one frame image is inserted into the frame image, that is, the frame image of the camera 2 is inserted every three frame images. However, since the time length of each frame image of the cameras 1 and 2 after the synthesis process cannot be set, as shown in FIG. 7D, the image data is simply synthesized as 60 fps image data.

  As shown in FIG. 7D, in this case, the time length for each frame image of the composite image data is not taken into account, but is simply equal (0.5 frame interval if considered based on a frame rate of 30 fps). Each frame image is lined up. However, the frame image F1. F2. Image data in which the frame image of the camera 2 is inserted for every three frame images in F3,.

  In the example shown in FIG. 7D, the frame image F1, the frame image F1 ′, the frame image F2, the frame image F2, the frame image F3, the frame image F3, the frame image F4, the frame image F2 ′,. As described above, the frame images F2 and F3 of the camera 1 are each continuous twice. This is due to the temporal positional relationship between the frame images F2 and F3 of the camera 1 and the frame image F1 ′ of the camera 2. is there. By using composite image data as shown in FIG. 7D, image data with more natural motion can be obtained.

  That is, as is clear from FIGS. 7A and 7B, the frame image F1 'of the camera 2 is image data captured earlier in time than the frame images F2 and F3 of the camera 1. Therefore, if the frame image F1 ′ of the camera 2 is inserted between the frame image F2 and the frame image F3 of the camera 1 or the like, the synthesized image data obtained thereby is image data that causes unnatural motion in the moving image. This is because there is a possibility of becoming.

  Note that the image data in FIG. 7D can be output to a network, an image display device such as a projector, or a recording medium such as a hard disk, as described above. For example, when output to an image display device such as a projector, the image is displayed as an image having a frame rate of 60 fps.

  FIG. 9 is a flowchart for explaining the image data composition processing in more detail. In FIG. 9, n represents an index. When the frame rate of the camera 1 is 30 fps and the frame rate of the camera 2 is 10 fps as in this embodiment, the frame image of the camera 2 is inserted for every three frame images of the camera 1 as described above. Perform the action. Therefore, in this case, an operation is performed in which the image data captured by the camera 2 is inserted into the image data captured by the camera 1 at a timing when n = 3. Further, when the frame rate of the camera 1 is 30 fps and the frame rate of the camera 2 is 15 fps, image data captured by the camera 2 with a low frame rate at the timing when n = 2 is obtained with the camera 1 with a high frame rate. An operation of inserting the captured image data is performed.

  In FIG. 9, first, the index n is set to n = 3 (step S21). Then, the presence / absence of image data captured by the camera 1 is determined (step S22), and the input unit 121 receives the image data captured by the camera 1 from the image data recording device 13 (step S23).

  Here, it is determined whether n = 3 (step S24). In this case, since n = 3, the presence / absence of image data captured by the camera 2 is determined (step S25), and the image is captured by the camera 2. If there is still image data, the input unit 121 receives the image data captured by the camera 2 from the image data recording device 13 (step S26).

  Then, n = 0 is set (step S27). Subsequently, the composite image data generation unit 123 combines the image data captured by the camera 1 and the image data captured by the camera 2 so as to be temporally continuous (step S28), and outputs the composite image data. (Step S29). The process in step S28 corresponds to, for example, a process in which the frame image F1 'is inserted after the frame image F1 in FIG.

  Subsequently, n is incremented by 1 (step S30), the process returns to step S22, and steps S22, S23, and S24 are performed. At this time, since n = 1, it is determined whether n = 3 in step S24. As a result, since n = 3, the process of step S31 is performed. That is, the composite image data generation unit 123 duplicates the image data captured by the camera 1 and combines the image data captured by the camera 1 so as to be continuous twice in time (step S31). Is output (step S29). The process in step S31 corresponds to, for example, a process in which the frame image F2 in FIG.

  Subsequently, n is incremented by 1 (step S30), the process returns to step S22, and steps S22, S23, and S24 are performed. At this time, since n = 2, it is determined whether n = 3 in step S24. As a result, since n = 3 is not satisfied, the process of step S31 is also performed this time. In this case, the process of step S31 corresponds to, for example, a process of continuing the frame image F3 in FIG. 7D twice.

Subsequently, n is incremented by 1 (step S30), the process returns to step S22, and steps S22, S23, and S24 are performed. At this time, since n = 3, it is determined whether n = 3 in step S24. As a result, the process of step S25 is performed this time. If there is no image data captured by the camera 2 in step S25, the process ends. If there is image data captured by the camera 2, the process proceeds to step S26. Since the processing after step S26 has been described above, the description thereof will be omitted here. Of the processing after step 26, the processing according to step S28 is performed, for example, after the frame image F4 in FIG. This corresponds to the process of inserting the image F2 ′.
By repeating the above operation, composite image data as shown in FIG. 7D is generated.

  As described above, in the first embodiment, by using a plurality of general-purpose cameras capable of capturing a moving image (two in the first embodiment), a high frame rate can be realized without sacrificing resolution. High-quality image data can be generated on the time axis. That is, the composite image data generated by the first embodiment is image data that can reproduce a moving image with smooth motion. Further, even when the frame rates of the cameras 1 and 2 are different, it is possible to obtain composite image data in which the image data input from the cameras 1 and 2 are temporally continuous.

  As shown in FIG. 7D, since this composite image data has the same time length for each frame image, for example, when such composite image data is displayed on an image display device such as a projector, an image is displayed. Even if the display device is an image display device corresponding to the image data processing device of the present invention (hereinafter referred to as a corresponding display device), an image display device not corresponding to the image data processing device of the present invention (hereinafter referred to as non-corresponding display) Even a device) can be displayed corresponding to the time length of each frame image.

  In addition, using two cameras, frame images F1, F2,... Are output from one camera, and frame images F1 ′, F2 ′,. As an effect of this, there is also an advantage that the so-called “twice writing” is performed on the image display device side.

  In order to realize the first embodiment (the same applies to the other embodiments described below), it is necessary to perform misalignment correction, color correction, luminance correction, and the like of each image data input from the cameras 1 and 2. Is desirable.

  The positional deviation correction of each image data input from the cameras 1 and 2 can be performed by using the image data of the camera 1 as reference image data. This is because, for example, a specific area of the image data of the camera 1 is searched from the image data of the camera 2 by block matching or the like, the amount of deviation of the image data of the camera 2 with respect to the image data of the camera 1 is detected, and 2 is a process of matching the entire image data 2 with the image data of the camera 1.

  In addition, color correction of each image data input from the cameras 1 and 2 is performed by using a color component (reference component) for each corresponding pixel between the image data of the camera 1 and the image data of the camera 2 after the above-described positional deviation correction. For each of red (R), green (G), blue (B), etc., the average value of both pixel values is obtained. Then, the pixel value is adjusted for each of the image data of the camera 1 and the image data of the camera 2 so that the obtained average value is obtained.

  Similarly to the above-described color correction, the luminance correction of each image data input from the cameras 1 and 2 corresponds to each pixel between the image data of the camera 1 and the image data of the camera 2 after the above-described misalignment correction. The average value of the two luminance values is obtained. Then, the brightness value is adjusted for each of the image data of the camera 1 and the image data of the camera 2 so that the obtained average value is obtained.

  The timing for performing each of these corrections may be before the start of imaging by the cameras 1 and 2, during the image data synthesis process by the synthesized image data generation unit 123 of the image data synthesis processing unit 12, after the imaging by the cameras 1 and 2 is completed It is done.

  For example, when the imaging is performed before the start of imaging by the cameras 1 and 2, the imaging is attempted using the test patterns and the like by the cameras 1 and 2 before the imaging is started. Here, correction is performed based on the captured image data. The correction parameters at this time are stored and applied at the time of image data synthesis.

  Further, when it is performed during the image data synthesis process, the image data of the cameras 1 and 2 are sequentially executed when the synthesized image data generation unit 123 synthesizes the image data. This can more accurately reflect environmental changes (such as changes in brightness) at the time of imaging, rather than predetermining correction parameters.

In addition, when the image capturing is performed after the image capturing is completed, the storage type including the first embodiment is particularly effective. That is, when there is a time margin until the final image data is output, correction can be performed with higher accuracy.
Note that various corrections such as misalignment correction, color correction, and luminance correction of the image data input from the cameras 1 and 2 are not necessarily performed, and may be performed as necessary.

[Embodiment 2]
In the first embodiment described above, in the first imaging, the camera 2 delays imaging with a time difference corresponding to the delay time set by the user with respect to the camera 1, but the imaging timing of the cameras 1 and 2 is set by the user. It is also possible to make it arbitrary from the first time without doing.

  The second embodiment is a case where the imaging timing of the cameras 1 and 2 is arbitrary from the first time in the first embodiment. In this case, the imaging timing of the cameras 1 and 2 may be the same timing, or the camera 1 may be earlier than the camera 2. Conversely, the camera 2 may be earlier than the camera 1, but in the second embodiment, the predetermined condition described in the first embodiment, that is, the camera 2 has a time difference of 0 to 1 frame with respect to the camera 1. It is assumed that imaging is started with a delay at an arbitrary timing within the range.

  Since the configuration of the image data processing apparatus according to the second embodiment can use the image data processing apparatus 10A according to the first embodiment as it is, the illustration as the image data processing apparatus according to the second embodiment is omitted. For convenience of explanation, the image data processing apparatus according to the second embodiment will be described as the image data processing apparatus 10B with the reference numeral 10B. Further, since the image pickup means control unit 11 and the image data composition processing unit 12 constituting the image data processing apparatus 10B can use those described in the first embodiment, their illustration is also omitted.

  FIG. 10 is a flowchart for explaining an image data processing procedure of the image data processing apparatus 10B according to the second embodiment. Prior to the start of processing of the image data processing apparatus 10B, first, the user inputs an imaging start command. Then, the imaging means control unit 11 sends an imaging start command to each of the cameras 1 and 2 (step S41). Thereby, the camera 1 starts imaging (step S42), and sends the captured image data to the image data recording device 13 (step S43).

  On the other hand, the camera 2 also starts imaging at a predetermined timing (step S44), and sends the captured image data to the image data recording device 13 (step S43). Note that the timing of the start of imaging by the camera 2 is performed based on the predetermined condition described above.

  The processes in steps S45, S46, S47, and S48 in FIG. 10 are the same as the processes in steps S7, S8, S9, and S10 in FIG.

  According to the second embodiment, the same effects as in the first embodiment can be obtained. In the second embodiment, each camera can be arbitrarily set even when the user does not set the imaging timing of the camera 2 with respect to the camera 1 at the start of imaging. Imaging can be performed at the timing of.

[Embodiment 3]
The image data processing apparatus according to the third embodiment sets a delay time according to the content, generates intermediate image data based on the set delay time, and uses the intermediate image data to generate composite image data. Is generated. In addition, although this Embodiment 3 is applicable to both Embodiment 1 and Embodiment 2, of course, the example applied to Embodiment 1 is demonstrated here. In the third embodiment, as in the first and second embodiments, a case where two cameras 1 and 2 are used as imaging means will be described.

  Since the configuration of the image data processing apparatus according to the third embodiment is the same as that of the image data processing apparatus 10A shown in FIG. 1, the illustration as the image data processing apparatus according to the third embodiment is omitted. For convenience of explanation, the image data processing apparatus according to the third embodiment will be described as the image data processing apparatus 10C with the reference numeral 10C.

  FIG. 11 is a diagram showing the configuration of the image data composition processing unit 12 used in the image data processing apparatus 10C. The image data composition processing unit 12 in FIG. 11 is different from that in FIG. 3 used in the description of the first embodiment in that it includes a content analysis unit 125.

That is, in the image data composition processing unit 12 used in the image data processing device 10C, the content analysis unit 125 performs content analysis between the image data of the cameras 1 and 2 recorded in the image data recording device 13, and the content A function of setting a delay time based on the analysis result (this delay time will be described later), generating intermediate image data based on the set delay time, and generating the generated intermediate image data and the plurality of imaging means An image data synthesizing processing unit 12 having a function of synthesizing image data input from an imaging means (camera 1 in the third embodiment) so as to be temporally continuous to generate synthesized image data. .
Further, since the imaging means controller 11 used in the image data processing apparatus 10C according to the third embodiment can be the same as that shown in FIG.

  FIG. 12 is a flowchart for explaining an image data processing procedure in the image data processing apparatus 10C. In the flowchart shown in FIG. 12, steps S51 to S57 are the same processing as steps S1 to S7 in the flowchart of FIG. 4, so here, there is an imaging end command in the determination of the presence or absence of the imaging end command in step S57. A processing procedure after the determination is made will be described.

  If it is determined in step S57 that there is an imaging end command, the content analysis unit 125 receives the image data of the cameras 1 and 2 recorded in the image data recording device 13 and performs content analysis (step S58). The composite image data generation unit 123 generates intermediate image data based on the content analysis result, combines the generated intermediate image data with the image data of the camera 1 (step S59), and outputs the composite image data (step S59). S60).

  FIG. 13 is a diagram for explaining the correspondence between the frame images of the cameras 1 and 2 when content analysis is performed. The input timing of each frame image of the cameras 1 and 2 in FIG. 13 shows the same input timing example as in FIGS. 7A and 7B used in the description of the first embodiment.

  As shown in FIGS. 13A to 13D, first, content analysis is performed on the frame image F1 of the camera 1 and the frame image F1 ′ of the camera 2, and then the frame image F2 of the camera 1 and the camera 2 are analyzed. Content analysis is performed on the frame image F1 ′, and then content analysis is performed on the frame image F3 of the camera 1 and the frame image F1 ′ of the camera 2, followed by the frame image F4 of the camera 1 and the frame image of the camera 2. Content analysis is performed in the order of content analysis with F2 ′.

  In the example of FIG. 13, since the frame rate of the camera 1 is 30 fps and the frame rate of the camera 2 is 10 fps, the correspondence between the frame images to be analyzed for content is that of the camera 2 for every three frame images of the camera 1. The frame image is updated. When the frame rate of the camera 1 is 30 fps and the frame rate of the camera 2 is 15 fps, the correspondence between the frame images for content analysis is updated every two frame images of the camera 1. Will be.

  Next, the operation of the content analysis unit 125 will be described. In the case of the third embodiment, the content analysis unit 125 performs content analysis between corresponding frame images of the camera 1 and the camera 2 as shown in FIGS. A function for detecting a change in the overall movement and / or a change in the brightness of the entire screen (in the third embodiment, both a change in the movement of the entire screen and a change in the brightness of the entire screen are detected) and its detection It has a function of setting a delay time (number of frames) according to the result, and a function of sending the set delay time (number of frames) to the composite image data generation unit 123.

  As a result of detecting a change in motion among changes in motion and brightness, the delay time is shortened when the change in motion is large, and the delay time is increased when the change in motion is small. . This is because if the delay time is set to be large in a scene where motion is intense, the visibility of the moving image is deteriorated and the afterimage feeling is further increased. On the other hand, in a slowly moving scene, even if the delay time is increased, the visibility of the moving image is not greatly affected.

  On the other hand, the delay time is shortened when the brightness change is large as a result of detecting the brightness change among the motion change and brightness change detection, and the delay time when the brightness change is small. Increase This is because if the change in brightness is large, the scene is likely to change, and if the delay time is set to be large in such a scene, the visibility of the video will deteriorate and the afterimage will become larger. . Conversely, when the change in brightness is small, the change in the scene is often small, and in this case, even if the delay time is increased, the visibility of the moving image is not greatly affected.

  Here, detection of a change in motion (referred to as motion detection) will be described. Motion detection can be performed by various methods. Here, an example in which a motion vector between frame images is obtained using a general block matching method will be described.

  FIG. 14 is a diagram for explaining a block matching method used in motion detection. The block matching method is a method for detecting motion in units of blocks, and uses a plurality of frame images including a frame image of interest at present. In the third embodiment, the currently focused frame image (referred to as t frame) and the immediately preceding frame image (referred to as t−1 frame) are used. For example, in the example of FIG. 13A, the t frame corresponds to the frame image F1 'of the camera 2, and the t-1 frame corresponds to the frame image F1 of the camera 1.

（１）式において、ＬはブロックＡ１のサイズ、ｘ，ｙはブロックＡ１のｘ軸方向及びｙ軸方向における移動方向、ｉ，ｊはブロックＡ１の移動量、ｔはフレームを表す。動きベクトルを求める最も簡単な方法は、（ｉ，ｊ）のすべての組み合わせについて（１）式を計算し、Ｄが最小となる（ｉ，ｊ）の組み合わせを求めるという方法である。 As shown in FIG. 14, the block A1 (assuming that the block size is L × L) in the t frame is moved in the t−1 frame, and the block having the closest data value to the block A1 in the t frame is changed to the t−1 frame. The shift width between the place and the original place (position indicated by a broken line) is used as a motion vector. That is, in the following formula, (i, j) when D is minimum is a motion vector.

In equation (1), L is the size of the block A1, x and y are the movement directions of the block A1 in the x-axis direction and the y-axis direction, i and j are the movement amounts of the block A1, and t is the frame. The simplest method for obtaining the motion vector is to calculate the equation (1) for all the combinations of (i, j) and obtain the combination of (i, j) that minimizes D.

と表すことができる。この（２）式によって得られた値がｔフレームにおける画面全体の動き量となる。 In this way, the motion vector is obtained for each block in the t frame, and the motion amount of the entire screen is calculated from the obtained motion vector for each block by the following equation. That is, if the motion vector is V and n motion vectors are obtained, the motion amount M of the entire screen is

It can be expressed as. The value obtained by equation (2) is the amount of motion of the entire screen in t frame.

  As described above, the motion amount may be calculated for the entire screen of each frame image. However, in order to reduce the calculation processing, the motion amount may be calculated only for the center area of the screen. . In many cases, this is because the viewer pays attention to the center of the screen.

  As described above, when the motion amount of t frame (amount indicating how much the frame moves with respect to t-1 frame) is detected, the delay time is set according to the detected motion amount. The delay time setting according to the amount of motion will be described below.

  FIG. 15 is a diagram illustrating a setting example of the delay time with respect to the motion amount. In this case, the motion amount is normalized to a value in the range of 0 to 220. As shown in FIG. 15, when the amount of motion is 0, the number of frames as the delay time is 1 (1 frame), and when the amount of motion is 220, the number of frames as the delay time is 0 (0 frames). ). When the motion amount is an intermediate value of 110, the number of frames as the delay time is 0.5 (0.5 frames). In this way, the delay time (number of frames) according to the amount of motion can be set in advance.

The delay time with respect to the motion amount obtained in this way can be prepared as a table from which the delay time can be obtained from the motion amount.
FIG. 16 is a diagram illustrating an example of a table capable of acquiring a delay time (the number of frames) from the motion amount. As shown in FIG. 16, for example, the delay time in the range of motion amount “0-20” is “1 frame”, the delay time in the range of motion amount “21-40” is “0.9 frame”, motion The delay time in the range of “201 to 220” has contents such as “0 frame”. If a motion amount is obtained by creating such a table, the delay with respect to the motion amount can be easily obtained. You can set the time.

  The example of setting the delay time according to the amount of motion obtained by performing motion detection has been described above. Next, an example of setting the delay time according to the detection result of the change in brightness will be described. Note that the detection of a change in brightness is herein a detection of a luminance difference.

  First, the luminance value of the pixel of each frame image is acquired for each frame image. Then, an average value of the acquired luminance values is obtained, and the average value of the obtained luminance values is assumed to be the overall brightness of the frame image. In this way, when the average value of the luminance values is obtained in each frame image, the difference between the average values of the luminance values between the frame images is used as the brightness change amount, and the delay time is determined based on the brightness change amount. Set the number of frames.

  FIG. 17 is a diagram illustrating a setting example of the delay time with respect to the amount of change in brightness. In this case, the brightness change amount is normalized to a value in the range of 0 to 220. As shown in FIG. 17, when the brightness change amount is 0, the number of frames as the delay time is 1 (1 frame), and when the brightness change amount is 220, the number of frames as the delay time. Is 0 (0 frame). When the brightness change amount is 110 as an intermediate value, the number of frames as the delay time is set to 0.5 (0.5 frames). Thus, the number of frames as the delay time according to the brightness change amount can be set in advance.

The delay time with respect to the brightness change amount obtained in this way can be prepared as a table in which the delay time can be obtained from the brightness change amount, similarly to the motion amount.
FIG. 18 is a diagram illustrating an example of a table capable of acquiring the delay time (number of frames) from the brightness change amount. As shown in FIG. 18, for example, the delay time in the range where the brightness change amount is “0-20” is “1 frame”, and the delay time in the range where the brightness change amount is “21-40” is “0.9”. The delay time in the range of “frame” and brightness change amount “201 to 220” has contents such as “0 frame”, and the brightness change amount is obtained by creating such a table. If so, it is possible to easily set a delay time for the brightness change amount.

  As described above, it is possible to set the delay time according to the motion amount and the delay time according to the brightness change amount. Although it is possible to use only one of the delay time according to the motion amount and the delay time according to the brightness change amount, each embodiment of the present invention is based on both the motion amount and the brightness change amount. The delay time is set. The delay time set based on both the motion amount and the brightness change amount will be referred to as “a delay time considering the motion amount and the brightness change amount”.

  The delay time considering the motion amount and the brightness change amount is an average value of the delay time (number of frames) corresponding to the motion amount and the delay time (number of frames) corresponding to the brightness change amount. For example, if the delay time (number of frames) according to the amount of motion is 0.3 frames and the delay time (number of frames) according to the amount of change in brightness is 0.7 frames, the average of both is obtained. , 0.5 frame is set as a delay time considering the motion amount and the brightness change amount.

  If the average value of the delay time (number of frames) according to the amount of motion and the delay time (number of frames) according to the amount of change in brightness is less than the settable accuracy, rounding, rounding down, Do one of the round ups.

  As an example, when the settable accuracy is up to the first decimal place (in units of 0.1 frame) as in the first embodiment, for example, “the obtained average value is a value less than the second decimal place. In some cases, it can be rounded off. As a result, when the average value is obtained as a value less than the second decimal place, such as 0.52, the delay time is 0.5 frame.

  FIG. 19 is a diagram illustrating an example of a delay time (number of frames) set in consideration of the motion amount and the brightness change amount and intermediate image data generated based on the delay time. As shown in FIG. 19, for example, when the frame images subjected to content analysis are the frame image F1 of the camera 1 and the frame image F1 ′ of the camera 2, the delay time obtained between the frame image F1 and the frame image F1 ′ The (delay time considering the amount of motion and the amount of change in brightness) is 0.3 frames, and the intermediate image data generated thereby is 1.3 frames.

  In FIG. 19, the intermediate image data of 1.3 frames, 2.5 frames, 3.4 frames,... Are stored in the frame numbers of the frame images F1, F2, F3,. This is expressed as a value (frame number + delay time) obtained by adding the delay time obtained by analysis. For example, when the delay time obtained between the frame image F1 of the camera 1 and the frame image F1 ′ of the camera 2 is 0.3 frame, 1 + 0.3 = 1.3. Three frames of intermediate image data are generated.

  FIG. 20 is a time chart for explaining the image data composition processing in the image data processing apparatus 10C. FIGS. 20A and 20B are diagrams showing the input timing of image data captured by the cameras 1 and 2, which are the same as FIGS. 7A and 7B used in the description of the first embodiment. Is the same.

  Assume that intermediate image data as shown in FIG. 19 is obtained as a result of content analysis of the image data input from the cameras 1 and 2 shown in FIGS. When the intermediate image data and the image data input from the camera 1 are combined, combined image data as shown in FIG. 20C is generated.

  In addition, although the imaging timing of each frame image of the camera 1 is not necessarily at equal intervals, the synthesis is performed on the assumption that the imaging timing of the camera 1 is at equal intervals. The frame rate at the time of synthesis is based on the frame rate of the camera 1 (30 fps in this example). In this case, the time length of each frame of the composite image data is adjusted in the data format of FIG. 20C so that the frame rate is apparently 60 fps.

  In the composite image data shown in FIG. 20C, 2.3 frames of intermediate image data are inserted at a position earlier in time than the frame image F2 of the second frame of the camera 1. Similarly, the intermediate image data 3.5 frame is inserted at a position earlier in time than the third frame image F3 of the camera 1. In this way, the intermediate image data 2.3 frame and the frame image F2 of the second frame of the camera 1, and the intermediate image data 3.5 frame and the frame image F3 of the third frame of the camera 1, respectively, It looks as if the order is reversed.

  This is due to the temporal positional relationship between the frame image of the camera 1 and the frame image of the camera 2 used for content analysis and generation of intermediate image data. For example, when the frame image F2 of the camera 1 and the frame image F1 ′ of the camera 2 used for content analysis and generation of intermediate image data are compared in terms of time, the frame image F1 ′ is more temporal than the frame image F2. It is very early image data. Similarly, when the frame image F3 of the camera 1 and the frame image F1 ′ of the camera 2 used for content analysis and generation of intermediate image data are compared in time position, the frame image F1 ′ has a longer time than the frame image F3. Image data.

  Since the cameras 1 and 2 are capturing images at arbitrary timings (asynchronous), the exact temporal relationship between them, that is, which camera captured earlier in each frame image is specified. In this case, it is assumed that the camera 2 takes an image later than the camera 1 under the predetermined condition described above.

  Therefore, the temporal positional relationship between each frame image of the camera 1 and each frame image of the camera 2 is temporally earlier than that of the frame image F1 ′ of the camera 2 because the frame image F1 of the camera 1 is temporally related. And the intermediate image data 1.3 frame generated by the frame image F1 ′ of the camera 2 is inserted at a position later in time than the frame image F1 of the camera 1.

  Further, since the frame image F2 of the camera 1 is later in time than the frame image F1 ′ of the camera 2, the intermediate image data 2.3 frame generated by the frame image F2 of the camera 1 and the frame image F1 ′ of the camera 2 is These are inserted at positions earlier in time than the frame image F2 of the camera 1.

  Similarly, since the frame image F3 of the camera 1 is later in time than the frame image F1 ′ of the camera 2, the intermediate image data 3.5 frames generated by the frame image F3 of the camera 1 and the frame image F1 ′ of the camera 2 are used. Is inserted at a position earlier in time than the frame image F3 of the camera 1.

FIG. 21 is a flowchart for explaining content analysis processing performed by the content analysis unit 125 of the image data composition processing unit 12 and image data composition processing performed by the composite image data generation unit 123.
In FIG. 21, first, the index n is set to a predetermined value (in this case, n = 3) (step S61). This index n is the same as that described in the flowchart of FIG. 9, and in each embodiment of the present invention, the frame rate of the camera 1 is 30 fps and the frame rate of the camera 2 is 10 fps. n = 3 is set.

  First, after setting n = 3 in step S61, it is determined whether there is image data captured by the camera 1 (step S62), and the input unit 121 captures image data captured by the camera 1 from the image data recording device 13. Is received (step S63).

  Here, it is determined whether or not n = 3 (step S64). At this time, since n = 3, the presence or absence of image data captured by the camera 2 is determined (step S65), and the image is captured by the camera 2. If there is any image data, the input unit receives the image data captured by the camera 2 (step S66).

  Then, n = 0 is set (step S67), and the content analysis unit 125 performs content analysis between the image data captured by the camera 1 and the image data captured by the camera 2 (step S68). Subsequently, the composite image data generation unit 123 generates intermediate image data based on the content analysis result, and combines the intermediate image data and the image data captured by the camera 1 so as to be temporally continuous (step S69), the composite image data is output (step S70). The process of step S69 corresponds to the process of inserting 1.3 frames of intermediate image data after the frame image F1 in FIG.

  Subsequently, n is incremented by 1 (step S71), the process returns to step S62, and steps S62, S63, and S64 are performed. At this time, since n = 1, it is determined whether n = 3 in step S64. As a result, since n = 3, the process of step S68 is performed.

  That is, in this case, the content analysis unit 125 performs content analysis between the image data captured by the camera 1 (frame image F2) and the image data captured by the camera 2 (frame image F1 ′). The composite image data generation unit 123 generates intermediate image data using the analysis result, combines the intermediate image data and the image data captured by the camera 1 so as to be temporally continuous, and outputs the composite image data. (Steps S68, S69, S70). The process in step S69 in this case corresponds to the process of inserting 2.3 frames of intermediate image data before the frame image F2 in FIG.

  Subsequently, n is incremented by 1 (step S71), the process returns to step S62, and steps S62, S63, and S64 are performed. At this time, since n = 2, it is determined whether or not n = 3 in step S64. As a result, since n = 3, the process of step S68 is performed. In this case, the content analysis unit 125 performs content analysis between the image data captured by the camera 1 (frame image F3) and the image data captured by the camera 2 (frame image F1 '). Then, the composite image data generation unit 123 generates intermediate image data using the content analysis result, combines the intermediate image data and the image data captured by the camera 1 so as to be temporally continuous, and generates a composite image. Data is output (steps S68, S69, S70). The process in step S69 in this case corresponds to the process of inserting 3.5 frames of intermediate image data before the frame image F3 in FIG.

  Subsequently, n is incremented by 1 (step S71), the process returns to step S62, and steps S62, S63, and S64 are performed. At this time, since n = 3, the process of step S65 is performed. That is, the presence / absence of image data captured by the camera 2 is determined (step S65). If there is no image data captured by the camera 2, the process ends. If there is image data captured by the camera 2, the input unit Receives the image data captured by the camera 2 (step S66).

Then, n = 0 is set (step S67), and the image data captured by the content analysis unit 125 with the camera 1 (in this case, the frame image F4) and the image data captured with the camera 2 (in this case, the frame image F2 ′). Content analysis is performed between the two (step S68). Subsequently, the composite image data generation unit 123 generates intermediate image data based on the content analysis result, and combines the intermediate image data and the image data captured by the camera 1 so as to be temporally continuous (step In step S69, the composite image data is output (step S70). The process of step S69 corresponds to the process of inserting 4.2 frames of intermediate image data after the frame image F4 in FIG.
By repeating the operation as described above, for example, composite image data as shown in FIG. 20C is output from the composite image data generation unit 123.

Incidentally, the composite image data as shown in FIG. 20 is image data in which the time lengths of the respective frame images are not uniform, that is, image data in which the time intervals of the respective frame images are unequal.
The composite image data in FIG. 20C apparently has a frame rate of 60 fps, but is not a general image data format because the time length of each frame image is not equal.

  When the composite image data as shown in FIG. 20C is displayed on an image display device such as a projector, for example, if the image display device is a compatible display device, the time of the frame image shown in FIG. Display corresponding to the length becomes possible. However, when the image display device is a non-compatible display device, display is performed as normal 60 fps image data.

  FIG. 22 is a diagram illustrating an example of the data format of the composite image data output from the composite image data generation unit 123. As shown in FIG. 22, the composite image data is roughly divided into an identifier for distinguishing it from image data of other formats, a header area where data size is written, and an image data writing area where actual image data is written. Have.

  In the header area, the total number of frames, the time length of each frame image, the screen size (width and height), and the like are written. In the image data writing area, image data for each frame image is written as composite image data.

  If the image display device that receives this data format is the corresponding display device of the present invention, data can be displayed based on the data format of FIG. 22, but if it is a non-compatible display device, The data format of FIG. 22 cannot be read.

  Therefore, the composite image data written in the image data writing area is data in the AVI (Audio Video Interleaving) format generally used, and “total number of frames” and “time length of each frame image” in the header area of FIG. Etc. can also be output as separate data.

  As described above, the “total number of frames”, the “time length of each frame image”, etc. in the header area of FIG. 22 are output as separate data. Both the described data such as “number” and “display time of each frame image” are read, and processing according to the contents is performed. On the other hand, the incompatible display device reads only the data in the AV1 format, and performs processing according to the content. As a result, the corresponding display device can display corresponding to the time length of each frame image as shown in FIG. 20C, and the non-compatible display device can display as normal 60 fps image data. .

  As described above, in the third embodiment, the same effect as in the first embodiment can be obtained, and a delay time is set according to the content, and intermediate image data is generated based on the set delay time. The intermediate image data and the image data captured by the camera 1 are synthesized so as to be temporally continuous to generate synthesized image data. The composite image data generated in this way can be used to reproduce a smooth video that follows, for example, scenes with intense movement, scenes with large changes in brightness, scenes with slow movement, or scenes with small changes in brightness. Image data.

In addition, in the case of the third embodiment, the content analysis between the image data captured by each camera associates the frame image of the camera 1 and the frame image of the camera 2 as shown in FIGS. Therefore, it is possible to generate appropriate intermediate image data corresponding to the difference in imaging timing between the camera 1 and the camera 2. The composite image data generated using such intermediate image data can be image data capable of reproducing a more natural moving image.
Of course, as described above, the third embodiment can also be applied to the second embodiment.

[Embodiment 4]
In Embodiment 3 described above, in the first imaging, the camera 1 delays imaging with a time difference corresponding to the delay time set by the user with respect to the camera 2, but the imaging timing of the cameras 1 and 2 is set by the user. It is also possible to make it arbitrary from the first time without doing.

  The fourth embodiment is a case where the imaging timing of the cameras 1 and 2 is arbitrary from the first time in the third embodiment. In this case, the imaging timing of the cameras 1 and 2 may be the same timing, or the camera 1 may be earlier than the camera 2. Conversely, the camera 2 may be earlier than the camera 1, but in the fourth embodiment, the predetermined condition described above, that is, the camera 2 is arbitrary within a time difference range of 0 to 1 frame with respect to the camera 1. It is assumed that imaging is started with a delay at the timing.

Since the configuration of the image data processing apparatus according to the fourth embodiment can use the image data processing apparatus according to the third embodiment as it is, the illustration as the image data processing apparatus according to the fourth embodiment is omitted.
For convenience of explanation, the image data processing apparatus according to the fourth embodiment will be described as the image data processing apparatus 10D with the reference numeral 10D. Further, since the imaging means control unit 11 and the image data composition processing unit 12 constituting the image data processing apparatus 10D can use those described in the third embodiment, their illustration is also omitted.

  FIG. 23 is a flowchart illustrating an image data processing procedure of the image data processing apparatus 10D according to the fourth embodiment. Prior to the start of processing of the image data processing apparatus 10D, first, an imaging start command is input by the user. Then, the imaging means control unit 11 sends an imaging start command to each of the cameras 1 and 2 (step S81). Thereby, the camera 1 starts imaging (step S82), and sends the captured image data to the content analysis unit 125 (step S83).

On the other hand, the camera 2 also starts imaging at a predetermined timing (step S84), and sends the captured image data to the content analysis unit 125 (step S83). Note that the timing of the start of imaging by the camera 2 is performed based on the predetermined condition described above.
The processes in steps S85 to S88 in FIG. 23 are the same as the processes in steps S57 to S60 in FIG.

  According to the fourth embodiment, the same effects as in the third embodiment can be obtained. In the fourth embodiment, each camera can be arbitrarily set even if the user does not set the imaging timing of the camera 2 with respect to the camera 1 at the start of imaging. Imaging can be performed at the timing of.

[Embodiment 5]
An image display system can be configured by using the image data processing devices 10A to 10D according to the first to fourth embodiments described so far. Embodiment 5 describes this image display system. Of course, the image display system can be configured using any of the image data processing devices 10A to 10D of the first to fourth embodiments, but here, the image according to the first embodiment is used. An image display system using the data processing apparatus 10A will be described.

  FIG. 24 is a diagram illustrating a configuration of an image display system according to the fifth embodiment. As schematically shown in FIG. 24, the image display system according to the fifth embodiment includes cameras 1 and 2, an image data processing device 10A, an image display device 20, and an information processing device 30 such as a PC. Have.

  In such an image display system, command input such as delay time setting and imaging start / end can be performed on the information processing apparatus 30 by operating a mouse or a keyboard. Then, the composite image data (for example, see FIG. 7D) output from the image data processing device 10A is sent to the image display device 20 and displayed.

  Note that the composite image data shown in FIG. 7D is image data having an apparent frame rate of 60 fps, and the time length of each frame image being equal. When such composite image data is displayed on the image display device 20, it can be displayed as 60 fps image data regardless of whether or not the image display device 20 is a compatible display device.

  By constructing an image display system as shown in FIG. 24, the image data processing apparatus 10A side can realize a high frame rate without sacrificing resolution, and output high-quality image data on the time axis. Can do.

  FIG. 25 is a diagram showing a modification of the image display system shown in FIG. The image display system shown in FIG. 25 uses an information processing apparatus 30 such as a PC as the image data processing apparatus 10A. Therefore, the information processing apparatus 30 includes software (image data synthesis processing program) that can operate as the image data synthesis processing unit 12 and software (imaging unit control program) that can operate as the imaging unit control unit 11. ) Is incorporated. Note that the delay time and the control command can be input on the information processing apparatus 30 by operating a mouse or a keyboard.

  In the image display system shown in FIG. 25, the same effect as that of the image display system in FIG. 24 can be obtained. Further, by adopting the configuration as shown in FIG. 25, the entire image display system can be further simplified. The image display system of the present invention may use not only the image data processing device 10A according to the first embodiment but also the image data processing devices 10B, 10C, and 10D according to the second to fourth embodiments. It is.

  When the image data processing apparatus 10C according to the third embodiment is used as the image data processing apparatus, the image data processing apparatus 10C receives composite image data, that is, each frame image as shown in FIG. Composite image data having unequal time lengths is output. When such composite image data is displayed on the image display device 20, for example, if the image display device 20 is a corresponding display device, display corresponding to the time length of each frame image is possible. When 20 is a non-compliant display device, display is performed as normal 60 fps image data.

  As described above, when the image data processing device 10C is used as the image data processing device, intermediate image data based on the delay time set in accordance with the content content is generated, and the generated intermediate image data and the camera 1 are input. Since the processed image data is temporally continuous, it is possible to generate composite image data capable of reproducing a moving image with smoother motion. Thereby, the image display device 20 can display high-quality image data.

  The present invention is not limited to the above-described embodiment, and various modifications can be made without departing from the gist of the present invention. For example, as a method of installing the cameras 1 and 2, as shown in FIG. 26A, the cameras 1 and 2 are installed so that their optical axes are orthogonal to each other, and the subject 3 is placed using the half mirror 4. The installation may be such that imaging is performed. Needless to say, the number of cameras may be three or more. For example, as shown in FIG. 26B, an example of providing a plurality of half mirrors 4 is conceivable. .

  In addition, each of the image data processing devices 10A, 10B, 10C, and 10D according to the first to fourth embodiments can be configured as a single device including the imaging unit, the imaging unit control unit 11, and the image data synthesis processing unit 12. is there.

  FIG. 27 is a diagram illustrating an example in which the image data processing apparatus is configured as a single apparatus including an imaging unit, an imaging unit control unit, and an image data synthesis processing unit. The image data processing apparatus shown in FIG. 27 corresponds to the image data processing apparatus 10A (see FIG. 1) according to the first embodiment, and includes two CCDs 5a and 5b that function as two cameras 1 and 2. The CCD drivers 6a and 6b, the lens 7, and the half mirror 4 provided corresponding to the CCDs 5a and 5b, the image pickup means controller 11 and the image data composition processor 12 shown in FIG.

  By configuring the image data processing device as shown in FIG. 27, the overall configuration can be simplified. It should be noted that such a single image data processing apparatus is configured not only for the image data processing apparatus 10A according to the first embodiment but also for the image data processing apparatuses 10B, 10C, and 10D according to the second to fourth embodiments. Can be similarly implemented.

  In each of the above-described embodiments, the image data recording device 13 for recording image data is provided. However, an input unit for inputting image data output from each camera is provided for each camera, and each of these input units is provided. If a buffer having a storage capacity necessary for storing image data necessary for content analysis or the like is provided, the image data recording device 13 does not necessarily have to be prepared separately.

  Also, the present invention can create an image data processing program in which the image data processing procedure for realizing the present invention described above is described, and the image data processing program can be recorded on various recording media. Therefore, the present invention also includes a recording medium on which the image data processing program is recorded. Further, the image data processing program may be obtained from a network.

1 is a diagram illustrating a configuration of an image data processing apparatus 10A according to a first embodiment. The figure which shows the structure of the imaging means control part 11 used for 10 A of image data processing apparatuses. The figure which shows the structure of the image data composition process part 12 used for 10A of image data processing apparatuses. The flowchart explaining the image data processing procedure of 10 A of image data processing apparatuses. 4 is a time chart showing the output timing of image data output from the cameras 1 and 2 when the imaging timings of the cameras 1 and 2 are synchronized with each other and imaging is started at the same time. The time chart which shows the input timing of the image data of the cameras 1 and 2 in 10 A of image data processing apparatuses. The time chart explaining the image data synthetic | combination process in 10 A of image data processing apparatuses. FIG. 4 is a diagram illustrating an example of a recording state of image data on the image data recording device 13. The flowchart explaining an image data synthetic | combination process in detail. 9 is a flowchart for explaining an image data processing procedure of the image data processing apparatus 10B according to the second embodiment. The figure which shows the structure of the image data synthetic | combination process part 12 used for 10C of image data processing apparatuses which concern on Embodiment 3. FIG. The flowchart explaining the image data processing procedure in 10 C of image data processing apparatuses. The figure explaining the correspondence of each frame image of the cameras 1 and 2 at the time of performing a content analysis. The figure explaining the block matching method used in a motion detection. The figure which shows the example of a setting of the delay time with respect to a motion amount. The figure which shows an example of the table which can acquire delay time (frame number) from motion amount. The figure which shows the example of the setting of the delay time with respect to the brightness variation The figure which shows an example of the table which can acquire delay time (frame number) from brightness variation. The figure which shows an example of the intermediate | middle image data produced | generated based on the delay time (frame number) which considered the amount of motion and the amount of brightness changes, and this delay time. The time chart explaining the image data synthetic | combination process in 10 C of image data processing apparatuses. 14 is a flowchart for explaining content analysis processing performed by a content analysis unit 125 of an image data composition processing unit 12 and image data composition processing performed by a composite image data generation unit 123 used in an image data processing apparatus 10C according to the third embodiment. The figure which shows an example of the data format of the composite image data output from the composite image data generation part 123 of 10C of image data processing apparatuses. 10 is a flowchart for explaining an image data processing procedure of the image data processing apparatus 10D according to the fourth embodiment. FIG. 10 is a diagram illustrating a configuration of an image display system according to a fifth embodiment. The figure which shows the modification of the image display system shown in FIG. The figure explaining the example of the installation method of the cameras 1 and 2. FIG. The figure which shows the example which comprised the image data processing apparatus as a single-piece | unit apparatus containing an imaging means, an imaging means control part, and an image data composition process part.

Explanation of symbols

  DESCRIPTION OF SYMBOLS 1, 2 ... Camera, 3 ... Subject, 10A, 10B, 10C, 10D ... Image data processing apparatus, 11 ... Imaging means control part, 12 ... Image data composition processing part, 13. ..Image data recording device, 111... User setting information input unit, 112... Delay time setting unit, 121... Input unit, 123. 125 ... Content analysis section

Claims (17)

Means for inputting a plurality of image data having different frame rates obtained by imaging the same subject within the same imaging range and without considering the synchronization between the plurality of imaging means;
Means for synthesizing the plurality of input image data so as to be temporally continuous to generate composite image data;
An image data processing apparatus comprising: The image data processing apparatus according to claim 1,
The means for generating the composite image data is a ratio of a frame rate of an imaging means having the highest frame rate to a frame rate of an imaging means other than the imaging means having the highest frame rate among the plurality of imaging means. And setting the insertion position of the image data of the other imaging means with respect to the image data of the imaging means having the highest frame rate, so that each frame image in each image data is temporally continuous at the insertion position. An image data processing apparatus that generates composite image data by combining the data. The image data processing apparatus according to claim 1 or 2,
An image data processing apparatus comprising: an imaging means control means capable of setting an imaging start time for at least one imaging means among the plurality of imaging means. In the image data processing device according to any one of claims 1 to 3,
An image data processing apparatus comprising: means for performing at least one correction among positional deviation correction, color correction, and luminance correction on the plurality of input image data. In the image data processing device according to any one of claims 1 to 4,
The image data recording device that stores the plurality of input image data, and the means for generating the composite image data generates the composite image data using the plurality of image data recorded in the image data recording device. An image data processing apparatus. A plurality of image pickup means inputting a plurality of image data having different frame rates obtained by imaging the same subject in the same image pickup range and without considering the synchronization between the plurality of image pickup means;
Combining the plurality of input image data so as to be temporally continuous to generate composite image data;
An image data processing method comprising: A plurality of image pickup means inputting a plurality of image data having different frame rates obtained by imaging the same subject in the same image pickup range and without considering the synchronization between the plurality of image pickup means;
Combining the plurality of input image data so as to be temporally continuous to generate composite image data;
An image data processing program characterized in that Means for inputting a plurality of image data having different frame rates obtained by a plurality of image pickup means picking up the same subject within the same image pickup range and without considering the synchronization between the plurality of image pickup means; and An image data processing apparatus having means for synthesizing a plurality of input image data so as to be temporally continuous to generate composite image data;
An image display device for displaying the composite image data output from the image data processing device;
An image display system comprising: Means for inputting a plurality of image data having different frame rates obtained by imaging the same subject within the same imaging range and without considering the synchronization between the plurality of imaging means;
Means for performing content analysis using the plurality of input image data;
Means for setting a delay time based on the result of the content analysis;
Means for generating intermediate image data based on the delay time;
Means for synthesizing the generated intermediate image data and at least one of the input plurality of image data so as to be temporally continuous to generate combined image data;
An image data processing apparatus comprising: The image data processing apparatus according to claim 9, wherein
The content analysis is performed by detecting a change in image motion and / or a change in brightness based on the plurality of input image data. The image data processing apparatus according to claim 9 or 10,
The means for performing content analysis is based on a ratio of a frame rate of an imaging unit having the highest frame rate to a frame rate of an imaging unit other than the imaging unit having the highest frame rate among the plurality of imaging units. The frame image of the imaging unit having the highest frame rate when performing the content analysis is associated with the frame image of the other imaging unit, and the content analysis is performed between the associated frame images An image data processing apparatus. The image data processing apparatus according to claim 11.
The means for generating the composite image data includes the intermediate image data generated by performing content analysis between the attached frame images and the image data input from the imaging means having the highest frame rate. An image data processing apparatus characterized in that it is synthesized so as to be temporally continuous in consideration of the temporal positional relationship of the associated frame images. In the image data processing device according to claim 9-12,
An image data processing apparatus comprising: means for outputting the composite image data as data in a data format to which information indicating a time length of each frame image in the composite image data is added. In the image data processing device according to claim 9-12,
An image data processing apparatus comprising: means for outputting the composite image data as information different from the composite image data, indicating information indicating a time length of each frame image in the composite image data. A plurality of image pickup means inputting a plurality of image data having different frame rates obtained by imaging the same subject in the same image pickup range and without considering the synchronization between the plurality of image pickup means;
Performing content analysis using the plurality of input image data;
Setting a delay time based on the result of the content analysis;
Generating intermediate image data based on the delay time;
Synthesizing the generated intermediate image data and at least one of the input plurality of image data so as to be temporally continuous to generate composite image data;
An image data processing method comprising: A plurality of image pickup means inputting a plurality of image data having different frame rates obtained by imaging the same subject in the same image pickup range and without considering the synchronization between the plurality of image pickup means;
Performing content analysis using the plurality of input image data;
Setting a delay time based on the result of the content analysis;
Generating intermediate image data based on the delay time;
Synthesizing the generated intermediate image data and at least one of the input plurality of image data so as to be temporally continuous to generate composite image data;
An image data processing program characterized in that Means for inputting a plurality of image data having different frame rates obtained by imaging a plurality of imaging means in the same imaging range and without considering synchronization between the plurality of imaging means; Means for analyzing content using a plurality of input image data; means for setting a delay time based on the result of the content analysis; means for generating intermediate image data based on the delay time; and the generated intermediate image An image data processing apparatus having means for generating combined image data by combining data and at least one of the input plurality of image data so as to be temporally continuous;
An image display device for displaying the composite image data output from the image data processing device;
An image display system comprising:

Images