JP3929811B2 - Image composition apparatus and image composition method - Google Patents

Description

[0001]
BACKGROUND OF THE INVENTION
The present invention relates to an image synthesizing apparatus and an image synthesizing method for synthesizing images. For example, the present invention relates to image synthesis used in television program production, movie production, and the like.
[0002]
[Prior art]
Conventionally, in sports broadcast broadcasting or the like, slow motion playback has been frequently used in order to express the movements of the players as subjects in an easy-to-understand manner for the viewer. In addition, a technique called multi-motion is also used in which a player's movement is expressed like a stroboscopic photograph and the still images are sequentially overwritten on a background image to be combined.
[0003]
[Problems to be solved by the invention]
In sports broadcasts, slow motion playback that slowly expresses the movement of fast players is easy to understand and effective for viewers. On the other hand, due to the slow movement, there was a drawback that it was difficult to grasp the trajectory and form transition that the player moved.
[0004]
Multi-motion has the advantage that the background of the movement is easy to understand because of the afterimage effect because the images of stationary players are superimposed one after another and synthesized. However, since images are sequentially superimposed according to time, when a baseball or golf swing is displayed, there is no significant movement in the position of the subject, so even if there is a scene of interest such as a hitting point, it is overwritten. There was a drawback that this scene of interest disappeared. In addition, the multi-motion display has a drawback in visual expression that it is flat and cannot give a stereoscopic effect to the viewer.
[0005]
The present invention has been made to solve such a problem, and provides an image composition device and an image composition method capable of effectively expressing the movement of a subject accompanied by a motion in the depth direction. Objective.
[0006]
[Means for Solving the Problems]
An image composition apparatus according to the present invention is an image composition apparatus that performs image composition of a subject with movement in a depth direction, and reads out image data from an image memory that stores image data and the image memory. Image extracting means for extracting image data, image combining means for performing image composition based on image data stored in the image memory and / or an extracted image extracted by the image extracting means, and reference image determining means for determining a reference image The image synthesizing unit performs an overlay process in which the extracted image is sequentially superimposed on the image before the reference image determined by the reference image determining unit, and is added to the image after the reference image. On the other hand, a lower superposition process is performed in which the extracted images are sequentially superimposed on the lower side. With such a configuration, the reference image is displayed at the top, and if the attention scene is selected as the reference image, the attention scene remains without being overwritten and the progress can be effectively confirmed. it can.
[0007]
In a preferred embodiment, the lower overlay process combines the first composite image generated by the upper overlay process and an image after the reference image, and the first composite image is combined with the composite image. This is realized by overwriting the composite image from which the subject is extracted.
[0008]
Further, the lower superimposing process is a second image generated by removing a portion of the image obtained by extracting the subject of the first composite image generated by the upper overlapping process from the image obtained by extracting the subject of the image after the reference image. This composite image may be realized by combining the first composite image with the first composite image.
[0009]
Furthermore, the lower superimposition process is an extraction in which the subject of the image after the reference image is extracted by the image extraction means so as not to extract the subject extraction range of the first composite image generated by the upper superimposition process. This may be realized by combining an image with the first combined image.
[0010]
The reference image determination unit may determine the reference image based on information for specifying the reference image input by the input unit.
[0011]
The reference image determining means may determine a reference image based on the sound information detected by the sound detecting means. With such a configuration, the reference image can be automatically determined, and the composition process can be easily performed.
[0012]
Further, the reference image determining means may determine the reference image based on the position information of all or a part of the subject detected by the position detecting means. With such a configuration, the reference image can be automatically determined, and the composition process can be easily performed. Further, according to such means, it is possible to automatically select an image that specifies a state in which all or a part of the subject is closest to the photographing position as the reference image.
[0013]
Further, the reference image determining means may determine the reference image based on the position information of all or part of the subject detected by the depth detecting means. With such a configuration, the reference image can be automatically determined, and the composition process can be easily performed. Further, according to such means, it is possible to automatically select an image that specifies a state in which all or a part of the subject is closest to the photographing position as the reference image.
[0014]
In particular, it is desirable that the reference image determination means selects an image that specifies a state in which all or a part of the subject is closest to the shooting position as the reference image. With such a configuration, since the displayed image is visually three-dimensional, a powerful expression effect can be obtained.
[0015]
Another image synthesizer according to the present invention is an image synthesizer that synthesizes an image of a subject that moves in a depth direction, and reads out image data from an image memory that stores image data and the image memory. Image extracting means for extracting the image, image composition means for performing image composition based on the image data stored in the image memory and / or the extracted image extracted by the image extracting means, and a reference image for determining a reference image Determining means, and the image synthesizing means performs a lower superposition process in which the extracted images are sequentially overlaid on an image before the reference image determined by the reference image determining means, and The image is subjected to an overlay process in which the extracted images are sequentially superimposed on the upper side. With such a configuration, the reference image is displayed at the top, and if the attention scene is selected as the reference image, the attention scene remains without being overwritten and the progress can be effectively confirmed. it can. This process is particularly effective when the scene of interest is in the back.
[0016]
On the other hand, an image composition method according to the present invention is an image composition method for performing image composition of a subject with a movement in a depth direction, the step of determining a reference image, and an image before the determined reference image. Thus, the method includes a step of performing an overlay process in which the extracted images are sequentially superimposed on the upper side, and a step of performing a lower overlay process in which the extracted images are sequentially superimposed on the lower side of the image after the reference image. Since the reference image is displayed at the top by such a method, if the attention scene is selected as the reference image, the attention scene remains without being overwritten and the progress can be effectively confirmed. it can.
[0017]
Another image composition method according to the present invention is an image composition method for performing image composition of a subject with a movement in a depth direction, wherein a step of determining a reference image, and an image before the reference image, The method includes a step of performing a lower superimposing process in which the extracted images are sequentially superimposed on the lower side, and a step of performing an upper superimposing process in which the extracted images are sequentially superimposed on the subsequent image of the determined reference image. Since the reference image is displayed at the top by such a method, if the attention scene is selected as the reference image, the attention scene remains without being overwritten and the progress can be effectively confirmed. it can. This process is particularly effective when the scene of interest is in the back.
[0018]
An image composition program according to the present invention is an image composition program for performing image composition of a subject with a movement in a depth direction, a step of determining a reference image for a computer, and a step before the determined reference image. A step of performing an overlay process in which an extracted image is sequentially superimposed on an image, and a step of performing a overlay process in which an extracted image is sequentially superimposed on an image subsequent to the reference image are performed. It is. By executing such a program with a computer, the reference image is displayed at the top, so if you select the scene of interest as the reference image, the scene of interest remains without being overwritten, and its progress is effective. Can be confirmed.
[0019]
Another image composition program according to the present invention is an image composition program for performing image composition of a subject with a movement in a depth direction, a step of determining a reference image for a computer, and a step before the reference image. A step of performing a superimposing process in which the extracted image is sequentially superimposed on the lower side of the image and a step of performing an upper superimposing process in which the extracted image is sequentially superimposed on the upper side of the image after the determined reference image are performed. It is something to be made. By executing such a program with a computer, the reference image is displayed at the top, so if you select the scene of interest as the reference image, the scene of interest remains without being overwritten, and its progress is effective. Can be confirmed. This process is particularly effective when the scene of interest is in the back.
[0020]
DETAILED DESCRIPTION OF THE INVENTION
Embodiment 1 of the Invention
FIG. 1 shows the configuration of an image composition apparatus according to Embodiment 1 of the present invention. The image composition apparatus 100 is connected to the photographing camera 1 by a cable or a wireless line.
[0021]
The photographing camera 1 photographs a moving subject and generates an image data signal. In this example, it is mainly used for photographing a subject that moves in the depth direction, that is, on a line connecting the subject from the photographing camera 1 and away from the subject. For example, a baseball batter's swing motion, a pitcher's pitching motion, a golf swing motion, a soccer shoot motion, a figure skating rotation motion, a judo throwing motion, and a car cornering motion in a car race. The photographing camera 1 inputs the generated image data signal to the image composition device 100 via a cable. Note that the image data signal from the photographing camera 1 may be temporarily stored in a storage medium such as a magnetic tape, and the image data signal may be read from the storage medium in the image composition device 100.
[0022]
The image composition apparatus 100 is configured on hardware by a dedicated computer having a CPU, ROM, RAM, hard disk, display, keyboard, and the like, for example. The image composition apparatus 100 can also be configured by installing an application program in a so-called general-purpose computer. Further, the image composition device 100 includes a buffer memory, and appropriately stores a composite image 5 with background, a composite image 6 without background, and an extracted image 7.
[0023]
As shown in FIG. 1, the image composition device 100 includes an image memory 2, an image extraction unit 3, and an image composition unit 4.
[0024]
The image memory 2 is a storage unit that accumulates image data signals input from the photographing camera 1. In the image memory 2, image data to be subjected to image synthesis processing according to the present invention is stored in association with time information for each frame. The number of frames of the image data is appropriately selected according to the speed and operation time of the subject performing the image composition process.
[0025]
The image extraction unit 3 inputs image data from the image memory 2, extracts a subject image, and performs a process of separating it from a background image. For example, in the case of an image relating to a baseball batter's swing motion, only the portion of the batter that is the subject is extracted. Extraction of image information by the image extraction unit 3 can be realized by an operator specifying an extracted image on the screen, and can also be automatically generated by the following method.
[0026]
(1) The difference from the background is taken and image processing is performed.
[0027]
(2) The position of the extraction target is detected using a detector to extract a specific portion in the video.
[0028]
(3) Using color information, a portion other than a specific color is extracted.
[0029]
The image extracted by the image extraction unit 3 is stored as an extracted image 7 in the buffer memory.
[0030]
The image composition unit 4 receives the image data stored in the image memory 2 and the extracted image data generated by the image extraction unit 3, and performs image composition processing. For example, the image composition unit 4 performs a process of superimposing the extracted image 7 input from the image extraction unit 3 on the image input from the image memory 2 to generate a composite image 5 with a background. The image composition unit 4 can also perform a process of superimposing a plurality of extracted images input from the image extraction unit 3 to generate a composite image 6 without a background.
[0031]
Here, an example of a method for synthesizing the image B behind the synthesized image A will be described. First, the extracted image of the image B is superimposed on the composite image A with the background. Thereafter, a composite image A without a background is further superimposed on the superimposed composite image. By such composition processing, the image B can be displayed so as to be positioned below the image A.
[0032]
FIG. 2 shows an image before composition and an image after composition in order to explain the image composition processing of the image composition apparatus according to the first embodiment of the present invention. Images 2a to 2f are images before synthesis, and image 20 is an image after synthesis.
[0033]
The images 2a to 2f are simply images obtained by capturing the baseball batter's swing motion at regular intervals. The image 2a is arranged in time series with the image 2a as the first, and the image 2f is the last. The background of each image is indicated by diagonal lines. In the present invention, a reference image is determined, and composition processing for an image temporally before the reference image and the reference image and composition processing for an image temporally after the reference image are changed. It has the characteristics. In the first embodiment of the present invention, the subject is superimposed on the image temporally before the reference image and the reference image, and the subject is superimposed on the image temporally after the reference image. Perform processing. There are various methods for determining the reference image, which will be described in detail in Embodiment 4 and later.
[0034]
In the example illustrated in FIG. 2, an image 2c that is a scene of interest indicating a hitting point is used as a reference image. In selecting the reference image, when preferentially selecting an image to be prevented from being overwritten by another image in this way, the subject or the subject (a part of the subject) in front is closest to the photographing camera 1. There are various cases, such as selecting an image where the
[0035]
In Embodiment 1 of the present invention, the image 2a, the image 2b, and the subject of the reference image 2c before the reference image 2c are overlaid. More specifically, the image of the batter in the image 2b is superimposed on the image of the batter in the image 2a (including the bat, the same applies hereinafter). Further, the image of the batter of the image 2c is overlaid on the composite image. Thereafter, the subject of the image 2d, the image 2e, and the image 2f after the reference image 2c is subjected to a lower overlay process. More specifically, the image of the batter of the image 2d is overlaid on the composite image on which the images 2a to 2c are overlaid. Furthermore, the image of the batter of the image 2e is overlaid on the composite image. Finally, the batter's image of the image 2f is overlaid on the composite image. The composite image generated in this way is an image 20.
[0036]
In the image 20, since the reference image with the target scene is displayed on the top, the target scene remains without being overwritten and the progress can be confirmed. When the target scene is on the shooting camera side, the displayed image is visually three-dimensional, so a powerful expression effect can be obtained.
[0037]
Next, the image composition process will be described in detail with reference to FIG. An example image is the same as that shown in FIG.
[0038]
First, the images 2a, 2b, and 2c are read from the image memory 2, and are synthesized by the image synthesizing unit 4 so that the images 2a, 2b, and 2c are sequentially overlaid, thereby generating an image 21. Since this synthesizing method is a well-known method performed in the conventional multi-motion, the description thereof is omitted. Next, the image extraction unit 3 reads the image 2d from the image memory 2, extracts only the subject image, and generates an extracted image 22. This extracted image 22 is an image without a background.
[0039]
Next, a composite process 23 is performed on the composite image 21 and the image 22 to obtain an image 24. Further, the image extraction unit 3 extracts only the image of the subject from the composite image 21 and generates an extracted image 25. This extracted image 25 is an image without a background. Then, a composite image 27 is generated by combining the composite image 24 and the extracted image 25. Similarly, a composite image 20 shown in FIG. 2 can be obtained by combining the composite image 27 with the images 2e and 2f.
[0040]
Embodiment 2 of the Invention
FIG. 4 shows a configuration of an image composition device according to the second exemplary embodiment of the present invention. The shooting camera 31 is the shooting camera 1 shown in FIG. 1, the image memory 32 is the image memory 2 shown in FIG. 1, the image extraction unit 33 is the image extraction unit 3 shown in FIG. Have functions equivalent to those of the image composition unit 4 shown in FIG.
[0041]
The image composition device 100 according to the second embodiment of the present invention includes an extraction area memory 36 as shown in FIG.
[0042]
The image compositing unit 34 according to the second embodiment of the present invention superimposes the extracted image 37 extracted by the image extracting unit 33 on the image data stored in the image memory 32, so that a composite image 35 with a background is obtained. Generate. Further, the image composition unit 34 appropriately captures and stores information indicating the range of the image extracted by the image extraction unit 33 in the extraction region memory 36.
[0043]
Here, an example of a method for synthesizing the image B behind the synthesized image A will be described. First, the range portion of the extracted image of the composite image A stored in the extraction area memory 36 is removed from the extracted image 37 of the image B. Then, the composite image A is superimposed on this image. By such composition processing, the image B can be displayed so as to be positioned below the image A.
[0044]
Next, an image composition process according to the second embodiment of the present invention will be specifically described with reference to FIG. An example image is the same as that shown in FIG.
[0045]
First, the images 2a, 2b, and 2c are read from the image memory 32, and the image composition unit 34 composes the images 2a, 2b, and 2c so that the images 2a, 2b, and 2c are sequentially overlaid, thereby generating an image 41. Next, the image extraction unit 33 extracts only the image of the subject from the image 41, generates an extracted image 44, and stores it in the extraction area memory 36. This extracted image 44 is an image without a background.
[0046]
Further, the image 2 d is read from the image memory 32, and only the image of the subject is extracted by the image extraction unit 33, and an extracted image 43 is generated and stored in the extraction area memory 36. This extracted image 43 is an image without a background.
[0047]
Next, the extracted image 43 and the extracted image 44 are read from the extraction region memory 36, and the image composition unit 34 performs a process of removing the extracted image 44 from the extracted image 43. By this processing, an image 46 is obtained. Subsequently, the synthesized image 42 is obtained by synthesizing the image 46 and the image 41. Similarly, the composite image 20 shown in FIG. 2 can be obtained by combining the composite image 42 with the images 2e and 2f.
[0048]
Embodiment 3 of the Invention
FIG. 6 shows the configuration of an image composition apparatus according to the third embodiment of the present invention. The shooting camera 51 is the shooting camera 1 shown in FIG. 1, the image memory 52 is the image memory 2 shown in FIG. 1, the image extraction unit 53 is the image extraction unit 3 shown in FIG. Has the same function as the image composition unit 4 shown in FIG.
[0049]
The image composition device 100 according to the third embodiment of the present invention includes an extraction area memory 56 as shown in FIG.
[0050]
The image compositing unit 54 according to the third embodiment of the present invention superimposes the extracted image 57 extracted by the image extracting unit 53 on the image data stored in the image memory 52, so that a composite image 55 with a background is obtained. Generate. Further, the image composition unit 54 appropriately captures and stores information indicating the range of the image extracted by the image extraction unit 53 in the extraction region memory 56.
[0051]
Here, an example of a method for synthesizing the image B behind the synthesized image A will be described. First, when the extraction image of the image B is generated by the image extraction unit 53, the extraction range of the composite image A stored in the extraction region memory 56 is processed so as not to be extracted. Then, the composite image A is superimposed on this image. By such composition processing, the image B can be displayed so as to be positioned below the image A.
[0052]
Next, an image composition process according to the third embodiment of the present invention will be specifically described with reference to FIG. An example image is the same as that shown in FIG.
[0053]
First, the images 2a, 2b, and 2c are read from the image memory 52, and are synthesized by the image synthesis unit 54 so that the images 2a, 2b, and 2c are overlaid in order. Next, the image extraction unit 53 extracts only the image of the subject from the image 61, generates an extracted image 62, and stores information on the extraction range in the extraction region memory 56.
[0054]
Further, the image 2d is read from the image memory 52, and only the image of the subject is extracted by the image extraction unit 53. At this time, based on the extraction range information of the extraction image 62 stored in the extraction region memory 56, the image in this range is Process not to extract. In this way, the extracted image 65 is generated. The extracted image 65 is an image without a background.
[0055]
Next, the combined image 67 is obtained by combining 66 the extracted image 65 and the combined image 61 by the image combining unit 54. Similarly, the composite image 20 shown in FIG. 2 can be obtained by combining the composite image 67 with the images 2e and 2f.
[0056]
Embodiment 4 of the Invention
The fourth embodiment of the present invention relates to a process for determining a reference image. As shown in FIG. 8, in the fourth embodiment of the present invention, a reference image determination device 73 is connected to the image composition device 72. The reference image determination device 73 may be configured to be incorporated in the image composition device 72. That is, the reference image determination device 73 can be realized as a function of the image composition device 72.
[0057]
Here, the photographing camera 71 corresponds to the photographing camera 1 shown in FIG. 1, and the image composition device 72 corresponds to the image composition device 100 shown in FIG.
[0058]
A keyboard 74, a microphone 75, and a camera 76 can be connected to the reference image determination device 73. The reference image determination device 73 according to the fourth embodiment of the present invention is described as being able to determine the reference image by various aspects, but is not limited to this, and the reference image is determined by one or any combination of the aspects. It is good also as what can determine an image.
[0059]
When the reference image determination device 73 is configured separately from the image composition device 72, the reference image determination device 73 includes a control circuit such as a CPU, a ROM, and a RAM, a memory, a display, and the like. In the memory, an algorithm for determining a reference image and a software program for performing processing are installed.
[0060]
The keyboard 74 is used as an input means for the operator to input the determination of the reference image while directly checking the image on the screen. As this input means, in addition to the keyboard 74, for example, a pointing device such as a mouse or a touch pad is included. At this time, the reference image determination device 73 extracts information for specifying the reference image based on the input signal from the input means such as the keyboard 74 and outputs the information to the image composition device 72.
[0061]
The microphone 75 is used as a sound detection unit that detects sound generated with the movement of the subject. For example, in the case of a swing action of a baseball batter, a hitting sound generated when a ball collides with a bat is detected. At this time, the reference image determination device 73 inputs the sound information detected by the microphone 75, and first extracts a sound to be specified, for example, a hitting sound from the sound information. This sound extraction can be realized, for example, by setting sound intensity and frequency data to be specified in advance and comparing the input sound information with the data. When a sound to be specified is extracted from the sound information, time information when the sound is detected is generated. Based on this time information, the time when the sound is detected from the image memory of the image synthesizer 72 or an image taken before or after a certain time is determined as the reference image. An algorithm for specifying the reference image based on the time information is configured to be preset by an operator.
[0062]
The camera 76 is a camera provided at a position different from the shooting camera 81 and is a camera for deriving time information of the reference image by shooting the motion of the subject. For example, in the case of a baseball batter's swing motion, a camera attached to the ceiling of the dome stadium is applicable. Since this camera can photograph the batter from above, the position of the bat can be clearly photographed. Then, time information when the bat passes the hitting point region is detected. Based on this time information, a reference image is determined. An algorithm for specifying the reference image based on the time information is configured to be preset by an operator.
[0063]
Embodiment 5 of the Invention
In the fifth embodiment of the present invention, as shown in FIG. 9, a position detection sensor 83 is connected to the image composition device 82. Here, the photographing camera 81 corresponds to the photographing camera 1 shown in FIG. 1, and the image composition device 82 corresponds to the image composition device 100 shown in FIG.
[0064]
The position detection sensor 83 acquires position information of the moving subject. The image composition device 82 acquires the position information of the subject from the position detection sensor 83, and determines the order in which the extracted images are superimposed so that the subject position and the composite image do not contradict each other. The position detection sensor 83 is an optical sensor, for example, and is installed at a predetermined position. For example, when shooting a golf swing motion, the golf club is installed at a position where it hits the ball. Based on the detection signal of the position detection sensor 83, it is possible to obtain time information when the golf club passes the position where it hits the ball. Based on this time information, a reference image is determined. An algorithm for specifying the reference image based on the time information is configured to be preset by an operator.
[0065]
Embodiment 6 of the Invention
In the sixth embodiment of the present invention, as shown in FIG. 10, a depth detection device 93 is connected to the image composition device 92. The depth detection device 93 may be provided in the image composition device 92. Here, the photographing camera 91 corresponds to the photographing camera 1 shown in FIG. 1, and the image composition device 92 corresponds to the image composition device 100 shown in FIG.
[0066]
The depth detection device 93 acquires position information in the depth direction of a specific portion of the moving subject, that is, the line direction connecting the photographing camera 91 and the subject. For example, in the case of a swing motion of a baseball batter, the position information of the tip portion of the bat. This position information can be detected using, for example, a Color Digiclops (Point Gray Co., Ltd.) three-dimensional image capturing device.
[0067]
Based on the position information in the depth direction detected by the depth detection device 93, the image composition device 92 detects time information at which the specific location and the photographing camera 91 are closest to each other, and the specific location is located at this position. An image obtained by photographing the state is set as a reference image. Then, the images are synthesized by the method as described in the above embodiments of the invention so that the reference image is displayed at the top.
[0068]
Note that when not all of the subject moves in the depth direction but all of the subject moves in the depth direction, the position information of all the subjects in the depth direction is detected, and based on this position information, the reference An image may be determined. Specifically, an image obtained by photographing a state where all of the subjects are closest to the photographing camera 91 is set as a reference image. Then, the images are synthesized by the method as described in the above embodiments of the invention so that the reference image is displayed at the top.
[0069]
In this way, an image obtained by photographing a state where a specific part or the entire subject is located closest to the photographing camera is used as a reference image, and the image is synthesized so that the reference image is displayed at the top. The object can be displayed in three dimensions.
[0070]
The photographing camera 91 and the depth detection device 93 may be realized by a single camera.
[0071]
Embodiment 7 of the Invention
In the image composition device according to the seventh embodiment of the present invention, a depth detection device 93 as shown in FIG. 10 is connected to the image composition device 92 as in the sixth embodiment. The depth detection device 93 may be provided in the image composition device 92. In the seventh embodiment of the present invention, in particular, the motion of the subject detected by the depth detection device 93 is further analyzed, and the position of the specific portion at the start and end of the operation is compared with the position of the specific portion at the reference time. Thus, the composition process is changed.
[0072]
FIG. 11 shows a processing flow of the image composition apparatus according to the seventh embodiment of the present invention. First, the subject is photographed by the photographing camera 91 (S101). Next, the image composition device inputs image data from the photographing camera 91 and stores it in the image memory (S102). A reference image is determined based on the image data stored in the image memory (S103). The determination of the reference image may be automatically determined using the depth detection device 93 as described in the sixth embodiment of the invention, or may be determined by the method described in the fourth and fifth embodiments of the invention. Good.
[0073]
Next, whether the position of the specific part at the start of the operation and the end of the operation is deeper than the position of the specific part at the reference time when the reference image is captured, that is, whether the position is away from the shooting camera 91. A determination is made based on the output signal of the depth detection device 93 (S104).
[0074]
As a result of the determination, if it is determined that the position of the specific part at the start and end of the operation is behind the position of the specific part at the time of reference when the reference image is captured, the reference image is overlaid. Then, after the reference image, a lower overlay process is performed (S105).
[0075]
As a result of the determination, if it is determined that the position of the specific location at the start and end of the operation is not behind the position of the specific location at the time of reference when the reference image is captured, the reference image is processed upside down. Then, after the reference image, the superposition process is performed (S106).
[0076]
For example, in the case of a baseball batter's swing motion, when shooting from the front as shown in FIG. 2, if a specific location is a bat, step S105 is performed. On the other hand, if the image is taken from behind the batter, step S106 is performed.
[0077]
In this example, the case where the depth detection device 93 automatically determines is described. However, the operator can set either the process in step S105 or the process in step S106 on the keyboard or the like. May be.
[0078]
Other embodiments.
Note that the above-described image synthesizing apparatus is not necessarily configured by a single computer, and may be configured by a plurality of computers. Further, the image synthesizing device, the photographing camera, and further the reference image determining device may be configured by a single device. For example, an image composition device or a reference image determination device may be incorporated in a home or business camera that captures a moving image. Further, an image composition device may be incorporated in a home digital video playback device or game machine. It is advisable to install this image composition device together with a photographing camera in a so-called batting center, golf driving range, game center or the like.
[0079]
Also, an image composition device is provided on the Internet server, image data is transmitted from a client computer via the Internet, the image composition processing according to the present invention is performed by this image composition device, and the composite image is transmitted to the client computer. It may be. At this time, the reference image may be designated by the client computer.
[0080]
Although some examples of determining the reference image have been described, information related to the reference image may be determined, not necessarily the reference image itself. For example, the reference image can be indirectly determined by determining the reference time.
[0081]
In the above example, the background image is incorporated on the finally obtained image, but it is not always necessary to incorporate it. In addition, other images may be incorporated instead of the background image at the time of shooting.
[0082]
Even if the composite image created by the image composition device according to the present invention is once displayed, the reference image can be easily changed by designating a subject to be a reference image among the subjects on the screen by a pointer or the like. Good.
[0083]
In the above example, when the image composition device is configured by a general-purpose computer, an application program for causing the general-purpose computer to function as the image composition device is installed. This application program can be stored in various types of storage media, and can be transmitted via a communication medium. Here, the storage medium includes, for example, a flexible disk, a hard disk, a magnetic disk, a magneto-optical disk, a CD-ROM, a DVD, a ROM cartridge, a battery-backed RAM memory cartridge, a flash memory cartridge, and a nonvolatile RAM cartridge. The communication medium includes a wired communication medium such as a telephone line, a wireless communication medium such as a microwave line, and the Internet.
[0084]
【The invention's effect】
According to the present invention, it is possible to provide an image composition device and an image composition method capable of effectively expressing the movement of a subject accompanied by an operation in the depth direction.
[Brief description of the drawings]
FIG. 1 is a block diagram showing a configuration of an image composition apparatus according to the present invention.
FIG. 2 is a diagram showing an image before composition and an image after composition.
FIG. 3 is a diagram illustrating a composition process by the image composition apparatus according to the present invention.
FIG. 4 is a block diagram showing a configuration of an image composition apparatus according to the present invention.
FIG. 5 is a diagram showing a composition process by the image composition apparatus according to the present invention.
FIG. 6 is a block diagram showing a configuration of an image composition apparatus according to the present invention.
FIG. 7 is a diagram showing a composition process by the image composition apparatus according to the present invention.
FIG. 8 is a block diagram showing a configuration of an image composition device according to the present invention.
FIG. 9 is a block diagram showing a configuration of an image composition apparatus according to the present invention.
FIG. 10 is a block diagram showing a configuration of an image composition apparatus according to the present invention.
FIG. 11 is a flowchart showing a composition process by the image composition apparatus according to the present invention.
[Explanation of symbols]
1 Camera for Shooting 100 Image Synthesizer
2 Image memory 3 Image extraction unit 4 Image composition unit
73 Reference Image Determination Device

Claims (14)

An image composition device that performs image composition of a subject with movement in a depth direction,
An image memory for storing image data;
Image extraction means for reading out image data from the image memory and extracting an image of a subject;
Image composition means for performing image composition based on the image data stored in the image memory and / or the extracted image extracted by the image extraction means;
A reference image determining means for determining a reference image;
The image synthesizing unit performs an overlay process in which the extracted image is sequentially overlaid on the image before the reference image determined by the reference image determining unit, and sequentially on the image after the reference image. An image synthesizing apparatus that performs a bottom overlay process in which an extracted image is superimposed below. The lower overlay process combines the first composite image generated by the upper overlay process and the image after the reference image, and the composite image obtained by extracting the subject of the first composite image from the composite image The image composition apparatus according to claim 1, wherein the image composition apparatus is realized by overwriting. The lower superimposition process is a second synthesis generated by removing a portion of the image obtained by extracting the subject of the first composite image generated by the upper superposition process from the image obtained by extracting the subject of the image after the reference image. The image synthesizing apparatus according to claim 1, wherein the image synthesizing apparatus is realized by synthesizing an image with the first synthesized image. In the lower superimposition process, the extracted image obtained by extracting the subject of the image after the reference image by the image extracting unit is not extracted so as not to extract the subject extraction range of the first composite image generated by the upper superposition process. The image synthesizing apparatus according to claim 1, wherein the image synthesizing apparatus is realized by synthesizing with one synthesized image. 5. The image synthesizing apparatus according to claim 1, wherein the reference image determining unit determines the reference image based on information for specifying the reference image input by the input unit. 5. The image synthesizing apparatus according to claim 1, wherein the reference image determining means determines a reference image based on the sound information detected by the sound detecting means. 5. The image synthesizing apparatus according to claim 1, wherein the reference image determining means determines the reference image based on position information of all or part of the subject detected by the position detecting means. . 5. The image synthesizing apparatus according to claim 1, wherein the reference image determining means determines the reference image based on position information of all or part of the subject detected by the depth detecting means. . 5. The image synthesizing apparatus according to claim 1, wherein the reference image determining unit selects, as a reference image, an image that specifies a state in which all or a part of a subject is closest to the photographing position. An image composition device that performs image composition of a subject with movement in a depth direction,
An image memory for storing image data;
Image extraction means for reading out image data from the image memory and extracting an image of a subject;
Image composition means for performing image composition based on the image data stored in the image memory and / or the extracted image extracted by the image extraction means;
A reference image determining means for determining a reference image;
The image synthesizing unit performs a lower superposition process in which the extracted image is sequentially superimposed on an image before the reference image determined by the reference image determining unit, and for an image after the reference image, An image composition apparatus for performing an overlay process in which extracted images are sequentially superimposed on the upper side. An image composition method for compositing an image of a subject with a movement in a depth direction,
Determining a reference image;
Performing an overlay process in which the extracted image is sequentially superimposed on the image before the determined reference image;
An image synthesizing method including a step of performing a lower superimposition process in which an extracted image is sequentially superimposed on an image subsequent to the reference image. An image composition method for compositing an image of a subject with a movement in a depth direction,
Determining a reference image;
Performing an overlay process in which the extracted images are sequentially superimposed on the previous image of the reference image;
And a step of performing an overlay process in which the extracted image is sequentially superimposed on the subsequent image of the determined reference image. An image synthesis program for synthesizing an image of a subject with a movement in the depth direction.
Determining a reference image;
Performing an overlay process in which the extracted image is sequentially superimposed on the image before the determined reference image;
An image composition program for executing a step of performing a lower superimposition process in which an extracted image is sequentially superimposed on an image subsequent to the reference image. An image synthesis program for synthesizing an image of a subject with a movement in the depth direction.
Determining a reference image;
Performing an overlay process in which the extracted images are sequentially superimposed on the previous image of the reference image;
An image synthesis program for executing a step of performing an overlay process in which an extracted image is sequentially overlaid on a subsequent image of a determined reference image.

Images