JP2019003325A - Image processing system, image processing method and program - Google Patents

Abstract

To reduce unnaturalness due to a difference of exposures in photographing of a foreground image and a background image when creating a virtual viewpoint image.SOLUTION: A back-end server (270) creates a virtual viewpoint image using multiple images photographed by multiple cameras (112). The back-end server (270) acquires a foreground image that is an image based on either one of the multiple images and corresponding to a prescribed object, acquires exposure information indicating an exposure when the foreground image is photographed, and acquires a background image that is an image based on either one of the multiple images and that is photographed with an exposure corresponding to the exposure information and that does not include the prescribed object on the basis of the exposure information. Then, the back-end server (270) creates a virtual viewpoint image using the foreground image and the background image.SELECTED DRAWING: Figure 1

Description

  The present invention relates to an image processing apparatus, an image processing method, a program, and an image processing system that process an image obtained by photographing a subject.

  Nowadays, multiple cameras are installed at different positions and synchronized shooting from multiple viewpoints. Using multiple viewpoint images obtained by the shooting, virtual viewpoints or virtual cameras different from these multiple cameras are used. A technique for generating virtual viewpoint content has attracted attention. According to the technology for generating virtual viewpoint content in which viewpoints can be arbitrarily changed from a plurality of viewpoint images, for example, a soccer or basketball highlight scene can be viewed from various angles. Therefore, the technology for generating the virtual viewpoint content can give the user a higher sense of presence compared with a normal image in which the viewpoint cannot be arbitrarily changed. The image generated in this way is called a virtual viewpoint image.

  Patent Document 1 discloses a technique for arranging and arranging a plurality of cameras so as to surround the same range, and generating and displaying a virtual viewpoint image corresponding to an arbitrary designation using an image obtained by photographing the same range. Has been.

  In order to generate a virtual viewpoint image, it is generally necessary to identify a portion where an object to be model generated exists in the image. The part where the object exists is called the foreground, otherwise it is called the background.

JP 2014-215828 A

  When a plurality of photographed images are combined to generate one image such as a virtual viewpoint image, it is desirable to prevent differences in brightness and color of the photographed images. However, if the foreground image and the background image used for composition when generating the virtual viewpoint image are different in timing at the time of shooting, for example, and the shooting device used for shooting, the exposure at the time of shooting may be different. When a foreground image and a background image having different exposures are used for generating a virtual viewpoint image, there may be a problem that an unnatural image is synthesized.

  Therefore, an object of the present invention is to reduce unnaturalness due to a difference in exposure during shooting of a foreground image and a background image when generating a virtual viewpoint image.

  The present invention is an image processing device that generates a virtual viewpoint image using a plurality of images photographed by a plurality of imaging devices, and is an image based on any of the plurality of images, wherein A first acquisition unit that acquires a foreground image corresponding to an object; a second acquisition unit that acquires exposure information indicating exposure when the foreground image acquired by the first acquisition unit is captured; An image based on any one of the images, which is captured with an exposure corresponding to the exposure information acquired by the second acquisition unit, and does not include the predetermined object, a background image is acquired by the second acquisition unit. A virtual viewpoint image is generated using third acquisition means acquired based on the acquired exposure information, the foreground image acquired by the first acquisition means, and the background image acquired by the third acquisition means. Characterized in that it has a generation unit configured to, a.

  Another aspect of the present invention provides a foreground image and a background used for generating a virtual viewpoint image from a plurality of foreground images and a plurality of background images obtained by separating a plurality of images taken by a plurality of imaging devices, respectively. Determining means for determining an image; acquisition means for acquiring exposure information at the time of shooting of the determined foreground image; and exposure information at the time of shooting of the determined background image; the foreground image and the background; Processing means for generating the virtual viewpoint image based on an image, and the processing means, when the exposure information of the determined foreground image and the background image is different, the determined foreground Performing the process of adjusting the exposure of the background image to the exposure of the image, and generating the virtual viewpoint image using the background image after the process of adjusting the exposure and the determined foreground image. You .

  According to the present invention, when generating a virtual viewpoint image, it is possible to reduce unnaturalness due to a difference in exposure during photographing between the foreground image and the background image.

It is a figure which shows the structural example of an image processing system. It is a figure which shows the function structural example of a camera adapter. It is a figure which shows the function structural example of the front end server of 1st Embodiment. It is a figure which shows the function structural example of the back end server of 1st Embodiment. It is a figure which shows the production | generation processing flow of the virtual viewpoint image of 1st Embodiment. It is a figure which shows the function structural example of the back end server of 2nd Embodiment. It is a figure which shows the function structural example of the back end server of 3rd Embodiment. It is explanatory drawing of the exposure relationship of a foreground image and a background image in 3rd Embodiment. It is a figure which shows the function structural example of the front end server of 4th Embodiment. It is a figure which shows the function structural example of the back end server of 4th Embodiment.

Hereinafter, an example of an embodiment of the present invention will be described in detail with reference to the drawings.
In the present embodiment, an example of an image processing system that installs a plurality of cameras in a stadium or the like, performs synchronous shooting, and generates a virtual viewpoint image using shooting data in which shooting parameters are changed during shooting will be described as an example. To do.
<First Embodiment>
In the first embodiment, when exposure information of an image used for generating a virtual viewpoint image differs due to a change in shooting parameters at the time of shooting, a virtual viewpoint image is generated by re-acquiring images with matching exposure information from the database. An example will be described.

FIG. 1 is a block diagram showing a schematic configuration of an image processing system 100 according to the present embodiment.
The image processing system 100 includes a sensor system 110a to a sensor system 110z, an image computing server 200, a controller 300, a switching hub 180, and an end user terminal 190.
The controller 300 includes a control station 310 and a virtual camera operation UI 330. The control station 310 performs operation state management, parameter setting control, and the like through the networks 310a to 310c, 180a and 180b, and the daisy chains 170a to 170y for each block constituting the image processing system 100. For each of these networks, for example, GbE (Gigabit Ethernet) or 10 GbE conforming to the IEEE standard, which is Ethernet (Ethernet is a registered trademark), can be used. Each network may be configured by combining interconnect Infiniband, industrial Ethernet, and the like. Further, the present invention is not limited to these, and other types of networks may be used.

First, an operation of transmitting 26 sets of images and sounds of the sensor systems 110a to 110z from the sensor system 110z to the image computing server 200 will be described. In the image processing system 100 of the present embodiment, the sensor systems 110a to 110z are connected by daisy chains 170a to 170y.
In addition, in this embodiment, when there is no special description, each system of 26 sets from the sensor system 110a to the sensor system 110z is described as the sensor system 110 without distinguishing. Similarly, each device in each sensor system 110 is described as a microphone 111, a camera 112, a camera platform 113, an external sensor 114, and a camera adapter 120 without distinguishing them unless there is a special description. In addition, although 26 sets are described as the number of sensor systems, this is only an example, and the number is not limited to this example. Further, in this embodiment, unless otherwise specified, the term “image” is described as including the concept of a moving image and a still image. That is, the image processing system 100 according to the present embodiment can process both still images and moving images. In the present embodiment, the virtual viewpoint content provided by the image processing system 100 includes a virtual viewpoint image and a virtual viewpoint sound, but is not limited thereto. For example, sound may not be included in the virtual viewpoint content. Further, for example, the sound included in the virtual viewpoint content may be a sound collected by a microphone closest to the virtual viewpoint. Further, in this embodiment, for the sake of simplicity of explanation, the description of the sound is partially omitted, but it is basically assumed that both the image and the sound are processed. Furthermore, in this embodiment, unless otherwise specified, image data and audio data transmitted via a network are simply described as images and audio.

  Each of the sensor systems 110a to 110z includes one camera 112a to 112z. That is, the image processing system 100 includes a plurality of cameras for photographing a subject from a plurality of directions. The plurality of sensor systems 110 are connected by a daisy chain 170. With this connection form, it is possible to reduce the number of connection cables and save wiring work in increasing the capacity of image data as the resolution of captured images increases to 4K, 8K, etc. and the frame rate. . Further, the connection form is not limited to the daisy chain 170, and each of the sensor systems 110 a to 110 z is connected to the switching hub 180, and is a star type network configuration that transmits and receives data between the sensor systems 110 via the switching hub 180. Also good.

  1 shows a configuration in which all of the sensor systems 110a to 110z are cascade-connected so as to form the daisy chain 170, but the present invention is not limited to this connection example. For example, the plurality of sensor systems 110 may be divided into several groups, and the sensor systems 110 may be daisy chain connected in divided groups. Then, the camera adapter 120 that is the end of the division unit may be connected to the switching hub 180 to input image data to the image computing server 200. Such a configuration is particularly effective in a stadium. For example, a case where a stadium is composed of a plurality of floors and the sensor system 110 is deployed on each floor can be considered. In this case, input to the image computing server 200 can be performed every floor or every half of the stadium, and the installation and system can be simplified even in places where wiring is difficult to connect all sensor systems 110 with a single daisy chain. Can be made flexible.

  In the present embodiment, the sensor system 110a includes a microphone 111a, a camera 112a, a pan head 113a, an external sensor 114a, and a camera adapter 120a. The front end server 230 may have at least part of the functions of the camera adapter 120. In the case of the present embodiment, the sensor systems 110b to 110z have the same configuration as the sensor system 110a, and thus description thereof is omitted. The sensor systems 110b to 110z are not limited to the same configuration as the sensor system 110a, and each sensor system 110 may have a different configuration.

The image captured by the camera 112a and the sound collected by the microphone 111a are subjected to various image processing and sound processing in the camera adapter 120a, and then the camera adapter 120b of the sensor system 110b through the daisy chain 170a. Is transmitted. The camera adapter 120 separates the foreground image including the object image and the other background images from the captured image. Details of the image processing performed by the camera adapter 120a will be described later. The camera 112a has a function of outputting camera exposure information such as an aperture, shutter speed, and ISO sensitivity at the time of shooting. The exposure information is also sent from the camera adapter 120a to the camera adapter 120b together with the data of the image taken by the camera 112a. In the present embodiment, an EV value is used as exposure information. When the shutter speed is T and the aperture is F, the EV value can be obtained by the following equation (1).
EV = log ₂ F ² -log ₂ T Formula (1)

  Similarly, the sensor system 110b transmits the image captured by the camera 112b and the sound collected by the microphone 111b together with the image and sound acquired from the sensor system 110a to the sensor system 110c. By continuing such an operation in each sensor system 110, the images and sounds acquired by the sensor systems 110a to 110z are transmitted from the sensor system 110z at the final stage to the switching hub 180 using the network 180b. . These image and audio data are transmitted from the switching hub 180 to the image computing server 200.

Next, the configuration and operation of the image computing server 200 will be described.
The image computing server 200 according to this embodiment processes data acquired from the sensor system 110z via the switching hub 180. The image computing server 200 includes a front-end server 230, a database 250 (the database is also referred to as DB as appropriate), a back-end server 270, and a time server 290.

  The time server 290 has a function of distributing the time and the synchronization signal, and distributes the time and the synchronization signal to the sensor systems 110a to 110z via the switching hub 180. The camera adapters 120a to 120z that have received the time and the synchronization signal perform image frame synchronization by genlocking the cameras 112a to 112z based on the time and the synchronization signal. That is, the time server 290 synchronizes the shooting timings of the plurality of cameras 112. As a result, the image processing system 100 can generate a virtual viewpoint image based on a plurality of captured images captured at the same timing, and can suppress deterioration in the quality of the virtual viewpoint image due to a shift in the capturing timing.

  The front-end server 230 reconstructs a segmented transmission packet from the image and sound acquired from the sensor system 110z via the switching hub 180 and converts the data format. Then, the front-end server 230 writes the image data after the data format conversion in the database 250 according to the camera identifier, data type, and frame number. At this time, the camera exposure information transmitted together with the foreground image and the background image from each camera adapter 120 is written in the database 250 in units of frames in association with the image data after the data format conversion.

  Next, the back-end server 270 accepts the designation of the viewpoint from the virtual camera operation UI 330, and based on the accepted viewpoint, the virtual end point content of the virtual viewpoint content is selected from a plurality of images and sounds stored in the database 250. Determine the image and sound required for generation. Then, the back-end server 270 reads the determined image and sound data from the database 250, and generates virtual viewpoint content by performing rendering processing and the like using the read data.

  The configuration of the image computing server 200 is not limited to the configuration example of FIG. For example, at least two of the front end server 230, the database 250, and the back end server 270 may be configured integrally. A plurality of at least one of the front-end server 230, the database 250, and the back-end server 270 may be included. In addition, devices other than these devices may be included in arbitrary positions in the image computing server 200. Furthermore, at least part of the functions of the image computing server 200 may be included in the end user terminal 190 and the virtual camera operation UI 330.

  The back-end server 270 generates virtual viewpoint content based on captured images (multi-viewpoint images) captured by the plurality of cameras 112 and viewpoint information by rendering processing. More specifically, the back-end server 270 includes image data of a predetermined area extracted by the corresponding camera adapter 120 from the captured images acquired by the plurality of cameras 112, and a viewpoint specified by a user operation. Based on the above, virtual viewpoint content is generated. Then, the back end server 270 transmits the generated virtual viewpoint content image and sound to the end user terminal 190. The end user terminal 190 has a display and a speaker, displays an image of the virtual viewpoint content transmitted from the back-end server 270 on the display, and outputs sound from the speaker.

  The virtual viewpoint content in the present embodiment is content including a virtual viewpoint image as an image obtained when a subject is photographed from a virtual viewpoint. In other words, the virtual viewpoint image is an image representing the appearance at the designated viewpoint. The virtual viewpoint (virtual viewpoint) may be arbitrarily specified by the user, or may be automatically specified based on the result of image analysis or the like. That is, the virtual viewpoint image includes an arbitrary viewpoint image (free viewpoint image) corresponding to the viewpoint arbitrarily designated by the user. An image corresponding to a viewpoint designated by the user from a plurality of viewpoint candidates and an image corresponding to a viewpoint automatically designated by the apparatus are also included in the virtual viewpoint image. In the present embodiment, the virtual viewpoint content is described with an example in which audio (audio data) is included, but the audio may not necessarily include audio. Further, the back-end server 270 converts the virtual viewpoint image into the H.264 format. The data may be compressed and encoded by a standard technique represented by H.264 or HEVC and transmitted to the end user terminal 190 using the MPEG-DASH protocol. Therefore, according to the present embodiment, the user who operates the end user terminal 190 can view a virtual viewpoint image and sound corresponding to an arbitrary viewpoint designated by the user, for example.

  As described above, the image processing system 100 according to the present embodiment has three functional domains: a video collection domain, a data storage domain, and a video generation domain. That is, in the present embodiment, the video collection domain includes the sensor systems 110a to 110z, and the data storage domain includes the database 250, the front end server 230, and the back end server 270. The video generation domain includes a virtual camera operation UI 330 and an end user terminal 190. For example, the virtual camera operation UI 330 can directly acquire images from the sensor systems 110a to 110z.

  However, in the image processing system 100 according to the present embodiment, for example, the virtual camera operation UI 330 is not configured to directly acquire images from the sensor systems 110a to 110z, but as illustrated in FIG. Is adopted. Specifically, the front-end server 230 of the data storage domain converts the image and sound acquired by the sensor systems 110 a to 110 z and meta information of the data into the common schema and data type of the database 250. Thereby, even if the camera 112 of the sensor systems 110a to 110z is changed to a camera of another model, for example, the difference due to the camera model change can be absorbed by the front end server 230 and registered in the database 250. As a result, when the camera 112 is changed to a camera of another model, it is possible to prevent a situation in which the virtual camera operation UI 330 stops operating properly.

  In the image processing system 100 according to the present embodiment, the virtual camera operation UI 330 is configured to be accessed via the back-end server 270 without directly accessing the database 250. That is, common processing related to image generation processing is performed by the back-end server 270, and a difference portion of an application related to the operation UI is performed by the virtual camera operation UI 330. Thus, for example, in the development of the virtual camera operation UI 330, it becomes possible to focus on the development of UI function devices and UI function requirements for operating a virtual viewpoint image to be generated. Further, the back-end server 270 can add or delete common processing related to image generation processing in response to a request from the virtual camera operation UI 330. As a result, it is possible to flexibly respond to the request of the virtual camera operation UI 330. As described above, in the image processing system 100 of the present embodiment, the virtual viewpoint image is generated by the back-end server 270 based on the image data obtained by photographing with the plurality of cameras 112 for photographing the subject from a plurality of directions. Generated. Note that the image processing system 100 according to the present embodiment is not limited to the physical configuration described above, but may be logically configured.

Next, functions of the camera adapter 120, the front-end server 230, and the back-end server 270 in the image processing system 100 according to the present embodiment will be described in detail with reference to FIGS.
FIG. 2 is a diagram illustrating functional blocks of the camera adapter 120 of the present embodiment.
As shown in FIG. 2, the camera adapter 120 includes a network adapter 6110, a transmission unit 6120, an image processing unit 6130, and an external device control unit 6140.

The network adapter 6110 includes a data transmission / reception unit 6111 and a time control unit 6112.
The data transmission / reception unit 6111 performs data communication with other camera adapters 120, the front-end server 230, the time server 290, the control station 310, and the like via the daisy chain 170, the switching hub 180, the networks 291 and 310a, and the like. The data transmission / reception unit 6111 outputs the foreground image and the background image separated from the captured image of the camera 112 by the foreground / background separation unit 6131 described later to another camera adapter 120.
Further, the data transmitting / receiving unit 6111 has a function of outputting the foreground image and the background image at different frame rates. In the case of this embodiment, the foreground image is output at a high frame rate, and the background image that does not include the shooting target (predetermined region) is output at a low frame rate. The output destination camera adapter 120 is the next camera adapter 120 in the order determined in advance according to the processing of the data routing processing unit 6122 described below among the camera adapters 120 in the image processing system 100. As each camera adapter 120 outputs the foreground image and the background image, the back-end server 270 can generate a virtual viewpoint image based on the foreground image and the background image taken from a plurality of viewpoints.

  The time control unit 6112 is based on, for example, the IEEE 1588 standard Ordinary Clock, and has a function of storing time stamps of data transmitted to and received from the time server 290, and performs time synchronization with the time server 290. Time synchronization is not limited to IEEE 1588, and time synchronization with a time server may be realized by other EtherAVB standards or a unique protocol.

  The transmission unit 6120 has a function of controlling data transmission to the switching hub 180 and the like via the network adapter 6110. The transmission unit 6120 includes functional units such as a data compression / decompression unit 6121, a data routing processing unit 6122, a time synchronization control unit 6123, an image / audio transmission processing unit 6124, and a data routing information holding unit 6125. .

The data routing information holding unit 6125 has a function of holding address information and the like for determining a transmission destination of data transmitted / received by the data transmitting / receiving unit 6111.
The data compression / decompression unit 6121 has a function of compressing data transmitted / received via the data transmission / reception unit 6111 by applying a predetermined compression method, compression rate, and frame rate, and a function of decompressing the compressed data And have.

  The data routing processing unit 6122 determines the routing destination of the data received by the data transmission / reception unit 6111 and the data processed by the image processing unit 6130 based on the address information held by the data routing information holding unit 6125. Further, the data routing processing unit 6122 also has a function of transmitting data to the determined routing destination. As a routing destination, the camera adapter 120 corresponding to the camera 112 focused on the same gazing point is suitable for performing image processing because the image frame correlation between the cameras 112 is high. In accordance with the determination by the data routing processing unit 6122 for each of the plurality of camera adapters 120, the order of the camera adapters 120 that output the foreground image and the background image in the relay format in the image processing system 100 is determined.

  The time synchronization control unit 6123 conforms to the IEEE 1588 standard PTP (Precision Time Protocol) and has a function of performing processing related to time synchronization with the time server 290. Note that time synchronization may be performed using not only PTP but also other similar protocols.

  The image / audio transmission processing unit 6124 has a function of creating a message for transferring image or audio data to another camera adapter 120 or the front-end server 230 via the data transmission / reception unit 6111. The message includes image or audio data and meta information of each data. In the case of the present embodiment, the meta information includes a time code or sequence number when an image is taken or audio is sampled, a data type, an identifier indicating the individual of the camera 112 or the microphone 111, and the like. The image / sound transmission processing unit 6124 receives a message from another camera adapter 120 via the data transmission / reception unit 6111. Then, the image / sound transmission processing unit 6124 restores the data information fragmented to the packet size defined by the transmission protocol into image or sound data according to the data type included in the message. If the data is compressed when the data is restored, the data compression / decompression unit 6121 performs decompression processing corresponding to the compression.

  The image processing unit 6130 has a function of processing image data captured by the camera 112 under the control of the camera control unit 6141 and image data received from another camera adapter 120. The image processing unit 6130 includes functional units such as a foreground / background separation unit 6131, a 3D model information generation unit 6132, and a calibration control unit 6133.

  The foreground / background separator 6131 has a function of separating image data captured by the camera 112 into a foreground image and a background image. That is, the foreground / background separation unit 6131 of each of the plurality of camera adapters 120 extracts a predetermined area from the image captured by the corresponding camera 112. The predetermined area in the present embodiment is, for example, a foreground image obtained as a result of object detection on the captured image, and the foreground / background separation unit 6131 extracts the predetermined area to separate the captured image into the foreground image and the background image. To do. Note that when the image processing system 100 is installed in a stadium, for example, as in the present embodiment, a person can be cited as an example of an object for a captured image. The person object in this case may be a specific person (for example, a player, a director, a referee, etc.), or may be an object with a predetermined image pattern such as a ball or a goal. The object may be not only a moving object such as a person but also a stationary object. In the image processing system 100 according to the present embodiment, the foreground image including important objects such as a person and the background area not including these objects are processed separately to correspond to the object of the generated virtual viewpoint image. The quality of the image of the part can be improved. Further, according to the present embodiment, the foreground image and the background image are separated by each of the plurality of camera adapters 120, whereby the load on the image processing system 100 including the plurality of cameras 112 can be distributed. The predetermined area is not limited to the foreground image, and may be a background image, for example.

  The 3D model information generation unit 6132 uses the foreground image separated by the foreground / background separation unit 6131 and the foreground image received from the other camera adapter 120, and uses the principle of a stereo camera, for example, to obtain image information related to the 3D model. It has the function to generate. Further, for example, a method using Visual Hull may be used for generating the three-dimensional model.

  The calibration control unit 6133 has a function of acquiring image data necessary for calibration from the camera 112 via the camera control unit 6141 to be described later and transmitting the image data to the front-end server 230 that performs arithmetic processing related to calibration. doing.

  The external device control unit 6140 includes functional units such as a camera control unit 6141, a microphone control unit 6142, a pan head control unit 6143, and a sensor control unit 6144. The camera control unit 6141 is connected to the camera 112 provided in the camera adapter 120. Similarly, the microphone control unit 6142 is the microphone 111, the camera platform control unit 6143 is the camera platform 113, and the sensor control unit 6144 is an external sensor. 114.

  The camera control unit 6141 has a function of controlling the connected camera 112, acquiring a captured image, providing a synchronization signal, setting a time, and the like. For the control of the camera 112, for example, setting and referring to shooting parameters (number of pixels, color depth, frame rate, white balance, etc.), acquisition of the state of the camera 112 (shooting, stopping, synchronizing, error, etc.), shooting Start and stop, focus adjustment, etc. The synchronization signal is provided by sending the shooting timing (control clock) to the camera 112 using the time synchronized with the time server 290 by the time synchronization control unit 6123. The time setting is performed by the time synchronization control unit 6123 sending the time synchronized with the time server 290 using, for example, a time code conforming to the SMPTE12M format. As a result, the time code is assigned to the image data received from the camera 112. Note that the format of the time code is not limited to SMPTE12M, and may be another format. In addition, the camera control unit 6141 may add time code to the image data received from the camera 112 without transmitting the time code to the camera 112.

  The microphone control unit 6142 has functions of controlling the connected microphone 111, starting and stopping sound collection, obtaining collected sound data, and the like. Examples of the control of the microphone 111 include gain adjustment and state acquisition. Similarly to the case of the camera control unit 6141, the audio sampling timing and time code are transmitted to the microphone 111. As clock information used as the timing of audio sampling, information obtained by converting time information from the time server 290 into, for example, a 48 KHz word clock is used.

The pan head control unit 6143 has a function of controlling the pan head 113 connected thereto. Control of the pan head 113 includes, for example, so-called pan / tilt control and state acquisition.
The sensor control unit 6144 has a function of acquiring sensor information sensed by the connected external sensor 114. For example, when a gyro sensor is used as the external sensor 114, the sensor control unit 6144 can acquire information representing vibration. The vibration information acquired by the sensor control unit 6144 is sent to the image processing unit 6130. Accordingly, the image processing unit 6130 can generate an image in which vibration is suppressed prior to the processing in the foreground / background separation unit 6131 using the vibration information acquired by the sensor control unit 6144.

FIG. 3 is a diagram showing functional blocks of the front-end server 230 of the present embodiment.
As shown in FIG. 3, the front-end server 230 includes a data input control unit 2120, a data synchronization unit 2130, an image processing unit 2150, a 3D model combination unit 2160, an image combination unit 2170, and a captured data file generation unit 2180. Further, the front end server 230 includes a control unit 2110, a CAD data storage unit 2135, a calibration unit 2140, a non-photographed data file generation unit 2185, and a DB access control unit 2190.

  The control unit 2110 includes a CPU, DRAM, a storage medium such as an HDD or NAND memory that stores program data and various data, and hardware such as Ethernet (registered trademark). The control unit 2110 controls each functional block of the front end server 230 and the entire system of the front end server 230. In addition, the control unit 2110 performs mode control to switch operation modes such as a calibration operation, a preparatory operation before photographing, and an operation during photographing. Further, the control unit 2110 receives a control instruction from the control station 310 via the network, and performs switching of each mode, input / output of data, and the like. The control unit 2110 acquires stadium CAD data from the control station 310 via the network, and transmits the stadium CAD data to the CAD data storage unit 2135 and the non-photographed data file generation unit 2185.

  The data input control unit 2120 is connected to the camera adapter 120 via the network and the switching hub 180. Then, the data input control unit 2120 acquires the above-described foreground image, background image, three-dimensional model of a subject such as the above-described object, audio data, and camera exposure information from the camera adapter 120. In addition, the data input control unit 2120 transmits the acquired foreground image and background image to the data synchronization unit 2130, and transmits camera calibration captured image data to the calibration unit 2140. The data input control unit 2120 has a function of performing compression / decompression of received data, data routing processing, and the like. Here, both the control unit 2110 and the data input control unit 2120 have a communication function via a network such as Ethernet (registered trademark), but the communication function may be shared by them. In that case, a method may be used in which the data input control unit 2120 receives an instruction based on a control command from the control station 310 or stadium CAD data and sends it to the control unit 2110.

  The data synchronizer 2130 temporarily stores data acquired from the camera adapter 120 on a DRAM (not shown), and buffers until the foreground image, background image, audio data, 3D model data, and camera exposure information are complete. To do. The foreground image, the background image, the sound data, the three-dimensional model data, and the camera exposure information are collectively referred to as shooting data in the following description. Meta information such as routing information, time code information (time information), and a camera identifier is added to the photographing data, and the data synchronization unit 2130 confirms the data attribute based on the meta information. As a result, the data synchronization unit 2130 determines that the data is ready by determining that the data is at the same time. When the data is ready, the data synchronization unit 2130 transmits the foreground image and the background image to the image processing unit 2150, the 3D model data to the 3D model combination unit 2160, and the audio data to the captured data file generation unit 2180. Note that the data aligned here is data necessary for generating a file in the shooting data file generation unit 2180 described later.

  The CAD data storage unit 2135 stores the three-dimensional data indicating the stadium shape received from the control unit 2110 (hereinafter referred to as stadium shape data) in a storage medium such as a DRAM, HDD, or NAND memory (not shown). The CAD data storage unit 2135 transmits the stadium shape data stored when the stadium shape data request is received to the image combining unit 2170.

  The calibration unit 2140 performs a calibration operation of the camera 112 and sends camera parameters obtained by the calibration to a non-photographed data file generation unit 2185 described later. At the same time, the calibration unit 2140 stores camera parameters in its own storage area, and transmits information about the stored camera parameters to a 3D model combining unit 2160 described later.

  The image processing unit 2150 matches the foreground image and the background image supplied via the data synchronization unit 2130 with colors and brightness values between cameras, and when RAW image data is input, develops the camera, Processing such as correction of lens distortion is performed. Then, the image processing unit 2150 transmits the foreground image subjected to the image processing to the shooting data file generation unit 2180 and the background image to the image combination unit 2170, respectively.

  The 3D model combining unit 2160 combines the 3D model data at the same time supplied via the data synchronization unit 2130 using the camera parameters generated by the calibration unit 2140. Then, the three-dimensional model combining unit 2160 generates three-dimensional model data of the foreground image in the entire stadium using a method called “VisualHull”. The 3D model generated by the 3D model combining unit 2160 is sent to the imaging data file generating unit 2180.

  The image combining unit 2170 acquires a background image from the image processing unit 2150, acquires stadium shape data from the CAD data storage unit 2135, and specifies the position of the background image with respect to the coordinates of the stadium shape data. When the position with respect to the coordinates of the stadium shape data can be specified for each of the background images, the image combining unit 2170 combines the background images into one background image. Note that the back-end server 270 may perform the creation of the three-dimensional shape data of the background image.

  The shooting data file generation unit 2180 receives audio and camera exposure information from the data synchronization unit 2130, foreground images from the image processing unit 2150, 3D model data from the 3D model combining unit 2160, and 3D shape from the image combining unit 2170. The background image combined with is acquired. Then, the shooting data file generation unit 2180 outputs the acquired data to the DB access control unit 2190. Here, the imaging data file generation unit 2180 outputs these data in association with each other based on the time information. Note that the shooting data file generation unit 2180 may output a part of these data in association with each other. For example, the shooting data file generation unit 2180 outputs the foreground image and the background image in association with each other based on the time information of the foreground image and the time information of the background image. Since the frame rate of the background image output from the camera adapter 120 is lower than the frame rate of the foreground image, the background image may not exist at a certain time. In this case, the shooting data file generation unit 2180 does not associate background images. Further, the shooting data file generation unit 2180 outputs the foreground image, the background image, and the 3D model data in association with each other based on the time information of the foreground image, the time information of the background image, and the time information of the 3D model data. To do. Note that the shooting data file generation unit 2180 may output the associated data as a file for each type of data, or output a plurality of types of data collectively as files for each time indicated by the time information. Also good. Then, the imaging data associated in this way is output to the database 250 via the DB access control unit 2190. As a result, the back-end server 270 can generate a virtual viewpoint image from the foreground image and the background image corresponding to the time information.

  The non-photographed data file generation unit 2185 acquires camera parameters from the calibration unit 2140 and stadium shape data from the control unit 2110, shapes them according to the file format, and transmits them to the DB access control unit 2190. Note that the non-photographed data file generation unit 2185 individually shapes camera parameters or stadium shape data, which are input data, according to the file format. That is, the non-photographed data file generation unit 2185, when receiving either one of the camera parameters and the stadium shape data, individually transmits them to the DB access control unit 2190.

  The DB access control unit 2190 is connected to the database 250 so that high-speed communication is possible by so-called InfiniBand. Then, the DB access control unit 2190 transmits the data received from the shooting data file generation unit 2180 and the non-shooting data file generation unit 2185 to the database 250. In the case of the present embodiment, the shooting data associated with the shooting data file generation unit 2180 based on the time information is output to the database 250 via the DB access control unit 2190.

  In the present embodiment, the front-end server 230 associates the foreground image with the background image. However, the present invention is not limited to this, and the database 250 may perform the association. In this case, the database 250 acquires a foreground image and a background image having time information from the front end server 230. Then, the database 250 may associate the foreground image and the background image with each other based on the time information of the foreground image and the time information of the background image, and output them to the storage unit included in the database 250.

FIG. 4 is a diagram illustrating functional blocks of the back-end server 270 according to the present embodiment.
As shown in FIG. 4, the back-end server 270 includes a data receiving unit 3001, a foreground texture determining unit 3003, a texture boundary color matching unit 3004, and a virtual viewpoint foreground image generating unit 3005 for foreground processing. Further, the back-end server 270 includes a background texture pasting unit 3002, a rendering mode management unit 3014, and a rendering unit 3006. Further, the back-end server 270 includes a virtual viewpoint audio generation unit 3007, a synthesis unit 3008, and an image output unit 3009. The back-end server 270 also includes a foreground object determination unit 3010, a request list generation unit 3011, and a request data output unit 3012 for reading out foreground data. In addition, the back-end server 270 includes a foreground image camera exposure information acquisition unit 3015, a background image camera exposure information acquisition unit 3016, a camera exposure information comparison unit 3017, a background image acquisition determination unit 3018, and a background image request unit 3019.

  The data receiving unit 3001 receives data transmitted from the database 250 and the controller 300. The data receiving unit 3001 receives stadium shape data, foreground image, background image, three-dimensional model of foreground image (hereinafter referred to as foreground three-dimensional model), and audio from the database 250. Furthermore, the data receiving unit 3001 also receives camera exposure information associated with the received foreground image and background image from the database 250.

  The data receiving unit 3001 receives from the controller 300 virtual camera parameters that specify a viewpoint related to generation of a virtual viewpoint image. The virtual camera parameter is data representing the position and orientation of the virtual viewpoint, and for example, an external parameter matrix and an internal parameter matrix are used. Note that data that the data receiving unit 3001 acquires from the controller 300 is not limited to virtual camera parameters. The information output from the controller 300 includes at least one of a viewpoint designation method, information that identifies an application that the controller 300 is operating, identification information of the controller 300, and identification information of a user who uses the controller 300. May be. Further, the data receiving unit 3001 may acquire the same information as the information output from the controller 300 from the end user terminal 190. Further, the data receiving unit 3001 may acquire information regarding the plurality of cameras 112 from devices such as the database 250 and the controller 300. The information regarding the plurality of cameras 112 is, for example, information regarding the number of the plurality of cameras 112, information regarding the operation state of the plurality of cameras 112, and the like. The operation state of the camera 112 includes, for example, at least one of a normal state, a failure state, a standby state, a start state, and a restart state of the camera 112.

  The background texture pasting unit 3002 acquires a background mesh model (stadium shape data) from the background mesh model management unit 3013. The background texture pasting unit 3002 creates a textured background mesh model by pasting the background image acquired from the database 250 as a texture to the three-dimensional space shape indicated by the background mesh model. A mesh model is data that represents a three-dimensional space shape, such as CAD data, as a set of surfaces. A texture is an image that is pasted to express the texture of the surface of an object. However, the background mesh model and the background image may not exist in the data received by the data receiving unit 3001 from the database 250. In this case, the background texture pasting unit 3002 generates a textured background mesh model using the background mesh model and the background image received from the database 250 immediately before.

The foreground texture determination unit 3003 determines texture information of the foreground 3D model from the foreground image and the foreground 3D model group.
The foreground texture boundary color matching unit 3004 performs texture matching between the texture information of each foreground 3D model and each 3D model group, and generates a colored foreground 3D model group for each foreground object.
The virtual viewpoint foreground image generation unit 3005 performs perspective transformation so that the foreground image group looks from the virtual viewpoint based on the virtual camera parameters.

  The rendering unit 3006 renders the background image and the foreground image based on the method used for generating the virtual viewpoint image determined by the rendering mode management unit 3014 to generate a virtual viewpoint image of the entire view. Note that examples of a virtual viewpoint image generation method include model-based rendering (Model-Based Rendering: MBR) and image-based (Image-Based Rendering: IBR).

  The rendering mode management unit 3014 determines a rendering mode to be used from among a plurality of rendering modes. This determination is made based on information acquired by the data receiving unit 3001. In the present embodiment, it is possible to flexibly configure the system by switching a plurality of rendering modes as required, and the image processing system 100 of the present embodiment can be applied to subjects other than the stadium. Applicable. Note that the rendering mode held by the rendering mode management unit 3014 may be a system preset in the system.

The virtual viewpoint sound generation unit 3007 generates sound (sound group) that can be heard at the virtual viewpoint based on the virtual camera parameters.
The synthesizing unit 3008 synthesizes the image group generated by the rendering unit 3006 and the audio generated by the virtual viewpoint audio generating unit 3007 to generate virtual viewpoint content. Note that the back-end server 270 may output the virtual viewpoint image that is generated by the rendering unit 3006 and does not include sound.
The image output unit 3009 outputs the virtual viewpoint content generated by the synthesis unit 3008 to the controller 300 and the end user terminal 190 via the network. However, the transmission to the outside is not limited to a network such as Ethernet (registered trademark), and a signal transmission path such as SDI, Display Port, and HDMI (registered trademark) may be used.

  The foreground object determination unit 3010 determines the foreground object group to be displayed from the virtual camera parameters and the position information of the foreground object indicating the position of the foreground object included in the foreground 3D model, and outputs the foreground object list To do. That is, the foreground object determination unit 3010 performs a process of mapping the virtual viewpoint image information to the physical camera 112. This virtual viewpoint has different mapping results depending on the rendering mode determined by the rendering mode management unit 3014. Therefore, the foreground object determination unit 3010 includes a control unit that determines a plurality of foreground objects, and the control unit performs control in conjunction with the rendering mode.

  The request list generation unit 3011 generates a request list for requesting the database 250 for the foreground image group, the foreground 3D model group, the background image, and the audio data corresponding to the foreground object list at the specified time. For the foreground object, data selected in consideration of the virtual viewpoint is requested from the database 250, but for the background image and the audio data, all data relating to the frame is requested. After the back-end server 270 is started, a background mesh model request list is generated until the background mesh model is acquired. The request list also includes camera exposure information associated with the requested foreground image group and background image.

The request data output unit 3012 outputs a data request command to the database 250 based on the input request list.
The background mesh model management unit 3013 stores the background mesh model received from the database 250.
The foreground image camera exposure information acquisition unit 3015 acquires all camera exposure information associated with the foreground image group acquired from the database 250.
The background image camera exposure information acquisition unit 3016 acquires camera exposure information associated with the background image acquired from the database 250.

  The camera exposure information comparison unit 3017 compares all the camera exposure information acquired by the foreground image camera exposure information acquisition unit 3015 with the camera exposure information acquired by the background image camera exposure information acquisition unit 3016. In the case of the present embodiment, the camera exposure information comparison unit 3017 performs comparison from the viewpoint of whether the camera exposure information of the background image matches the camera exposure information of the foreground image group. First, the camera exposure information comparison unit 3017 confirms whether the EV value of the camera exposure information of each foreground image is the same for the acquired camera exposure information of the foreground image group. Next, when there are a plurality of different numerical values among the camera exposure information of the foreground image group, the camera exposure information comparison unit 3017 determines the EV value of the camera exposure information having the largest number for each exposure information as the camera of the foreground image. Let it be exposure information. Then, the camera exposure information comparison unit 3017 calculates the difference between the EV value of the camera exposure information of the foreground image and the EV value of the camera exposure information of the background image acquired by the background image camera exposure information acquisition unit 3016. The value is output to the background image acquisition determination unit 3018. In the present embodiment, the difference between the EV values of the foreground image and the background image is used as the comparison result, but a flag indicating a match or mismatch may be used.

  The background image acquisition determination unit 3018 generates and outputs a background image reacquisition instruction according to the comparison result by the camera exposure information comparison unit 3017. For example, when the comparison result indicates that the camera exposure information of the foreground image and the background image match, the background image acquisition determination unit 3018 outputs the generated textured background mesh model to the background texture pasting unit 3002 to the rendering unit 3006. Instruct. On the other hand, for example, when the comparison result indicates a mismatch between the camera exposure information of the foreground image and the background image, the background image acquisition determination unit 3018 instructs the background image request unit 3019 to reacquire the background image. At that time, the background image acquisition determination unit 3018 also acquires the camera exposure information of the foreground image from the camera exposure information comparison unit 3017 and outputs it to the background image request unit 3019.

  In response to an instruction from the background image acquisition determination unit 3018, the background image request unit 3019 outputs a command for requesting a background image having camera exposure information that matches the camera exposure information of the foreground image to the database 250. In the present embodiment, the background image requested from the database 250 is a background image having camera exposure information that matches the camera exposure information of the foreground image among the background images that can be used to generate the virtual viewpoint image. Specifically, the background image requested from the database 250 is a background image at the closest shooting time that is a time after the shooting time in the background image acquired immediately before by the data receiving unit 3001. As a result, background image data having the same camera exposure information as the camera exposure information of the foreground image is input to the background texture pasting 03002.

  In this embodiment, the case where the back-end server 270 performs both the determination of the generation method of the virtual viewpoint image and the generation of the virtual viewpoint image will be mainly described, but the present invention is not limited to this. That is, the device that has determined the generation method may output data corresponding to the determination result. For example, the front-end server 230 determines a generation method used for generating a virtual viewpoint image based on information on a plurality of cameras 112, information output from a device that specifies a viewpoint for generating a virtual viewpoint image, and the like. May be. Then, the front-end server 230 outputs the image data based on the photographing by the camera 112 and the information indicating the determined generation method to at least one of a storage device such as the database 250 and an image generation device such as the back-end server 270. May be. In this case, for example, the back-end server 270 generates a virtual viewpoint image based on information indicating the generation method output by the front-end server 230. As described above, the front-end server 230 determines the generation method, thereby reducing the processing load caused by the database 250 and the back-end server 270 processing data for image generation using a method different from the determined method. Can be reduced. On the other hand, when the back-end server 270 determines a generation method as in the present embodiment, the database 250 holds data that can correspond to a plurality of generation methods, and thus a plurality of virtual models corresponding to each of the plurality of generation methods. A viewpoint image can be generated.

Next, a virtual viewpoint image generation method according to this embodiment will be described with reference to FIG.
FIG. 5 illustrates the virtual camera operation UI 330, the back-end server 270, and the database from when the operator (user) operates the virtual camera operation UI 330 to when the virtual viewpoint image is generated and displayed on the end user terminal 190. 250 processing flow.

  First, in S3300 of FIG. 5, the virtual camera operation UI 330 obtains an input for operating the virtual camera from an operator (user). As an input device to the virtual camera operation UI 330, a joystick, a jog dial, a touch panel, a keyboard, a mouse, and the like are used. Here, as an operation of the virtual camera by the operator, for example, it is assumed that an operation instruction such as the position and orientation of the virtual camera and the zoom magnification is input. When a virtual camera operation instruction is input from the operator, the virtual camera operation UI 330 derives (calculates) a virtual camera parameter corresponding to the operation instruction in S3301. The virtual camera parameters in this case include external parameters indicating the position and orientation of the virtual camera and internal parameters indicating the zoom magnification of the virtual camera. In step S3302, the virtual camera operation UI 330 transmits the derived virtual camera parameter to the back-end server 270.

  When receiving the virtual camera parameter, the back-end server 270 requests the foreground three-dimensional model group from the database 250 in S3303. In step S <b> 3304, the database 250 transmits a foreground three-dimensional model group including foreground object position information to the back-end server 270 in response to a request from the back-end server 270. Thereby, in S3305, the back-end server 270 geometrically derives (calculates) a foreground object group that falls within the field of view of the virtual camera based on the virtual camera parameters and the position information of the foreground objects included in the foreground three-dimensional model. To do. In step S <b> 3306, the back-end server 270 transmits a request for requesting the foreground object group to the database 250.

  Upon receiving a request from the backend server 270, the database 250 reads out data corresponding to the request and transmits it to the backend server 270 in S3307. When the back-end server 270 receives data from the database 250, in S3308, the back-end server 270 calculates the EV value of the camera exposure information of the foreground image from all the camera exposure information associated with the foreground image of the foreground object group. Further, the back-end server 270 compares the calculated EV value with the camera exposure information EV value of the received background image. The back-end server 270 uses the average EV value of the plurality of foreground images used for generating the virtual viewpoint image as the camera exposure information EV value of the foreground image, and uses the average EV value of the plurality of background images used for generating the virtual viewpoint image. The camera exposure information EV value of the background image may be used. Note that the back-end server 270 may determine whether or not the difference between the camera exposure information EV value of the foreground image and the camera exposure information EV value of the background image is smaller than a predetermined threshold value. If the camera exposure information (EV value) of the foreground image and the background image does not match, the back-end server 270 requests the database 250 to transmit the background image associated with the same camera exposure information as the foreground image in S3309. To do. Note that the back-end server 270 may acquire a plurality of foreground images used for generating a virtual viewpoint image, and request a background image having an EV value determined based on the acquired foreground image from the database 250.

  Upon receiving the request from the back-end server 270, the database 250 reads out the background image data associated with the same camera exposure information as the foreground image and transmits it to the back-end server 270 in response to the request in S3310. . In S3311, the back-end server 270 generates a virtual viewpoint foreground image and a background image based on the foreground image, the foreground three-dimensional model, and the background image received from the database 250, and synthesizes them to generate a virtual viewpoint whole scene image. Generate. Further, in S3311, the back-end server 270 synthesizes audio data according to the position of the virtual camera based on the audio data group, and integrates it with the full-view image of the virtual viewpoint to generate a virtual viewpoint image and audio. Then, the back-end server 270 transmits the generated virtual viewpoint image and sound to the virtual camera operation UI 330 in S3312. The virtual camera operation UI 330 realizes display of a captured image of the virtual camera by displaying the received image on the end user terminal 190, for example.

As described above, in the image processing system 100 according to the first embodiment that outputs the foreground image and the background image at different frame rates, when the camera exposure information differs between the foreground image and the background image due to a change in shooting parameters at the time of shooting or the like. A method for generating a virtual viewpoint image has been described.
In the image processing system 100 of the present embodiment, the camera exposure information of the foreground image and the background image used for generating the virtual viewpoint image can be matched. Thus, according to the image processing system 100 of the present embodiment, when generating a virtual viewpoint image based on image data captured by a plurality of cameras 112, the foreground image and the background image with matching camera exposure information are displayed. It is possible to generate a virtual viewpoint image. That is, according to the present embodiment, it is possible to generate a natural virtual viewpoint image without unnaturalness such that the luminance becomes discontinuous. Note that if the camera exposure information differs between the foreground image and the background image, the back-end server 270 gives priority to the background image, and generates the virtual viewpoint image from the foreground image captured with the exposure when the background image is captured. It is good also as a structure to use.

<Second Embodiment>
Hereinafter, as the second embodiment, when the exposure information of the foreground image and the background image used for generating the virtual viewpoint image is different due to the shooting parameter change at the time of shooting or the like, the image quality of the background image is adjusted to be suitable for the foreground image. A method for generating a virtual viewpoint image will be described.

  The schematic configuration of the image processing system 100 in the second embodiment is the same as that of the first embodiment described above. Since the sensor system 110, the switching hub 180, the end user terminal 190, the front end server 230, the database 250, the time server 290, the control station 310, and the virtual camera operation UI 330 are the same as described above, the description thereof is omitted.

  FIG. 6 is a diagram illustrating functional blocks of the back-end server 270 in the second embodiment. The back-end server 270 according to the second embodiment does not include the camera exposure information comparison unit 3017, the background image acquisition determination unit 3018, and the background image request unit 3019 of the back-end server 270 illustrated in FIG. An image quality correction unit 3020 is provided. Note that the data receiving unit 3001 to background image camera exposure information acquisition unit 3016 in FIG. 6 have substantially the same configuration as the data reception unit 3001 to background image camera exposure information acquisition unit 3016 shown in FIG. Omitted.

The background image image quality correction unit 3020 corrects the image quality of the background image based on the camera exposure information of the foreground image and the background image. Therefore, the background image quality correction unit 3020 first acquires all the camera exposure information acquired by the foreground image camera exposure information acquisition unit 3015 and the camera exposure information acquired by the background image camera exposure information acquisition unit 3016. Also in the second embodiment, as in the first embodiment, when there are different numerical values among the camera exposure information of the foreground image group, the largest number of camera exposures among the camera exposure information of the plurality of foreground images. The EV value of the information is used as camera exposure information for the foreground image. The background image image quality correction unit 3020 corrects the image quality of the background image supplied from the database 250 via the data reception unit 3001 based on the camera exposure information of the foreground image and the background image. Specifically, when the camera exposure information of the foreground image is expressed as EVf, the camera exposure information of the background image is expressed as EVb, and the background image data is expressed as BP, the background image quality correction unit 3020 corrects by performing the following equation (2). Later background image data BPc is calculated.
BPc = BP * 2 ^(EVf-EVb) formula (2)

  In the case of the second embodiment, the background image (BPc) after the image quality correction by the background image image quality correction unit 3020 is output to the background texture pasting unit 3002. In the case of the second embodiment, the background texture pasting unit 3002 creates a textured background mesh model by pasting the background image after image quality correction into a three-dimensional space shape.

  When the camera exposure information of the foreground image and the camera exposure information of the background image are different from each other, the back-end server 270 of the second embodiment performs unnaturalness by correcting the image quality of the background image based on the camera exposure information of the foreground image. A virtual viewpoint image can be obtained. Further, the back-end server 270 can associate the brightness and color of the foreground image with the brightness and color of the background image by performing image processing of the background image based on the camera exposure information of the foreground image. Therefore, since a virtual viewpoint image is generated using such a foreground image and a background image, a virtual viewpoint image without unnaturalness can be obtained.

  In the case of the second embodiment, the background image correction processing by the background image image quality correction unit 3020 is executed every time the back-end server 270 acquires the foreground image and the background image from the database 250. It is not limited. For example, the background image quality correction unit 3020 compares the camera exposure information of the foreground image and the camera exposure information of the background image, and corrects the background image data only when these two camera exposure information does not match. May be.

  Further, when there are different values among the camera exposure information of the foreground image group, the back-end server 270 of this embodiment uses the EV value of the most camera exposure information as the camera exposure information of the foreground image, and other EVs. A foreground image having a value may be corrected in the same manner as the background image. This makes it possible to easily adjust the brightness mismatch between the foreground image and the background image when the exposure differs between the cameras. Note that the back-end server 270 may be configured to correct the tone and brightness of the foreground image by performing image processing so that the foreground image corresponds to the background image when the camera exposure information differs between the foreground image and the background image. .

<Third Embodiment>
In the above-described embodiment, when the camera exposure information of the foreground image and the camera exposure information of the background image no longer match, an appropriate background image is reacquired (first embodiment), or the image quality of the background image is corrected (first image). Embodiment 2) is performed, and the brightness mismatch between the foreground image and the background image is eliminated. However, for example, when the camera exposure information of the foreground image is changed abruptly before and after a certain time, the generated virtual viewpoint image becomes an unnatural image with luminance discontinuity in the time direction. There is a risk of it.

Therefore, in the third embodiment, even when the camera exposure information of the foreground image is abruptly changed at a certain time, it is possible to generate a natural virtual viewpoint image in which the luminance does not become discontinuous in the time direction. The process of making will be described.
The schematic configuration of the image processing system 100 in the third embodiment is the same as that of the first embodiment described above. Since the sensor system 110, the switching hub 180, the end user terminal 190, the front end server 230, the database 250, the time server 290, the control station 310, and the virtual camera operation UI 330 are the same as described above, the description thereof is omitted.

  FIG. 7 is a diagram illustrating functional blocks of the back-end server 270 according to the third embodiment. The back-end server 270 of the third embodiment does not include the camera exposure information comparison unit 3017, the background image acquisition determination unit 3018, and the background image request unit 3019 of the back-end server 270 of FIG. 3020 and a time information request unit 3021. Note that the data receiving unit 3001 to the background image camera exposure information acquisition unit 3016 in FIG. 7 have substantially the same configuration as the data reception unit 3001 to the background image camera exposure information acquisition unit 3016 shown in FIG. Omitted.

  The time information request unit 3021 requests the time information of the foreground image and the background image acquired by the back-end server 270 and the time information of the background image associated with the same value as the camera exposure information of the foreground image to the database 250.

  The background image quality correction unit 3020 uses the camera exposure information of the foreground image and the background image and the time information acquired from the database 250 to obtain the camera exposure information of the background image suitable for the time of the frame for which the virtual viewpoint image is to be generated. Calculate and perform image quality correction of the background image.

Hereinafter, the image quality correction processing of the background image in the third embodiment will be described in detail with reference to FIG.
In FIG. 8, the time of the frame for which the virtual viewpoint image is to be generated is tx, and the time when the background image exists in the database 250 is t1, t2 (t1 <t2). The exposure change is performed immediately before the time tx. The camera exposure information before the exposure change is EVα, and the camera exposure information after the exposure change is EVβ.

  First, the foreground image camera exposure information acquisition unit 3015 acquires camera exposure information of the foreground image, and the background image camera exposure information acquisition unit 3016 acquires camera exposure information of the background image. Here, since the exposure change is performed immediately before time tx, the camera exposure information of the foreground image acquired by the foreground image camera exposure information acquisition unit 3015 is EVβ. On the other hand, since there is no corresponding data in the background image, the camera exposure information of the background image acquired by the background image camera exposure information acquisition unit 3016 is the camera exposure information of the background image acquired immediately before by the back-end server 270. EVα.

  Next, the time information request unit 3021 stores the time information of the background image acquired immediately before by the back-end server 270 and the time information of the background image having the same value as the camera exposure information EVβ of the foreground image. To request.

  The database 250 outputs predetermined time information to the back-end server 270 in response to a request from the time information request unit 3021. Note that the time information of the background image acquired immediately before by the back-end server 270 is t1, and the time information where the background image having the same value as the camera exposure information EVβ of the foreground image exists is t2.

  When the back-end server 270 acquires time information from the database 250, the background image image quality correction unit 3020 acquires camera exposure information from the foreground image camera exposure information acquisition unit 3015 and the background image camera exposure information acquisition unit 3016, respectively. Also in the third embodiment, as in the first and second embodiments, when there are different numerical values among the camera exposure information of the foreground image group, the EV values of the most camera exposure information are the values of the foreground image. Camera exposure information is used.

In the case of the third embodiment, the background image quality correction unit 3020 calculates the camera exposure information EVx at the frame time tx at which the virtual viewpoint image is generated by the calculation of the following formula (3).
EVx = EVβ * (tx−t1) / (t2−t1) + EVα * (t2−tx) / (t2−t1) Equation (3)

Further, the background image image quality correction unit 3020 performs the calculation of the following equation (4) to correct the background image data. In Equation (4), BP represents background image data before correction, and BPc represents background image data after correction.
BPc = BP * 2 ^{(EVx-EVα )} Formula (4)

  Then, the background image quality correction unit 3020 outputs the background image whose image quality has been corrected to the background texture pasting unit 3002. The background texture pasting unit 3002 pastes the background image after the image quality correction into a three-dimensional space shape, and generates a textured background mesh model.

  As described above, the back-end server 270 of the third embodiment is based on the camera exposure information and time information of the foreground image and the background image when the camera exposure information of the foreground image and the camera exposure information of the background image are different. The image quality of the background image is corrected. Thus, according to the third embodiment, a natural virtual viewpoint image can be generated in which the luminance does not become discontinuous in the time direction.

<Fourth Embodiment>
In the above-described embodiment, if the camera exposure information of the foreground image and the background image does not match when the virtual viewpoint image is generated, an appropriate background image is reacquired (first embodiment), or the image quality of the background image is corrected (the second image). In the third embodiment, the brightness mismatch between the foreground image and the background image is eliminated.
In the fourth embodiment, a process of detecting a mismatch between camera exposure information of a foreground image and a background image before storing shooting data in the database 250 and generating a background image in which the mismatch of luminance is eliminated will be described.
The schematic configuration of the image processing system 100 in the fourth embodiment is the same as that of the first embodiment described above. Since the sensor system 110, the switching hub 180, the end user terminal 190, the database 250, the time server 290, the control station 310, and the virtual camera operation UI 330 are the same as described above, the description thereof is omitted.

  FIG. 9 is a functional block diagram of the front-end server 230 in the fourth embodiment. The front-end server 230 according to the fourth embodiment has a configuration in which a camera exposure information comparison unit 2200 is added to the front-end server 230 illustrated in FIG. The data input control unit 2120 to DB access control unit 2190 in FIG. 9 has substantially the same configuration as the data input control unit 2120 to DB access control unit 2190 shown in FIG.

  Similar to the first embodiment, the data synchronization unit 2130 of the front-end server 230 buffers data until image data at the same time is collected. When the photographing data is prepared, the data synchronization unit 2130 transmits the foreground image and the background image to the image processing unit 2150, the three-dimensional model data to the three-dimensional model combination unit 2160, and the audio data to the photographing data file generation unit 2180. . Further, the data synchronization unit 2130 outputs the camera exposure information to the camera exposure information comparison unit 2200.

  The camera exposure information comparison unit 2200 acquires camera exposure information of foreground images and background images of all sensor systems 110 from the data synchronization unit 2130. First, the camera exposure information comparison unit 2200 sets the EV value of the largest number of camera exposure information as the camera exposure information of the foreground image when there are different numerical values among the plurality of camera exposure information of the foreground image. Next, the camera exposure information comparison unit 2200 compares the camera exposure information of all the background images with the camera exposure information of the foreground image. In the fourth embodiment, the camera exposure information comparison unit 2200 calculates a difference between the EV value of the camera exposure information of the foreground image and the EV value of the background image of each sensor system 110, and uses the difference value as a camera identifier. Together, it is output to the image processing unit 2150.

  As in the first embodiment, the image processing unit 2150 according to the fourth embodiment develops the raw image data of the foreground image and the background image, corrects camera lens distortion, and the like. Perform the process. The image processing unit 2150 corrects the image quality of the background image having a value different from the camera exposure information value of the foreground image based on the EV value difference value and the camera identifier input from the camera exposure information comparison unit 2200. To do. When the image quality of the background image is corrected, correction processing is performed using Expression (2) of the second embodiment, Expression (3), Expression (4), or the like of the third embodiment. Then, the image processing unit 2150 outputs the foreground image after the image processing to the shooting data file generation unit 2180, and outputs the background image after the image processing to the image combining unit 2170.

  The image combining unit 2170 acquires the background image from the image processing unit 2150, acquires the stadium three-dimensional shape data (stadium shape data) from the CAD data storage unit 2135, and obtains the background image with respect to the coordinates of the acquired stadium shape data. Identify the location. Then, when the position relative to the coordinates of the stadium shape data can be specified for each of the background images, the image combining unit 2170 combines the background images into one background image.

  FIG. 10 is a diagram illustrating functional blocks of the back-end server 270 according to the fourth embodiment. The back-end server 270 of the fourth embodiment includes a foreground image camera exposure information acquisition unit 3015, a background image camera exposure information acquisition unit 3016, a camera exposure information comparison unit 3017, a background image acquisition determination unit 3018, and a background image request in FIG. The portion 3019 is excluded. Note that the data reception unit 3001 to the rendering mode management unit 3014 in FIG. 10 have substantially the same configuration as the data reception unit 3001 to the rendering mode management unit 3014 illustrated in FIG.

  The back-end server 270 in the fourth embodiment does not need to acquire the camera exposure information of the foreground image and the background image and perform the camera exposure information comparison process. Accordingly, the back-end server 270 reads the corresponding foreground image, background image, and audio data from the database 250 based on the viewpoint designated via the virtual camera operation UI 330, and performs rendering processing to generate a virtual viewpoint image. .

  That is, in the image processing system 100 according to the fourth embodiment, when the camera exposure information of the foreground image is different from the camera exposure information of the background image, the image quality of the background image is corrected by the front-end server 230, so that the luminance is low. Natural virtual viewpoint images that do not become continuous can be generated.

<Fifth Embodiment>
In the fifth embodiment, description will be given of processing for eliminating the mismatch between the camera exposure information of the foreground image and the background image by changing the transmission timing of the background image in the sensor system 110.
The schematic configuration of the image processing system 100 in the fifth embodiment is the same as that of the first embodiment described above. Since the sensor system 110, the switching hub 180, the end user terminal 190, the front end server 230, the database 250, the time server 290, the control station 310, and the virtual camera operation UI 330 are the same as described above, the description thereof is omitted. Further, the back-end server 270 of the fifth embodiment has the same function as that of the above-described fourth embodiment, and thus the description thereof is also omitted.

Hereinafter, with reference to FIG. 2, background image transmission timing change processing performed by the camera adapter 120 in the sensor system 110 in the fifth embodiment will be described.
A camera control unit 6141 of the camera adapter 120 of the fifth embodiment is connected to the camera 112 and performs camera control in response to an instruction to change shooting parameters (such as the number of pixels, color depth, frame rate, and white balance). . At the same time, the camera control unit 6141 notifies the data transmission / reception unit 6111 that the shooting parameters have been changed.

  Similar to the first embodiment, the data transmission / reception unit 6111 outputs the foreground image and the background image separated from the captured image of the camera 112 by the foreground / background separation unit 6131 to another camera adapter 120. At this time, the data transmission / reception unit 6111 thins out some frames of the background image separated from the captured image of the camera 112 and outputs the background image at a lower frame rate than the foreground image. However, when the data transmission / reception unit 6111 receives the notification of the imaging parameter change from the camera control unit 6141, the data transmission / reception unit 6111 cancels the background image thinning process immediately after the imaging parameter change and outputs the background image to another camera adapter 120. .

  As described above, in the image processing system 100 of the fifth embodiment, both the foreground image and the background image immediately after the shooting parameter change are stored in the database 250, and the back-end server 270 has discontinuous luminance. It is possible to generate a virtual viewpoint image that never becomes.

<Sixth Embodiment>
In the fifth embodiment, an example is given of processing for eliminating the mismatch between the camera exposure information of the foreground image and the background image by changing the transmission timing of the background image in the sensor system 110. In the sixth embodiment, when a shooting parameter change instruction is input from the outside of the camera adapter 120, the camera exposure information mismatch between the foreground image and the background image is resolved by controlling the timing of setting the shooting parameter in the camera 112. Processing to be performed will be described. Note that the configuration of the image processing system 100 of the sixth embodiment is the same as that of the fifth embodiment.

  In the case of the sixth embodiment, first, the camera control unit 6141 acquires the background image thinning timing performed by the data transmission / reception unit 6111. The camera control unit 6141 then changes the shooting parameter after the change from the shot image in which both the foreground image and the background image are output to the other camera adapter 120 even if the shooting parameter change instruction is input. The camera 112 is controlled so that is applied.

  According to the sixth embodiment, both the foreground image and the background image immediately after the shooting parameter change are stored in the database 250, and the back-end server 270 generates a virtual viewpoint image that does not have a discontinuous luminance. it can.

  The present invention supplies a program that realizes one or more functions of the above-described embodiments to a system or apparatus via a network or a storage medium, and one or more processors in a computer of the system or apparatus read the program. It can also be realized by processing to be executed. It can also be realized by a circuit (for example, ASIC) that realizes one or more functions.

  The above-described embodiments are merely examples of implementation in carrying out the present invention, and the technical scope of the present invention should not be construed as being limited thereto. That is, the present invention can be implemented in various forms without departing from the technical idea or the main features thereof.

  100: Image processing system, 110: Sensor system, 112: Camera, 120: Camera adapter, 180: Switching hub, 200: Image computing server, 230: Front end server, 250: Database, 270: Back end server, 290: Time server, 300: Controller, 310: Control station, 330: Virtual camera operation UI, 190: End user terminal

Claims (22)

An image processing device that generates a virtual viewpoint image using a plurality of images photographed by a plurality of imaging devices,
First acquisition means for acquiring a foreground image corresponding to a predetermined object, the image being based on any of the plurality of images;
Second acquisition means for acquiring exposure information indicating exposure when the foreground image acquired by the first acquisition means is captured;
A background image that is based on any one of the plurality of images and is photographed with an exposure corresponding to the exposure information acquired by the second acquisition means and does not include the predetermined object, Third acquisition means for acquiring based on the exposure information acquired by the acquisition means;
Generating means for generating a virtual viewpoint image using the foreground image acquired by the first acquisition means and the background image acquired by the third acquisition means;
An image processing apparatus comprising:   The image processing apparatus according to claim 1, wherein the third acquisition unit acquires the background image captured with the same exposure as the exposure indicated by the exposure information acquired by the second acquisition unit. The first acquisition means acquires a plurality of foreground images corresponding to the predetermined object,
The said 3rd acquisition means acquires the said background image image | photographed with the exposure defined based on the exposure information of these foreground images acquired by the said 2nd acquisition means. An image processing apparatus according to 1.   The third acquisition unit requests the background image captured at an exposure determined based on the exposure information acquired by the second acquisition unit from a database, and acquires the background image from the database. The image processing apparatus according to any one of claims 1 to 3. Determining means for determining a foreground image and a background image to be used for generating a virtual viewpoint image from a plurality of foreground images and a plurality of background images obtained by separating a plurality of images captured by a plurality of imaging devices, respectively;
Acquisition means for acquiring exposure information at the time of shooting of the determined foreground image and exposure information at the time of shooting of the determined background image;
Processing means for generating the virtual viewpoint image based on the foreground image and the background image,
When the exposure information of the determined foreground image and the background image is different, the processing means performs a process of adjusting the exposure of the background image to the exposure of the determined foreground image, and a process of adjusting the exposure An image processing apparatus that generates the virtual viewpoint image using the subsequent background image and the determined foreground image.   The processing means, as the processing for adjusting the exposure, is a predetermined adjustment for adjusting the image quality of the foreground image with respect to the determined background image based on the determined exposure information of the foreground image and the background image. The image processing apparatus according to claim 5, wherein image quality correction processing is performed. The processing means includes
When there are a plurality of the determined foreground images and the exposure information of the plurality of foreground images is different, the exposure information having the largest number for each exposure information among the plurality of exposure information of the plurality of foreground images. Set as exposure information of the determined foreground image,
The image processing according to claim 6, wherein the image quality correction process is performed on a foreground image having exposure information different from the set exposure information so as to match an image quality with the foreground image having the set exposure information. apparatus.   The processing means obtains a background image of exposure information that matches the exposure information of the foreground image from a plurality of background images that can be used to generate the virtual viewpoint image as the processing for adjusting the exposure. The image processing apparatus according to claim 5, wherein the image processing apparatus is performed.   When there are a plurality of background images of exposure information that match the exposure information of the foreground image, the processing means, based on the determined shooting time of the background image, among the plurality of background images that match the exposure information. The image processing apparatus according to claim 8, wherein a background image at a shooting time closest to the shooting time is acquired later. Holding means for holding a plurality of foreground images and a plurality of background images obtained by respectively separating a plurality of images taken by a plurality of imaging devices;
When the foreground image and the background image from which the captured image is separated have different exposure information at the time of shooting, the exposure information of the foreground image and the background image is used for the background image. Processing means for performing a predetermined image quality correction process for matching the image quality to the foreground image, and holding the background image after the image quality correction process is performed in the holding means;
Determining means for determining a foreground image and a background image used for generating a virtual viewpoint image from among a plurality of foreground images and a plurality of background images held in the holding means;
Generating means for generating a virtual viewpoint image using the determined foreground image and the background image;
An image processing apparatus comprising:   When the exposure at the time of shooting is changed by the imaging device, the processing means applies the exposure to the background image based on exposure information at shooting times before and after the exposure is changed as the process of adjusting the exposure. The image processing apparatus according to claim 5, wherein the image quality correction process for adjusting an image quality to the foreground image is performed.   The processing means performs the image quality correction processing on the background image based on exposure information calculated by a predetermined calculation using the exposure information at shooting times before and after the exposure at the time of shooting is changed. The image processing apparatus according to claim 11.   When the processing means includes a plurality of foreground images used for generating the virtual viewpoint image and the exposure information of the plurality of foreground images is different, exposure is performed among the plurality of exposure information of the plurality of foreground images. The image processing apparatus according to claim 8, wherein exposure information having the largest number of pieces of information is set as exposure information of the determined foreground image.   The image processing apparatus according to claim 5, wherein a frame rate of the foreground image is higher than a frame rate of the background image. Control means for separating a captured image into a foreground image and a background image, performing a thinning process on the separated background image, and outputting a frame rate of the background image as a frame rate lower than the frame rate of the foreground image; ,
Processing means for generating a virtual viewpoint image based on the foreground image and the background image output from the control means,
The image processing apparatus according to claim 1, wherein the control unit cancels the thinning-out process for the background image when an exposure is changed at the time of photographing the image. Control means for separating a captured image into a foreground image and a background image, and outputting the foreground image and the background image at different frame rates;
Processing means for generating a virtual viewpoint image based on the foreground image and the background image output from the control means,
When an instruction to change the exposure at the time of capturing the image is input, the control unit adjusts the exposure in accordance with a timing to output both the foreground image and the background image separated from the captured image. An image processing apparatus characterized by applying the change. A determination step of determining a foreground image and a background image used for generating a virtual viewpoint image from a plurality of foreground images and a plurality of background images obtained by separating a plurality of images captured by a plurality of imaging devices, respectively;
An acquisition step of acquiring exposure information at the time of shooting of the determined foreground image and exposure information at the time of shooting of the determined background image;
A process for generating the virtual viewpoint image based on the foreground image and the background image, and
In the processing step, if the exposure information of the determined foreground image and the background image is different, a process of adjusting the exposure of the background image to the exposure of the determined foreground image, and the process of adjusting the exposure An image processing method for an image processing apparatus, wherein the virtual viewpoint image is generated by using the background image after and the determined foreground image. A holding step of holding a plurality of foreground images and a plurality of background images obtained by separating a plurality of images taken by a plurality of imaging devices, respectively;
When the foreground image and the background image from which the captured image is separated have different exposure information at the time of shooting, the exposure information of the foreground image and the background image is used for the background image. Performing a predetermined image quality correction process for matching the image quality to the foreground image, and holding the background image after the image quality correction process in the holding process;
A determining step for determining a foreground image and a background image used for generating a virtual viewpoint image from a plurality of foreground images and a plurality of background images held in the holding step;
Generating a virtual viewpoint image using the determined foreground image and the background image;
An image processing method for an image processing apparatus, comprising: A control step of separating the captured image into a foreground image and a background image, performing a thinning process on the separated background image, and outputting the frame rate of the background image as a frame rate lower than the frame rate of the foreground image; ,
A processing step of generating a virtual viewpoint image based on the foreground image and the background image output from the control step,
An image processing method of an image processing apparatus, wherein, in the control step, the thinning process for the background image is canceled when an exposure is changed when the image is captured. A control step of separating the captured image into a foreground image and a background image and outputting the foreground image and the background image at different frame rates;
A processing step of generating a virtual viewpoint image based on the foreground image and the background image output from the control step,
In the control step, when an instruction to change the exposure at the time of capturing the image is input, the exposure is performed in accordance with a timing for outputting both the foreground image and the background image separated from the captured image. An image processing method for an image processing apparatus, characterized by applying a change of   The program for functioning a computer as each means of the image processing apparatus of any one of Claim 1 to 16. An imaging device that separates a captured image into a foreground image and a background image and outputs them at different frame rates, and outputs exposure information at the time of capturing the image together with the foreground image and the background image;
The image processing device according to any one of claims 5 to 14, wherein the foreground image and the background image output from the imaging device and the exposure information are acquired.
An image processing system.

Images