JP2021056900A - Image processor, image processing method, and program - Google Patents

Abstract

To make it easy to view a region where a specific event within a photographing region is performed in a virtual viewpoint image generated based on a photographed image.SOLUTION: An image generator 120 acquires a plurality of viewpoint images based on photographing performed by a plurality of photographing devices 110 which photograph a photographed region 200 from mutually different directions. The image generator 120 acquires viewpoint information indicating the position and the direction of a virtual viewpoint and specifies a partial region in the photographing region 200 where a specific event included in those performed in the photographed region 200 is performed. Then the image generator 120 generates, based on the plurality of viewpoint images and viewpoint information, a virtual viewpoint image which has limitation on display of an object not located within the specified partial region included in the object in the photographed image 200.SELECTED DRAWING: Figure 8

Description

The present invention relates to a technique for generating a virtual viewpoint image.

There is a technique in which a plurality of photographing devices are installed at different positions to perform synchronous photography, and a virtual viewpoint image in which the viewpoint can be arbitrarily changed is generated by using a plurality of captured images obtained by the photographing. Specifically, a virtual viewpoint image is generated by generating three-dimensional shape data of an object included in the captured image based on a plurality of captured images and performing rendering processing according to the position and orientation of the virtual viewpoint. To.

Patent Document 1 discloses that a camera installed around a stadium captures a soccer competition or an athletics competition to generate a virtual viewpoint image. By viewing the virtual viewpoint image generated in this way, the user can watch the competition from various directions.

Japanese Unexamined Patent Publication No. 2017-21128

When athletics are held in the stadium, multiple types of competition such as 100m dash, long jump, javelin throw, etc. may proceed in parallel in the same stadium. When the virtual viewpoint image is generated based on the image of the stadium in such a situation, the visibility of the virtual viewpoint image for the user may be deteriorated. For example, when a user tries to watch the 100m dash, the long jump player hides the 100m dash player, or the long jump that is reflected in the image makes the user unable to concentrate on the 100m dash. Can be considered.

Such problems are not limited to athletics, but can occur similarly when multiple sports such as judo and gymnastics are held at the same venue. Further, even when a circus or the like in which a plurality of performances are performed is targeted for shooting, the same problem may occur, such as difficulty in seeing a person or the like within a specific range that the user wants to see.

The present invention has been made in view of the above problems, and an object of the present invention is to make it easier to see an area in a photographed area where a specific event is performed in a virtual viewpoint image generated based on a photographed image.

In order to solve the above-mentioned problems, the image processing apparatus according to the present invention has, for example, the following configuration. That is, an image acquisition means for acquiring a plurality of images based on shooting by a plurality of shooting devices that shoot a shooting area from different directions, an information acquisition means for acquiring viewpoint information indicating the position and direction of a virtual viewpoint, and the shooting. A specific means for specifying a partial area in the shooting area where a specific event is performed among the events performed in the area, and a virtual viewpoint image according to the position and orientation of the virtual viewpoint, which is an object in the shooting area. Among the virtual viewpoint images in which the display of objects not located in the partial area specified by the specific means is restricted, the plurality of images acquired by the image acquisition means and the viewpoint acquired by the information acquisition means. It has an image generation means for generating based on information.

According to the present invention, in a virtual viewpoint image generated based on a captured image, it is possible to make it easier to see an area in the captured area where a specific event is performed.

It is a block diagram which shows the structure of an image generation system. It is a figure which shows the installation example of the photographing apparatus. It is a figure which shows the hardware configuration example of an image generator. It is a block diagram which shows the functional structure example of an image generation apparatus. It is a figure for demonstrating the example of the competition area information. It is a flowchart for demonstrating an example of processing of an image generation apparatus. It is a figure which shows the example of the display screen of a user terminal. It is a figure for demonstrating the example of the generated virtual viewpoint image. It is a figure for demonstrating the example of the position history of an object. It is a flowchart for demonstrating an example of processing of an image generation apparatus. It is a figure which shows the example of the information which associates a virtual viewpoint with a competition.

[System configuration]
FIG. 1 is a block diagram showing a configuration example of the image generation system 100 according to the present embodiment. As shown in FIG. 1, the image generation system 100 includes a plurality of photographing devices 110, an image generation device 120, and a user terminal 130.

As shown in FIG. 2, the plurality of photographing devices 110 photograph a part or the whole of the photographing area 200 from different directions. Each photographing device 110 is, for example, a digital camera capable of photographing a still image and a moving image. The plurality of photographing devices 110 are installed at different positions so as to surround the photographing area 200, and photograph in synchronization with each other. The plurality of photographing devices 110 may not be installed over the entire circumference of the photographing area 200, and may be installed only in a part around the photographing area 200 depending on the limitation of the installation place or the like. Further, the number of photographing devices 110 is not limited to the example shown in the figure. For example, when the photographing area 200 is used as an athletics stadium, even if about 100 photographing devices 110 are installed around the stadium. Good. Further, a photographing device 110 having different functions such as a telephoto camera and a wide-angle camera may be installed.

The photographing area 200 in the present embodiment is a stadium where a plurality of types of athletics are simultaneously carried out. However, the photographing area 200 is not limited to this, and may be a stadium where a competition other than athletics is performed, or a stage where a circus performance or the like is performed. In the following, competitions held at the stadium and performances performed on the stage will be referred to as events. For example, in an athletics stadium, a plurality of different events such as 100m dash, long jump, and javelin throw are performed in different areas.

Images taken by the plurality of photographing devices 110 are transmitted to the image generating device 120 via a communication medium such as a LAN cable. The image may be transmitted wirelessly to the image generator 120.

The image generation device 120 is, for example, a server device or the like, and has a database function and an image processing function. The image generation device 120 acquires and stores a plurality of images (multi-viewpoint images) based on the images taken by the plurality of photographing devices 110. The image included in the multi-viewpoint image may be a captured image, or may be an image obtained by performing image processing such as extraction of a predetermined region on the captured image. Then, the image generation device 120 generates a virtual viewpoint image representing the appearance from the designated virtual viewpoint based on the plurality of viewpoint images and the virtual viewpoint designated via the user terminal 130.

The virtual viewpoint image in the present embodiment is also called a free viewpoint image, but is not limited to an image corresponding to a viewpoint freely (arbitrarily) specified by the user, for example, a viewpoint selected by the user from a plurality of candidates. Corresponding images and the like are also included in the virtual viewpoint image. Further, in the present embodiment, the case where the virtual viewpoint is specified based on the user operation will be mainly described, but the virtual viewpoint may be automatically specified based on the result of image analysis or the like. Further, in the present embodiment, the case where the virtual viewpoint image is a moving image will be mainly described, but the virtual viewpoint image may be a still image.

The image generation device 120 has a database function of storing an image taken in advance of a stadium in a state where a predetermined object such as a player does not exist, such as before the start of a competition, as background image data. Further, the image generation device 120 stores the image obtained by separating the foreground of the athlete or the like during the competition from the captured image by image processing as the foreground image data. As a method for separating the foreground, for example, an image processing of object extraction that extracts the foreground based on the difference between the captured image and the background image is used. In addition, as another method, a moving object detected from the captured image may be separated as the foreground. The object to be the foreground may be not only the athlete performing the competition but also another person (for example, a referee), or an instrument such as a javelin throw or high jump mat or pole.

The image generation device 120 generates a virtual viewpoint image corresponding to the designated virtual viewpoint from the held background image data and foreground image data. Then, the image generation device 120 outputs the generated virtual viewpoint image to the user terminal 130 via a LAN cable or the like.

As a virtual viewpoint image generation method, for example, model-based rendering (MBR) can be used. The MBR is a method of generating a virtual viewpoint image by using a three-dimensional model generated based on a plurality of captured images of an object captured from a plurality of directions. Specifically, by projecting a plurality of captured images corresponding to the appearance of the three-dimensional shape from the virtual viewpoint as a rendered image on the three-dimensional shape of the target object obtained by the three-dimensional shape restoration method, it is virtual. This is a method for generating a viewpoint image. As the three-dimensional shape restoration method, a method such as a visual volume crossing method or an MVS (Multi-View-Stereo) can be used. However, the method of generating the virtual viewpoint image is not limited to this, and various methods such as a method of generating a virtual viewpoint image by projective transformation of the captured image without using the three-dimensional model can be used.

The user terminal 130 is, for example, a PC or a tablet terminal, and has an operation unit that accepts user operations such as a mouse, keyboard, 6-axis controller, or touch panel, and a display unit that displays an image. The user terminal 130 receives an operation for selecting a competition of interest and an operation for designating a virtual viewpoint from the user, and outputs information according to the user operation to the image generator 120 via a LAN cable or the like. .. Further, the user terminal 130 causes the user to view the virtual viewpoint image according to the user operation by displaying the virtual viewpoint image received from the image generation device 120 on the display unit.

In the present embodiment, the information input from the user terminal 130 to the image generation device 120 in response to the operation for designating the virtual viewpoint is the viewpoint information used for generating the virtual viewpoint image, and the position and orientation of the virtual viewpoint. (Gaze direction) is shown. Specifically, the viewpoint information is a parameter set including a parameter representing a three-dimensional position of the virtual viewpoint and a parameter representing the orientation of the virtual viewpoint in the pan, tilt, and roll directions. The parameter representing the position of the virtual viewpoint represents a three-dimensional position on the virtual space having an origin associated with the position in the shooting area 200, and is a relative position of the virtual viewpoint with respect to a predetermined position such as the center of the shooting area 200. Is shown. The parameter representing the direction of the virtual viewpoint indicates in which direction the virtual viewpoint image is generated from the position of the virtual viewpoint.

The content of the viewpoint information is not limited to the above. For example, the parameter set as the viewpoint information may include a parameter representing the size (angle of view) of the field of view of the virtual viewpoint. Further, the viewpoint information may have a plurality of parameter sets. For example, the viewpoint information may have a plurality of parameter sets corresponding to a plurality of frames constituting a moving image of the virtual viewpoint image, and may be information indicating the position and orientation of the virtual viewpoint at each of a plurality of consecutive time points. .. The user terminal 130 may input operation information indicating the content of the user operation to the image generation device 120, and the image generation device 120 may generate viewpoint information based on the operation information.

The configuration of the image generation system 100 is not limited to that shown in FIG. For example, a plurality of user terminals 130 may be connected to the image generation device 120, or the plurality of image generation devices 120 may acquire a plurality of viewpoint images from the photographing device 110 and generate a virtual viewpoint image.

[Hardware configuration]
FIG. 3 is a diagram showing a hardware configuration example of the image generation device 120. The configuration of the user terminal 130 is also the same as that of the image generation device 120 described below. The image generation device 120 includes a CPU 301, a ROM 302, a RAM 303, an HDD 304, an input unit 305, a display unit 306, and a communication unit 307.

The CPU 301 realizes each function of the image generation device 120 by executing arithmetic processing and various programs using the control programs stored in the ROM 302 and the HDD 304. The image generator 120 may have one or more dedicated hardware different from the CPU 301, and the dedicated hardware may execute at least a part of the processing by the CPU 301. Examples of dedicated hardware include ASICs (application specific integrated circuits), FPGAs (field programmable gate arrays), and DSPs (digital signal processors). The ROM 302 has a storage area for storing a boot program, a control program, parameters set in each part of the image generation device 120, and the like. The RAM 303 is used as a matching storage area for the main memory, work area, etc. of the CPU 301. The HDD 304 stores various data and programs. In addition, other auxiliary storage devices such as SSD may be used instead of HDD 304.

The input unit 305 is composed of, for example, a keyboard, a mouse, a joystick, a touch panel, or the like, and inputs various instructions to the CPU 301 in response to an operation by the user. The display unit 306 is composed of, for example, a liquid crystal display, an LED, or the like, and displays a GUI (Graphical User Interface) or the like for the user to operate the image generation device 120. The CPU 301 operates as a display control unit that controls the display unit 306. The communication unit 307 executes communication processing with external devices such as the photographing device 110 and the user terminal 130. The communication by the communication unit 307 may be performed via a wired network such as a LAN, or may be performed via a wireless network.

In the present embodiment, it is assumed that the display unit 306 and the input unit 305 exist inside the image generation device 120, but at least one of the display unit 306 and the input unit 305 exists as another device outside the image generation device 120. You may be doing it.

[Functional configuration of image generator]
FIG. 4 is a block diagram showing a functional configuration of the image generation device 120 according to the present embodiment. The image acquisition unit 401 receives the captured image from the photographing device 110 and transmits it to the foreground background separating unit 402. The foreground background separation unit 402 extracts a foreground area corresponding to an object such as a player from the captured image received from the image acquisition unit 401, and stores it in the data holding unit 403 as foreground image data. In addition to the foreground image data received from the foreground / background separation unit 402, the data holding unit 403 holds as background image data a photographed image of the stadium in a state where no subject exists, such as before the start of the competition.

The model generation unit 404 generates model information of the object based on the foreground image data acquired from the data holding unit 403 and the camera parameters indicating the position and orientation of the photographing device 110. The model information includes point cloud data representing the shape of the model, position information representing the position of the model, and the like. That is, the model information is data representing the three-dimensional shape of the object and the position of the object.

The information acquisition unit 405 receives the input corresponding to the user operation from the user terminal 130 as user command data. When the user command data is selection information corresponding to the operation of selecting the competition of interest, the information acquisition unit 405 transmits the selection information to the area management unit 407. The selection information indicates, for example, a plurality of competition events and whether or not the competition event has been selected. Further, when the user command data is the viewpoint information corresponding to the operation of designating the virtual viewpoint, the information acquisition unit 405 transmits the viewpoint information to the viewpoint setting unit 406.

The viewpoint setting unit 406 sets the position and orientation of the virtual viewpoint in the virtual space based on the information received from the information acquisition unit 405, and transmits the viewpoint information indicating the set virtual viewpoint to the image generation unit 409. The area management unit 407 manages competition area information indicating the competition area in which each competition to be photographed is performed and the competition designated as the target of attention. Then, the competition area information is updated based on the selection information received from the information acquisition unit 405, and the model display determination unit 408 is notified of the update.

An example of competition area information managed by the area management unit 407 will be described with reference to FIG. As shown in FIG. 5A, the photographing area 200 includes an area 501 which is a partial area where the long jump is performed, an area 502 which is a partial area where the pole vault is performed, and an area which is a partial area where the 100m run is performed. 503 is included. In this example, the areas are set so as not to overlap, but a part of a plurality of areas may overlap.

FIG. 5B shows an example of the competition area information 510. The competition area information 510 includes a plurality of competition events, coordinate information indicating a competition area in which each competition is held, and competition designation information indicating whether each competition is designated as a target of attention. In the example of FIG. 5B, pole vaulting is designated as the sport of interest. The coordinate information of the competition area shall be preset according to the stadium to be photographed. However, the coordinate information of the competition area may be modified according to the actual progress of the competition, or may be modified according to the input from the user terminal 130 via the information acquisition unit 405. ..

Further, in FIG. 5B, an example in which the competition area is represented by the two-dimensional coordinates of the X-axis and the Y-axis is shown, but the competition area is represented by the three-dimensional coordinates including the coordinates in the Z-axis direction (height direction). May be done. Further, in the example of FIG. 5, the competition area is assumed to be a rectangle having the coordinates of two points as vertices, but the shape of the competition area is not limited to this, and the shape of the competition area differs depending on the competition. You may. Further, when the content of the competition to be performed in the photographing area 200 and the competition area thereof change depending on the time, the competition area information 510 may have time information of each competition in addition to the above information. In this case, the time information of the competition may be input in advance based on the competition progress schedule, or the time information may be updated according to the actual progress status of the competition and the shooting status.

Based on the competition area information, the model display determination unit 408 is one or more of the plurality of regions in the shooting region 200 where different events are performed, and one or more corresponding to the competition designated as the target of attention. Identify the area. When the competition area information has the time information of each competition, the model display determination unit 408 may specify the area where the event to be noted is performed based on the time information. As a result, it is possible to reduce the time and effort of the user operation related to the selection of the competition.

Then, the model display determination unit 408 acquires model information from the model generation unit 404, determines whether or not to display each object in the shooting area 200 represented by the model information in the virtual viewpoint image, and displays the determination result as an image. It is transmitted to the generation unit 409. As a determination method in this embodiment, it is determined whether or not each object is located in the competition area of the competition designated as the object of interest, and only the objects located in the designated competition area are in the virtual viewpoint image. It shall be the display target.

In the present embodiment, as a criterion for determining whether an object is located in a specific competition area, when the entire object (in the case of a person, the entire body) is included in the area, the object is included in the area. It shall be determined that it is located. However, the present invention is not limited to this, and for example, when a part of half or more of an object is included in the area, it may be determined that the object is located in the area. Alternatively, if a specific part of the object (for example, a part in contact with the center of gravity or the ground) is included in the area, it may be determined that the object is located in the area.

The image generation unit 409 receives the model display determination result from the model display determination unit 408, and acquires the model information of the object determined to be displayed from the model generation unit 404. Then, the image generation unit 409 is based on the model information acquired from the model generation unit 404, the foreground image data and the background image data acquired from the data holding unit 403, and the viewpoint information indicating the virtual viewpoint set by the viewpoint setting unit 406. To generate a virtual viewpoint image. Specifically, the rendering process is performed on the model viewed from the virtual viewpoint using the color information of the image data. The image generation unit 409 transmits the generated virtual viewpoint image to the image output unit 410. The image output unit 410 transmits the virtual viewpoint image received from the image generation unit 409 to the user terminal 130.

The configuration of the image generation device 120 is not limited to the example shown in FIG. 4, and some of the components shown in FIG. 4 may be mounted on a device different from the image generation device 120. For example, the foreground / background separation unit 402 may be mounted on the photographing device 110, and the image generation device 120 may acquire the image data obtained by the foreground / background separation from the photographing device 110.

[Operation flow]
Next, the processing flow of the image generation device 120 according to the present embodiment will be described with reference to FIG. The process shown in FIG. 6 is realized by the CPU 301 of the image generator 120 expanding the program stored in the ROM 302 into the RAM 303 and executing the program. At least a part of the processing shown in FIG. 6 may be realized by one or a plurality of dedicated hardware different from the CPU 301. The process shown in FIG. 6 is started at the timing when the image generation device 120 is connected to the photographing device 110 and the user terminal 130 and an instruction for performing the process related to the generation of the virtual viewpoint image is input to the image generation device 120. .. However, the start timing of the process shown in FIG. 6 is not limited to this. In the following description using FIG. 6, it is assumed that a virtual viewpoint image of a moving image is generated based on a plurality of viewpoint images of a moving image composed of a plurality of frames previously photographed by the photographing device 110. However, a virtual viewpoint image of a moving image may be generated in real time in parallel with the shooting by the shooting device 110. Further, a virtual viewpoint image of a still image may be generated.

In S600, the image acquisition unit 401 acquires a plurality of viewpoint images from the photographing device 110, and the foreground background separation unit 402 extracts the foreground image data from the plurality of viewpoint images. Then, the model generation unit 404 acquires the model information of the object based on the foreground image data. In S601, the information acquisition unit 405 acquires selection information according to the operation of selecting the competition of interest from the user terminal 130, and the area management unit 407 updates the competition area information based on the selection information.

FIG. 7 shows an example of a GUI screen displayed on the user terminal 130 at the time of competition selection based on the display control by the image generation device 120. The screen 700 is displayed on the display unit 710 of the user terminal 130. The screen 700 shows an overall view of the stadium and the competition area of each competition managed by the area management unit 407. The user can select the competition of interest while recognizing the competition area of each competition in the stadium (in the example of FIG. 7, the long jump area 701, the pole vault area 702, and the 100m running area 703). In addition, the competition selection menu 704 on the screen 700 shows the competition currently selected as the target of interest (pole vault in the example of FIG. 7) and other competitions (long jump and 100m run in the example of FIG. 7) among the selectable competitions. ) Is identifiable. As the selectable competition, only the competition in progress may be displayed, or the competition that has already been completed or the competition that will be held in the future may also be displayed. Further, when a new competition is started, a message notifying the user of the new competition may be displayed on the screen 700.

In the example of FIG. 7, one competition is selected, but two or more competitions may be selected. Further, in the present embodiment, the user selects the competition and the image generator 120 specifies the area corresponding to the selected competition, but the present invention is not limited to this, and the user directly selects the area of interest. May be good. Further, the user may select a competition or region of interest, and the image generator 120 may specify the region of interest based on the input according to the selection.

In S602, the model display determination unit 408 acquires model information from the model generation unit 404, and selects an object to be determined from a plurality of objects in the photographing area 200 indicated by the model information. In S604, the model display determination unit 408 acquires the competition area information from the area management unit 407, and determines whether the object selected in S602 is located in the competition area of the competition designated as the attention target. When the selected object is located in the competition area, the model display determination unit 408 sets the object as a display target in the virtual viewpoint image in S604. On the other hand, when the selected object is not located in the competition area, the model display determination unit 408 does not set the object as a display target in the virtual viewpoint image in S605.

In S606, the model display determination unit 408 determines whether the display or non-display is set for all the objects indicated by the model information. If there is an unset object, the process returns to S602 and the object to be determined is newly selected. On the other hand, when the setting is completed for all the objects, the process proceeds to S607.

In S607, the viewpoint setting unit 406 sets a virtual viewpoint based on the information input from the user terminal 130 via the information acquisition unit 405, updates the viewpoint information, and transmits it to the image generation unit 409. In S608, the image generation unit 409 acquires the model information of the object set as the display target in S604 from the model generation unit 404. Then, the image generation unit 409 virtualizes according to the position and orientation of the virtual viewpoint based on the model information of the object to be displayed, the viewpoint information received in S607, and the foreground and background image data acquired from the data holding unit 403. Generate a viewpoint image. The virtual viewpoint image generated here includes objects in the field of view according to the position and orientation of the virtual viewpoint among the objects set to be displayed (objects located in the area corresponding to the competition of interest). Is done. In S609, the image output unit 410 outputs the virtual viewpoint image generated in S608 to the user terminal 130.

[Example of virtual viewpoint image]
The virtual viewpoint image generated in S608 of FIG. 6 will be specifically described with reference to FIG. FIG. 8A shows the positional relationship between the pole vault area 801 designated as the object of interest in the shooting area 200, the virtual viewpoint 802 set in S607, and the objects 803 and 804. The object 803 is located in the area of the pole vault area 801 and the object 804 is located outside the area of the pole vault area 801. Further, it is assumed that the object 803 and the object 804 are included in the field of view according to the position and orientation of the virtual viewpoint 802.

FIG. 8B shows a display example of virtual viewpoint image generation generated in S608 according to the situation shown in FIG. 8A. The virtual viewpoint image 800 is displayed on the display unit 710 of the user terminal 130. In the virtual viewpoint image 800, the object 803 located in the area of the pole vault area 801 is displayed, but the object 804 located outside the area of the pole vault area 801 is not displayed. That is, the image generation device 120 generates a virtual viewpoint image 800 in which the display of the object 804 outside the designated pole vault area 801 is restricted and only the object 803 within the designated pole vault area 801 is displayed. .. Although the object 804 is schematically shown by a dotted line in FIG. 8B, the object 804 is not actually displayed on the virtual viewpoint image 800 and cannot be visually recognized by the user who sees the virtual viewpoint image 800.

By displaying such a virtual viewpoint image 800, the user who sees the virtual viewpoint image 800 using the user terminal 130 can concentrate on the image in the area designated as the attention target. For example, when the object of interest is the pole vault, the long jump player is not displayed, so that the visibility of the virtual viewpoint image 800 of the pole vault competition is improved.

In the above description, it is assumed that the objects outside the competition area (designated area) corresponding to the competition designated as the attention target are not displayed, but the image generator 120 displays the objects not located in the designated area. You can limit it. That is, the image generation device 120 may generate a virtual viewpoint image in which the display of objects not located in one or more designated areas among the objects in the shooting area 200 is restricted, and the content of the restriction is the restriction on the presence or absence of display. Not limited. For example, by performing transparency processing on an object outside the designated area and displaying it semi-transparently, the display may be restricted so as to reduce the visibility of the display of the object. Further, for example, an object outside the designated area may be colored with a specific single color to make it inconspicuous. Further, among the objects outside the designated area, the display mode of the object on the front side of the designated area and the object on the back side of the designated area when viewed from the virtual viewpoint may be different.

Further, the model display determination unit 408 may set whether or not to restrict the display of objects outside the designated area, or set the content of the restriction on the display of objects outside the designated area based on the input from the user terminal 130 or the like. Good. Then, the image generation unit 409 may generate a virtual viewpoint image according to the setting.

Further, for the objects outside the designated area, the display mode may be different for each object type, and the display / non-display may be selected for each object type. For example, among the objects included in the field of view according to the position and orientation of the virtual viewpoint and not located in the specified area, the object of the person is not displayed, and the object of the instrument such as the mat or the pole is displayed. Good. As a result, the visibility of the objects in the designated area can be improved by hiding the person outside the designated area, and the presence of the stadium can be maintained by displaying the equipment outside the designated area. The type of object is not limited to a person and an instrument, and for example, a moving object and a stationary object may be treated as different types of objects.

Further, the initial value of the virtual viewpoint when setting the virtual viewpoint in response to the input from the user terminal 130 may be different depending on the competition designated as the target of attention. For example, the initial position of the virtual viewpoint may be set to the center position of the designated area, or the most notable points according to the competition (100m dash goal, high jump bar, etc.) may be set so that they can be seen. Is also good. Further, the initial state of the virtual viewpoint may be set to the state of the virtual viewpoint when the same competition was designated in the past. In this case, the image generation device 120 acquires viewpoint information indicating the position and orientation of the virtual viewpoint according to the competition selected by the user operation. Then, the information indicating the virtual viewpoint for each competition is managed by the area management unit 407 and transmitted to the user terminal 130. Further, when setting the virtual viewpoint in response to the input from the user terminal 130, the position of the virtual viewpoint may be restricted so that the position of the virtual viewpoint can be specified only within the designated area. Further, it may be restricted so that the direction of the virtual viewpoint can be specified only in a certain direction in the designated area. By such processing, it is possible to provide the user with a virtual viewpoint image according to an easy-to-see viewpoint for each competition.

Further, the area management unit 407 may hold correspondence information for associating a plurality of virtual viewpoints (virtual viewpoints A to F) with the competition as shown in FIG. In this case, the user selects a specific virtual viewpoint from the virtual viewpoints A to F via the user terminal 130. The information acquisition unit 405 receives the input corresponding to this selection operation, and the viewpoint setting unit 406 sets a virtual viewpoint according to the selection. Then, the area management unit 407 sets the competition area associated with the selected virtual viewpoint as a designated area. As a result, even if the competition selection of S601 is not performed, a virtual viewpoint image in which only the objects in the competition area associated with the virtual viewpoint are displayed is generated. According to the correspondence information shown in FIG. 11, since a plurality of virtual viewpoints can be associated with one competition, the user can watch the competition while switching between the plurality of virtual viewpoints. The position and orientation of each virtual viewpoint may be predetermined according to the competition, or the position and orientation of each virtual viewpoint can be changed to hold the changed position and orientation of each virtual viewpoint. You may. The correspondence relationship between the virtual viewpoint and the competition in the correspondence information held by the area management unit 407 may be predetermined, or may be set and changed according to the operation of the user terminal 130. In addition, the user may be able to set a name for identifying each virtual viewpoint.

In the above example using FIG. 11, the designated area is set according to the user operation of selecting a specific virtual viewpoint from a plurality of virtual viewpoints, but the present invention is not limited to this, and the virtual viewpoint designated by the user is not limited to this. A designated area may be set based on viewpoint information indicating the position and orientation of the viewpoint. That is, the user can freely specify the virtual viewpoint via the user terminal 130, the area management unit 407 specifies the competition area corresponding to the designated virtual viewpoint from the plurality of competition areas, and the competition area is designated as the designated area. It may be set as. The competition area corresponding to the designated virtual viewpoint may be, for example, a competition area closest to the virtual viewpoint, a competition area most visible in the virtual viewpoint image, or the like.

Further, in the above-described embodiment, when the object outside the designated area is hidden, the image generation unit 409 acquires the model information of the object in the designated area, and obtains the model information and the viewpoint information indicating the virtual viewpoint. Based on this, a virtual viewpoint image is generated. That is, among the model information generated by the model generation unit 404, the virtual viewpoint image is generated without using the model information of the object not located in the designated area. However, the method of generating the virtual viewpoint image is not limited to this. For example, the image generation unit 409 may specify a range included in the field of view of the virtual viewpoint and acquire model information of an object located within the range of the field of view among the objects in the designated area. As a result, the amount of model information data processed by the image generation unit 409 can be reduced.

Further, the image generation device 120 accepts the selection of the designated area before acquiring the model information, generates the model information of the object in the designated area by the model generation unit 404, and generates the model information of the object outside the designated area. It does not have to be. In this case, since only the model of the object in the limited area is generated, there is an effect that the model generation time can be shortened. However, this does not apply when a plurality of user terminals 130 are connected to the image generator 120 and different competitions can be selected by the plurality of user terminals 130. In such a case, the image generation device 120 generates a model of the entire object in the shooting area 200, and based on the information of the competition selection from each user terminal 130, the virtual viewpoint image for each user terminal 130 is displayed in parallel. May be generated.

[Example using object position history]
In the following, a case of determining whether or not to display each object in the virtual viewpoint image will be described using the position history of the objects. In this case, the model display determination unit 408 acquires model information indicating the positions of the objects at a plurality of times and holds the position history for each model. Then, the model display determination unit 408 determines whether or not to display each object based on the competition area information acquired from the area management unit 407 and the position history of each object.

The position change and the position history of the object will be described with reference to FIG. 9 (A) shows the positional relationship between the pole vault area 901, the long jump area 902, the object 903, and the object 904 at a plurality of shooting times (t1, t2, t3). FIG. 9B shows an example of the history information 910 of the object corresponding to the situation shown in FIG. 9A. The history information 910 indicates a competition area in which each object is located at each shooting time.

In the example of FIG. 9, the object 904 (object B) is in the long jump area 902 at time t1, but is located in the pole vault area 901 at time t2, and is again located in the long jump area 902 at time t3. In the example of FIG. 9B, the history information 910 indicates the history of the competition area where the object is located, but the history information 910 may indicate the history of the coordinates of the object, or the competition in which the object is located. The history of area changes may be shown.

Next, the processing flow of the image generator when the position history of the object is used will be described with reference to FIG. In FIG. 10, the same reference numerals are given to the same processes as those described with reference to FIG. In the following, the differences from the processing flow described with reference to FIG. 6 will be mainly described.

In S600, when the model information is generated by the model generation unit 404, the model display determination unit 408 generates and holds the history information indicating the history of the position of the object based on the model information. The processing of S601 to S603 is the same as the processing described with reference to FIG.

When the object to be determined is located in the designated area designated as the object of interest, the model display determination unit 408 in S1001 determines whether the object to be determined is located outside the designated area at another time based on the history information. judge. Then, if the target object is located in the designated area at another time, the object is set as the display target in S604. On the other hand, when the target object is located outside the designated area at another time, the object is set to be hidden in S605. The processing after S606 is the same as the processing described with reference to FIG.

In the example shown in FIG. 9, when pole vault is selected as the competition of interest and a virtual viewpoint image at time t2 is generated, the object 904 is located in the pole vault area 901 at time t2. However, since the object 904 is located outside the pole vault area 901 (long jump area 902) at other times t1 and t3, the object 904 is set to be hidden and is not displayed in the virtual viewpoint image at time t2.

By doing so, when a player in a competition different from the attention competition temporarily invades the area of the attention competition, the player can be prevented from being displayed in the virtual viewpoint image. As a result, it is possible to suppress a decrease in the visibility of the virtual viewpoint image due to the display and disappearance of objects that are not related to the competition of interest.

In the example of FIG. 9, the history information of the objects corresponding to the three times t1 to t3 is shown, but the image generator 120 shows the positions of the objects at each time included in the entire period in which the competition is photographed. History information may be retained. However, when determining S1001, the position history of the object before or after the time of the generated virtual viewpoint image may be the determination target, or the object in a certain period including the time of the virtual viewpoint image may be determined. The position history may be the determination target. For example, in a virtual viewpoint image at a certain time, among the objects located in the specified area at that time, the objects that are not located in the specified area within a period of several minutes before and after that time are not displayed and are specified within the period. Objects located within the area may be displayed.

Further, an object whose time located in the designated area is longer than the time located outside the designated area may be displayed, or a model located in the designated area at least one time may be displayed. Also, even if the object is outside the specified area at another time, the object located near the center of the specified area at the time of the generated virtual viewpoint image will be displayed, and the position of the object within the area will be changed. Whether or not to display may be determined accordingly.

Similarly, the position history of the object may be used to leave the object temporarily out of the designated area as the display target. For example, in a virtual viewpoint image at a certain time, among the objects located outside the specified area at that time, the objects that are not located within the specified area within a period of several minutes before and after that time are not displayed and are specified within the period. Objects located within the area may be displayed. In the example of FIG. 9, the object 904 is located outside the long jump area 902 (pole vault area 901) at time t2, but is in the long jump area 902 at other times t1 and t3. The object 904 may be displayed as a display target. As a result, even if a player of the competition to be focused on temporarily leaves the designated area between competitions, the state of the player can be continuously projected on the virtual viewpoint image.

Further, the competition corresponding to each object (competition in which each player participates) may be determined based on the position history of the object. In that case, the competition in the area where the object to be determined is located for the longest time may be the competition corresponding to the object. Further, when an object is located in two or more areas for the same amount of time, two or more competitions may be associated with the object. When a part of two or more competition areas overlaps, it is possible to determine which competition the object corresponds to by using the position history of the objects located in the overlapping areas.

As described above, the image generation device 120 according to the present embodiment acquires a plurality of viewpoint images based on shooting by a plurality of shooting devices 110 that shoot the shooting area 200 from different directions. Further, the image generation device 120 acquires viewpoint information indicating the position and orientation of the virtual viewpoint, and identifies a partial area in the shooting area 200 in which a specific event is performed among the events performed in the shooting area 200. Then, the image generation device 120 generates a virtual viewpoint image in which the display of the object not located in the specified partial area among the objects in the shooting area 200 is restricted, based on the plurality of viewpoint images and the viewpoint information.

According to such a configuration, in the virtual viewpoint image generated based on the captured image, it is possible to easily see the region in the captured area where a specific event is performed. For example, when the user tries to watch the 100m dash, the long jump player hides the 100m dash player, or the long jump that is reflected in the image makes the user unable to concentrate on the 100m dash. It is possible to provide a virtual viewpoint image that is easy to see.

In this embodiment, a plurality of types of athletics are mainly performed in the athletics stadium, but the present invention is not limited to this, and a plurality of athletics or a plurality of athletics such as gymnastics, judo, and wrestling are described. This embodiment can be applied even when the games are going on at the same time. That is, it is possible to generate a virtual viewpoint image in which the display of an object located outside the area where a specific competition or match is held is restricted among a plurality of competitions or matches played in the same stadium. Alternatively, the present embodiment can be similarly applied to the case where a plurality of events, not limited to competitions, are simultaneously held at the same venue.

The present invention supplies a program that realizes one or more functions of the above-described embodiment to a system or device via a network or storage medium, and one or more processors in the computer of the system or device reads and executes the program. It can also be realized by the processing to be performed. It can also be realized by a circuit (for example, ASIC or the like) that realizes one or more functions. Further, the program may be recorded and provided on a computer-readable recording medium.

100 Image generation system 110 Imaging device 120 Image generator 130 User terminal 200 Imaging area

Claims (24)

An image acquisition means for acquiring a plurality of images based on shooting by a plurality of shooting devices that shoot a shooting area from different directions, and an image acquisition means.
Information acquisition means for acquiring viewpoint information indicating the position and orientation of the virtual viewpoint,
A specific means for specifying a partial area in the shooting area where a specific event is performed among the events performed in the shooting area, and
A virtual viewpoint image according to the position and orientation of the virtual viewpoint, wherein the display of objects not located in the partial area specified by the specific means among the objects in the shooting area is restricted. An image processing apparatus comprising, an image generation means for generating based on the plurality of images acquired by the image acquisition means and viewpoint information acquired by the information acquisition means. In the virtual viewpoint image generated by the image generation means, an object included in the field of view corresponding to the position and orientation of the virtual viewpoint and located in the partial region specified by the specific means is displayed. The image processing apparatus according to claim 1. The image processing apparatus according to claim 1 or 2, wherein in the virtual viewpoint image generated by the image generation means, an object not located in the partial area specified by the specific means is not displayed. The image processing according to claim 1 or 2, wherein in the virtual viewpoint image generated by the image generation means, an object not located in the partial region specified by the specific means is displayed semi-transparently. apparatus. In the virtual viewpoint image generated by the image generation means, among the objects included in the field of view corresponding to the position and orientation of the virtual viewpoint and not located in the partial area specified by the specific means, the first type. The image processing apparatus according to claim 1 or 2, wherein the object is not displayed and the second type object is displayed. The image processing apparatus according to claim 5, wherein the first type object is an instrument, and the second type object is a person. In a virtual viewpoint image generated by the image generation means at a predetermined time, among the objects included in the visual field according to the position and orientation of the virtual viewpoint and not located in the partial region at the predetermined time, a predetermined The image processing apparatus according to claim 1 or 2, wherein the object not located in the partial area is not displayed within the period, and the object located in the partial area is displayed within the predetermined period. .. In a virtual viewpoint image generated by the image generation means at a predetermined time, among the objects included in the visual field according to the position and orientation of the virtual viewpoint and located in the partial region at the predetermined time, a predetermined value is obtained. The image processing apparatus according to claim 1, wherein the object not located in the partial area is not displayed within the period, and the object located in the partial area is displayed within the predetermined period. The image processing apparatus according to any one of claims 1 to 8, wherein the specifying means specifies one or more of the partial areas based on a user operation of selecting one or more areas. The image processing apparatus according to any one of claims 1 to 8, wherein the specifying means specifies one or more of the partial regions based on a user operation of selecting one or more events. The specific means is characterized in that it identifies one or more of the partial areas corresponding to the viewpoint information among a plurality of predetermined areas in the photographing region based on the viewpoint information acquired by the information acquisition means. The image processing apparatus according to any one of claims 1 to 8. The specifying means according to any one of claims 1 to 8, wherein the specifying means specifies the partial area based on information regarding at least one of a plurality of events performed in the photographing area. Image processing equipment. It has a model generating means for generating model information representing a three-dimensional shape of an object in the photographing area based on the plurality of images acquired by the image acquiring means.
The first or second aspect of the present invention, wherein the image generation means generates the virtual viewpoint image based on the model information generated by the model generation means and the viewpoint information acquired by the information acquisition means. Image processing equipment. The image processing apparatus according to claim 13, wherein the model generating means does not generate model information of an object not located in the partial region specified by the specific means. The image generation means generates the virtual viewpoint image without using the model information of an object not located in the partial region specified by the specific means among the model information generated by the model generation means. The image processing apparatus according to claim 13. The shooting area is a stadium where multiple types of competitions are held.
The specific means according to any one of claims 1 to 15, wherein the region where the specific event is performed is specified as the partial region among a plurality of regions where competitions of different events are performed. Image processing equipment. It has a setting means for setting whether or not to limit the display of an object not located in the partial area specified by the specific means.
The image processing apparatus according to any one of claims 1 to 16, wherein the image generation means generates the virtual viewpoint image according to the setting by the setting means. A holding means that holds correspondence information that associates multiple virtual viewpoints with events,
It has a receiving means for receiving an input according to a user operation for selecting one of the plurality of virtual viewpoints.
The specific means is a partial area in which an event corresponding to a virtual viewpoint selected by the user operation is performed based on the input received by the receiving means and the corresponding information held by the holding means. The image processing apparatus according to any one of claims 1 to 17, wherein the image processing apparatus is specified as. Any of claims 1 to 18, wherein the display unit includes a display control means for displaying on the display unit an image that identifiablely indicates the partial area specified by the specific means among the plurality of areas in the photographing area. The image processing apparatus according to item 1. The image processing apparatus according to any one of claims 1 to 19, wherein the information acquisition means acquires viewpoint information indicating the position and orientation of a virtual viewpoint according to an event selected by a user operation. .. An image acquisition process for acquiring a plurality of images based on shooting by a plurality of shooting devices that shoot the shooting area from different directions, and
An information acquisition process for acquiring viewpoint information indicating the position and orientation of a virtual viewpoint,
A specific step of specifying a partial area in the shooting area where a specific event is performed among the events performed in the shooting area, and
A virtual viewpoint image according to the position and orientation of the virtual viewpoint, wherein the display of objects not located in the partial area specified in the specific step among the objects in the shooting area is restricted. An image processing method comprising the plurality of images acquired in the image acquisition step and an image generation step of generating based on the viewpoint information acquired in the information acquisition step. In the virtual viewpoint image generated by the image generation step, an object included in the visual field according to the position and orientation of the virtual viewpoint and located in the partial region is displayed, and the object is positioned in the partial region. The image processing method according to claim 21, wherein an object that does not display is not displayed. The shooting area is a stadium where multiple types of competitions are held.
The image processing method according to claim 22, wherein the specific step specifies a region in which the specific event is performed among a plurality of regions in which competitions of different events are performed as the partial region. A program for causing a computer to function as each means of the image processing apparatus according to any one of claims 1 to 20.

Images