JP7494153B2 - Generation device, generation method, and program - Google Patents

Description

This disclosure relates to technology for generating virtual viewpoint images.

In recent years, a technology that uses multiple image capture devices arranged around a capture area to capture images and generates an image (virtual viewpoint image) viewed from a specified viewpoint (virtual viewpoint) using the multiple captured images acquired from each of the capture devices has been attracting attention. This technology can generate images that allow users to view sporting events such as soccer and rugby, concerts, and dances from any viewpoint, providing a highly realistic experience to the user.

Patent Document 1 discloses a technology for generating and displaying arbitrary virtual camera images using images of a subject captured by multiple cameras arranged to surround the subject. According to Patent Document 1, in viewing content using the technology for generating virtual camera images, it becomes possible to view the dance, performance, etc. of a filmed performer (subject) from various angles.

JP 2008-15756 A

However, for example, in dancing or acting, the subject may be in a position different from the desired position during filming. As a result, the subject's position in the virtual viewpoint image generated based on the filmed image will also be different from the desired position. Furthermore, if the subject's position is misaligned during filming, depending on the specified virtual viewpoint position, the user may not notice that the subject's position in the virtual viewpoint image is different from the desired position.

This disclosure has been made in consideration of the above problems. Its purpose is to generate a virtual viewpoint image that can accommodate cases where the subject's position is different from the desired position.

The generating device according to the present disclosure is characterized in that it has an acquisition means for acquiring shape data representing the shape of a subject based on multiple captured images obtained by multiple imaging devices capturing images of the subject in a shooting area, a first identification means for identifying multiple subject positions corresponding to the positions of the multiple subjects in the shooting area, a second identification means for identifying a reference position based on the multiple subject positions identified by the first identification means, and a generating means for generating a virtual viewpoint image based on the shape data acquired by the acquisition means according to the deviation between the subject positions identified by the first identification means and the reference position identified by the second identification means.

According to the present disclosure, it is possible to generate a virtual viewpoint image that can accommodate the subject's position being different from the desired position.

FIG. 1 is a diagram illustrating a configuration of an image processing system. FIG. 1 is a diagram showing an example of installation of a plurality of image capturing devices. FIG. 2 is a diagram illustrating a hardware configuration of an image generating apparatus. FIG. 2 is a diagram for explaining the functional configuration of an image generating apparatus according to the first embodiment. 4 is a flowchart for explaining a process performed by the image generating apparatus according to the first embodiment. 5 is a diagram showing an example of an arrangement of shape data of a subject in the first embodiment; FIG. FIG. 4 is a diagram showing an example of a virtual viewpoint image generated in the first embodiment. FIG. 11 is a diagram for explaining the functional configuration of an image generating apparatus according to a second embodiment. 10 is a flowchart illustrating a process performed by an image generating apparatus according to a second embodiment. FIG. 11 is a diagram showing an example of an arrangement of shape data of a subject in the second embodiment. FIG. 13 is a diagram for explaining the functional configuration of an image generating apparatus according to a third embodiment. 13 is a flowchart illustrating a process performed by an image generating apparatus according to a third embodiment. FIG. 13 is a diagram showing an example of an arrangement of shape data of a subject in the third embodiment. FIG. 13 is a diagram for explaining the functional configuration of an image generating apparatus according to a fourth embodiment. FIG. 13 is a diagram illustrating an example of installation of a virtual camera in the fourth embodiment. 1A and 1B are diagrams illustrating an example of a subject to be photographed and a virtual viewpoint image to be generated. 5 is a diagram showing an example of arrangement of object shape data outside a shooting area in the first embodiment; FIG.

Embodiments of the present disclosure will be described below with reference to the drawings. Note that the components described in the following embodiments are examples of embodiments, and the present disclosure is not limited to these.

First Embodiment
1 is a diagram showing an image processing system 100 according to this embodiment. The image processing system 100 includes a plurality of image capture devices 110, an image generation device 120, and a terminal device 130. Each image capture device 110 and the image generation device 120 are connected via a communication cable such as a LAN (Local Area Network) cable. Note that in this embodiment, the communication cable is a LAN cable, but the communication cable is not limited to this embodiment.

The image capture device 110 is, for example, a digital camera capable of capturing images (still images and video). Each image capture device 110 is installed to surround a capture area in a photography studio or the like, and captures images (video). In this embodiment, an example will be described in which multiple performers are captured as subjects, such as in a dance scene. The captured images are sent from the image capture device 110 to the image generation device 120. Figure 2 is a diagram showing an example of the installation of the image capture device 110. In this embodiment, the multiple image capture devices 110 are each installed to capture all or part of the photography studio. In other words, the image processing system 100 of this embodiment includes multiple image capture devices 110 for capturing images of a subject from multiple directions.

The image generating device 120 accumulates the captured images obtained by the image capturing device 110, and when virtual viewpoint information and playback time information are input by a user operation on the terminal device 130, the image generating device 120 generates a virtual viewpoint image based on the captured image and the virtual viewpoint. Here, the virtual viewpoint information includes information indicating the position of a virtual viewpoint (virtual viewpoint) in a virtual space constructed from the captured image and the line of sight from the virtual viewpoint. Note that the information included in the virtual viewpoint information is not limited to this. For example, information regarding the width of the field of view (angle of view) of the virtual viewpoint may be included. Also, a configuration may be adopted in which information indicating at least one of the position of the virtual viewpoint, the line of sight from the virtual viewpoint, and the angle of view of the virtual viewpoint is included. The playback time information is time information from the start time of recording of the captured image. For example, the user can generate a virtual viewpoint image of a scene corresponding to the specified playback time in the recorded captured image by operating the terminal device 130 described later to specify the playback time.

The image generating device 120 is, for example, a server device, and has a database function and an image processing function. In the database, images captured in advance of a scene in which no subject is present, such as before the start of actual filming, are stored as background images via the filming device 110. In addition, in a scene in which a subject is present, the image generating device 120 separates areas of the captured image that correspond to people such as performers and specific objects such as tools used by performers (hereinafter also referred to as foreground images) through image processing, and stores the areas as foreground images. Note that the specific objects may be objects such as props, whose image patterns are predetermined.

The virtual viewpoint image corresponding to the virtual viewpoint information is generated from a background image and a specific object image managed in a database. For example, model-based rendering (MBR) is used as a method for generating the virtual viewpoint image. MBR is a method for generating a virtual viewpoint image using a three-dimensional shape generated based on multiple captured images of a subject taken from multiple directions. Specifically, it is a technology that uses a three-dimensional shape (model) of a target scene obtained by a three-dimensional shape restoration method such as a visual volume intersection method or Multi-View-Stereo (MVS) to generate an image of the scene as seen from a virtual viewpoint. Note that the method for generating the virtual viewpoint image may use a rendering method other than MBR. The generated virtual viewpoint image is transmitted to the terminal device 130 via a LAN cable or the like.

The terminal device 130 is, for example, a PC (Personal Computer) or a tablet. The controller 131 is, for example, a mouse, a keyboard, a six-axis controller, or a touch panel, which are operated by the user to display still images and moving images on the screen. The terminal device 130 displays, for example, a virtual viewpoint image received from the image generating device 120 on the display unit 132. The terminal device 130 further receives instructions for the playback time and movement of the virtual viewpoint (instructions regarding the amount of movement and the direction of movement) in response to user operations on the connected controller 131, and transmits a transmission signal indicating instruction information corresponding to the received instruction to the image generating device 120.

Figure 3 shows the hardware configuration of the image generating device 120.

The image generating device 120 has a CPU 301, a ROM 302, a RAM 303, a HDD 304, a display unit 305, an input unit 306, and a communication unit 307. The CPU 301 reads out a control program stored in the ROM 302 and executes various processes. The RAM 303 is used as a temporary storage area such as the main memory and work area of the CPU 301. The HDD 304 stores various data and various programs. The display unit 305 displays various information. The input unit 306 has a keyboard and a mouse, and accepts various operations by the user. The communication unit 307 performs communication processing with an external device such as the imaging device 110 via a network. Note that an example of the network is Ethernet (registered trademark). As another example, the communication unit 307 may communicate with an external device wirelessly.

In the example shown in FIG. 3, the HDD 304, the display unit 305, and the input unit 306 are included inside the image generating device 120, but this is not limiting. For example, at least one of the HDD 304, the display unit 305, and the input unit 306 may be connected to the outside of the image generating device 120 as another device. Furthermore, the functions and processing of the image generating device 120 described below are realized by the CPU 301 reading out a program stored in the ROM 302 or the HDD 304 and executing this program. Furthermore, the hardware configuration of the terminal device 130 is similar to the hardware configuration of the image generating device 120.

Figure 4 is a diagram showing the functional configuration of the image generating device 120. Here, the processing performed by the image generating device 120 in this embodiment will be described. In this embodiment, it is assumed that a dance scene is shot and a virtual viewpoint image is generated based on the shot image. However, the subject's standing position during the dance may be a position different from the desired standing position. Figure 16 (a) is a conceptual diagram of a subject in a virtual space generated by shooting a shooting area surrounded by a line with multiple cameras, and a virtual camera (hereinafter referred to as a virtual camera) corresponding to a virtual viewpoint. A virtual camera that shoots a subject from a free angle can be specified, such as a case where the subject is shot from the front and viewed as in the virtual camera 2101, or a case where the subject is shot from the side and viewed as in the virtual camera 2102. Figure 16 (c) is a screen display of a virtual viewpoint image corresponding to the virtual camera 2102. Here, it is assumed that the performer wants to perform from a standing position that will be in a straight line when shot from the virtual camera 2102. However, when actually photographing, the subjects' positions may shift, resulting in a virtual viewpoint image in which the subjects' positions are not aligned in a straight line, as shown in Figure 16(c).

Another possible use would be for an actor to check his/her standing position by looking at the virtual viewpoint image. In this case, depending on the virtual camera specified, it may not be easy to recognize the deviation in standing position. FIG. 16(b) is a screen display of a virtual viewpoint image corresponding to virtual camera 2101. In the virtual viewpoint image corresponding to virtual camera 2101, the subjects appear to be positioned in the same way. However, in reality, the subjects are positioned out of place in the virtual viewpoint image captured by virtual camera 2102. The image generating device 120 in this embodiment aims to solve the above problem.

The image generating device 120 has a captured image input 401, a foreground/background separation unit 402, a captured image data storage unit 403, a camera parameter holding unit 404, a subject shape generation unit 405, a subject shape center of gravity calculation unit 406, a subject shape movement unit 407, and a subject position determination unit 408. The image generating device 120 also has a user input unit 409, a virtual viewpoint information setting unit 410, a coloring information calculation unit 411, a virtual viewpoint image generation unit 412, and an image output unit 413. Each processing unit will be described below.

The captured image input unit 401 converts a transmission signal input from the image capture device 110 via a LAN cable into captured image data and outputs it to the foreground/background separation unit 402. The foreground/background separation unit 402 outputs, from the captured images input from the captured image input unit 401, images captured of a scene in which the subject is not present beforehand, such as before the subject begins their performance, as background image data to the captured image data storage unit 403. In addition, the unit extracts the subject from an image captured during the subject's performance, and outputs it to the captured image data storage unit 403 as foreground image data.

The photographed image data storage unit 403 is a database, and among the photographed image data input from the foreground/background separation unit 402, the unit stores an image that was previously photographed without the presence of a subject as background image data in the HDD 304. The photographed image data storage unit 403 also stores difference data between the background image data and the photographed image data in which a subject is present as foreground image data in the HDD 304. The photographed image data storage unit 403 also outputs foreground image data to the subject shape generation unit 405. The unit also outputs the background image data and foreground image data specified by the coloring information calculation unit 411 to the coloring information calculation unit 411.

The camera parameter holding unit 404 holds, as camera parameter information, information on the shooting positions of the multiple image capture devices 110, the focal length of the lenses of the image capture devices 110, and camera setting information such as the shutter speed of the image capture devices 110. The multiple image capture devices 110 are installed at predetermined positions, and the camera parameter information is acquired in advance. The camera parameter holding unit 404 also outputs the camera parameter information to the subject shape generation unit 405 and the coloring information calculation unit 411.

The subject shape generation unit 405 generates shape data representing the shape of the subject using the foreground image data and the camera parameter information. The subject shape generation unit 405 generates the shape data of the subject using a three-dimensional shape restoration method such as a volume intersection method. The subject shape generation unit 405 also outputs the shape data to the subject shape center of gravity calculation unit 406 and the coloring information calculation unit 411.

The subject shape center of gravity calculation unit 406 identifies the position of the subject in the shooting area. Specifically, the subject shape center of gravity calculation unit 406 uses the shape data input from the subject shape generation unit 405 to identify the center of gravity of the subject's shape as the subject position. At this time, the subject shape center of gravity calculation unit 406 calculates the center of gravity position when the subject is viewed from a specified viewpoint position. The subject shape center of gravity calculation unit 406 calculates, for example, the center of gravity position at a viewpoint looking down on the subject from directly above as the subject position. The subject shape center of gravity calculation unit 406 outputs subject center of gravity information consisting of the subject center of gravity position to the subject shape movement unit 407.

The subject shape moving unit 407 determines the position at which to place the subject's shape data based on the subject's center of gravity information input from the subject shape center of gravity calculation unit 406 and the subject's destination position information input from the subject position setting unit 408 described below. Here, the subject's destination position information is information that represents the reference position (hereinafter also referred to as the reference position) at which the subject is to be placed. The subject shape moving unit 407 determines the placement of the shape data depending on the deviation between the reference position and the position of the subject.

When the subject destination position information is lattice point information at a predetermined interval (e.g., 3 meter intervals) set on the floor surface, the subject shape moving unit 407 uses the position of the lattice point as a reference position and arranges the shape data of the subject so that the position of the lattice point coincides with the position of the center of gravity of the subject. Note that the subject shape data may be regenerated based on the subject destination position information. Furthermore, when the deviation between the reference position and the position of the subject is equal to or greater than a predetermined threshold, the subject shape moving unit 407 may change the position of the shape data so that the deviation becomes smaller than the predetermined threshold, and arrange the shape data at the changed position. Note that the subject shape moving unit 407 outputs the moved shape data to the virtual viewpoint image generating unit 412.

The subject position setting unit 408 outputs subject destination position information in three-dimensional space, which has been set in advance by the user, to the subject shape moving unit 407. For example, it outputs lattice point information at intervals of 3 meters corresponding to the floor surface so that multiple subjects are placed in a specified section on a specified straight line. Note that the subject destination position information is not limited to lattice points, and may be, for example, a straight line or a curve.

The user input unit 409 converts a transmission signal input from the terminal device 130 via a LAN cable into user input data. If the user input data is playback time information and virtual viewpoint information, the user input unit 409 outputs the playback time information and the virtual viewpoint information to the virtual viewpoint information setting unit 410.

The virtual viewpoint information setting unit 410 updates the current position, direction, and playback time in the virtual space based on the playback time information and virtual viewpoint information input from the user input unit 409.

Then, the playback time information and virtual viewpoint information are output to the subject shape generation unit 405, the coloring information calculation unit 411, and the virtual viewpoint image generation unit 412. Note that the origin of the virtual space is set in advance to the center of the stadium, etc.

The coloring information calculation unit 411 inputs foreground image data and background image data based on the playback time information and virtual viewpoint information input from the virtual viewpoint information setting unit 410 from the captured image data storage unit 403. It also inputs camera parameters from the camera parameter holding unit 404 and shape data from the subject shape generation unit 405. Next, the subject shape as seen from the virtual viewpoint position is rendered (colored) using color information of image data captured by a real camera at the corresponding time, and coloring information of the subject shape is stored. For example, when a subject based on shape data is visible from the virtual viewpoint, and real camera position information is within a specified range from the virtual viewpoint position, the foreground image data of the real camera is used as the color of the shape. The coloring information calculation unit 411 also outputs coloring information to the virtual viewpoint image generation unit 412.

The virtual viewpoint image generating unit 412 inputs the foreground image data and background image data based on the playback time information and virtual viewpoint information input from the virtual viewpoint information setting unit 408 from the captured image data storage unit 403. It also inputs the camera parameters from the camera parameter holding unit 404, and also inputs the moving shape data from the subject shape moving unit 407. After that, the background image data is subjected to projection transformation and image processing so that it is seen as the background from the virtual viewpoint position, and becomes the background of the virtual viewpoint image. Next, for the moving subject shape seen from the virtual viewpoint position, coloring information based on the image data captured by the shooting device at the corresponding time is input from the coloring information calculation unit 411, and rendering (coloring processing) is performed with the color information to generate a virtual viewpoint image. Finally, the virtual viewpoint image generated by the virtual viewpoint image generating unit 412 is output to the image output unit 413. The image output unit 413 converts the image data input from the virtual viewpoint image generating unit 412 into a transmission signal that can be transmitted to the terminal device 130, and outputs it to the terminal device 130.

Next, the operation of the image generating device 120 will be described. FIG. 5 is a flowchart showing the operation of the image generating device 120 according to the first embodiment. The CPU 301 reads out and executes a program stored in the ROM 302 or the HDD 304, thereby performing the following processing. When an input is made from the terminal device 130 to specify virtual viewpoint information and a playback time, the processing is started.

The virtual viewpoint information setting unit 410 determines whether or not virtual viewpoint information and playback time information have been input via the user input unit 409 (S501). If virtual viewpoint information and playback time information have not been input (No in S501), the unit waits. On the other hand, if virtual viewpoint information and playback time information have been input (Yes in S501), the unit outputs the virtual viewpoint information and playback time information to the subject shape generation unit 405, the coloring information calculation unit 411, and the virtual viewpoint image generation unit 412.

Next, the subject shape generation unit 405 reads the foreground image data input from the captured image data storage unit 403 based on the playback time information input from the virtual viewpoint information setting unit 410, and the camera parameter information input from the camera parameter holding unit 404 (S502). Next, the subject shape generation unit 405 estimates the three-dimensional shape of the subject (S503). For example, it is assumed that the shape data of the subject is generated using a three-dimensional shape restoration method such as a visual volume intersection method. Here, the shape data is assumed to be composed of a group of multiple points, with each point including position information.

Next, the coloring information calculation unit 411 inputs from the captured image data storage unit 403 the foreground image data and background image data based on the playback time information and virtual viewpoint information input from the virtual viewpoint information setting unit 410. It also inputs the camera parameters from the camera parameter holding unit 404 and the shape data from the subject shape generation unit 405. Next, the subject shape as seen from the virtual viewpoint position is rendered (coloring process) using the color information of the image data captured by the real camera at the corresponding time, and the coloring information of the subject shape is stored (S504).

Next, the subject shape center of gravity calculation unit 406 identifies the center of gravity position of the subject shape input from the subject shape generation unit 405 when viewed from a specified viewpoint position as the subject position (S505). In this embodiment, the center of gravity position when viewed from directly above is used as the subject shape center of gravity information. FIG. 6(a) is a conceptual diagram of the shape data of the subject viewed from directly above in virtual space and the center of gravity position of the shape data. In this embodiment, in order to rearrange multiple subjects on specified lattice points, the center of gravity position is calculated by calculating the center of the front-back axis and the center of the left-right axis of the subject from a viewpoint looking down from directly above the subject. The center of gravity position is indicated by a black dot on the subject, and the center of gravity position is positioned on a straight line or a lattice point.

Returning to the explanation of FIG. 5, the subject shape moving unit 407 determines whether or not there is subject destination position information in three-dimensional space from the subject position setting unit 408 (S506). If there is subject destination position information in three-dimensional space from the subject position setting unit 408 (Yes in S506), the subject destination position information is input (S507). On the other hand, if there is no subject destination position information (No in S506), the process proceeds to S509. FIG. 6(b) is a conceptual diagram of lattice point positions on the floor surface, which represent the reference position of the subject based on the subject destination position information. For example, assume that lattice point information at 3-meter intervals corresponding to the floor surface is output so that multiple subjects are placed in a specified section on a specified straight line.

Returning to the explanation of FIG. 5, next, the subject shape moving unit 407 moves the shape data based on the subject centroid information input from the subject shape centroid calculation unit 406 and the subject destination position information input from the subject position setting unit 408 (S508). FIG. 6(c) is a conceptual diagram of the subject shape after movement. It shows that the subject position has been changed so that the grid point position based on the subject destination position information matches the centroid position of the subject shape when viewed from directly above the subject shape, and the shape data has been moved to the changed subject position. This means that the position of the moved subject shape has been moved to a certain distance apart. Note that the subject position may be changed and the shape data may be positioned so that the reference position and subject position are smaller than a specified threshold.

Returning to the explanation of FIG. 5, the virtual viewpoint image generating unit 412 generates a virtual viewpoint image by rendering (coloring) the shape data seen from the virtual viewpoint position using image data captured by the image capturing device 110 at a specified time based on the moved shape data (S509). That is, the color information of the subject determined and held before the subject movement is rendered after the subject movement, so that the subject is displayed at the position to which it has been moved. The virtual viewpoint image generating unit 412 outputs the virtual viewpoint image generated by the virtual viewpoint image generating unit 412 to the image output unit 413 (S510). The above-described processing may be performed based on recorded and stored image data, or may be performed in real time in parallel with the image capturing by the image capturing device 110.

Note that, in the explanation of FIG. 5 above, it is explained that the color information of the subject that was determined and held before the subject moved is rendered after the subject moved, but this is not limited to this. For example, the target subject is moved to a specified position, and the shooting positions of all the shooting devices 110 and positions other than the target subject are moved relatively to maintain their positional relationship with the target subject, and only the target subject is rendered. This may be performed sequentially for each subject, and the virtual viewpoint image may be generated by combining the results. In other words, instead of determining the color information before the subject moves, the color information may be calculated and rendered after the subject moves.

Figure 7 is a diagram showing a virtual viewpoint image generated by performing the processing shown in Figure 5 when the subject shown in Figure 16 (a) is photographed. Figure 7 (a) shows the position of the shape data of the subject in three-dimensional space corresponding to the photographing area. According to Figure 7 (a), multiple subjects, while performing a dance or other performance, are moved to a specified lattice point position, which is a different standing position from when performing the performance, and the virtual camera photographs them performing the same movement from a standing position at a certain distance. Whether the subjects are photographed from the position of virtual camera 901 or the position of virtual camera 902, they will be in a different standing position compared to the standing position when they actually performed a dance or other performance.

Figure 7(b) is an example of a virtual viewpoint image corresponding to virtual camera 901, displayed on the terminal device 130. Even if the subject is photographed from the front, it is possible to have the subject perform the action while always maintaining a constant position. Also, Figure 7(c) is an example of a virtual viewpoint image corresponding to virtual camera 902. Even if the subject is photographed from the side, it can be rearranged and displayed in a straight line, so it is possible to have the subject perform the action while always maintaining the same position.

As described above, according to the first embodiment, when multiple subjects are photographed by multiple cameras and photographed from a virtual viewpoint and displayed, it is possible to generate a virtual viewpoint image in which the subjects are repositioned at a desired position. Compared to when the performance is photographed, it is possible to view a performance with the subjects aligned in the image of the virtual camera. In the above example, the reference position is represented by a lattice point, but the reference position may be represented by a predetermined straight line or curve. In this case, if the subject position deviates from the line, the shape data is positioned so that the subject position coincides with the position of an arbitrary point on the line (for example, a point on the line closest to the current subject position). The reference position may also be a point (coordinate) at an arbitrary position. In this case, if there are multiple subjects, a reference position point may be set for each subject.

The reference position may also be specified based on the subject positions of multiple subjects. For example, suppose that when three subjects are to be photographed, it is desired to set the standing positions so that the three subjects are positioned in a straight line. In this case, a line connecting the subject positions of two of the three subjects is calculated, and a point on the calculated line is used as the reference position, and the subject position of the remaining person is changed. In this way, a virtual viewpoint image is generated in which shape data is arranged so that the three subjects are positioned in a straight line. Note that this can also be applied to cases where the number of subjects is other than three.

The reference position is not limited to placing the subject at a three-dimensional space position corresponding to the shooting area as shown in FIG. 7(a), but may be set to any three-dimensional space position outside the shooting area. FIG. 17 shows an example in which the shape data of the subject is placed at a three-dimensional space position outside the shooting area. In the example shown in FIG. 17, the reference position for moving the shape data of the target subject is set to a position outside the shooting area. According to FIG. 17, even if multiple subjects are actually photographed performing a dance or other performance within the shooting area, they are photographed from the position of the virtual camera 901 as if they are performing a dance or other performance in an area larger than the shooting area.

Second Embodiment
In this embodiment, the position of the subject is changed based on a predetermined feature of the subject. FIG. 8 is a diagram showing a functional configuration of an image generating device 1100 according to the second embodiment. The image generating device 1100 has a subject feature generating unit 1101 instead of the subject shape center of gravity calculation unit 406 of the image generating device 120 according to the first embodiment shown in FIG. 4. Note that the hardware configuration of the image generating device 1100 is assumed to be the same as that of the image generating device 120 according to the first embodiment. Also, the same reference numerals are used for the same components as those of the image generating device 120, and the description thereof will be omitted.

The subject feature generation unit 1101 calculates the recognition of specific features of the subject and their positions from the shape data and coloring information corresponding to the shape data input from the subject shape generation unit 405. For example, when rearranging the faces of multiple subjects as features on a straight line or on specific grid points on the floor surface, the subject's faces are recognized using the shape data and coloring information, and the position of the faces is identified. As a result, in subsequent processing, the shape data is arranged so that the face position is on a specific straight line or grid point when viewed from directly above. In other words, the subject feature generation unit 1101 identifies a specific part of the subject's shape as the subject position. The subject feature generation unit 1101 outputs the subject feature position information to the subject shape movement unit 407.

Figure 9 is a flowchart showing image processing by the image processing device 1100 according to the second embodiment. Note that steps S501 to S504 are the same as those in Figure 5, and therefore their explanations are omitted. Steps S506 to S510 are also the same as those in Figure 5, and therefore their explanations are omitted.

In S1201, the subject shape generation unit 405 identifies the position of a predetermined part as the subject position. At this time, the subject shape generation unit 405 identifies a position having a predetermined characteristic in the subject's shape data by using image analysis such as face recognition.

Figure 10 shows an example of a case where the subject's face is recognized and analyzed from shape data and coloring information, and the position of the face is identified and taken as the subject position. The subject feature generation unit 1101 identifies the position of the subject's face, as shown in Figure 10(a). The subject shape movement unit 407 also positions the shape data based on the subject position identified by the subject feature generation unit 1101. As a result, the shape data is positioned so that the position of the face coincides with the lattice points when viewed from above, as shown in Figure 10(b).

Note that feature recognition (e.g., facial recognition) and position identification of the subject are performed by extracting features (faces) from multiple captured images obtained by the image capture device 110, and then calculating the positions of the features (positions in three-dimensional space) based on camera parameter information. However, this is not limited to the above, and for example, a virtual camera may be set at a predetermined position to generate a virtual viewpoint image, and feature recognition and position identification may be performed using the generated virtual viewpoint image.

As described above, the image generating device 1100 of this embodiment identifies the position of a specific part of the subject, arranges shape data according to the deviation between the identified position and a reference position, and generates a virtual viewpoint image. The specific part is not limited to the face, but may be the hands, feet, shoes, etc. As a result, when shooting a shoe commercial, for example, it is possible to identify the characteristics of the shoes and arrange the shape data so that the position of the shoes matches the reference position or the position deviation is smaller than a specific threshold. In this way, a virtual viewpoint image can be generated in which the shoes are positioned in a desired position.

Third Embodiment
The third embodiment is an example in which the position of the subject is changed based on the position of the center of gravity of the subject in the time axis direction. Fig. 11 is a diagram showing the functional configuration of an image generating device 1500 according to the third embodiment. The image generating device 1500 has a subject shape average center of gravity calculation unit 1601 instead of the subject shape center of gravity calculation unit 406 of the image generating device 120 according to the first embodiment shown in Fig. 4. Note that the hardware configuration of the image generating device 1500 in this embodiment is assumed to be the same as that of the above-mentioned embodiment. Also, the same reference numerals are given to the same functional configurations, and the description will be omitted.

The subject shape average center of gravity calculation unit 1501 calculates the center of gravity position of the subject in each video frame of the captured image obtained by the imaging device 110. For example, if a 10-minute dance scene is captured, the center of gravity position from a viewpoint looking down on the subject from directly above is identified in each frame, and the average position of the center of gravity positions of each video frame is calculated. The subject shape average center of gravity calculation unit 1501 sets the calculated average position of the centers of gravity as the subject position, and outputs information on the average position to the subject shape movement unit 407.

Figure 12 is a flowchart showing image processing by the image processing device 1500 according to the third embodiment. Note that, among the processes shown in Figure 16, the description of the same processes as those shown in Figure 5 will be omitted.

After the processing of S504, the subject shape average center of gravity calculation unit 1501 calculates the center of gravity position when viewed from a specified viewpoint position of the subject shape of the multiple shooting frames input from the subject shape generation unit 405. In addition, the subject shape average center of gravity calculation unit 1501 calculates the average position of the center of gravity based on the center of gravity of the subject in each video frame, and outputs information on the average position to the subject shape movement unit 407 (S1601).

Figure 13 is a diagram showing the average position of the center of gravity of a subject. For example, suppose that in a series of dance scenes, the subject moves in the direction of the arrow shown in Figure 13(a). Here, the position of the center of gravity of the subject is calculated for a frame before the movement during the subject's performance and a frame after the movement during the performance. Furthermore, the average position of the center of gravity of each frame is calculated. In Figure 13(a), point 1301 is identified as the average position of the center of gravity.

Figure 13 (b) shows the case where the average position of the center of gravity is moved based on the reference position. When viewed from directly above the subject, the shape data is positioned so that the average center of gravity position of the subject coincides with the grid point position based on the subject destination position information. As a result, the subject is positioned at a constant position on average during a series of shots, and moves around that position.

As described above, according to this embodiment, the average position of the center of gravity of the subject over the shooting time is calculated, and the subject is repositioned at a predetermined height viewpoint, making it possible to reposition the subject without creating a sense of incongruity even with a series of movements in the scene. Note that, although this embodiment calculates the average center of gravity position, this is not limited to this, and the center of gravity position of the subject in any frame of shooting may be used as the basic position. For example, it is possible to reposition the center of gravity position of the subject in the frame at the start of shooting to match the grid point position of the floor surface, or the center of gravity position of the subject in the frame immediately before the end of shooting may be used. Also, in this embodiment, the center of gravity of the subject is identified, and the average position of the center of gravity is used as the subject position, but this is not limited to this. For example, the position of the specified part described in the second embodiment may be identified, and the average position may be used as the subject position.

Fourth Embodiment
In the fourth embodiment, a method is described in which the subject is not moved, but the virtual camera is moved relatively based on the positional relationship before and after the movement of the subject, and the captured image is synthesized with the pre-movement virtual camera image. Fig. 14 has a subject position difference calculation unit 1901, a multiple chaos viewpoint information setting unit 1902, and a multiple virtual viewpoint image generation unit 1903. Note that the hardware configuration of the image generation device 1100 is the same as that of the above-mentioned embodiment. Also, the same reference numerals are used for the same configurations, and the description will be omitted.

The subject position difference calculation unit 1901 inputs subject center of gravity information from the subject shape center of gravity calculation unit 405. It also inputs information on the reference position of the subject from the subject position setting unit 408, and calculates the difference between the reference position and the subject center of gravity position.

The multiple virtual viewpoint setting unit 1902 updates the current position, direction, and playback time in the virtual space based on the playback time information and virtual viewpoint information input from the user input unit 409, and outputs the playback time information and virtual viewpoint information to the multiple virtual viewpoint image generation unit 1903. In addition, when the multiple virtual viewpoint setting unit 1902 receives the positional relationship between the pre-movement position of the subject and the post-movement position of the subject as subject difference information from the subject position difference calculation unit 1901, it generates difference virtual viewpoint information based on the subject difference information.

The multiple virtual viewpoint image generating unit 1903 inputs the foreground image data and background image data based on the playback time information and virtual viewpoint information input from the multiple virtual viewpoint information setting unit 1902 from the captured image data storage unit 403. The camera parameters are also input from the camera parameter holding unit 404, and the shape data after movement is input from the object shape moving unit 407. After that, the background image data is subjected to projection transformation and image processing so that it can be seen as the background from the virtual viewpoint position, and becomes the background of the virtual viewpoint image. Next, the moving object shape as seen from the virtual viewpoint position is rendered (colored) with color information based on the image data captured by the real camera at the corresponding time to generate a virtual viewpoint image. In addition, when difference virtual viewpoint information resulting from the object position difference calculation unit 1901 calculating the object difference information is input, a virtual viewpoint image is generated at the specified position of the difference virtual viewpoint information, and is composited with the virtual viewpoint image based on the user input. Finally, the virtual viewpoint image is output to the image output unit 413.

Figure 15 is a conceptual diagram of the virtual camera position at the subject position to be moved in virtual space. The center (0,0) of the floor surface is set as the origin, and the subject position information after the movement is also set as the center (0,0) of the floor surface. The unit is meters. If the position of the subject before the movement is (0,2), when placing the subject at the reference position (0,0), the position (x,y) of the virtual camera 2001 is moved by (0,2) to a position (x,y+2). Then, the subject is photographed from the moved virtual camera 2001 to generate a virtual viewpoint image. Furthermore, the generated virtual viewpoint image is composited with a virtual viewpoint image corresponding to the virtual camera 2001 that photographed an object other than the subject to be moved.

As described above, according to this embodiment, a virtual viewpoint image is generated by moving a virtual camera relative to a subject to be moved, and is then composited with a virtual viewpoint image corresponding to a virtual camera capturing an image of an object other than the subject to be moved. This makes it possible to move the position of the subject simply by composite of the virtual viewpoint images.

Other Embodiments
In the above-mentioned embodiment, an example was described in which the shape data of the subject is rearranged according to the deviation between the subject position and the reference position to generate a virtual viewpoint image. However, the present invention is not limited to this, and a configuration may be used in which a virtual viewpoint image is generated that includes information that can identify that the subject position is offset from the reference position. For example, shape data may be arranged at a specified subject position, and information indicating the reference position (for example, a lattice point or a straight line) may be displayed. In addition, shape data may be arranged at the subject position, and a display that makes it possible to identify a subject that is offset from the reference position may be performed. For example, the shape data of a subject that is offset from the reference position may be surrounded by a line, or the color or brightness of the subject may be changed to highlight the subject. With these methods, a user who views the virtual viewpoint image can recognize that the subject position is offset from the reference position. For example, when a performer wants to check his/her standing position by looking at the generated virtual viewpoint image, the offset of the performer's position can be easily identified regardless of the position of the virtual viewpoint.

The present disclosure can also be realized by supplying a program that realizes one or more of the functions of the above-described embodiments to a system or device via a network or a storage medium, and having one or more processors in the computer of the system or device read and execute the program. It can also be realized by a circuit (e.g., an ASIC) that realizes one or more of the functions.

120 Image generating device 405 Subject shape generating unit 406 Subject shape center of gravity calculating unit 408 Subject position setting unit 412 Virtual viewpoint image generating unit

Claims (17)

an acquisition means for acquiring shape data representing a shape of a subject based on a plurality of captured images obtained by capturing images of the subject in a capturing area using a plurality of image capturing devices;
a first specifying means for specifying a plurality of subject positions corresponding to positions of a plurality of subjects in the shooting area;
a second specifying means for specifying a reference position based on a plurality of subject positions specified by the first specifying means;
and a generation means for generating a virtual viewpoint image according to a deviation between a subject position identified by the first identification means and a reference position identified by the second identification means, based on the shape data acquired by the acquisition means. The generating device according to claim 1, characterized in that the generating means generates a virtual viewpoint image by arranging the shape data based on the reference position in response to the deviation being equal to or greater than a predetermined threshold value. The generating means includes:
changing the subject position identified by the first identification means so that the deviation becomes smaller than a predetermined threshold value;
The generating device according to claim 1 , further comprising: a virtual viewpoint image generating unit configured to generate the virtual viewpoint image by arranging the shape data based on a changed subject position. The generating means includes:
changing the subject position based on the deviation so that the subject position identified by the first identification means coincides with the reference position;
The generating device according to claim 1 , further comprising: a virtual viewpoint image generating unit configured to generate the virtual viewpoint image by arranging the shape data based on a changed subject position. The generating device according to any one of claims 1 to 4, characterized in that the reference position is identified based on lattice points arranged at a predetermined interval. The generating device according to any one of claims 1 to 5, characterized in that the reference position is a position outside the imaging area. The generating device according to any one of claims 1 to 6, characterized in that the second identification means identifies the reference position so that the multiple subject positions are spaced apart by a predetermined distance. The generating device according to any one of claims 1 to 7, characterized in that the second identification means identifies the reference position so that the multiple subject positions are positions on a predetermined straight line. The generating device according to any one of claims 1 to 8, characterized in that the first identification means identifies the subject position based on the shape of the subject represented by the shape data acquired by the acquisition means. The generating device according to claim 9, characterized in that the first identification means identifies, as the subject position, the position of the center of gravity in the shape of the subject represented by the shape data. The generating device according to claim 9, characterized in that the first identification means identifies, as the subject position, the position of a predetermined part in the shape of the subject represented by the shape data. The generating device according to any one of claims 1 to 11, characterized in that the generating means generates a virtual viewpoint image including information that allows the deviation to be identified. The generating device according to claim 12, characterized in that the generating means generates a virtual viewpoint image including information representing the subject position identified by the first identifying means and the reference position identified by the second identifying means as information that can identify the deviation. The generating device according to claim 12 or 13, characterized in that the generating means generates a virtual viewpoint image including information that can identify the deviation and that can identify a subject whose position that is deviated from the reference position is identified as the subject position by the first identifying means, as the information that can identify the deviation. an acquisition step of acquiring shape data representing a shape of a subject based on a plurality of captured images obtained by capturing images of the subject in a capturing area using a plurality of image capturing devices;
a first specifying step of specifying a plurality of subject positions corresponding to positions of a plurality of subjects in the shooting area;
a second specifying step of specifying a reference position based on the plurality of subject positions specified by the first specifying step ;
and a generation step of generating a virtual viewpoint image according to a deviation between the position identified in the first identification step and the reference position identified in the second identification step, based on the shape data acquired in the acquisition step. A program for causing a computer to function as a generating device according to any one of claims 1 to 14. an acquisition means for acquiring shape data representing a shape of a subject based on a plurality of captured images obtained by capturing images of the subject in a capturing area using a plurality of image capturing devices;
a first specifying means for specifying a plurality of subject positions corresponding to positions of a plurality of subjects in the photographing area;
a second specifying means for specifying a reference position based on a plurality of subject positions specified by the first specifying means;
and a generation means for generating a virtual viewpoint image according to a deviation between the subject position identified by the first identification means and the reference position identified by the second identification means, based on the shape data acquired by the acquisition means.

Images