JP5672109B2 - Imaging apparatus, imaging method, and program - Google Patents

Description

  The present invention relates to an imaging apparatus, an imaging method, and a program that enable a user to visually recognize an image having a desired composition only by moving the imaging apparatus while keeping his body fixed.

2. Description of the Related Art Conventionally, an imaging apparatus such as a digital camera has a display unit on its back surface, and the subject is captured in the imaging direction (the optical axis of the lens) from the stage before a recording instruction operation such as a shutter button pressing operation is performed. An image showing a state seen from the direction of the image is displayed on the display unit in real time.
Such an image is generally called a “live view image” or a “through image”. In the present specification, the term “through image” will be used in the description.

The user moves or rotates the image pickup apparatus while being held by a hand or the like, and checks whether or not a desired composition is obtained by looking at the through image each time. When the user confirms that a through image having a desired composition is displayed on the display unit, the user performs a recording instruction operation such as a shutter button pressing operation while maintaining the state.
Then, the image capturing apparatus records data of an image captured at the stage when the recording instruction operation is performed, that is, data of an image substantially the same as the through image of the desired composition on a memory card or the like.

As described above, a user often uses a through image to determine a desired composition before performing a recording instruction operation.
However, the through image, as described above, is an image showing a state in which the subject is viewed from the current imaging direction (the direction of the optical axis of the lens). For this reason, when the user cannot move upright and cannot move the imaging apparatus only by hand, the user's own body must be moved. Therefore, the user determines the desired composition. Operation is burdensome.
From such a situation, many users have recently demanded that a user wants to visually recognize an image having a desired composition only by moving the imaging apparatus while the body is fixed. However, in order to meet this requirement, a technique for displaying what cannot be seen with the naked eye at the present time is required.

As a conventional technique for displaying what cannot be seen with the naked eye at present, a technique for displaying a special image with an infrared camera, a night vision camera, an infrared thermography camera, or the like is known.
However, these special images are only images that fall within the field angle range of the current imaging direction (the direction of the optical axis of the lens).
However, an object to be displayed as being invisible to the naked eye at the present time is an object that cannot be viewed with the naked eye from the current viewpoint, that is, an object outside the range of the angle of view in the current imaging direction. In other words, if the viewpoint is moved, it is an object that can be seen with the naked eye, and in order to move the viewpoint, it is necessary to display an object on which the user must move the body. For example, when the imaging device faces the front side of a predetermined object, it is necessary to display the back side of the predetermined object.

As a conventional technique that can meet such a need, there is a CG (Computer Graphics) technique (hereinafter referred to as “CG technique”) that can construct a virtual world.
For example, Non-Patent Document 1 discloses a CG technique capable of drawing an image showing a state of viewing a model from a predetermined viewpoint using a three-dimensional model prepared in advance (Non-Patent Document 1). reference).
In response to such a technique of Non-Patent Document 1, Patent Documents 1 and 2 show images showing the target object viewed from an arbitrary viewpoint based on the three-dimensional shape data of the target object prepared in advance. Is described.

JP 2002-298160 A JP 2004-271671 A

Eijiro Shibusawa “Research on 3D reconstruction from uncalibrated multi-view images based on minimization of algebraic error” Doctoral dissertation, University of Electro-Communications Graduate School of Electro-Communications, March, 2010.

However, in order to display an image of a predetermined object viewed from an arbitrary viewpoint by applying conventional CG techniques including the CG techniques described in Non-Patent Document 1 and Patent Documents 1 and 2, It is assumed that a three-dimensional model (three-dimensional shape data) is prepared in advance.
Therefore, even if the conventional CG technology that requires such a premise is applied as it is, it is possible to satisfy the above-mentioned demand, that is, the user can move only the imaging device while fixing the body, and the desired composition can be obtained. It is difficult to meet the demand for viewing images.

  The present invention has been made in view of such a situation, and an object of the present invention is to make it possible to visually recognize an image having a desired composition only by moving the imaging apparatus while the user is fixing his body.

In order to achieve the above object, an imaging device of one embodiment of the present invention includes:
Imaging means for imaging one or more subjects from a predetermined imaging direction and outputting data of a captured image obtained as a result;
Subject-of-interest determining means for determining a subject of interest to be noted as a processing target from among the one or more subjects based on the data of the captured image output from the imaging means;
3D model generation and acquisition means for generating or acquiring the 3D model of the target object determined by the target object determination means;
Position information acquisition means for acquiring position information indicating the position and orientation of the imaging means;
The viewpoint identified according to the position information acquired by the position information acquisition means, and generated or acquired by the three-dimensional model generation acquisition means from a viewpoint in a direction different from the predetermined imaging direction. Virtual image generation means for generating, as virtual image data, image data indicating a view of the three-dimensional model of the subject of interest;
Display control means for controlling display of the virtual image generated as data by the virtual image generation means;
It is characterized by providing.

  According to the present invention, it is possible to visually recognize an image with a desired composition simply by moving only the imaging device while the body is fixed.

It is a block diagram which shows the structure of the hardware of the imaging device which concerns on one Embodiment of this invention. It is a functional block diagram which shows the functional structure which implement | achieves the execution function of the imaging recording process containing a virtual through display process among the functional structures of the imaging device of FIG. 3 is a flowchart illustrating a flow of an imaging recording process executed by the imaging apparatus of FIG. 1 having the functional configuration of FIG. 2. It is a figure explaining comparing the result of virtual through display processing with the result of normal through display processing. FIG. 4 is a flowchart illustrating an example of details of a flow of a virtual through display process in the imaging recording process of FIG. 3. It is a flowchart explaining an example of the detail of the flow of a three-dimensional model generation process among the virtual through display processes of FIG.

  Hereinafter, embodiments of the present invention will be described with reference to the drawings.

FIG. 1 is a block diagram showing a hardware configuration of an imaging apparatus 1 according to an embodiment of the present invention.
The imaging device 1 is configured as a digital camera, for example.

  The imaging device 1 includes a CPU (Central Processing Unit) 11, a ROM (Read Only Memory) 12, a RAM (Random Access Memory) 13, an image processing unit 14, a bus 15, an input / output interface 16, and an imaging unit. 17, an operation unit 18, a sensor unit 19, a display unit 20, a storage unit 21, a communication unit 22, and a drive 23.

  The CPU 11 executes various processes according to a program recorded in the ROM 12 or a program loaded from the storage unit 21 to the RAM 13.

  The RAM 13 appropriately stores data necessary for the CPU 11 to execute various processes.

  The image processing unit 14 is configured by a DSP (Digital Signal Processor), a VRAM (Video Random Access Memory), and the like, and performs various image processing on image data in cooperation with the CPU 11.

  The CPU 11, ROM 12, RAM 13, and image processing unit 14 are connected to each other via a bus 15. An input / output interface 16 is also connected to the bus 15. An imaging unit 17, an operation unit 18, a sensor unit 19, a display unit 20, a storage unit 21, a communication unit 22, and a drive 23 are connected to the input / output interface 16.

  Although not shown, the imaging unit 17 includes an optical lens unit and an image sensor.

The optical lens unit is configured by a lens that collects light, for example, a focus lens or a zoom lens, in order to photograph a subject.
The focus lens is a lens that forms a subject image on the light receiving surface of the image sensor. The zoom lens is a lens that freely changes the focal length within a certain range.
The optical lens unit is also provided with a peripheral circuit for adjusting setting parameters such as focus, exposure, and white balance as necessary.

The image sensor includes a photoelectric conversion element, AFE (Analog Front End), and the like.
The photoelectric conversion element is composed of, for example, a CMOS (Complementary Metal Oxide Semiconductor) type photoelectric conversion element or the like. A subject image is incident on the photoelectric conversion element from the optical lens unit. Therefore, the photoelectric conversion element photoelectrically converts (captures) the subject image, accumulates the image signal for a predetermined time, and sequentially supplies the accumulated image signal as an analog signal to the AFE.
The AFE performs various signal processing such as A / D (Analog / Digital) conversion processing on the analog image signal. Through various signal processing, a digital signal is generated and output as an output signal of the imaging unit 17.
Such an output signal of the imaging unit 17 is hereinafter referred to as “captured image data”. The captured image data is appropriately supplied to the CPU 11 or the like.

  The operation unit 18 includes various buttons such as a shutter button, and inputs various information in accordance with a user instruction operation.

The sensor unit 19 includes, for example, a triaxial geomagnetic sensor, a triaxial acceleration sensor, a tilt sensor, a distance sensor, and the like, and detects the posture and position of the imaging unit 17.
Based on the detection result of the sensor unit 19, the CPU 11 determines the orientation and position of the imaging device 1 relative to the subject, such as the azimuth angle of the imaging device 1, the tilt of the imaging device 1, and the distance between the subject and the imaging device 1. (Hereinafter referred to as “positional information of the imaging device 1”) is calculated.
For example, the triaxial geomagnetic sensor of the sensor unit 19 detects triaxial (X, Y, Z) components in the direction of geomagnetism. The CPU 11 calculates the azimuth angle of the imaging device 1 based on the detection result of the triaxial geomagnetic sensor.
Further, for example, the tilt sensor is constituted by, for example, a piezoelectric vibration gyro, and outputs a voltage value corresponding to the applied angular velocity. The CPU 11 integrates the voltage value output from the tilt sensor to calculate the change amount of the tilt, and calculates the tilt of the imaging device 1 based on the change amount of the tilt.
The CPU 11 corrects the detection result of the triaxial geomagnetic sensor or the tilt sensor using the detection result of the triaxial acceleration sensor capable of measuring the azimuth and the tilt even in an arbitrary posture (tilt angle) state, and the correction. The azimuth angle and inclination of the imaging device 1 may be calculated based on the result.
The distance sensor detects a distance from a non-contact object. The CPU 11 calculates the distance between the subject and the imaging device 1 based on the detection result of the distance sensor.

The display unit 20 includes a liquid crystal display and displays various images such as a captured image.
The storage unit 21 is configured with a hard disk, a DRAM (Dynamic Random Access Memory), or the like, and stores various image data.
The communication unit 22 controls communication with other devices (not shown) via a network including the Internet.

  A removable medium 31 made of a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, or the like is appropriately attached to the drive 23. The program read from the removable medium 31 by the drive 23 is installed in the storage unit 21 as necessary. The removable medium 31 can also store various data such as image data stored in the storage unit 21 in the same manner as the storage unit 21.

The imaging apparatus 1 having such a configuration executes a series of processes (hereinafter referred to as “imaging recording process”) until the subject is imaged and the data of the captured image obtained as a result is recorded in the removable medium 31 or the like. be able to.
In the imaging / recording process of the present embodiment, the CPU 11 recognizes that the operation of pressing the shutter button of the operation unit 18 is an instruction to record the captured image, and the CPU 11 captures the captured image captured immediately after the pressing operation. Data is recorded on the removable medium 31 or the like.

Prior to such an instruction to record a captured image, the CPU 11 displays a through image on the display unit 20 by executing a through imaging process and a through display process.
Specifically, when the imaging recording process is started, the CPU 11 continues the imaging operation by the imaging unit 17. Then, while the imaging operation by the imaging unit 17 is continued, the CPU 11 temporarily stores the captured image data sequentially output from the imaging unit 17 in the memory (the storage unit 21 in the present embodiment). Such a series of control processes is the “through imaging process” referred to herein.
Further, the CPU 11 sequentially reads the data of each captured image temporarily recorded in the memory (the storage unit 21 in the present embodiment) during the through imaging process, and sequentially displays the captured image on the display unit 20. . Such a series of control processes is the “through display process” here, and the captured image displayed on the display unit 20 by the through display process is the “through image” here.

In the present embodiment, the imaging apparatus 1 has a normal display mode and a virtual display mode as operation modes during the through display process (hereinafter referred to as “through display mode”), and these two types of operation modes. Can be switched freely.
The “normal display mode” refers to a mode in which the display unit 20 displays the captured image itself captured at the present time as a through image as usual. Hereinafter, the through display process in the normal display mode is referred to as “normal through display process”.
The “virtual display mode” refers to a mode in which a through image of a virtual image (hereinafter referred to as “virtual through image”) obtained as a result of the following series of processes is displayed on the display unit 20.
That is, the CPU 11 recognizes a subject to be noted as a processing target (hereinafter referred to as “target subject”) among subjects included in the captured image based on data of the captured image captured at the present time. A three-dimensional model of the subject of interest is created or acquired. Based on the three-dimensional model, the CPU 11 estimates and calculates virtual image data obtained when it is assumed that the imaging unit 17 has captured the subject of interest with a posture or position different from the current state. Then, the CPU 11 causes the display unit 20 to display the virtual image as a virtual through image. Hereinafter, such a series of processes, that is, the through display process in the virtual display mode is referred to as “virtual through display process”.

  FIG. 2 is a functional block diagram illustrating a functional configuration that realizes an execution function of an imaging recording process including such a virtual through display process, among the functional configurations of the imaging apparatus 1.

When the imaging recording process is executed, the through display mode selection unit 51, the switching unit 52, the display control unit 53, and the recording control unit 54 function in the CPU 11.
Note that the functions of the through display mode selection unit 51 to the recording control unit 54 will be described together with the description of the processing of related steps in the imaging recording process of FIG. 3 to be described later.
In the present embodiment, each of the through display mode selection unit 51 to the recording control unit 54 is configured as a combination of hardware called the CPU 11 and a program (software) stored in the ROM 12 or the like in the configuration shown in FIG. ing.
However, this is merely an example, and at least a part of the functions of the through display mode selection unit 51 to the recording control unit 54 may be transferred to other components (image processing unit 14 or the like) other than the CPU 11. Is possible.

When the imaging and recording process is executed, in the image processing unit 14, the target subject determination unit 61, the 3D model generation / acquisition unit 62, and the virtual through image generation unit 63 function.
Note that the functions of the target subject determination unit 61 to the virtual through image generation unit 63 will be described together with the description of the processing of related steps in the imaging recording processing of FIG. 3 to be described later. .
In the present embodiment, each of the subject-of-interest determination unit 61 to the virtual through image generation unit 63 is included in the image processing unit 14 alone or the hardware such as the image processing unit 14 and the ROM 12 in the configuration illustrated in FIG. It is configured as a combination with a stored program (software).
However, this is merely an example, and at least a part of the functions of the subject-of-interest determination unit 61 to the virtual through image generation unit 63 may be transferred to other components (CPU 11 or the like) other than the image processing unit 14. Of course it is possible.

In the present embodiment, a captured image storage unit 71 and a three-dimensional model storage unit 72 are provided as one area of the storage unit 21.
Specific examples of the stored contents of the captured image storage unit 71 and the three-dimensional model storage unit 72 will be shown together with the description of the processing of related steps in the imaging recording process of FIG. 3 to be described later. .
Note that the captured image storage unit 71 and the three-dimensional model storage unit 72 are provided as one area of the storage unit 21, but this is only an example, and the captured image storage unit 71 and the three-dimensional model storage unit 72 are stored in the storage unit 21. A storage area other than the unit 21, for example, an area of the removable medium 31 may be provided. Further, the captured image storage unit 71 and the three-dimensional model storage unit 72 are not particularly required to be provided in the imaging device 1, and may be provided in another device connected via the communication unit 22, for example. .

Next, with reference to FIG. 3, an imaging recording process executed by the imaging apparatus 1 having the functional configuration of FIG. 2 will be described.
FIG. 3 is a flowchart for explaining an example of the flow of imaging recording processing executed by the imaging apparatus 1 having the functional configuration of FIG.

For example, when the operation mode of the imaging apparatus 1 is switched to a specific mode such as an imaging mode by a predetermined operation on the operation unit 18 by the user, the imaging recording process is started as a trigger.
The imaging unit 17 continues the imaging operation from the start to the end of the imaging recording process, and the captured image data sequentially output from the imaging unit 17 is stored in the captured image storage unit 71 during that time. Shall be.

In step S1, the through display mode selection unit 51 determines whether or not the virtual through display mode is selected as the through display mode.
That is, the user can select a desired mode of the normal through display mode and the virtual through display mode as the through display mode by performing a predetermined operation on the operation unit 18.

When the normal through display mode is selected by such user operation, it is determined as NO in step S1. In this case, the through display mode selection unit 51 switches the output end of the switching unit 52 to the display control unit 53 side. Then, the captured image data output from the imaging unit 17 and temporarily stored in the captured image storage unit 71 is supplied to the display control unit 53 via the switching unit 52.
In step S <b> 2, the display control unit 53 of the CPU 11 executes normal through display processing, and displays such a captured image on the display unit 20 as a through image as it is.

On the other hand, when the virtual through display mode is selected by the user's operation, it is determined as YES in Step S1. In this case, the through display mode selection unit 51 switches the output end of the switching unit 52 to the target subject determination unit 61 side of the image processing unit 14. Then, the captured image data output from the imaging unit 17 and temporarily stored in the captured image storage unit 71 is supplied to the target subject determination unit 61 of the image processing unit 14 via the switching unit 52.
In step S <b> 3, the image processing unit 14 and the CPU 11 execute a virtual through display process to display a virtual through image on the display unit 20.

In this way, when a normal through image is displayed in the normal through display process in step S2, or when a virtual through image is displayed in the virtual through display process in step S3, the process proceeds to step S4.
In step S4, the recording control unit 54 determines whether or not the shutter button of the operation unit 18 has been pressed.
If the shutter button is not pressed by the user, it is determined as NO in step S4, the process returns to step S1, and the subsequent processes are repeated. That is, until the shutter button is pressed, the loop processing of steps S1 to S4 is repeatedly executed, and a normal through image or virtual through image is displayed on the display unit 20.

For example, when the virtual through display mode is maintained, the loop process of step S1: YES, step S3, and step S4: NO is repeatedly executed, and the virtual through image is displayed as if it were a real-time display. That is, the imaging unit 17 continuously and repeatedly captures images, and sequentially outputs captured image data obtained each time. In this case, the subject-of-interest determination unit 61, the three-dimensional model generation / acquisition unit 62, the position information acquisition unit 55, the virtual through image generation unit 63, and the display control unit 53 sequentially output captured image data from the imaging unit 17. Each time, processing based on each function described later is executed. Thereby, each of the virtual through images corresponding to each of the captured images output as sequential data from the imaging unit 17 is continuously and repeatedly displayed on the display unit 20.
Therefore, the user moves the position and posture of the imaging device 1 while appropriately viewing the virtual through image with the body substantially fixed, and confirms whether the image of the desired composition is displayed as the virtual through image. Can do.
Here, as described above, as a mode for displaying a through image under the control of the display control unit 53, a normal through display mode for displaying a captured image sequentially output as data from the imaging unit 17 as a normal through image, and imaging There is a virtual through display mode in which each virtual through image corresponding to each captured image output as sequential data from the unit 17 is displayed. The through display mode selection unit 51 can select one of the normal through display mode and the virtual through display mode.
Therefore, when an image having a desired composition is obtained as a virtual through image, the user can instruct the normal display through mode as the through display mode by operating the operation unit 18 this time. Thereby, the through display mode selection unit 51 selects the normal through display mode. Then, the loop process of Step S1: NO, Step S2, and Step S4: NO is repeatedly executed, and this time, the normal through image is displayed in real time.
As a result, the user moves the position and posture of the imaging apparatus 1 until a normal through image with a desired composition is displayed, and moves the body as necessary.
In this way, the user moves the position and posture of the imaging apparatus 1 while appropriately viewing the normal through image or virtual through image, or moves the body as appropriate, so that the normal through image of the desired composition is displayed. If it is confirmed that the shutter button is pressed, the shutter button of the operation unit 18 is pressed at that stage.
Then, it determines with it being YES in step S4, and a process progresses to the process of step S5. In step S <b> 5, the recording control unit 54 acquires the captured image data stored in the captured image storage unit 71 at that time, and records it in the removable medium 31.
Thereby, the imaging recording process ends.

  Further, the virtual through display process in step S3 will be described below among such imaging and recording processes.

FIG. 4 is a diagram for explaining the result of the virtual through display process while comparing it with the result of the normal through display process.
FIG. 4A shows a state in which the imaging device 1 has captured the person Ma in an initial state immediately after the start of the imaging recording process of FIG. 3 in a posture facing the front surface of the person Ma.
In the initial state, since it is necessary to determine the reference posture of the imaging apparatus 1, not only when the normal through display process is executed, but also when the virtual through display process is executed, FIG. As illustrated, a captured image obtained as a result of the imaging device 1 imaging a person Ma is displayed as it is as a through image SP1.
FIG. 4B shows that the imaging apparatus 1 rotates to the left as viewed from a user (an operator of the imaging apparatus 1) (not shown) with respect to the reference posture in the initial state, and faces the oblique side surface of the person Ma. The normal through display process (step S2 in FIG. 3) is executed in such a posture as to capture the person Ma.
In this case, as shown in FIG. 4B, the captured image obtained as a result of the imaging device 1 imaging the person Ma is displayed as it is on the display unit 20 as the through image SP2.
FIG. 4C shows a posture in which the imaging device 1 is rotated to the left as viewed from a user (an operator of the imaging device 1) (not shown) with respect to the reference posture in the initial state, that is, FIG. The virtual through display process (step S3 in FIG. 3) is executed with the same posture and the person Ma is imaged.
In this case, the oblique side surface of the person Ma within the range of the field of view of the imaging device 1 is originally shown (see the through image SP2 in FIG. 4B), and the back surface of the person Ma outside the range of the field of view of the imaging device 1 A virtual through image VSP <b> 2 in which (the back of the head) is reflected is displayed on the display unit 20.

As described above, the user (operator of the imaging apparatus 1) operates the operation unit 18 and selects the virtual through display mode, and then the imaging apparatus 1 does not actually move around and move around a long distance. The virtual through display process (step S3 in FIG. 3) is executed by the imaging device 1 by slightly moving the position and orientation of the image. As a result, the user can easily visually recognize a virtual through image equivalent to a captured image obtained when the user moves around and captures an image.
There are also places where it is impossible to go around depending on the position of the subject. Even in such a case, the user can easily visually recognize a virtual through image equivalent to a captured image obtained when capturing an image in such a place.
As a result, for the user, the task of determining the imaging angle of the subject is remarkably facilitated, and the time required for photographing the subject is shortened. Also, the model (subject) is freed from the pain of being forced to take the same pose for a long time or changing the standing position to take the same pose.
In summary, the user holds the imaging device 1 with his / her hand, tilts and turns it, and performs an intuitive operation that slightly changes the position and orientation of the imaging device 1. It can be easily viewed from various viewpoints. As a result, the user does not need to go around the subject or move the subject. For this reason, the user can easily understand the appearance and shape of the subject in an easy-to-understand manner or can examine the details in detail. That is, the user can easily examine and examine ideal imaging conditions such as the imaging direction, imaging angle, composition, and the like without moving greatly from the spot, and can easily image the subject under the desired imaging conditions.
That is, it is possible to visually recognize an image with a desired composition as a virtual through image only by moving only the imaging device 1 while the user is fixed.

Further details of such virtual through display processing will be described below.
FIG. 5 is a flowchart illustrating an example of details of the flow of the virtual through display process in step S3 in the imaging recording process of FIG.

In step S21, the target subject determination unit 61 of the image processing unit 14 in FIG. 2 determines a target subject from the subjects included in the captured image based on the captured image data.
Note that the method of determining the subject of interest is not particularly limited. For example, although not shown, a method of determining a subject selected by the user by operating the operation unit 18 as the subject of interest may be adopted. Alternatively, a method for determining a subject of interest based on a result of a face detection process or a region-of-interest detection process that is publicly known or will appear in the future can be adopted as a method of determining a subject of interest.
When the information related to the subject of interest determined in the process of step S21 and the data of the captured image including the subject of interest are supplied from the subject of interest determination unit 61 to the 3D model generation / acquisition unit 62, the process proceeds to step S22. move on.

In step S <b> 22, the position information acquisition unit 55 of the CPU 11 captures images such as the azimuth angle of the imaging device 1, the tilt of the imaging device 1, the distance between the subject of interest and the imaging device 1 based on the detection result of the sensor unit 19. Information indicating the position and orientation of the apparatus 1 is acquired as position information of the imaging apparatus 1 relative to the subject of interest.
When the position information of the imaging device 1 is supplied from the position information acquisition unit 55 to the three-dimensional model generation acquisition unit 62, the process proceeds to step S23.

In step S23, the 3D model generation / acquisition unit 62 of the image processing unit 14 determines whether or not a 3D model of the subject of interest has been generated.
When a three-dimensional model that is the same as or similar to the subject of interest is already stored in the three-dimensional model storage unit 72, for example, as a result of the three-dimensional model generation process in step S25 executed in the past (the processing content will be described later). If stored, it is determined as YES in step S23, and the process proceeds to step S24.
In step S <b> 24, the 3D model generation / acquisition unit 62 acquires the 3D model of the subject of interest from the 3D model storage unit 72.
On the other hand, if no three-dimensional model identical or similar to the subject of interest is stored in the three-dimensional model storage unit 72, it is determined as NO in Step S23, and the process proceeds to Step S25.
In step S25, the three-dimensional model generation / acquisition unit 62 generates a three-dimensional model of the subject of interest and stores it in the three-dimensional model storage unit 72 (hereinafter, such a series of processes is referred to as “3 (Referred to as “dimensional model generation process”). The details of the three-dimensional model generation process will be described later with reference to the flowchart of FIG.
In this way, the three-dimensional model of the subject of interest acquired in the process of step S24 or generated in the process of step S25 is the position information of the imaging apparatus 1 and the captured image acquired in the process of step S22. When the data is supplied from the three-dimensional model generation / acquisition unit 62 to the virtual through image generation unit 63 together with the data, the process proceeds to step S26.

In step S26, the virtual through image generation unit 63 of the image processing unit 14 generates virtual through image data based on the position information of the imaging device 1, the three-dimensional model of the subject of interest, and the captured image data. .
In step S27, the display control unit 53 of the CPU 11 causes the display unit 20 to display the virtual through image generated as data by the virtual through image generation unit 63 in the process of step S26.

For example, the virtual through image generation unit 63 performs the processing in step S26 from a viewpoint associated with the position information of the imaging device 1 from a viewpoint corresponding to a two-dimensional image (viewed from the viewpoint). In this case, synthesized image data obtained by synthesizing the image of the subject of interest in this case with an image such as a background of the captured image is generated as virtual through image data.
Here, the viewpoint associated with the position information of the imaging apparatus 1 is a viewpoint from a direction different from the imaging direction of the imaging apparatus 1 (the optical axis direction of the imaging unit 17). Specifically, in the example of FIG. 4, a state in which the person Ma is viewed from the viewpoint of the image capturing direction of the image capturing apparatus 1 (the optical axis direction of the image capturing unit 17) (a state in which the oblique side of the person Ma is viewed) is illustrated in FIG. 4B is a normal through image SP2. On the other hand, a state in which the three-dimensional model of the person Ma is viewed from a viewpoint from a direction different from the imaging direction of the imaging device 1 (the optical axis direction of the imaging unit 17) (the back of the three-dimensional model of the person Ma, that is, the back of the head) Is a virtual through image VSP2 shown in FIG. That is, in the process of step S27, the virtual through image VSP2 shown in FIG.
More specifically, if the horizontal angle of the reference posture of the image pickup apparatus 1 shown in FIG. 4A is set to 0 degree, the position of the actual rotation angle θ in the horizontal direction of the image pickup apparatus 1 is adopted as a viewpoint. Then, only an image equivalent to the normal through image SP2 shown in FIG. 4B is displayed. Therefore, in the example of FIG. 4, a rotation angle α (= A × θ) obtained by multiplying the actual rotation angle θ by a predetermined coefficient A (> 1), that is, a rotation angle α larger than the actual rotation angle θ. Is used as the viewpoint, and the state of viewing the three-dimensional model of the person Ma from the viewpoint is displayed on the display unit 20 as the virtual through image VSP2 shown in FIG. It is.
In this way, when the virtual through image VSP2 or the like is displayed on the display unit 20, the virtual through display process ends. That is, the process of step S3 in FIG. 3 ends, and the process proceeds to step S4.

Next, details of the three-dimensional model generation process of step S25 in the virtual through display process of FIG. 5 will be described.
FIG. 6 is a flowchart for explaining an example of the details of the flow of the three-dimensional model generation process of step S25 in the virtual through display process of FIG.

In step S <b> 41, the three-dimensional model generation / acquisition unit 62 acquires data of a plurality of captured images obtained as a result of imaging the subject of interest from different viewpoints from the captured image storage unit 71 via the switching unit 52.
Note that it is sufficient that data of a plurality of captured images exist at the time of processing in step S41, and the imaging timing of the plurality of captured images is not particularly limited. Further, it is not necessary to capture the subject of interest itself (original), and captured image data obtained as a result of capturing a similar object in the past may be acquired. Further, the acquisition destination of the data of the plurality of captured images is not particularly required to be the captured image storage unit 71, and may be other devices that can communicate with the removable medium 31 or the communication unit 22. Good.

  In step S42, the three-dimensional model generation / acquisition unit 62 performs preprocessing (for example, various image processing such as sharpening, noise removal, and inclination correction) on the plurality of image data acquired in step S21. .

In step S43, the three-dimensional model generation / acquisition unit 62 determines whether or not the position information of the imaging device at the time when the captured image data is captured is acquired regarding the data of the plurality of captured images acquired in the process of step S21. .
Here, when a plurality of captured images are captured by the image capturing unit 17, the “position information of the image capturing apparatus 1” acquired by the position information acquiring unit 55 based on the detection result of the sensor unit 19 is step S43. Corresponds to “position information of imaging device”. On the other hand, when a plurality of captured images are captured by another imaging device, information equivalent to “position information of imaging device 1” acquired by the other imaging device is “position information of imaging device”. It corresponds to.
When these “position information of the imaging device” cannot be acquired, it is determined as NO in Step S43, and the process proceeds to Step S47. However, the processing after step S47 will be described later.
On the other hand, when these “position information of the imaging device” can be acquired, it is determined as YES in Step S43, and the process proceeds to Step S44.

  In step S44, the three-dimensional model generation / acquisition unit 62 extracts the contour (edge) of the subject of interest from each of the plurality of captured image data acquired in the process of step S41, and determines the contour that can identify the contour. Data (hereinafter referred to as “contour image data”) is generated.

In step S45, the three-dimensional model generation / acquisition unit 62 recognizes the subject of interest based on the “position information of the imaging device” and the contour image data for each of the plurality of captured images acquired as data in the process of step S41. A three-dimensional model (three-dimensional shape data) is generated.
In step S <b> 46, the three-dimensional model generation / acquisition unit 62 stores the three-dimensional model of the subject of interest generated in this manner in the three-dimensional model storage unit 72.
Note that the data structure when the 3D model of the subject of interest is stored in the 3D model storage unit 72 is not particularly limited. For example, a wire frame model, a surface model, a solid model, CSG (Constructive Solid Geometry) is used. A data structure based on an expression, a boundary expression, or the like can be employed.

  In this way, when “positional information of the imaging device” regarding each of the plurality of captured image data acquired in the process of step S41 can be acquired, it is determined that YES is determined in the process of step S43. Steps S44 to S46 are executed, and a three-dimensional model of the subject of interest is generated and stored in the three-dimensional model storage unit 72. Thereby, the three-dimensional model generation process ends, that is, the process of step S25 of FIG. 5 ends, and the process proceeds to step S26.

On the other hand, if “position information of the imaging device” regarding each of the plurality of captured image data acquired in the process of step S41 cannot be acquired, it is determined as NO in the process of step S43. The process proceeds to step S47.
In step S47, the three-dimensional model generation / acquisition unit 62 determines whether to apply the factorization method.

If it is determined that the factorization method is to be applied, YES is determined in step S47, and the process proceeds to step S48.
In step S48, the three-dimensional model generation / acquisition part 62 generates a three-dimensional model of the subject of interest by a factorization method.
In the factorization method, when the depth of the subject is sufficiently smaller than the distance to the camera (here, the imaging device 1), the formula for obtaining the three-dimensional shape (three-dimensional model) of the subject is a factorization of the matrix. This is a technique that uses the fact that it becomes a linear simultaneous equation that can be solved by The factorization method is preferably applied when obtaining a three-dimensional shape (three-dimensional model) from data of a plurality of captured images taken at a sufficient distance from the subject.
Specifically, the following process is executed as the process of step S48.
The three-dimensional model generation / acquisition unit 62 represents the contour outline of the subject of interest and the characteristic part of the face from the data of a plurality of captured images (captured images of the subject of interest from different viewpoints) acquired in the process of step S41. For example, elements such as line segments, curves, and feature points are extracted.
The three-dimensional model generation / acquisition unit 62 extracts the point feature of the principal point from the data of the plurality of captured images acquired in the process of step S41 and associates it with each feature point.
The three-dimensional model generation / acquisition unit 62 generates a three-dimensional model of the main subject by restoring the three-dimensional shape information of the main subject based on such association.
Refer to Non-Patent Document 1 for further details of the factorization method.

If it is determined that such a factorization method is not applied, it is determined as NO in step S47, and the process proceeds to step S49.
In step S49, the three-dimensional model generation / acquisition unit 62 generates a three-dimensional model of the subject of interest by the visual volume intersection method.
Specifically, the following process is executed as the process of step S49.
The three-dimensional model generation / acquisition unit 62 prepares a three-dimensional voxel space for storing the shape and divides it into a cubic lattice.
The three-dimensional model generation / acquisition unit 62 acquires the silhouette image data of each multi-viewpoint image to be processed from the data of a plurality of captured images (captured images of each subject of interest from different viewpoints) acquired in step S41. Is generated.
The three-dimensional model generation / acquisition unit 62 performs back projection by orthographic projection on each voxel divided into cubic lattices using the silhouette image data of each multi-viewpoint image to be processed.
The three-dimensional model generation / acquisition unit 62 determines whether or not the silhouette of the processing target image is inherent for each voxel, leaves the inherent voxel, and deletes the other voxels.
The three-dimensional model generation acquisition unit 62 generates a three-dimensional shape model of the main subject based on the remaining set of voxels.

  In this way, when the three-dimensional model of the subject of interest is generated by the factorization method in the process of step S48 or the visual object intersection method in the process of step S49, the process in the next step S46 A three-dimensional model of the subject of interest is stored in the three-dimensional model storage unit 72. Thereby, the three-dimensional model generation process ends, that is, the process of step S25 of FIG. 5 ends, and the process proceeds to step S26.

  Note that the method of generating the three-dimensional model of the subject of interest is not particularly limited to the method illustrated in FIG. 6. For example, a binocular camera, a stereo camera, or the like is employed as the imaging device 1. For example, a method using captured image data may be employed.

As described above, the imaging apparatus 1 according to the present embodiment includes the imaging unit 17, the target subject determination unit 61, the 3D model storage unit 72, the 3D model generation acquisition unit 62, and the virtual through image generation unit 63. And comprising.
The imaging unit 17 images one or more subjects from a predetermined imaging direction, and outputs captured image data obtained as a result.
The target subject determination unit 61 determines a target subject to be noted as a processing target from one or more subjects based on the captured image data output from the imaging unit 17.
The three-dimensional model generation / acquisition unit 62 generates or acquires a three-dimensional model of the target subject determined by the target subject determination unit 61.
The position information acquisition unit 55 acquires position information of the imaging device 1 that indicates the position and orientation of the imaging device 1, more precisely position information that indicates the position and orientation of the imaging unit 17.
The virtual through image generation unit 63 is a viewpoint identified according to the position information acquired by the position information acquisition unit 55, and is generated by the three-dimensional model generation acquisition unit 62 from a viewpoint different from a predetermined imaging direction. Data of an image showing a view of the generated or acquired three-dimensional model of the subject of interest is generated as virtual through image data.
The display control unit 53 controls the display of the virtual through image generated as data by the virtual through image generation unit 63.
As a result, the user can hold the imaging device 1 with his hand, tilt and rotate, and perform an intuitive operation that slightly changes the position and posture of the imaging device 1. Can be easily viewed from various viewpoints. As a result, the user does not need to go around the subject of interest or move the subject of interest. For this reason, the user can easily understand the appearance and shape of the subject of interest in an easy-to-understand manner or can examine it in detail. That is, the user can easily examine and examine ideal imaging conditions such as the imaging direction, imaging angle, composition, and the like without moving greatly from the spot, and can easily image the subject under the desired imaging conditions.
That is, it is possible to visually recognize an image with a desired composition as a virtual through image only by moving only the imaging device 1 while the user is fixed.

  The embodiment of the present invention has been described above. However, the present invention is not limited to the above-described embodiment, and modifications, improvements, and the like within a scope that can achieve the object of the present invention are included in the present invention.

For example, as the display unit 20, a flat planar display panel such as a liquid crystal or an organic EL (Electro-Luminescence) is employed in the above-described embodiment.
In this case, if the amount (size) of the attitude angle or tilt angle of the imaging device 1 is used as it is for the amount of change in the viewing direction (for example, the direction of the observation angle of the object image), the angle between the display surface and the viewing direction. Becomes smaller, making it difficult to see the image of the main subject displayed in the virtual through image, and limiting the range of angles that can be displayed. For example, there is a problem that even if the side surface of the main subject is seen, the back side (back side) cannot be seen.
Therefore, in the above-described embodiment, the following technique is adopted as a technique for solving the problem. That is, the line-of-sight direction (virtual rotation angle) α of the image of the main subject is made larger than the rotation angle θ (attitude angle θ) of the imaging device 1, for example, α = A × θ (A is greater than 1). A method of increasing the value to a large integer value, for example, about 1.5 to 3) is employed in the present embodiment. As a result, a virtual through image including an image of the main subject in a wider range of the line-of-sight direction can be seen according to a small change in posture of the imaging device 1.
However, the technique for solving the above-described problem is not particularly limited to the technique of the present embodiment (hereinafter referred to as “first technique”), and various various techniques such as the following second to seventh techniques. Techniques can be employed.

  The second method is a method in which the display surface of the display unit 20 is spherical, hemispherical, or convex so that the range of angles that can be easily displayed is widened.

The third method is the line-of-sight direction of the image of the main subject as compared to the change in the attitude angle Δθ (change in the rotation angle Δθ) of the imaging device 1, such as a ship rudder or a steering wheel operation of the vehicle. The change Δα (virtual rotation angle Δα) is increased, for example, Δα = C × Δθ (C is an integer value greater than 1).
By adopting the third method, when the user (operator of the imaging apparatus 1) changes the line-of-sight direction, the attitude of the imaging apparatus 1 is tilted. By returning to the original position, it is possible to maintain the viewing direction and angle. As a result, a situation in which the angle between the display surface of the display unit 20 and the line-of-sight direction becomes small or the range of angles that can be displayed is limited hardly occurs.

The fourth method is a so-called “differential handle”, that is, a steering wheel capable of making the actual steering angle of the vehicle tire proportional to the derivative of the steering angle θ of the vehicle. The change Δα (virtual rotation angle Δα) in the line-of-sight direction of the image of the main subject is changed in accordance with the amount of change (angular velocity) dθ / dt.
Specifically, a method of calculating Δα = C × dθ / dt is a fourth method.
By applying such a fourth method, it is possible to achieve the same effect as the “differential handle”, that is, if it is turned fast, a large actual rudder angle is obtained, and if it is slowly turned, a small actual rudder angle is obtained. become.
Specifically, in the “differential handle”, in order to maintain a constant actual steering angle θ, it is necessary to continue to rotate at a constant angular velocity, but when the desired direction is reached, the rotation is stopped (the angular velocity is 0). Since the actual rudder angle θ becomes 0 and the vehicle can return to the straight traveling direction, there is no need to return the steering wheel. Similarly, after the imaging apparatus 1 is rotated at a constant angular velocity and then turned in a desired direction, it is not necessary to return the orientation of the imaging apparatus 1 (equivalent to a handle) to its original position if the rotation is stopped.

The fifth method is a combination of the third method and the fourth method, that is, control is performed by combining a normal handle (third method) and a “differential handle” (fourth method). It is a technique.
Specifically, a method of calculating α = C1 × θ + C2 × dθ / dt is the fifth method.

The sixth method is to change the line-of-sight direction by changing the direction and size of the imaging device 1 while the user touches the touch panel or the like on the display surface of the display unit 20 with the finger. This is a method of maintaining the line-of-sight direction at the time when the button is moved away from the touch panel.
Thereby, the operation of returning the posture of the imaging device 1 to the original state can be omitted.

  The seventh method is displayed according to a stroke operation (sweep) with the finger on the touch panel of the display unit 20 or a dial rotation operation, regardless of the position and orientation of the main body of the imaging apparatus 1. This is a technique of displaying a virtual through image obtained by rotating a main subject in a desired direction or sequentially switching and displaying virtual through images viewed from several viewpoint directions.

  Note that when there is no particular need to solve the above-described problem, the above-described first to seventh methods may not be employed.

In the above-described embodiment, the imaging apparatus 1 to which the present invention is applied has been described using a digital camera as an example, but is not particularly limited thereto.
For example, the present invention can be applied to general electronic devices having an imaging function. Specifically, for example, the present invention can be applied to a notebook personal computer, a video camera, a portable navigation device, a mobile phone, a portable game machine, and the like.

The series of processes described above can be executed by hardware or can be executed by software.
In other words, the functional configuration of FIG. 2 is merely an example and is not particularly limited. That is, it is sufficient that the imaging apparatus 1 has a function capable of executing the above-described series of processing as a whole, and what functional blocks are used to realize this function is not particularly limited to the example of FIG.
In addition, one functional block may be constituted by hardware alone, software alone, or a combination thereof.

When a series of processing is executed by software, a program constituting the software is installed on a computer or the like from a network or a recording medium.
The computer may be a computer incorporated in dedicated hardware. The computer may be a computer capable of executing various functions by installing various programs, for example, a general-purpose personal computer.

  The recording medium including such a program is not only constituted by the removable medium 31 of FIG. 1 distributed separately from the apparatus main body in order to provide the program to the user, but also in a state of being incorporated in the apparatus main body in advance. It is comprised with the recording medium etc. which are provided in. The removable medium 31 is composed of, for example, a magnetic disk (including a floppy disk), an optical disk, a magneto-optical disk, or the like. The optical disk is composed of, for example, a CD-ROM (Compact Disk-Read Only Memory), a DVD (Digital Versatile Disk), or the like. The magneto-optical disk is configured by an MD (Mini-Disk) or the like. In addition, the recording medium provided to the user in a state of being preinstalled in the apparatus main body includes, for example, the ROM 12 in FIG. 1 in which the program is recorded, the hard disk included in the storage unit 21 in FIG.

In the present specification, the step of describing the program recorded on the recording medium is not limited to the processing performed in time series along the order, but is not necessarily performed in time series, either in parallel or individually. The process to be executed is also included.
Further, in the present specification, the term “system” means an overall apparatus configured by a plurality of devices, a plurality of means, and the like.

  As mentioned above, although several embodiment of this invention was described, these embodiment is only an illustration and does not limit the technical scope of this invention. The present invention can take other various embodiments, and various modifications such as omission and replacement can be made without departing from the gist of the present invention. These embodiments and modifications thereof are included in the scope and gist of the invention described in this specification and the like, and are included in the invention described in the claims and the equivalent scope thereof.

The invention described in the scope of claims at the beginning of the filing of the present application will be appended.
[Appendix 1]
Imaging means for imaging one or more subjects from a predetermined imaging direction and outputting data of a captured image obtained as a result;
Subject-of-interest determining means for determining a subject of interest to be noted as a processing target from among the one or more subjects based on the data of the captured image output from the imaging means;
3D model generation and acquisition means for generating or acquiring the 3D model of the target object determined by the target object determination means;
Position information acquisition means for acquiring position information indicating the position and orientation of the imaging means;
The viewpoint identified according to the position information acquired by the position information acquisition means, and generated or acquired by the three-dimensional model generation acquisition means from a viewpoint in a direction different from the predetermined imaging direction. Virtual image generation means for generating, as virtual image data, image data indicating a view of the three-dimensional model of the subject of interest;
Display control means for controlling display of the virtual image generated as data by the virtual image generation means;
An imaging apparatus comprising:
[Appendix 2]
The imaging unit continuously and repeatedly captures images, sequentially outputs captured image data obtained each time,
The target subject determination unit, the three-dimensional model generation / acquisition unit, the position information acquisition unit, the virtual image generation unit, and the display control unit are processed each time the captured image data is sequentially output from the imaging unit. By executing the above, each of the virtual images corresponding to each of the captured images sequentially output as data from the imaging unit is displayed continuously and repeatedly.
The imaging apparatus according to appendix 1.
[Appendix 3]
As a mode for displaying an image under the control of the display control unit, a normal through display mode for displaying the captured image sequentially output as data from the imaging unit as a normal through image, and a sequential data output from the imaging unit. The imaging apparatus according to appendix 2, further comprising mode selection means for selecting one of a virtual through display mode in which each of the virtual images corresponding to each of the captured images is displayed as a virtual through image.
[Appendix 4]
In an imaging method executed by an imaging apparatus including an imaging unit that images one or more subjects from a predetermined imaging direction and outputs data of a captured image obtained as a result,
An object-of-interest determination step for determining an object of interest to be noted as a processing object from among the one or more objects, based on the data of the captured image output from the imaging unit;
A three-dimensional model generation acquisition step of generating or acquiring a three-dimensional model of the target subject determined by the processing of the target subject determination step;
A position information acquisition step of acquiring position information indicating the position and orientation of the imaging means;
Generated by the process of the 3D model generation acquisition step from the viewpoint specified according to the position information acquired by the process of the position information acquisition step and in a direction different from the predetermined imaging direction. A virtual image generation step of generating, as virtual image data, data of an image showing a view of the acquired three-dimensional model of the subject of interest;
A display control step for controlling display of the virtual image generated as data by the processing of the virtual image generation step;
An imaging method including:
[Appendix 5]
A computer that controls an imaging apparatus including an imaging unit that captures one or more subjects from a predetermined imaging direction and outputs data of a captured image obtained as a result;
A target subject determination function for determining a target subject to be noted as a processing target from among the one or more subjects based on the data of the captured image output from the imaging unit;
A three-dimensional model generation / acquisition function for generating or acquiring a three-dimensional model of the target object determined by the target object determination function;
A position information acquisition function for acquiring position information indicating the position and orientation of the imaging means;
The viewpoint identified according to the position information acquired by the position information acquisition function, and generated or acquired by the three-dimensional model generation acquisition function from a viewpoint in a direction different from the predetermined imaging direction. A virtual image generation function for generating, as virtual image data, image data showing a view of a three-dimensional model of the subject of interest;
A display control function for controlling display of the virtual image generated as data by the virtual image generation function;
A program to realize

  DESCRIPTION OF SYMBOLS 11 ... CPU, 12 ... ROM, 13 ... RAM, 14 ... Image processing part, 17 ... Imaging part, 18 ... Operation part, 19 ... Sensor part, ... Display unit, 21 ... storage unit, 22 ... communication unit, 23 ... drive, 31 ... removable media, 51 ... through display mode selection unit, 52 ... switching unit, 53 ..Display control unit 54... Recording control unit 55 .. position information acquisition unit 61... Subject of interest determination unit 62... 3D model generation and acquisition unit 63. Generation unit, 71 ... captured image storage unit, 72 ... 3D model storage unit

Claims (6)

Imaging means for imaging one or more subjects from a predetermined imaging direction and outputting data of a captured image obtained as a result;
Subject-of-interest determining means for determining a subject of interest to be noted as a processing target from among the one or more subjects based on the data of the captured image output from the imaging means;
3D model generation and acquisition means for generating or acquiring the 3D model of the target object determined by the target object determination means;
Position information acquisition means for acquiring position information indicating the position and orientation of the imaging means;
The viewpoint identified according to the position information acquired by the position information acquisition means, and generated or acquired by the three-dimensional model generation acquisition means from a viewpoint in a direction different from the predetermined imaging direction. Virtual image generation means for generating, as virtual image data, image data indicating a view of the three-dimensional model of the subject of interest;
Display control means for controlling display of the virtual image generated as data by the virtual image generation means;
An imaging apparatus comprising: The imaging unit continuously and repeatedly captures images, sequentially outputs captured image data obtained each time,
The target subject determination unit, the three-dimensional model generation / acquisition unit, the position information acquisition unit, the virtual image generation unit, and the display control unit are processed each time the captured image data is sequentially output from the imaging unit. By executing the above, each of the virtual images corresponding to each of the captured images sequentially output as data from the imaging unit is displayed continuously and repeatedly.
The imaging device according to claim 1. As a mode for displaying an image under the control of the display control unit, a normal through display mode for displaying the captured image sequentially output as data from the imaging unit as a normal through image, and a sequential data output from the imaging unit. The imaging apparatus according to claim 2, further comprising mode selection means for selecting one of a virtual through display mode in which each of the virtual images corresponding to each of the captured images is displayed as a virtual through image. The viewpoint in a direction different from the predetermined imaging direction is a viewpoint in which a change in the line-of-sight direction α in the image of the subject of interest is larger than a rotation angle θ of the imaging apparatus. 4. The imaging device according to any one of 3. In an imaging method executed by an imaging apparatus including an imaging unit that images one or more subjects from a predetermined imaging direction and outputs data of a captured image obtained as a result,
An object-of-interest determination step for determining an object of interest to be noted as a processing object from among the one or more objects, based on the data of the captured image output from the imaging unit;
A three-dimensional model generation acquisition step of generating or acquiring a three-dimensional model of the target subject determined by the processing of the target subject determination step;
A position information acquisition step of acquiring position information indicating the position and orientation of the imaging means;
Generated by the process of the 3D model generation acquisition step from the viewpoint specified according to the position information acquired by the process of the position information acquisition step and in a direction different from the predetermined imaging direction. A virtual image generation step of generating, as virtual image data, data of an image showing a view of the acquired three-dimensional model of the subject of interest;
A display control step for controlling display of the virtual image generated as data by the processing of the virtual image generation step;
An imaging method including: A computer that controls an imaging apparatus including an imaging unit that captures one or more subjects from a predetermined imaging direction and outputs data of a captured image obtained as a result;
A target subject determination function for determining a target subject to be noted as a processing target from among the one or more subjects based on the data of the captured image output from the imaging unit;
A three-dimensional model generation / acquisition function for generating or acquiring a three-dimensional model of the target object determined by the target object determination function;
A position information acquisition function for acquiring position information indicating the position and orientation of the imaging means;
The viewpoint identified according to the position information acquired by the position information acquisition function, and generated or acquired by the three-dimensional model generation acquisition function from a viewpoint in a direction different from the predetermined imaging direction. A virtual image generation function for generating, as virtual image data, image data showing a view of a three-dimensional model of the subject of interest;
A display control function for controlling display of the virtual image generated as data by the virtual image generation function;
A program to realize

Images