JPH10149455A - Picture generation/display device and generated/ displayed picture compiling device - Google Patents

Abstract

PROBLEM TO BE SOLVED: To easily generate/display a moving picture whose movement is smooth by deforming a sample picture corresponding to plural sample positions adjacent to a virtual camera positions so that feature points are matched and synthesizing them by means of weighting average with a distance between the position of the virtual camera and the sample positions as weight. SOLUTION: A sample picture generation part 3 generates the picture obtained by projecting a display object viewed from the sample positions on a plane. A feature point candidate picture generation part 4 generates a feature point candidate picture constituted of the picture obtained by projecting a point on the surface of the object on the plane on which a two-dimensional picture is projected. An adjacent sample position calculation part 5 calculates the sample position adjacent to a camera position. A sample picture deformation part 7 deforms the sample picture obtained by photographing it from the sample position so that the feature points are matched. A sample picture synthesis part 8 executes synthesis by weighting average where the distance between the position of the virtual camera and the sample position is set to be weight. A synthesized picture display part 9 displays the synthesized picture on a screen.

Description

DETAILED DESCRIPTION OF THE INVENTION

[0001]

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to an image generating and displaying apparatus for generating an image by replacing a three-dimensional image with a two-dimensional image and displaying the image, and a generating and displaying image editing apparatus for performing editing for displaying the image. In particular, in the fields of computer-aided design (CAD), scientific and technological simulation, and video game machines, an image generation / display apparatus for displaying a screen by applying a deformation process to a computer graphics or a real video image edited. Also, the present invention relates to a generated display image editing apparatus for generating and editing information for performing image generation and display in this apparatus in advance.

[0002]

2. Description of the Related Art In recent years, image generation and display devices have been used in various designs, simulations, game machines, and the like as tools for communicating between computers and humans using realistic screen displays.

[0003] An example of the above-described conventional image generating and displaying apparatus will be described below with reference to the drawings.

FIG. 20 shows a configuration of a conventional image generating and displaying apparatus.

[0005] In Fig. 20, reference numeral 991 denotes an internal surface creating unit;
Reference numeral 992 denotes an inclusion plane projection unit, 993 denotes a projection image storage unit, 99
4 is a camera designation unit, 995 is a texture mapping unit,
Reference numeral 996 denotes a generated image display unit.

Next, the operation will be described.

[0007] First, the included surface creation unit 991 creates a curved surface that wraps around the display target object.

As such a curved surface, for example, there is a spherical surface or a cylindrical surface which includes or circumscribes an object, but such a spherical surface or a cylindrical surface is actually formed by a polygon.

Next, a pattern of the surface of the object is projected on one of the surfaces constituting the above-mentioned curved surface in the inclusion surface projection unit 992.

In this projection, for example, an intersection point between a half line passing through an arbitrary point on the surface of the display target object and a curved line, starting from a point inside the curved surface, is calculated. There is a method of projecting the brightness and color tone of the image.

Then, in the projection image storage unit 993, the pattern projected on the curved surface is developed and stored as a two-dimensional image.

On the other hand, if the object to be displayed is supposed to be seen by a camera, the camera designation unit 994 designates the position of the camera and the direction of the line of sight.

That is, the camera designating section 994 designates from which position to which plane the object to be displayed is to be displayed when displayed on the screen.

Next, in the texture mapping unit 995, the projection plane viewed from the camera is attached to a plane.
Image generation is performed using a so-called texture mapping method.

Finally, in the generated image display unit 996,
The generated image is displayed on a screen as an image of the display target object viewed from the camera.

[0016]

However, in the above configuration, an image corresponding to a change in the line of sight of the display target object, that is, a change in the position or direction of the camera, is generated and displayed. When trying to change the line of sight continuously and slowly, if the number of sample images sampled at spatial imaging intervals is small,
The motion is unnatural, such as dropped frames, and there is a problem that it is very difficult to display a moving image with smooth motion.

FIG. 21 shows an example of a method for photographing this sample image, and shows a case where a subject is photographed from a spherical surface. As shown in FIG. 21, when photographing from above a spherical surface covering a display target object (subject) a, photographing positions arranged at certain intervals, that is, longitude c and latitude formed at regular intervals on a spherical surface covering the subject. The image is shot from the intersection b of d, and when the number of sample images, which are the images shot from the intersection b of the longitude c and the latitude d, is small, a moving image with smooth motion is displayed as described above. It becomes very difficult.

The present invention has been made in view of the above-mentioned problems, and when generating an image corresponding to a case where the position and direction of a camera is continuously and gradually changed, a realistic moving image with a smooth motion can be easily generated. It is an object of the present invention to provide an image generation and display device capable of displaying, and a generation and display image editing device for editing information of a sample image and feature points required for the image generation and display in advance.

[0019]

In order to solve the above-mentioned problems, an image generating and displaying apparatus according to a first aspect of the present invention comprises a latitude line and a longitude line formed at predetermined intervals on a virtual spherical surface including a display object. A sample position storage unit that saves the position of the intersection as a sample position, a sample image creation unit that creates a sample image that is an image obtained by projecting an image to be displayed on the plane from the sample position, A feature point candidate image producing unit for producing a feature point candidate image composed of an image obtained by projecting a point including a feature point on the plane, and a feature from points on the surface of the display target viewed from a plurality of sample positions adjacent to each other. A feature point designating unit that designates a point on the sample image and the feature point candidate image; and a position of a virtual camera that captures an arbitrary point on the virtual sphere (hereinafter, referred to as a “camera”). A camera position input unit designated as a virtual camera position), an adjacent sample position calculation unit for calculating a plurality of sample positions adjacent to the virtual camera position, and a sample image corresponding to the plurality of adjacent sample positions. And a sample image deforming unit that deforms so that the characteristic points of the sample images coincide with each other, and the deformed sample image are combined by a weighted average in which the distance between the virtual camera position and the sample position where the sample image is located is weighted. A sample image combining unit that obtains a combined image as a display target image viewed from the virtual camera position, and displays the combined image on the screen as an image obtained by projecting a display target viewed from the virtual camera position onto a two-dimensional plane. And a combined image display unit.

According to a second aspect of the present invention, there is provided the image generating and displaying apparatus according to the first aspect, wherein the sample image producing unit converts the two-dimensional image into a two-dimensional image by photographing the display object from the sample position. And a sample image storage unit for storing the two-dimensional image as a sample image.

According to a third aspect of the present invention, there is provided the image generating and displaying apparatus according to the first aspect, wherein the sample image producing unit is configured to store the object model including the shape and attribute data of the video to be displayed. A model storage unit, a sample image generation unit configured to generate an image of the object model using the sample position as a virtual camera, and a sample image storage unit configured to store an image generated by the sample image generation unit as a sample image. Things.

According to a fourth aspect of the present invention, there is provided the image generation and display apparatus according to the first aspect, wherein the feature point candidate image producing unit is configured to generate a feature point candidate on the surface of the display target object from the sample position. A feature point candidate photographing unit for photographing a display target object to which a mark is added and converting the object into a two-dimensional image, and a feature point image storage unit for saving the two-dimensional image as a feature point candidate image It is.

According to a fifth aspect of the present invention, there is provided the image generation and display apparatus according to the first aspect, wherein the feature point candidate image producing unit is configured to determine an arbitrary point on the surface of the display target object as a feature point candidate. A feature point candidate position storage unit that stores the three-dimensional position as a model, a feature point candidate generation unit that generates an image of the feature point candidate model using the sample position as the virtual camera position, and a feature point candidate generation unit. And a feature point image storage unit for storing the image as the feature point candidate image.

According to a sixth aspect of the present invention, there is provided the image generation and display apparatus according to the first aspect, wherein the feature point designating unit alpha-combines the sample image and the feature point candidate image at an arbitrary ratio. An image synthesis display unit to be displayed, a feature point selection unit that interactively selects a feature point from feature point candidates of the synthesized image, and a feature point storage unit that stores a two-dimensional position of the feature point It was done.

According to a seventh aspect of the present invention, there is provided a generated and displayed image editing apparatus for storing a position of an intersection of a latitude and a longitude formed at predetermined intervals on a virtual spherical surface including a display object as a sample position. A position storage unit, a sample image production unit for producing a sample image which is an image obtained by projecting the display target image onto the plane from the sample position, and projecting points on the surface of the display target, including the feature points, onto the plane Feature point candidate image producing section for producing a feature point candidate image composed of the extracted image, and a feature point on the sample image and the feature point candidate image from the points on the surface of the display target viewed from a plurality of sample positions adjacent to each other. And a feature point designating unit that designates the feature point.

According to an eighth aspect of the present invention, there is provided the generated and displayed image editing apparatus according to the seventh aspect, wherein the two-dimensional image of the object to be displayed is photographed from the sample position by the camera. And a sample image storage unit for storing the two-dimensional image as a sample image.

According to a ninth aspect of the present invention, in the generation and display image editing apparatus of the seventh aspect, the sample image production unit stores the object model including the shape and attribute data of the display target video. And a sample image storage unit for generating an image of the object model using the sample position as a virtual camera, and a sample image storage unit for storing an image generated by the sample image generation unit as a sample image. It was made.

According to a tenth aspect of the present invention, there is provided the generation display image editing apparatus according to the seventh aspect, wherein the feature point candidate image producing unit is provided with a feature point candidate image creation unit which is provided on the surface of the display target object from the sample position. A feature point candidate photographing unit for photographing a display target object to which a mark as a candidate is added with a camera and converting it into a two-dimensional image, and a feature point image storage unit for saving the two-dimensional image as a feature point candidate image. It was done.

According to an eleventh aspect of the present invention, in the generation and display image editing apparatus of the seventh aspect, the feature point candidate image producing unit is characterized in that an arbitrary point on the surface of the display target object is characterized by A feature point candidate position storage unit that stores the three-dimensional position as a point candidate model; a feature point candidate generation unit that generates an image of the feature point candidate model using a sample position as the virtual camera position; and a feature point candidate generation unit. And a feature point image storage unit for storing the generated image as the feature point candidate image.

According to a twelfth aspect of the present invention, there is provided the generated and displayed image editing apparatus according to the seventh aspect, wherein the feature point designating unit performs alpha synthesis on the sample image and the feature point candidate image at an arbitrary ratio. An image synthesizing display unit for displaying a screen, a feature point selecting unit for interactively selecting a feature point from the feature point candidates of the synthesized image, and a feature point storing unit for storing a two-dimensional position of the feature point. It is like that.

Further, according to a thirteenth aspect of the present invention, there is provided the image generation / display apparatus, wherein an arbitrary point on a virtual spherical surface including the display object is positioned at a position of a virtual camera which captures the display object (hereinafter, a virtual camera). And calculating a plurality of sample positions adjacent to the virtual camera position from among the sample positions obtained by the camera position input unit designated as (position) and the generated display image editing apparatus according to any one of claims 7 to 12. An adjacent sample position calculating unit;
2. a sample image deforming unit that deforms the sample images corresponding to the plurality of adjacent sample positions obtained by any one of the generated display image editing apparatuses so that their characteristic points match each other; A sample image obtained as a display target image viewed from the virtual camera position by synthesizing the weighted average using the distance between the virtual camera position and the sample position where the sample image is positioned as a weight. A combining unit and a display object that can display the combined image from the virtual camera position by two
And a composite image display unit that displays a screen as an image projected on a two-dimensional plane.

[0032]

BEST MODE FOR CARRYING OUT THE INVENTION

Embodiment 1 FIG. Hereinafter, the image generation and display device according to the first embodiment of the present invention will be described with reference to the drawings.

FIG. 1 is a configuration diagram of an image generating and displaying apparatus according to Embodiment 1 of the present invention.

In FIG. 1, reference numeral 1 denotes a camera position input unit;
Is a sample position storage unit, 3 is a sample image production unit, 4 is a feature point candidate image production unit, 5 is an adjacent sample position calculation unit,
Reference numeral 6 denotes a feature point designation unit, 7 denotes a sample image transformation unit, 8 denotes a sample image synthesis unit, and 9 denotes a synthesized image display unit.

The operation of the image generating and displaying apparatus configured as described above will be described below with reference to FIGS.

First, in the sample position storage unit 2 of FIG. 1, latitude lines and meridians formed at regular intervals on a virtual spherical surface containing a real object or a virtual object for displaying computer graphics (CG). Is stored as the sample position.

For example, when a still image (sample image) is created by actual shooting, a dome e as shown in FIG. 14 is actually created, and its shooting position in a three-dimensional space is stored.

When a still image (sample image) is created by computer graphics, an object, a space,
Since the cameras are all virtual, a dome as described above is virtually created inside the computer, and the shooting position is stored based on the three-dimensional position in the virtual space.

In computer graphics,
The subject is not yet an image-generated one. To put it simply, the whole subject g is represented by one coordinate system (coordinate system for creating the subject) h, and the coordinate system h defines the dome as described above. Only the position in the three-dimensional coordinate system f is located as information.

The coordinate value of each vertex of the subject (virtual object) g is indicated by a coordinate system f, and the value is shown in FIG.
Is stored in the object model storage unit 31 in the sample image production unit 3.

FIG. 7 is a conceptual diagram of a viewpoint position of the image generating and displaying apparatus according to the first embodiment of the present invention. In FIG. 7, reference numeral 71 denotes a display target object (subject);
Reference numeral 3 denotes a spherical surface (hereinafter, referred to as a virtual sphere) which is a virtual curved surface including the display target object, and reference numeral 74 denotes a center of the virtual sphere 73.

As an example of a sample position for photographing the display target object, the center of gravity of the display target object 71, that is, the center of gravity of a box-shaped space (boundary box) separated from the surrounding space by its boundary line is taken as the center. Virtual sphere 7
There are points A to D on the surface of No. 3. Points A to D are points A and B
Are located at the same latitude, points C and D are located at the same latitude, different from points A and B, points A and C are located at the same longitude, and points B and D are located at point A. And C located at the same longitude.

Next, the sample image producing section 3 of FIG. 1 produces a sample image, that is, an image in which the display target object viewed from the sample position is projected onto a plane. here,
The line-of-sight vector starts from the above sample position, and the virtual sphere 7
The center 74 of 3 is the end point.

FIG. 8 is a conceptual diagram of a sample image of the image generating and displaying device according to the first embodiment of the present invention. In FIG. 8, reference numerals 81 to 84 denote sample images.

The sample images 81 to 84 are examples of sample images obtained by capturing images to be displayed from the sample positions A to D in FIG.

FIG. 2 shows the sample image producing section 3 of the image generating and displaying apparatus according to the first embodiment of the present invention shown in FIG.
FIG. 1 is a diagram showing a first configuration example of the present invention, and this configuration example shows a case where a sample image is produced when an object to be displayed is a real object, that is, when the display object is a real video. In FIG.
Reference numeral 21 denotes an object photographing unit, and reference numeral 22 denotes a sample image storage unit.

In the sample image producing section 3, first, in the object photographing section 21, an object to be displayed is photographed from each sample position by a camera and is converted into a two-dimensional image.

Next, in the sample image storage unit 22,
The two-dimensional image is stored as a sample image. As a result, a still image photographed from multiple viewpoints, which is necessary for generating a moving image using the still image, can be obtained as a sample image.

FIG. 3 is a diagram showing a second configuration example of the sample image production section of the image generation and display apparatus according to the first embodiment of the present invention shown in FIG. In the case of a graphics (CG) video, a case where a sample image is produced is shown. In FIG. 3, 31 is an object model storage unit, 32 is a sample image generation unit, and 33 is a sample image storage unit.

In the sample image producing section 3, first, the object model storing section 31 stores an object model including the shape and attribute data of the display target object.

This object model is not a computer graphics image but a virtual object, and vertex coordinate values constituting the object in a coordinate system for defining the object, which vertices are connected to form a plane (polygon) It consists of shape data consisting of information on whether or not it is done, and attribute data such as the type of texture to be attached to the plane, coordinate values of the texture itself, and transparency.

FIG. 15 shows the correspondence between one polygon i existing in the object model and the coordinate system j of the texture.

Next, the sample image generation unit 32 of FIG. 3 generates an image of an object model when the object model is viewed by using the sample positions as virtual camera positions.

The order of the image generation processing differs depending on the ray tracing (ray tracing method) or the Z-buffer method, but is generally performed in the order shown in FIG.

In FIG. 16, reference numeral 321 denotes world coordinate conversion means, 322 denotes viewpoint coordinate conversion means, 323 denotes clipping processing means, 324 denotes perspective conversion means, 325 denotes luminance calculation means, 326 denotes texture mapping means, and 327 denotes hidden surface processing. Means.

First, the world coordinate conversion unit 321 converts the world coordinates between the polygon and the texture, and then the viewpoint coordinate conversion unit 322 converts the viewpoint coordinates.
Then, clipping processing is performed by the clipping processing means 323, and perspective transformation is performed by the perspective transformation means 324. Then, luminance calculation of the texture is performed by the luminance calculation unit 325, and texture mapping (texture attachment) is performed by the texture mapping unit 326. Finally, hidden surface processing is performed by the hidden surface processing means 327. Note that the image generation processing of the present invention is not limited to this procedure or method.

Finally, the image generated by the sample image generation unit 32 is stored as a sample image in the sample image storage unit 33 of FIG.

Then, the feature point candidate image producing section 4 in FIG. 1 produces a feature point candidate image composed of an image obtained by projecting a point on the object surface onto a plane on which the two-dimensional image is projected.

FIG. 4 is a diagram showing a first configuration example of a feature point candidate image producing unit of the image generation and display device according to the first embodiment of the present invention. The case where a feature point candidate image is produced is shown.
In FIG. 4, reference numeral 41 denotes a feature point candidate, 42 denotes a feature point candidate photographing unit, and 43 denotes a feature point candidate image storage unit.

First, feature point candidate 4 is displayed on the surface of the display target object.
A mark of 1 is added by manual marking or the like.

Next, in the feature point candidate photographing section 42, the display object to which the mark is added is photographed by the camera from the sample position, and the photographed object is converted into a two-dimensional image.

Finally, in the feature point image storage unit 43,
The two-dimensional image is stored as a feature point candidate image, that is, an image including a point on the object surface that may be a feature point. The feature point candidate image is, for example, an image obtained by capturing a motion capture with a camera. Note that, for example, when a motion capture is captured by a camera, the feature point corresponds to the marker. In this case, various objects other than markers and others are shown, and these correspond to feature point candidate images.

FIG. 5 is a diagram showing a second configuration example of the feature point candidate image producing unit of the image generation and display device according to the first embodiment of the present invention. This configuration example generates and displays a computer graphic (CG) video. In this case, the case where the feature point candidate image is produced is shown. In FIG. 5, reference numeral 51 denotes a feature point candidate position storage unit, 52 denotes a feature point candidate generation unit, and 53 denotes a feature point candidate image storage unit.

First, the feature point candidate position storage unit 51 stores an arbitrary point on the surface of the display target object as a feature point candidate model and stores its three-dimensional position.

The feature point candidates are selected with reference to vertices and edges of the object, points at which the color and brightness change rapidly, and the like.

As the method, since the shape model is known, there is a method in which the operator inputs the feature point candidates numerically. Also, as described above, based on the shape model and the attribute data such as texture. There is a method of actually drawing by a simple image generation method, designating feature point candidates on a screen, and storing a three-dimensional position.

FIG. 17 shows a method of storing the three-dimensional position. In FIG. 17, 511 is a step of drawing an object model, 512 is a step of selecting a feature point candidate, 51
3 is a step of obtaining three-dimensional coordinates of an intersection between the object data and a half-line connecting the viewpoint of the virtual camera and the marker, and 514 is a step of converting to a three-dimensional position in the object coordinate system. 515 is whether or not a plurality of feature points have been obtained. Step 516 is a step of determining whether or not the position of the virtual camera has been changed. Step 517 is a step of changing the position of the virtual camera.

The method of storing the three-dimensional position is as follows.

That is, (1) A virtual camera position and a line-of-sight direction are selected, and an object model viewed from the position is actually drawn by using an image generation technique (step 511).

(2) A position (point) to be a feature point candidate is specified on the drawn screen using a mouse or the like, and a marker is attached on the drawn screen (step 512).

(3) A half line passing through a designated point (marker) on the drawing screen is drawn in the three-dimensional space from the viewpoint position of the virtual camera, and the intersection of the half-line and the shape data arranged in the three-dimensional space is drawn. A three-dimensional spatial position is calculated (step 513).

(4) Since the three-dimensional position of the intersection is a value in the viewpoint coordinate system, it is converted to a three-dimensional position in the object coordinate system (step 514).

(5) Steps (2) to (4) are repeated to obtain a plurality of feature point candidates (step 515).

(6) Returning to (1), the virtual cameras are arranged at different positions (steps 516 and 517), and (2) to (4) are repeated.

Next, the feature point candidate generation unit 52 shown in FIG. 5 generates an image of the feature point candidate model using the sample position as a virtual camera position.

In this processing, a virtual camera is arranged at an appropriate sample position, and feature point candidates at three-dimensional space positions are drawn by an image generation technique. Then, the feature point candidate is specified by a marker or the like.

Finally, the feature point candidate image storage unit 53 stores the image generated by the feature point candidate generation unit as a feature point candidate image.

FIG. 9 is a conceptual diagram of feature point candidates of the image generating and displaying apparatus according to the first embodiment of the present invention. 9, reference numerals 91 to 94 denote sample images, and the sample images 91 to 94 are examples of feature point candidate images corresponding to the sample positions A to D in FIG. E to K are feature point candidates of the sample image.

Next, the feature point designating section 6 shown in FIG. 1 designates points on the object surface that can be seen from a plurality of adjacent sample positions as feature points on the sample image and the feature point candidate images.

FIG. 6 is a configuration diagram of a feature point designation unit of the image generation and display device according to the first embodiment of the present invention. In FIG. 6, reference numeral 61 denotes an image synthesis display unit; 62, a feature point selection unit;
Reference numeral 3 denotes a feature point storage unit.

First, in the image synthesizing display section 61, the sample image and the feature point candidate image are alpha-synthesized at an arbitrary ratio and displayed on the screen.

FIG. 18 shows an example of the configuration of means for performing this alpha synthesis. In FIG. 18, reference numeral 611 denotes a multiplier for multiplying a feature point candidate image by α (α is a combination ratio and any value satisfying 0 ≦ α ≦ 1), and 612 multiplies a sample image by (1−α). A multiplier 613 is provided for these multipliers 61.
This is an adder for taking the sum of the outputs of 1,612 and outputting a composite image.

This alpha synthesis takes a weighted average of two images. The reason for performing this alpha synthesis is as follows.
This is because, in the case of a CG video, it is not clear which feature point candidate should be selected as a feature point unless the sample image and the feature point candidate image are combined. In the case of a real video, the display target object is also reflected in the feature point candidate image, so that it is not always necessary to perform the alpha synthesis.

Next, in the feature point selecting section 62 in FIG.
A feature point image consisting of only feature points is selected interactively or automatically by image recognition from feature point candidates of the composite image.

By selecting feature points from the feature point candidates by the feature point selection section 62, not only feature points such as noise are removed, but also the number of feature points is reduced as much as possible. The amount of calculation in the processing can be reduced.

At this time, the feature points to be selected include:
It is general to select a portion where the brightness or color of the image changes abruptly, that is, a contour or a corner of the image.

The following is a method of automatically selecting a feature point by image recognition, for example.

(1) Unnecessary feature point candidates are removed by moving a weighted block composed of n × n pixels on a sample image, multiplying each image data by a weight, and calculating the sum thereof. I do.

(2) Pixels of feature point candidates having a value equal to or less than a certain luminance value are removed.

However, since the above method cannot be used for the feature point candidates obtained by computer graphics, the feature point candidates must be manually selected and the final feature points must be manually selected.

Then, the two-dimensional position of the feature point is stored in the feature point storage section 63 of FIG.

Next, in the camera position input unit 1 of FIG. 1, an arbitrary point on the spherical surface is designated as a virtual camera position.

Then, the adjacent sample position calculator 5 calculates a sample position adjacent to the camera position.

Further, in the sample image deforming section 7,
A sample image obtained by photographing from the sample position is deformed so that each feature point matches.

FIG. 10 is a conceptual diagram of image deformation of the image generating and displaying device according to the first embodiment of the present invention. In FIG. 10, reference numerals 101 to 104 denote sample images after deformation.

FIG. 11 is a conceptual diagram of texture coordinates of the image generating and displaying apparatus according to the embodiment of the present invention. FIG.
, 111 is a texture coordinate axis, and 112 is a sample image.

The calculation of adjacent sample positions in the adjacent sample position calculation unit 5 of FIG. 1 is performed by the following method.

First, the coordinates of the camera position M specified by the camera position input unit 1 and the coordinates of the adjacent sample positions A to D calculated by the adjacent sample position calculation unit 5 on the virtual sphere 73 are represented by M (φ, θ), respectively. , A (φ1, θ1), B (φ1, θ1)
2), C (φ2, θ1) and D (φ2, θ2).

Here, it is assumed that θ and φ each satisfy the following equation.

Θ1 ≦ θ <θ2 φ1 ≦ φ <φ2 Texture images of feature points E at sample positions A to D before deformation are respectively _expressed by A (U _ea , V _ea ), B (U _eb , V _eb ),
_Assuming that C (U _ec , V _ec ) and D (U _ed , V _ed ), the texture coordinates E of the feature points E at the sample positions A to D after deformation.
(U _e , V _e ) is calculated by the following equation.

[0101] _{U e = W a · U ea} + W b · U eb + W c · U
_{ec + W d · U ed V} e = W a · V ea + W b · V eb + W c · V ec + W d · V
_ed Here, W _a , W _b , W _c , and W _d satisfy the following equations.

W _a = (1-α) (1-β) W _b = α (1-β) W _c = (1-α) β W _d = αβ α = (θ-θ1) / (θ2-θ1) ) Β = (φ−φ1) / (φ2−φ1) Similarly, the characteristic points F to
Sample positions A to A after deformation using texture coordinates of K
The texture coordinates of the feature points F to K of D are calculated.

Next, in the sample image synthesizing section 8 of FIG. 1, a synthesized image is created from the deformed sample image by weighted averaging using the distance between the camera position and the sample position as a weight.

Hereinafter, the deformation method will be described with reference to FIGS.

FIG. 12 is a conceptual diagram of a composite image of the image generating and displaying device according to the first embodiment of the present invention. In FIG. 12, reference numeral 121 denotes a transformed composite image, and reference numeral 122 denotes a composite image.

FIG. 13 is a conceptual diagram of a weighted average of the image generating and displaying device according to the first embodiment of the present invention. In FIG. 13, A to D are the viewpoint positions of the sample, and M is the viewpoint position.

The weights W _a , W _b , W _c , and W _d of the weighted average are calculated by the following equations.

W _a = (1-α) (1-β) W _b = α (1-β) W _c = (1-α) β W _d = αβ where α and β satisfy the following equations. I do.

Α = (θ−θ1) / (θ2−θ1) β = (φ−φ1) / (φ2−φ1) The luminance of the image is also synthesized by the same algorithm as the coordinates described above. I do.

Finally, in the composite image display section 9, the composite image is displayed as two objects to be displayed from the camera position.
It is displayed on the screen as an image projected on a two-dimensional plane.

As a result, using the sample images generated by actual shooting or computer graphics, an image from a virtual camera position which is not a sample position is obtained from a virtual camera position of the plurality of sample images which the current virtual camera wants to photograph. Generates this by interpolating a sample image taken (or generated) from a nearby camera position,
When generating images that can be displayed and corresponding to continuous and gradual changes in camera position and direction, even if the number of sample images is small, it can complicate realistic moving images with smooth motion. It can be easily generated and displayed without using complicated calculations.

For this reason, a moving image corresponding to a case where the line of sight is slowly changed with a processing time much shorter than the processing time required to generate all images from an arbitrary virtual camera position using computer graphics images. Can be easily and smoothly generated and displayed.

As described above, according to the apparatus of the first embodiment, the display object viewed from an arbitrary camera position is generated using the sample image, and the display object is displayed. When generating an image equivalent to a continuous and gradual change of, even if the sample image is small, a realistic moving image with smooth motion can be easily and quickly generated and displayed. can do.

Embodiment 2 Hereinafter, a generated display image editing device and an image generation display device according to Embodiment 2 of the present invention will be described with reference to the drawings.

In the second embodiment, the image generation and display device according to the first embodiment is different from the image generation and display device for performing editing, which is a preparation stage for the generation and display, and the image generation and display device for generating and displaying an original image. And these are realized as separate devices.

FIG. 19 is a configuration diagram of a generated display image editing apparatus and an image generation display apparatus according to Embodiment 2 of the present invention.

In the figure, reference numeral 200 denotes a generated display image editing apparatus, and 300 denotes an image display apparatus.

The generated display image editing apparatus 200 includes a sample position storage unit 2, a sample image production unit 3, a feature point candidate image production unit 4, and a feature point designating unit 6, and generates a sample image relating to a display object. Specify feature points for

The image generating and displaying apparatus 300 comprises a camera position input section 1, an adjacent sample position calculating section 5, a sample image deforming section 7, a sample image synthesizing section 8, and a synthetic image displaying section 9. Camera position input unit 1 based on the sample images and feature points obtained by
The sample image is deformed using the line-of-sight information obtained by the above, and is displayed.

As described above, according to the apparatus of the second embodiment, the apparatus includes a generation / display image editing apparatus for performing editing, which is a preparation stage of image generation / display, and an image generation / display apparatus for generating and displaying an original image. The image generation and display device, such as a video game machine, performs image generation and display exclusively. The feature point information can be generated in advance and provided to this.

Therefore, by using such a generated display image editing apparatus, the position and direction of the camera can be continuously and gently changed without imposing too much load on the central processing unit of the image generation display apparatus. Equivalent to
A realistic moving image with smooth movement can be easily generated and displayed.

[0122]

As described above, according to the image generation and display apparatus of the first aspect of the present invention, the position of the intersection of the latitude and longitude lines formed at predetermined intervals on a virtual spherical surface enclosing the display target is determined. A sample position storage unit for storing as a sample position, a sample image production unit for producing a sample image which is an image obtained by projecting an image to be displayed on the plane from the sample position, and a point including a feature point on the surface of the display object And a feature point candidate image producing unit for producing a feature point candidate image composed of an image projected on the plane, and a feature point candidate image from the points on the surface of the display target viewed from a plurality of sample positions adjacent to each other. And a feature point designating unit that designates a feature point candidate image, and an arbitrary point on the virtual sphere, a position of a virtual camera that captures the display target (hereinafter, referred to as a virtual camera position). A camera position input unit, a plurality of sample positions adjacent to the virtual camera position, and an adjacent sample position calculation unit, and the feature points of the sample images corresponding to the plurality of adjacent sample positions match. By deforming the sample image deforming section and the deformed sample image, by synthesizing the weighted average with the distance between the virtual camera position and the sample position where the sample image is positioned as a weight,
A sample image combining unit that obtains a combined image as a display target image viewed from the virtual camera position, and a combined image display unit that displays the combined image on the screen as an image obtained by projecting a display target viewed from the camera position onto a two-dimensional plane Therefore, there is an effect that a realistic and realistic moving image having a smooth motion in accordance with a continuous and gentle change in the position and direction of the camera can be generated and displayed.

According to the image generation and display device of the second aspect of the present invention, in the image generation and display device of the first aspect,
Since the sample image producing unit is constituted by an object photographing unit for photographing an object to be displayed from the sample position with a camera and converting it into a two-dimensional image, and a sample image storing unit for storing the two-dimensional image as a sample image. In addition, when an image is generated and displayed based on a real video image, there is an effect that a sample image can be easily produced.

According to the third aspect of the present invention, in the first aspect of the present invention,
A sample image production unit, an object model storage unit for storing an object model composed of shape and attribute data of a display target video, a sample image generation unit for generating an image of the object model using the sample position as a virtual camera, and a sample image generation unit And a sample image storage unit that stores the image generated by the unit as a sample image. Therefore, when an image is generated and displayed based on a computer graphics image, there is an effect that the sample image can be easily produced. .

According to a fourth aspect of the present invention, in the first aspect of the present invention,
A feature point candidate image producing unit, a feature point candidate adding unit that adds a mark serving as a candidate for a feature point to the surface of the display target object, and a camera object photographing the display target object to which the mark is added from a sample position into a two-dimensional image A feature point candidate photographing unit to be converted and a feature point image storage unit that saves the two-dimensional image as a feature point candidate image are used. There is an effect that candidate image production can be easily realized.

According to a fifth aspect of the present invention, in the first aspect of the present invention,
A feature point candidate image production unit, a feature point candidate position storage unit that stores an arbitrary point on the surface of the display target object as a feature point candidate model and saves its three-dimensional position; Since a feature point candidate generation unit for generating an image of a candidate model and a feature point image storage unit for storing the image generated by the feature point candidate generation unit as the feature point candidate image, the computer graphics image When an image is generated and displayed based on the feature point, there is an effect that the production of the feature point candidate image can be easily realized.

According to the sixth aspect of the present invention, in the first aspect of the present invention,
An image synthesis display unit that alpha-combines a sample image and a feature point candidate image at an arbitrary ratio and displays the image on a screen, and a feature point selection unit that interactively selects feature points from the feature point candidates of the synthesized image. And a feature point storage unit for storing the two-dimensional positions of the feature points, so that a point to be a feature point can be arbitrarily determined with reference to an image obtained by combining a sample image and a feature point candidate image. Since it is possible to select the desired point, it is possible to interactively select a desired point suitable as the feature point, thereby providing an effect of specifying the desired point.

Further, according to the generated display image editing apparatus of the present invention, the position of the intersection of the latitude and the longitude formed at a predetermined interval on the virtual spherical surface including the display object is stored as the sample position. A sample position storage unit, a sample image production unit that produces a sample image that is an image obtained by projecting an image to be displayed on the plane from the sample position, and a point including a feature point on the surface of the display object on the plane. And a feature point candidate image producing unit for producing a feature point candidate image composed of images projected on the surface of the display target viewed from a plurality of sample positions adjacent to each other. Since it is composed of a feature point designating part that designates on the image, a smooth and realistic moving image corresponding to continuous and gentle changes in the position and direction of the camera can be created. There is an effect that can be acquired sample image and feature points necessary for forming the display.

According to the generated display image editing apparatus of the present invention, the sample image producing section is configured to shoot the object to be displayed from the sample position with a camera. Since it is constituted by an object photographing unit for converting into a two-dimensional image and a sample image storing unit for storing the two-dimensional image as a sample image,
When editing for generating and displaying an image based on a real video image is performed, there is an effect that a sample image can be easily produced.

According to the generated display image editing apparatus of the ninth aspect of the present invention, in the generated display image editing apparatus of the seventh aspect, the sample image production unit is provided with an object model comprising the shape and attribute data of the video to be displayed. An object model storage unit for storing the image data, a sample image generation unit for generating an image of the object model using the sample position as a virtual camera, and a sample image storage unit for storing the image generated by the sample image generation unit as a sample image Therefore, when editing for generating and displaying an image based on a computer graphics image, it is possible to easily realize the production of a sample image.

According to a tenth aspect of the present invention, there is provided the generated and displayed image editing apparatus according to the seventh aspect, wherein the feature point candidate image producing unit is provided with a feature point candidate image forming part on the surface of the display target object. A feature point candidate adding unit that adds a mark as a target, a feature point candidate shooting unit that shoots a display target object to which the mark is added from the sample position by a camera, and converts the two-dimensional image into a feature point candidate image The feature point image storage unit stores the feature point candidate image when generating and displaying an image based on a real video image.

According to the generation display image editing apparatus of the present invention, in the generation display image editing apparatus of the present invention, the feature point candidate image producing unit is provided with an arbitrary point on the surface of the display target object. A feature point candidate storage unit that stores the three-dimensional position as a feature point candidate model, a feature point candidate generation unit that generates an image of the feature point candidate model using a sample position as the virtual camera position, and a feature point candidate generation And a feature point image storage unit that saves the image generated by the unit as the feature point candidate image. Therefore, when editing for generating and displaying an image based on a computer graphics image, the feature point There is an effect that candidate image production can be easily realized.

According to the generated display image editing apparatus of the twelfth aspect of the present invention, in the generated display image editing apparatus of the seventh aspect, the feature point designating unit may be configured to set the sample image and the feature point candidate image at an arbitrary ratio. An image synthesis display unit for alpha-synthesizing and displaying a screen, a feature point selection unit for interactively selecting feature points from feature point candidates of the synthesized image, and a feature point storage unit for storing the two-dimensional position of the feature points With this configuration, it is possible to arbitrarily select a point to be a feature point while referring to an image obtained by synthesizing the sample image and the feature point candidate image, and interactively select a desired point suitable as the feature point. This has the effect that this can be specified.

Further, according to the image generating and displaying apparatus of the thirteenth aspect of the present invention, an arbitrary point on a virtual spherical surface including a display object is positioned at a position of a virtual camera (hereinafter, referred to as an image) for imaging the display object. A camera position input unit designated as a virtual camera position), and a plurality of sample positions adjacent to the virtual camera position from among sample positions obtained by the generated display image editing apparatus according to any one of claims 7 to 12. 13. An adjacent sample position calculation unit for calculating, and a sample image corresponding to the plurality of adjacent sample positions obtained by the generated display image editing apparatus according to any one of claims 7 to 12 so that respective feature points match. A sample image deforming unit, and the deformed sample image between the virtual camera position and the sample position where the sample image is located. A sample image synthesis unit that obtains a synthesized image as a display target image viewed from the virtual camera position by synthesizing by a weighted average using the separation as a weight, and a display target that can view the synthesized image from the virtual camera position. And a synthetic image display unit that displays a screen as an image projected on a two-dimensional plane, so that the position and direction of the continuous and gentle camera can be controlled using information obtained in advance by an externally generated and displayed image editing device. There is an effect that it is possible to generate and display a realistic moving image with a smooth movement in accordance with the change of.

[Brief description of the drawings]

FIG. 1 is a configuration diagram of an image generation and display device according to a first embodiment of the present invention.

FIG. 2 is a configuration diagram in a first example of a sample image production unit of the image generation and display device according to the first embodiment of the present invention.

FIG. 3 is a configuration diagram in a second example of the sample image production unit of the image generation and display device according to the first embodiment of the present invention.

FIG. 4 is a configuration diagram in a first example of a feature point candidate image creation unit of the image generation and display device according to the first embodiment of the present invention.

FIG. 5 is a configuration diagram in a second example of the feature point candidate image creation unit of the image generation and display device according to the first embodiment of the present invention.

FIG. 6 is a configuration diagram of a feature point designation unit of the image generation and display device according to the first embodiment of the present invention.

FIG. 7 is a conceptual diagram of a viewpoint position of the image generation and display device according to the first embodiment of the present invention.

FIG. 8 is a conceptual diagram of a sample image of the image generation and display device according to the first embodiment of the present invention.

FIG. 9 is a conceptual diagram of a feature point candidate of the image generation and display device according to the first embodiment of the present invention.

FIG. 10 is a conceptual diagram of image deformation of the image generation and display device according to the first embodiment of the present invention.

FIG. 11 is a conceptual diagram of texture coordinates of the image generation and display device according to the first embodiment of the present invention.

FIG. 12 is a conceptual diagram of a composite image of the image generation and display device according to the first embodiment of the present invention.

FIG. 13 is a conceptual diagram of a weighted average of the image generation and display device according to the first embodiment of the present invention.

FIG. 14 is a diagram showing the relationship between the coordinate system of the dome and the coordinate system for creating a virtual object.

FIG. 15 is a diagram showing the correspondence between one polygon existing in the object model and the coordinate system of the texture.

FIG. 16 is a diagram showing the order of a general image generation processing method.

FIG. 17 is a flowchart illustrating a method for storing a three-dimensional position.

FIG. 18 is a diagram illustrating a configuration example of a unit that performs alpha synthesis.

FIG. 19 is a configuration diagram of a generated display image editing device and an image generation display device according to Embodiment 2 of the present invention.

FIG. 20 is a configuration diagram of a conventional example of an image generation and display device.

FIG. 21 is a diagram illustrating an example of a method for capturing a sample image.

[Explanation of symbols]

DESCRIPTION OF SYMBOLS 1 ... Camera position input part 2 ... Sample position storage part 3 ... Sample image production part 4 ... Feature point candidate image production part 5 ... Adjacent sample position calculation part 6 ... Feature point designation part 7 ... Sample image transformation part 8 ... Sample image synthesis Unit 9 Composite image display unit 21 Object imaging unit 22 Sample image storage unit 31 Object model storage unit 32 Sample image generation unit 33 Sample image storage unit 41 Feature point candidate 42 Feature point candidate imaging unit 43 Feature point candidate image storage section 51 ... Feature point candidate position storage section 52 ... Feature point candidate generation section 53 ... Feature point candidate image storage section 61 ... Image synthesis display section 62 ... Feature point selection section 63 ... Feature point storage section 71 ... Display Target object 72 Coordinate axis 73 Virtual sphere 81,82,83,84 Sample image 91,92,93,94 Sample image 01,102,103,104 ... sample image 111 ... texture coordinate axis 112 ... sample image 121 ... Synthetic image 122 ... Synthetic image A after deformation after deformation, B, C, viewpoint position M ... viewpoint position of the D ... sample

 ──────────────────────────────────────────────────続 き Continued on the front page (72) Inventor Noriyuki Hidaka 1006 Kazuma Kadoma, Kadoma City, Osaka Matsushita Electric Industrial Co., Ltd.

Claims (13)

[Claims] A sample position storage unit for storing, as a sample position, a position of an intersection of a latitude line and a meridian line formed at a predetermined interval on a virtual spherical surface including a display object; A sample image producing unit for producing a sample image which is an image projected on a plane; and a feature point candidate for producing a feature point candidate image composed of an image obtained by projecting a point including a feature point on the surface of a display target onto the plane. An image producing unit; a feature point designating unit for designating feature points from among points on the surface of the display target viewed from a plurality of sample positions adjacent to each other on the sample image and the feature point candidate image; A camera position input unit for designating an arbitrary point on a simple spherical surface as a position of a virtual camera that captures the display target (hereinafter, referred to as a virtual camera position); An adjacent sample position calculation unit that calculates a plurality of sample positions in contact with each other; a sample image deformation unit that deforms the sample images corresponding to the plurality of adjacent sample positions so that their respective feature points match each other; Of the virtual camera position and the sample position where the sample image is located is weighted averaged to obtain a composite image as a display target image viewed from the virtual camera position. An image generating and displaying apparatus, comprising: an image synthesizing unit; and a synthesized image display unit that displays the synthesized image on a screen as an image obtained by projecting a display target viewed from the virtual camera position onto a two-dimensional plane. 2. The image generation and display device according to claim 1, wherein the sample image production unit comprises: an object photographing unit for photographing a display target object with a camera from the sample position and converting the image into a two-dimensional image; An image generation and display device comprising: a sample image storage unit for storing an image. 3. The image generation and display device according to claim 1, wherein the sample image production unit comprises: an object model storage unit for storing an object model including shape and attribute data of a display target video; and the sample position as a virtual camera. An image generation and display device, comprising: a sample image generation unit that generates an image of the object model; and a sample image storage unit that stores an image generated by the sample image generation unit as a sample image. 4. The image generation and display apparatus according to claim 1, wherein the feature point candidate image producing unit performs camera photographing of a display target object in which a mark as a feature point candidate is added to a surface of the display target object from a sample position. An image generation and display apparatus comprising: a feature point candidate photographing unit that converts a two-dimensional image into a two-dimensional image; and a feature point image storage unit that saves the two-dimensional image as a feature point candidate image. 5. The image generation and display device according to claim 1, wherein the feature point candidate image producing unit stores a three-dimensional position of the arbitrary point on the surface of the display target object as a feature point candidate model. A position storage unit, a feature point candidate generation unit that generates an image of the feature point candidate model using the sample position as the virtual camera position, and a feature point that stores the image generated by the feature point candidate generation unit as the feature point candidate image An image generation and display device comprising an image storage unit. 6. The image generation and display device according to claim 1, wherein the feature point designating section alpha-combines the sample image and the feature point candidate image at an arbitrary ratio and displays the screen on a screen. An image generation and display apparatus comprising: a feature point selection unit that interactively selects a feature point from feature point candidates; and a feature point storage unit that stores a two-dimensional position of the feature point. 7. A sample position storage unit for storing, as a sample position, a position of an intersection of a latitude line and a meridian line formed at a predetermined interval on a virtual spherical surface including a display target, and an image of the display target from the sample position. A sample image producing unit for producing a sample image which is an image projected on a plane; and a feature point candidate for producing a feature point candidate image composed of an image obtained by projecting a point including a feature point on the surface of a display target onto the plane. An image production unit, and a feature point designating unit that designates feature points from among points on the surface of the display target viewed from a plurality of sample positions adjacent to each other on the sample image and the feature point candidate image. A generation display image editing apparatus characterized by the following. 8. The generated and displayed image editing apparatus according to claim 7, wherein the sample image producing unit comprises: an object photographing unit for photographing a display target object with a camera from the sample position and converting it into a two-dimensional image; A generated display image editing device, comprising: a sample image storage unit that stores a sample image. 9. The generation and display image editing apparatus according to claim 7, wherein the sample image production unit stores an object model including a shape and attribute data of a display target video, and a virtual camera that stores the sample position. A sample image generation unit for generating an image of the object model, and a sample image storage unit for storing an image generated by the sample image generation unit as a sample image. 10. The generated display image editing apparatus according to claim 7, wherein the feature point candidate image producing unit shoots a display target object having a mark as a feature point candidate added to a surface of the display target object from a sample position by a camera. And a feature point candidate photographing unit for converting the two-dimensional image into a two-dimensional image, and a feature point image storage unit for saving the two-dimensional image as a feature point candidate image. 11. The generation and display image editing apparatus according to claim 7, wherein the feature point candidate image producing unit stores a three-dimensional position of an arbitrary point on the surface of the display target object as a feature point candidate model. A candidate position storage unit, a feature point candidate generation unit that generates an image of the feature point candidate model using the sample position as the virtual camera position, and a feature that stores the image generated by the feature point candidate generation unit as the feature point candidate image. A generated display image editing device, comprising: a point image storage unit. 12. The generated display image editing apparatus according to claim 7, wherein the feature point designating section alpha-combines the sample image and the feature point candidate image at an arbitrary ratio and displays the screen on the screen. A feature point selecting unit for interactively selecting a feature point from the feature point candidates described above, and a feature point storage unit for storing a two-dimensional position of the feature point. 13. A camera position input unit for designating an arbitrary point on a virtual spherical surface including a display target as a position of a virtual camera (hereinafter, referred to as a virtual camera position) for imaging the display target, An adjacent sample position calculation unit that calculates a plurality of sample positions adjacent to the virtual camera position from among sample positions obtained by the generated display image editing apparatus according to any one of claims 7 to 12, and 12. A sample image deforming unit that deforms the sample images corresponding to the plurality of adjacent sample positions obtained by the generated display image editing apparatus according to any one of 12 so that their characteristic points match each other, and the deformed sample. Images are synthesized by weighted averaging using the distance between the virtual camera position and the sample position where the sample image is located as a weight. With this, a sample image combining unit that obtains a combined image as a display target image viewed from the virtual camera position, and displays the composite image on the screen as an image obtained by projecting a display target viewed from the virtual camera position onto a two-dimensional plane. An image generation and display device comprising a composite image display unit.