JP3403143B2 - Image processing method, apparatus and storage medium - Google Patents

Description

Detailed Description of the Invention

[0001]

BACKGROUND OF THE INVENTION The present invention collects ray space data.
Generate an image of virtual space according to user's operation using
It is for doing.

Now, a method of recording the light space data will be described.

As shown in FIG. 1, the coordinate system OXY-
Install Z. A ray passing through a reference plane P (Z = z) perpendicular to the Z axis is represented by a position (x, y) where the ray crosses P and variables θ and φ indicating the direction of the ray. That is, one ray is uniquely determined by the five variables (x, y, z, θ, φ). If the function expressing the light intensity of this ray is defined as f, the ray group data in this space can be expressed by f (x, y, z, θ, φ). This five-dimensional space is called "ray space"
Call.

Here, if the reference plane P is set to z = 0 and the parallax information in the vertical direction of the light beam, that is, the degree of freedom in the φ direction is omitted, the degree of freedom of the light beam is reduced to two dimensions (x, θ). Can be made. This x-θ two-dimensional space is a subspace of the ray space. Then, if a ray (Fig. 2) passing through a point (X, Z) in the real space is set as u = tan θ, as shown in Fig. 3 in x-u space, [Equation 1] X = It is mapped on the line x + u ・ Z.

Shooting with a camera corresponds to an operation of receiving a light ray passing through an entrance pupil of the camera on an image pickup surface and imaging its brightness and color. In other words, a group of light rays passing through one point in the real space called the entrance pupil position is acquired as an image for the number of pixels. Here, since the degree of freedom in the φ direction is omitted and the behavior of light rays is considered only in the XZ plane, only pixels on a line segment that intersects the plane orthogonal to the Y axis in the image are considered. Become. In this way, the light rays that pass through one point can be collected by taking an image, and x
-Data on one line segment in u space can be acquired.

If a large number of this photographing is performed by changing the viewpoint position, a ray group passing through a large number of points can be obtained. When the real space is photographed using N cameras as shown in FIG. 4, it corresponds to the entrance pupil position (Xn, Zn) of the n-th (n = 1, 2, ..., N) camera Cn. , As shown in FIG. 5, it is possible to input the data on the straight line of [Equation 2] x + u · Zn = Xn. In this way, by shooting from a sufficient number of viewpoints,
The xu space can be densely filled with data.

On the contrary, an observation image from a new arbitrary viewpoint position can be generated from the data in the xu space (FIG. 6) (FIG. 7). As shown in this figure, the observation image from the new viewpoint position E (X, Z) expressed in the form of eyes is
It can be generated by reading the data on the straight line of Expression 1 on the xu space from the xu space.

One of the major characteristics of ray space data is that ray space data is defined for each pixel. That is, the frame data of one scene is represented by the ray space data for the number of pixels of the frame. Therefore, the data amount of the ray space data does not depend on the complexity of the scene, but the calculation amount depends only on the size and resolution of the scene, that is, only the total number of pixels of the scene to be generated. In the case of normal CG data, if the scene becomes complicated, the complexity cannot be expressed unless the number of polygons is increased. As a result, the amount of calculation increases, and thus the rendering performance deteriorates. In this case, if the total number of pixels of the image to be drawn is constant, the drawing performance will be constant regardless of the complexity of the scene.

Consider a case where a user walks through a virtual space created using such ray space data. At this time, in order to give the user the feeling of walking through, it is necessary to generate and present an image of several tens of frames per second.

However, if the total number of pixels of the image reconstructed from the ray space data in each frame is large, it takes time to draw one frame, and the drawing frame rate may not follow the moving speed.

Even when the user operates (moves or rotates) an object in the scene at high speed, if the object is generated and drawn from the ray space data, the number of pixels of the object is large. Drawing cannot be done in time.

The present invention has been proposed in order to solve the above-mentioned problems of the prior art, and its object is to reconstruct an image from image data including spatial information such as ray space data. In this case, if there is no problem in lowering the resolution, such as when moving in space, lower the sampling rate and read the image information at a resolution lower than the resolution of the image information during recording. An object of the present invention is to propose an image processing apparatus and method that secure a high drawing frame rate by reconstructing an image.

The objects of the present invention are achieved.
Therefore, for example, the image processing apparatus of the present invention has the following configuration.
Get That is, by the user using the ray space data
Image processing for generating virtual space images according to operations
A device for recording the light space data, and a device for reading from the recording device in response to an operation by a user.
Setting means for setting a sampling rate indicating a pixel interval of the light ray space data to be emitted, and light ray sky from the recording means according to the sampling rate set by the setting means.
A reconstruction means for reconstructing an image of the virtual space is read between the data, the reconstructed image by the reconstruction means
To enlarge the image so that it has the set image size.
Characterized in that it comprises an interpolation unit for performing pixel interpolation processing.

When the sampling rate is determined based on, for example, the moving speed of the virtual space, the image information is read out at the sampling rate and the image is about to be reconstructed. Since this rate is lower than the resolution at which it was recorded, the time required for image reconstruction is shortened. Moreover, the decrease in resolution is compensated by pixel interpolation (for example, texture mapping).

In order to achieve the object of the present invention, for example, a book
The image processing method of the invention has the following configuration. That is,
The user can use the ray space data recorded in the memory.
Image processing to generate an image of virtual space according to the operation
An image processing method performed by a processing device, which is operated by a user.
Image of the ray space data read from the memory according to
A setting step of setting a sampling rate that indicates the elementary interval, according to the sampling rate set in the setting step, the reading light field data from the memory
A reconstruction step for reconstructing an image of the virtual space, the reconstruction
Image reconstructed in process, set image size
Characterized in that it comprises an interpolation step of performing pixel interpolation processing so as to expand so as to.

The above object can also be achieved by an image processing method or a storage medium for storing a program for realizing this method.

[0015]

DETAILED DESCRIPTION OF THE INVENTION Referring to the accompanying drawings,
An image processing apparatus and an image processing method to which the present invention is applied will be described in detail. The image processing apparatus and the image processing method set the reading sampling rate of the ray space data when walking through the virtual space or when manipulating (moving or rotating) an object in the virtual space represented by the ray space. By lowering the resolution by lowering the resolution, the drawing frame rate is made to follow the spatial movement speed, and on the other hand, when the resolution is lowered, pixel interpolation is performed by texture mapping.

FIG. 8 illustrates the quantization of the ray space data in this embodiment. That is, the ray space data handled by the image processing apparatus of this embodiment is a camera having a CCD with a number of pixels N in the main scanning direction, an angle of view w, and an intersection angle α between the optical axis and the Z axis is α. It is assumed that the image data is obtained by using.
Each pixel of the CCD is sequentially numbered 0, ..., N-1, the angle formed by the ray passing through the jth pixel with the Z axis is θ, and the ray passing through the jth pixel intersects with the X axis. When the position is x, [Equation 3] N / 2tan (θ−α) = (N / 2−j) tan (w / 2) holds. In FIG. 8, the Y axis is mapped. Therefore, if Expression 3 holds for the j-th pixel on the arbitrary Y = m line, the angle θ corresponding to the pixel j can be obtained from Expression 3. That is, when the i-th image data is represented by I ″ (i), this is converted into image data I ′ in the (x, θ) space, and u = tan θ is further converted into the image data I in the (x, u) space. If conversion is performed, [Equation 4] I ″ (i) = I ′ (x, θ) = I (x, u).

U and x are quantized so as to preserve the resolution of the original image. For example, regarding the x-axis, the minimum value of the interval between two adjacent rays on the x-axis is obtained, and quantization is performed at this interval. The u axis is quantized by the tangent of the angle formed by two adjacent pixels from the camera viewpoint position.

By doing so, as shown in FIG. 10, the ray space data becomes points sampled discretely on a straight line.

The smoothness of movement when the user or object moves at high speed in the virtual space depends on the drawing frame rate. That is, the faster the drawing frame rate, the more the user or the object feels as if moving smoothly in the virtual space, and the slower the motion becomes jerky, the more uncomfortable the movement in the space becomes. Therefore, in order to realize smooth movement in the virtual space even when the moving speed of the user or the object is high, it is necessary to improve the drawing rate. However, this means increasing the total number of pixels in the image reconstructed from the ray space data, and therefore
There is a feeling that the drawing is not in time. Therefore, in this embodiment, the sample rate in FIG. 10 is changed. That is, 2
If every third pixel is suitable for drawing, then every second pixel is sampled. The user's moving speed is
In this embodiment, the movement speed of the mouse is used for detection. If this speed is set to vcm / s, the degree of load for drawing at this time is kv (k is a predetermined constant). The sample rate t should be set as large as possible. Load kv
The sample rate t (every pixel) corresponding to is the load kv
Is a function of. The sample rate t is a numerical value such that drawing will not be completed in time if set to a value lower than that, and can be determined in advance in accordance with the drawing speed of the present system and the value of the load kv. When such t is determined, the ray space data is sampled at every t sample points in FIG. The amount of processing data is reduced by sampling every t pixels to cope with the movement or scrolling of the user or the object. It should be noted that even if the amount of data is reduced and the resolution is degraded, the visual characteristics of the user or the object observing the same while moving in space, that is, the details of the moving object or the scene of the moving object or scene. The reduced resolution space is not a problem because the discriminative ability of the is reduced.

FIG. 11 shows the system configuration of this embodiment. The hardware configuration shown in FIG. 11 is that of a normal workstation. In the figure, in particular, the disk 25 stores a large amount of ray space data as shown in FIG. In this system, the virtual space CRT23 is provided to the user.
It is presented above. At this time, the user uses the mouse 2
8 can be operated to freely walk through the virtual space, or an object in the virtual space can be operated (moved, rotated, etc.). The speed at which the walkthrough or the object is operated is the moving speed of the mouse 28. FIG. 12 shows the first
The control procedure of the system of FIG. 1 is shown.

When the system is started in step S2,
In step S4, the moving direction and the moving amount v of the mouse are obtained based on the user's operation on the mouse 28.
In step S6, the image size is determined from the size of the display range of the CRT 23. In step S8, the sample rate t (pixels) is determined from the movement amount v of the mouse, and the light space data is read from the disk 25 at the sample rate t (pixels) to reconstruct the image.

In step S10, the image is enlarged. This enlargement reduces the image because the image reconstruction performed in step S8 is thinning at the sample rate t. Therefore, the image is enlarged so that the image size is determined in step S6, and the texture mapper 24, which is well-known hardware, is used. By doing so, image interpolation is performed and image display at a desired resolution is realized.

The present invention can be modified in various ways.

Modification 1: In the above embodiment, the case of reconstructing an image from the image database of the ray space data format at the same site was dealt with, but the present invention is applied to the case where the database is remote via the network. Is also applicable. This is because when the system is used via a network, the transfer rate may fluctuate due to the load of the network. In the above embodiment, when the observer (user) walks through, the drawing rate is in time with the moving speed. This is because the same problem as when there is no case occurs. In other words, if there is little change or movement of the image through the network, the frame rate will decrease, so the CPU of the image processing device on the receiving side has a margin, but there is no data, so the problem is that the image becomes jerky. . When the present invention is applied to deal with such a problem, although the image data is deteriorated in image quality by setting the sampling rate of the image data between the transmitting side and the receiving side according to the data transfer rate of the network. It becomes possible to provide a smooth motion picture.

The processing of this modified example is shown in FIG. In the flowchart of the figure, the same processes as those in FIG. 12 are designated by the same reference numerals, and the description thereof will be omitted. The difference from FIG. 12 is that the data transfer rate of the network is determined in the process of step S31 and the image transfer rate is also referred to in the image generation process of step S81 to set the sampling rate of the image.

Modification 2: The movement speed can be detected by means other than the mouse. The point is that the moving speed such as walk-through or scrolling should be detected.

Further, in the above embodiment, it is assumed that the load increase with respect to the moving speed of the user is linear, but the present invention is not limited to this form. For example, the resolution (total number of pixels of the image) may be set to be inversely proportional to the square of the speed. Further, the actual measurement value may be actually obtained and determined.

Modified Example 3: In the above embodiment, the ray space data is obtained by calculation, but RAM or ROM which is tabulated in advance may be used.

Modification 4: The display device is not limited to the CRT. It can be applied to lenticular type and HMD type display devices.

[0030]

As described above, according to the present invention, even if the virtual space based on the ray space data has a large processing load on the computer, the load is suppressed by lowering the sampling rate and the resolution. be able to. In particular, if the increase in load processing is caused by the movement of the virtual space or the movement of the object in the virtual space by the user operation, the reduction in the resolution by lowering the sampling rate does not pose a problem to the user's vision. . Note that the deterioration of the image quality can be suppressed to some extent because the deterioration of the resolution is compensated by the pixel interpolation. Also
According to the invention described in claim 6, the set sump
Reading of light space data by ring rate is relatively
It can be done easily.

[Brief description of drawings]

FIG. 1 is a diagram illustrating the definition of a ray space.

FIG. 2 is a diagram showing a ray group passing through a point (x, z) in a real space.

FIG. 3 is a diagram when a ray group passing through a point (x, z) in the real space is mapped to the ray space.

FIG. 4 is a diagram showing sampling of a ray group by a plurality of cameras.

FIG. 5 is a diagram showing a state in which a ray group sampled by a plurality of cameras is mapped to a ray space.

FIG. 6 Point (x, z) in real space from ray space data
For explaining the principle of reconstructing an image that can be observed in.

FIG. 7 is a diagram illustrating a relationship between a plurality of cameras used for sampling rays and a viewpoint position at which an image is to be reconstructed.

FIG. 8 is a diagram illustrating quantization in the embodiment.

FIG. 9 is a diagram illustrating storage of ray space data according to the embodiment.

FIG. 10 is a diagram illustrating sampling storage of ray space data according to the embodiment.

FIG. 11 is a block diagram showing a hardware configuration of the embodiment.

FIG. 12 is a flowchart showing a control procedure of the embodiment.

FIG. 13 is a flowchart of processing according to the first modification of the present invention.

─────────────────────────────────────────────────── ─── Continuation of the front page (56) References Interactive operation of 3D live-action space based on ray description, IEICE Transactions, Japan, The Institute of Electronics, Information and Communication Engineers, May 25, 1998, VOL ． J81-D-2 NO. 5, p850-860 Realization of Time Critical Rendering using adaptive texture mapping, IPSJ Research Report 96-CG-79, Japan, Information Processing Society of Japan, February 23, 1996, p39-46. (58) Fields investigated (Int.Cl. ⁷ , DB name) G06T 15/00-17/50 H04N 13/00-13/04

Claims (13)

(57) [Claims] 1. A user operation using ray space data.
Image processing device for generating virtual space images according to the work
And recording means for recording the light space data, and reading from the recording means in response to a user operation.
Said reading and setting means for setting the sampling rate of a pixel interval of the light space data, the light space data from said recording means in accordance with a sampling rate set by the setting means
The reconstruction means for reconstructing the image of the virtual space and the image reconstructed by the reconstruction means are set.
Pixel interpolation processing to enlarge so that the image size is
The image processing apparatus characterized by comprising an interpolation means for performing. 2. The image processing apparatus according to claim 1, wherein the setting unit determines the sampling rate based on the moving speed of the virtual space designated by the user. 3. The image processing apparatus according to claim 1, wherein the setting unit determines the sampling rate based on an operation speed of an object in the virtual space designated by the user. 4. The ray space data includes rays intersecting the X axis.
Where x is the tangent of the angle the ray makes with the Z axis
Characterized by being managed in the indicated (x, u) space
The image processing apparatus according to claim 1. 5. The x is on the x-axis of two adjacent rays.
Is quantized by the minimum value of the interval at
Quantized by the tangent of the angle between two adjacent pixels from the position
The image processing apparatus according to claim 4, wherein the image processing apparatus is provided. 6. A light beam read by the reconstructing means.
The spatial data is the sample data set by the setting means.
On a straight line in (x, u) space according to
Characterized by discretely sampled data
The image processing apparatus according to claim 4. 7. The light space data recorded in the memory is used.
And generate an image of the virtual space according to the user's operation.
An image processing method because the image processing apparatus is performed in response to operation by a user, the light read from the memory
A setting step of setting a sampling rate of a pixel spacing of line space data in accordance with a sampling rate set in the setting step, the from the memory reads the light space data temporary
The reconstruction process for reconstructing the image of the virtual space and the image reconstructed in the reconstruction process are set.
An image processing method characterized by comprising the <br/> interpolation step of performing pixel interpolation processing so as to expand so that the image size. 8. The image processing method according to claim 7 , wherein in the setting step , the sampling rate is determined based on the moving speed of the virtual space designated by the user. 9. The image processing method according to claim 7 , wherein in the setting step , the sampling rate is determined based on an operation speed of an object in the virtual space designated by the user. 10. The ray space data is such that a ray is on the X axis.
U, which is the tangent of the intersecting position x and the angle the ray makes with the Z axis
Is managed in the (x, u) space indicated by
The image processing method according to claim 7. 11. The x is the x-axis of two adjacent rays.
Is quantized by the minimum value of the above interval,
Quantize with the tangent of the angle between two adjacent pixels from the point position
The image processing method according to claim 10, wherein
Law. 12. A ray sky read in the reconstructing step.
Interval data is the sampling level set in the setting process.
The discretization on a straight line in (x, u) space according to
The data is sampled in
10. The image processing method according to 10. 13. A computer recorded in a memory
Virtual according to user operation using ray space data
Functions as an image processing device for generating spatial images.
Computer reading that stores the program for
The storage medium is a storage medium that can be read, and the program is an optical read from the memory in response to a user operation.
Set the sampling rate of a pixel interval of the line space data
According to the setting process to be set and the sampling rate set in the setting process.
Read out the light space data from the memory,
The reconstruction process for reconstructing the image of the virtual space and the image reconstructed in the reconstruction process are set.
Pixel interpolation processing is performed to enlarge the image size.
Computer reading characterized by comprising an interpolation step
Removable storage medium.