JP6917140B2 - Image processing device - Google Patents

Description

The present invention relates to image processing based on 3D modeling data.

There is a technique for generating modeling data representing a three-dimensional shape of a virtual object or a real object by 3D CAD, laser scanning, or the like (so-called modeling). By performing drawing processing (so-called 3D rendering) based on the generated modeling data, viewpoint information, light source information, and the like, images of the object viewed from various viewpoints can be displayed on the screen.

Japanese Unexamined Patent Publication No. 2016-170488

Generally, the size of modeling data is large and requires complicated calculations during display processing. When the drawing process is performed on the user terminal using the modeling data acquired from the content server, it takes time to download the modeling data from the content server and the processing load on the user terminal is large. In addition, an application that performs rendering processing must be installed on the terminal. In addition, there is a risk that the modeling data will be duplicated and distributed without permission by a malicious user.
An object of the present invention is to realize a display based on 3D data on a user terminal without providing modeling data to the user terminal.

In one embodiment, the present invention is an acquisition means for acquiring modeling data representing a three-dimensional shape of an object, and generation of generating a plurality of two-dimensional image data representing the object viewed from one direction based on the modeling data. An image processing apparatus having a means and an output means for collectively outputting the generated two-dimensional image data or outputting a two-dimensional image selected by the user from the generated two-dimensional image data. offer.

According to the present invention, the display based on the modeling data is performed on the user terminal without passing the modeling data to the user terminal.

The schematic diagram of the image processing system 10. The functional diagram of the image processing server 100. The figure for demonstrating the calculation of a feature amount. The functional diagram of the user terminal 200. The conceptual diagram for demonstrating the process of generating 2D data from 3D data. An example of the structure of a 2D dataset generated from a single 3D data. An operation example of the image processing system 10. An example of a download site displayed on the screen of the user terminal 200. A screen example (No. 1) displayed when the image display application is executed on the user terminal 200. A screen example (No. 2) displayed when the image display application is executed on the user terminal 200. A screen example (No. 3) displayed when the image display application is executed on the user terminal 200. A screen example (No. 4) displayed when the image display application is executed on the user terminal 200. A screen example (No. 5) displayed when the image display application is executed on the user terminal 200. Another example of the operation of the image processing system 10. A screen example (No. 6) displayed when the image display application is executed on the user terminal 200.

<Example>
FIG. 1 shows an outline of the image processing system 10. The image processing system 10 includes an image processing server 100 and a user terminal 200. The image processing server 100 is, for example, a server managed by a business operator that publishes object contents and provides them to a user terminal 200. The user terminal 200 is, for example, a general-purpose personal computer. Reference numeral 300 is, for example, the Internet or a leased line. The image processing server 100 and the user terminal 200 are connected via 300.

FIG. 2 shows the function of the image processing server 100. The image processing server 100 includes a control unit 110, a communication unit 120, a storage unit 130, a display unit 140, and 150. The communication unit 120 is implemented as a communication interface for transmitting and receiving data to and from the user terminal 200 in accordance with a predetermined communication standard, and includes a transmission unit 121 and a reception unit 122.

The receiving unit 122 acquires modeling data (hereinafter, referred to as “3D data”) representing the three-dimensional shape of the object. The 3D data is data showing the three-dimensional shape of a real object or a virtual object, and is, for example, STL, OBJ, PLY, OFF, 3DS, COLLADA, PTX, V3D, PTS, APTS, XYZ, GTS, TRI. , ASC, X3D, X3DV, VRML, ALN, DXF, GTS, U3D, IDTF, X3D and the like. The 3D data may be data created by 3DCG software, 3D CAD software, or the like and representing a virtual object, or may be data obtained by 3D scanning a real object. Further, the data may be obtained from a 3D data sharing site or the like, and the generation method or acquisition method thereof is not limited.

The acquired 3D data is stored in the storage unit 130 and read out in response to a request received from the user terminal 200. This request includes at least information that specifies an object to be displayed, and may also include information regarding the direction (viewpoint) of the object. The direction is represented, for example, as a position (Cartesian coordinates or polar coordinates) in three-dimensional space with a certain point in the virtual space where the object is placed as the origin. Alternatively, the direction may be defined as a range of directions such as front, back, right side, and left side.

The requirement may further include designation of one or more parts of the object. For example, the object is a doll and contains information that specifies a part of the object such as "face", "eyes", and "body". In short, it is a part of the three-dimensional space occupied by an object (a set of coordinates).

The transmission unit 121 outputs a plurality of two-dimensional image data (hereinafter referred to as 2D data) corresponding to one 3D data generated by the data generation unit 112. The 2D data has a format such as JPEG, BMP, PDF, TIF and the like. The plurality of 2D data may be output in whole or in part at the same time, or may be output one by one according to a request received from the user. Further, the output timing may be different as long as the plurality of 2D data are transmitted in association with all or part of the data as data related to a specific object.

The storage unit 130 is a storage device for a hard disk, a semiconductor memory, or the like, and stores a program executed by the control unit 110 and 3D data read by the program.

The display unit 140 is an input device operated by the administrator of the image processing server 100 such as a keyboard, a touch panel, a mouse, and a stylus. Reference numeral 150 denotes a display device such as a liquid crystal display for presenting information such as the operating status of the image processing server 100 to the administrator.

The image processing server 100 is implemented as one or more dedicated processors such as a general-purpose processor (CPU or the like) and / or a drawing processing processor (GPU), and has a feature determination unit 111, a data generation unit 112, and a viewpoint setting unit 113. And include.

The feature determination unit 111 acquires the feature amount related to the object represented by the 3D data and outputs it to the data generation unit 112 or the viewpoint setting unit 113.
The feature amount related to the object corresponds to the direction of the 2D data generated by the data generation unit 112 by the feature determination unit 111 calculating the feature amount in each direction based on the 3D data acquired from the storage unit 130. The feature amount may be added to each 2D data generated by the viewpoint setting unit 113.
Alternatively, the feature amount related to the object may be calculated by the feature determination unit 111 based on the 2D data generated by the data generation unit 112. As a specific method of calculating based on the 2D data, the viewpoint setting unit 113 determines based on at least one of the structural feature and the color (pattern) feature of the object according to a predetermined algorithm. Other structural features are defined, for example, by the texture (degree of unevenness) on the surface of the object and the number of vertices and faces. Color or pattern features are defined, for example, as the variety of colors used.

FIG. 3 is a diagram for explaining a feature amount. Doll 900 is an example of an object represented by 3D data.
(A) is an image of the doll 900 viewed from the front, an image of the doll 900 rotated by 45 ° in the horizontal plane (left and right direction of the paper surface) (an image in which the viewpoint is rotated by 45 ° in the horizontal plane), and rotated by 90 °. The image, the image rotated by 135 °, and the image rotated by 180 ° are shown.

FIGS. (B) and (c) show examples of parameters representing the feature quantities calculated corresponding to each angle. (B) shows the feature quantities calculated according to the angles when 2D data of θ = 0 °, 45 °, 90 °, 135 °, 180 °, 225 °, and 270 ° are generated as 2D data. .. In this example, the front side is structurally more complicated than the back side, and it is determined that the image viewed from the front has a relatively large feature amount. FIG. (C) is an example in which the feature amount is calculated as a function of θ directly from the 3D data.

The feature amount related to the object may be acquired separately from the 3D data and the 2D data. For example, in the image of the viewpoint in the range of the front side (−45 ° to −45 °), the weighting parameter is defined as + Δw1, the back side (135 ° to 225 °) is defined as −Δw1, and the side surfaces on both sides are defined as 0. The feature amount may be input to the user terminal 200 by the user and transmitted to the image processing server 100. For example, it is defined that the direction in which the face of the doll 900 is facing is "front", and whether or not the face is reflected by performing image analysis on the image represented by the 2D data generated by the data generation unit 112. It is possible to give a larger feature amount to the image in which the face is shown than in the image in which the face is not shown.

In the above example, for convenience of explanation, the feature amount is calculated for one rotation direction, but the feature amount corresponding to any direction represented by the two angle parameters θ and φ can be calculated. For example, the feature amount depending on the orientation (φ direction) in the vertical direction of the paper may be calculated from the 3D data or the 2D data, or may be acquired separately from the 3D data or the 2D data. For example, in the case of the doll shown in the figure (a), in general, an image viewed from directly above (θ = 0 °, φ = 90 °) or directly below (θ = 0 °, φ = −90 °) is viewed. The feature amount of the 2D data represented is smaller than the feature amount of the 2D data representing the image viewed from the front (θ = 0 °, φ = 0 °).

The viewpoint setting unit 113 determines from which direction the 2D data to be generated should be viewed from the doll 900. In other words, the viewpoint setting unit 113 determines the direction in which the doll 900 is virtually photographed or the viewpoint of the photographing.
Information indicating the determined direction is output to the data generation unit 112. For example, the viewpoint setting unit 113 may determine the direction based on the feature amount supplied from the feature determination unit 111. Further, when the image processing server 100 receives the designation of the part of the doll 900 from the user terminal 200, the viewpoint setting unit 113 determines the above direction so that the designated part always appears in one or more images. May be good.
Alternatively, the feature amount may be determined by a device other than the image processing server 100 and supplied to the image processing server 100.

The data generation unit 112 generates a plurality of 2D data representing the doll 900 viewed from one direction, respectively, based on one 3D data. Specifically, parameters such as a light source are set as necessary according to a predetermined rendering algorithm, and rendering processing is performed based on the viewpoint information supplied from the viewpoint setting unit 113.
In the rendering process, the feature amount supplied from the feature determination unit 111 may be used. When the feature amount is generated based on the 2D data, even if the 2D data generated by the data generation unit 112 is first output to the feature determination unit 111 and the feature amount generated by the feature determination unit 111 is acquired. good.
The data generation unit 112 may determine the resolution of one or more 2D data included in the generated plurality of 2D data according to the feature amount. As an example of setting the resolution based on the feature, the process of lowering the resolution is performed as the 2D data has a smaller feature amount. For such an image with a relatively small amount of features (in other words, an image from a viewpoint that the object viewed from that angle has few features or is presumed to be of low importance or interest to the user). By relatively lowering the resolution, it is possible to reduce the total amount of 2D data while suppressing a decrease in discomfort given to the user when drawing processing is performed based on a plurality of 2D data sets. The resolution is defined by, for example, the number of vertical and horizontal pixels.
The generated 2D data is supplied to the storage unit 130 or the communication unit 120.

FIG. 4 is a functional diagram of the user terminal 200. The user terminal 200 includes a communication unit 210, an input unit 220, a display unit 230, a storage unit 240, and an image processing unit 250.

The storage unit 240 is a storage device for a hard disk, a semiconductor memory, or the like, and stores a program executed by the image processing unit 250 and 2D data acquired from the image processing server 100.

The input unit 220 is an input device operated by a user (hereinafter, simply referred to as a user) of the user terminal 200 such as a keyboard, a touch panel, and a mouse, and receives an instruction from the user. This instruction includes the direction of the doll 900 to be displayed. Further, this instruction may include information used when the image processing server 100 generates 2D data.

The communication unit 210 is implemented as a communication interface for transmitting and receiving data to and from the user terminal 200 in accordance with a predetermined communication standard. Specifically, the request including the designation of the display target and the designation of the direction of the display target, which are input by the input unit 220 and necessary for displaying on the display unit 230, is transmitted to the image processing server 100, and the request is made. 2D data transmitted from the image processing server 100 is received and supplied to the image processing unit 250 according to the above.

The image processing unit 250 is implemented as one or more dedicated processors such as a general-purpose processor (CPU or the like) and / or a drawing processing processor (GPU), and is instructed by the 2D data and the input unit 220 received from the image processing server 100. The drawing process is performed based on the orientation. Specifically, among the plurality of 2D data stored in the storage unit 240 corresponding to one 3D data, one or a plurality of 2D data corresponding to the direction specified by the input unit 220 is specified. It performs image processing such as necessary encoding and outputs it to the image processing unit 250 together with a drawing command.

The display unit 230 is a display device such as a liquid crystal display, and displays an image under a command from the image processing unit 250.

FIG. 5 is a conceptual diagram for explaining a process of generating 2D data from 3D data. The doll 900 is arranged in a three-dimensional (XYZ) virtual space whose origin is P. Set the central axis CL that penetrates the center of the doll 900.
The image processing server 100 generates a two-dimensional image of the doll 900 viewed from a plurality of viewpoints set in the virtual space. In other words, it corresponds to "shooting" the doll 900 with a virtual camera (corresponding to a viewpoint; hereinafter simply referred to as a camera) installed in a virtual space. In the figure, A to N represent cameras. θ is the angle in the XY plane, and φ is the angle in the XZ plane. L1 to L5 define the position where the camera is arranged, and are XY planes having different φs. In other words, φ represents the height of the camera (angle in the vertical direction), and θ represents the angle in the plane of the camera. It is assumed that the direction in which the face of the doll 900 is facing (θ = 0 °) is defined as the “front”. The reference direction of the doll 900, such as the front, may be determined based on the feature amount of the 3D data, or may be separately specified by the image processing server 100 or the user terminal 200. In this example, all cameras are equidistant from P.
That is, the plurality of generated 2D data represent an image whose viewpoint is a position equidistant from the doll 900 and having a different direction with respect to the object.

As shown in the figure, a total of 6 cameras from A to H are arranged in L3, I and J are arranged slightly diagonally above the doll 900 in L2, and near the overhead of the doll 900 in L1. In L4, K is arranged at a position slightly diagonally downward, L and M are arranged, and in L5, N is arranged near the feet. As a result, a total of 14 2D data obtained by setting the viewpoint positions to A to N in one 3D data are generated.
As shown in the figure, in this example, the camera is mainly arranged in front of the doll 900. More specifically, the cameras are densely arranged at positions close to θ = −90 and φ = 0, and sparsely arranged at distant positions (upper, lower, side, back). This is because the feature determination unit 111 calculates that the image of the feature amount viewed from the front side is relatively large, and the viewpoint setting unit 113 determines the camera position based on the feature amount.

The arrangement shown in FIG. 5 is only an example. The placement of the camera is a hardware resource such as the characteristics of the object (3D data), the user's preference, the communication environment such as restrictions on the data size of the generated 2D data, and the conditions related to the processing load allowed by the image processing server 100. And can be determined according to conditions relating to at least one of the software resources.

Regarding the number of cameras, for example, as the number of cameras decreases, the total amount of 2D data generated decreases, but the number of viewpoints that can be selected by the user decreases, so that the reproducibility of 3D data decreases. On the contrary, as the number of cameras increases, there is an increased risk that the user of the user terminal 200 will generate the original 3D data (data close to) by performing interpolation processing based on the 2D data set.

Regarding the position of the camera, the characteristic portion of the object differs depending on the object, and may depend on the user's subjectivity such as where the user is paying attention. For example, even if an object can be thought of as "front", the image from the front is not always the most important. In addition, there may be a case where you want to check the whole image of an object evenly from various directions. In such a case, the viewpoint setting unit 113 may determine the arrangement of the cameras without using the feature amount. As an example, the number arranged in each layer is set to be the same. Then, the distance between the cameras on the same layer is set to be equal. For example, when 30 pieces are arranged on a layer, θ is arranged at a position shifted by 12 °, and 2D data for a total of 150 sheets is generated.

Also, it is not necessary for each camera to be equidistant from the reference point. For example, the camera position may be determined so that the distance from the object surface is substantially equidistant. The method of setting the origin P is not limited to the mode shown in FIG. P may be set outside the space occupied by the doll 900.
The cameras do not have to have the same performance. For example, a plurality of cameras having different angles of view and focus may be arranged.
In short, it suffices to generate a plurality of 2D data corresponding to each viewpoint based on one 3D data.

FIG. 6 shows an example of the data structure of the data D generated from one 3D data. (A) shows the overall structure of the data. The data D is composed of a header portion Dh and an image data main body Dm composed of a plurality of 2D data. The header part Dh includes an identifier for identifying the 3D data of the generation source and information defining the structure of the data. This information defines the relationship between a plurality of two-dimensional image files included in the image data main body Dm and a plurality of viewpoints.
(B) shows the details of the image data main body Dm. In this example, the viewpoint setting unit 113 shows a case where five layers are set and 30 cameras (every 12 °) are arranged for each layer. The Dm contains image data for a total of 150 images of IM1-1 to IM5-30, which are arranged in the order specified in the header portion Dh (in this example, each layer is grouped).

The generated 2D data may be in a format recognized as one file as a whole as shown in FIG. 6, or may be recognized as a set of individual image files. In short, the format may be such that when it is handled by the user terminal 200, it is recognized that each is generated based on one original 3D. In the latter case, for example, each 2D data is generated as one independent image file containing the identification information of the original 3D data and the information indicating the viewpoint in the header.

FIG. 7 is an example of the operation of the image processing system 10. The user terminal 200 starts a web browser or the like, accesses the website applied by the image processing server 100, and displays the content on the screen of the image processing server 100 (S102). FIG. 8 is an example of the screen W2 displayed when the image display application is executed on the user terminal 200. A 2D image (image viewed from the front in the figure), which is a representative of an object represented by 3D data, is displayed as a thumbnail.
When the user specifies an image to be viewed (S104), a request including identification information of the image is transmitted to the image processing server 100 (S106).

The image processing server 100 acquires 3D data of the designated object (S108) and generates a 2D data set based on the 3D data (S110). The generated 2D data sets are associated with each other, transmitted to the user terminal 200 (S112), and stored in the display unit 230.

When the drawing application is started automatically or by the user's instruction in the user terminal 200, the 2D data set is read and the screen W3 as shown in FIG. 9 is displayed. Here, one 2D data corresponding to a representative direction (here, the front) of the 2D data set is selected as the default, and the image is displayed. In addition, a message prompting the user to specify the desired direction is displayed on the screen W3.

When the user specifies a direction (for example, a position rotated by 45 ° in the horizontal plane) (S114), the image processing unit 250 reads out and displays one 2D data corresponding to the specified direction (S116). As a result, a screen such as W4 is displayed. If there is no 2D data corresponding to the direction specified by the user (for example, the direction rotated by 14 °), such designation is ignored or the 2D data in the direction closest to the specified direction is extracted. .. For example, when an instruction is received to continuously rotate 45 times in the same direction from this state, the screens shown in W5, W6, and W7 are continuously displayed, and the user can see the doll 900 as if it were rotated. felt.

As described above, according to the above embodiment, the user can give a feeling as if the result of calculation based on the 3D data possessed by the user terminal 200 is displayed. However, in reality, since the 3D data itself is not provided to the user terminal 200, the risk of duplication of the 3D data is reduced. Further, since the amount of data exchanged is smaller than that of the original 3D data, the response between the image processing server 100 and the user terminal 200 is improved. In addition, it is not necessary to install an application having a rendering function on the terminal.

<Other Examples>
The method of providing 2D data is not limited to the mode shown in FIG. FIG. 13 is another example of the operation of the image processing system 10. The same reference numerals as those in FIG. 7 indicate the same operation contents. The difference from the operation of FIG. 7 in this example is that the user terminal 200 does not receive the 2D data sets in a batch, but receives them as needed. That is, when the user specifies the display target, one direction is further specified (S103). Specifically, when the drawing application receives a new direction designation from the user while displaying one 2D data associated with a certain 3D data like W3, the drawing application is added to the identification information of the 3D data. , Information indicating the direction is transmitted to the image processing server 100 (S106).

The image processing server 100 reads the 3D data if it is stored in the storage unit 130, and newly acquires the 3D data if it is not stored (S108).
Subsequently, the image processing server 100 extracts one 2D data corresponding to the designated direction (S120). If such data is already stored in the storage unit 130, it is read out. Specifically, if the 2D data viewed from the same direction as the direction designated as the 2D data related to the 3D data already exists, the data is read out. If the data of the same angle does not exist, the 2D data of the closest angle is read out. Alternatively, the control unit 110 may newly generate 2D data viewed from a newly designated direction.
One extracted or generated 2D data is transmitted to the user terminal 200 (S122). The drawing display application on the image processing server 100 executes drawing processing based on the received 2D data. The point at which the image corresponding to the direction specified by the user is displayed (S116) is the same as that of the above-described embodiment.

For example, when a plurality of 2D data corresponding to a plurality of angles are received at one time as shown in FIG. 7, a plurality of images may be displayed on one screen as shown in the screen W8 shown in FIG. In this case, what is displayed on one screen may be a part of the acquired 2D data. In the case of FIG. 15, of the 150 acquired 2D data, only 5 data in (θ: 0 ° to 180 °, φ = 0) are displayed at one time. That is, the direction in which the user specifies the display is not limited to one direction, and may be a range of a plurality of directions or directions. This designation method can also be applied in the aspect shown in FIG.

The viewpoint setting unit 113 may arrange two or more cameras in the same direction. Specifically, the user specifies the distance from the object to the viewpoint in addition to the direction, and the image processing server 100 provides the user terminal 200 with 2D data according to the specified direction and distance (enlargement / reduction ratio). .. As a result, it is possible to give the user the feeling of performing a zoom operation in addition to the rotation operation.

In short, in the system of the present invention, there are a step of acquiring modeling data representing a three-dimensional shape of an object and a step of generating a plurality of two-dimensional image data representing the object viewed from one direction based on the modeling data. , A step of outputting at least a part of a plurality of two-dimensional image data may be executed. Here, the plurality of two-dimensional image data to be output may be those specified by the user, may be output at the same time regardless of the user's specification, or may be output in association with each other. May be good. In short, it suffices to output in such a manner that the receiving terminal can grasp that the plurality of two-dimensional image data are all generated based on one original modeling data.

100 ... image processing server, 200 ... user terminal, 110 ... control unit, 111 ... feature determination unit, 112 ... data generation unit, 113 ... viewpoint setting unit, 130 ... Storage unit, 140 ... Display unit, 120 ... Communication unit, 121 ... Transmit unit, 122 ... Receiver unit, 230 ... Display unit, 210 ... Communication unit, 220 ... Input Unit, 240 ... Storage unit, 250 ... Image processing unit

Claims (7)

An acquisition means for acquiring modeling data representing a three-dimensional shape of an object,
A second acquisition means for acquiring a feature amount related to the object, and
A generation means for generating a plurality of two-dimensional image data representing the object based on the modeling data, and
An output means for outputting all or a part of the generated two-dimensional image data at the same time or in association with each other, or outputting a two-dimensional image selected by the user from the generated two-dimensional image data. Prepare ,
The generation means determines the resolution of the plurality of two-dimensional image data according to the feature amount.
Image processing device. The plurality of two-dimensional image data represents an image whose viewpoint is a position equidistant from the object and having a different direction with respect to the object.
The image processing apparatus according to claim 1. The generation means determines the direction for each of the plurality of two-dimensional image data based on the feature amount.
The image processing apparatus according to claim 2. The feature amount is calculated for each two-dimensional image data generated by the generation means.
The image processing apparatus according to claim 2 or 3. Further provided with a receiving means for receiving the designation of the part of the object from the user,
The direction is determined for each of the plurality of two-dimensional image data so that the designated portion appears in one or more images.
The image processing apparatus according to claim 2 or 3. On the computer
Steps to acquire modeling data representing the three-dimensional shape of an object,
The step of acquiring the feature amount related to the object and
Based on the modeling data, a step of generating a plurality of two-dimensional image data representing the object viewed from one direction, and
A program for executing a step of collectively outputting the generated two-dimensional image data or outputting a two-dimensional image selected by the user from the generated two-dimensional image data .
In the step of generating a plurality of the two-dimensional image data, the resolution of the plurality of two-dimensional image data is determined according to the feature amount.
program. Steps to acquire modeling data representing the three-dimensional shape of an object,
The step of acquiring the feature amount related to the object and
Based on the modeling data, a step of generating a plurality of two-dimensional image data representing the object viewed from one direction, and
Possess and outputting the two-dimensional images selected by the user among the plurality of two-dimensional image data output or the product together a plurality of two-dimensional image data the generated,
In the step of generating a plurality of the two-dimensional image data, the resolution of the plurality of two-dimensional image data is determined according to the feature amount.
Image processing method.

Images