WO2017149755A1

WO2017149755A1 - Search device, search method, program, and non-transitory computer-readable information recording medium

Info

Publication number: WO2017149755A1
Application number: PCT/JP2016/056782
Authority: WO
Inventors: アディヤンムジビヤ
Original assignee: 楽天株式会社
Priority date: 2016-03-04
Filing date: 2016-03-04
Publication date: 2017-09-08

Abstract

An acceptance unit (210) accepts a shape model which represents a desired three-dimensional shape. For each product which is a search subject, an estimation unit (231) estimates, from a two-dimensional photographic image which represents the product, a photography direction at which the product has been photographed. For each of the estimated photography directions, a generating unit (232) generates a two-dimensional observation image in which the shape model has been observed from said photography direction. For each of the photographic images, an assessment unit (233) assesses whether said photographic image is similar to the observation image of the corresponding photography direction. An output unit (240) outputs product information for the photographic images which have thus been assessed as being similar.

Description

Search device, search method, program, and non-transitory computer-readable information recording medium

The present invention relates to a search device, a search method, a program, and a non-transitory computer-readable information recording medium.

In recent years, sales sites for electronic commerce have become popular on the Internet. For example, in a sales site (shopping mall site) composed of many stores (virtual stores), not only handles products of a wide category (genre) as a whole, but also price competition between stores is held. Sometimes it has gained many users.
In such a sales site, for example, a user performs a search by inputting a product name or a keyword, and searches for a target product from the product information displayed in a list.

Recently, proposals have been made to search for products by image search. For example, Patent Document 1 discloses a prior art that searches (identifies) a similar object from a photographic image.

JP 2011-128934 A

When applying the technology of Patent Document 1 described above to a sales site, it was necessary to register a three-dimensional model for each product. However, since the products on the sales site are enormous and new products are frequently registered, it is expected that it is difficult to register a three-dimensional model for each product.

For this reason, there has been a demand for a technique for appropriately searching for a product similar to a desired three-dimensional model without registering the three-dimensional model of the product on the sales site.

The present invention has been made in view of the above circumstances, and a search device, a search method, a program, and non-transitory computer-readable information capable of appropriately searching for a product similar to a desired three-dimensional model An object is to provide a recording medium.

The search device according to the first aspect of the present invention provides:
A reception unit for receiving a shape model representing a desired three-dimensional shape;
For each product to be searched, an estimation unit that estimates a shooting direction in which the product is photographed from a two-dimensional photographic image representing the product,
For each of the estimated shooting directions, a generating unit that generates a two-dimensional observation image obtained by observing the shape model from the shooting direction;
For each of the photographic images, a determination unit that determines whether the shooting direction is similar to the corresponding observation image,
An output unit that outputs product information about the photographic image determined to be similar;
It is characterized by providing.

In the search device according to the above aspect,
A storage unit that stores relevant information that associates the shooting direction estimated by the estimation unit with the corresponding photographic image;
The generation unit may generate an observation image obtained by observing the shape model from the shooting direction according to related information stored in the storage unit.

In the search device according to the above aspect,
The estimation unit may identify one or more vanishing points according to a plurality of edge portions extracted from the photographic image, and estimate a shooting direction based on the identified vanishing points.

In the search device according to the above aspect,
The determination unit obtains a feature amount for a shape from each of the photograph image and the observation image, and determines whether the photograph image and the observation image are similar based on the obtained feature amounts. Also good.

The search method according to the second aspect of the present invention is:
A search method in a search device,
An accepting step in which the search device accepts a shape model representing a desired three-dimensional shape;
An estimation step in which the search device estimates, for each product to be searched, a shooting direction in which the product is photographed from a two-dimensional photograph image representing the product;
A generation step in which the search device generates a two-dimensional observation image obtained by observing the shape model from the shooting direction for each of the estimated shooting directions;
A determination step for determining whether or not the retrieval direction is similar to the corresponding observation image for each of the photographic images;
An output step in which the search device outputs product information about the photographic image determined to be similar;
It is characterized by providing.

The program according to the third aspect of the present invention is:
Computer
A reception unit for receiving a shape model representing a desired three-dimensional shape;
For each product to be searched, an estimation unit that estimates a shooting direction in which the product is photographed from a two-dimensional photographic image representing the product,
For each of the estimated shooting directions, a generating unit that generates a two-dimensional observation image obtained by observing the shape model from the shooting direction;
For each of the photographic images, a determination unit that determines whether the shooting direction is similar to the corresponding observation image,
An output unit that outputs product information about the photographic image determined to be similar;
It is made to function as.

The above program can be distributed and sold via a computer communication network independently of the computer on which the program is executed.

A non-transitory computer-readable information recording medium according to the fourth aspect of the present invention is provided.
A reception unit for receiving a shape model representing a desired three-dimensional shape;
For each product to be searched, an estimation unit that estimates a shooting direction in which the product is photographed from a two-dimensional photographic image representing the product,
For each of the estimated shooting directions, a generating unit that generates a two-dimensional observation image obtained by observing the shape model from the shooting direction;
For each of the photographic images, a determination unit that determines whether the shooting direction is similar to the corresponding observation image,
An output unit that outputs product information about the photographic image determined to be similar;
A program characterized by functioning as a program is recorded.

The information recording medium is a non-transitory recording medium, and can be distributed and sold independently of the computer. Here, the non-temporary recording medium refers to a tangible recording medium. Non-temporary recording media are, for example, compact disks, flexible disks, hard disks, magneto-optical disks, digital video disks, magnetic tapes, semiconductor memories, and the like. The transitory recording medium refers to the transmission medium (propagation signal) itself. The temporary recording medium is, for example, an electric signal, an optical signal, an electromagnetic wave, or the like. The temporary storage area is an area for temporarily storing data and programs, and is, for example, a volatile memory such as a RAM (Random Access Memory).

According to the present invention, a product similar to a desired three-dimensional model can be appropriately searched.

It is a block diagram which shows an example of the whole structure of the sales system which concerns on this embodiment. It is a block diagram which shows an example of schematic structure of the typical information processing apparatus with which a search device and a user terminal are implement | achieved. It is a block diagram which shows an example of schematic structure of the search device which concerns on this embodiment. It is a schematic diagram for demonstrating an example of merchandise information. It is a figure which shows an example of a photograph image. It is a schematic diagram for demonstrating a mode that the edge part was extracted. It is a schematic diagram for demonstrating a mode that a vanishing point is calculated | required. It is a schematic diagram for demonstrating a mode that an imaging | photography direction is calculated | required. It is a schematic diagram which shows an example of a list page (search result). It is a flowchart for demonstrating the search process which concerns on this embodiment.

Embodiments of the present invention will be described below. In the embodiment of the present invention, a sales site (shopping mall site) constituted by a collection of many stores (virtual stores) will be described as an example. However, the present invention can be appropriately applied to a sales site by a single store (trader). .
Further, the following embodiments are for explanation, and do not limit the scope of the present invention. Therefore, those skilled in the art can employ embodiments in which each or all of these elements are replaced with equivalent ones, and these embodiments are also included in the scope of the present invention.

(overall structure)
A sales system 100 according to an embodiment of the present invention is configured by connecting a search device 200 and each user terminal 300 via the Internet 900 as shown in FIG. Although simplified in the figure, it is assumed that there are a large number of user terminals 300 according to the users to be used.

The search device 200 includes, for example, a sales server (server computer) and the like, and provides information related to products to the user terminal 300 and sells products desired by the user.
For example, the search device 200 sells products of many stores (virtual stores), and presents a list page of products to the user terminal 300 that has accessed. As will be described later, a shape model (three-dimensional model) representing a desired three-dimensional shape is transmitted from the user terminal 300 to the search device 200 as a query. Then, the search device 200 compares an observation image appropriately generated from the shape model with a photographic image (product image) of the product, and presents a list page of products determined to be similar to the user terminal 300.

The user terminal 300 includes, for example, a personal computer or a smartphone, and accesses the search device 200 via the Internet 900 to instruct the search device 200 to search for a product.
As will be described later, when searching for a product, it is possible to instruct image search using a shape model (three-dimensional shape model) created by a user or the like.

(Outline configuration of information processing device)
A typical information processing apparatus 400 in which the search apparatus 200 and the user terminal 300 according to the embodiment of the present invention are realized will be described.

As shown in FIG. 2, the information processing apparatus 400 includes a CPU (Central Processing Unit) 401, a ROM (Read Only Memory) 402, a RAM (Random Access Memory) 403, a NIC (Network Interface Card) 404, an image A processing unit 405, an audio processing unit 406, a DVD-ROM (Digital Versatile Disc Disc ROM) drive 407, an interface 408, an external memory 409, a controller 410, a monitor 411, and a speaker 412 are provided.

The CPU 401 controls the overall operation of the information processing apparatus 400 and is connected to each component to exchange control signals and data.

The ROM 402 records an IPL (Initial Program Loader) that is executed immediately after the power is turned on, and when this is executed, a predetermined program is read into the RAM 403 and the CPU 401 starts executing the program. The ROM 402 stores an operating system program and various data necessary for operation control of the information processing apparatus 400 as a whole.

The RAM 403 is for temporarily storing data and programs, and holds programs and data read from the DVD-ROM and other data necessary for communication.

The NIC 404 is used to connect the information processing apparatus 400 to a computer communication network such as the Internet, and conforms to the 10BASE-T / 100BASE-T standard used when configuring a LAN (Local Area Network). Analog modem for connecting to the Internet using a telephone line, ISDN (Integrated Services Digital Network) modem, ADSL (Asymmetric Digital Subscriber Line) modem, cable modem for connecting to the Internet using a cable television line, etc. These are configured by an interface (not shown) that mediates between these and the CPU 401.

The image processing unit 405 processes the data read from the DVD-ROM or the like by a CPU 401 or an image arithmetic processor (not shown) provided in the image processing unit 405, and then processes this data in a frame memory provided in the image processing unit 405. (Not shown). The image information recorded in the frame memory is converted into a video signal at a predetermined synchronization timing and output to the monitor 411. Thereby, various page displays are possible.

The audio processing unit 406 converts audio data read from a DVD-ROM or the like into an analog audio signal, and outputs the analog audio signal from the speaker 412 connected thereto. Further, under the control of the CPU 401, a sound to be generated during the progress of the processing performed by the information processing apparatus 400 is generated, and a sound corresponding to the sound is output from the speaker 412.

The DVD-ROM mounted on the DVD-ROM drive 407 stores, for example, a program for realizing the search device 200 according to the embodiment. Under the control of the CPU 401, the DVD-ROM drive 407 performs a reading process on the DVD-ROM loaded therein, reads necessary programs and data, and these are temporarily stored in the RAM 403 or the like.

External memory 409, controller 410, monitor 411, and speaker 412 are detachably connected to interface 408.

The external memory 409 stores data relating to the user's personal information in a rewritable manner.

The controller 410 accepts operation inputs performed at various settings of the information processing apparatus 400. The user of the information processing apparatus 400 can record these data in the external memory 409 as appropriate by inputting instructions through the controller 410.

The monitor 411 presents the data output by the image processing unit 405 to the user of the information processing apparatus 400.

The speaker 412 presents the audio data output by the audio processing unit 406 to the user of the information processing apparatus 400.

In addition, the information processing apparatus 400 uses a large-capacity external storage device such as a hard disk so as to perform the same functions as the DVD-ROM mounted on the ROM 402, RAM 403, external memory 409, and DVD-ROM drive 407 You may comprise.

Hereinafter, the configuration of the search device 200 realized in the information processing apparatus 400 will be described with reference to the drawings. When the information processing apparatus 400 is turned on, for example, when an access from the user terminal 300 is received, each program causing the information processing apparatus 400 to function as the search apparatus 200 according to the present embodiment is executed, thereby realizing the search apparatus 200 according to the present embodiment. Is done.
Each user terminal 300 is similarly realized in the information processing apparatus 400, but these configurations are omitted, and the most characteristic search apparatus 200 in the present embodiment will be described below.

(Outline configuration of search device)
FIG. 3 is a block diagram illustrating an example of a schematic configuration of the search device 200 according to the present embodiment. As illustrated, the search device 200 includes a reception unit 210, a storage unit 220, a control unit 230, and an output unit 240.

The accepting unit 210 accepts various information sent from each user terminal 300 via the Internet 900.
For example, the accepting unit 210 accepts from the user terminal 300 a product search instruction and various operation instructions (for example, a mouse click operation by the user) in a list page presented in response to the search instruction.
Note that the search instruction from the user terminal 300 includes not only a character search such as a keyword but also an image search. The image search includes not only a search using a two-dimensional image as a query but also a search using a shape model representing a three-dimensional shape as a feature of the present invention as a query.
That is, when performing an image search using a three-dimensional shape model as a query, the accepting unit 210 accepts a shape model representing a three-dimensional shape (a shape model in a predetermined format) from the user terminal 300.
The above-described NIC 404 or the like can function as such a reception unit 210.

The storage unit 220 stores information about products, information about stores, and information about users.
For example, the storage unit 220 stores product information 221 as illustrated in FIG. As an example, the product information 221 includes a store ID 221a, a product ID 221b, a category name 221c, a product name 221d, a price 221e, and a product image 221f.
This product image 221f is a two-dimensional photographic image representing a product. For example, a shooting direction estimated by an estimation unit 231 described later can be added to the product image 221f as attribute information. Instead of adding a shooting direction to the product image 221f, the shooting direction estimated by the estimation unit 231 may be included in the product information 221 as related information of the product image 221f. 4 shows a case where there is only one product image 221f for the product. For example, a plurality of product images 221f having different shooting directions may be stored. Also good. In that case, the estimation unit 231 estimates the shooting direction for each of the plurality of product images 221f, even for the same product.
In the product information 221 of FIG. 4, the category name 221c indicates the product in the table, but the actual storage unit 220 stores information on all products in all stores (virtual stores).

In addition, the storage unit 220 stores store information and user information.
The above-described RAM 403, external memory 409, and the like can function as such a storage unit 220.

3, the control unit 230 controls the entire search device 200. The control unit 230 includes an estimation unit 231, a generation unit 232, and a determination unit 233, and performs a product search and the like.

The estimation unit 231 estimates, for each product to be searched, a shooting direction in which the product is photographed from a two-dimensional photograph image (product image) representing the product.
For example, the estimation unit 231 identifies one or more vanishing points according to a plurality of edge portions extracted from the photographic image, and estimates the shooting direction based on the identified vanishing points.
Hereinafter, the estimation of the shooting direction will be described using the photographic image shown in FIG.

First, the estimation unit 231 extracts an edge portion as shown in FIG. 6 from the photographic image of FIG. Such extraction of the edge portion is performed by, for example, a method of taking out the hue value and luminance value of the pixel in the photographic image through a differential filter and picking out a portion with a large amount of change every week. Note that the edge portion may be extracted by other methods.
Next, the estimation unit 231 obtains a point (1 to 3 points) where the straight line of the edge portion is stretched and the stretched straight line is converged as a vanishing point. Specifically, the estimation unit 231 obtains two vanishing points SP as shown in FIG. In FIG. 7, for the convenience of explanation, the vanishing point SP is obtained at a close position, but the actual vanishing point SP is obtained at a farther position. Note that the number of vanishing points SP obtained varies depending on the photographic image. For example, only one vanishing point SP can be obtained for a photographic image obtained by photographing a product (table or the like) from the front. In addition, three vanishing points SP are obtained for a photographic image obtained by obliquely photographing a product at a short distance.
Finally, the estimation unit 231 obtains a photographing point based on the vanishing point, and estimates a photographing direction passing through the obtained photographing point. Specifically, the estimation unit 231 obtains a shooting point VP in which two vanishing points SP are orthogonal to each other as shown in FIG. 8, and estimates a shooting direction Z passing through the shooting point VP. Note that the actual position of the shooting point VP changes according to the focal length of the camera (lens) that has shot, but the shooting direction Z does not change.

As described above, the estimation unit 231 estimates the shooting direction in which the product is photographed from the photographic image for each of the products to be searched.
The estimation unit 231 may store the estimated shooting direction in the storage unit 220 and reuse it when a new shape model is received. For example, the estimation unit 231 adds the estimated shooting direction as attribute information to the product image 221f in the product information 221 of FIG. 4 described above. Further, instead of adding the shooting direction to the product image 221f, the estimation unit 231 may add the shooting direction to the product information 221 as related information of the product image 221f.

Returning to FIG. 3, the generation unit 232 observes a shape model (a three-dimensional shape model accepted by the reception unit 210 as a query) from each of the shooting directions estimated by the estimation unit 231 2. A dimensional observation image is generated. For example, the generation unit 232 expands a shape model in a three-dimensional virtual space, arranges a virtual camera (viewpoint) in the shooting direction in the virtual space, and performs a perspective transformation to generate a two-dimensional observation image. To do. At this time, the viewpoint position in the photographing direction may be determined so that the size of the observation image is equal to that of the photographic image (product size).

As described above, the generation unit 232 generates a two-dimensional observation image obtained by observing the shape model from each of the estimated shooting directions.

The determination unit 233 determines whether each of the photographic images is similar to a corresponding observation image (an observation image obtained by observing the shape model from the same shooting direction). For example, the determination unit 233 obtains a feature amount for the shape from each of the photographic image and the observation image, and determines whether the photographic image and the observation image are similar based on the obtained feature amounts. Note that similarity determination between a photographic image and an observation image may be performed by other methods.

In addition, the control unit 230 performs, for example, a predetermined settlement process when a product is purchased by the user.
The above-described CPU 401 and the like can function as the control unit 230 having such a configuration.

The output unit 240 outputs various information to the user terminal 300 via the Internet 900.
For example, the output unit 240 outputs product information about a photographic image that is determined to be similar by the determination unit 233. Specifically, the output unit 240 presents the search result list page 500 as shown in FIG. 9 to the user terminal 300. That is, even if the direction (photographing direction) of the photographic image registered on the sales site side varies, a product similar to the shape model accepted as a query is appropriately searched and presented to the user. .
The above-described CPU 401, NIC 404, and the like can function as such an output unit 240.

(Operation of search device)
Hereinafter, the operation of the search apparatus 200 having such a configuration will be described with reference to the drawings. FIG. 10 is a flowchart showing the flow of search processing executed by the search device 200. This search process is executed each time a three-dimensional shape model is given as a query from the user terminal 300 and a search is instructed.

First, the search device 200 verifies the received shape model (step S11).
That is, the control unit 230 verifies that the shape model received by the receiving unit 210 satisfies a predetermined requirement. For example, the control unit 230 verifies whether the shape model is a three-dimensional shape model in a predetermined format or a shape model for a product or the like. If the requirement is not satisfied, an error screen or the like is output from the output unit 240 without performing the subsequent processing.
In addition, when the shape model is verified to determine which category (product category) is the shape model, the migration processing is performed by narrowing down the products in the identified category to the search target products. .

The search device 200 sets the initial value 1 to the variable n (step S12). This variable n is used when information about the nth product is read from the storage unit 220.

The search device 200 estimates the shooting direction of the nth product (step S13).
That is, the estimation unit 231 estimates the shooting direction in which the product is captured from the photographic image of the nth product. For example, the estimation unit 231 identifies one or more vanishing points according to a plurality of edge portions extracted from the photographic image, and estimates the shooting direction based on the identified vanishing points.
Specifically, the estimation unit 231 extracts the edge portion as shown in FIG. 6 described above from the above-described photographic image of FIG. Next, the estimation unit 231 obtains two vanishing points SP as shown in FIG. Finally, the estimation unit 231 obtains a shooting point VP in which two vanishing points SP as shown in FIG. 8 are orthogonal to each other, and estimates a shooting direction Z passing through the shooting point VP.

The search device 200 generates an observation image based on the estimated shooting direction (step S14).
That is, the generation unit 232 generates a two-dimensional observation image obtained by observing the shape model from the shooting direction estimated in step S13. For example, the generation unit 232 expands a shape model in a three-dimensional virtual space, arranges a virtual camera (viewpoint) in the shooting direction in the virtual space, and performs a perspective transformation to generate a two-dimensional observation image. To do.

The search device 200 determines whether or not the generated observation image is similar to the photographic image of the nth product (step S15).
That is, the determination unit 233 determines whether or not the photographic image of the nth product is similar to the observation image generated in step S14. For example, the determination unit 233 obtains a feature amount for the shape from each of the photographic image and the observation image, and determines whether the photographic image and the observation image are similar based on the obtained feature amounts.

If the search device 200 determines that they are not similar (step S15; No), the search device 200 proceeds to step S17 described later.

On the other hand, when it is determined that they are similar (step S15; Yes), the search device 200 reads the information of the nth product from the storage unit 220 and adds it to the search result (step S16).

The search device 200 determines whether there is a next product (step S17).

When the search device 200 determines that there is the next product (step S17; Yes), the search device 200 adds 1 to the variable n (step S18). Then, the search device 200 returns the process to step S13 described above.

On the other hand, when it is determined that there is no next product (step S17; No), the search device 200 outputs a search result (step S19).
That is, the output unit 240 presents the search result list page 500 as shown in FIG.

By such a search process, it is possible to appropriately search for a product similar to a desired three-dimensional model without registering the three-dimensional model for the product at the sales site.

(Other embodiments)
In the above embodiment, the merchandise sales site has been described as an example. However, in addition to such merchandise, the present invention can be appropriately applied to a search site for searching for an image.

As described above, according to the present invention, a search device, a search method, a program, and a non-transitory computer-readable information recording medium capable of appropriately searching for products similar to a desired three-dimensional model Can be provided.

DESCRIPTION OF SYMBOLS 100 Sales system 200 Search apparatus 210 Reception part 220 Storage part 230 Control part 231 Estimation part 232 Generation part 233 Determination part 240 Output part 300 User terminal 400 Information processing apparatus 401 CPU
402 ROM
403 RAM
404 NIC
405 Image processing unit 406 Audio processing unit 407 DVD-ROM drive 408 Interface 409 External memory 410 Controller 411 Monitor 412 Speaker 900 Internet

Claims

A reception unit for receiving a shape model representing a desired three-dimensional shape;
For each product to be searched, an estimation unit that estimates a shooting direction in which the product is photographed from a two-dimensional photographic image representing the product,
For each of the estimated shooting directions, a generating unit that generates a two-dimensional observation image obtained by observing the shape model from the shooting direction;
For each of the photographic images, a determination unit that determines whether the shooting direction is similar to the corresponding observation image,
An output unit that outputs product information about the photographic image determined to be similar;
A search device comprising:
The search device according to claim 1,
A storage unit that stores relevant information that associates the shooting direction estimated by the estimation unit with the corresponding photographic image;
The generation unit generates an observation image obtained by observing the shape model from the shooting direction according to related information stored in the storage unit.
A search device characterized by that.
The search device according to claim 1,
The estimation unit specifies one or more vanishing points according to a plurality of edge portions extracted from the photographic image, and estimates a shooting direction based on the specified vanishing points.
A search device characterized by that.
The search device according to claim 1,
The determination unit obtains a feature amount for a shape from the photographic image and the observation image, and determines whether the photographic image and the observation image are similar based on the obtained feature amounts.
A search device characterized by that.
A search method in a search device,
An accepting step in which the search device accepts a shape model representing a desired three-dimensional shape;
An estimation step in which the search device estimates, for each product to be searched, a shooting direction in which the product is photographed from a two-dimensional photograph image representing the product;
A generation step in which the search device generates a two-dimensional observation image obtained by observing the shape model from the shooting direction for each of the estimated shooting directions;
A determination step for determining whether or not the retrieval direction is similar to the corresponding observation image for each of the photographic images;
An output step in which the search device outputs product information about the photographic image determined to be similar;
A search method comprising:
Computer
A reception unit for receiving a shape model representing a desired three-dimensional shape;
For each product to be searched, an estimation unit that estimates a shooting direction in which the product is photographed from a two-dimensional photographic image representing the product,
For each of the estimated shooting directions, a generating unit that generates a two-dimensional observation image obtained by observing the shape model from the shooting direction;
For each of the photographic images, a determination unit that determines whether the shooting direction is similar to the corresponding observation image,
An output unit that outputs product information about the photographic image determined to be similar;
A program characterized by functioning as
Computer
A reception unit for receiving a shape model representing a desired three-dimensional shape;
For each product to be searched, an estimation unit that estimates a shooting direction in which the product is photographed from a two-dimensional photographic image representing the product,
For each of the estimated shooting directions, a generating unit that generates a two-dimensional observation image obtained by observing the shape model from the shooting direction;
For each of the photographic images, a determination unit that determines whether the shooting direction is similar to the corresponding observation image,
An output unit that outputs product information about the photographic image determined to be similar;
A non-transitory computer-readable information recording medium having a program recorded thereon, which is made to function as: