JP2011113197A - Method and system for image search - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To provide a method and a system for image search capable of accurately identifying corresponding points even in the case of comparing object images having many objects of similar shapes like building images having a large number of objects of similar shapes such as windows and pillars. <P>SOLUTION: From a pair of a query image Iq in Fig.10(a) and a similar image candidate (street image) Iwc in Fig.10(b), two sets of corresponding point candidates [q1, w1] and [q2, w2] are extracted. In regard to one corresponding point candidate [q1, w1], the vertical position of the corresponding point q1 at the query image Iq side is located at a position N1 in an appearance pattern [N1-N2-N3...] while the corresponding point w1 on the similar image candidate Iwc side is located at a position N3. Since both the corresponding points have a large amount of deviation, they are discarded. In regard to the other corresponding point candidate [q2, w2], the vertical position of the corresponding point q1 on the query image Iq side is located at a position N2 in the appearance pattern [N1-N2-N3...], and the corresponding point w2 on the similar image candidate Iwc is similarly located at the position N2. Because both the corresponding points are close in distance, they are determined to be corresponding points. <P>COPYRIGHT: (C)2011,JPO&INPIT

Description

  The present invention relates to an image search method and system for searching an image similar to a query image from among a large number of search target images, and in particular, an image for determining similarity between images by comparing local feature amounts of each image. The present invention relates to a search method and system.

  Non-Patent Document 1 discloses SIFT (Scale Invariant Feature Transform) as one method for robustly detecting an image of a rigid body such as a building. In this SIFT, local feature amounts are extracted in advance from both the query image and the search target image, and the nearest neighbor search is executed based on the Euclidean distance L between the local feature amounts of each image. Then, local feature quantities having a short distance are used as corresponding point pairs, and finally, a search target image having many corresponding point pairs is used as a search result.

David G. Lowe, "Distinctive image features from scale-invariant keypoints" International Journal of Computer Vision, 60, 2 (2004), pp.91-110.

  In an image of a building structure, particularly a building image in which components based on straight lines such as windows, columns, or walls are uniformly combined like a building, the object difference between the buildings is small in the shape of these components. Therefore, even when images of different buildings are compared with each other, local feature amounts such as window portions show close values, and individual corresponding points may not actually show correct corresponding points. That is, in the comparison between building images, the corresponding points may be mistaken as compared with the case where images other than buildings are compared.

  Also, even if some of the corresponding points are misidentified, if the misidentification is between the same building images, a similar image including the same building as the query image can be searched from the search target based on the corresponding points There is. However, if an attempt is made to further specify the shooting position of the query image, that is, the user position based on each corresponding point from the similar images of the search results, the shooting position may not be accurately estimated due to the influence of the misidentified corresponding point. It was.

  The object of the present invention is to solve the above-mentioned problems of the prior art, and even when comparing images of a subject having many similar shaped objects, such as a building image having many identical shaped objects such as windows and columns. It is an object of the present invention to provide an image search method and system that can identify an image with high accuracy.

  In order to achieve the above object, the present invention is characterized in that the following measures are taken in an image search system that searches an image similar to a query image from a set of search target images.

  (1) Means for geometrically transforming the query image and each search target image so that the parallel component of the subject is also parallel on the image, and a local feature amount that extracts a local feature amount from the feature points of the query image and each search target image A feature amount extraction unit, a corresponding point candidate extraction unit that compares local feature amounts extracted from each feature point of the query image and the search target image, and extracts a feature point having a higher similarity as a corresponding point candidate, and is subjected to geometric transformation Means for extracting the edge component from the query image and the search target image, means for detecting the appearance pattern of the edge component from the query image and each search target image, and corresponding from the corresponding point candidates based on the appearance pattern of the edge component Corresponding point extracting means for extracting points and similar image determining means for determining a search target image similar to the query image based on the extracted corresponding points And features.

  (2) Corresponding point candidate extracting means extracts, for each feature point of the query image, a feature point having the highest N-best similarity from each search target image; A means for calculating the distribution of the plot defined by the local area scale and the difference in orientation compared to the local feature of the corresponding feature point, and a plurality of likelihoods as the corresponding points based on the distribution of the plot. Means for extracting a feature point pair as a corresponding point candidate, and each plot is given a weight value according to the similarity of local feature amounts of each feature point of the query image and each search target image It is characterized by.

  According to the present invention, the following effects are achieved.

  (1) An appearance pattern related to an edge component of the query image and each search target image is detected based on the query image and the edge component extracted from each search target image, and the query image Iq of each corresponding point candidate based on the appearance pattern Since the amount of deviation between the position on the search side and the position on the search target image Iw (k) side is calculated, and only corresponding point candidates with a small amount of deviation are regarded as true corresponding points, It becomes possible to prevent errors that are judged.

  (2) Before detecting the appearance pattern related to the edge component from the query image Iq and each search target image Iw, the parallel component of the subject is parallel to the query image Iq and each search target image Iw even on the image. Thus, the appearance pattern detection accuracy for the edge component can be improved.

  (3) When extracting the N best feature points whose local feature values are similar to each feature point of the query image Iq from each search target image, the similarity regarding the local feature values between the feature points of each image is expressed as `` Orientation Angle Since it is expressed by the distribution of plots defined by two parameters of “difference” and “scale ratio”, each plot is given a weight value depending on the Euclidean distance regarding the similarity between each local feature amount. The similarity between the feature quantities can be accurately obtained from the distribution of the plot.

It is a block diagram of the position guidance system to which the image search system of the present invention is applied. 4 is a block diagram of a corresponding point extraction unit 304. FIG. It is the figure which showed the calculation method of difference distribution typically. It is the figure which showed an example of difference distribution. It is the figure which showed an example of the extraction method of a corresponding point candidate. It is the figure which showed another example of the extraction method of a corresponding point candidate. It is a figure for demonstrating the geometric transformation by the image correction part 407. FIG. FIG. 10 is a diagram for explaining region division by a region division unit 408; It is the figure which showed the detection method of the edge appearance pattern. It is the figure which showed the evaluation method of the corresponding point candidate based on an edge appearance pattern. It is the main flow which showed operation | movement of one Embodiment of this invention. It is the flowchart which showed the procedure of the corresponding point candidate extraction process. It is the flowchart which showed the procedure of the corresponding point extraction process.

  Hereinafter, embodiments of the present invention will be described in detail with reference to the drawings. FIG. 1 is a block diagram showing the configuration of the main part of a position guidance system to which the image search system of the present invention is applied. Here, the configuration unnecessary for the description of the present invention is omitted.

  The position guidance system is mounted on a moving body such as a vehicle and shoots a street with an omnidirectional camera, generates a number of street images Iw to be searched from captured still images, An image search unit 3 that extracts a local feature quantity fw from the parallel image Iw and creates a database, and searches the database for a street image Iw similar to the landmark image (query image Iq) captured by the user terminal 2; Based on the street image Iw (best) that is most similar to the query image Iq, the current position of the user is estimated, and the position estimation unit 4 that responds to the user terminal 2 is the main component.

  In the photographing apparatus 1, the omnidirectional camera 101 includes a large number of imaging units and photographs omnidirectional still images. The position detection unit 102 measures the current position (shooting position) of the shooting timing using, for example, a plurality of GPS signals. Each captured still image is linked to the shooting position information and is stored in the image storage unit 103 one after another. If an external database constructed by photographing the street image Iw, such as street view of Google Maps, can be used, the streetscape can be obtained from such an existing external database instead of the photographing device 1. An image may be acquired.

  In the image search unit 3, the cityscape image generation unit 301 generates a panoramic image based on a large number of still images captured by the omnidirectional camera 101, and a large number of images are converted from the panoramic image according to a predetermined rule. Cut out as image Iw (k). A symbol k is an identifier of each street image Iw.

  The local feature quantity extraction unit 302 extracts local feature quantities fq (i) and fw (k, j) with reference to the feature points i and j of the query image Iq and the street image Iw (k). In this embodiment, feature points are extracted and associated using SIFT. For specifying the local region, feature point extraction based on the extreme value in the scale space by Difference of Gaussian (DoG) disclosed in Non-Patent Document 1 is used. As a result of this feature point extraction, the position of the feature point and the range (scale) of the local region are calculated, and a luminance histogram direction histogram is used as the feature descriptor of the local region.

  Such a direction histogram calculates the luminance gradient of each pixel in the feature region, weights it to generate a histogram, and rotates the feature region in that direction based on the direction of the most bin region (orientation) The direction histogram in the luminance direction is generated again, the histogram is further divided into blocks, the direction histogram is calculated in each block, and this is normalized and vectorized. In this embodiment, since the luminance direction in the block is divided into 8 directions and the target area is divided into 16 areas, one feature descriptor has 8 * 16 = 128 dimensions.

  As feature of these feature descriptors, local region features are generated, so it is resistant to occlusion, scales are determined for feature points, and the image size is unchanged, and orientation is set in the image plane based on the luminance gradient. Since it is performed, it is invariable to rotation with respect to the image plane. Furthermore, since the edge component is used, it is resistant to luminance changes. Such feature point detection is performed on all pixels of the image, but even if a certain feature point takes an extreme value, it is excluded from the feature region if it is inappropriate as a feature point.

In the present embodiment, each local feature quantity fq (i) of the query image Iq is expressed by the following equation (1). pq (i) is the position of the feature point expressed in homogeneous coordinates, op (i) is the orientation of the feature point, σq (i) is the scale at which the feature point was found, dq (i) is the feature descriptor, Symbol i is a feature point identifier of the query image Iq.

fq (i) = {pq (i), op (i), σq (i), dq (i)} (1)

Similarly, the local feature amount fw (k, j) of each street image Iw (k) is expressed by the following equation (2). pw (k, j) is the position of the feature point expressed in homogeneous coordinates, ow (k, j) is the orientation of the feature point, σw (k, j) is the scale where the feature point was found, dw (k, j ) Is a feature descriptor, and symbol j is a feature point identifier of the street image Iw (k).

fw (k, j) = {pw (k, j), ow (k, j), σw (k, j), dw (k, j)} (2)

  The local feature quantity fw (k, j) extracted in this way is stored in the image database 303 together with the street image Iw (k).

As will be described later in detail, the corresponding point extraction unit 304 compares each local feature quantity fq (i) of the query image Iq with each local feature quantity fw (k, j) of each street image Iw (k). In addition to obtaining corresponding point candidates, the similar image candidate Iwc (k) is extracted from each street image Iw (k) based on the number of corresponding point candidates, and the corresponding point candidates for each similar image candidate Iwc (k) are further extracted. Based on the geometric constraint conditions of the query image Iq and each similar image candidate Iwc (k), narrow down to true corresponding points with high likelihood.
The similar image determination unit 305 extracts each similar image candidate Iwc (k) having the largest number of corresponding points, and outputs this as a search result Iw (best). Alternatively, projective transformation is performed using the plurality of extracted corresponding points, and among these, similar image candidates Iwc (k) having the largest number of corresponding points that have been projective transformed are obtained as search results Iw (best). You may make it do.

  The position estimation unit 4 estimates the shooting position and shooting direction of the user based on the query image Iq and the street image Iw (best) that is most similar to the query image Iq, and responds to this to the user terminal 2.

  FIG. 2 is a block diagram showing a configuration of a main part of the corresponding point extraction unit 304, and the same reference numerals as those described above indicate the same or equivalent contents.

  The search range narrowing unit 401 narrows down the search range of the cityscape image Iw (k) in advance based on the current position of the user that is measured by the GPS system of the user terminal 2. The N best extraction unit 402 compares the local feature amount of each feature point of the query image Iq with the local feature amount of each feature point of all the street image Iw (k) within the search range, and the query image Iq For each feature point, the top N feature points with high local feature similarity are extracted as N best feature points from each street image Iw (k). Therefore, if m feature points are set in the query image Iq, (m × N) N best feature points are extracted from the street image Iw (k) within the search range. .

In this embodiment, for each feature point of the query image Iq, a local feature nearest neighbor search is performed for each feature point of all the street image Iw (k), and for each street image Iw (k). The similarity is calculated based on the distance between the local feature amounts. In this embodiment, the similarity between the local feature quantities fq (i) and fw (k, j) of each feature point is represented by the feature descriptors dq (i), dw (k, represented by Euclidean distance L between j).

L = | dq (i) −dw (k, j) | (3)

  The difference distribution calculation unit 403 compares the local feature amount of the N best feature point with the local feature amount of the corresponding feature point of the query image Iq, and calculates a distribution related to the difference of each local region. In the present embodiment, first, a distribution relating to a local area scale ratio (or difference) and an orientation angle difference (or angle ratio) is calculated.

Next, as shown in FIG. 3, for each N best feature point, the size ratio of the local region (scale) between the local feature amount and the corresponding local feature amount of the query image Iq and the angular difference of the orientation are calculated. Then, the distribution is performed for all the N best feature points and plotted on a virtual two-dimensional space, thereby calculating the difference distribution shown in FIG.
In the present embodiment, a weight value based on the Euclidean distance L is set for each plot. That is, a larger weight value is assigned to the N best feature point having a shorter Euclidean distance L. In the example shown in FIG. 4, the weight value of each plot is represented by the size of the plot for convenience.
Returning to FIG. 2, the corresponding point candidate extraction unit 404 extracts a plurality of corresponding point candidates having high likelihood as corresponding points based on the difference distribution.

  FIG. 5 is a diagram showing an example of a method for extracting corresponding point candidates. In this embodiment, N best feature points existing within a predetermined range centering on a position having the highest distribution density are set as corresponding point candidates. The other N best feature points are removed as noise. Note that the method of extracting corresponding point candidates based on the distribution density is not limited to the above, and as shown in an example in FIG. 6, the difference distributions are clustered, and the N best feature greatly deviates from each cluster. The points may be removed and the remaining points may be used as corresponding point candidates.

Returning to FIG. 2, the similar image candidate extraction unit 405 extracts the top M cityscape images Iw having many corresponding point candidates as similar image candidates Iwc. The normalizing unit 406 normalizes the size of each similar image candidate Iwc (k) based on the scale ratio of the local feature quantity fwc (k) of the corresponding point candidate.
The image correction unit 407 extracts a horizontal component and a vertical component from each similar image candidate Iwc (k) and the query image Iq, and obtains a shooting viewpoint direction by extracting a vanishing point based on each component. Based on the direction, geometric transformation is performed on each image so that the parallel lines of the subject are parallel on the image. Thereafter, an image in which the parallel lines of the subject are parallel even on the image by geometric transformation may be expressed as a “planarized image”.

  FIG. 7 is a diagram for explaining the geometric transformation performed on the query image Iq and each similar image candidate Iwc (k) so that the parallel lines of the subject are parallel on the image in the image correction unit 407. FIG. 5A shows an original image to be subjected to geometric transformation.

  In this embodiment, first, the Canny method [J. Canny: “A computational approach to edge detection,” IEEE Transactions on Pattern Analysis and Machine Intelligence, 8 (6), November 1986] is applied to detect edges. Next, Hough transformation [Duda, RO and PE Hart, "Use of the Hough Transformation to Detect Lines and Curves in Pictures," Comm. ACM, Vol. 15, pp. 11-15 (January , 1972).] Is executed, and the edges of the vertical component and the horizontal component are extracted as shown in FIG.

  Next, vanishing point extraction method that transforms an infinite plane into a finite deformation region [Matsufuji, Saito: "Disappearance of vanishing points in an infinite plane," IPSJ Research Report, Vol.99, No.70, pp.25- 30, 1999], the vanishing point is obtained as shown in FIG. Next, the viewpoint direction at the time of shooting is obtained based on this vanishing point, and each image is geometrically transformed based on this viewpoint direction, so that the parallel lines of the subject are imaged as shown in FIG. A parallel planarized image can be obtained even above.

  The region dividing unit 408 divides the query image Iq and each similar image candidate Iwc (k) into regions, and excludes regions that are inappropriate for image recognition from the search target in advance. FIG. 8 is a diagram showing a state of region division by the region division unit 408. In this embodiment, (1) a white line in a road region, (2) a moving object such as a car or a tree, or an object whose form changes. As described above, the region of the object that adversely affects the corresponding point detection is excluded in advance from the original image of FIG.

In this embodiment, first, region labeling is manually performed on the learning streetscape image, and each region in the image is classified into a plurality of categories such as buildings, sky, roads, cars, people, and plants. Is done. Next, learning is performed using SIFT feature points in each region, and using this learning data, the query image Iq and each similar image candidate Iw (k) are divided into a plurality of blocks, and each block Fall into one of the categories. For this learning, for example, SVM (Support Vector Machine) can be used. As shown in Fig. 5 (b), this region division can be used to detect objects that have a negative effect on detection of corresponding points, such as moving objects, objects that change according to the season, or white lines on roads that have many similar shapes. The area is excluded from the search target in advance, and only the area indicated by the square frame is set as the search target.
The edge detection unit 409 detects an edge component from the query image Iq and each similar image candidate Iwc (k). The vertical edge extraction unit 410 extracts a vertical edge from each edge component. The edge appearance pattern detection unit 411 detects the vertical edge appearance pattern of the query image Iq and each similar image candidate Iwc (k).

  The corresponding point extraction unit 412 obtains the position of each corresponding point candidate on the query image side and the position on the similar image candidate side for the image pair of the query image Iq and the similar image candidate Iwc (k) from which the corresponding point candidates are extracted. A shift amount calculation unit 412a that calculates the shift amount based on the appearance pattern of the vertical edge of each image is provided, and a true corresponding point is extracted from the corresponding point candidate based on the calculation result of the shift amount.

  FIG. 9 is a view showing an example of the vertical edge obtained by the edge detection and the vertical edge extraction. A large number of vertical edges are detected in the region where the window is arranged. Since such vertical edge appearance patterns are unique to each building, in the present embodiment, the edge appearance pattern detection unit 411 performs vertical edge counts at predetermined unit intervals for the query image Iq and each similar image candidate Iwc (k). Nx is counted, and the vertical edge appearance pattern [N1-N2-N3-N4...] Is detected.

  The corresponding point extraction unit 412 grasps the relative positional relationship between the images based on the edge appearance pattern of the query image Iq and each similar image candidate Iwc (k), and detects the shift amount of each corresponding point candidate. . Then, the corresponding point candidate with a small amount of deviation is regarded as a true corresponding point, while the corresponding point candidate with a large amount of deviation is discarded as an incorrect correspondence.

  FIG. 10 is a diagram for explaining a method of evaluating a corresponding point candidate based on the vertical edge appearance pattern and extracting a true corresponding point. FIG. 10A shows an example of a query image Iq. FIG. 6B shows an example of a similar image candidate Iwc (k) from which corresponding point candidates are extracted from the query image Iq. Here, in order to make it easy to see the drawing, the display of the vertical edge is omitted.

  In FIG. 10, two pairs of corresponding point candidates [q1, w1] and [q] between a pair of query image Iq and similar image candidate Iwc from which corresponding point candidates have been extracted (hereinafter also referred to as corresponding image pairs). q2, w2] are extracted. For one corresponding point candidate [q1, w1], the vertical position of the corresponding point q1 on the query image Iq side is the position of N1 of the appearance pattern [N1-N2-N3-N4 ...], The vertical position of the corresponding point w1 on the similar image candidate Iwc side is the position of N2 to N3 of the appearance pattern [N1-N2-N3-N4...], And both are discarded because of the large amount of deviation.

  On the other hand, for the other corresponding point candidate [q2, w2], the vertical position of the corresponding point q1 on the query image Iq side is the position of N2 of the appearance pattern [N1-N2-N3-N4. Similarly, the vertical position of the corresponding point w2 on the image candidate Iw side is also the position of N2 in the appearance pattern [N1-N2-N3-N4.

  Returning to FIG. 1, the similar image determination unit 305 extracts a similar image candidate Iwc (k) having the largest number of true corresponding points extracted as described above, and outputs this as a search result Iw (best). Alternatively, projective transformation is performed using the plurality of extracted corresponding points, and among these, the similar image candidate Iwc (k) having the largest number of corresponding points that can be projective transformed is set as the search result Iw (best). You may do it.

  The position estimation unit 4 extracts a streetscape image similar to the streetscape image Iw (best) as the search result from the database 303. In this embodiment, since the cityscape images to be searched are discretely taken along the road, the cityscape images Iw (best-1) and Iw (best + 1) on both sides of the cityscape image Iw (best) are taken. ) Of the street images Iw (best) (here, for example, Iw (best-1) is extracted from the database 303. Next, these two images Iq and Iw (best-1) are extracted. Corresponding points are detected in the same manner as described above, epipolar constraints are calculated, and the three-dimensional position of the feature point is estimated from two images Iq and Iw (best-1) whose shooting positions are known.

The point x = [uv 1] ^T projected from the point X in the unknown space onto the image Iw (best) and the point x '= [u' v '1] ^T projected onto the image Iw (best-1) The following equation (4) is established.

x ^T Fx '= 0… (4)

  This is called epipolar constraint, and F is the fundamental matrix. After the three-dimensional position of the corresponding point in the database of cityscape images is estimated based on the above relationship, the correspondence relationship is similarly calculated for the query image Iq and the cityscape image Iw (best). The user position is estimated.

  Next, the operation of the embodiment of the present invention will be described in detail with reference to a flowchart. FIG. 11 is a main flow showing the operation of the embodiment of the present invention, and mainly shows the operation of the corresponding point extraction unit 304.

  In step S1, based on the current position of the user terminal 2 received together with the query image Iq from the user terminal 2, the search range of the street image Iw is narrowed down to the vicinity of the user position by the search range narrowing unit 401. . In step S2, the N best extraction unit 402 determines that the local feature amount of each feature point of the query image Iq and the local feature amount of each feature point of each street image Iw (k) are the query image Iq and the street image Iw. For each feature point of the query image Iq, N best feature points having similar local feature amounts are extracted from each street image Iw (k). In step S3, corresponding point candidates are extracted from the N best feature points.

  FIG. 12 is a flowchart showing the procedure for extracting the corresponding point candidates, and mainly shows the operations of the different distribution calculating unit 403 and the corresponding point candidate extracting unit 404.

  In step S101, one of the cityscape images Iw (k) within the search range is selected as the current attention image. In step S102, attention is paid to one of the N best feature points extracted from the current street image Iw, and the feature points of the query image Iq corresponding to the N best feature points are acquired. In step S103, the local feature amounts of the acquired feature points are compared with each other, and the size ratio and orientation angle difference are calculated. In step S104, as described with reference to FIG. 3, the current N best feature points are plotted in a virtual two-dimensional space based on the size ratio and the orientation angle difference.

  In step S105, it is determined whether or not the plot has been completed for all N best feature points extracted from the current street image Iw. If not completed, the process returns to step S102, and the same procedure is repeated for the remaining N best feature points.

  When plotting is completed for all N best feature points and the difference distribution of FIG. 4 is completed, the process proceeds to step S106, where the maximum position of the distribution density is detected (FIG. 5) or clustering is performed (FIG. 6). In step S107, N best feature points within a predetermined range are determined as corresponding point candidates based on the distribution density, and other N best feature points are discarded as miscorresponding noise.

  Returning to FIG. 11, in step S <b> 4, the similar image candidate determination unit 405 extracts the top M cityscape images Iw (k) with many corresponding point candidates, and outputs these as similar image candidates Iwc. In step S5, a corresponding point search is performed again on the M similar image candidates Iwc, and true corresponding points are extracted.

  FIG. 13 is a flowchart showing a processing procedure for extracting true corresponding points from the M similar image candidates Iwc, and mainly shows the operation of the corresponding point extracting unit 412.

  In step S201, one of the similar image candidates Iwc is selected as the current attention image. In step S202, one of the corresponding point candidates extracted from the current attention image Iwc is selected as the current attention candidate. In step S203, the edge appearance pattern detection unit 411 separately detects a shift amount between one of the corresponding point candidates of the current image of interest Iwc and the feature point on the query image Iq side corresponding to the corresponding point candidate. Detected based on the edge appearance pattern. In step S204, it is determined whether or not the calculation result of the deviation amount is within a predetermined allowable range. If it is out of the allowable range, the process proceeds to step S205, where it is determined as an erroneous correspondence point and discarded.

  In step S206, it is determined whether or not the above processing has been completed for all corresponding point candidates extracted from the current attention image Iwc. If not completed, the process returns to step S202, and the above processing is repeated while switching the corresponding candidate point of interest. When the above processing is completed for all corresponding point candidates of the current image of interest Iwc, the process proceeds to step S207, and all corresponding point candidates remaining without being discarded are set as true corresponding points. In step S208, it is determined whether or not the above process has been completed for all similar image candidates Iwc. If not, the process returns to step S201, and the above processes are repeated while switching the similar image candidate Iwc of interest. .

  The similar image determination unit 305 extracts a similar image candidate Iwc (k) having the largest number of corresponding points, and outputs this as a search result Iw (best). Alternatively, projective transformation is performed using the plurality of extracted corresponding points, and among them, the similar image candidate Iwc (k) having the largest number of corresponding points that can be projective transformed is the search result Iw (best). Is output.

  DESCRIPTION OF SYMBOLS 1 ... Imaging device, 2 ... User terminal, 3 ... Image search part, 4 ... Position estimation part, 101 ... Omnidirectional camera, 102 ... Position detection part, 103 ... Image storage part, 301 ... Streetscape image generation part, 302 ... Local feature amount extraction unit, 303 ... database, 304 ... corresponding point extraction unit, 305 ... similar image determination unit, 401 ... search range narrowing unit, 402 ... N best extraction unit, 403 ... difference distribution calculation unit, 404 ... corresponding point Candidate extraction unit, 405 ... Similar image candidate extraction unit, 406 ... Normalization unit, 407 ... Image correction unit, 408 ... Region division unit, 409 ... Edge detection unit, 410 ... Vertical edge extraction unit, 411 ... Edge appearance pattern detection unit , 412 ... corresponding point extraction unit

Claims (5)

In an image search system that searches an image similar to a query image from a set of search target images,
A local feature amount extraction means for extracting a local feature amount from the query image and feature points of each search target image;
Means for geometrically transforming the query image and each search target image so that the parallel component of the subject is parallel on the image;
A corresponding point candidate extracting unit that compares local feature amounts extracted from the feature points of the query image and the search target image, and extracts a feature point having a higher similarity as a corresponding point candidate;
Means for detecting an appearance pattern of an edge component from the geometrically transformed query image and search target image;
For pairs of query images and search target images from which corresponding point candidates have been extracted, the amount of deviation between the position of each corresponding point candidate on the query image side and the position on the search target image side is used as the appearance pattern of the edge component of each image. Means for calculating on the basis of;
Corresponding point extraction means for extracting corresponding points from the corresponding point candidates based on the calculation result of the deviation amount;
An image search system comprising: similar image determination means for determining a search target image similar to a query image based on the extracted corresponding points. The corresponding point candidate extraction means includes:
For each feature point of the query image, means for extracting feature points with the highest N best similarity from each search target image;
Means for comparing the local feature amount of the N best feature points with the local feature amount of the corresponding feature point of the query image, and calculating a distribution of a plot defined by the difference in scale and orientation of the local region;
A plurality of feature point pairs having a high likelihood as corresponding points based on the distribution of the plot,
2. The image search system according to claim 1, wherein a weight value corresponding to the similarity of local feature amounts of each feature point of the query image and each search target image is assigned to each plot. The means for geometric transformation is:
Means for extracting a linear component from the query image and each search target image;
Means for detecting the vanishing point based on the linear component,
Means for calculating a viewpoint direction of each image based on the vanishing point,
3. The image search system according to claim 1, wherein each image is geometrically transformed based on the designated direction so that a parallel component of the subject is parallel on the image.   The image search system according to any one of claims 1 to 3, wherein the search target image is a street image in which a building is a subject. In an image search method for searching an image similar to a query image from a set of search target images,
A procedure for geometrically transforming the query image and each search target image so that the parallel component of the subject is parallel on the image,
A procedure for extracting a local feature amount from a feature point of a query image and each search target image;
A procedure for comparing local feature amounts extracted from each feature point of the query image and the search target image and extracting a feature point having a higher similarity as a corresponding point candidate;
A procedure for detecting an edge component appearance pattern from the geometrically transformed query image and search target image;
For pairs of query images and search target images from which corresponding point candidates have been extracted, the amount of deviation between the position of each corresponding point candidate on the query image side and the position on the search target image side is used as the appearance pattern of the edge component of each image. Based on the calculation procedure,
A procedure for extracting corresponding points from the corresponding point candidates based on the calculation result of the deviation amount;
And a procedure of determining a search target image similar to a query image based on the extracted corresponding points.