JP2023005682A - Dimension measurement method - Google Patents

Abstract

To provide a method capable of measuring a dimension by calculating a feature amount without being affected by the change in illumination and efficiently learning a regression model necessary for dimension measurement.SOLUTION: A dimension measurement method for a test object according to the present invention comprises: an image capturing step of capturing an image of the test object; a feature amount calculation step of extracting at least two partial region image data including a portion of each of both ends in the dimension measurement direction of the test object from the image and calculating a feature amount by inputting the partial region image data into a feature amount calculation model; a dimension calculation process of obtaining an explanatory variable by coupling the feature amount output from each partial region image data and inputting the explanatory variable to a learned regression model to calculate a distance between both ends; and a dimension outputting step of outputting the calculated distance between both ends.SELECTED DRAWING: Figure 1

Description

The present invention relates to a dimension measuring method for measuring dimensions of a test object.

Press molding using plastic working technology is a very efficient manufacturing method that enables the mass production of metal products of the same shape at low cost. In press molding, the dimensions in the plane direction, which depend on the mold, can be processed with high accuracy, but the dimensions in the vertical direction are affected by factors such as material thickness, mold temperature, amount of lubricating oil, and mechanism for vertical movement of the mold. It will fluctuate.

Therefore, generally, the dimensions of metal products after press forming are measured by sampling inspection to control the quality. However, in the sampling inspection, the discovery of defects is delayed and a large number of defective products are produced. Therefore, it is desired to measure dimensions in real time immediately after press molding to control quality.
Here, as a method of measuring the dimensions of an object by image processing, the methods of Patent Document 1 and Patent Document 2 are disclosed.

The method of Japanese Patent Laid-Open No. 2002-200303 performs a process of matching information on feature points of a measurement site shown in an image with information on feature points registered in advance, that is, template information. The above processing is performed at both ends of the measurement site, and the length of the line connecting the feature points is output as the dimension. The method of Patent Document 2 uses a region division method in an image called semantic segmentation of deep learning to extract a region to be measured, and outputs the distance connecting the endpoints of the extracted region as a dimension.

WO2019/045089 JP 2020-184295 A

In the method disclosed in Japanese Patent Laid-Open No. 2002-200010, the change in brightness of adjacent pixels is used as the feature point information, and the feature points are matched based on the similarity of the change. There was also the issue of many things.

In Patent Document 2, semantic segmentation technology, which has been rapidly progressing in recent years, is used. In addition, in order to perform semantic segmentation with high accuracy, there is also a problem that a large amount of learning for all pixel numbers in the segmented image is indispensable.

Accordingly, the present invention provides a method that enables dimension measurement by calculating feature quantities without being affected by changes in illumination, and that enables efficient learning of a regression model necessary for dimension measurement.

A dimension measuring method according to the present invention is a method for measuring dimensions of a test object, and comprises an image capturing step of capturing an image of the test object; a feature quantity calculation step of extracting at least two partial region image data including a part, inputting the partial region image data into a feature quantity calculation model, and calculating a feature quantity; A size calculation step of combining the output feature values to form an explanatory variable, inputting the explanatory variable into a learned regression model, and calculating the distance between the two ends, and the calculated distance between the two ends. has a dimension output step of outputting

Further, a plurality of learning partial area image data obtained from the image while changing the position for extracting the partial area image data is input to the feature amount calculation model to calculate a learning feature amount, and a plurality of learning feature amounts are calculated. combining the learning feature values to obtain a learning explanatory variable, calculating a learning objective variable according to the number of pixels by which the position of the partial area image data is moved, and calculating the learning explanatory variable and the learning objective variable from the learning explanatory variable and the learning objective variable; It is preferable to further include a learning step of learning a regression model to obtain the learned regression model.

Also, the feature quantity calculation model is preferably a trained convolutional neural network.

Also, the regression model is preferably ensemble machine learning based on a decision tree.

According to the present invention, the regression model can be learned efficiently, and the feature amount can be calculated to measure dimensions without being affected by changes in illumination.

It is an example of the configuration of a computer for measuring dimensions. It is an example of a product manufactured by press molding. It is an example of a state in which a product is held by a jig. It is an example of a captured image. It is an example of definition of a rectangular area and a measurement site. It is an example of the flowchart which shows the procedure of dimension measurement. It is an example of the flowchart which shows the learning procedure of a regression model. It is an example of a moved rectangular area. It is an example of combined features and dimensions for learning. It is an example of selected effective feature amounts and dimensions for learning.

Hereinafter, the dimension measuring method of the present invention will be described in detail using embodiments. In this embodiment, an example of measuring two dimensions of a metal product formed by press molding will be described.

FIG. 1 is an example of the configuration of a computer for measuring dimensions. In the computer 11, an arithmetic device 31, a primary storage device 32, and a secondary storage device 33 are connected by a bus 35, and data and control signals are transmitted and received via the bus 35. FIG. The computer 11 also includes input/output units 21, 22, 23, 24, and 25 based on international standards such as USB, which are also connected by a bus 35 for transmitting and receiving data and control signals. Connected to the input/output unit are a camera 12 capable of capturing an image as an external device, a display 41 for a person to operate the computer, a keyboard 42, a mouse 43, and the like. In addition, it is connected to a local area network via a network interface card 34 to transmit and receive data to and from an external database system or the like. The dimension measuring method of the present invention is stored as a computer program in the secondary storage device 33 of the computer 11. When measuring the dimension, the computer program is read into the primary storage device 32 via the bus 35 and executed. be done. Also, an image for measuring the dimensions is taken by the camera 12 and taken in from the input/output unit 21 . The timing of image pickup is sent as a serial signal from a programmable logic controller (PLC) that controls the operation of the press molding processing apparatus, and the serial signal is taken in from the input/output unit 22 .

FIG. 2 is an example of a metal product 50 formed by press molding. 2(a) is a side view of the manufactured metal product 50, FIG. 2(b) is a top view, and FIG. 2(c) is a bottom view. This metal product 50 has a shape in which a flange portion 61 is attached to the upper portion of a cylindrical portion 62 . The critical dimensions are the thickness of the flange portion 61 and the height of the cylindrical portion 62, and in this embodiment, these two dimensions are measured.

FIG. 3 shows a state in which the metal product 50 shown in FIG. 2 is sandwiched between jigs 71 and 72 at the unloading section of the press molding machine. 3(a) is a side view of the metal product 50 and jigs 71 and 72, that is, the side to be carried out, FIG. 3(b) is a top view, and FIG. 3(c) is a bottom view. FIG. 3(d) is a sectional view of FIG. 3(a). As shown in FIG. 3(d), the jigs 71 and 72 are provided with grooves corresponding to the flange portion and the cylindrical portion having different diameters. 50 is sandwiched and held.

FIG. 4 is an example of an image captured by the camera 12 in the state of FIG. 3(a). Here, in the present embodiment, the metal product 50 is imaged using the camera 12 from a direction perpendicular to the thickness of the flange portion 61 and the height of the cylindrical portion 62 of the metal product 50. There is no limitation, and for example, an image may be captured from an oblique direction.

The coordinate system of the image 80 has a vertical axis 81 and a horizontal axis 82, with the origin at the upper left vertex of the image. Coordinates in the image are represented by the number of pixels from the origin. For example, coordinates (50, 60) mean the 50th pixel below the origin and the 60th pixel to the right. The image 80 shows the flange portion 61 of the metal product, the cylindrical portion 62, the jigs 71 and 72, and the background.

Rectangular area data 91, 92, and 93 indicate partial area image data (hereinafter referred to as rectangular area data) used for dimension measurement in this embodiment. Here, the "partial area" of the partial area image data indicates a partial area including a part of the edge of the object, not the entire edge. The rectangular area data 91, 92, 93 include portions of both ends of the flange portion 61 in the thickness direction and the cylindrical portion 62 in the height direction, which are objects of dimension measurement in this embodiment. That is, the rectangular area data 91 shows a part of one end side of the flange part 61 necessary for measuring the thickness of the flange part 61, and the rectangular area data 92 shows the thickness of the flange part 61. In addition, part of the edge of the boundary between the flange portion 61 and the cylindrical portion 62, which is necessary for measuring the height of the cylindrical portion 62, is shown. Furthermore, a part of the other end side of the cylindrical portion 61 necessary for measuring the height of the cylindrical portion 62 is shown in the rectangular area 93 .

The thickness of the flange portion 61 and the height of the cylindrical portion 62 can be measured in real time by inputting each pixel value in the rectangular area data into a feature amount calculation model, which will be described later. The sizes of the rectangular area data 91, 92, 93 are determined by variations in the positions of the metal products held by the jig. In the present embodiment, one pixel corresponds to 10 μm×10 μm when converted from the resolution of the entire image and the size of the metal product or jig shown. For example, if the positional variation of the metal product 50 is within 2 mm due to the jig that holds the metal product 50, the size of the rectangular area data 91, 92, 93 can be determined to be 200 pixels×200 pixels. The larger the size of the rectangular area data 91, 92, 93, the longer the calculation time for dimension measurement, so it is important to set the size appropriately.

FIG. 5 is an example of the definition text file 130 used in this embodiment. The definition text file 130 is stored in the secondary storage device 33, and the definition text file 130 is read from the secondary storage device 33 to the primary storage device 32 when the dimension measurement process or the regression model learning process is executed. use it. The definition text file 130 of this embodiment describes two types of data: rectangular area data and measurement objects.

The information of the rectangular area data includes an image ID that identifies an image captured by the camera 12, a rectangle ID that identifies a rectangle in the image ID, a vertical start coordinate of the rectangle, a vertical end coordinate of the rectangle, and a width of the rectangle. There are the start coordinate of the direction and the end coordinate of the horizontal direction of the rectangle. is the rectangle ID, the third column defines the vertical start coordinates, the fourth column defines the vertical end coordinates, the fifth column defines the horizontal start coordinates, and the sixth column defines the horizontal end coordinates.
Here, information on rectangular area data 91 is described on the first line immediately below the description of [window], information on rectangular area data 92 is described on the second line, and information on rectangular area data 93 is described on the third line. That is, in the case of the rectangular area data 91, the image ID is 0, the rectangle ID is 0, the coordinates of the upper left vertex are (338, 968), and the coordinates of the lower right vertex are (562, 1192). there is According to this definition, the rectangular area data 91, 92, and 93 have the same image ID and horizontal coordinates, but different rectangle IDs and vertical coordinates. Rectangular area data 91, 92 and 93 have the same size and information of 224 pixels×224 pixels.
Rectangular area data 91, 92, and 93 drawn in FIG. 4 are results drawn based on this information. That is, the rectangular area data 91 is the position of the edge of the flange portion 61 where the vicinity of the boundary between the background and the flange portion 61 is captured, and the rectangular area data 92 is the location where the flange portion 61 and the positions of the ends of the cylindrical portion 62, and the rectangular area data 93 defines the positions of the ends of the cylindrical portion 62 where the vicinity of the boundary between the cylindrical portion 62 and the background appears.

Next, the measurement object is described immediately below the description of [objective] in the definition 130 text file. Information about the object to be measured includes the name of the part whose dimensions are to be measured, the image ID, the rectangle ID, and the like. is the image ID including the first rectangle, the third column is the first rectangle ID, and the fourth column is the image ID including the second rectangle. The fifth column describes the second rectangle ID. Here, in the case of this embodiment, the definition text file 130 describes the name of the part whose dimensions are to be measured, and the image ID and rectangle ID used for the measurement. That is, the part named "FLANGE" is a rectangle with image ID 0 and rectangle ID 0, that is, rectangular region data 91, and a rectangle with image ID 0 and rectangle ID 1, that is, rectangular region data 91. 92 is used to define the dimensions to be calculated. The part named "LEG" is a rectangle with image ID 0 and rectangle ID 1, that is, rectangular region data 92, and a rectangle with image ID 0 and rectangle ID 2, that is, rectangular region data 93. is defined to be used to calculate the dimensions. In this way, in this embodiment, only one image with the image ID of 0 is used as an example. If the image does not fit, it can be handled even if the two ends are different images.

FIG. 6 is an example of a flowchart showing the procedure for measuring dimensions, and the dimension measuring method according to this embodiment will be described along this flowchart.
(preparatory process)
First, in step 121 , the rectangular area data and the definition text file 130 of the measurement site are read from the secondary storage device 33 to the primary storage device 32 .
In step 122 , the definitions of effective feature quantities, which will be described later, are read from the secondary storage device 33 to the primary storage device 32 .
At step 123 , the feature quantity calculation model and the regression model are read from the secondary storage device 33 to the primary storage device 32 . A feature amount calculation model is a machine learning model that extracts feature amounts summarizing an image when an image is input. As a feature amount calculation model, trained VGG16, MobileNet, etc., which are widely open to the public on the Internet, can be used. VGG16 and MobileNet are convolutional neural networks that can classify images containing cats, dogs, etc., but the convolutional layers, excluding the fully connected layers of the neural network that finally perform classification processing, extract feature values from images. It can be used as a feature value calculation model for In this embodiment, VGG16 is used.
On the other hand, the regression model is a machine learning model that calculates dimensions as objective variables when inputting feature amounts, that is, explanatory variables. For example, well-known regression models include multiple regression analysis, random forest, support vector machine, neural network, and gradient boosting, and any method may be used. However, the inventors have determined that gradient boosting can compute the most accurate dimensions. We also confirm that random forests can compute dimensions with almost as much accuracy as gradient boosting. Therefore, since both random forest and gradient boosting are methods of ensemble machine learning based on decision trees, it is preferable to use ensemble machine learning based on decision trees for the regression model of the present invention. Gradient boosting was used in this embodiment.

(Image capturing process)
In the image capturing step, an image of a product, which is an object to be inspected, is captured. That is, in the present embodiment, an image including the thickness of the flange portion 61 and the height of the cylindrical portion 62 of the metal product 50 is captured. In this embodiment,
At step 124 , image 80 is captured by camera 12 and read into primary storage device 32 . An image 80 shows a serial signal from the programmable logic controller (PLC) of the press molding machine to the input/output unit 22 when the metal product 50 is held by the jigs 71 and 72 at the unloading section of the press molding machine. The image is captured at the timing of reception.

(Feature amount calculation process)
In the feature amount calculation step, at least two or more partial area image data are extracted from the image including both ends of the object in the dimension measurement direction and a part of each, and the partial area image data are input to the feature amount calculation model. In step 125, the rectangular area data in the image is input to the feature amount calculation model, that is, the learned convolution layer of the VGG 16, and the feature amount is calculated. In this embodiment, since rectangular area data 91, 92, and 93 are defined, three types of feature amounts are calculated. Since the size of each rectangle is 224 pixels×224 pixels, the VGG 16 outputs 25088 feature amounts per rectangular area data.
In step 126, according to the definition text file 130, the feature amount calculated by putting the data of the rectangular area data 91 into the feature amount calculation model and the feature amount calculated by putting the data of the rectangular area data 92 into the feature amount calculation model and select valid features. The method of combining features and the method of selecting effective features will be explained together when explaining the learning method of the regression model. Further, the feature amount calculated by putting the data of the rectangular area data 92 into the feature amount calculation model and the feature amount calculated by putting the data of the rectangular area data 93 into the feature amount calculation model are combined to obtain an effective feature amount. to select.

(Dimension calculation process)
In the dimension calculation step, feature amounts output from each partial area image data are combined to form an explanatory variable, and the explanatory variable is input to a learned regression model to calculate the distance between the ends.
In step 127, according to the definition text file 130, the selected effective feature quantities are input to the regression model, and two dimensions of "FLANGE" and "LEG" are calculated.

(Dimension output process)
The output dimensions step outputs the distance between the calculated edges.
At step 128 , the calculated two dimensions are output to the secondary storage device 33 .

In addition, it is preferable that the regression model learn the regression model in order to improve the dimension measurement accuracy. Therefore, a step of learning a regression model (learning step) can be added before the dimension calculating step. Here, in the learning step, a plurality of learning partial area image data obtained while changing the position for extracting the partial area image data from the image is input to the feature amount calculation model to calculate the learning feature amount, Multiple learning feature values are combined to form a learning explanatory variable, a learning objective variable is calculated according to the number of pixels by which the position of the partial area image data is moved, and a regression model is derived from the learning explanatory variable and the learning objective variable. is learned and used as a trained regression model. In particular, the method of acquiring the partial area image data from the image while changing the extraction position moves the partial area image data by a predetermined number of pixels in the direction perpendicular to the dimension measurement direction or in the dimension measurement direction. It was performed by extracting at the position where
In addition, since the regression model is learned using a plurality of images, the data of various lighting conditions are mixed, and the dimension measurement robust to the lighting conditions becomes possible.

FIG. 7 is an example of a flowchart showing the learning procedure of the regression model in this embodiment. First, in step 140, the rectangular region data and the measurement site definition text file 130 are read from the secondary storage device 33 to the primary storage device 32, as in step 121 of FIG.
From step 141 to step 148, loop processing is repeated for the number of products actually measured for regression model learning.
In step 142 , the image and the measured dimension values are read from the secondary storage device 33 to the primary storage device 32 .
Between steps 143 and 147 is a loop process of repeatedly acquiring learning partial area image data (hereinafter referred to as learning rectangular area data) while moving the position of the rectangular area data.
In step 144, the learning rectangular area data whose position has been moved is input to the feature amount calculation model to calculate the learning feature amount. The same model as that used in FIG. 6 was used as the feature amount calculation model. That is, the feature amount calculation model is a convolution layer of VGG16.
In step 145, the feature values for learning are combined according to the definition text file 130 to obtain explanatory variables for learning.
At step 146, a learning objective variable is calculated according to the number of pixels by which the position of the learning rectangular area data has been moved. That is, the learning dimensions are calculated from the direction in which the position of the learning rectangular area data is moved, the number of pixels, and the actually measured dimensions. Specifically, when the two rectangles move away from each other, the learning dimension is shortened by the following formula (Equation 1).

Also, when two rectangles are close to each other, the dimension for learning is lengthened by the following formula (Equation 2).

In step 149, the zero ratio of the feature quantity is calculated for each column from the list of the feature quantity and the learning dimensions generated by combining the positions of various objects and various learning rectangular area data. % or less is selected as an effective learning feature amount, and the effective learning feature amount is defined and written to the secondary storage device 33 .
In step 150 , a regression model, that is, various parameters of gradient boosting are learned using a list of valid learning features and learning dimensions, and the regression model is written to the secondary storage device 33 .

FIG. 8 is an example of a diagram for explaining how to move the rectangular area data between steps 143 to 147 in the flow chart showing the regression model learning procedure. The rectangular area data 101 is a rectangle obtained by moving the rectangular area data 91 upward by several pixels. If the rectangular area data 102 does not move the rectangular area data 92, the rectangular area data 101 and the rectangular area data 102 are separated from the rectangular area data 91 and the rectangular area data 92 by the number of pixels moved. Since the rectangular area data 101 has moved upward from the rectangular area data 91, the flange portion 61 appearing within the rectangle appears to have shifted downward. Similarly, the rectangular area data 103 is a rectangle obtained by moving the rectangular area data 93 upward by several pixels. If the rectangular area data 102 does not move the rectangular area data 92, the rectangular area data 1022 and the rectangular area data 103 are closer than the rectangular area data 92 and the rectangular area data 93 by the number of pixels moved. Since the rectangular area data 103 has moved upward from the rectangular area data 93, the columnar portion 62 appearing within the rectangle appears to have shifted downward.
The change in the part reflected in the rectangle in this way gives the regression model an illusion and allows it to learn different dimensions. In this embodiment, an example in which the rectangular area data 92 is not moved has been described, but it is possible to move a plurality of rectangles in various ways and create data for learning various dimensions according to the distances between the rectangles.

Rectangular area data 111, 112, and 113 are rectangles obtained by moving the rectangular area data 91, 92, and 93 by the same number of pixels in the direction perpendicular to the dimension measurement direction. By moving the positions of the rectangles while keeping the distance between the rectangles constant, it is possible to increase the list of learning features and dimensions for learning the regression model. For example, it can learn various conditions, such as when a metal product has micro-scratches, or when the lighting is subtly different depending on the position. Furthermore, by moving the rectangular area data 111, 112, 113 in the dimension measurement direction, it is possible to increase the amount of learning by combining various rectangular positions.

FIG. 9 is an example of an excerpt from the list of learning feature values and dimensions before effective learning feature values are selected in step 149 . As described above, when rectangular data of 224 pixels×224 pixels is input to the convolution layer of the VGG 16, 25088 learning feature values are calculated. Columns F0 to F25087 are learning feature amounts calculated by moving the rectangular area data 91, columns F25088 to F50175 are learning feature amounts calculated by moving the rectangular area data 92, and column FLANGE is a rectangle. It is the learning dimension calculated by Equation 1 or Equation 2 by moving the area data 91 and the rectangular area data 92 . In this way, the learning feature amount calculated by moving the rectangular area data 91, which is the learning feature amount before selecting the effective learning feature amount, and the learning feature amount calculated by moving the rectangular area data 92. The features will be combined by listing them together in the list shown in FIG.

Columns F0 and F2 are arranged vertically with only 0.000. Such a phenomenon occurs in VGG16 because ReLu is used as the activation function. A feature value of 0.000 across all rows is useless even if it is input to the regression model, and it is unnecessary because it only prolongs the calculation time. , are deleted as unnecessary learning features, and the remaining learning features are defined as "effective learning features".

FIG. 10 is an example of an excerpt from a list of effective learning features and dimensions in step 149 . As compared with FIG. 9, columns F0 and F2 are deleted. Column F12 and column 50172 seem to have only 0.000 at first glance, but there are values other than 0.000 in rows that are omitted and hidden, and they remain effective feature values for learning. At step 150, the regression model is trained using this list as training data. Also, when measuring the dimensions, the columns selected here are input to the regression model as effective feature quantities to calculate the dimensions.

According to the present invention, the site to be measured of the subject is imaged with a camera, a plurality of rectangular area data are set from the imaged image, and each rectangular area data is input to a deep learning convolutional neural network, You can measure the dimensions of the part you want to measure. In addition, by moving the position of the set rectangular area data, the regression model can be learned efficiently with only a small number of images obtained by imaging a small number of subjects. The learning method of the present invention is different from the conventional data augmentation method in which images are rotated or resized to increase the number of images. , can vary in dimension, which is the objective variable. Therefore, a regression model can be generated that can measure various dimensions without intentionally making small or large products.

Although the present invention has been described using the above embodiments, the present invention is not limited to the above embodiments.

11 Computer 12 Cameras 21, 22, 23, 24, 25 Input/output unit 31 Arithmetic device 32 Primary storage device 33 Secondary storage device 34 Network interface card 41 Display 42 Keyboard 43 Mouse 50 Metal product 61 Flange portion 62 Cylindrical portion 71, 72 jig 80 image 81 vertical axis,
82 horizontal axis 91, 92, 93 rectangular area data 101, 102, 103, 111, 112, 113 rectangular area data 130 definition text file

Claims (4)

A method for measuring dimensions of an object,
an image capturing step of capturing an image of the subject;
At least two partial area image data are extracted from the image, including both ends of the object in the dimension measurement direction and a part of each, and the partial area image data are input to a feature amount calculation model to obtain a feature amount. A feature quantity calculation step of calculating
a size calculation step of combining the feature amounts output from the respective partial area image data to form an explanatory variable, inputting the explanatory variable into a learned regression model, and calculating a distance between the two ends;
and a dimension output step of outputting the calculated distance between the two ends. A plurality of learning partial area image data obtained from the image while changing the position for extracting the partial area image data is input to the feature amount calculation model to calculate a learning feature amount, and a plurality of learning feature amounts are calculated. Combine the features for learning as explanatory variables for learning,
A learning step of calculating a learning objective variable corresponding to the number of pixels by which the position of the partial area image data is moved, and learning a regression model from the learning explanatory variable and the learning objective variable to obtain the learned regression model. The dimension measuring method according to claim 1, further comprising: 3. The dimension measuring method according to claim 1, wherein the feature amount calculation model is a trained convolutional neural network. 4. The dimension measuring method according to at least one of claims 1 to 3, wherein the regression model is ensemble machine learning based on a decision tree.

Images