WO2020183837A1

WO2020183837A1 - Counting system, counting device, machine learning device, counting method, component arrangement method, and program

Info

Publication number: WO2020183837A1
Application number: PCT/JP2019/049274
Authority: WO
Inventors: 祐也島▲崎▼
Original assignee: 三菱電機株式会社
Priority date: 2019-03-08
Filing date: 2019-12-17
Publication date: 2020-09-17
Also published as: JPWO2020183837A1; JP7134331B2; CN113518998A; CN113518998B

Abstract

An image acquisition unit (11) of this counting device (1) acquires a captured image from an image-capturing device (2) that captures a stationary component. A binarization unit (12) binarizes the captured image acquired by the image acquisition unit (11). An area calculation unit (14) calculates the area of a group into which the pixel distribution of the captured image binarized by the binarization unit (12) is classified. A number-of-components calculation unit (15) calculates the number of components for each group from the area of the group calculated by the area calculation unit (14), sums the number of components for each group, and generates number-of-components information indicating the total number of components captured in the captured image. An information output unit (16) outputs the number-of-components information generated by the number-of-components calculation unit (15).

Description

Counting system, counting device, machine learning device, counting method, component placement method, and program

This disclosure relates to a counting system, a counting device, a machine learning device, a counting method, a component placement method, and a program.

As a technique for counting articles, there is a technique for detecting and counting articles from photographed images by comparing a photographed image of the articles with a reference image of the articles. However, in this technique, it is necessary to store a reference image of the article. Therefore, as an article counting device that does not need to store a reference image of an article, Patent Document 1 describes an article that is continuously transferred, and binarizes the photographed image to obtain the area of the article. A counting device for counting articles is disclosed.

JP-A-9-124142

The counting device described in Patent Document 1 photographs articles continuously transferred by a conveyor and measures the number of articles existing between the dividing lines in which the articles do not exist. Therefore, the articles on the conveyor must be arranged so that there is a dividing line. In addition, the number of articles that can be transferred per unit time is limited, and counting takes time. Even if it is configured to count articles in a predetermined measurement range, there is a problem that accurate counting cannot be performed when articles exist across the boundary of the measurement range.

The present disclosure has been made in view of the above-mentioned problems, and it is not necessary to store a reference image of an article in a counting device that counts an article from a photographed image, and the time is shorter. The purpose is to enable accurate counting.

In order to achieve the above object, the counting system according to the present disclosure includes a photographing device for photographing a stationary article and a counting device for counting the articles photographed by the photographing device. The counting device has an image acquisition unit, a binarization unit, an area calculation unit, an article number calculation unit, and an information output unit. The image acquisition unit acquires a captured image from the photographing device. The binarization unit binarizes the captured image acquired by the image acquisition unit. The area calculation unit calculates the area of a group that classifies the pixel distribution of the captured image that has been binarized by the binarization unit. The article number calculation unit calculates the number of articles for each group from the area of the group calculated by the area calculation unit, totals the number of articles for each group, and indicates the total number of articles shown in the photographed image. To generate. The information output unit outputs the number of articles information generated by the number of articles calculation unit.

According to the present disclosure, a counting device that counts articles from captured images binarizes captured images of stationary articles and classifies the pixel distribution of the binarized captured images based on the area of the group. By counting, it is not necessary to store a reference image of the article, and accurate counting can be performed in a shorter time.

The figure which shows the structure of the counting system which concerns on Embodiment 1. The figure which shows the functional configuration example of the counting apparatus which concerns on Embodiment 1. The figure which shows the example of the non-countable information which concerns on Embodiment 1. The figure which shows the example of the part number information which concerns on Embodiment 1. A flowchart showing a counting process executed by the counting device according to the first embodiment. The figure which shows the functional configuration example of the counting apparatus which concerns on Embodiment 2. The figure which shows the example of the group image information which concerns on Embodiment 1. A flowchart showing a counting process executed by the counting device according to the second embodiment. The figure which shows the structure of the counting system which concerns on Embodiment 3. The figure which shows the functional configuration example of the counting apparatus which concerns on Embodiment 3. A flowchart showing a counting process executed by the counting device according to the third embodiment. The figure which shows the structure of the counting system which concerns on Embodiment 4. The figure which shows the functional configuration example of the counting apparatus which concerns on Embodiment 4. The figure which shows the example of the part number information which concerns on Embodiment 4. A flowchart showing a counting process executed by the counting device according to the fourth embodiment. The figure which shows the structure of the counting system which concerns on Embodiment 5. The figure which shows the functional structure example of the counting apparatus which concerns on Embodiment 5. A flowchart showing a counting process executed by the counting device according to the fifth embodiment. The figure which shows the structure of the counting system which concerns on Embodiment 6. The figure which shows the functional configuration example of the counting device and the machine learning device which concerns on Embodiment 6. The figure which shows the example of the neural network generated by the machine learning apparatus which concerns on Embodiment 6. The figure which shows another example of the structure of the counting system which concerns on Embodiment 6. The figure which shows another example of the structure of the counting system which concerns on Embodiment 7. A flowchart showing a counting process executed by the component arranging device according to the seventh embodiment. The figure which shows an example of the hardware configuration of the counting apparatus which concerns on Embodiments 1 to 7.

The counting system, counting device, machine learning device, counting method, component placement method, and program according to the present embodiment will be described in detail below with reference to the drawings. The same or corresponding parts are designated by the same reference numerals in the drawings. This embodiment is an example of a counting system that counts parts.

(Embodiment 1)
As shown in FIG. 1, the counting system 100 includes a counting device 1 for counting parts P, a photographing device 2 for photographing a photographing range C, and a user terminal 3 used by a user. The counting device 1, the photographing device 2, and the user terminal 3 are connected by wire or wirelessly.

When the user puts the component P into the photographing range C, the photographing device 2 photographs the photographing range C while the component P is stationary, and transmits the photographed image to the counting device 1. The timing at which the photographing device 2 shoots may be the timing at which the user inputs a shooting instruction to the photographing device 2, or may be the timing at which the photographing device 2 detects that the component P to be counted has been put into the shooting range C. The counting device 1 binarizes the captured image received from the photographing device 2. The counting device 1 classifies the pixel distribution of the binarized captured image into a group. The counting device 1 calculates and totals the number of parts P for each group from the area of the group, and calculates the total number of parts P shown in the captured image. The counting device 1 generates component number information indicating the calculated total number of components P and transmits it to the user terminal 3. The number of parts information is an example of the number of articles information.

When there is a non-countable group whose number of parts P cannot be calculated from the area of the group, the counting device 1 generates non-countable information indicating the non-countable group and transmits it to the user terminal 3. The user terminal 3 displays the non-countable information received from the counting device 1. The user visually counts the number of parts P of the uncountable group indicated by the uncountable information displayed on the user terminal 3. The user inputs numerical information indicating the number of parts P of the uncountable group to the user terminal 3. The user terminal 3 transmits the input numerical information to the counting device 1. When the counting device 1 receives the numerical information from the user terminal 3, the counting device 1 adopts the number of the parts P of the group that cannot be counted indicated by the numerical information, and calculates the total number of the parts P shown in the captured image. When the parts P do not overlap as in the case where the shape of the parts P is a sphere, a group that cannot be counted does not occur. The counting device 1 generates component number information indicating the calculated total number of components P and transmits it to the user terminal 3. The user terminal 3 outputs the component number information received from the counting device 1 by a method such as screen display or voice output. As a result, the user can grasp the number of inserted parts P.

Here, the functional configuration of the counting device 1 according to the first embodiment will be described with reference to FIG. As shown in FIG. 2, the counting device 1 has, as a functional configuration, an image acquisition unit 11 that receives a captured image from the photographing device 2, a binarization unit 12 that binarizes the captured image, and a binarized photographing. A reduction processing unit 13 that reduces the image, an area calculation unit 14 that calculates the area of the group that classifies the pixel distribution of the reduced captured image, and a component number calculation unit 15 that calculates the total number of parts P from the area of the group. It includes an information output unit 16 that outputs information to the user terminal 3.

The image acquisition unit 11 receives the captured image from the photographing device 2. The captured image is digital data. The binarization unit 12 converts the captured image received by the image acquisition unit 11 into grayscale, and removes noise by a smoothing filter such as a Gaussian filter or a moving average filter. The binarization unit 12 calculates a threshold value for a grayscale photographed image from which noise has been removed. As a method of calculating the threshold value of binarization, there are a method of calculating the threshold value while performing iterative calculation by two-class clustering, a discriminant analysis method of calculating the threshold value at which the degree of separation is maximized, and the like. According to the discriminant analysis method, since it is not necessary to perform the iterative calculation, the threshold value for binarization can be calculated at high speed. The binarization unit 12 binarizes the grayscale photographed image from which noise has been removed by using the calculated threshold value.

The reduction processing unit 13 reduces the captured image binarized by the binarization unit 12. When the component P has a fine structure such as a lead wire portion of an electronic component, the fine structure of the component P shown in the captured image can be removed by reducing the binarized captured image. .. As a result, erroneous detection of the contour of the component P can be suppressed. If the component P does not have a fine structure, the counting device 1 does not have to include the reduction processing unit 13.

The area calculation unit 14 classifies the pixel distribution of the captured image reduced by the reduction processing unit 13 into groups, and calculates the number of pixels of each group as the area. As a method of classifying into a group, for example, there is a K-means method. When the K-means method is used, the area calculation unit 14 counts the number of edges in the binarized captured image and sets it as the value of the group number k. For counting the number of edges, for example, the findcons function of OpenCV (OpenSource Computer Vision Library) is used. When the counting device 1 does not include the reduction processing unit 13, the area calculation unit 14 classifies the pixel distribution of the captured image binarized by the binarization unit 12 into groups, and sets the number of pixels of each group as the area. Calculate as.

The part number calculation unit 15 determines that the area of the smallest group excluding noise among the areas of each group calculated by the area calculation unit 14 is the area of one part P, and uses it as the reference area. As a method of removing noise, for example, there is the following method. A plurality of experiments are performed in advance using the counting device 1 to calculate the lower and upper limits of the area that the group of parts P can take. Alternatively, the component number calculation unit 15 arranges the area values of each group calculated by the area calculation unit 14 in ascending or descending order. The part number calculation unit 15 compares the value of the area of each group from the center to the smaller one, and if there is a difference of more than a predetermined value between adjacent values, the larger value is set as the lower limit. The number of parts calculation unit 15 compares the value of the area of each group from the center to the larger one, and if there is a difference of more than a predetermined value between adjacent values, the smaller value is set as the upper limit. The component number calculation unit 15 removes a group having an area smaller than the lower limit and a group having an area larger than the upper limit as noise.

The number of parts calculation unit 15 calculates the number of parts P of the group of the area within the first range including 1 times the reference area as one. The part number calculation unit 15 calculates the number of parts P of the group of the area within the second range including twice the reference area as two. The part number calculation unit 15 calculates the number of parts P of the group of the area within the third range including three times the reference area as three. In this way, the component number calculation unit 15 calculates the number of components P of the group of the area within the Nth range including N times the reference area as N, and adds them. N is a positive integer. The Nth range is the minimum range that absorbs errors due to the shape and orientation of the component P. The component number calculation unit 15 determines that the group whose area is out of the Nth range has overlapping components P, and cannot count the group. The part number calculation unit 15 generates non-countable information indicating a non-countable group. The information output unit 16 transmits the non-countable information generated by the component number calculation unit 15 to the user terminal 3.

The component number calculation unit 15 receives numerical information indicating the number of components P in a group that cannot be counted from the user terminal 3. The part number calculation unit 15 adopts the number of parts P of the uncountable group indicated by the numerical information, and calculates the total number of parts P shown in the photographed image. The component number calculation unit 15 generates component number information indicating the total number of calculated components P. The number of parts calculation unit 15 is an example of the number of articles calculation unit. The information output unit 16 transmits the component number information generated by the component number calculation unit 15 to the user terminal 3.

Next, the non-countable information and the number of parts information will be described with reference to FIG. FIG. 3A shows an example of non-countable information. In the example of FIG. 3A, images of all populations except noise are displayed. The group that the number of parts calculation unit 15 has determined to be uncountable is surrounded by a square frame, and the characters "uncountable" are displayed. The characters "Uncountable" are also displayed in the "Number of parts" column. The non-countable information is not limited to the example of FIG. 3A, and for example, the group that the part number calculation unit 15 has made uncountable may be displayed in a different color from other groups, or may be blinked. There is no need to have a column for "number of parts". Alternatively, the non-countable information may be information indicating the coordinates of the non-countable group on the photographing range C.

When the non-countable information shown in FIG. 3A is displayed on the user terminal 3, the user visually counts the number of parts P in the group surrounded by the square frame, and obtains numerical information indicating the number "5" of the parts P. Input to the user terminal 3. The user terminal 3 transmits the input numerical information to the counting device 1. The component number calculation unit 15 of the counting device 1 adds the number of components P "5" indicated by the numerical information to the total "46" obtained by adding the number of components P for each group calculated from the area, and adds the number of components P indicated by the numerical information to the captured image. Calculate the total number "51" of the parts P in the picture. The component number calculation unit 15 generates component number information indicating the total number of components P “51” as shown in FIG. 3B. The information output unit 16 transmits the component number information generated by the component number calculation unit 15 to the user terminal 3. The user terminal 3 displays the received component number information. As a result, the user knows that the number of input parts P is 51.

In the example of FIG. 3B, images of all groups excluding noise are displayed. A group in which the number of parts P is other than one is surrounded by a square frame, and the number is displayed. The number "5" of the parts P indicated by the numerical information input by the user is also reflected in the group displayed as the group that cannot be counted in FIG. 3A. In the "Number of parts" column, "51", which is the total number of parts P, is displayed. The number of parts information is not limited to the example of FIG. 3B, and for example, only the total number of parts P may be displayed, or the previous counting result, the cumulative number of parts P counted in a predetermined period, and the like may be displayed. May be good. Alternatively, it may be voice data notifying the total number of parts P.

Note that the information output unit 16 of the counting device 1 may output the non-counting information and the number of parts information by a method such as screen display or voice output without transmitting the information to the user terminal 3. In this case, the counting system 100 does not have to include the user terminal 3. The user visually counts the number of parts P of the uncountable group indicated by the uncountable information output by the information output unit 16, and inputs numerical information indicating the number of parts P of the uncountable group to the part number calculation unit 15. To do.

Here, the flow of the counting process executed by the counting device 1 will be described. The counting process shown in FIG. 4 starts when the power of the counting device 1 is turned on. When the image acquisition unit 11 of the counting device 1 does not receive the captured image from the photographing device 2 (step S11; NO), the image acquisition unit 11 repeats step S11 and waits for the reception of the captured image. When a captured image is received from the photographing device 2 (step S11; YES), the binarization unit 12 converts the captured image received by the image acquisition unit 11 into grayscale, and removes noise by a smoothing filter. The binarization unit 12 calculates the binarization threshold value for the grayscale photographed image from which noise has been removed (step S12). At this time, if the discriminant analysis method is used, the threshold value for binarization can be calculated at high speed. The binarization unit 12 binarizes the grayscale photographed image from which noise has been removed by using the calculated threshold value (step S13).

The reduction processing unit 13 reduces the captured image binarized by the binarization unit 12 (step S14). If the counting device 1 does not include the reduction processing unit 13, step S13 is omitted. The area calculation unit 14 classifies the pixel distribution of the captured image reduced by the reduction processing unit 13 into groups, and calculates the area of each group (step S15). When step S13 is omitted, the area calculation unit 14 classifies the pixel distribution of the captured image binarized by the binarization unit 12 into groups, and calculates the area of each group. The component number calculation unit 15 calculates the smallest area of each group calculated by the area calculation unit 14 excluding noise as a reference area which is the area of one component P (step S16). At this time, the component number calculation unit 15 determines a lower limit and an upper limit, and removes a group having an area smaller than the lower limit and a group having an area larger than the upper limit as noise.

The number of parts calculation unit 15 determines whether or not the area of each group classified by the area calculation unit 14 is within the Nth range including N times the reference area (step S17). When the area is within the Nth range including N times the reference area (step S17; YES), the number of parts P of the group is set to N and added to the number of parts P (step S20). When the area is outside the Nth range including N times the reference area (step S17; NO), the component number calculation unit 15 determines that the components P overlap and cannot count. The part number calculation unit 15 generates non-countable information indicating a non-countable group. The information output unit 16 transmits the non-countable information generated by the component number calculation unit 15 to the user terminal 3 (step S18). When the counting system 100 does not include the user terminal 3, the information output unit 16 displays the non-countable information in step S18.

In the example of non-countable information in FIG. 3A, images of all groups excluding noise are displayed. In step S17, a group whose area is determined to be outside the Nth range, that is, a group that cannot be counted is surrounded by a square frame, and the characters “cannot be counted” are displayed. The characters "Uncountable" are also displayed in the "Number of parts" column.

Returning to FIG. 4, when the component number calculation unit 15 does not receive the numerical information indicating the number of the uncountable group of components P from the user terminal 3 (step S19; NO), the component number calculation unit 15 repeats step S19 to receive the numerical information. Wait. When the component number calculation unit 15 receives numerical information indicating the number of components P in a group that cannot be counted from the user terminal 3 (step S19; YES), the number of components P indicated by the numerical information is converted into the number of components P. Add (step S20). When the counting system 100 does not include the user terminal 3, in step S19, the component number calculation unit 15 determines whether or not the numerical information has been input.

If there is a next group (step S21; YES), the process returns to step S17 and repeats steps S17 to S21. Steps S17 to S21 are repeated for all the groups classified by the area calculation unit 14, and when there is no next group (step S21; NO), the part number calculation unit 15 totals the number of parts P of all the groups. Generates component number information indicating the total number of component Ps. The information output unit 16 transmits the component number information generated by the component number calculation unit 15 to the user terminal 3 (step S22). If the counting system 100 does not include the user terminal 3, the information output unit 16 displays the number of parts information in step S22.

In the example of the number of parts information in FIG. 3B, images of all groups excluding noise are displayed. In step S17, the group determined to have an area within the second range is surrounded by a square frame, and the number “2” of the parts P is displayed. In step S17, the group determined to have an area within the third range is surrounded by a square frame, and the number of parts P “3” is displayed. The group to which the area is determined to be out of the Nth range in step S17 and the uncountable information shown in FIG. 3A is transmitted in step S18 is also surrounded by a square frame and the numerical information received in step S19 is displayed. The number "5" of the indicated component P is displayed. In the "Number of parts" column, "51", which is the total number of parts P, is displayed.

Returning to FIG. 4, when the power of the counting device 1 is not turned off (step S23; NO), the process returns to step S11 and repeats steps S11 to S23. When the power of the counting device 1 is turned off (step S23; YES), the process ends. In addition, when the parts P do not overlap as in the case where the shape of the parts P is a sphere, a group that cannot be counted does not occur, so that steps S18 and S19 may be omitted.

According to the counting system 100 according to the first embodiment, the counting device 1 that counts the component P from the captured image binarizes the captured image obtained by capturing the stationary component P, and obtains the pixel distribution of the binarized captured image. By counting the parts P based on the area of the classified group, it is not necessary to store the reference image of the parts P, and accurate counting can be performed in a shorter time.

(Embodiment 2)
In the second embodiment, the counting device 1 stores the group image information in which the image of the group and the number of parts P of the group are associated with each other, and corresponds to the image of the group similar to the image of the group that cannot be counted. The number of attached parts P is adopted as the number of parts P of a group that cannot be counted.

The functional configuration of the counting device 1 according to the second embodiment will be described with reference to FIG. As shown in FIG. 5, the counting device 1 has an image acquisition unit 11, a binarization unit 12, a reduction processing unit 13, an area calculation unit 14, a component number calculation unit 15, and an information output unit 16 as functional configurations. In addition, a group image storage unit 17 for storing group image information is provided.

When the component number calculation unit 15 receives numerical information indicating the number of parts P of the uncountable group from the user terminal 3, the group image is associated with the image of the uncountable group and the number of components P indicated by the numerical information. Generate information. The group image storage unit 17 stores the group image information generated by the component number calculation unit 15. An example of group image information is shown in FIG. For example, in the first group image information, the image of the uncountable group shown in the non-countable information of FIG. 3A is associated with the number “5” of the parts P indicated by the numerical information. That is, the group image information is information in which the image of the group that could not be counted is associated with the number of parts P of the group.

Returning to FIG. 5, when there is a group that cannot be counted, the component number calculation unit 15 refers to the group image storage unit 17, and whether or not there is group image information in which the image of the group that cannot be counted and the image of the group are similar. To judge. As a method of determining whether or not the two images are similar, for example, there is a method of extracting feature points of the two images, calculating the similarity, and determining that the two images are similar if the similarity is higher than the threshold value.

When there is group image information in which the image of the group that cannot be counted and the image of the group are similar, the component number calculation unit 15 determines the number of parts P associated with the image of the group that is similar to the image of the group that cannot be counted. It is adopted as the number of parts P in a group that cannot be counted. When there is no group image information in which the non-countable group image and the group image are similar, the component number calculation unit 15 generates the non-countable information and the information output unit 16 cannot count, as in the first embodiment. The information is transmitted to the user terminal 3. The part number calculation unit 15 adopts the number of parts P of the uncountable group indicated by the numerical information received from the user terminal 3. The component number calculation unit 15 generates group image information each time numerical information indicating the number of uncountable group components P is received from the user terminal 3. Group image information is cumulatively stored in the group image storage unit 17. Other functions of the counting device 1 are the same as those of the first embodiment.

Here, the flow of the counting process executed by the counting device 1 will be described. The counting process shown in FIG. 7 starts when the power of the counting device 1 is turned on. Since steps S31 to S36 are the same as steps S11 to S16 of the flowchart shown in FIG. 4, the description thereof will be omitted. The component number calculation unit 15 determines whether or not the area of each group classified by the area calculation unit 14 is within the Nth range including N times the reference area (step S37). When the area is within the Nth range including N times the reference area (step S37; YES), the number of parts P of the group is set to N and added to the number of parts P (step S42). When the area is outside the Nth range including N times the reference area (step S37; NO), the component number calculation unit 15 refers to the group image storage unit 17, and the group image and the group image that cannot be counted. Determines if there is similar group image information (step S38).

When there is group image information in which the non-countable group image and the group image are similar (step S38; YES), the part number calculation unit 15 is associated with the group image similar to the non-countable group image. The number of parts P is adopted as the number of parts P in a group that cannot be counted, and is added to the number of parts P (step S42).

When there is no group image information in which the non-countable group image and the group image are similar (step S38; NO), the component number calculation unit 15 generates non-countable information indicating the non-countable group. The information output unit 16 transmits the non-countable information generated by the component number calculation unit 15 to the user terminal 3 (step S39). When the component number calculation unit 15 does not receive the numerical information indicating the number of the non-countable group of components P from the user terminal 3 (step S40; NO), the component number calculation unit 15 repeats step S40 and waits for the reception of the numerical information. When the component number calculation unit 15 receives numerical information indicating the number of parts P of the uncountable group from the user terminal 3 (step S40; YES), the image of the uncountable group and the number of components P indicated by the numerical information The group image information is generated in association with the above, and the group image information is stored in the group image storage unit 17 (step S41).

For example, in the first group image information shown in FIG. 6, the image of the uncountable group shown in the uncountable information of FIG. 3A is associated with the number “5” of the parts P indicated by the numerical information. There is.

Returning to FIG. 7, the component number calculation unit 15 adds the number of components P indicated by the received numerical information to the number of components P (step S42). If there is a next population (step S43; YES), the process returns to step S37 and repeats steps S37-S43. Steps S37 to S43 are repeated for all the groups classified by the area calculation unit 14, and when there is no next group (step S43; NO), the part number calculation unit 15 totals the number of parts P of all the groups. Generates component number information indicating the total number of component Ps. The information output unit 16 transmits the component number information generated by the component number calculation unit 15 to the user terminal 3 (step S44). When the power of the counting device 1 is not turned off (step S45; NO), the process returns to step S31 and repeats steps S31 to S45. When the power of the counting device 1 is turned off (step S45; YES), the process ends.

According to the counting system 100 according to the second embodiment, the counting device 1 that counts the component P from the captured image binarizes the captured image obtained by capturing the stationary component P, and obtains the pixel distribution of the binarized captured image. By counting the parts P based on the area of the classified group, it is not necessary to store the reference image of the parts P, and accurate counting can be performed in a shorter time. Further, by adopting the number of parts P associated with the image of the group similar to the image of the group that cannot be counted as the number of parts P of the group that cannot be counted, the user can use the parts P of the group that cannot be counted. It is possible to reduce the work of visually counting the numbers and inputting the numerical information to the user terminal 3.

(Embodiment 3)
In the third embodiment, the user can visually count the number of parts P in a group that cannot be counted, and instead of inputting numerical information to the user terminal 3, the parts P on which the leveling operation devices overlap can be leveled and counted. Perform a leveling operation.

As shown in FIG. 8, the counting system 100 includes a counting device 1, a photographing device 2, and a user terminal 3, as well as a leveling operation device 4 that performs a leveling operation. The counting device 1 and the leveling operation device 4 are connected by wire or wirelessly.

When there is an uncountable group in which the number of parts P cannot be calculated from the area of the group, the counting device 1 transmits the leveling operation instruction information instructing the leveling operation to the leveling operation device 4. When the leveling operation device 4 receives the leveling operation instruction information from the counting device 1, the leveling operation device 4 performs the leveling operation. The leveling operation device 4 breaks the overlap of the parts P by, for example, vibrating the table on which the parts P are loaded. Alternatively, the leveling operation instruction information includes the coordinates of the non-countable group on the photographing range C, and the leveling operation device 4 injects air with respect to the coordinates of the non-countable group to overlap the parts P. It may be a device that breaks or breaks the overlap of parts P with a robot arm.

The photographing device 2 photographs the photographing range C after the leveling operation device 4 breaks the overlap of the parts P. As for the timing of shooting by the shooting device 2, the user may input a shooting instruction to the shooting device 2, or the shooting device 2 may detect the leveling operation of the leveling operation device 4 and shoot. Other configurations of the counting system 100 are the same as those in the first embodiment.

Here, the functional configuration of the counting device 1 according to the third embodiment will be described with reference to FIG. As shown in FIG. 9, the counting device 1 has an image acquisition unit 11, a binarization unit 12, a reduction processing unit 13, an area calculation unit 14, a component number calculation unit 15, and an information output unit 16 as functional configurations. In addition, a leveling operation instruction unit 18 is provided. The image acquisition unit 11, the binarization unit 12, the reduction processing unit 13, and the area calculation unit 14 perform the same processing as in the first embodiment.

The part number calculation unit 15 calculates the reference area regarded as the area of one part P, and calculates the number of parts P of the group of the area within the Nth range including N times the reference area as N pieces. The component number calculation unit 15 determines that the group whose area is out of the Nth range has overlapping components P, and cannot count the group. When there is a group that cannot be counted, the component number calculation unit 15 generates leveling operation instruction information that instructs the leveling operation device 4 to perform the leveling operation. When the leveling operation instruction information includes the coordinates of the uncountable group on the photographing range C, the component number calculation unit 15 includes the coordinates of all the uncountable groups on the photographing range C. To generate. The information output unit 16 transmits the leveling operation instruction information generated by the component number calculation unit 15 to the leveling operation device 4.

The leveling operation device 4 performs a leveling operation when it receives the leveling operation instruction information from the counting device 1. The photographing device 2 photographs the photographing range C after the leveling operation device 4 breaks the overlap of the parts P by the leveling operation. The image acquisition unit 11, the binarization unit 12, the reduction processing unit 13, and the area calculation unit 14 perform the same processing as in the first embodiment. When the number of uncountable groups disappears, the number of parts calculation unit 15 totals the number of parts P of all the groups and calculates the total number of parts P shown in the captured image. The component number calculation unit 15 generates component number information indicating the total number of calculated components P. The information output unit 16 transmits the component number information generated by the component number calculation unit 15 to the user terminal 3. Other functions of the counting device 1 are the same as those of the first embodiment.

Here, the flow of the counting process executed by the counting device 1 will be described. The counting process shown in FIG. 10 starts when the power of the counting device 1 is turned on. Since steps S51 to S56 are the same as steps S11 to S16 of the flowchart shown in FIG. 4, the description thereof will be omitted. The component number calculation unit 15 determines whether or not the area of each group classified by the area calculation unit 14 is within the Nth range including N times the reference area (step S57). When the area is outside the Nth range including N times the reference area (step S57; NO), the component number calculation unit 15 generates leveling operation instruction information for instructing the leveling operation device 4 to perform the leveling operation. To do. The information output unit 16 transmits the leveling operation instruction information generated by the component number calculation unit 15 to the leveling operation device 4 (step S58).

When the leveling operation instruction information includes the coordinates of a group that cannot be counted on the photographing range C, the component number calculation unit 15 determines in step S57 whether or not all the groups are within the Nth range. If there is a group outside the Nth range, leveling operation instruction information including the coordinates of all the uncountable groups on the photographing range C is generated. In step S58, the information output unit 16 transmits the leveling operation instruction information generated by the component number calculation unit 15 to the leveling operation device 4. In this case, in step S59, the component number calculation unit 15 totals the number of component Ps of all the groups, generates component number information indicating the total number of component Ps shown in the captured image, and omits step S60. ..

The leveling operation device 4 performs a leveling operation when it receives the leveling operation instruction information from the counting device 1. The photographing device 2 photographs the photographing range C after the leveling operation device 4 breaks the overlap of the parts P. The process returns to step S51 and repeats steps S51 to S57. When the area is within the Nth range including N times the reference area (step S57; YES), the number of parts P of the group is set to N and added to the number of parts P (step S59).

If there is a next group (step S60; YES), the process returns to step S57 and repeats steps S57 to S60. Steps S57 to S60 are repeated for all the groups classified by the area calculation unit 14, and when there is no next group (step S60; NO), the part number calculation unit 15 totals the number of parts P of all the groups. Generates component number information indicating the total number of component Ps. The information output unit 16 transmits the component number information generated by the component number calculation unit 15 to the user terminal 3 (step S61). If the power of the counting device 1 is not turned off (step S62; NO), the process returns to step S51 and repeats steps S51 to S62. When the power of the counting device 1 is turned off (step S62; YES), the process ends.

According to the counting system 100 according to the third embodiment, the counting device 1 that counts the component P from the captured image binarizes the captured image obtained by capturing the stationary component P, and obtains the pixel distribution of the binarized captured image. By counting the parts P based on the area of the classified group, it is not necessary to store the reference image of the parts P, and accurate counting can be performed in a shorter time. Further, since the parts P of the non-countable group are automatically leveled and can be counted, it is not necessary for the user to visually count the number of the non-countable group parts P and input the numerical information to the user terminal 3. It becomes.

(Embodiment 4)
In the fourth embodiment, the counting device 1 counts the number of parts to be delivered. The counting device 1 stores the shooting record information in which the information for identifying the part and the shot image of the part are associated with each other, and determines whether or not the part to be delivered and the part P shown in the shot image match. judge. Further, the counting device 1 reflects the number of parts P to be delivered in the inventory information indicating the number of stocks of the parts P.

As shown in FIG. 11, the counting system 100 includes a counting device 1, a photographing device 2, a user terminal 3, a production management system 5 for instructing the delivery of parts, and an inventory management system 6 for managing the inventory of parts. And. The counting device 1 is connected to the production management system 5 and the inventory management system 6 via a network.

The production management system 5 transmits the delivery information indicating the parts to be delivered to the user terminal 3 and the counting device 1. The user terminal 3 outputs the delivery information received from the production management system 5 by a method such as screen display or voice output. When the user terminal 3 outputs the delivery information, the user puts the parts indicated by the delivery information into the shooting range C. Hereinafter, the work of putting the parts indicated by the shipping information into the shooting range C is referred to as the shipping work. The photographing device 2 photographs the photographing range C, and transmits the photographed image to the counting device 1.

The counting device 1 receives the shipping information from the production management system 5, and each time it receives the shooting image from the shooting device 2, it determines whether or not there is shooting record information in which the information for identifying the goods to be shipped matches the information for identifying the goods. To do. If there is shooting record information in which the information for identifying the articles matches, the counting device 1 refers to the information and determines whether or not the parts to be delivered and the parts P shown in the shot image match. If they do not match, the counting device 1 outputs error information indicating a warning. If there is no photographing record information that matches the information for identifying the article, the counting device 1 corresponds the photographed image received from the photographing device 2 with the information for identifying the part included in the delivery information received from the production management system 5. Attach to generate and store shooting record information. If the counting device 1 generates and stores the shooting record information and it is found that the component P put into the shooting range C by the user is not a component to be delivered, the corresponding shooting record information is deleted. It may be configured.

The inventory management system 6 stores inventory information indicating the number of parts P in stock. The counting device 1 calculates the total number of parts P shown in the captured image, that is, the total number of parts P to be delivered in one delivery operation. The inventory information is updated by subtracting the total number of parts P to be delivered in one delivery operation from the number of parts P in the inventory information stored in the inventory management system 6. Other configurations of the counting system 100 are the same as those in the first embodiment.

Here, the functional configuration of the counting device 1 according to the fourth embodiment will be described with reference to FIG. As shown in FIG. 12, the counting device 1 has an image acquisition unit 11, a binarization unit 12, a reduction processing unit 13, an area calculation unit 14, a component number calculation unit 15, and an information output unit 16 as functional configurations. In addition, a correctness / rejection determination unit 19 that determines whether or not the parts to be delivered and the component P shown in the photographed image match, the photography record storage unit 20 that stores the shooting record information, and the inventory update that updates the inventory information. A unit 21 is provided.

When the correctness determination unit 19 receives the delivery information from the production management system 5 and the image acquisition unit 11 receives the captured image from the photographing device 2, the correctness determination unit 19 refers to the photographing result storage unit 20 and identifies the parts included in the issuing information. It is determined whether or not there is shooting record information in which the information and the information for identifying the part match.

When there is no shooting record information that matches the information for identifying the parts, the correctness / rejection determination unit 19 is included in the shot image received by the image acquisition unit 11 from the shooting device 2 and the shipping information received from the production management system 5. The shooting record information is generated in association with the information for identifying the parts, and is stored in the shooting record storage unit 20.

When there is shooting record information in which the information for identifying the parts matches, the correctness determination unit 19 compares the shot image of the part indicated by the shooting record information with the shot image received by the image acquisition unit 11, and the same component. It is determined whether or not it is. As a method of determining whether or not the components are the same, for example, there is a method of extracting feature points from two captured images, calculating the similarity, and determining that the components are the same if the similarity is higher than the threshold value.

The correctness determination unit 19 generates error information indicating a warning when it is determined that the parts are not the same. The information output unit 16 transmits the error information generated by the correctness determination unit 19 to the user terminal 3. The user terminal 3 displays the error information received from the counting device 1. As a result, the user can know that the inserted component P is incorrect. When the correctness determination unit 19 determines that the parts are the same, the binarization unit 12, the reduction processing unit 13, and the area calculation unit 14 perform the same processing as in the first embodiment. It should be noted that the delivery information is transmitted to the counting device 1 not only in the production management system 5. For example, the user may input the delivery information into the correctness determination unit 19, or the correctness determination unit 19 may acquire the delivery information input by the user into the user terminal 3 from the user terminal 3.

When the number of parts calculation unit 15 calculates the total number of parts P shown in the photographed image, the number of parts information indicating the calculated total number of parts P is generated as shown in FIG. The information output unit 16 transmits the component number information generated by the component number calculation unit 15 to the user terminal 3. The user terminal 3 displays the received component number information. In the example of FIG. 13, in addition to the part number information indicating the total number of parts P “51” shown in FIG. 3B, the “cumulative number of parts” 114, which is the cumulative number of parts P counted after receiving the shipping information, and the shipping information. The number of shipments of the component P indicated by "the number of delivery instructions" 200 is displayed. As a result, the user can grasp the number of the input parts P and the remaining 86 parts P need to be delivered.

Returning to FIG. 12, when the number-of-parts calculation unit 15 calculates the total number of parts P shown in the photographed image, that is, the total number of parts P to be delivered in one delivery operation, the inventory management system 6 causes the inventory update unit 21 to calculate. The inventory information is updated by subtracting the total number of parts P to be delivered in one delivery operation from the number of parts P in stock in the stored inventory information. It should be noted that the inventory information is not limited to the inventory management system 6. For example, the counting device 1 may store the inventory information, or an external storage device may store the inventory information. Other functions of the counting device 1 are the same as those of the first embodiment.

Here, the flow of the counting process executed by the counting device 1 will be described. The counting process shown in FIG. 14 starts when the power of the counting device 1 is turned on. The correctness determination unit 19 of the counting device 1 receives the delivery information from the production management system 5 (step S71). When the image acquisition unit 11 does not receive the captured image from the photographing device 2 (step S72; NO), the image acquisition unit 11 repeats step S72 and waits for the reception of the captured image. When the image acquisition unit 11 receives the captured image from the photographing device 2 (step S72; YES), the correctness determination unit 19 refers to the photographing result storage unit 20, and the information for identifying the component included in the delivery information and the component. It is determined whether or not there is shooting record information that matches the information that identifies the image (step S73).

When there is no matching shooting record information (step S73; NO), the correctness / rejection determination unit 19 is included in the shot image received by the image acquisition unit 11 from the shooting device 2 and the shipping information received from the production management system 5. The shooting record information is generated by associating it with the information that identifies the part. The correctness / rejection determination unit 19 stores the generated shooting record information in the shooting record storage unit 20 (step S76), and the process proceeds to step S77.

When there is matching shooting record information (step S73; YES), the correctness determination unit 19 compares the shot image of the component indicated by the shooting record information with the shot image received by the image acquisition unit 11, and the same component. It is determined whether or not it is (step S74). When it is determined that the parts are not the same (step S74; NO), the correctness determination unit 19 generates error information indicating a warning. The information output unit 16 transmits the error information generated by the correctness determination unit 19 to the user terminal 3 (step S75). The user terminal 3 displays the error information received from the counting device 1. The process returns to step S72, and steps S72 to S74 are repeated. If it is determined that the parts are the same (step S74; YES), the process proceeds to step S77. Since steps S77 to S87 are the same as steps S12 to S22 of the flowchart shown in FIG. 4, the description thereof will be omitted.

When the part number calculation unit 15 calculates the total number of parts P shown in the photographed image, that is, the total number of parts P to be delivered in one delivery operation, the inventory update unit 21 calculates the inventory stored in the inventory management system 6. The inventory information is updated by subtracting the total number of parts P to be delivered in one delivery operation from the number of parts P in stock in the information (step S88). If the power of the counting device 1 is not turned off (step S89; NO), the process returns to step S71, and steps S71 to S89 are repeated. When the power of the counting device 1 is turned off (step S89; YES), the process ends.

According to the counting system 100 according to the fourth embodiment, the counting device 1 that counts the component P from the captured image binarizes the captured image obtained by capturing the stationary component P, and obtains the pixel distribution of the binarized captured image. By counting the parts P based on the area of the classified group, it is not necessary to store the reference image of the parts P, and accurate counting can be performed in a shorter time. In addition, the counting device 1 stores the shooting record information in which the information for identifying the component and the captured image of the component are associated with each other, and whether or not the component to be delivered and the component P shown in the captured image match. Is determined, and if they do not match, error information is output to prevent the wrong parts from being delivered. Further, by updating the inventory information every time the warehousing operation is performed, it is possible to shorten the time during which the inventory quantity of the component P indicated by the inventory information does not match the actual inventory quantity of the component P.

(Embodiment 5)
In the fifth embodiment, the counting device 1 detects a marker that defines the counting range from the captured image and counts the number of parts P in the counting range.

As shown in FIG. 15, the counting system 100 includes a counting device 1, a photographing device 2, and a user terminal 3. When the user puts the component P into the photographing range C, the photographing device 2 photographs the photographing range C and transmits the photographed image to the counting device 1. A marker M is assigned to the photographing range C, and the counting device 1 calculates the number of parts P in the counting range R surrounded by the marker M. The marker M may be given in advance or may be given by the user. Other configurations of the counting system 100 are the same as those in the first embodiment.

Here, the functional configuration of the counting device 1 according to the fifth embodiment will be described with reference to FIG. As shown in FIG. 16, the counting device 1 has functional configurations such as an image acquisition unit 11, a binarization unit 12, a reduction processing unit 13, an area calculation unit 14, a component number calculation unit 15, and an information output unit 16. In addition, a marker storage unit 22 for storing marker information indicating the marker M is provided.

The binarization unit 12 refers to the marker information and detects the marker M in the captured image received by the image acquisition unit 11. The binarization unit 12 identifies the counting range R surrounded by the markers M from the captured image received by the image acquisition unit 11. The binarization unit 12, the reduction processing unit 13, the area calculation unit 14, the number of parts calculation unit 15, and the information output unit 16 perform the same processing as in the first embodiment on the captured image of the counting range R. ..

Here, the flow of the counting process executed by the counting device 1 will be described. The counting process shown in FIG. 17 starts when the power of the counting device 1 is turned on. When the image acquisition unit 11 of the counting device 1 does not receive the captured image from the photographing device 2 (step S91; NO), the image acquisition unit 11 repeats step S91 and waits for the reception of the captured image. When the image acquisition unit 11 receives the captured image from the photographing device 2 (step S91; YES), the binarization unit 12 refers to the marker information and displays the marker M in the captured image received by the image acquisition unit 11. To detect. The binarization unit 12 identifies the counting range R surrounded by the markers M from the captured image received by the image acquisition unit 11 (step S92). The binarization unit 12, the reduction processing unit 13, the area calculation unit 14, the number of parts calculation unit 15, and the information output unit 16 perform the processes of steps S93 to S104 on the captured image of the counting range R. Since steps S93 to S104 are the same as steps S12 to S23 of the flowchart shown in FIG. 4, the description thereof will be omitted.

According to the counting system 100 according to the fifth embodiment, the counting device 1 that counts the component P from the captured image binarizes the captured image obtained by capturing the stationary component P, and obtains the pixel distribution of the binarized captured image. By counting the parts P based on the area of the classified group, it is not necessary to store the reference image of the parts P, and accurate counting can be performed in a shorter time. Further, since the user can set the counting range, it is possible to prevent the parts in the range not intended by the user from being counted and the inserted parts P from going out of the counting range.

(Embodiment 6)
In the sixth embodiment, as shown in FIG. 18, the counting system 100 includes a counting device 1, a photographing device 2, a user terminal 3, and a machine learning device 7 that learns the counting result of the counting device 1. The machine learning device 7 connects to the counting device 1 and the user terminal 3 via a network. The machine learning device 7 may be incorporated in the counting device 1 or may exist on the cloud server.

When the user puts the component P into the photographing range C, the photographing device 2 photographs the photographing range C while the component P is stationary, and transmits the photographed image to the counting device 1. The counting device 1 calculates the total number of parts P appearing in the captured image based on the captured image received from the photographing device 2. The counting device 1 generates component number information indicating the calculated total number of components P and transmits it to the user terminal 3. Further, the counting device 1 transmits the calculated number of parts information indicating the total number of parts P and the photographed image used for calculating the total number of parts P in association with each other to the machine learning device 7. When the user terminal 3 outputs the component number information, the user counts the total number of the actual components P and inputs the correctness information indicating whether or not the counting result of the counting device 1 is correct to the user terminal 3. The user terminal 3 associates the input correctness information with the number of parts information and transmits the input to the machine learning device 7.

The machine learning device 7 learns the counting result of the counting device 1 from the data set generated based on the captured image and the number of parts information received from the counting device 1 and the correctness information received from the user terminal 3. As a result of learning the counting result of the counting device 1, the machine learning device 7 generates a learned model that outputs the total number of parts P shown in the input captured image. Other configurations of the counting system 100 are the same as those in the first embodiment.

Here, the functional configuration of the counting device 1 and the machine learning device 7 will be described with reference to FIG. The image acquisition unit 11 of the counting device 1 sends the captured image received from the photographing device 2 to the information output unit 16. The information output unit 16 transmits the captured image received from the image acquisition unit 11 and the component number information generated by the component number calculation unit 15 to the machine learning device 7. Other functional configurations of the counting device 1 are the same as those in the first embodiment.

The machine learning device 7 learns the data acquisition unit 71 that receives the captured image and the number of parts information from the counting device 1, the correctness information acquisition unit 72 that receives the correctness information from the user terminal 3, and the counting result of the counting device 1. It includes a learning unit 73 that generates a trained model, and a storage unit 74 that stores the trained model generated by the learning unit 73. The data acquisition unit 71 sends the captured image and the number of parts information received from the counting device 1 to the learning unit 73. The correctness information acquisition unit 72 sends the correctness information received from the user terminal 3 to the learning unit 73.

The learning unit 73 generates a data set for machine learning based on the captured image and the number of parts information received from the data acquisition unit 71 and the correctness information received from the correctness information acquisition unit 72. The learning unit 73 learns the counting result of the counting device 1 from the generated data set.

The learning unit 73 learns the counting result of the counting device 1 by supervised learning according to, for example, a neural network model. Supervised learning is a model in which a learning device learns features in those data sets and estimates the result from the input by giving a large amount of data sets of inputs and results (labels) to the learning device. The data set generated by the learning unit 73 for supervised learning uses the captured image in which the counting result of the counting device 1 indicated by the correctness information is correct as input data, and labels the total number of parts P indicated by the corresponding number of parts information. It is a data set associated as data. The total number of parts P indicated by the number of parts information corresponding to the photographed image in which the counting result of the counting device 1 indicated by the correctness information was correct is the total number of parts P in which the counting result of the counting device 1 indicated by the correctness information was correct. Hereinafter, the captured image in which the counting result of the counting device 1 indicated by the correctness information is correct is simply referred to as a captured image in which the counting result is correct. The total number of parts P for which the counting result of the counting device 1 indicated by the correctness information is correct is simply referred to as the total number of parts P for which the counting result is correct.

A neural network is composed of an input layer composed of a plurality of neurons, an intermediate layer composed of a plurality of neurons, and an output layer composed of a plurality of neurons. The intermediate layer may be one layer or two or more layers. For example, in the case of a three-layer neural network as shown in FIG. 20, when a plurality of input data are input to neurons X1 to X3, which are input layers, the values are multiplied by weights w11 to w16 to form an intermediate layer. It is input to neurons Y1 and Y2. The results output from the neurons Y1 to Y2 are further multiplied by the weights w21 to w26 and output from the output layers neurons Z1 to Z3. The results output from neurons Z1 to Z3 vary depending on the values of weights w11 to w16 and weights w21 to w26.

When the learning unit 73 learns the counting result of the counting device 1 according to the neural network model, the captured image with the correct counting result is input to the input layer of the neural network. The neural network adjusts the weights and learns to bring the result output from the output layer closer to the total number of parts P for which the counting result is correct, that is, the label data. When the learning is completed, the learning unit 73 stores in the storage unit 74 a learned model of the neural network that outputs the total number of parts P shown in the input captured image. The learning is completed, for example, when the data set for the test is input and the correct answer rate of the output exceeds the threshold value.

Further, the learning unit 73 may classify the supervised learning and learn the counting result. In this case, the learning unit 73 classifies the input data into classes corresponding to the label data. That is, the captured image with the correct counting result is classified into the class corresponding to the total number of parts P with the correct counting result. The learning unit 73 learns the characteristics of the captured image for which the counting result is correct for each class, that is, for each total number of parts P for which the counting result is correct. When the learning is completed, the learning unit 73 classifies the input captured image into one of the classes, and stores in the storage unit 74 a learned model that outputs the total number of parts P which are labels corresponding to the classified classes. ..

Note that the learning unit 73 may learn the counting result according to the data set generated for the plurality of parts. In this case, the data acquisition unit 71 receives the information for identifying the parts from the counting device 1 in addition to the captured image and the information on the number of parts. The learning unit 73 may generate a data set based on captured images of a plurality of parts collected from the same counting device 1, information on the number of parts, information for identifying the parts, and correctness / rejection information, or may generate a plurality of countings. A data set may be generated based on a photographed image of a plurality of parts collected from the device 1, information on the number of parts, information for identifying the parts, and correctness information. In addition, the parts for which the data set is to be collected may be added or deleted in the middle. Further, the machine learning device 7 that has learned the counting result of a certain part of a certain counting device 1 may be attached to another counting device 1 to relearn the counting result of another part.

As the learning algorithm used in the learning unit 73, deep learning, which learns the extraction of the feature amount itself, can also be used. In addition, the learning unit 73 may execute machine learning according to other known methods such as genetic programming, functional logic programming, and support vector machines.

The correctness information acquisition unit 72 of the machine learning device 7 may acquire correctness information from other than the user terminal 3. For example, as shown in FIG. 21, the counting system 100 measures the total weight of the parts P put into the shooting range C in addition to the counting device 1, the photographing device 2, the user terminal 3, and the machine learning device 7. The weight measuring device 8 may be provided. The machine learning device 7 and the weight measuring device 8 are connected by wire or wirelessly.

In this case, the weight measuring device 8 transmits the total part weight information indicating the total weight of the parts P put into the photographing range C to the machine learning device 7. The correctness / rejection information acquisition unit 72 stores in advance unit weight information indicating the weight of one component P. The correctness / rejection information acquisition unit 72 divides the total weight of the parts P indicated by the total weight information received from the weight measuring device 8 by the weight of one part P, and calculates the total number of parts P put into the photographing range C. calculate. In the correctness / rejection information acquisition unit 72, the total number of parts P indicated by the number of parts information received from the measuring device 1 by the data acquisition unit 71 and the total number of parts P calculated based on the total weight information received from the weight measuring device 8 are calculated. If they match, the correctness information indicating that the counting result of the counting device 1 is correct is generated. If they do not match, the correctness / rejection information acquisition unit 72 generates correctness / rejection information indicating that the counting result of the counting device 1 is incorrect.

According to the counting system 100 according to the sixth embodiment, the counting device 1 that counts the component P from the captured image binarizes the captured image obtained by capturing the stationary component P, and obtains the pixel distribution of the binarized captured image. By counting the parts P based on the area of the classified group, it is not necessary to store the reference image of the parts P, and accurate counting can be performed in a shorter time. Further, by using the trained model generated by the machine learning device 7, the total number of parts P shown in the captured image can be calculated faster than that of the counting device 1. Further, the trained model generated by the machine learning device 7 is stored in a computer-readable recording medium and distributed, and the trained model is installed in a computer capable of acquiring captured images to obtain the counting device 1. A similar device can be easily realized.

(Embodiment 7)
In the seventh embodiment, the component P counted by the counting device 1 is arranged on the substrate. As shown in FIG. 22, the counting system 100 includes a counting device 1, a photographing device 2, a user terminal 3, and a component arranging device 9 for arranging the component P on a substrate. The component placement device 9 is connected to the counting device 1 and the user terminal 3. The counting device 1 may be incorporated in the component arranging device 9.

The component arranging device 9 stores the component P counted by the counting device 1 in a component supply unit (not shown). When the counting device 1 can count and the component arranging device 9 has a plurality of components to be arranged on the substrate, the component arranging device 9 includes a component supply unit for each component. When the counting device 1 can count and the component arranging device 9 has a plurality of components to be arranged on the substrate, for example, the user inputs information for identifying the components put into the photographing range C into the user terminal 3. The user terminal 3 transmits information for identifying the input component to the counting device. The counting device 1 associates the information for identifying the parts received from the user terminal 3 with the parts number information. The component arranging device 9 arranges the components housed in the component supply unit on the substrate. After that, the substrate on which the component P is arranged is sent to the solder flow tank, so that the component P is soldered and mounted on the substrate. Alternatively, the component P arranged on the substrate is soldered and mounted by the soldering device. Alternatively, the component P placed on the board by the operator is manually soldered and mounted.

The information output unit 16 of the counting device 1 transmits the number of parts information to the parts arranging device 9 instead of the user terminal 3. Other functional configurations of the counting device 1 are the same as those in the first embodiment. When the component arranging device 9 receives the component number information from the counting device 1, the component arranging device 9 stores the component P counted by the counting device 1 in the component supply unit while adding the total number of components P indicated by the component number information. When the total number of added parts P, that is, the number of parts P stored in the parts supply unit reaches a certain number, the parts arranging device 9 indicates to the user terminal 3 that the number of parts P has reached a certain number. Send arrival information. A certain number may be, for example, the number of parts P that the component arranging device 9 can arrange on the board during the operating time of one day, that is, the required number of components P of the component arranging device 9 per day. , The number may be determined based on the upper limit of the number of parts P that can be stored in the parts supply unit.

The user terminal 3 outputs the arrival information received from the component arranging device 9 by a method such as screen display or voice output. When the user terminal 3 outputs the arrival information, the user stops putting the component P into the shooting range C. When the counting device 1 can count and the component arranging device 9 has other components to be arranged on the substrate, the user puts the next component into the photographing range C. The arrival information may include information that identifies the next component. In this case, the user puts the next component into the shooting range C based on the information for identifying the next component included in the arrival information output by the user terminal 3.

Subsequently, the flow of the counting process executed by the counting device 1 and the component placement process executed by the component placement device 9 will be described. The flow of the counting process is the same as the flowchart of the counting process of the first embodiment shown in FIG. However, in step S22, the information output unit 16 transmits the component number information not only to the user terminal 3 but also to the component arranging device 9. The component placement process shown in FIG. 23 starts when the power of the component placement device 9 is turned on. When the component arranging device 9 does not receive the component number information from the counting device 1 (step S111; NO), the component arranging device 9 repeats step S111 and waits for the reception of the component number information.

When the number of parts information is received from the counting device 1 (step S111; YES), the parts arranging device 9 adds the total number of parts indicated by the number of parts information (step S112), and supplies the parts counted by the counting device 1. It is stored in the unit (step S113). The component arranging device 9 determines whether or not the total number of added components, that is, the number of components stored in the component supply unit has reached a certain number (step S114). If a certain number has not been reached (step S114; NO), the process returns to step S111, and steps S111 to S114 are repeated. When a certain number is reached (step S114; YES), the component arranging device 9 transmits arrival information indicating that the number of components has reached a certain number to the user terminal 3 (step S115). The user terminal 3 outputs the arrival information received from the component arranging device 9. When the user terminal 3 outputs the arrival information, the user stops putting the parts into the shooting range C.

When the counting device 1 is capable of counting and the component arranging device 9 has other components to be arranged on the substrate (step S116; YES), the user puts the next component into the photographing range C. The arrival information may include information that identifies the next component. In this case, the user sees the information for identifying the next component output by the user terminal 3 and puts the component into the shooting range C. The counting device 1 calculates the total number of the next parts put into the photographing range C, and transmits the part number information to the parts arranging device 9. The process returns to step S111, and the component arranging device 9 receives component number information from the counting device 1 (step S111; YES). The component arranging device 9 repeats steps S111 to S116. When the counting device 1 can count and the component arranging device 9 has no other components to be arranged on the board (step S116; NO), the component arranging device 9 arranges the components stored in the component supply unit on the board. (Step S117), and the process ends.

According to the counting system 100 according to the seventh embodiment, the counting device 1 that counts the component P from the captured image binarizes the captured image obtained by capturing the stationary component P, and obtains the pixel distribution of the binarized captured image. By counting the parts P based on the area of the classified group, it is not necessary to store the reference image of the parts P, and accurate counting can be performed in a shorter time. Further, the component arranging device 9 stores the components P counted by the counting device 1 in the component supply unit until a certain number is reached. As a result, the component arranging device 9 can smoothly arrange the components on the substrate without running out of the components stored in the component supply unit.

The hardware configuration of the counting device 1 will be described with reference to FIG. 24. As shown in FIG. 24, the counting device 1 includes a temporary storage unit 101, a storage unit 102, a calculation unit 103, an operation unit 104, an input / output unit 105, and a display unit 106. The temporary storage unit 101, the storage unit 102, the operation unit 104, the input / output unit 105, and the display unit 106 are all connected to the calculation unit 103 via the BUS.

The calculation unit 103 is, for example, a CPU (Central Processing Unit). According to the control program stored in the storage unit 102, the calculation unit 103 includes a binarization unit 12, a reduction processing unit 13, an area calculation unit 14, a component number calculation unit 15, and a leveling operation instruction unit 18. Each process of the correctness determination unit 19 and the inventory update unit 21 is executed.

The temporary storage unit 101 is, for example, a RAM (Random-Access Memory). The temporary storage unit 101 loads the control program stored in the storage unit 102 and is used as a work area of the calculation unit 103.

The storage unit 102 is a non-volatile memory such as a flash memory, a hard disk, a DVD-RAM (Digital Versatile Disc-Random Access Memory), and a DVD-RW (Digital Versatile Disc-ReWritable). The storage unit 102 stores in advance a program for causing the calculation unit 103 to perform the processing of the counting device 1, and supplies the data stored by this program to the calculation unit 103 according to the instruction of the calculation unit 103, and the calculation unit 102. The data supplied from 103 is stored. The group image storage unit 17, the shooting record storage unit 20, and the marker storage unit 22 are configured in the storage unit 102.

The operation unit 104 is an interface device that connects an input device such as a keyboard and a pointing device and an input device such as a keyboard and a pointing device to the BUS. For example, in the case of a configuration in which information is directly input to the counting device 1, the input information is supplied to the calculation unit 103 via the operation unit 104.

The input / output unit 105 is a network termination device or wireless communication device connected to the network, and a serial interface or LAN (Local Area Network) interface connected to them. The input / output unit 105 functions as an image acquisition unit 11, a component number calculation unit 15, an information output unit 16, a leveling operation instruction unit 18, a correctness determination unit 19, and an inventory update unit 21.

The display unit 106 is a display device such as a CRT (Cathode Ray Tube) or an LCD (Liquid Crystal Display). For example, in the case of a configuration in which information is directly input to the counting device 1, the display unit 106 displays an operation screen. Further, when the information output unit 16 displays the non-countable information and the number of parts information, the display unit 106 functions as the information output unit 16.

Image acquisition unit 11, binarization unit 12, reduction processing unit 13, area calculation unit 14, number of parts calculation unit 15, information output unit of the counting device 1 shown in FIGS. 2, 5, 9, 19 and 16. 16. The control program controls the processing of the group image storage unit 17, the leveling operation instruction unit 18, the shooting result storage unit 20, the correctness / rejection determination unit 19, the inventory update unit 21, and the marker storage unit 22. It is executed by processing using the calculation unit 103, the storage unit 102, the operation unit 104, the input / output unit 105, the display unit 106, and the like as resources.

In addition, the above hardware configuration and flowchart are examples, and can be changed and modified arbitrarily.

The central part of the counting device 1, such as the calculation unit 103, the temporary storage unit 101, the storage unit 102, the operation unit 104, the input / output unit 105, and the display unit 106, is a normal computer regardless of a dedicated system. It can be realized using a system. For example, a computer program for executing the above operation can be a computer-readable recording medium such as a flexible disk, a CD-ROM (Compact Disc-Read Only Memory), or a DVD-ROM (Digital Versatile Disc-Read Only Memory). The counting device 1 that executes the above-mentioned processing may be configured by storing and distributing the computer program in the computer and installing the computer program in the computer. Further, the counting device 1 may be configured by storing the computer program in a storage device of a server device on a communication network such as the Internet and downloading it by a normal computer system.

Further, when the function of the counting device 1 is realized by sharing the OS (Operating System) and the application program, or by coordinating the OS and the application program, only the application program part is stored in a recording medium, a storage device, or the like. You may.

It is also possible to superimpose a computer program on a carrier wave and provide it via a communication network. For example, the computer program may be posted on a bulletin board system (BBS, Bulletin Board System) on the communication network, and the computer program may be provided via the communication network. Then, by starting this computer program and executing it in the same manner as other application programs under the control of the OS, the above processing may be executed.

In the fourth embodiment, the counting device 1 has an image acquisition unit 11, a binarization unit 12, a reduction processing unit 13, an area calculation unit 14, a component number calculation unit 15, and an information output unit 16 as functional configurations. In addition, a correct / reject determination unit 19, a shooting record storage unit 20, and an inventory update unit 21 are provided, but the present invention is not limited to this. The counting device 1 includes an image acquisition unit 11, a binarization unit 12, a reduction processing unit 13, an area calculation unit 14, a component number calculation unit 15, an information output unit 16, a correctness determination unit 19, and a shooting record storage. A configuration may be configured in which the unit 20 is provided but the inventory update unit 21 is not provided, or the correctness determination unit 19 and the shooting record storage unit 20 are not provided but the inventory update unit 21 is provided.

The counting device 1 of the above-described

embodiments

4 and 5 adds a function to the counting device 1 of the first embodiment, but the present invention is not limited to this. Both functions of the functions of the fourth and fifth embodiments may be added to the counting device 1 of the first embodiment, or the counting devices 1 of the second embodiment or the third embodiment may have the functions of the fourth and fifth embodiments. One or both of the functions of the above may be added.

The machine learning device 7 of the sixth embodiment learns the counting result of the counting device 1, but the present invention is not limited to this, and the correctness of the counting result of the counting device 1 may be learned. In this case, the data set generated by the learning unit 73 for supervised learning uses the total number of parts P indicated by the captured image and the number of parts information as input data, and is the counting result of the counting device 1 indicated by the corresponding correctness information. It is a data set in which correctness is associated as label data. The learning unit 73 learns the correctness of the counting result of the counting device 1 from the generated data set. As a result of learning the correctness of the counting result of the counting device 1, the learning unit 73 generates a learned model that outputs the correctness of the counting result with respect to the total number of input captured images and parts P. For example, another counting method in which the photographed image of the counting device 1 and the number of parts information are input to this trained model, and when the counting result of the counting device 1 is output to be incorrect, the user visually counts. By using, the accuracy rate of the counting result of the total number of parts P can be increased.

The counting system 100 of the above-described

embodiments

6 and 7 adds a machine learning device 7 or a component arranging device 9 to the counting system 100 of the first embodiment, but the present invention is not limited to this. The machine learning device 7 or the component placement device 9 may be added to the counting system 100 of the first to fifth embodiments, or both the machine learning device 7 and the component placement device 9 may be added.

The counting device 1 of the above embodiment 7 transmits the component number information to the component arranging device 9 instead of the user terminal 3, and the component arranging device 9 reaches a certain number of components P to the user terminal 3. Send arrival information indicating that. When the arrival information is displayed on the user terminal 3, the user stops putting the component P into the shooting range C. Not limited to this, the counting device 1 transmits the component number information to the user terminal 3 and the component arranging device 9, adds the number of components P indicated by the component number information received by the user terminal 3, and reaches a certain number. Then, the arrival information may be displayed. In this case, the component arranging device 9 does not have to transmit the arrival information to the user terminal 3.

In the above embodiment, an example of the counting system 100 that counts parts has been described, but the present invention is not limited to this. The counting system 100 may be any counting system that counts articles.

It should be noted that the present disclosure allows various embodiments and modifications without departing from the broad spirit and scope of the present disclosure. Moreover, the above-described embodiment is for explaining this disclosure, and does not limit the scope of the present disclosure. That is, the scope of the present disclosure is indicated not by the embodiment but by the claims. And various modifications made within the scope of the claims and within the equivalent meaning of disclosure are considered to be within the scope of this disclosure.

This application is based on Japanese Patent Application No. 2019-042438, which was filed on March 8, 2019. The specification of Japanese Patent Application No. 2019-042438, the scope of claims, and the entire drawing shall be incorporated into this specification as a reference.

1 Counting device, 2 Imaging device, 3 User terminal, 4 Leveling operation device, 5 Production management system, 6 Inventory management system, 7 Machine learning device, 8 Weight measuring device, 9 Parts placement device, 11 Image acquisition unit, 12 2. Value conversion unit, 13 reduction processing unit, 14 area calculation unit, 15 parts number calculation unit, 16 information output unit, 17 group image storage unit, 18 leveling operation instruction unit, 19 correctness judgment unit, 20 shooting record storage unit, 21 Inventory update unit, 22 marker storage unit, 71 data acquisition unit, 72 correct / false information acquisition unit, 73 learning unit, 74 storage unit, 100 counting system, 101 temporary storage unit, 102 storage unit, 103 calculation unit, 104 operation unit, 105 Input / output unit, 106 display unit, C shooting range, M marker, P parts, R counting range.

Claims

A counting system including a photographing device for photographing a stationary article and a counting device for counting the articles photographed by the photographing device.
The counting device is
An image acquisition unit that acquires a photographed image from the photographing device, and
A binarization unit that binarizes the captured image acquired by the image acquisition unit,
An area calculation unit that calculates the area of a group in which the binarization unit classifies the pixel distribution of the captured image that has been binarized.
The number of articles for each group is calculated from the area of the group calculated by the area calculation unit, the number of articles for each group is totaled, and the number of articles information indicating the total number of articles shown in the photographed image is generated. The number of goods calculation unit and
An information output unit that outputs the article number information generated by the article number calculation unit, and
Counting system with.
The counting system according to claim 1, wherein the binarization unit calculates a binarization threshold by a discriminant analysis method.
The counting device further includes a reduction processing unit in which the binarization unit reduces the captured image that has been binarized.
The area calculation unit calculates the area of a group that classifies the pixel distribution of the captured image reduced by the reduction processing unit.
The counting system of claim 1 or 2.
Further equipped with a user terminal used by the user
When there is an uncountable group whose number of articles cannot be calculated from the area of the group calculated by the area calculation unit, the article number calculation unit generates uncountable information indicating the uncountable group.
The information output unit transmits the non-countable information generated by the article number calculation unit to the user terminal.
When the article number calculation unit receives numerical information indicating the number of articles in the uncountable group from the user terminal, the article number calculation unit adopts the number of articles in the uncountable group indicated by the numerical information and takes the photographed image. Calculate the total number of items shown in
The counting system according to any one of claims 1 to 3.
The counting device is
It further has a group image storage unit that stores group image information in which a group image is associated with the number of articles in the group.
When there is a non-countable group and there is the group image information in which the image of the non-countable group and the image of the group are similar, the article number calculation unit is associated with the image of the similar group. The number of articles is adopted as the number of articles in the uncountable group to calculate the total number of articles in the photographed image, and the group image in which the image of the uncountable group and the image of the group are similar. When there is no information, the total number of articles in the photographed image is calculated by adopting the number of articles in the uncountable group indicated by the numerical information, and the image of the uncountable group and the numerical value are calculated. The group image information is generated in association with the number of articles in the uncountable group indicated by the information.
The counting system according to claim 4.
Further provided with a leveling operation device that performs a leveling operation that breaks the overlap of the stationary articles and enables counting.
The counting device is
When there is an uncountable group whose number of articles cannot be calculated from the area of the group calculated by the area calculation unit, the leveling operation device further has a leveling operation instruction unit for instructing the leveling operation.
The counting system according to any one of claims 1 to 3.
The counting device is
A shooting record storage unit that stores shooting record information in which information for identifying an article and the photographed image of the article are associated with each other.
When the delivery information including the information for identifying the goods to be delivered is acquired and the shooting record information in which the information for identifying the goods to be delivered and the information for identifying the goods match is available, the goods to be delivered are identified. By comparing the photographed image associated with the information for identifying the article that matches the information with the photographed image acquired by the image acquisition unit, the article to be delivered and the article appearing in the photographed image are compared. It determines whether or not they match, generates error information that indicates a warning if they do not match, and if there is no shooting record information that matches the information that identifies the article to be delivered and the information that identifies the article, A correctness determination unit that generates the shooting record information by associating the information for identifying the article to be delivered with the captured image acquired by the image acquisition unit.
Further has
The information output unit outputs the error information generated by the correctness determination unit.
The counting system according to any one of claims 1 to 6.
The counting device is
It further has an inventory update unit that updates inventory information indicating the inventory number of articles by subtracting the total number of articles calculated by the article number calculation unit from the inventory number of the articles.
The counting system according to any one of claims 1 to 7.
The counting device is
It further has a marker storage unit that stores marker information indicating a marker that defines a counting range.
The binarization unit detects the marker indicated by the marker information from the captured image, specifies the counting range, and binarizes the captured image in the counting range.
The counting system according to any one of claims 1 to 8.
A data acquisition unit that acquires data including the photographed image and the number of articles information,
A correctness information acquisition unit that acquires correctness information indicating the correctness of the total number of articles shown in the photographed image indicated by the item number information, and
A learning unit that learns the correctness of the counting result of the counting device or the counting result of the counting device from the photographed image, the data including the article number information, and the data set generated based on the correctness information.
Have more,
The counting system according to any one of claims 1 to 9.
The article counted by the counting device is a component mounted on a substrate.
A component arranging device for storing the parts counted by the counting device in a component supply unit and arranging the components stored in the component supply unit on the substrate is further provided.
The information output unit outputs the article number information to the component arranging device.
The component arranging device stores a certain number of the components in the component supply unit based on the total number of components shown in the photographed image indicated by the article number information.
The counting system according to any one of claims 1 to 10.
An image acquisition unit that acquires a photographed image from a photographing device that photographs a stationary article,
A binarization unit that binarizes the captured image acquired by the image acquisition unit,
An area calculation unit that calculates the area of a group in which the binarization unit classifies the pixel distribution of the captured image that has been binarized.
The number of articles for each group is calculated from the area of the group calculated by the area calculation unit, the number of articles for each group is totaled, and the number of articles information indicating the total number of articles shown in the photographed image is generated. The number of goods calculation unit and
An information output unit that outputs the article number information generated by the article number calculation unit, and
A counting device equipped with.
A machine learning device that learns the counting result of the counting device according to claim 12 or the correctness of the counting result of the counting device.
A data acquisition unit that acquires data including the photographed image and the number of articles information,
A correctness information acquisition unit that acquires correctness information indicating the correctness of the total number of articles shown in the photographed image indicated by the item number information, and
A learning unit that learns the correctness of the counting result of the counting device or the counting result of the counting device from the photographed image, the data including the article number information, and the data set generated based on the correctness information.
A machine learning device equipped with.
The shooting device executes,
Shooting steps to shoot stationary objects and
Performed by the counting device,
A binarization step of binarizing the captured image captured by the imaging device, and
An area calculation step for calculating the area of a group in which the pixel distribution of the captured image binarized in the binarization step is classified, and
The number of articles for each group is calculated from the area of the group calculated in the area calculation step, the number of articles for each group is totaled, and the number of articles information indicating the total number of articles shown in the photographed image is generated. Item number calculation step and
An information output step that outputs information on the number of articles generated in the step for calculating the number of articles, and
A counting method comprising.
The shooting device executes,
Shooting steps to shoot stationary parts and
Performed by the counting device,
A binarization step of binarizing the captured image captured by the imaging device, and
An area calculation step for calculating the area of a group in which the pixel distribution of the captured image binarized in the binarization step is classified, and
The number of the parts for each group is calculated from the area of the group calculated in the area calculation step, the number of the parts for each group is totaled, and the total number of the parts shown in the photographed image is calculated. Calculation steps and
The component placement device executes,
A storage step for storing a certain number of the parts in the parts supply unit based on the total number of the parts shown in the photographed image calculated in the part number calculation step, and a storage step.
An arrangement step of arranging the components housed in the component supply unit on the board, and
Parts placement method.
Computer,
Binarization unit, which binarizes captured images of stationary objects,
An area calculation unit that calculates the area of a group in which the binarization unit classifies the pixel distribution of the captured image that has been binarized.
The number of articles for each group is calculated from the area of the group calculated by the area calculation unit, the number of articles for each group is totaled, and the number of articles information indicating the total number of articles shown in the photographed image is generated. Item number calculation unit and
An information output unit that outputs information on the number of articles generated by the article number calculation unit,
A program that functions as.