JP6824398B2 - Image processing equipment, multiplex communication system and image processing method - Google Patents

Description

The present invention relates to an image processing apparatus for processing image data obtained by capturing an image of the working state of a working machine, a multiplex communication system, and an image processing method.

A work machine, for example, a component mounting machine that mounts electronic components on a circuit board, recognizes marks and codes written on the circuit board transported on the production line, and inspects the state of the circuit board and electronic components. Equipped with a camera for. As a technique related to this type of camera, for example, there is a technique of setting a region of interest (ROI) in an imaging region and outputting image data (for example, Patent Document 1). The image pickup apparatus disclosed in Patent Document 1 can read image data from each area by setting a plurality of areas of interest in the image area and assigning a read address to each area of interest. ing.

Japanese Unexamined Patent Publication No. 2006-109001

By the way, for example, in the above-mentioned component mounting machine, complicated operations such as an imaging operation and a moving operation are repeatedly performed during a mounting operation for mounting an electronic component on a circuit board. Therefore, the component mounting machine captures a plurality of images in each step during the mounting work. Therefore, it is desired to improve the accuracy of image processing by analyzing a plurality of image data captured during the mounting work.

The present invention has been made in view of the above problems, and is an image processing apparatus capable of improving the accuracy of image processing by adding classification information for machine learning to image data obtained by capturing the working state of a working machine. An object of the present invention is to provide a communication system and an image processing method.

In order to solve the above problems, the present specification describes an imaging device that captures the working state of a work machine that performs work on a work, and when the image data is captured with respect to the image data captured by the imaging device. Input the machine learning classification information addition unit that adds and outputs machine learning classification information indicating the state of the working machine, and the image data to which the machine learning classification information is added from the machine learning classification information addition unit. and, the example includes a machine learning classification information processing unit based on the machine learning classification information performs processing on the image data, wherein the machine learning classification information processing unit, in addition to the machine learning classification information, An image processing device that classifies the image data based on the error history information indicating the history of errors that have occurred in the working machine, and applies the machine learning algorithm by using the classified image data as teacher data for machine learning. Disclose.
Further, the present specification discloses a multiplex communication system including an image processing device and an image processing method.

According to the present disclosure, the machine learning classification information processing unit can classify, associate, and the like with respect to a plurality of image data obtained by capturing a working state by an imaging device based on the machine learning classification information. As a result, for example, by analyzing each classification or associated image data and feeding back (applying) the analysis result to the image processing conditions of the image data, the accuracy of the image processing after the feedback can be improved.

It is a figure which shows the structure of the board-to-board work system in embodiment. It is a block diagram which shows the structure of the component mounting machine. It is a schematic diagram which shows the state of the mounting work by the mounting head. It is a figure which shows the data structure of the image data with a header information. It is a flow chart which shows the classification process of image data. It is a flow chart which shows the classification process of image data. It is a figure which shows the data structure of the error history information.

Hereinafter, the substrate-to-board work system 10 which is an embodiment of the image processing apparatus of the present application will be described in detail with reference to the drawings.

(1. Configuration of board work system 10)
FIG. 1 shows the configuration of the board-to-board work system 10 of the present embodiment. The board-to-board work system 10 of the present embodiment includes a plurality of component mounting machines 11, an inspection device 12, a host computer 13, and a learning PC 14. The plurality of component mounting machines 11 are arranged in parallel and connected to each other, for example, to form a production line for mounting electronic components on a circuit board. The inspection device 12 inspects the mounting state of the circuit board on which the electronic components are mounted by the plurality of component mounting machines 11.

The host computer 13 is a device that is connected to a plurality of component mounting machines 11 and inspection devices 12 and controls the component mounting machines 11 and the like in a centralized manner. The host computer 13 creates and manages, for example, a control program (so-called recipe) for controlling the operation of the component mounting machine 11 and the inspection device 12. The learning PC 14 is mainly composed of a computer equipped with, for example, a CPU, RAM, and the like, and includes a storage device 14A. The storage device 14A includes, for example, a hard disk, a non-volatile memory, or the like. The learning PC 14 is connected to a plurality of component mounting machines 11 and an inspection device 12. The learning PC 14 receives image data with header information from each of the component mounting machines 11, stores the received image data in the storage device 14A, and performs machine learning. The interface of the learning PC 14 for connecting to each of the component mounting machines 11 is not particularly limited. As the interface, for example, a device such as an Ethernet (registered trademark) standard, a USB standard, or a Thunderbolt (registered trademark) standard can be used. Further, the component mounting machine 11 and the inspection device 12 transmit information regarding the error (error history information ER) to the learning PC 14.

(2. Configuration of component mounting machine 11)
Next, the configuration of the component mounting machine 11 will be described. FIG. 2 is a block showing the configuration of the component mounting machine 11. As shown in FIG. 2, the component mounting machine 11 includes a device main body 21, a mounting head 22, an X-axis slider 23, a Y-axis slider 24, and the like. For example, the component mounting machine 11 mounts the electronic components held by the suction nozzles of the mounting head 22 on the circuit board conveyed on the production line based on the control of the mounting control board 32 of the apparatus main body 21. To carry out.

The device main body 21 is connected to the multiplex communication device 25 and the Y-axis slider 24 on the main body side. The multiplexing communication device 25 on the main body side is connected to the multiplexing communication device 26 on the mounting head 22 side and the multiplexing communication device 27 on the X-axis slider 23 side. The multiplex communication device 25 is connected to each of the multiplex communication devices 26 and 27 via, for example, an optical fiber cable 28, and performs multiplexing communication. The optical fiber cable 28 preferably has high bending resistance. The wires that connect the multiplex communication devices 25, 26, and 27 to each other are not limited to optical fiber cables, but are, for example, LAN cables that comply with the communication standard of Gigabit Ethernet (registered trademark) and USB (Universal Serial Bus) 3. A USB cable conforming to the communication standard of 0.0 may be used. Further, the multiplex communication device 25 is not limited to wired communication, and may carry out multiplex communication with each of the multiplex communication devices 26 and 27 by wireless communication.

The multiplexing communication devices 25 to 27 transmit and receive image data GD1, I / O information IO, encoder information ED, trigger signal TRG, and the like via the optical fiber cable 28. The multiplexing communication devices 25 to 27 multiplex and transmit and receive image data GD1 and the like by, for example, time division multiplexing (TDM) type multiplexing communication. The communication method by the multiplex communication devices 25 to 27 is not limited to the time division multiplexing method, and other multiplexing methods (frequency division, etc.) may be used.

The apparatus main body 21 includes an image processing board 31 and an amplifier board 33 in addition to the mounting control board 32 described above. The image processing board 31 transmits a trigger signal TRG to the IPS camera 41 of the mounting head 22 and the mark camera 51 of the X-axis slider 23 based on the imaging instruction from the mounting control board 32. The trigger signal TRG is transmitted to the IPS camera 41 or the like via multiplexing communication. For example, the IPS camera 41 takes an image in response to the reception of the trigger signal TRG.

The amplifier board 33 receives the encoder information ED from the encoder mounted on the servomotor 43 of the mounting head 22 and the servomotor 53 of the X-axis slider 23. The amplifier board 33 receives the encoder information ED from the mounting head 22 and the like via the multiplexing communication. The amplifier board 33 transmits the received encoder information ED to the mounting control board 32. The mounting control board 32 determines, for example, a position for moving the mounting head 22 based on the encoder information ED. The mounting control board 32 transmits information such as a position change to the amplifier board 33. The amplifier board 33 controls the servomotor 53 and the like by changing the power supplied to the servomotor 53 and the like based on the information received from the mounting control board 32, and determines the position and moving speed of the mounting head 22 and the like. Control.

The mounting head 22 is connected to the multiplexing communication device 26 on the head side. The mounting head 22 includes an IPS camera 41, various I / O elements 42, and servomotors 43, 44, 45 for each axis. The mounting head 22 is mounted on the X-axis slider 23. The mounting head 22 is moved in the X-axis direction at the upper part of the apparatus main body 21 by, for example, the X-axis slider 23, and is moved in the Y-axis direction by the Y-axis slider 24.

The IPS camera 41 is, for example, a machine vision camera corresponding to the camera link (registered trademark) standard. The IPS camera 41 is a camera that images electronic components sucked by the suction nozzle of the mounting head 22 from the side. The image processing board 31 receives the image data GD1 from the IPS camera 41 of the mounting head 22 via the multiplexing communication, and detects the electronic component in the attracted state from the image data GD1. The mounting control board 32 determines whether the suction posture is good or not, whether there is a suction error, or the like, based on the detection information from the image processing board 31. The IPS camera 41 and the like are not limited to the camera link (registered trademark) standard, and may be a camera corresponding to other standards such as the GigE Vision (registered trademark) and CoaXpress (registered trademark) standards.

The I / O element 42 is a sensor or relay element that detects the position of the suction nozzle of the mounting head 22, the suction status of parts, the separation status between the suction nozzle and the circuit board, and the like. The I / O element 42 transmits the I / O information IO such as the detection result to the mounting control board 32 via the multiplexing communication.

The servomotor 43 rotates, for example, a rotary head (rotary head) for holding a plurality of suction nozzles provided on the mounting head 22, so that the plurality of suction nozzles are collectively rotated around the vertical axis (R-axis direction). It functions as a drive source for moving. The servomotor 44 functions as, for example, a drive source for moving individual suction nozzles in the vertical direction (Z-axis direction). Further, the servomotor 45 functions as a drive source for rotating each suction nozzle around the vertical axis (in the θ-axis direction). An encoder for detecting a rotation position or the like is attached to each of the servomotors 43 to 45. The amplifier board 33 of the apparatus main body 21 receives the encoder information ED of the encoders of the servomotors 43 to 45 via the multiplexing communication. The amplifier board 33 controls the electric power supplied to the servomotors 43 to 45 based on the received encoder information ED, and controls the position of the suction nozzle.

The X-axis slider 23 is a slider device that moves the position of the mounting head 22 in the X-axis direction. The X-axis slider 23 includes a mark camera 51, various I / O elements 52, and a servomotor 53. The mark camera 51 is a device that images the circuit board from above, and captures identification information (marks, etc.) written on the circuit board. The image processing board 31 receives the image data GD1 from the mark camera 51 of the X-axis slider 23 via the multiplexing communication, and detects the identification information from the image data GD1. The mounting control board 32 determines the work content and the like based on the detection result of the image processing board 31.

The I / O element 52 is, for example, a position sensor that outputs a signal corresponding to the position of the X-axis slider 23. The I / O element 52 transmits the I / O information IO indicating the detection result and the like to the mounting control board 32 via the multiplexing communication.

The servomotor 53 functions as a drive source for moving the mounting head 22 in the X-axis direction. The amplifier board 33 of the apparatus main body 21 receives the encoder information ED from the encoder attached to the servomotor 53 via multiplexing communication. The amplifier board 33 controls the electric power supplied to the servomotor 53 based on the received encoder information ED, and controls the rotation position and rotation speed of the servomotor 53. As a result, the amplifier board 33 can control the position of the mounting head 22 in the X-axis direction.

The Y-axis slider 24 is a slider device that moves the position of the X-axis slider 23 in the Y-axis direction. The Y-axis slider 24 includes a servomotor 29 as a drive source. The amplifier board 33 is connected to the Y-axis slider 24 without using multiplexed communication. The amplifier board 33 controls the power supplied to the servomotor 29 based on the encoder information ED received from the encoder of the servomotor 29. Then, as described above, the mounting head 22 is attached to the X-axis slider 23. Therefore, the mounting head 22 can move in the X-axis direction and the Y-axis direction above the circuit board in response to the drive of the X-axis slider 23 and the Y-axis slider 24.

FIG. 3 schematically shows a state in which the component mounting machine 11 is viewed from above. For example, as shown by the arrow 71 in FIG. 3, the mounting control board 32 drives a transport device 74 having a conveyor for transporting the circuit board 72, and carries in the circuit board 72 from the component mounting machine 11 on the upstream side. .. The mounting control board 32 fixedly holds the carried-in circuit board 72 at the mounting position (position shown in FIG. 3). The mounting control board 32 starts the mounting operation after holding the circuit board 72.

The component mounting machine 11 includes, for example, a plurality of tape feeders 77 as devices for supplying the electronic components 75. Each of the tape feeders 77 sequentially sends out the electronic components 75 to the supply position 77A under the control of the mounting control board 32. When the mounting operation is started, the mounting control board 32 drives the X-axis slider 23 and the Y-axis slider 24 via multiplexing communication, and moves the mounting head 22 to the supply position 77A of the electronic component 75. The mounting head 22 drives one suction nozzle 78 out of a plurality of suction nozzles 78 (only one is shown in FIG. 3) to suck and hold the electronic component 75.

After sucking the electronic component 75 by the suction nozzle 78, the mounting head 22 moves to the upper side of the circuit board 72 in response to the drive of the X-axis slider 23 and the Y-axis slider 24 (see arrow 79). The mounting control board 32 drives the servomotors 43 to 45 of the mounting head 22 to change the position of the suction nozzle 78, and mounts the electronic component 75 on the circuit board 72.

When the mounting operation is completed, the mounting head 22 moves to the supply position 77A in order to acquire the next electronic component 75 (see arrow 81). The mounting head 22 repeatedly executes such mounting work on the same circuit board 72. When the mounting control board 32 completes the mounting work of all the electronic components 75, the mounting control board 32 carries out the circuit board 72 to the component mounting machine 11 on the downstream side (see arrow 82).

On the other hand, the mounting control board 32 captures a plurality of image data GD1 by the IPS camera 41 and the mark camera 51 during the series of operations described above. For example, the mounting control board 32 uses the IPS camera 41 to image the suction nozzle 78 that sucks and holds the electronic component 75 in a constant imaging region 84 that moves from the supply position 77A to the circuit board 72. Further, for example, the mounting control board 32 uses the IPS camera 41 to image the suction nozzle 78 after mounting, that is, the suction nozzle 78 that does not hold the electronic component 75, in a constant imaging region 85 above the circuit board 72. To do. Then, as shown in FIG. 2, the multiplexing communication device 25 of the present embodiment transfers the image data GD2 to which the header information HD is added to the image data GD1 to the learning PC 14 in addition to the image data GD1 transmitted to the device main body 21. Send to.

(3. About image data GD2 for learning)
As described above, the multiplexing communication device 25 transmits the image data GD2 in which the header information HD is added to the image data GD1 to the learning PC 14. FIG. 4 shows a data structure in which the header information HD is added to the image data GD1 of the IPS camera 41 as an example of the image data GD2 with the header information HD. As shown in FIG. 4, various information such as width information 61 is added as header information HD to the image data GD2. The header information HD of the present embodiment includes, for example, width information 61, height information 62, camera number (camera No.) 64, trigger count information 65, I / O information IO, error detection code (checksum, etc.) 68, and Encoder information ED1, ED2, ED3, ED4, ED5.

The multiplex communication device 25 of the present embodiment generates image data GD2 for each image capture of the image data GD1, and transmits the image data GD2 to the learning PC 14, for example. The multiplex communication device 25 may collectively transmit a plurality of image data GD2 to the learning PC 14. Further, the device for adding the header information HD to the image data GD1 is not limited to the multiplex communication device 25, and may be another device, for example, the multiplex communication device 26. In this case, the multiplexing communication device 26 may add the header information HD to the image data GD1 before performing the multiplexing process.

The width information 61 shown in FIG. 4 is information related to the width W of the image data GD1, and is, for example, information indicating the number of pixels in the horizontal direction of the image data GD1. Further, the height information 62 is information related to the height H of the image data GD1, and is, for example, information indicating the number of pixels in the vertical direction of the image data GD1. The image data GD1 has, for example, the number of pixels of 1024 (height) × 1024 (width) pixels. Further, the image data GD1 is, for example, black and white image data, and each color is represented by 8 bits (256 gradations), and each pixel has a data amount of 2 bytes.

The image data GD1 may be the image data itself (whole image) captured by the IPS camera 41, or the data (a part of the image data) of the region of interest (ROI) set in the imaging region. Good. When the region of interest is set, for example, the height information 62 is information indicating the height (number of pixels) of the region of interest.

The camera number 64 is a device-specific number for identifying a plurality of cameras (IPS camera 41, mark camera 51, etc.) from each other. The trigger count information 65 is information for identifying the order in which imaging instructions are given to the IPS camera 41 or the like. More specifically, for example, the multiplexing communication device 25 counts the number of times the trigger signal TRG (imaging instruction) is transmitted from the image processing board 31 to the IPS camera 41. The multiplexing communication device 25 detects, for example, the rising edge of the trigger signal TRG and performs count-up processing. When the image is captured by the IPS camera 41, the multiplexing communication device 25 adds the count value at that time to the image data GD1 as the trigger count information 65.

Further, the multiplex communication device 25 can reset the count value based on the control of the mounting control board 32, for example. The mounting control board 32 resets the count value of the multiplexing communication device 25 every hour, for example. As a result, the learning PC 14 can collate not only the storage time (acquisition time) of the image data GD2 but also the error history information ER of the apparatus main body 21 described later based on the trigger count information 65, for example.

The I / O information IO is, for example, detection information of the I / O element 42. The error detection code 68 is data for detecting an error in the header information HD or the image data GD1, and is, for example, data such as a checksum or CRC (Cyclic Redundancy Check). When the learning PC 14 detects an error in the header information HD of the received image data GD2 based on the detection code 68, for example, the learning PC 14 requests the multiplexing communication device 25 to retransmit the image data GD2.

The encoder information ED1 is the encoder information ED of the servomotor 43 (R-axis direction). The encoder information ED2 is the encoder information ED of the servomotor 44 (Z-axis direction). The encoder information ED3 is the encoder information ED of the servomotor 45 (in the θ-axis direction). For example, the multiplexing communication device 25 adds encoder information ED1 to ED3 of the servomotor 43 or the like separately acquired from the multiplexing data 26 as header information HD to the image data GD1.

Further, the encoder information ED4 is the encoder information ED of the servomotor 53 (X-axis direction) of the X-axis slider 23. The multiplexing communication device 25 adds, for example, the encoder information ED4 of the servomotor 53 (X-axis slider 23) separately acquired from the multiplexing data 27 as header information HD to the image data GD1. Further, the encoder information ED5 is the encoder information ED of the servomotor 29 (Y-axis direction) of the Y-axis slider 24. For example, the multiplexing communication device 25 inputs the encoder information ED5 of the servomotor 29 from the Y-axis slider 24 or the amplifier board 33 and adds it to the image data GD1 as header information HD.

The multiplexing communication device 25 adds, for example, encoder information ED1 to ED5 when the image data GD1 is imaged to the image data GD1 as header information HD. More specifically, the encoder information ED1 to ED5 are, for example, the position information of the mounting head 22 and the suction nozzle (classification information for machine learning indicating the state of the working machine) when the image data GD1 is imaged by the IPS camera 41. is there. As a result, the learning PC 14 can classify the image data GD2 and perform machine learning based on the encoder information ED1 to ED5.

The header information HD shown in FIG. 4 is an example, and other header information HD may be added. Further, the multiplex communication device 25 may transmit the image data GD2 to which the same or different header information HD as the image data GD2 is added to the device main body 21 instead of the image data GD1. As a result, the apparatus main body 21 can perform processing (classification and the like) based on the header information HD. Further, the multiplex communication device 25 adds header information HD to the image data GD1 of the mark camera 51 of the X-axis slider 23 and transmits the image data GD1 to the learning PC 14 in the same manner as in the above processing.

(4. Machine learning of image data GD2 by learning PC14)
(4-1. Automatic classification before and after mounting)
Next, the processing of the image data GD2 by the learning PC 14 will be described. 5 and 6 show a flow of classification processing of the image data GD2. First, in step 11 of FIG. 5 (hereinafter, simply referred to as “S”) 11, the learning PC 14 receives the image data GD2 received from the plurality of multiplex communication devices 25 (parts mounting machine 11) arranged on the production line. It is stored in the storage device 14A.

Next, the learning PC 14 reads the image data GD2 from the storage device 14A and decodes the header information HD added to the read image data GD2. The learning PC 14 executes automatic classification of the image data GD2 for use as data for machine learning based on the decoded header information HD (S12).

The learning PC 14 is a plurality of suction nozzles 78 provided on the mounting head 22 based on, for example, encoder information ED1 (R-axis direction), encoder information ED2 (Z-axis direction), and encoder information ED3 (θ-axis direction). Detect the position. The learning PC 14 determines which suction nozzle 78 is the suction nozzle 78 imaged in the image data GD2, that is, the suction nozzle 78 to be imaged, based on the detected position of the suction nozzle 78. The learning PC 14 is not limited to the encoder information ED1 to ED3, and may determine the suction nozzle 78 to be imaged based on other information, for example, the I / O information IO.

It is assumed that the mounting head 22 is mounted with, for example, 23 suction nozzles 78. Further, in the storage device 14A, a storage area (data folder or the like) for storing (classifying) the image data GD2 before or after mounting is secured for each suction nozzle 78 according to the number of suction nozzles 78. .. The learning PC 14 determines the type, identification number, etc. of the suction nozzle 78 to be imaged in the image data GD2 based on the header information HD, and determines the data folder (in FIG. 5) according to the suction nozzle 78 as the determination result. The image data GD2 is stored in the nozzles 1 to 23) (S13). The type of data folder shown in FIG. 5 is an example, and the number of data folders may be increased according to the type of the electronic component 75, for example.

Further, the learning PC 14 detects the position of the mounting head 22 (suction nozzle 78) based on, for example, the encoder information ED4 (X-axis direction) and the encoder information ED5 (Y-axis direction). The learning PC 14 determines, for example, whether it is the image data GD2 before mounting the electronic component 75 (see FIG. 3) or the image data GD2 after mounting based on the encoder information ED4 and ED5. As shown in FIG. 3, in the component mounting machine 11 of the present embodiment, the imaging region 84 that images the suction nozzle 78 that has attracted the electronic component 75 and the imaging region that captures the suction nozzle 78 that does not attract the electronic component 75. 84 is set. The imaging regions 84 and 85 have different positions (coordinates) in the X-axis direction and the Y-axis direction. Therefore, the learning PC 14 determines the value of the XY coordinates based on the encoder information ED4 and ED5, and determines which of the imaging regions 84 and 85 is the image data GD2 captured. This makes it possible for the learning PC 14 to determine whether it is the image data GD2 before mounting (imaging area 84) or the image data GD2 after mounting (imaging area 85).

The method of classifying the above-mentioned image data GD2 before and after mounting is an example. For example, the learning PC 14 may determine before and after wearing based on the I / O information IO. When the I / O element 42 detects the electronic component 75, the learning PC 14 determines that it is the image data GD2 before mounting. If the I / O element 42 cannot detect the electronic component 75, the learning PC 14 may determine that it is the image data GD2 after mounting. Further, the learning PC 14 may determine before and after mounting based on the encoder information ED2 (Z-axis direction). For example, when the learning PC 14 has the encoder information ED2 indicating that the position of the suction nozzle 78 is raised to the upper end in the Z-axis direction, it may be determined that the learning PC 14 has not been mounted. Further, when the learning PC 14 has the encoder information ED2 indicating that the position of the suction nozzle 78 is lowered in the Z-axis direction, it may be determined that the learning PC 14 has been mounted. In this case, the mounting control board 32 takes an image of the suction nozzle 78 immediately after mounting the electronic component 75 on the circuit board 72 by the IPS camera 41 before moving to the upper end in the Z-axis direction.

(4-2. Automatic classification based on the presence or absence of errors)
Next, a process of classifying the image data GD2 based on the error history information ER will be described. The learning PC 14 classifies the image data GD2 classified by the process of S13 described in more detail based on the criteria of whether or not an error has occurred at the time of imaging. As shown in FIG. 1, each of the component mounting machines 11 (mounting control board 32) transmits error history information ER, which is information about an error that occurred during work, to the learning PC 14. Further, the inspection device 12 transmits information determined to be defective in the inspection of the circuit board 72 after mounting (position information of the electronic component 75 having defective mounting, etc.) as error history information ER to the learning PC 14. The learning PC 14 stores the received error history information ER in the storage device 14A (S15 in FIG. 6).

FIG. 7 shows an example of the error history information ER received from the component mounting machine 11. As shown in FIG. 7, the error history information ER has time, trigger count information, camera number (No), and error content information. The learning PC 14 reads out the image data GD2 saved in each data folder of the storage device 14A in S13. Further, the learning PC 14 reads out the error history information ER stored in the storage device 14A.

The learning PC 14 collates, for example, the storage time of the image data GD2 read from the storage device 14A, the trigger count information 65 (see FIG. 4), the camera number 64, and the time of the error history information ER (S15 in FIG. 6). ). More specifically, the learning PC 14 determines whether or not there is a time that matches the imaging (storing) time of the image data GD2 in the time information of the error history information ER (time shown in the first column from the left in FIG. 7). Search from inside. Further, the learning PC 14 has the trigger count information of the target data searched and detected from the error history information ER (the number of times of imaging shown in the second column from the left in FIG. 7) and the trigger count information 65 of the image data GD2 (see FIG. 4). ) Matches the number of imagings. Further, the learning PC 14 determines whether the camera number 64 of the image data GD2 and the camera number of the error history information ER (No shown in the third column from the left in FIG. 7) match. In this way, the learning PC 14 searches the error history information ER for error information that matches the header information HD of the image data GD2 (S15). As a result, the learning PC 14 can determine whether or not an error has occurred during the imaging of the image data GD2. Further, for example, even if the system time between the multiplex communication device 25 and the learning PC 14 is different, an error corresponding to the image data GD2 is made based on information other than the time (trigger count information 65, etc.). Information can be detected. The learning PC 14 may classify the image data GD2 based on the error history information ER of the inspection device 12 (position information of the electronic component 75 having a mounting defect, etc.).

Further, as shown in FIG. 6, for example, in the storage device 14A, a storage area (subfolder or the like) for further classifying each storage area (data folder or the like) before and after mounting according to the presence or absence of an error is secured. Has been done. The learning PC 14 classifies the image data GD2 according to the presence or absence of an error based on the header information HD and the error history information ER, and saves the image data GD2 in each data folder (S16). In the above classification process, when there is image data GD2 that is difficult to classify, the learning PC 14 may notify the user of the classification error. As a result, the user can manually classify some image data GD2 that are difficult to classify.

Then, the learning PC 14 performs machine learning based on the image data GD2 classified in S16. Machine learning here is, for example, a technology that realizes a function similar to human learning ability on a computer. More specifically, for example, by learning teacher data which is a plurality of events (image data) that have occurred, analysis of features of teacher data, weighting of features, patterning, etc. are performed, and the learning results are used. Performs processing to improve the accuracy of image processing (error detection accuracy, etc.). Machine learning is, for example, deep learning or a support vector. For example, the learning PC 14 applies a deep learning algorithm using the classified image data GD2 as teacher data.

As shown in FIG. 6, as a result of classification, the image data GD2 of the present embodiment is classified into four data folders for one suction nozzle 78. The first is, for example, an image of the suction nozzle 78 before mounting, and the image data GD2 in a normal state in which the electronic component 75 is sucked. The second is, for example, an image of the suction nozzle 78 before mounting, and an image of an error in which the suction nozzle 78 that could not suck the electronic component 75 from the supply position 77A of the tape feeder 77 (see FIG. 3) was imaged. Data GD2. Alternatively, it is image data GD2 in which an error occurs even though the electronic component 75 is attracted. As an error in this case, for example, the suction nozzle 78 sucks the electronic component 75 in an inappropriate state (a state in which the lead is turned sideways and the electronic component 75 stands, or a state in which the electronic component 75 is tilted), so that the circuit board is used. It is conceivable that the installation of 72 has failed.

The third is, for example, an image of the suction nozzle 78 after mounting, and is image data GD2 in a normal state in which the electronic component 75 is not sucked (mounting is completed). The fourth is, for example, an image of the suction nozzle 78 after mounting, and is image data GD2 in an error state in which the electronic component 75 is sucked. As an error in this case, for example, it is conceivable that the electronic component 75 fails to be detached from the suction nozzle 78 to the circuit board 72 at the mounting position of the circuit board 72. The mounting head 22 takes an image of the suction nozzle 78 in the imaging region 85 (see FIG. 3) when the electronic component 75 that has not been mounted is attracted to the suction nozzle 78 and heads for the next supply position 77A.

The learning PC 14 reads, for example, the image data GD2 (teacher data) to which the tags (information) before mounting (normal), before mounting (error), after mounting (normal), and after mounting (error) are added. , Automatically extract feature points of image data GD2 and the like. Thereby, for example, the learning PC 14 can determine the occurrence of an error by machine learning from a plurality of image data GD2 (big data) without setting what the user should pay attention to to determine the occurrence of the error. The feature points of the image data GD2 (such as the degree of inclination of the electronic component 75) can be automatically extracted. More specifically, it is possible to automatically learn how much the electronic component 75 is tilted with respect to the suction nozzle 78 before mounting to cause an error. Further, if an error actually occurs even though the error is not detected by the image processing of the mounting control board 32 or the image processing board 31, the image data GD2 is machine-learned and the feature points are fed back. For example, the accuracy of image processing of the mounting control board 32 and the like can be improved.

Further, as described above, the learning PC 14 (classification information processing unit for machine learning) classifies the image data GD1 based on the header information HD, and uses the classified image data GD1 as the teacher data for machine learning for machine learning. Apply the algorithm of. According to this, for example, the image data GD2 that captures the working state of the component mounting machine 11 (working machine) is classified based on the header information HD (classification information for machine learning) and used as teacher data for machine learning. By learning, the accuracy of image processing for the image data GD1 can be improved based on the learning result.

Further, the header information HD (classification information for machine learning) gives an imaging instruction to the encoder information ED of the motor (servo motor 43, etc.) functioning as the drive source of the component mounting machine 11 (working machine), the IPS camera 41, and the like. At least one of the trigger count information 65 (trigger information) for identifying the order and the I / O information IO of the detection devices (I / O elements 42 and 52) included in the component mounting machine 11 (working machine). ..

The encoder information ED is, for example, information indicating an operating state (position, etc.) of a movable portion (mounting head 22) of the component mounting machine 11 (working machine). In the case of the component mounting machine 11, the information indicates, for example, the positions of the mounting head 22 and the suction nozzle 78 during the suction operation of the electronic component 75 before and after mounting the electronic component 75 on the circuit board 72. Thereby, a plurality of image data GD2 can be classified according to the operating state of the component mounting machine 11 based on the encoder information ED.

Further, for example, when an imaging instruction is sent from the mounting control board 32 to the IPS camera 41 or the like, the number of times the imaging instruction is counted on the mounting control board 32 and the error information are associated with each other, and the trigger count added to the image data GD2. By collating with the information 65, it is possible to determine later when and in what state the image data GD1 is captured (whether an error has occurred). Thereby, for example, a plurality of image data GD2 can be classified according to the presence or absence of an error based on the trigger count information 65.

Further, the I / O information IO is, for example, output information of the I / O element 42 (sensor) indicating whether or not the electronic component 75 is attracted to the suction nozzle 78 of the mounting head 22. Thereby, a plurality of image data GD2 can be classified according to the presence or absence of adsorption of the electronic component 75 based on the I / O information IO.

Further, the header information HD (classification information for machine learning) includes encoder information ED of a plurality of motors (servo motor 53, etc.) that function as a drive source of the component mounting machine 11 (working machine). The learning PC 14 (machine learning classification information processing unit) is an encoder of a plurality of motors (servo motor 53, servo motor 29, etc.) as information related to the imaging position (before and after mounting, etc.) of the IPS camera 41 (imaging device). The image data GD1 is classified based on the information ED. According to this, the learning PC 14 can classify the image data GD1 for each imaging position by classifying the image data GD1 based on the encoder information ED. The learning PC 14 is not limited to the encoder information ED, and may classify the image data GD1 based on, for example, whether or not the illumination of the IPS camera 41 is turned on. For example, when the learning PC 14 detects that the illumination of the IPS camera 41 is lit based on the I / O information IO of the header information HD, the IPS camera 41 (mounting head 22) is set to a predetermined image pickup position. The image data GD1 may be classified as being arranged.

Further, the component mounting machine 11 includes a mounting head 22 (movable part) that performs work on the circuit board 72, an image processing board 31 (image processing unit) that inputs image data GD1 and performs image processing on the image data GD1, and an image. A mounting control board 32 (control unit) that controls the mounting head 22 based on the result of image processing of the processing board 31 is provided. The image processing board 31 is provided separately from the learning PC 14 (machine learning classification information processing unit).

According to this, a learning PC 14 for processing the image data GD2 to which the header information HD is added and an image processing board 31 (image processing unit) for processing the image data GD1 to which the header information HD is not added are separately provided. .. The component mounting machine 11 needs to execute predetermined operations one after another in real time based on, for example, the image data GD1 captured by the IPS camera 41 or the like. For this reason, by providing the image processing board 31 and the learning PC 14 separately, the image processing board 31 is provided with a delay in processing in the learning PC 14, an update work for repairing a defect in the learning PC 14, and the like. It is possible to prevent the processing, that is, the processing of the component mounting machine 11 from being delayed (the production line is stopped, etc.).

Further, the substrate working system 10 (image processing device) further includes a storage device 14A for storing the image data GD2 processed by the learning PC 14. According to this, the image data GD2 processed (classified or the like) by the learning PC 14 is stored (stored) in the storage device 14A. Thereby, for example, a plurality of image data GD2 stored in the storage device 14A can be analyzed collectively. Further, for example, by providing the storage device 14A separately from the component mounting machine 11, the image data GD2 can be analyzed in a state (offline) in which communication with the component mounting machine 11 is disconnected.

Further, the learning PC 14 classifies the image data GD2 based on the error history information ER indicating the history of errors that have occurred in the component mounting machine 11 in addition to the header information HD (machine learning classification information), and the classified image data. A machine learning algorithm is applied using GD2 as machine learning teacher data. According to this, the learning PC 14 classifies the image data GD2 in more detail based on the error history information ER of the component mounting machine 11 in addition to the header information HD indicating the state of the component mounting machine 11, and performs machine learning. Use as teacher data. As a result, if the learning result is fed back, the accuracy of image processing can be improved more reliably.

Further, the component mounting machine 11 includes a multiplex communication system composed of multiplex communication devices 25, 26, 27 and the like. The component mounting machine 11 multiplexes the image data GD1 with other work data (encoder information ED and I / O information IO) and transmits the image data GD1. According to this, the component mounting machine 11 can multiplex a plurality of data and transmit them on the same transmission line to save wiring.

Incidentally, the board-to-board work system 10 is an example of an image processing device and a multiplex communication system. The component mounting machine 11 is an example of a working machine. The learning PC 14 is an example of a machine learning classification information processing unit. The mounting head 22 is an example of a drive unit and a movable unit. The multiplex communication device 25 is an example of a classification information addition unit for machine learning. Servo motors 29, 43 to 45, 53 are examples of motors. The image processing board 31 is an example of an image processing unit. The mounting control board 32 is an example of a control unit. The IPS camera 41 and the mark camera 51 are examples of an imaging device. The trigger count information 65 is an example of the trigger information. The circuit board 72 is an example of a work. The I / O elements 42 and 52 are examples of detection devices. The header information HD is an example of classification information for machine learning.

(5. Effect)
According to the above-described embodiment described in detail above, the following effects are obtained.
The image processing device (working system for board 10) of the present embodiment is an imaging device (IPS camera 41, mark camera 51) that captures the working state of the component mounting machine 11 (working machine) that performs work on the circuit board 72 (work). ) And the header information HD (classification information for machine learning) indicating the state of the component mounting machine 11 (working machine) when the image data GD1 is imaged is added to the image data GD1 captured by the IPS camera 41 or the like. The multiplex communication device 25 (classification information addition unit for machine learning) and the image data GD2 to which the header information HD is added are input from the multiplex communication device 25, and the image data GD2 is processed based on the header information HD. PC 14 (classification information processing unit for machine learning) is provided.

According to this, the learning PC 14 (machine learning classification information processing unit) has header information HD (machine learning classification information) for a plurality of image data GD1s whose work state is captured by an IPS camera 41 or the like (imaging device). ), Classification, association, etc. can be carried out. As a result, for example, by analyzing each classification or associated image data GD2 and feeding back (applying) the analysis result to the image processing conditions of the image data GD1 (image data GD2), the accuracy of the image processing after the feedback is achieved. Can be improved.

(6. Deformation mode)
The present application is not limited to the above embodiment, and can be implemented in a form in which various modifications and improvements have been made based on the knowledge of those skilled in the art.
For example, in the above embodiment, machine learning is performed by the learning PC 14 using the classified image data GD2 to improve the accuracy of image processing, but the present invention is not limited to this. For example, the user may check the automatically classified image data GD2 and manually change the image processing settings (extraction of feature points, etc.) by the apparatus main body 21. In this case, since the classification is automatically performed in advance, the work load on the user can be reduced.
Further, the processing by the machine learning classification information processing unit (learning PC14) of the present application is not limited to the classification processing of the image data GD2 based on the machine learning classification information (header information HD). For example, the learning PC 14 may detect an error in the position of the suction nozzle 78 and correct the misalignment of the suction nozzle 78 based on the encoder information ED. As a result, the misalignment of the suction nozzle 78 is suppressed, so that the accuracy of image processing can be improved.
Further, in the above embodiment, the learning PC 14 for performing machine learning is provided separately from the device main body 21, but the present invention is not limited to this. For example, the mounting control board 32 may perform machine learning in addition to supervising the mounting work.
Further, the position information of the drive unit in the present application is not limited to the encoder information ED of the motor (servo motor 43, etc.) that functions as the drive source of the component mounting machine 11 (working machine), for example, each part of the component mounting machine 11. It may be the position information of the output shaft of the actuator (driving unit) to be driven.
Further, the classification information for machine learning in the present application is not limited to the header information HD, and may be, for example, information added to the footer of the image data GD1.
Further, the trigger information in the present application is not limited to the trigger count information 65 indicating the number of times of imaging, and for example, other trigger information that can identify the image data GD1 (work process name at the time of starting imaging, suction nozzle 78, electronic component 75). Information indicating the order of imaging for each) may be used.
Further, in the above embodiment, the component mounting machine 11 is adopted as the target work machine for performing image processing based on the machine learning classification information, but the present invention is not limited to this. For example, an inspection device 12 that inspects the circuit board after work may be adopted as the work machine to be image-processed. Further, for example, another working robot used in the FA field may be adopted as the working machine to be image-processed.

10 anti-board work system (image processing device, multiplex communication system), 11 component mounting machine (working machine), 14 learning PC (classification information processing unit for machine learning), 14A storage device, 22 mounting head (drive unit, movable) Unit), 25 Multiplex communication device (classification information addition unit for machine learning), 29,43 to 45,53 Servo motor (motor), 31 Image processing board (image processing unit), 32 Mounting control board (control unit), 41 IPS camera (imaging device), 42, 52 I / O element (detecting device), 51 mark camera (imaging device), 65 trigger count information (trigger information), 72 circuit board (work), ED, ED1 to ED5 encoder Information, ER error history information, HD header information (classification information for machine learning), IO I / O information, GD1, GD2 image data.

Claims (9)

An imaging device that captures the working state of the work equipment that performs work on the work,
A machine learning classification information addition unit that adds and outputs machine learning classification information indicating the state of the working machine when the image data is imaged to the image data captured by the image pickup device.
A machine learning classification information processing unit that inputs the image data to which the machine learning classification information is added from the machine learning classification information addition unit and processes the image data based on the machine learning classification information.
Bei to give a,
The machine learning classification information processing unit classifies the image data based on the error history information indicating the history of errors that have occurred in the work machine in addition to the machine learning classification information, and the classified image data is machined. An image processing device that applies machine learning algorithms by using it as learning teacher data. The machine learning classification information processing unit classifies the image data based on the machine learning classification information, and applies the machine learning algorithm by using the classified image data as machine learning teacher data. The image processing apparatus according to 1. The image processing apparatus according to claim 1 or 2, wherein the machine learning classification information is position information of a drive unit that functions as a drive source for the work machine. The machine learning classification information includes encoder information of a motor that functions as a drive source of the work machine, trigger information that identifies the order in which an image pickup instruction is given to the image pickup device, and I / O of the detection device included in the work machine. The image processing apparatus according to claim 1 or 2, which is at least one piece of information. The machine learning classification information includes encoder information of a plurality of motors that function as a drive source for the work machine.
The image processing apparatus according to claim 1 or 2, wherein the machine learning classification information processing unit classifies the image data based on the encoder information of the plurality of motors as information related to the imaging position of the imaging device. .. The working machine has a movable unit that performs work on the work, an image processing unit that inputs the image data and performs image processing on the image data, and the movable unit based on the result of image processing of the image processing unit. Equipped with a control unit to control
The image processing device according to any one of claims 1 to 5, wherein the image processing unit is provided separately from the machine learning classification information processing unit. The image processing device according to any one of claims 1 to 6, further comprising a storage device for storing the image data processed by the machine learning classification information processing unit. The image processing apparatus according to any one of claims 1 to 7 is provided.
A multiplex communication system that multiplexes and transmits the image data with other work data. Steps to image the working state of the work machine that performs work on the work,
A step of adding and outputting classification information for machine learning indicating the state of the working machine when the image data is captured to the captured image data.
A step of inputting the image data to which the machine learning classification information is added and performing processing on the image data based on the machine learning classification information.
Only including,
In the step of processing the image data based on the machine learning classification information, the image data is classified based on the error history information indicating the history of errors that have occurred in the working machine in addition to the machine learning classification information. An image processing method in which a machine learning algorithm is applied by using the classified image data as machine learning teacher data .

Images