JP7481939B2 - How to identify insects - Google Patents

Description

The present invention relates to a method for identifying insects.

A method for identifying captured insects is known (for example, Patent Document 1). This method reads an image of the sticky sheet of an insect trapping device that has captured an insect. The method extracts at least the body region, leg region, and wing region from the insect region, calculates the body length, body width, and the ratio of body length to body width from the extracted body region, calculates the leg length and the ratio of leg length to body length from the leg region, and calculates the wing area and the ratio of wing area to body area from the wing region to obtain landmark data of morphological characteristics. The method identifies the insect by comparing the obtained landmark data with standard landmark data that is set and stored in advance for each insect.

An image processing device that counts the number of insects is also known (see, for example, Patent Document 2). This image processing device counts the number of insects by performing processing using features when the resolution of the target image is high, and by performing processing using template matching when the resolution of the target image is low.

Patent No. 5690856 Patent No. 5812321

By the way, information about the wing venation of insects is useful for identifying the species of insect.

The technology in Patent Document 1 extracts the wing region of insects through image processing, but it uses the area of the wing region and does not use information on the wing veins. In addition, the technology in Patent Document 2 does not use information on the wing veins either.

For this reason, the techniques of Patent Documents 1 and 2 above have the problem that they cannot identify the type of insect using information about the insect's wing veins.

The present invention was made in consideration of the above facts, and aims to identify the type of insect using information about the insect's wing veins.

The first insect identification method according to the present invention is an insect identification method in which a computer executes a process of acquiring three-dimensional data representing an insect's wing, extracting a plurality of bending points from the acquired three-dimensional data, generating a plurality of polygons based on the extracted bending points, generating a vein image, which is a two-dimensional image including the insect's vein, based on the generated polygons, and identifying the type of insect corresponding to the generated vein image using a database storing a plurality of data in which the type of insect is associated with an image representing the insect's vein or a trained model trained in advance from the plurality of data, and using the trained model to identify the type of insect depicted in the vein image from the vein image. According to the insect identification method according to the present invention, the type of insect can be identified using information about the insect's vein.

A second insect identification method according to the present invention is an insect identification method in which a computer executes a process of acquiring three-dimensional data representing the wings of an insect, extracting polyline data including the outline and wing veins of the wing from the acquired three-dimensional data, and identifying the type of insect corresponding to the acquired three-dimensional data based on features extracted from the extracted polyline data using a database storing a plurality of data in which the type of insect is associated with features representing the wing veins of the insect, or a trained model trained in advance from the plurality of data, and using the trained model to identify the type of insect from the features. According to the insect identification method according to the present invention, the type of insect can be identified using information about the wing veins of the insect.

In addition, the insect identification method according to the present invention can generate the wing vein image by referring to predetermined rules derived from the features of the ground-truth data on the insect's wing veins, so that the wing vein image is generated in such a way that overlaps or bends in the insect's wings are eliminated. This makes it possible to appropriately generate a wing vein image for identifying the type of insect.

The present invention has the effect of being able to identify the type of insect using information about the wing veins of the insect.

1 is a block diagram showing an image processing system according to a first embodiment. 10 is a diagram for explaining how an image of an adhesive paper is captured by a camera. FIG. FIG. 13 is a diagram for explaining extraction of a region representing a wing portion of an insect. FIG. 13 is a diagram for explaining the generation of three-dimensional data representing an insect wing. 11A and 11B are diagrams for explaining extraction of bending points and generation of polygons; 11 is a diagram for explaining predetermined rules derived from feature amounts of correct answer data regarding insect wing veins. FIG. FIG. 13 is a diagram for explaining generation of a vein image taking rules into consideration. FIG. 2 is a diagram for explaining data stored in a database. FIG. 13 is a diagram for explaining identification of insect species. FIG. 2 is a diagram for explaining a basic wing vein image and wing vein images of various insects. FIG. 13 is a diagram illustrating an example of feature amounts of an insect wing. 5 is a flowchart showing the contents of an insect identification processing routine of the image processing system according to the first embodiment. FIG. 11 is a block diagram showing an image processing system according to a second embodiment. FIG. 13 is a diagram for explaining extraction of polyline data. 13 is a flowchart showing the contents of an insect identification processing routine of the image processing system according to the second embodiment. FIG. 13 is a diagram showing an example of a trained model used to identify the type of insect. FIG. 11 is a diagram illustrating an example of a feature amount.

The following describes in detail an embodiment of the present invention with reference to the drawings.

[First embodiment]

<Configuration of the image processing system of this embodiment>

Figure 1 is a block diagram showing an example of the configuration of an image processing system 10 according to the first embodiment. Functionally, the image processing system 10 can be represented as including a camera 12, a computer 14, and a display unit 16, as shown in Figure 1.

Camera 12 captures an image of the sticky paper to which the insects are attached.

Figure 2 shows a diagram for explaining how the camera 12 captures an image of adhesive paper. As shown in Figure 2, an insect is attached to the adhesive paper A. In this embodiment, the camera 12 captures an image Im that shows the insect attached to the adhesive paper A.

In this embodiment, the camera 12 is a stereo camera or a movable camera. If the camera 12 is a movable camera, images of the insects attached to the adhesive paper A are captured from a number of different directions. Therefore, based on the images captured by the camera 12, it is possible to generate three-dimensional data of the insects captured in the images.

The computer 14 is configured to include a CPU (Central Processing Unit), a ROM (Read Only Memory) that stores programs for implementing each processing routine, a RAM (Random Access Memory) that temporarily stores data, a memory as a storage means, and a network interface. As shown in FIG. 1, the computer 14 functionally includes an image storage unit 20, an image processing unit 22, a three-dimensional data acquisition unit 24, a bending point extraction unit 26, a polygon generation unit 28, a vein rule storage unit 30, a vein image generation unit 32, a database 34, and an insect identification unit 36.

The image memory unit 20 stores images of insects captured by the camera 12.

The image processing unit 22 detects an area representing the wing of an insect (hereinafter simply referred to as the "wing area") from an image representing the insect stored in the image storage unit 20.

Figure 3 is a diagram for explaining the process of detecting the wing parts of an insect. As shown in Figure 3, the image processing unit 22 detects the wing region of an insect from the image of the sticky paper using a known segmentation algorithm. For example, the image processing unit 22 detects the wing region of an insect from an image representing the insect using a trained model for wing region detection that has been generated in advance by a known machine learning algorithm.

The three-dimensional data acquisition unit 24 acquires three-dimensional data representing the insect's wings using the images of the insect stored in the image storage unit 20 that correspond to the insect's wing region detected by the image processing unit 22.

Insects that are stuck to sticky paper often have distorted or folded wings. For this reason, even if you use an image of a distorted or folded wing to identify the type of insect, the accuracy is low.

In this embodiment, the image processing system 10 generates three-dimensional data representing the wings of an insect, uses the three-dimensional data to generate a two-dimensional image that is information about the wing veins, and then uses the two-dimensional image to identify the type of insect. The three-dimensional data includes wing height information. Therefore, when generating a two-dimensional image from the three-dimensional data, the image processing system 10 of this embodiment generates the two-dimensional image in such a way that the distortion and bending of the wings in the three-dimensional data is eliminated. This eliminates distorted or overlapping parts of the wings, making it possible to accurately identify the type of insect from the wing veins.

Figure 4 shows a diagram for explaining the generation of three-dimensional data representing an insect wing. As shown in Figure 4, the three-dimensional data acquisition unit 24 generates three-dimensional data T representing an insect wing using a known three-dimensional image processing algorithm. The three-dimensional data acquisition unit 24 may also generate three-dimensional data representing an insect wing using the technology disclosed in the following reference information.

Reference information: "Digital microscope 3D display function," [searched March 23, 2020], Internet: <https://www.keyence.co.jp/ss/products/microscope/beginner/technology/3d-displayfunction.jsp>

By performing the processes described below, a vein image W is generated from the three-dimensional data T as shown in Figure 4.

The bending point extraction unit 26 extracts multiple bending points from the three-dimensional data acquired by the three-dimensional data acquisition unit 24.

Fig. 5 is a diagram for explaining the extraction of bending points and the generation of polygons. As shown in Fig. 5, for example, the bending point extraction unit 26 uses a known three-dimensional image processing algorithm to extract a plurality of bending points _P1 , ..., _PN from the three-dimensional data T representing the wing of an insect. For example, the bending point extraction unit 26 extracts bending lines from the three-dimensional data T, and extracts the intersections of these bending lines as bending points.

The polygon generation unit 28 generates a plurality of polygons based on the plurality of bending points extracted by the bending point extraction unit 26. For example, as shown in FIG. 5 above, the polygon generation unit 28 generates a plurality of polygons D using a known image processing algorithm. Note that a vein image W is generated based on the plurality of polygons D as shown in FIG. 5. Note that the polygon generation unit 28 may generate the vein image W after performing a predetermined angle correction on the plurality of polygons.

The vein rule storage unit 30 stores predetermined rules derived from the features of the correct answer data on the vein patterns of insects. For example, the rules are stored in the vein rule storage unit 30 in a table format as shown in FIG. 6.

Table R1 shown in FIG. 6 stores multiple rules Z1, Z2, ... Zn. For example, the table stores a rule derived from the features of correct answer data regarding insect wing veins, such as "there is a wing chamber d, and the anterior vein C ends at the wing tip." Or, the table stores a rule such as "the wing chamber bm is closed by the apical transverse vein bm-cu, and is always separated from the wing chamber br by the midvein M."

The vein image generating unit 32 generates a vein image, which is a two-dimensional image including the insect's vein, based on the multiple polygons generated by the polygon generating unit 28. When generating the vein image, the vein image generating unit 32 refers to the rules stored in the vein rule storage unit 30 to generate the vein image in such a way that overlaps or bends in the insect's wings are eliminated.

Specifically, for example, when the wing vein image generating unit 32 generates a wing vein image, if lines _l1 and _l2 intersect in a polygon within At of image Im as shown in FIG. 7 (if there are pixels among the pixels constituting line _l1 and the pixels constituting line _l2 that are located at the same position on the image), it determines that lines _l1 and _l2 are not wing veins but overlapping wings, and that _l1 and _l2 are the outline lines of wings, and reconstructs a wing vein image W from multiple polygons so that lines _l1 and _l2 do not overlap.

Furthermore, for example, when the angle of curvature of a line _l1 representing a certain vein contained within a polygon is smaller than X degrees (not shown), the vein image generating unit 32 determines that the vein represented by the line _l1 is in a bent state, and reconstructs a vein image W from multiple polygons so that the line _l1 is not bent.

In this way, the vein image generation unit 32 generates a vein image from multiple polygons so that each rule stored in the vein rule storage unit 30 is satisfied.

Database 34 stores multiple data items that associate the type of insect with a basic vein image, which is an image that represents the wing veins of that insect. For example, data is stored in database 34 in a table format as shown in FIG. 8. Note that a basic vein image is a typical vein image of the corresponding insect type, and the type of insect is identified based on this basic vein image.

The insect identification unit 36 refers to multiple data stored in the database 34 to identify the type of insect that corresponds to the vein image generated by the vein image generation unit 32.

Specifically, the insect identification unit 36 extracts a predetermined feature amount from the wing vein image generated by the wing vein image generation unit 32. Then, the insect identification unit 36 compares the feature amount of the wing vein image generated by the wing vein image generation unit 32 with the feature amount extracted from the basic wing vein image stored in the database 34 and the wing vein image of the correct answer data of each insect stored in the database 34, and identifies which type of insect the wing vein image generated by the wing vein image generation unit 32 corresponds to.

Fig. 9A is a diagram for explaining the identification of insect types. As shown in Fig. 9A, the insect identification unit 36 extracts the position, shape, area, aspect ratio, etc. of the wing chambers r1, d, m2 of the wing vein image _Im1 generated by the wing vein image generation unit 32 as feature quantities. The insect identification unit 36 also extracts similar feature quantities from the basic wing vein image _ImK1 . Furthermore, the insect identification unit 36 extracts feature quantities from the correct wing vein image of each insect.

9B is a diagram for explaining the basic vein image and the vein image of the correct data of each insect. As shown in FIG. 9B, in addition to the basic vein image Im _K1 , vein images Im,X, Im,Y, and Im,Z of each insect are stored in the database 34. For example, vein image Im,X is the correct vein image of insect X, vein image Im,Y is the correct vein image of insect Y, and vein image Im,Z is the correct vein image of insect Z.

For this reason, the insect identification unit 36 extracts features from the basic wing vein image Im _K1 , and also extracts features from the correct wing vein images Im,X, Im,Y, and Im,Z of each insect.

Then, the insect identification unit 36 compares the features of the wing vein image _Im1 with the features of the basic wing vein image _ImK1 stored in the database 34 and the features of the correct wing vein images Im,X, Im,Y, and Im,Z for each insect, and identifies the type of insect corresponding to the wing vein image _Im1 .

9A, the feature amount may be the position, shape, length, or vein contact point of the wing vein extracted after binarizing the wing vein image Im2. In this case, the insect identification unit 36 compares the feature amount of the wing vein image _Im2 with the feature amount of the basic wing vein image _ImK2 stored in the database ₃₄ and the feature amount of the correct wing vein image of each insect (not shown), and identifies the type of insect corresponding to the wing vein image _Im2 .

Furthermore, for example, when multiple features ( _A1 , h, bM, Sc, r.m, Rs, C, _CuA2 , _CuA1 , _M1 , _M2 , _R5 ) are extracted from a wing vein image as shown in Figure 10, it is also possible to identify the fly as a sciarid fly based only on feature F, which represents the degree of curvature of _M1 and _M2 .

The insect identification unit 36 outputs, for example, candidates for the type of insect that corresponds to the vein image generated by the vein image generation unit 32, together with the probability. For example, the insect identification unit 36 outputs information such as Chironomidae 93%, Mycetophilidae 5%, and Cecidomyiidae 2%. Alternatively, the insect identification unit 36 may output the type of insect whose corresponding probability is equal to or exceeds a predetermined value.

The display unit 16 displays the information output by the insect identification unit 36. The display unit 16 is realized, for example, by a display. The user checks the information displayed on the display unit 16 and determines what type of insect is attached to the adhesive paper.

<Function of the image processing system>

Next, the operation of the image processing system 10 of the first embodiment will be described with reference to FIG. 11. When the computer 14 of the image processing system 10 receives an image from the camera 12, it stores the image in the image storage unit 20. Then, when the computer 14 of the image processing system 10 receives an instruction signal to start the insect type identification process, it executes the insect identification process routine shown in FIG. 11.

<Insect identification processing routine>

In step S100, the image processing unit 22 reads an image representing an insect stored in the image storage unit 20, and detects the wing region of the insect from the image.

In step S102, the 3D data acquisition unit 24 acquires 3D data representing the insect's wing using the information on the insect's wing region detected in step S100 above.

In step S104, the bending point extraction unit 26 extracts multiple bending points from the three-dimensional data acquired in step S102.

In step S106, the polygon generation unit 28 generates multiple polygons based on the multiple bending points extracted in step S104.

In step S108, the vein image generation unit 32 generates a vein image, which is a two-dimensional image including the insect's vein, based on the multiple polygons generated in step S106, so as to eliminate overlaps or bends in the insect's wings.

In step S110, the insect identification unit 36 refers to multiple pieces of data stored in the database 34 to identify the type of insect that corresponds to the wing vein image generated in step S108.

In step S112, the insect identification unit 36 outputs the type of insect identified in step S110 above as a result.

The display unit 16 displays information output from the computer 14. The user checks the information displayed on the display unit 16 and determines what type of insect is stuck to the sticky paper.

As described above in detail, the insect identification method executed in the image processing system of the first embodiment detects the wing region of an insect from an image representing the insect. The insect identification method then acquires three-dimensional data representing the insect's wings from the image representing the insect, and extracts a plurality of bending points from the acquired three-dimensional data. The insect identification method then generates a plurality of polygons based on the extracted bending points. The insect identification method then generates a wing vein image, which is a two-dimensional image including the wing veins of the insect, based on the generated plurality of polygons. The insect identification method then refers to a database that stores a plurality of data that associates the type of insect with an image representing the wing veins of the insect, and identifies the type of insect that corresponds to the generated wing vein image. This makes it possible to identify the type of insect using information about the wing veins of the insect.

More specifically, the insect identification method of this embodiment acquires wing regions that are distorted or overlapped on a two-dimensional image as three-dimensional data, and generates a two-dimensional image, a wing vein image, from the three-dimensional data. The insect identification method then uses the wing vein image to identify the type of insect. This makes it possible to accurately identify the type of insect using information about the insect's wing veins.

In addition, even users without specialist knowledge can perform detailed classification of insects, enabling pest monitoring surveys to be conducted with less effort and time. Furthermore, since it is possible to perform analysis with a higher degree of accuracy than conventional systems, it can contribute to identifying insects and their sources of occurrence or attraction.

[Second embodiment]

Next, the second embodiment will be described. In the second embodiment, the method of generating a two-dimensional image of the vein from three-dimensional data differs from that of the first embodiment. In the second embodiment, the wing outline and vein are extracted from the three-dimensional data as polyline data, and the two-dimensional image is generated using the polyline data. Note that parts with the same configuration as the first embodiment are given the same reference numerals and descriptions are omitted.

Fig. 12 is a block diagram showing an example of the configuration of an image processing system 210 according to the second embodiment. Functionally, as shown in Fig. 12, the computer 214 of the image processing system 210 according to the second embodiment includes an image storage unit 20, an image processing unit 22, a three-dimensional data acquisition unit 24, a polyline data extraction unit 226, a vein rule storage unit 30, a vein image generation unit 232, a database 34, and an insect identification unit 36.

The polyline data extraction unit 226 extracts polyline data including the wing outline and wing veins from the three-dimensional data acquired by the three-dimensional data acquisition unit 24.

Fig. 13 is a diagram for explaining the extraction of polyline data. As shown in Fig. 13, for example, the polyline data extraction unit 226 performs a known edge extraction process on the three-dimensional data to extract polyline data E including the wing outline and wing veins. In this case, as shown in Fig. 13, intersections _P1 ... _P4 are extracted, so it is also possible to simultaneously extract lengths _L1 , _L2 , etc., of the wing veins as feature quantities. These feature quantities are used by the insect identification unit 36.

The vein image generating unit 232 generates a vein image, which is a two-dimensional image including the vein of an insect, based on the polyline data extracted by the polyline data extracting unit 226. For example, the vein image generating unit 232 generates a two-dimensional vein image by projecting three-dimensional polyline data onto a two-dimensional plane.

<Function of the image processing system>

Next, the operation of the image processing system 210 of the second embodiment will be described with reference to FIG. 14. When the computer 214 of the image processing system 210 receives an image from the camera 12, it stores the image in the image storage unit 20. Then, when the computer 214 of the image processing system 210 receives an instruction signal to start the process of identifying the type of insect, it executes the insect identification process routine shown in FIG. 14.

<Insect identification processing routine>

Steps S100 to S102 and steps S110 to S112 are executed in the same manner as in the first embodiment.

In step S204, the polyline data extraction unit 226 extracts polyline data including the wing outline and wing veins from the three-dimensional data acquired in step S102.

In step S208, the vein image generation unit 32 generates a vein image, which is a two-dimensional image including the vein of the insect, based on the polyline data generated in step S204.

Note that other configurations and operations of the image processing system 210 according to the second embodiment are similar to those of the first embodiment, and therefore will not be described.

As described above in detail, the insect identification method in the image processing system of the second embodiment detects the area of an insect's wing from an image representing the insect. The insect identification method then acquires three-dimensional data representing the insect's wings from the image representing the insect, and extracts polyline data including the wing outline and wing veins from the acquired three-dimensional data. The insect identification method then generates a wing vein image, which is a two-dimensional image including the insect's wing veins, based on the extracted polyline data. The insect identification method then refers to a database that stores multiple data that associates insect types with images representing insect wing veins, and identifies the type of insect that corresponds to the generated wing vein image. This makes it possible to identify the type of insect using information about the insect's wing veins.

The present invention is not limited to the above-described embodiment, and various modifications and applications are possible without departing from the spirit of the invention.

For example, in the above embodiment, a database storing multiple data items correlating the type of insect with an image showing the wing veins of the insect is referenced to identify the type of insect corresponding to the generated wing vein image, but the present invention is not limited to this. For example, the type of insect may be identified using a trained model for identifying the type of insect depicted in the wing vein image from the wing vein image. In this case, the trained model is trained in advance using, for example, multiple data items stored in the database 34 of the above embodiment as training data. Figure 15 shows an example of a trained model for identifying the type of insect depicted in the wing vein image from the vein image.

For example, as shown in FIG. 15, when a wing vein image is input to a trained model such as a neural network, the probability of the type of insect to which the wing vein image corresponds is output. For example, the trained model outputs probability information such as 0.8 for Chironomidae and 0.1 for Mycetophilidae. Therefore, the insect identification unit 36 may use this probability information to identify which type the wing vein image corresponds to.

In the above embodiment, polygon data or polyline data is used to generate a vein image from three-dimensional data, but the present invention is not limited to this. For example, the techniques disclosed in the following reference information 3 and 4 may be used.

Reference information 3: [Searched on March 23, 2020], Internet <https://tamasoft.co.jp/pepakura/productinfo/index.html>
Reference information 4: [Searched on March 23, 2020], Internet: <https://www.cadmac.net/products/seg5>

In the above embodiment, the wing vein image generating unit 32 generates a wing vein image so that predetermined rules related to insect wing veins are satisfied. However, the wing vein image may be corrected after it is generated. In this case, for example, the wing vein image generating unit 32 may correct the generated wing vein image using the technology disclosed in Japanese Patent No. 3837710 or Japanese Patent No. 4095768. Furthermore, the wing vein image generating unit 32 may repeat the correction of the wing vein image until the rules are satisfied.

In the above embodiment, an example has been described in which three-dimensional data is generated using images acquired by the camera 12, but the present invention is not limited to this. For example, the sticky paper on which the insects are attached may be sensed by a sensor such as a CT scan, laser, pattern light irradiation, or confocal microscope to directly acquire three-dimensional data. In this case, information is acquired as three-dimensional point cloud data.

For example, if multiple insects are attached to the adhesive paper, the types of insects can be counted, and an alert can be issued if the count value exceeds a certain value.

In the second embodiment, a vein image, which is a two-dimensional image including the vein of an insect, is generated based on polyline data, and features are extracted from the vein image. However, the present invention is not limited to this. For example, features may be extracted directly from polyline data, and the type of insect corresponding to the acquired three-dimensional data may be identified based on the features extracted from the polyline data. FIG. 16 shows an example of features extracted from polyline data. As shown in FIG. 16, it is determined which part of the vein corresponds to the polyline data (outline, C1, R1, R2, etc.), and their relative length and relative position are extracted as features. Then, the features extracted from the polyline data are compared with the features representing the type of insect and the vein of the insect stored in the database, and the type of insect is identified. Note that a pre-trained model may be used to identify the type of insect. Note that the features shown in FIG. 16 are features that can be used in each of the above embodiments.

In addition, while the above describes an embodiment in which the program according to the present invention is pre-stored (installed) in a storage unit (not shown), the program according to the present invention can also be provided in a form in which it is recorded on a recording medium such as a CD-ROM, DVD-ROM, or microSD card.

Reference Signs List 10, 210 Image processing system 12 Camera 14, 214 Computer 16 Display unit 20 Image storage unit 22 Image processing unit 24 3D data acquisition unit 26 Bend point extraction unit 28 Polygon generation unit 30 Wing vein rule storage unit 32, 232 Wing vein image generation unit 34 Database 36 Insect identification unit 226 Polyline data extraction unit

Claims (3)

Obtain 3D data representing the insect's wings,
Extracting a plurality of bending points from the acquired three-dimensional data;
generating a plurality of polygons based on the extracted plurality of bending points;
generating a vein image, which is a two-dimensional image including the vein of the insect, based on the generated polygons;
Identifying the type of insect corresponding to the generated wing vein image using a database storing a plurality of data items in which the type of insect is associated with an image showing the wing vein of the insect, or a trained model trained in advance from the plurality of data items, and using the trained model for identifying the type of insect depicted in the wing vein image from the wing vein image;
A method for identifying insects in which processing is performed by a computer. Obtain 3D data representing the insect's wings,
Extracting polyline data including a wing outline and wing veins from the acquired three-dimensional data;
Identifying the type of insect corresponding to the acquired three-dimensional data based on the feature values extracted from the extracted polyline data, using a database storing a plurality of data in which the type of insect is associated with a feature value representing the wing veins of the insect, or a trained model trained in advance from the plurality of data, and using the trained model for identifying the type of insect from the feature values.
A method for identifying insects in which processing is performed by a computer. When generating the vein image,
generating the vein image of the insect so as to eliminate overlapping or bending of the insect's wings by referring to a predetermined rule derived from a feature amount of ground-truth data on the insect's vein;
The method of claim 1 .

Images