JP7539110B2 - Visual inspection support system and visual inspection support method - Google Patents

Description

The present invention relates to a computer-based system and method for assisting in the visual inspection of man-made or natural objects larger than the human body.

For example, in a tank yard filled with huge storage tanks, personnel inspect each tank's exterior (for example, by visual inspection and photographing with a mobile phone camera) to check for abnormalities such as deterioration of the paint or corrosion. If the area of the abnormal part is larger than a certain extent, the tank is repaired.

When inspecting such large objects, a person must walk around the object to find any abnormalities and take photographs. The work of creating an inspection result report using the many photographs taken requires a considerable amount of time and effort. In addition, there are many areas, such as the high parts of the tank, that are difficult for a person to get close to and observe.

One way to help solve these problems is to use unmanned aerial vehicles. For example, Patent Document 1 discloses a system that generates a three-dimensional model of an object to be inspected based on multiple images of the object taken by an aircraft equipped with a camera. This system detects abnormalities in the object to be inspected based on the multiple images, identifies the position of the detected abnormality in a three-dimensional coordinate system, stores the image in association with the position of the abnormality, accepts the specification of any position (coordinate) on the three-dimensional model, and identifies and displays the image corresponding to the specified position.

JP 2019-211257 A

According to the system disclosed in Patent Document 1, detected abnormalities are mapped onto a three-dimensional model that is generated based on multiple images of the test object. This makes it possible to view the photographic image that corresponds to the abnormality.

However, when judging whether an object needs repair or to estimate the cost, it is necessary to identify quantitative information about the abnormal part, such as where the abnormal part is located on the object and its area or percentage. In other words, this quantitative information needs to be accurately expressed in a document such as a report using drawings or text. For this purpose, it is not enough to simply be able to see a photographic image of the abnormal part on a three-dimensional model displayed on a display screen.

In addition, there are objects for which it is restricted to photograph them with unmanned aerial vehicles. For example, if a storage tank is located close to another tank and the area between the two tanks is too narrow for an unmanned aerial vehicle to fly, it is difficult to photograph the part of the tank facing that area with an unmanned aerial vehicle. Or, if it is prohibited to fly an unmanned aerial vehicle above an object for safety reasons, it is not possible to photograph the roof of the object from directly above it with an unmanned aerial vehicle.

The present invention was made in consideration of the above problems, and one of its objectives is to provide a visual inspection support system that can support object inspection that is more adapted to real-world requirements and limitations.

The visual inspection support system according to one embodiment includes an image acquisition unit that acquires a plurality of images in which an object is displayed, a development drawing generation unit that generates a development drawing of the object, and a state detection unit that detects a predetermined area on the development drawing in which a predetermined state of the object appears based on the images, and provides a user with a display of the development drawing and information about the predetermined area.

FIG. 1 is a functional configuration diagram of an appearance inspection support system. FIG. 1 is a hardware configuration diagram of an appearance inspection support system. FIG. 1 is an explanatory diagram of an example of a process from image acquisition to generation of a three-dimensional model and then generation of a development view. FIG. 11 is an explanatory diagram of another example of a process from the generation of a three-dimensional model to the generation of a development view. FIG. 11 is an explanatory diagram of yet another example of a process from the generation of a three-dimensional model to the generation of a development view. FIG. 13 is an explanatory diagram of an example of a detection result development diagram. FIG. 13 is a diagram illustrating an example of an edited development diagram of detection results. 3 is an overall flowchart of the processing performed by the appearance inspection support system. 9 is a flowchart showing the development drawing generation in step S4 of FIG. 8 . 10 is a flow chart showing the generation of a three-dimensional point group in step S15 of FIG. 9 . 10 is a flowchart showing the generation of a simplified model in step S17 of FIG. 9 . 10 is a flowchart showing the coloring process of the simplified model in step S18 of FIG. 9 . 9 is a flowchart showing the abnormality detection in step S5 of FIG. 8 .

Figure 1 is a schematic diagram of an appearance inspection support system 1 according to one embodiment. The appearance inspection support system 1 supports the appearance inspection of a fuel tank as an example of an "object". Note that the appearance inspection support system 1 is not limited to the appearance inspection of fuel tanks, and may also support the appearance inspection of other types of objects, such as bridges, ships, or rock walls. In other words, the appearance inspection support system 1 supports the task of inspecting man-made or natural objects larger than the human body, for example, for the presence or absence and degree of rust as an example of a "predetermined state", or the presence or absence and degree of abnormalities such as damage, deformation, or alteration, through visual observation.

The visual inspection support system 1 supports visual inspections based on multiple images showing a fuel tank, acquired from at least one of an unmanned aerial vehicle 2 (hereinafter sometimes referred to as a drone 2) having a camera 24 as an example of a "mobile body" and a mobile camera 3 carried by a user as an example of a "portable imaging device". The visual inspection support system 1 is connected, for example, to a user terminal 4 used by a user so as to be capable of two-way communication, and acquires multiple images of an object from either the drone 2 or the mobile camera 3 or both via the user terminal 4 (or without via the user terminal 4), and registers the acquired multiple images of the object in the visual inspection support system 1 in association with the object.

The drone 2 is not limited to a mobile object that flies in the air, but may be another type of mobile object that has the function of photographing a fuel tank, i.e., an object. Other types of mobile objects may be, for example, a device that runs on a solid surface, a device that navigates on the surface of a liquid, or a device that submerges in a liquid, or even a device that flies in space, such as an artificial satellite. Furthermore, instead of or in combination with the drone 2, a manned mobile object that has the function of photographing an object, or a photographing device located in a specific place or area, such as a fixed camera installed near the object, may also be used to obtain an image of the object. The following describes an example in which the drone 2 flies in the air and is used to visually inspect the rust condition of a fuel tank.

The drone 2 photographs the exterior of the fuel tank by flying around or above the fuel tank. The drone 2 has, is equipped with, or is attached to, for example, a flight device 21, a position measuring device 22, an orientation measuring device 23, and a camera 24. The flight device 21 flies the drone 2 and controls its flight.

The position measuring device 22 measures three-dimensional position information of the drone 2 or the camera 24. The three-dimensional position information is, for example, three-dimensional geographic coordinates such as latitude, longitude, or altitude. For example, the position measuring device 22 may use GNSS (Global Navigation Satellite System) or RTK (Real Time Kinematic). Note that the position measuring device 22 is not limited to devices that use GNSS or RTK.

The orientation measurement device measures the line of sight (optical axis) of the camera 24 or the three-dimensional orientation of the drone 2. The three-dimensional orientation is set by, for example, a roll angle, a pitch angle, and a yaw angle. The orientation measurement device 23 is, for example, a gyro sensor. Note that the orientation measurement device 23 is not limited to a gyro sensor.

Camera 24 captures the exterior of the fuel tank. In this embodiment, images 7(1) and 7(2) (see FIG. 3) captured by camera 24 are described as still images showing the fuel tank, but the captured images 7(1) and 7(2) are not limited to still images and may be videos. Images 7(1) and 7(2) may be set as one still image in the captured video. Hereinafter, when images 7(1) and 7(2) are not to be distinguished from each other, they may be referred to as image 7.

The portable camera 3 is used to manually photograph the exterior of the fuel tank. The portable camera 3 has, is equipped with, or is attached to, for example, a photographing device 31, a position measuring device 32, and an orientation measuring device 33. The photographing device 31 photographs the exterior of the fuel tank and saves the image 7. The photographing device 31 is, for example, a camera part such as a lens or a shutter.

The position measuring device 32 measures the three-dimensional position of the mobile camera 3. The orientation measuring device 33 measures the three-dimensional orientation of the line of sight (optical axis) of the mobile camera 3. The three-dimensional orientation includes, for example, an azimuth angle on a horizontal plane and an azimuth angle on a vertical plane.

The portable camera 3 may be used in place of the drone 2 or as an aid to photography by the drone 2. For example, there may be locations where flying the drone 2 is difficult or prohibited. Such locations include, for example, the narrow space between a fuel tank and a nearby structure, or an area where flight of the drone 2 is restricted to ensure safety. Photography may be performed using the portable camera 3 in such locations. Also, if there is a part of the fuel tank that is hidden behind an object and cannot be photographed from a location where the drone 2 can fly, the portable camera 3 may be used to photograph that part.

That is, the portable camera 3 may supplement images 7 that are insufficient for visual inspection by capturing images of a range other than the range of the external appearance of the fuel tank that can be captured by the drone 2. Note that if the visual inspection support system 1 can generate a report describing the rust condition of the fuel tank using only images 7 from the drone 2, the portable camera 3 does not necessarily have to be used. If the visual inspection support system 1 can generate a report describing the rust condition of the fuel tank using only the portable camera 3, the drone 2 does not necessarily have to be used.

The user terminal 4 is an information processing and communication terminal used by a user of the visual inspection support system 1 (a person who performs a visual inspection of an object such as a fuel tank, a person who takes a photograph of an object, a person who receives and uses the results of the visual inspection, etc.). The user terminal 4 is, for example, a personal computer, a smartphone, a tablet terminal, etc. Note that the user terminal 4 is not limited to a personal computer, a smartphone, or a tablet terminal, and may be a device formed integrally with the drone 2 or the mobile camera 3.

The user terminal 4 acquires multiple images of the fuel tank from the drone 2 and/or the handheld camera 3. Each acquired image 7 includes data for the image itself and associated data corresponding to each image 7.

The associated data for each image 7 includes, for example, the exterior orientation elements and interior orientation elements for that image 7. The data for the exterior orientation elements includes, for example, parameters representing the three-dimensional position of the drone 2, camera 24, or mobile camera 3 when the drone 2 or mobile camera 3 photographed the fuel tank, and parameters representing the three-dimensional orientation of the drone 2, camera 24, or mobile camera 3.

The internal orientation elements include, for example, the focal length parameters of the camera 24 or the mobile camera 3, and/or parameters such as lens distortion coefficients. The user terminal 4 sends the multiple captured images 7 to the visual inspection support system 1.

Furthermore, the user terminal 4 includes a display device 41. The user terminal 4 displays on the display device 41 a GUI (Graphical User Interface) for controlling a number of functions 11 to 17 included in the visual inspection support system 1, which will be described later.

The visual inspection support system 1 generates a report that describes rust locations as an example of a "predetermined area" of a fuel tank, based on a large number of images 7 of the fuel tank taken from a user terminal 4. The visual inspection support system 1 includes, for example, a database 10, an image acquisition unit 11, a three-dimensional model generation unit 12, a design drawing acquisition unit 13, an unfolded view generation unit 14, a condition detection unit 15, an unfolded view editing unit 16, and a report generation unit 17. Note that in the figures, the "units" may be omitted.

2, the visual inspection support system 1 typically includes a storage (storage unit) 61, a CPU 62, a memory 63, and a communication unit 64. The storage 61 stores computer programs that, when executed by the CPU 62, cause the CPU 62 to operate as, for example, a database 10, an image acquisition unit 11, a three-dimensional model generation unit 12, a design drawing acquisition unit 13, an unfolded drawing generation unit 14, a state detection unit 15, a unfolded drawing editing unit 16, and a report generation unit 17.

The CPU 62 executes the computer programs loaded from the storage 61 to the memory 63 to manipulate and process various data stored in the storage 61. In this way, the visual inspection support system 1 realizes functions 10 to 17.

The database 10 stores and manages in the storage 6 a number of images 7 taken of each of the one or more fuel tanks being inspected, as well as various data used in the visual inspection, such as three-dimensional models, blueprints, simplified models, development drawings, and reports, which will be described later. The database 10 provides each of the other functional units 11-17 with the data that each functional unit requests to read, and also receives and stores data from each functional unit that the function requests to be saved.

The image acquisition unit 11 acquires multiple (usually a large number) captured images of a fuel tank from the user terminal 4. The image acquisition unit 11 registers the acquired images of the fuel tank in the database 10 in association with the identification information of the fuel tank. Thus, images of different fuel tanks can be registered in the database 10 in association with each fuel tank. The large number of images of each fuel tank registered in the database 10 can be supplied (read) to various functions such as the image 3D model generation unit 12, the development drawing generation unit 14, and the report generation unit 17.

The three-dimensional model generating unit 12 generates a three-dimensional model 121 of a fuel tank (e.g., three-dimensional model 121(1) shown in FIG. 3 or three-dimensional model 121(2) shown in FIG. 4) based on multiple captured images of the fuel tank. The three-dimensional model generating unit 12 associates the generated three-dimensional model 121 of the fuel tank with the fuel tank and registers it in the database 10. The registered three-dimensional model 121 can be supplied (read) to the development drawing generating unit 14 and/or the state detecting unit 15, etc.

The three-dimensional model generating unit 12 may color the surface of the three-dimensional model 121 based on multiple images 7 of the fuel tank. That is, the three-dimensional model generating unit 12 may project the images 7 onto the three-dimensional model 121, thereby coloring the surface of the three-dimensional model 121 with a color that is substantially the same as the surface color of the fuel tank based on the images 7, a color similar to this, or at least a color that allows the rusted areas of the fuel tank to be detected to be distinguished by the naked eye. Hereinafter, such a colored three-dimensional model 121 may be referred to as a colored three-dimensional model 121.

The three-dimensional model generating unit 12 is not limited to generating the three-dimensional model 121 based on an image of the fuel tank. The three-dimensional model generating unit 12 may generate the three-dimensional model 121 based on a design drawing of the fuel tank acquired from the design drawing acquisition unit 13.

The design drawing acquisition unit 13 acquires the design drawing of the fuel tank from the user terminal 4. The design drawing acquisition unit 13 associates the design drawing of the fuel tank with the fuel tank and registers it in the database 10. The design drawing registered in the database 10 can be supplied to the 3D model generation unit 12 and/or the development drawing generation unit 14.

Based on a 3D model 121 of a fuel tank registered in the database 10, the development drawing generator 14 creates a simplified model 122 (e.g., simplified model 122(1) shown in FIG. 3, simplified model 122(2) shown in FIG. 4, or simplified model 122(3) shown in FIG. 5, etc.) that simplifies the 3D shape of the fuel tank to a geometric shape close to the 3D shape. The development drawing generator 14 further generates a development drawing 141 (e.g., development drawing 141(1) of FIG. 3 or FIG. 4, or development drawing 141(2) of FIG. 5, etc.) consisting of one or more planar views in which the surface of the simplified model 122 is developed.

In the example shown in FIG. 3, a simple cylindrical simplified model 122(1) is created based on a three-dimensional model 121(1) of a fuel tank whose roof is slightly lower than the top of the side wall. The side walls and roof of the simplified model 121(1) are then each developed into a plan view to create an expanded view 141(1) of the fuel tank. On the other hand, in the example shown in FIG. 4, a simple cylindrical simplified model 122(2) similar to that in FIG. 3 is created based on a three-dimensional model 121(2) of a fuel tank with a pyramidal roof. The expanded view 141(1) similar to that in FIG. 3 is created from the simplified model 122(2). Alternatively, in the example shown in FIG. 5, a simplified model 122(3) with a conical roof is created based on a three-dimensional model 121(2) of a fuel tank with a pyramidal roof. The side walls and roof of the simplified model 122(3) are then each developed into a plan view to create an expanded view 141(2). In this way, the development drawing generating unit 14 can generate simplified models 122 with different shapes and configurations, and generate development drawings 144 from each simplified model 122. Note that the examples shown in the figures are not limiting, and simplified models and development drawings with other shapes or structures may be generated.

The development drawing generator 14 assigns color to the development drawing 141 of the simplified model 122 based on the multiple images 7 of the fuel tank, and as a result, the development drawing 141 is colored. That is, the development drawing generator 14 projects the colors of the corresponding pixels of the multiple images 7 onto the surface of the simplified model 122, thereby assigning to the surface of the simplified model 122 a color that is substantially the same as the fuel tank, or at least a color that allows the rusted area of the fuel tank to be detected to be identified by the naked eye or by automatic image processing. By converting the simplified model 122 to which color has been assigned in this way into the development drawing 141, the development drawing 141 is assigned the above color.

Hereinafter, the simplified model 122 and the development 141 to which colors have been added by the development generating unit 14 may be referred to as the colored simplified model 122 and the colored development 141, respectively. The development generating unit 14 associates the colored simplified model 122 and the colored development 141 of the fuel tank with the fuel tank and registers them in the database 10. The colored simplified model 122 and the colored development 141 registered in the database 10 can be supplied to the state detecting unit 15.

The development drawing generating unit 14 is not limited to generating the simplified model 122 and the development drawing 141 based on the three-dimensional model 121. The development drawing generating unit 14 may generate the simplified model 122 and the development drawing 141 based on a design drawing acquired from the design drawing acquiring unit 13, or may create the simplified model 122 and the development drawing 141 by manual operation by the user.

The unfolded drawing generating unit 14 is not limited to applying color to the simplified model 122 and the unfolded drawing 144 based on multiple images 7. The unfolded drawing generating unit 14 may also apply color to the simplified model 122 and the unfolded drawing 144 based on a colored three-dimensional model 121.

The condition detection unit 15 detects specific parts of the fuel tank, such as predetermined abnormal parts, such as rusted parts, based on the colored development 141. That is, the condition detection unit 15 detects rusted parts of the fuel tank from the surface color of the development 141, and marks the development 141 with rust marks 151 (see FIG. 6) as an example of a "mark" to indicate the rusted parts.

Hereinafter, the development diagram 141 showing the rusted areas may be referred to as the detection result development diagram 152. The condition detection unit 15 associates the detection result development diagram 152 of the fuel tank with the fuel tank and registers it in the database 10. The registered detection result development diagram 152 can be supplied to the development diagram editing unit 16.

The condition detection unit 15 is not limited to detecting specific condition locations, such as rust locations, based on the colored development diagram 141. The detection may also be performed based on the images 7 of the object, or on other models onto which the images 7 are projected, such as the three-dimensional model 121 or the simplified model 122.

When requested by any of the user terminals 4, the development diagram editing unit 16 incorporates the detection result development diagram 152 into a GUI (graphical user interface) configured to be editable by the user, and displays the GUI on the display device 41 of that user terminal 4. By editing the detection result development diagram 152 displayed in the GUI on the display device 41, the user can create a report development diagram 153 (see FIG. 7) suitable for inclusion in an inspection result report.

As illustrated in FIG. 6, the detection result development diagram 152 may be composed of multiple two-dimensional diagrams 143, 144 each representing a different portion of the fuel tank. In the example of FIG. 6, one two-dimensional diagram 143 represents the sidewall surface of the fuel tank, and another two-dimensional diagram 144 represents the roof surface of the fuel tank. By operating the GUI described above, the user can edit the relative positions of the multiple two-dimensional diagrams 143, 144 that make up the detection result development diagram 152. For example, the position of the two-dimensional diagram 144 representing the roof surface of the detection result development diagram 152 illustrated in FIG. 6 can be moved to the position illustrated in FIG. 7.

In addition, by operating the GUI described above, for example, by operating the correction tool 8 illustrated in FIG. 7, the user can correct the rust marks 151 automatically detected by the state detection unit 15 on the detection result development diagram 152. For example, it is possible to delete erroneously detected rust marks or add new rust marks to rust locations that were not detected. In particular, rust often dissolves in rainwater and flows down to the area below the rust location, turning the area rust-colored, and such an area (for example, the part surrounded by the correction tool 8 of the rust mark 151 shown in FIG. 7) may also be automatically detected as an erroneous rust location. Therefore, there is a great advantage in being able to observe the detection result development diagram 152 with the naked eye and correct the automatically detected results.

The development diagram editing unit 16 performs quantitative analysis of the rusted areas, such as calculating the ratio of the area of the rusted areas to the surface area of the fuel tank, based on the report development diagram 153 created by editing the detection result development diagram 152. The development diagram editing unit 16 registers the report development diagram 153 of the fuel tank and the quantitative analysis results in the database 10 in association with the fuel tank. The registered report development diagram 153 and quantitative analysis results can be supplied to the report generation unit 17.

The report generation unit 17 generates a report showing the results of the visual inspection based on the report development 153 generated by the development editing unit 16 and the quantitative analysis results of the rust locations. When the report generation unit 17 receives a report from any user terminal 4, it incorporates the report into a GUI configured to enable the user to edit the report, and displays the GUI on the display device 41 of the user terminal 4. At least the results of the quantitative analysis of the rust locations and the report development 153 are automatically included in the report.

When the report development diagram 153 is edited by the user on the GUI that allows editing of the report, the report generation unit 17 may call the development diagram editing unit 16 to regenerate the report development diagram 153. The report generation unit 17 may regenerate the report based on the regenerated report development diagram 153, and display the regenerated report on the display device 41.

In response to a user request from the user terminal 4, the report generation unit 17 can display an image 7 of a portion of the fuel tank desired by the user on the display device 41 and add the image 7 to the report. That is, for example, when the user selects one of the rust locations displayed on the GUI, the report generation unit 17 can obtain the image 7 of the rust location selected by the user from the database 10 and display it on the GUI. The report generation unit 17 can add the displayed image 7 to the position desired by the user in the report.

Figure 8 is an overall flow diagram of the processing performed by the visual inspection support system 1.

In step S1, the image acquisition unit 11 acquires multiple images of one or more inspection objects, for example, one or more fuel tanks, from any one of the user terminals 4, and registers the multiple acquired images of each fuel tank in the database 10 in association with each fuel tank.

Then, in step S2, upon receiving an object selection request from any of the user terminals 4, the development drawing generating unit 14 selects one inspection object, for example one fuel tank, from one or more inspection objects registered in the database 10. Then, in step S3, the development drawing generating unit 14 determines whether or not an development drawing of the selected fuel tank is already present in the database 10. If the result of this determination is Yes, control proceeds to step S5, and if the result is No, control proceeds to step S4.

In step S4, the development drawing generating unit 14 creates a development drawing of the selected fuel tank. At that time, the 3D model generating unit 12 or the design drawing acquiring unit 13 also participates in the development drawing creation as necessary. The development drawing created here may be a colored development drawing in which the color of the image of the fuel tank is projected. After that, control proceeds to step S5.

In step S5, the condition detection unit 15 detects abnormal areas, for example rusted areas, from, for example, a colored development of the fuel tank being inspected. Then, in step S6, the condition detection unit 15 projects the detected abnormal areas, for example rusted areas, onto the development of the fuel tank (for example, attaches a rust mark to the rusted area on the development). The development onto which the rusted areas are projected (with the rust mark attached) is the aforementioned detection result development.

The detection of abnormalities in step S5 may be performed before the creation of the development diagram in step S4. In that case, the state detection unit 15 can detect abnormalities based on the images 7 of the fuel tank or based on the three-dimensional model 121 onto which these images 7 are projected.

Then, in step S7, the development diagram editing unit 16, in response to a request from any of the user terminals 4, displays the detection result development diagram on that user terminal 4, allowing the user to confirm the automatic inspection results. Then, in response to user operations input to the user terminal 4, the development diagram editing unit 16 edits the detection result development diagram and corrects it if necessary. Furthermore, the development diagram editing unit 16 performs quantitative calculations such as the area and area ratio of rusted areas based on the detection result development diagram that no longer requires further correction.

Then, in step S8, in response to a request from any of the user terminals 4, the report generation unit 16 creates a report showing the inspection results of the fuel tank using a development diagram of the detection results of the fuel tank and, if necessary, images of specific locations and other registered data.

Figure 9 shows the flow of creating an unfolded view in step S4 of Figure 3.

In step S11, the user selects a creation method from the user terminal 4 for the development drawing generating unit 14. In step S12, it is determined whether the selected creation method is manual or automatic, and if it is the former, control proceeds to step S13.

In step S13, the user specifies the specifications of the simplified model from the user terminal 4 to the development drawing generation unit 14. The specifications include shape parameters for specifying the three-dimensional shape of the simplified model, such as the shape type (e.g., cylinder, sphere, rectangular solid, cone, etc.) and dimensions (e.g., height, radius, length, width, etc.). Then, in step S15, a simplified model is created based on the specified shape parameters. Then, in step S14, the development drawing generation unit 14 creates a simplified model of the fuel tank based on the specified specifications.

If automatic creation of a development view is selected in step S12 above, control proceeds to step S15, where the 3D model generation unit 12 generates a large number of 3D point clouds of the fuel tank to be inspected. Here, the 3D point cloud of the fuel tank means the 3D position data of each of the many characteristic points of the fuel tank. Then, in step S16, the 3D point clouds are connected in a mesh shape to generate a 3D model (3D mesh model) 121 (see Figures 3, 4, and 5) of the fuel tank.

Then, in step S17, a simplified model 122 of the fuel tank (see Figures 3, 4, and 5) is generated.

The three-dimensional shape of the simplified model 122 is not particularly limited as long as it can be expanded into a two-dimensional development, but it is preferable that it can be expanded into a development structure that allows the user to easily grasp the positional relationship between the actual object to be inspected and the development (for example, a set of a side view and a plan view, or a set of a front view, a right side view, a left side view, a back view, and a plan view, etc.). For example, the shape of the simplified model 122 may be a shape of the three-dimensional shape of a fuel tank in which small shape elements that do not significantly affect the inspection purpose have been simplified or omitted. The shape of the simplified model 122 may also be a geometric shape such as a cylinder, a rectangular parallelepiped, a cone, a polygonal pyramid, a sphere, or a combination of these.

After the simplified model 122 of the fuel tank is generated in step S14 or S17, a color based on the image 7 of the fuel tank is applied to the simplified model 122 in step S18. Then, in step S19, the colored simplified model 122 is converted into a two-dimensional development 141 (see Figures 3, 4, and 5).

The development 141 generated in step S18 may include auxiliary information indicating the positional or directional relationship between the development 141 and the object of inspection. For example, the development 141(1) of the fuel tank shown in FIG. 3, which is simplified into a cylindrical shape, includes auxiliary information indicating the directions of the fuel tank, such as 0° (front), 90° (left side), 180° (rear), and 270° (right side).

Figure 10 shows the flow of 3D point cloud generation in step S15 of Figure 9.

In step S21, the three-dimensional model generation unit 12 reads multiple images 7 of the fuel tank from the database 10. Then, in step S22, the three-dimensional model generation unit 12 extracts a large number of pixels corresponding to the characteristic points of the fuel tank from the images 7. The three-dimensional model generation unit 12 may extract points of parts of the fuel tank that have a characteristic shape, color, or pattern as the characteristic points. For example, the three-dimensional model generation unit 12 may extract points of steps, convex parts, concave parts, windows, etc. provided on the fuel tank as the characteristic points.

Then, the three-dimensional model generating unit 12 calculates the exterior orientation parameters and the interior orientation parameters from the multiple images 7 of the fuel tank (S23). The three-dimensional model generating unit 12 calculates the three-dimensional positions of the multiple feature points of the fuel tank (i.e., a three-dimensional point cloud) based on the pixels corresponding to the multiple feature points extracted from the images 7 and the exterior orientation parameters and the interior orientation parameters (S24).

The method of generating the three-dimensional model 121 is not limited to the above-mentioned method. The three-dimensional model generating unit 12 may generate the three-dimensional model 121, for example, by using a neural network that has learned a method of generating the three-dimensional model 121 from the image 7.

Figure 11 shows the flow of generating a simplified model in step S17 of Figure 9.

In step S31, the development drawing generating unit 14 determines whether or not there is a design drawing for the fuel tank to be inspected in the database 10. If the determination result is Yes, the design drawing acquiring unit 13 reads the design drawing for the fuel tank from the database 10 (S32). If the determination result is No, the development drawing generating unit 14 reads the 3D model of the fuel tank from the database 10 (S33).

Then, the development drawing generating unit 14 determines the specifications of the simplified model, such as the shape type and dimensions, based on the design drawings or 3D model of the fuel tank (S34). The development drawing generating unit 14 may determine the shape type of the simplified model depending on the shape or type of the inspection object. The development drawing generating unit 14 may also select specific specifications from among different types of specifications that are set in advance, depending on the inspection object. The development drawing generating unit 14 then generates a simplified model based on the determined specifications (S35).

Figure 12 shows the flow of the simplified model coloring process in step S18 of Figure 9.

In step S41, the development drawing generating unit 14 reads multiple images 7 of the fuel tank to be inspected from the database 10, and obtains the external and internal evaluation elements of those images 7. If a three-dimensional model of the fuel tank has already been created, the external and internal evaluation elements of each image 7 have already been calculated and registered in the database 10 in step S23 of FIG. 10, so it is sufficient to read them. On the other hand, if the external and internal evaluation elements of the image 7 have not yet been calculated, they can be calculated in the same manner as steps S21 to S23 of FIG. 10.

Then, in step S42, the development drawing generating unit 14 associates all coordinate points on the surface of the simplified model with pixels of one or more images 7 in which each coordinate point appears, based on the external and internal orientation elements of each image 7. In other words, it identifies which pixel of which image 7 each coordinate point of the simplified model appears in. In most cases, each coordinate point appears in pixels in two or more different images 7, so two or more pixels are identified as the pixel corresponding to each coordinate point.

Then, in step S43, the development drawing generating unit 14 selects, for each coordinate point on the surface of the simplified model, one corresponding pixel from among the two or more corresponding pixels identified in step S42, which corresponds to the coordinate point photographed at a position closest to the front of the camera. This is because it is determined that the result of photographing the coordinate point with the camera positioned directly in front of it most accurately represents the color of the coordinate point.

Then, in step S44, the development drawing generating unit 14 sets the pixel value of the corresponding pixel selected in step 43 to the color of that coordinate point of the simplified model. In step 45, the processing of steps S43 and S44 is performed for all coordinate points of the simplified model, thereby setting the color of all coordinate points of the simplified model. This results in a colored simplified model.

Figure 13 shows the flow of abnormality detection in step S5 of Figure 8.

In step S51, the user selects an analysis method for anomaly detection for the state detection unit 15 through the user terminal 4. If the selected analysis method is machine learning (S52, Yes), control proceeds to step S53, and in step S15, the state detection unit 15 detects anomalies such as rusted areas in the fuel tank using a previously prepared trained anomaly detection model (for example, a neural network that has learned a method for detecting anomalies such as rusted areas based on the surface color of the fuel tank to be inspected). At this time, for example, the colored development diagram 141 or the colored simplified model 122 is input to the anomaly detection model, and the anomaly detection model detects areas of anomalies such as rusted areas from the surface of the colored development diagram 141 or the colored simplified model 122.

In step S52, if the selected analysis method is image processing (S52, No), the state detection unit 15 converts the color data of the colored development diagram 141 or the colored simplified model 122 into color data in a predetermined color space suitable for detecting abnormalities by image processing, as necessary. For example, when the original color data is in the RGB system, this is converted into color data in the HSV (Hue, Saturation, Brightness) system or the HLS (Hue, Lightness, Saturation) system (S54).

Then, in step S55, the condition detection unit 15 detects abnormalities such as rust by evaluating the color data at each position of the colored development diagram 141 or the colored simplified model 122 in a multidimensional feature space that has been prepared in advance to identify abnormalities such as rust.

The visual inspection support system 1 according to the embodiment described above is equipped with a development diagram editing unit 16 and a report generating unit 17, and is therefore capable of displaying a report based on quantitative information of predetermined state locations, such as rusted locations, on the display device 41. This allows the visual inspection support system 1 to accurately express the quantitative information of predetermined state locations in the report using drawings and text. As a result, the visual inspection support system 1 can support object inspection that is more suited to actual requirements.

The visual inspection support system 1 according to one embodiment includes an unfolded view editing unit 16, which allows the user to edit the unfolded view 121 on the GUI. This allows the visual inspection support system 1 to include in the report an explanatory unfolded view 153 that is more easily visible to the user.

The visual inspection support system 1 according to one embodiment includes a development view editing unit 16, which allows the user to edit the results of automatic detection such as rust marks 151. This allows the visual inspection support system 1 to include more accurate inspection results in the report.

The visual inspection support system 1 according to one embodiment includes a report generating unit 17, which allows an image showing the part of the subject of inspection desired by the user to be displayed on the GUI. This improves the convenience of the visual inspection support system 1.

The appearance inspection support system 1 according to one embodiment is equipped with an image acquisition unit 11, and is therefore capable of acquiring an image 7 from at least one of the drone 2 or the portable camera 3. This allows the range of the appearance of the inspection object that cannot be photographed by the drone 2 to be supplemented by the image 7 from the portable camera 3. As a result, the appearance inspection support system 1 can support the inspection of objects that are more suited to real-world limitations.

The visual inspection support system 1 according to one embodiment can be applied to visual inspection of various types of artificial or natural inspection objects other than fuel tanks.

Although one embodiment of the present invention has been described above, this is merely an example for explaining the present invention, and the present invention can be embodied in various different forms. For example, the condition detection unit 15 may detect abnormalities such as rusted areas based on the colored three-dimensional model 121 from the three-dimensional model generation unit 12 or the image 7 of the inspection target, instead of a colored development view or a colored simplified model.

For example, the development drawing generating unit 14 may generate the development drawing 141 based on the design drawing of the inspection object instead of the simplified model 122. In this case, the development drawing generating unit 14 can create a colored development drawing by associating each coordinate point of the development drawing created from the design drawing with one or more pixels in the multiple images 7 of the inspection object and setting the color of each coordinate point based on the pixel value of the one or more corresponding pixels.

For example, the development drawing generating unit 14 may generate the development drawing 141 based on the design drawing of the inspection object instead of the simplified model 122. In this case, the development drawing generating unit 14 can create a colored development drawing by associating each coordinate point of the development drawing created from the design drawing with one or more pixels in the multiple images 7 of the inspection object and setting the color of each coordinate point based on the pixel value of the one or more corresponding pixels.

1...Appearance inspection support system, 11...Image acquisition unit, 12...3D model generation unit, 13...Design drawing acquisition unit, 14...Unfolded drawing generation unit, 15...State detection unit, 16...Unfolded drawing editing unit, 17...Report generation unit, 2...Drone, 21...Flying device, 22...Position measurement device, 23...Orientation measurement device, 24...Camera, 3...Mobile camera, 31...Photographing device, 32...Position measurement device, 33...Orientation measurement device, 4...User terminal, 41...Display device

Claims (8)

an image acquisition unit that acquires a plurality of images in which an object is displayed;
a development generating unit for generating a colored development of the object onto which the colors of the plurality of images are projected;
a state detection unit that detects a predetermined area on the development view in which a predetermined state of the object appears based on the image;
means for outputting a display of the development view and information regarding the predetermined area ;
a three-dimensional model generating unit that generates a three-dimensional model of the object based on the plurality of images,
the development generating unit generates a colored development onto which the images are projected, based on the plurality of images and the three-dimensional model;
The state detection unit detects the predetermined area based on the colored development view.
Visual inspection support system. Further, the visual inspection support system includes a development view editing unit that displays a plurality of two-dimensional views constituting the development view in an adjustable manner with respect to relative positions.
2. The visual inspection support system according to claim 1. The visual inspection support system further includes a report generating unit that generates a report explaining the predetermined area based on the development view.
3. The visual inspection support system according to claim 1 or 2. means for calculating quantitative information of the predetermined region,
The report generation unit generates the report based on the quantitative information.
The visual inspection support system according to claim 3 . the state detection unit marks a detection result of the predetermined area on the development view;
The development diagram editing unit displays the mark in a manner that allows the mark to be added or deleted.
The visual inspection support system according to claim 2 . means for generating a simplified model by simplifying the three-dimensional model;
The development drawing generating unit generates the development drawing based on the simplified model.
2. The visual inspection support system according to claim 1 . The image acquisition unit acquires the image captured by at least one of a moving body equipped with a camera and a portable photographing device.
The visual inspection support system according to any one of claims 1 to 6 . Acquiring a number of images in which the object is displayed;
generating a colored development of the object onto which the colors of the plurality of images are projected;
detecting a predetermined area on the development where a predetermined state of the object appears based on the image;
Displaying the development and information about the predetermined area ;
generating a three-dimensional model of the object based on the plurality of images;
generating the colored development onto which the images are projected based on the plurality of images and the three-dimensional model;
Detecting the predetermined area based on the colored development view.
A method for supporting visual inspection.

Images