JP7406740B2 - System for inspecting objects and creating reports - Google Patents

Description

The present invention relates to a computer-based system for supporting the work of inspecting, inspecting, or evaluating an object using images and creating a report describing the results.

For the purpose of surveying and maintenance inspection of man-made facilities and natural objects, the target land or facility is photographed using a camera mounted on a moving vehicle above the site or on the ground, or carried by a person, and a large number of images are obtained. 2. Description of the Related Art Work is being done in various industrial fields in which people visually inspect or evaluate objects and create reports that express the results using a combination of photographed images and accompanying marks and text comments.

Systems for creating reports are disclosed in, for example, Patent Document 1 and Patent Document 2.

JP2005-352848A Japanese Patent Application Publication No. 2007-34453

For example, when inspecting the condition of a vast object such as a solar power generation facility based on images taken by a camera mounted on an unmanned aerial vehicle, the number of images taken is larger than the total number of images taken for the entire object. It becomes very large. It is not an easy task for humans to grasp the entire picture of an object by looking at such a large number of images, or to inspect individual parts in detail.

Furthermore, in order to create a report that allows others to easily understand the inspection results, it is necessary to skillfully arrange many images on the report and add marks and comments to those images. It is. Creating reports takes a lot of time. For example, it is said that it usually takes about a week to inspect a solar power generation facility and create a report based on images taken from an unmanned flying vehicle.

However, the conventional techniques disclosed in Patent Documents 1 and 2 are not aimed at solving the above problem.

One object is to provide a computer-based system that supports the task of inspecting an object using a large number of images obtained from the object, or the task of compiling the inspection results into a report.

In accordance with one embodiment, a system for inspecting an object and generating a report includes means for communicating with a user terminal, means for providing a plurality of images, each representing an entire object and a different portion of the object; means for displaying a work screen displaying a requested image and a comment input tool on the user terminal; automatically creating a report in accordance with the report form using means for storing it in association with an image, means for storing a report form, the saved comment and the image associated therewith; and means.

1 shows the overall structure of an inspection report support system according to an embodiment. The basic communication flow between the data processing server and the user in the same system is shown. The flow of inspection processing performed by the data processing server is shown. An example of a screen display for allowing a user to view an entire image of an object to be inspected in an inspection process is shown. An example of a screen display is shown to allow a user to divide an inspection target into a plurality of sub-areas in an inspection process. An example of a screen display for allowing a user to inspect a portion to be inspected in detail during inspection processing is shown. An example of a screen display is shown for allowing the user to list the points of interest of the object to be inspected and check the inspection results of each point in the inspection process. The flow of comparison test processing performed by the data processing server is shown. An example of a screen display for allowing a user to perform an inspection by comparing and observing two images of a target in an inspection process is shown. The flow of report creation processing performed by the data processing server is shown. A portion of an example report automatically created by the data processing server is shown.

FIG. 1 shows the overall structure of an inspection report support system according to one embodiment.

As shown in FIG. 1, the inspection report support system 1 has a data processing server 3, and the data processing server 3 may be configured by one or more server computer machines. The data processing server 3 provides each of one or more users with a service that helps them inspect one or more objects and compile the inspection results into a report. The data processing server 3 can perform data communication via the web server 5 with a web browser of an information processing terminal (hereinafter referred to as user terminal) 7 such as a personal computer or a smartphone used by each user. In FIG. 1, only one user terminal 7 is illustrated for convenience, and the details of this system will be described below by taking as an example a case where one user uses the data processing server 3.

The data processing server 3 includes a processing request reception section 11, a data storage/search section 13, an image registration section 15, an image combination section 17, a map/image display section 19, a comment registration section 21, a report creation section 23, and a report display section. It has processing components such as a section 25, a report editing section 27, and a report output section 29.

The processing request receiving unit 11 receives various processing requests from users (for example, various processing requests for performing an inspection, various processing requests for creating a report, etc.), and performs the requested processing. Pass the request to a processing component that does the following:

The data storage/retrieval unit 13 stores data such as maps, images, test results, and reports of various inspection targets, and stores data such as maps, images, test results, and reports of various inspection targets, and stores data such as maps, images, test results, and other data in response to data requests from other processing components. Read data and pass it to its processing component.

The image registration unit 15 receives images of the inspection object from the user terminal 7 (including not only general visible light photographic images, but also photographic images using non-visible light such as infrared rays, and other types of images). (may also be included), and register those images in the data storage/retrieval unit 13 in association with the inspection object and the user. Usually, many images are registered for one inspection object. For example, if a camera mounted on an unmanned aerial vehicle takes a picture of a vast object such as a solar power generation facility, many high-quality photographic images of various parts of the facility will be associated with the solar power generation facility. will be registered.

The image combining unit 17 generates various images (hereinafter referred to as inspection images) that the user will observe to inspect the object by combining a large number of images of each registered inspection object. . The inspection image is stored in the data storage/retrieval section 13.

Images with different resolutions at multiple levels from the lowest resolution to the highest resolution may be created as inspection images of the object. For example, relatively low-resolution images can be used to provide an overview of the entire object, relatively high-resolution images can be used to examine individual parts of the object in detail, and images with resolutions in between can be used for inspection. Can be created as an image. The format of the inspection image and provision to the user can be performed, for example, in accordance with WMTS (Web Map Tile Service) or WMS (Web Map Service). The inspection image may be an orthoimage or a mosaic image. The inspection image of the ortho image may be created using SfM (Structure from motion).

The map/image display unit 19 sends a map or an inspection image of the inspection object to be displayed on the screen of the user terminal 7 to the Web server 5 in response to a processing request from the user. The web server 5 sends the work screen incorporating the map or inspection image to the user terminal 7 and displays it on the screen of the web browser.

The comment registration unit 21 receives comments, etc. (text, options, marks, figures, etc.) input by the user into the user terminal 7 when inspecting an object, and records them as inspection result data associated with the object. , is saved in the data storage/search section 13.

In response to a request from a user, the report creation unit 23 creates an inspection report for each inspection target using data such as maps, inspection images, and test results for each inspection target stored in the data storage/retrieval unit 13. automatically created. The automatically created report is stored in the data storage/search unit 13 in association with the subject.

The report display unit 25 sends the automatically created inspection report to the Web server 5 in response to a request from a user. The web server 5 sends the report to the user terminal 7 and displays it on the screen of the web browser.

The report editing section 27 allows the user to freely edit the report displayed on the screen of the user terminal 7. The report edited by the user is stored in the data storage/search unit 13 in association with the subject.

The report output unit 29 sends the edited report to the Web server 5 in response to a request from a user. The web server 5 allows the user to download the report to the user terminal 7.

FIG. 2 shows the basic flow of communication between the data processing server 3 and the user 9 in the inspection report support system 1.

In FIG. 2, in step 31, the user 9 registers an image of a certain inspection object (usually there are many images) in the data processing server 3. In step 33, the data processing server 3 saves the registered images, and in step 35, the saved images are combined to create inspection images of different resolutions of the inspection object.

In step 37, the user selects a desired inspection image (for example, an image for overlooking the entire object, or an image for observing a specific part of the object in detail) in order to inspect the object. request. In response to the request, in step 39 the data processing server 3 selects and reads the requested inspection image from among the saved inspection images, and in step 41 displays the inspection image on the user terminal 7. .

In step 43, the user views and examines the displayed inspection image, makes a necessary judgment regarding the inspection object, and in step 45, the user makes comments (such as text that describes the judgment result or opinion) in accordance with the judgment result. input options, graphic marks, etc.). Then, in step 47, the data processing server 3 stores the examined image and the input comment etc. in association with each other.

Steps 37 to 47 may be repeated multiple times in the inspection process for one inspection target. For example, the user first looks at the entire object and enters comments, etc., then looks at detailed images of each part of the object, inspects that part, and enters comments, etc. The test will be repeated until the test is completed. Steps 37 to 47 are performed each time the repetitive work is performed.

Thereafter, in step 49, the user requests a report summarizing the test results for the test target. Upon receiving the request, the data processing server 3 automatically creates a report using the stored test results in step 51, and displays the report on the user terminal 7 in step 53.

In step 55, the user confirms the automatically created report, and in step 57, the user makes desired edits to the automatically created report and sends the edited report to the data processing server 3. In step 59, data processing server 3 saves the edited report and in step 61 outputs the report. At step 63, the user downloads the output report.

FIG. 3 shows the flow of inspection processing (processing corresponding to steps 39, 41, and 47 in FIG. 2) performed by the data processing server 3.

As shown in FIG. 3, upon starting the inspection process, the data processing server 3 performs steps S1, S2, and S3 in parallel. In step S1, it is determined whether an image display request has been received from the user and what kind of area, such as the entire inspection object or a specific portion, of the image has been requested to be displayed from the user. In step S2, it is determined whether the user has requested zooming or scrolling of the displayed image. In step S3, it is determined whether the image corresponds to an area selected by the user within the displayed image.

When it is determined in step S1 that display of an image of a certain area of a certain inspection target is requested, the data processing server 3 prepares an image of the area of the inspection target in step S4, and displays the image. Display the incorporated work screen on the screen of the user terminal 7. For example, if a user requests the display of an image to provide an overview of the entire inspection object, a work screen such as the one illustrated in FIG. Image 71 is incorporated. For example, if a user requests an image to examine a specific part of the object in detail, a work screen like the one shown in Figure 6 is displayed, which shows a narrow area of the specific part. image 91 is incorporated.

Referring again to FIG. 3, if it is determined in step S2 that zooming (enlargement or reduction) or scrolling of the display image is requested, the data processing server 3, in step S5, Zoom or scroll. For example, on the work screen illustrated in FIG. 4, the user can zoom in (enlarge) the image 71 showing the entire object to view a portion of the object in higher resolution and in more detail. Alternatively, for example, on the work screen illustrated in FIG. 6, the user may scroll through the image 91 showing a particular portion of the object to view other portions of the object.

Referring again to FIG. 3, if it is determined in step S3 that a certain area within the image displayed on the work screen has been selected by the user, data processing server 3 selects the selected area in step S6. Displays the outline of the area on the image. Furthermore, in step S7, the data processing server 3 provides one or more tools (hereinafter referred to as comment input tools) for the user to input comments such as the area name and test results of the selected area. It is displayed on the work screen via the server 5. For example, on the work screen shown in FIG. 4, an outline frame 75 of the area selected by the user (in this example, the selected area is the entire site of the solar power generation facility to be inspected) is displayed on the image. At the same time, comment input tools 76 and 77 for inputting an area name and an arbitrary comment regarding the selected area are also displayed.

Referring again to FIG. 3, when the user inputs the area name of the area or a comment regarding the area using the displayed comment input tool, the data processing server 3 inputs the input in step S8. Display the area name or comment on the work screen. For example, on the work screen illustrated in FIG. 4, the input area name is displayed in the comment input tool 76, and the input comment is displayed in the comment display field 79 along with the time stamp at the time of input. has been done.

Referring again to FIG. 3, in step S9, the data processing server collects the image displayed on the work screen, the position coordinates of the area (or area frame) selected by the user, the area name and comment input by the user. , is stored in association with the selected area as a data element constituting the inspection result of the selected area.

4 to 7 show some examples of work screens for inspecting an object that are displayed on the screen of the user terminal 7 by the inspection process shown in FIG. 3. In the examples shown in FIGS. 4 to 7, a certain solar power generation facility is the subject of inspection.

FIG. 4 shows an example of a work screen that allows the user to view the entire solar power generation facility to be inspected and select the entire area of the site.

As shown in FIG. 4, the image 71 displayed on the work screen is a relatively low-resolution image showing the entire solar power generation facility. The user can zoom in on the displayed image to display higher resolution images to view solar power facility details.

The user can use the area selection tool 73 on the displayed image 71 to select the inspection target, that is, the entire site of the solar power generation facility (typically, by drawing an outline of the entire site). Then, an outline frame 75 of the selected area is displayed on the image 71. At the same time, tools 76 and 77 (comment input tools) for inputting an area name and a comment for the selected area are displayed on the work screen.

The user can use the input tool 76 to input the area name of the selected area (for example, "A Prefecture, B City, C Town, E Facility"), and also use the input tool 77 to enter the inspection results and explanation of the selected area. You can enter any comments you like. When the user clicks the send button on the input tools 76 and 77, the input area name and comment are sent to the data processing server 3 and are associated with the display image 71 and the position coordinates of the selected area (or area frame). and saved. In the example of FIG. 4, in addition to the comment input tools 76 and 77, the comment display tool 79 also displays comments that were previously input and saved for the selected area.

When the user double-clicks the outline frame 75 of the inspection target on the work screen as illustrated in FIG. A work screen for inspection while investigating will be displayed on the screen.

FIG. 5 shows a display example of a work screen that allows setting one or more sub-areas within an already selected arbitrary area.

The work screen illustrated in FIG. 5 exemplifies a case where a plurality of sub-areas are set within the area of the entire site of the solar power generation facility selected in the screen example of FIG. 4 . That is, when the user draws the outline of each sub-region using the area selection tool 73, the drawn outline frames 81A and 81B are displayed on the display image 75. At the same time, comment input tools 83A, 85A, 83B, and 85B for inputting area names and comments for the respective subareas are also displayed on the work screen. When the user inputs the names of the respective regions (for example, "Archive A" and "Archive B") using the input tools 83A and 83B, the respective region names are displayed in the outline frame 81A of the respective sub-regions on the display image 75. , 81B. Further, the user inputs comments regarding each area using the input tools 85A and 85B, and these comments are displayed on the input tools 85A and 85B, respectively. In the example of FIG. 5, comments input in the past regarding the sub-area named "A Ward" are also displayed in the comment display field 87A.

The user can additionally set a new sub-area. The comments, etc. input tool of the added sub-area is added below the comment, etc. input tool 85B at the bottom position shown in FIG. The user can access any input tool such as comments by scrolling the group of input tools such as comments on the work screen 75.

When the user double-clicks the outline frame 81A, 81B of an arbitrary sub-area or its area name on the work screen as illustrated in FIG. 5, or specifies an arbitrary sub-area using another predetermined method, the A work screen will be displayed on the screen for inspecting each part of the designated sub-area in detail.

FIG. 6 shows an example of a work screen for performing an inspection while examining each part to be inspected in detail.

The example shown in FIG. 6 is a work screen for inspecting in detail each part of the sub-area "Archive A" shown in FIG. 5. On this work screen, a relatively high-resolution image 91 showing in detail the state of a part of the A area is displayed. In addition, an auxiliary image 95 is also displayed on the work screen, showing in a frame 94 which part of the inspection object the area represented by the display image 91 corresponds to.

When the user selects a noteworthy point on the display screen 91 using the area selection tool 73 (for example, by drawing a frame surrounding that point), the frame 91 of the noted point is displayed on the display screen 91. be done. At the same time, a mark 96 is displayed on the auxiliary image 95 indicating where the point of interest is located on the inspection object. Furthermore, at the same time, tools (comment input tools) 97, 99, 101, 103 for inputting the location number, location information, maintenance urgency, condition evaluation comments, etc. regarding the point of interest are also displayed on the work screen. Is displayed.

Here, the location number may be automatically set, or the user may be able to input an arbitrary number. As for the degree of urgency, a standard level such as "high," "medium," or "low" may be selected from a menu, for example. Further, as for comments for condition evaluation, it may be possible to select one of several standard comments from a pull-down menu 105 as illustrated in FIG. 5, and in addition, it may be possible to input free text. It's fine.

When the location number (for example, "(1)") of the point of interest is set or input, that point number is displayed on the display screen 91 in association with the frame 93 of the point of interest.

FIG. 7 shows a display example of a work screen for displaying a list of already inspected points of interest within the inspection target and allowing confirmation of the inspection results of each point of interest.

As shown in FIG. 7, place marks 113 indicating respective points of interest are displayed on an image 71 that provides an overview of the entire inspection target. When the user specifies an arbitrary point mark 113 with the cursor 111, an image 115 showing the detailed state of the corresponding point of interest and an inspection result 117 of the point of interest are displayed on the work screen.

FIG. 8 shows the flow of comparison test processing that can be performed by the data processing server in the above-described test processing process.

Comparative inspection processing is processing that displays a plurality of images of different inspection targets side by side on a work screen, and allows the user to compare and examine these images to perform an inspection. The multiple images displayed include, for example, images taken of the same part of the inspection target at different times, or images taken of the same part using different methods (for example, a visible light photographic image and an infrared photographic image). obtain.

As shown in FIG. 8, the data processing server 3 displays a plurality of images requested by the user in parallel on the work screen in step S21.

Thereafter, in step S22, when the user requests zooming or scrolling of any of the displayed images, in step S23, the plurality of images displayed in parallel on the work screen are synchronized to have the same zoom magnification. Or, it is zoomed or scrolled only in the scrolling direction and distance.

Further, when the user draws a frame indicating an arbitrary area on any of the displayed images in step S24, the area frame is displayed on the plurality of images in step S25, and in step S26, the area frame is displayed on a plurality of images. A tool for inputting inspection results and the like regarding the area (comment etc. input tool) is displayed on the work screen.

Further, in step S27, when the user inputs a comment etc. using the comment input tool, the input comment etc. is displayed on the work screen, and in step S28, the input comment etc. and the area (or The coordinates of the frame) and the plurality of images on the work screen are associated with each other and saved as inspection result data.

In step S27, when the user requests the end of the test, the comparison test process ends.

FIG. 9 shows a display example of a work screen for comparative inspection processing.

On the work screen illustrated in FIG. 9, two images 121 and 123 representing the same part of the inspection target are displayed side by side. For example, one image 121 is an infrared photographic image and the other image 123 is a visible light photographic image, and the images displayed side by side may be images taken at different times using the same or different sensors. It's okay.

When the user zooms or scrolls one image (eg, image 121), both images 121, 123 are similarly zoomed or scrolled at the same time.

Furthermore, when the user draws a region frame on one image (for example, image 121) using the region selection tool 73, the region frames 125 and 127 are displayed on both images 121 and 123. In addition, a comment etc. input tool 129 regarding the area is also displayed.

FIG. 10 shows the flow of report creation processing performed by the data processing server 3.

As shown in FIG. 10, in step S31, a previously saved report form (report structure (for example, what titles are arranged in what order, what images are included in each item, etc.) data that defines the layout in which text comments are arranged, etc.) is read out. Here, report forms can be prepared in several different ways. One method is for the user to define the structure of the report through programming. Another method is to use an existing printed report image or document file (e.g., an image file obtained by scanning the report with an image printer, or a report document file created with a word processor, etc.). ) is sent from the user terminal 7 to the data processing server 3, and the data processing server 3 analyzes the file and automatically creates a report form.

At step 32, uncreated items are extracted from among the items to be reported defined in the report form. Then, in step S32, images, area coordinates, area names, comments, etc. corresponding to the constituent elements of each uncreated item are read out from the saved inspection result data, and in step S34, these are read out. Based on the output data, images, area names, area frames, comments, etc. are arranged and written in the corresponding item part of the report according to the layout defined in the report form. A report is created by repeating steps S33 and S34 until there are no more uncreated items.

Thereafter, in step S35, the report is saved as a file in a format that can be edited later by the user.

FIG. 11 shows a portion of an example report automatically created by the process shown in FIG.

As shown in Figure 11, the automatically created report includes the display image, area frame on the image, area name, input comment, etc. used on the work screen as illustrated in Figures 4 to 7. Written according to the order and layout defined in the report form. The user can complete the report by downloading such an automatically created report file to the user terminal 7 and editing it as necessary.

Although one embodiment has been described above, the scope of the present invention is not limited to this embodiment, and includes, for example, area frames, area names, input comments, etc. on images included in automatically generated reports. may be automatically created by artificial intelligence or the like, and may be implemented in various other forms without departing from the gist thereof.

1 Inspection report support system 3 Data processing server 7 User terminal 11 Processing request reception unit 13 Data storage/search unit 15 Image registration unit 17 Image combination unit 19 Map/image display unit 21 Comment registration unit 23 Report creation unit 25 Report Display section 27 Report editing section 29 Report output section

Claims (1)

In the system for inspecting objects and creating reports,
means for communicating with a user terminal used by the user;
means for preparing a plurality of images each representing the entire object and different parts of the object;
means for displaying on the user terminal a work screen displaying the image and comment input tool requested by the user;
means for receiving a comment input by the user on the work screen and storing the comment in association with the image displayed on the work screen;
a means for storing a report form;
means for automatically creating a report according to the report form using the saved comment and the image associated therewith;
When the user requests display of a plurality of images, the display means displays the plurality of images requested by the user on the work screen, and displays the plurality of images displayed on the work screen. When any one of zooming or scrolling is requested by the user, synchronously zooming or scrolling the plurality of images displayed on the work screen;
When the user specifies an arbitrary area on any one of the plurality of images displayed on the work screen, the display means displays each of the plurality of images displayed on the work screen. display the specified area in
When the user inputs a comment using the comment input tool regarding the specified area of the work screen where the plurality of images are displayed, the storing means saves the input comment and the specified area. associate and save the coordinates of and the plurality of images on the work screen ;
The means for automatically creating the report reads the comment, the coordinates of the specified area, and the image that are stored in association with each other, and reads the comment, the coordinates of the specified area, and the image. arranging and describing the comment, the frame of the specified area, and the image in the report according to the report form based on the image;
A system configured as follows.

Images