AU2018100324B4

AU2018100324B4 - Image Analysis

Info

Publication number: AU2018100324B4
Application number: AU2018100324A
Authority: AU
Inventors: Simon Fonteyn
Original assignee: Lis Pty Ltd
Current assignee: Lis Pty Ltd
Priority date: 2017-12-18
Filing date: 2018-03-15
Publication date: 2018-07-19
Anticipated expiration: 2026-03-15
Also published as: AU2018100324A4; WO2019119030A1

Abstract

Abstract Described is an image analysing system which includes a conversion module, an indicia recognition module, a classifier, and a feature locator. The conversion module provides an image that includes segmented data including a first set of indicia. The indicia recognition module recognises indicia in the first set of indicia, and the indicia recognition module generates a set of resolved indicia and a set of unresolved indicia. The classifier classifies resolved indicia to find at least one feature that includes at least one indicium from the set of resolved indicia. The feature locator determines at least one indicia location in the image associated with one or more indicia in the set of unresolved indicia.

Description

Technical Field [0001] The present disclosure relates to an image analysing system and a method of analysing an image that includes both text and images.

Background [0002] Images in electronic format can be challenging to analyse, navigate and/or extract information from. If there is information in an image, there are limited tools available to find that information, or to search the image for the information. Existing optical character recognition (OCR) technology is able to find text within images, but does not always provide an accurate result. Also, there are limited tools with which to navigate through or edit images that have been OCR-ed. If the same type of information is required from different types or styles of images, it can be difficult for a user to find that information when visually inspecting the images.

[0003] Any discussion of documents, acts, materials, devices, articles or the like which has been included in the present specification is not to be taken as an admission that any or all of these matters form part of the prior art base or were common general knowledge in the field relevant to the present disclosure as it existed before the priority date of each claim of this application.

Summary [0004] In one aspect there is provided an image analysing system which includes: a conversion module that converts an image to segmented data including a first set of indicia and a first set of images; an image recognition module that selects a first image subset from the first set of images and extracts a second set of indicia from the first image subset; an indicia recognition module that recognises indicia in the first set of indicia and indicia in the second set of indicia, wherein the indicia recognition module

2018100324 15 Mar 2018 generates a set of resolved indicia and a set of unresolved indicia; a classifier that classifies resolved indicia by: comparing the set of resolved indicia with a classification framework, and extracting at least one feature that includes at least one indicium from the set of resolved indicia; and a feature locator that determines at least one indicia location in the image associated with one or more indicia in the set of unresolved indicia, wherein the feature locator bookmarks the at least one indicia location with an indicia bookmark.

[0005] The system may further include a user interface enabling a user to: access the at least one indicia location via the indicia bookmark, and manipulate the unresolved indicia to form resolvable indicia.

[0006] The classifier may classify the resolvable indicia to extract at least one further feature from the resolvable indicia.

[0007] The image recognition module may select a second image subset from the first set of images, the feature locator may determine at least one image location in the image associated with one or more images in the second image subset, and the feature locator may bookmark the at least one image location with an image bookmark.

[0008] The user interface may further enable the user to: access the at least one image location via the image bookmark, and manipulate the one or more images in the second image subset to form further resolvable indicia.

[0009] The classifier may classify the further resolvable indicia to extract at least one additional feature from the further resolvable indicia.

[0010] The user interface may display the indicia bookmark and the image bookmark to be visible on the image at the at least one indicia location and the at least one image location respectively.

2018100324 15 Mar 2018 [0011 ] Extracting the at least one feature may include displaying the at least one extracted feature on the user interface. The at least one extracted feature may be displayed in a segmented and editable format.

[0012] In another aspect there is provided an image analysing system which includes: a conversion module that provides an image that includes segmented data including a first set of indicia; an indicia recognition module that recognises indicia in the first set of indicia, wherein the indicia recognition module generates a set of resolved indicia and a set of unresolved indicia; a classifier that classifies resolved indicia to find at least one feature that includes at least one indicium from the set of resolved indicia; and a feature locator that determines at least one indicia location in the image associated with one or more indicia in the set of unresolved indicia.

[0013] The feature locator may bookmark the at least one indicia location with an indicia bookmark.

[0014] The conversion module may further provide a first set of images; the system may further include an image recognition module that extracts a second set of indicia from the first set of images; and the indicia recognition module may recognise further indicia in the second set of indicia, and may add the further indicia to at least one of the set of resolved indicia and the set of unresolved indicia.

[0015] The classifier may classify the set of resolved indicia including the further indicia.

[0016] The classifier may classify the set of resolved indicia by: comparing the set of resolved indicia with a classification framework, and extracting at least one feature that includes at least one indicium from the set of resolved indicia.

[0017] Extracting the at least one feature may include displaying the at least one extracted feature on the user interface.

2018100324 15 Mar 2018 [0018] In another aspect there is provided a method of analysing an image, the method including: providing an image that includes segmented data including a first set of indicia; recognising indicia in the first set of indicia; generating a set of resolved indicia and a set of unresolved indicia; classifying resolved indicia to find at least one feature that includes at least one indicium from the set of resolved indicia; and determining at least one indicia location in the image associated with one or more indicia in the set of unresolved indicia.

[0019] The method may further include bookmarking the at least one indicia location with an indicia bookmark.

[0020] The image may further include a first set of images, and the method may further include: extracting a second set of indicia from the first set of images; recognising further indicia in the second set of indicia; and adding the further indicia to at least one of the set of resolved indicia and the set of unresolved indicia.

[0021] The providing may include converting the image to the segmented data including the first set of indicia and the first set of images.

[0022] The classifying may include: comparing the set of resolved indicia with a classification framework, and extracting at least one feature that includes at least one indicium from the set of resolved indicia.

[0023] The extracting at least one feature may include displaying the at least one extracted feature on a user interface.

[0024] Throughout this specification the word comprise, or variations such as comprises or comprising, will be understood to imply the inclusion of a stated element, integer or step, or group of elements, integers or steps, but not the exclusion of any other element, integer or step, or group of elements, integers or steps.

2018100324 15 Mar 2018

Brief Description of Drawings [0025] Embodiments of the disclosure are now described by way of example with reference to the accompanying drawings in which:[0026] Fig. 1 is a schematic representation of an embodiment of an image analysing system;

[0027] Fig. 2 is a representation of an embodiment of a user interface of the image analysing system; and [0028] Fig. 3 is a flow diagram of an embodiment of a method of analysing an image.

[0029] In the drawings, like reference numerals designate similar parts.

Description of Embodiments [0030] Referring to Fig. 1 of the drawings, an image analysing system 100 includes a conversion module 102 that converts an image 104 to segmented data. The segmented data includes a first set of indicia 106 and a first set of images 108. The system 100 includes an image recognition module 110 that selects a first image subset from the first set of images and extracts a second set of indicia 112 from the first image subset. The image recognition module 110 may include a known optical character recognition application such as OmniPage.

[0031] The system 100 includes an indicia recognition module 114 that recognises indicia in the first set of indicia and indicia in the second set of indicia. The indicia recognition module 114 generates a set of resolved indicia 116 and a set of unresolved indicia 118. The system includes a classifier 120 that classifies resolved indicia by comparing the set of resolved indicia with a classification framework, and extracting at least one feature 122 that includes at least one indicium from the set of resolved indicia. The system 100 includes a feature locator 124 that determines at least one

2018100324 15 Mar 2018 indicia location in the image associated with one or more indicia in the set of unresolved indicia, and the feature locator 124 bookmarks the at least one indicia location with a bookmark 126, in particular an indicia bookmark.

[0032] The system 100 also includes a user interface 130 that enables a user to access the at least one indicia location via the indicia bookmark 126, and to manipulate the unresolved indicia to form resolvable indicia 132. The classifier 120 then classifies the resolvable indicia 132 to extract at least one further feature 122 from the resolvable indicia 132.

[0033] In some embodiments the image recognition module 110 may select a second image subset from the first set of images, and then the feature locator 124 determines at least one image location in the image associated with one or more images in the second image subset. The feature locator 124 bookmarks the at least one image location with an image bookmark 126. In these embodiments, the user interface 130 further enables the user to access the at least one image location via the image bookmark 126, and to manipulate the one or more images in the second image subset to form further resolvable indicia 132. The classifier 120 classifies the further resolvable indicia 132 to extract at least one additional feature 122 from the further resolvable indicia 132.

[0034] The system 100 may be implemented on a suitable standard computer. The computer may be set up to run a virtual machine that has dedicated CPUs, for example 4 virtual CPUs, each being an Intel Core 2 Duo T770 at 2.40GHz, and the virtual machine having at least 8GB RAM. The system 100 may be implemented using any suitable software, for example Python v3.6 using virtualenv and Python modules as required.

[0035] Fig. 2 of the drawings shows an example of an embodiment of the user interface 130 of the image analysing system 100 that includes a display 200. The image 104 (or a portion of the image 104) is displayed on the user interface 130 at a first display location 201 in the bottom right hand side of the display 200. The image

2018100324 15 Mar 2018

104 (or portion of the image) is displayed in such a manner that a user can scroll, pan or otherwise navigate around the image 104 to view any part of the image 104. In this example the image 104 includes a scanned electronic document, for example a Portable Document Format (pdf) document. The image 104 is converted to segmented data, for example with the use of known optical character recognition (OCR) technology. As used herein “segmented data” refers to data including one or more segments of data that can be electronically edited, searched, and/or otherwise processed.

[0036] The segmented data includes indicia in the form of text 202. The indicia recognition module 114 includes text recognition functionality, and as such is able to recognise at least some of the text 202. Text that is recognised forms part of a set of resolved indicia. Text that is not recognised (or not recognised with a certainty above a defined indicia threshold) forms part of a set of unresolved indicia.

[0037] The classifier 120 classifies one or more features that contain at least one number, letter, symbol, or word of text from the recognised text. Classification includes matching a feature label defined in a classification framework with one or more suitable features present in the image 104. The classifier 120 includes a machine learning module, and classification may be performed using any suitable machine learning process, for example named-entity recognition (NER). NER is used for information extraction in order to locate and classify named entities, and the predefined categories are defined in the classification framework. NER methods that may be used include known methods such as Stanford NER and/or NeuroNER.

[0038] Several features 122 are displayed at a second display location 206 on the left hand side of the display 200. Each feature 122 has a feature label 208. As described above, the system 100 extracts the features 122 from the image 104. These extracted features 122 are displayed on the display 200 of the user interface 130, identified by the relevant feature labels 208 associated with the respective features as illustrated.

2018100324 15 Mar 2018 [0039] When a feature is found and matched to a feature label, the feature location where that feature is located in the image 104 is bookmarked and tagged with the feature label.

[0040] The display allows the user to search and locate instances of the features by selecting the feature label displayed within the second display location 206. The system 100 locates the feature location by navigating through the image 104 using the relevant feature bookmark that is tagged by the feature label that the user searches. An image subset 212 containing an instance of the feature associated with the selected feature label is displayed at a third display location 214. Referring to the image subset 212, the user is able to verify the details of the feature as it appears in the original image 104, and the user can then amend the feature as displayed in the relevant feature field if necessary.

[0041] In some embodiments the features 122 are extracted with an associated measure of accuracy. The measure of accuracy lies above an accuracy threshold for features considered to have been relatively accurately extracted, or the measure of accuracy lies below the accuracy threshold for extracted features that may include an error. The display may include an indicator of the measure of accuracy associated with a particular feature 122. For example, a feature field 210 may display a coloured border indicative of the relevant measure of accuracy. In this example, the feature fields include a broken border 230 for features considered to have been accurately extracted, the feature fields include a solid border 232 for features with an associated measure of accuracy below the accuracy threshold, and the feature fields include a double-lined border 234 for features which have not been extracted.

[0042] The first display location 201 includes a feature locator 216 in the form of a search field. When a user enters a feature label into the feature locator field 218, the user is able to navigate through the image to view instances of features that are associated with the entered feature label. In some embodiments, the feature locator 216 allows the user to navigate through the image to view possible instances of features associated with the entered feature label, “possible instances” being identified

2018100324 15 Mar 2018 as potential features with an associated measure of accuracy below the defined accuracy threshold.

[0043] The feature locator 216 also allows the user to search for indicia by entering letters, numbers or symbols into the feature locator field 218, and navigating through the image 104 using the navigation controls 240.

[0044] The display 200 allows the user to navigate or browse through the image 104 and to place one or more feature labels at selected locations within the image to associate the placed feature labels with one or more features identified by the user within the image 104. The display 200 also allows the user to navigate or browse through the image 104 and to insert data and/or metadata, for example in the form of text. In this way the user is able to manipulate unresolved indicia to form resolvable indicia, and/or manipulate one or more images (for example a picture or hand-writing) to form further resolvable indicia. This resolvable indicia added by the user is also classified as appropriate based on the feature labels defined in the classification framework. Manipulating the image in this way may be performed, for example, by an annotation tool such as BRAT.

[0045] Unrecognised text in the set of unresolved indicia may include classifiable features. In order to facilitate further processing of the unresolved indicia, the feature locator 124 determines one or more indicia locations associated with one or more unrecognised numbers, letters, symbols, or words of text. In some embodiments, the feature locator 124 bookmarks the indicia locations.

[0046] Where a feature associated with a particular feature label has not been extracted, the user interface 130 allows a user to enter feature data into the relevant feature field 210. Similarly, the user interface 130 allows a user to amend any of the features displayed in the feature fields.

[0047] The image 104 may include a set of images having one or more images, for example scanned in hand-written words 204. In embodiments that include an image

2018100324 15 Mar 2018 recognition module 110, a hand-written word, referred to herein as “an image subset”, is selected and processed by the image recognition module 110 in order to extract a set of indicia, i.e. one or more numbers, letters, symbols, or words of text, from the handwritten word. These extracted indicia may include resolved and unresolved indicia, and the resolved indicia are also classified, labelled, and bookmarked.

[0048] Fig. 3 of the drawings illustrates an embodiment of a method 300 of analysing an image. At 302 an image 104 is provided. The image 104 includes segmented data including a first set of indicia 304. In some embodiments the providing includes converting the image to the segmented data including the first set of indicia and a first set of images. In other embodiments the image 104 is input to the method already segmented, for example a scanned document may be uploaded. At 308 indicia in the first set of indicia 304 are recognised, and at 310 a set of resolved indicia 312 and a set of unresolved indicia 314 are generated. Steps 308 and 310 may be executed for example using known text recognition or OCR tools.

[0049] At 316 one or more resolved indicia are classified to find at least one feature 318 that includes at least one indicium from the set of resolved indicia 312. The classifying includes comparing one or more indicia in the set of resolved indicia 312 with a classification framework, and extracting at least one feature 318 that includes at least one indicium from the set of resolved indicia 312. When the feature is extracted, the feature is displayed on the user interface 130 as illustrated in Fig. 2 of the drawings. Classification may be performed using any suitable machine learning process, for example named-entity recognition (NER). NER methods that may be used include known methods such as Stanford NER and/or NeuroNER.

[0050] At 320 at least one indicia location 322 in the image associated with one or more indicia in the set of unresolved indicia is determined. In some embodiments, the method further includes bookmarking 324 the at least one indicia location 322 with an indicia bookmark 326.

2018100324 15 Mar 2018 [0051] Where the image 104 also includes a set of images 306 (including, for example, hand-written words or other pictures), the method further includes extracting 330 a further set of indicia 332 from the set of images 306, recognising further indicia in the second set of indicia (as at 308 and using known text recognition or OCR tools), and adding the further indicia to at least one of the set of resolved indicia 312 and the set of unresolved indicia 314.

[0052] The system and methods described herein facilitate navigating through an image based on the bookmarks and the feature labels. These bookmarks and feature labels are inserted into locations of the image automatically by the system, but bookmarks and feature labels can also be edited or added by a user on inspection of the image or a part of the image. Furthermore, where the system is unable to resolve indicia or accurately identify features within the image, the user interface and the bookmarks facilitate inspection of such indicia or features. By selecting feature labels the user can edit the details of the feature associated with that label. The user is also able to insert feature labels into the image at user-defined locations. Accordingly, the system and methods described herein more efficiently display information about unresolved indicia, and also allow a user to quickly navigate and resolve the indicia to create an updated or new image or document.

[0053] The user interface provides a side by side view of two representations: one part that typically has a familiar layout (e.g. the second display location 206) that a user would be familiar with and know how and where to locate information, and another part (e.g. the first display location 201) with an appearance varying depending on the particular image being considered. Where different images are likely have different appearances, this two-part display facilitates the analysis and understanding of the content of images.

[0054] It will be appreciated by persons skilled in the art that numerous variations and/or modifications may be made to the above-described embodiments, without departing from the broad general scope of the present disclosure. The present

2018100324 15 Mar 2018 embodiments are, therefore, to be considered in all respects as illustrative and not restrictive.

2018100324 15 Mar 2018

Claims

CLAIMS:

1. An image analysing system which includes:

a conversion module that converts an image to segmented data including a first set of indicia and a first set of images;

an image recognition module that selects a first image subset from the first set of images and extracts a second set of indicia from the first image subset;

an indicia recognition module that recognises indicia in the first set of indicia and indicia in the second set of indicia, wherein the indicia recognition module generates a set of resolved indicia and a set of unresolved indicia;

a classifier that classifies resolved indicia by:

comparing the set of resolved indicia with a classification framework, and extracting at least one feature that includes at least one indicium from the set of resolved indicia; and a feature locator that determines at least one indicia location in the image associated with one or more indicia in the set of unresolved indicia, wherein the feature locator bookmarks the at least one indicia location with an indicia bookmark.

2018100324 15 Mar 2018
2.

An image analysing system which includes:

a conversion module that provides an image that includes segmented data including a first set of indicia;

an indicia recognition module that recognises indicia in the first set of indicia, wherein the indicia recognition module generates a set of resolved indicia and a set of unresolved indicia;

a classifier that classifies resolved indicia to find at least one feature that includes at least one indicium from the set of resolved indicia; and a feature locator that determines at least one indicia location in the image associated with one or more indicia in the set of unresolved indicia.
3. The system of claim 2, wherein the feature locator bookmarks the at least one indicia location with an indicia bookmark.
4. The system of claim 2 or 3:

wherein the conversion module further provides a first set of images;

the system further including an image recognition module that extracts a second set of indicia from the first set of images; and wherein the indicia recognition module recognises further indicia in the second set of indicia, and adds the further indicia to at least one of the set of resolved indicia and the set of unresolved indicia.
5. A method of analysing an image, the method including:

providing an image that includes segmented data including a first set of indicia;

2018100324 15 Mar 2018 recognising indicia in the first set of indicia;

generating a set of resolved indicia and a set of unresolved indicia;

classifying resolved indicia to find at least one feature that includes at least one indicium from the set of resolved indicia; and determining at least one indicia location in the image associated with one or more indicia in the set of unresolved indicia.

1/3

2018100324 15 Mar 2018

104

Fig. 1

2018100324 15 Mar 2018

Fig. 2

2018100324 15 Mar 2018

3/3

326

300

Fig. 3