RU2326435C2 - Electronic ink processing - Google Patents

Abstract

FIELD: physics, computing.

SUBSTANCE: invention relates to means of analysis, arrangement, and recognition of electronic ink. In this method, document data, with electronic ink contained in them, are taken from the software application running in the first processing thread. The first processing thread is used for providing the document data to the electronic ink analysis process for the analysis in the second processing thread, the first processing thread control is then returned to the software application, after the analysis results reception, the results are matched with the updated document data.

EFFECT: detection of dynamic displacement and alteration of arrangement of electronic ink in annotated documents.

19 cl, 49 dwg

Description

FIELD OF THE INVENTION

The present invention relates to the processing of electronic ink. Various aspects of the present invention are particularly applicable to electronic ink analysis, including layout analysis, classification and recognition of electronic ink. Additional aspects of the invention relate to the use of compiled, classified, and recognized electronic inks, for example, when creating flexible annotations with extensive features in an electronic ink document.

State of the art

Since the role of computers has spread throughout society, all sorts of different techniques have been developed for entering data into computers. One particularly applicable data presentation technique is the use of handwritten text. As a result of entering with a pen or other object on a digital converter for receiving “electronic ink”, a computer user can refuse bulky and inconvenient keyboards. Handwriting can be conveniently used, for example, by visiting physicians, site architects, couriers delivering parcels, warehouse workers walking around a warehouse, and in many situations where using a keyboard would be difficult or inconvenient. Although handwriting is more convenient than keyboard input in many situations, text written in electronic ink usually cannot be directly processed by most software applications. Instead, text written in electronic ink should be analyzed to convert it to another form, such as characters of the American Standard Code for the Exchange of Information (ASCII). This analysis includes a handwriting recognition process in which characters are recognized based on various relationships between the individual strokes of the electronic ink constituting the electronic ink word.

Handwriting recognition algorithms have improved significantly in recent years, but their accuracy may decline when electronic ink is recorded at an angle. Similarly, when it is not easy to distinguish between separate groups of ink strokes, for example, when two words are written close to each other, many recognition algorithms cannot accurately recognize electronic ink. Some recognition algorithms may also incorrectly recognize electronic ink as text when, in fact, electronic ink is supposed to be a pattern. For example, a user can annotate typewritten text by writing a stroke of electronic ink that emphasizes, highlights, surrounds, or crosses out some portion of typewritten text. The handwriting recognition algorithm may then incorrectly recognize these strokes of annotations as a dash, a number zero, or the letter “O”.

The accuracy of many recognition algorithms can be greatly enhanced by “parsing” (for example, by analyzing the layout and / or “classification”) of electronic ink before using the handwriting recognition algorithm. The classification process usually determines whether a stroke of electronic ink is part of a drawing (i.e., a stroke of a drawing ink) or part of a handwritten text (i.e. a stroke of ink of a text). Classification algorithms are also possible to identify other types of strokes. The layout analysis process typically groups the strokes of electronic ink into meaningful associations, such as words, lines, and paragraphs. The layout and classification analysis processes can thus be used to identify which strokes in the electronic ink collection belong to one word, which electronic ink words are associated with one line of text recorded by electronic ink, and which lines of text recorded by electronic ink are associated with a paragraph.

Although layout analysis and classification can greatly improve electronic ink recognition, many software developers are unaware of the importance of these steps to electronic ink recognition. Until recently, layout and classification analysis algorithms were not readily available for use with existing software applications. For example, the Microsoft® Windows XP Tablet PC Edition Version 2002 operating system was usually sold with the Microsoft® Windows Journal software application for storing, displaying, and managing electronic ink. Although the Microsoft® Windows Journal software application uses an internal parsing program, until recently this parsing program was not available for other software applications running under the operating system.

Although the parsing process from the Windows Journal software application is now separately available for other software applications, the use of this parsing program is not well known, and this parsing program cannot be easily used with many software applications into which the user may want to enter handwritten text. In addition, even if the software application developer had to create a parsing program specifically for use with the desired software application (which in itself can be a difficult and time-consuming process), the execution of the parsing process can be completely time-consuming. For example, parsing only a few strokes of electronic ink using a relatively fast microprocessor can take several seconds or even several minutes for a parsing program. If a software application must stop working until the parsing process is completed, then the software application becomes too slow for most users to use.

Therefore, there is a need for electronic ink processing techniques that can be used by various software applications, for example, to analyze the layout, classification and / or recognition of electronic ink. Further, there is a need for electronic ink processing techniques that can process electronic ink, while at the same time allowing the software application to apply the techniques to accept a new input of electronic ink without invalidating the ink processing results.

Summary of the invention

Mostly, various examples of the invention provide electronic ink processing techniques that can be used by various electronic ink processing software applications. Further, these electronic ink processing techniques allow electronic ink processing to be performed asynchronously with respect to the operation of a software application that implements the techniques, so that electronic ink can be processed without stopping or significantly delaying the operation of the software application. The software application may even continue to accept a new input of electronic ink, while the previous input of electronic ink is processed.

In accordance with various examples of the invention, elements in a file or document may be described based on their spatial position relative to each other. For example, both a stroke of electronic ink and typewritten text can be described on the basis of a single spatial coordinate system. Using spatial information to describe elements of a document, a document management software application can maintain a data structure that describes the relationship between its document elements. In particular, a software application can support a data structure that describes a class of various elements of a document, and defines associations between different elements of a document. These associations can be defined, for example, as the information used to link the data of electronic ink strokes or their collections with other elements in an electronic document (such as words, lines, paragraphs, figures, table cells, etc.).

By describing the elements of a document in the data structure of a file or document, based on their spatial position, document elements for various types of files can use common techniques for identifying and managing their document elements. More specifically, various software applications can describe elements of a document in a document based on their spatial position, and apply this spatial position, referring to the use of general methods for analyzing electronic ink. Further, by defining a specific area of the document for analysis, each software application can limit the analysis process to only the required elements in the document.

To analyze the new input of electronic ink into a document in accordance with various examples of the invention, the document management software application modifies the data structure associated with the document in order to include new ink for analysis. The software application then feeds this data structure (or its corresponding parts) to the ink analysis tool, which copies some or all of the data structure for analysis (and works on this copy of the data, which is independent of the data structure of the application program document). The ink analysis tool transmits a copy to an analysis process, such as a parsing process (for example, a layout analysis process and / or a classification process). The software application may resume its normal operation, including receiving a new input of electronic ink and / or other data, while the ink analysis process (s) is in progress. In addition to receiving new electronic ink, the application program can receive any “other data”, for example, data modifying the size, placement or contents of existing ink, text, images, graphs, tables, flowcharts, diagrams, etc .; data adding additional text, images, graphs, tables, flowcharts, diagrams, etc .; data deleting existing text, images, graphs, tables, flowcharts, diagrams, etc. After completing all the required analysis processes, the analysis results are returned to the ink analysis tool.

Thus, various examples of systems and methods in accordance with the invention make it possible to perform ink analysis processes asynchronously with the operation of a software application using the ink analysis process. This asynchronous operation allows the user to continue using the software application without the delay caused by the analysis process. Further, it allows you to simultaneously perform multiple analysis processes.

In response to receiving the analysis results, the ink analysis tool receives the current version of the electronic document data structure (which may contain new and / or modified data entered during the analysis process) from the software application and reconciles the analysis results with the current version of the data structure. By matching the analysis results with the current version of the data structure, various examples of the invention can eliminate more complex techniques, such as “locking”, for asynchronous access to the data used by the software application. Instead, reconciliation can be triggered by any software application without the need for complex internal locking.

After matching the analysis results with the current version of the data structure, the ink analysis tool can then provide a copy of the agreed analysis results to another analysis process, such as a handwriting recognition process. Again, the software application can resume its normal operation, including receiving a new input of electronic ink and / or other data, while the process (s) of the second ink analysis is performed. After all the required second analysis processes have been completed, the results of the second analysis process are returned to the ink analysis tool. The ink analysis tool then obtains the current version of the data structure from the software application (which again may include new and / or modified data) and reconciles the results of the second analysis with the current version of the data structure. The ink analysis tool then updates the data structure using the consistent results of the second analysis process. Of course, any number of ink analysis procedures and / or steps can be used without departing from the invention.

The use of various ink analysis processes described above and spatial information that matches or links electronic ink data with other features of an electronic document can also be used to create valuable, flexible and natural ink annotations in an electronic document. For example, aspects of the invention can be used to create annotations through electronic inks that dynamically move and / or otherwise change based on the changes made to the underlying annotated elements of the document. Users typically annotate documents in many different ways, for example, they can circle, underline, highlight, or cross out words; write marginal notes; draw arrows or other pointers to annotations located in margins, etc. In addition, users make annotations on documents of various different types, including, for example, text, spreadsheets, drawings, slide presentations, tables, charts, graphs, flowcharts, etc.

The smooth incorporation of electronic ink annotations into an electronic document requires that the annotations change accordingly when, for any reason, the underlying electronic document changes. For example, if a user encircles a word in an electronic document (as an annotation) and then adds text somewhere in the document before the given word, this may cause the circled word to move. In this case, the annotation in the form of a circle should move and remain with the word. As another example, if the user adds or removes characters from the circled word or otherwise resizes it in any way, the annotation in the form of a circle should be stretched or compressed to adapt to the new word size. Preferably, when for any reason the text of the underlying electronic document is reformatted and / or the provisions of its constituent elements (paragraphs, figures, columns, etc.) are updated relative to each other and / or the page, the annotations made by electronic ink also move and remain correctly positioned relative to the base text or other information. If the annotations are not changed in this way relative to the text of the underlying electronic document, then it becomes inappropriate to edit or share active electronic documents after annotation, and printing on paper will still be the simplest and most applicable annotation method.

Determining how to modify the annotation accordingly will depend on various factors related to the annotation and the electronic document. For example, a corresponding change in the annotation in the case of reformatting the text may depend, for example, on the capabilities of the computing system (for example, parsing tools) with respect to: (a) identification of electronic ink as an annotation; (b) identifying the type of annotation made by electronic ink; and (c) identifying the dependence of electronic ink on a particular element in a basic electronic document. Although it is possible to request some or all of this type of information from the user, such systems would require significantly more onerous actions from the user than annotating on paper. Therefore, in accordance with various aspects of the invention, the above information can be obtained from: (1) the electronic ink itself and (2) the contents of the base document, including the spatial position of the various elements in the document relative to electronic ink.

Since determining the value of annotations made by ink can be complicated and difficult, it is extremely impractical to expect that each application program, a document of which the user may want to annotate, will individually implement the annotation identification logic. Rather, it would be preferable to create a reusable component that provides this annotation function, which can easily be integrated into any application program. For example, since the operating system of a pen-based computing system provides components for collecting and reproducing electronic ink, it would be preferable for this operating system to provide a component that determines the value of annotations made with ink (also called “annotation parser”) which is quite effective to work with the ink in question.

The identification of the contents of a basic electronic document, however, poses significant difficulties in providing the capabilities of “intellectual” annotation. For example, all kinds of different application programs have largely incomparable ways to store, manage, and reformat documents. The present invention provides a reusable annotation parser capable of determining the dependence of possible annotations made by ink on various types of electronic documents. More specifically, in accordance with some aspects of the present invention, there is provided a mechanism with parsing annotations that calls back to an application program to obtain relevant information related to parsed inks (e.g., information related to an electronic document in a spatial domain corresponding to the ink annotation to obtain a “context” to the ink annotation). This mechanism is simple enough for its practical integration into any document-based application, and it is effective enough to work with very large documents (which can be processed in parts, such as pages, etc.).

Ink processing techniques in accordance with various examples of the invention thus allow various software applications to perform numerous processes on electronic ink using an ink analysis tool. In addition, a software application using the above methods can continue its normal operation during the analysis process, including accepting a new input of electronic ink, without the need to invalidate the results of the analysis processes.

List of drawings

The above and other objects, features and advantages of the present invention are obvious and fully understood from the following detailed description, taken in conjunction with the accompanying drawings, in which:

figure 1 is a schematic representation of a general-purpose digital computing environment in which some aspects of the present invention can be implemented;

figure 2 - the environment of a personal computer (PC) with a pen input, which can be implemented some aspects of the present invention;

3 and 4 are various features of exemplary implementations of aspects of the invention relating to the annotation of electronic documents;

figure 5 - General features of parsing ink electronic document;

6A-6I are exemplary data structures applicable in the implementation of at least some aspects of the present invention;

7A-12B are exemplary electronic documents and related exemplary data structures applicable to processing annotations of electronic documents made by electronic ink;

13A-14B are exemplary electronic documents and associated exemplary data structures applicable to processing features of electronic block diagrams in electronic documents;

figa-15C - exemplary electronic documents and related exemplary data structures applicable to the processing of signs of spreadsheets in electronic documents;

figa-16D is a method of processing electronic ink in accordance with various examples of the invention;

17, 19-21 and 23-26 are illustrations of transferring data objects during exemplary ink analysis processes in accordance with various examples of the invention;

Figs. 18 and 22 are simple data trees that can be manipulated in accordance with various examples of the invention;

Fig. 27 is a flowchart depicting a method for matching analysis results with the current state of a document; and

Fig - means for the asynchronous analysis of electronic ink in accordance with other examples of the invention.

Detailed Description of Drawings

Terms

In the present description of the invention, the following terms are used and, unless otherwise specified or clearly follows from the context, the terms have the meaning given below:

“Play” or “reproduced” or “reproduction” - The process of determining how information (including text, graphics and / or electronic ink) should be displayed, for example, on a screen, printed or printed in some other way.

"Computer-readable computer medium" is any available medium that a user can access on a computer system. By way of example, and not limitation, a “computer readable medium” may include a computer storage medium and a data medium. “Computer storage medium” includes volatile and non-volatile, removable and non-removable media made in any way or by any technology for storing information, such as computer-readable instructions, data structures, program modules or other data. “Computer storage medium” includes, but is not limited to, random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other technology memory; compact disc, digital multifunctional disc (DVD) or other optical storage devices; magnetic cassettes, magnetic tape, magnetic disk storage devices or other magnetic storage devices; or any other medium that can be used to store the required information and to which the computer can access. A “data medium” typically embodies machine-readable instructions, data structures, program modules or other data in a data-modulated signal, such as a carrier wave or other transport mechanism, and includes any information delivery medium. The term "modulated data signal" means a signal, one or more of whose characteristics are set or changed so that encoding is provided to encode information in this signal. By way of example, and not limitation, a communication medium includes wired media such as a wired network or a direct wired connection, and wireless media such as acoustic, radio frequency, infrared, and other wireless media. Combinations of any of the above storage media and transmission media are also encompassed by the term “computer readable medium”.

Overview

In accordance with various examples of the invention, the properties of an element in a document (or other type of file) may include information regarding the spatial position of the element in the document. Thus, both a stroke of electronic ink and typewritten text can be described based on one spatial coordinate system for a document and / or based on a spatial relationship or other relationship with other elements in the document. In addition, related elements in a document can be identified simply by identifying the spatial area of the document containing these elements and / or by linking these elements.

Using the spatial relationship between the various elements of the document, the software application can create and maintain a data structure that describes other relationships between the elements of the document. For example, a software application may support a data structure, such as a data tree, that defines a class for various elements in a document. Thus, the node in the data structure can correspond to a handwritten word or drawing containing one or more strokes of ink, and the data stored in the node can additionally indicate the position of the word on the page. Any other suitable or standard commonly stored data related to the ink stroke and / or the word electronic ink can also be stored in the word node.

A data structure of this type can also associatively link document elements (such as single ink strokes or typewriter characters) into meaningful groups such as words, lines of words, sentences, paragraphs, etc. Thus, if the software application supports a tree-like structure of a document that describes a paragraph of handwritten electronic ink, the end nodes of the data structure may include data related to individual strokes of electronic ink, and one or more strokes may be associated together in the data structure as word nodes corresponding to words in a paragraph (for example, words defined by parser and / or recognition tool). The tree structure can then associatively associate the word nodes with the line nodes corresponding to the lines in the paragraph. Each line node can optionally be associated with a node corresponding to a paragraph. Further, the software application may support a tree or other data structure that associatively associates a node corresponding to a stroke of electronic ink and / or some other grouping of electronic ink with another node corresponding to a non-ink element of a document, such as an image; ink drawing; typewriter character, word, line, paragraph or the like. The given data structures can thus be used to determine the relationships between the associated strokes of electronic ink to distinguish strokes of electronic ink forming a handwritten text from strokes of electronic ink annotating non-ink elements of a document, and / or to associate strokes of electronic ink with other elements of a document .

As described in detail below, these data structures can be used with an ink analysis tool in accordance with various examples of the invention to analyze electronic ink in a document. According to various examples of the invention, a software application can analyze electronic ink in a document by first creating a data structure for the document. This data structure describes the relationship between document elements that have already been analyzed (if any), and thus provides the context in which any new electronic ink will be analyzed. Said data structure or “analysis context object” also includes any new electronic ink that has not been analyzed. Those. The analysis context object also includes electronic ink, for which interconnection with other elements of the document has not yet been established. For some examples of the invention, the software application creates its own analysis context object. According to other examples of the invention, however, the software application uses an ink analysis tool or other tool to create an analysis context object.

After a software application provides an analysis context object with an ink analysis tool (or an ink analysis tool creates an analysis context object), the ink analysis tool copies or retrieves information related to at least a part of the analysis context object that contains unanalyzed electronic ink . By making a copy or receiving information related to the desired part of the analysis context object, the ink analysis tool creates a data structure that can subsequently be analyzed without changing the analysis context object supported by the software application. Those. the copy is independent of the actual electronic document used in the software application, and thus may be referred to below as a “document-independent” analysis context object.

As soon as the ink analysis tool creates a document-independent analysis context object, the ink analysis tool provides this document-independent analysis context object to one or more analysis processes. For example, if handwriting recognition is to be performed for unparsed electronic ink in a document, then the ink analysis tool can provide a document-independent context analysis object to classification and / or layout analysis processes to classify ink into strokes of text and drawing (if necessary, or if required) and then grouping the strokes of the text of the unanalyzed electronic ink into associated groups, based on the layout of the ink. While the classification and / or layout analysis processes analyze a document-independent analysis context object, the software application can continue its normal operation. In particular, the software application may continue to accept new electronic ink input and / or any other data in the electronic document executed by the application program.

When an analysis process, such as a parsing process, completes its analysis of a document-independent analysis context object, it returns the analysis results to the ink analysis tool. More specifically, the parsing process (which may include, in particular, the processes of classifying and analyzing the layout as described above) will return a modified version of the document-independent analysis context object depicting new relationships for previously unanalyzed electronic ink. Although the software application is free to accept new electronic ink input and / or any other data for the document during the parsing operation described above, however, the current version of the analysis context object for the document (i.e., the version supported by the application) may be different from being independent of an analysis context object document originally provided to the ink analysis tool, and from the parsing results provided by the parsing process.

Therefore, in accordance with some examples of the invention, the ink analysis tool may receive the current version of the analysis context object from the software application and match the parsing results with the current version of the analysis context object. During this matching process, the ink analysis tool updates the current version of the analysis context object to reflect the results of the parsing process. The ink analysis tool then passes consistent data from the current analysis context object to the handwriting recognition process for recognition. According to other examples of the invention, however, the ink analysis tool may skip the matching process and, instead, pass the parsing results directly to the handwriting recognition process.

Once the parsing results are consistent with the current version of the analysis context object, the software application can return to its usual work again, and thus it can continue to accept new electronic ink input and / or any other data related to the document. In the meantime, the recognition process analyzes consistent data from the current analysis context object (or, alternatively, parsing results). After the recognition process completes its analysis of the agreed data (or parsing results), it returns its recognition results to the ink analysis tool. Again, since the software application may have received a new input of electronic ink and / or any other data for the document during the recognition process, the ink analysis tool receives the current version of the analysis context object from the software application. The ink analysis tool then matches the results of the recognition process with the current version of the analysis context object to update the current version of the analysis context object with recognition results.

Exemplary Operating Environment

1 is a schematic representation of a general-purpose digital computing environment that can be used to implement various aspects of the present invention. 1, a computer 100 includes a data processing unit 110, a system memory 120, and a system bus 130 that connects various system components, including a system memory 120, to a data processing unit 110. The system bus 130 may be any of several types of bus structures, including a memory bus or memory controller, a peripheral bus, and a local bus using any of a variety of bus architectures. System memory 120 may include read only memory (ROM) 140 and random access memory (RAM) 150.

A basic input / output system (BIOS) 160 comprising basic procedures that facilitate transferring information between elements in a computer 100, for example, during startup, is stored in a ROM 140. The computer 100 may also include a hard disk drive 170 for reading and writing to a hard disk (not shown), a magnetic disk drive 180 for reading or writing to a removable magnetic disk 190, and an optical disk drive 191 for reading or writing to a removable optical disk 192, such as a compact disk or other optical carriers. A hard disk drive 170, a magnetic disk drive 180, and an optical disk drive 191 are connected to the system bus 130 via a hard disk drive interface 192, a magnetic disk drive interface 193, and an optical disk drive interface 194, respectively. These drives and drives and related computer-readable media provide non-volatile storage of computer-readable instructions, data structures, program modules and other data for a personal computer 100. For a person skilled in the art, it will be appreciated that other types of computer-readable media may also be used in an exemplary operating environment. that can store data that the computer can access, such as magnetic tapes, flash memory cards, digital video discs, cartridges Ulli, random access memories (RAMs), read only memories (ROM), etc.

Several software modules may be stored on a hard disk drive 170, magnetic disk 190, optical disk 192, in ROM 140 or RAM 150, including operating system 195, one or more application programs 196, other program modules 197, and program data 198. The user can enter commands and information into the computer 100 using input devices such as a keyboard 101 and a pointing device 102 (such as a mouse). Other input devices (not shown) may include a microphone, joystick, game pad, satellite dish, scanner, and the like. The above and other input devices are often connected to the data processing unit 110 via the serial port interface 106, which is connected to the system bus 130, but they can also be connected using other interfaces, such as a parallel port, a game port, or a universal serial bus (USB) etc. Further, the above devices can be connected directly to the system bus 130 through an appropriate interface (not shown).

A monitor 107 or other type of display device may also be connected to the system bus 130 via an interface, such as a video adapter 108. In addition to the monitor 107, personal computers typically include other peripheral output devices (not shown), such as speakers and printers. In one example, a pen input digitizer 165 and an accompanying pen or pen 166 are provided for hand-held digital input. Although FIG. 1 shows the connection between the pen input 165 and the serial port interface 106, in practice, the pen input 165 can be directly connected to the data processing unit 110, or it can be connected to the data processing unit 110 in any suitable manner, for example, using a parallel port or other interface and system bus 130, as is known in the art. In addition, although the digital converter 165 is shown separately from the monitor 107 in FIG. 1, the usable input region of the digital converter 165 may have a spatial extension with the display area of the monitor 107. Further, the digital converter 165 may be integrated in the monitor 107, or it may exist in the form of a separate device, superimposed or otherwise attached to the monitor 107.

Computer 100 may operate in a network environment using logical connections to one or more remote computers, such as remote computer 109. Remote computer 109 may be a server, router, network PC, peer device, or other common network node, and it typically includes many or all of the elements described above with respect to the computer 100, although for simplicity only the data storage device 111 is shown in FIG. The logical connections shown in FIG. 1 include a local area network (LAN) 112 and a wide area network (WAN) 113. Such network environments are common in offices, enterprise-wide computer networks, intranets, and the Internet, using both wired and wireless connections.

When used in a LAN network environment, the computer 100 is connected to the local area network 112 via a network interface or adapter 114. When used in a network environment, the personal computer 100 typically includes a modem 115 or other means for establishing a communication link over a wide area network 113, such as the Internet . The modem 115, which may be internal or external to the computer 100, may be connected to the system bus 130 via the serial port interface 106. In a networked environment, program modules described with respect to the personal computer 100, or parts thereof, may be stored on a remote storage device.

It should be understood that the network connections shown are exemplary, and other techniques may be used to establish a communication link between computers. It is assumed that any of various well-known protocols exists, such as Transmission Control Protocol / Internet Protocol (TCP / IP), Ethernet, File Transfer Protocol (FTP), Hypertext Transfer Protocol (HTTP), User Datagram Protocol (UDP), etc. , and the system can operate in a user-server configuration, allowing the user to retrieve Web pages from a Web server. Any of various conventional web browsers can be used to display and manage data on web pages.

Although the environment of FIG. 1 depicts an exemplary environment, it is understood that other computing environments may also be used. For example, one or more examples of the present invention can use media having a smaller amount than all of the various aspects shown in FIG. 1 and described above, and these aspects can manifest themselves in various combinations and subcombinations that are obvious to a person skilled in the art .

Figure 2 shows a personal computer (PC) 201 with a pen input, which can be used in accordance with various aspects of the present invention. Any or all of the signs, subsystems, and functions in the system of FIG. 1 may be included in the computer of FIG. 2. A pen input personal computer system 201 includes a large display surface 202, for example, a digitizing flat screen display, such as a liquid crystal display (LCD) screen, on which a plurality of windows 203 are displayed. Using the pen 204, a user can Select, highlight, and record on the digitizing display area. Examples of suitable digitizing display panels include pen-type electromagnetic digital converters, thus pen-type digital converters supplied by Mutoh Co. (now known as FinePoint Innovations Co.) or Wacom Technology Co. Other types of pen-type digital converters may also be used, for example, optical digital converters and touch-sensitive digital converters. Pen-based computing system 201 interprets gestures made using pen 204 to manipulate data, enter text, and perform common computer application tasks, such as creating, editing, and editing spreadsheets, word processing programs, and the like.

Pen 204 may be equipped with buttons or other features to complement its capabilities. In one example, the pen 204 can be in the form of a “pencil” or “pen”, in which one end makes up the writing part, and the other end makes up the end of the “eraser”, and which when moving along the display indicates the parts of electronic ink on the display that should to be erased. Other types of input devices may also be used, such as a mouse, trackball, keyboard, or the like. Additionally, the user's own finger can be used to select or indicate portions of the displayed image on a touch or non-contact display. Therefore, the term “user input device” as used herein is intended to be broadly defined and encompasses many variations of well-known input devices.

In various examples, the system provides an ink processing platform in the form of a set of component object model (COM) services that the application program can use to collect, manage, and store ink. The ink processing platform may also include a markup language, including a language similar to Extensible Markup Language (XML). Further, the system can use the distributed component object model (DCOM) as another implementation option. Other implementations may be used, including the Win32 programming model and Microsoft Corporation's Net programming model. Mentioned platforms are commercially available and known in the art.

In addition to using full-performance pen computing systems or “tablet PCs” (eg, convertible laptops or slate type tablet PCs), aspects of the present invention can be used in conjunction with other types of pen computing systems and / or other devices that receive data in the form of electronic ink and / or accept input with an electronic pen or pen, such as: handheld or handheld computing systems; personal digital information devices; handheld personal computers; mobile and cell phones, pagers and other communication devices; clock; electrical appliances; and any other devices or systems that include a monitor or other display device and / or digitizer, which presents printed or graphic information to users and / or allows input using an electronic pen or pen, or which can process electronic ink, collected by another device (such as a regular desktop computer that can process electronic ink collected by a tablet).

The invention is described below in connection with the rest of the figures, which show various examples of the invention and information that helps explain the invention. The specific figures and information contained in this detailed description should not be construed as limiting the invention.

Generalization of the spatial appearance of the document

As described above, some aspects of the present invention relate mainly to systems and methods for providing more versatile, high-powered annotations using electronic ink in electronic documents. The following describes in detail various aspects and examples of the present invention.

A. Overview of the invention

Figure 3 shows, in General terms, the principle of operation of systems and methods in accordance with at least some examples of the present invention. Specifically, as shown in FIG. 3, the user can interact in some way with the electronic document 300 (document “A” in FIG. 3), for example, by adding electronic ink to the document 300. In the illustrated example, the electronic document 300 includes electronic or typescript 302 (“Sample Text” in the illustrated example) and ink drawing 304 (home in the illustrated example). Of course, the electronic document 300 may include any data or information without departing from the scope of the invention, such as electronic text, electronic ink (drawing or text), images, graphs, tables, charts, other graphic or electronic information and / or their combinations. For the purposes of this example, suppose that the original electronic document 300 (also called the “base document” or “base part” in this application) includes electronic typewriting text 302 and electronic ink drawing 304. When the user views Document A 300 in this example, he decides to include the annotation made by electronic ink in the document 300. Specifically, in the illustrated example, the annotation includes a circle 306 around some portion 308 of the electronic text 302.

In this example, after the user enters the annotation, the application program calls the parsing program 310 and requests that the newly entered electronic ink data be parsed (as noted above, “parsing” may include, for example, classifying electronic ink into various types ink (e.g., drawings, text, flow charts, music, etc.) and / or analysis of ink layout (e.g., refinement of spatial and positional relationships among ink strokes and grouped e them to the appropriate groups) and other analysis processes). Specifically, the application program sends input electronic ink data (representing annotation 306 in this example) to parser 310 in the form of ungraded ink data. The above is depicted in FIG. 3 by an input arrow 312.

Optionally, at the same or about this point in time, the application program may send some or all of the data related to the underlying electronic document 300 to the parser 310. Said data may include, for example, data related to the spatial arrangement of information in a document 300, for example, the position of electronic text 302, figure 304, etc. Additionally, the application program can send data showing the hierarchical structure of the data in the document, which is described in detail below. As another alternative, as also described in more detail below, the parser 310 can call back the application program by requesting some data related to the electronic document 300, such as information related to the electronic document 300 in the spatial domain associated with the new unclassified ink (in the aforementioned example, data representing typewritten text and information located spatially near annotation 306). Said data input to the parser 310 is depicted in FIG. 3 by an input arrow 314. Of course, both unclassified ink and previously analyzed data can be simultaneously sent to the parser and / or made available to it without deviating from the invention.

The parser 310 receives the input data 314 of the electronic document and the input data 312 of the ungraded ink and uses the information to classify the new electronic ink data into ink types. Inks can be classified into all kinds of different types of ink without departing from the invention, and various examples of possible types of ink are described in more detail below. For the present example, based on the input ink 306 and the information 302 contained in the base document 300, the parser 310 in accordance with this example of the invention can determine and classify the input ink 306 as an annotation, or even more specifically as an ink drawing corresponding to the annotation - a container (for example, a circle) that contains a specific typescript. Of course, other types of annotations are also possible, such as highlight annotations, “pointing to” annotations, “pointing from” annotations, anchored annotations (for example, without pointing to or pointing from), horizontal coverage, vertical coverage, etc. ., and various examples are explained in detail below.

After parsing, the data associated with annotation 306 can be included in the hierarchical data structure of the “document tree” associated with electronic document 300 (or any other suitable or required data structure) and said data structure (or instructions for modifying an existing data structure supported by the application program) can be sent back to the application program, as shown in figure 3 by arrow 316. If required, other data can also be returned from the tool 310 parsing an application program without departing from the invention. For example, as shown in FIG. 3, data associated with the results of a recognition analysis process (such as handwriting recognition tool or other recognition tool) can be returned to the application program if the actions of the parser 310 included calls to one or more modules, components and / or recognition software programs.

Figure 4 shows an example implementation of 400 various features used in some examples of the present invention. As shown, an electronic document (such as Document A 300 in FIG. 3) is generated, stored and / or maintained by the application program 400a as a data structure or document model 402. Said data structure 402 may be a hierarchical data structure or any other desired or suitable data structure without departing from the invention. The application program 400a includes an “Analysis Context” object 404 that contains a mirror copy, a selective copy, or other relevant information related to the data structure 402 (for example, the Analysis Context object 404 may be a “transform” level at which queries to the Analysis Context object 404 are converted to queries on the structure of the document’s native modes - thus, parsing technology in accordance with aspects of the present invention can be converted for use with all kinds of GOVERNMENTAL different applications their own programs). The "platform" 400b or the operating system side of this exemplary implementation 400 includes an Ink Analyzer method 406 that is invoked by the application program 400a. Application program 400a calls Ink Analyzer method 406 and passes it a reference to the Analysis Context object 404. Ink Analyzer method 406 calls parser 408 (also part of platform 400b in this example) that classifies and analyzes the layout of the input data, as briefly described above. The principle of operation of an exemplary implementation in accordance with the invention is described below in more detail in the section "Ink analysis". If the parser 408 completes its analysis and processing, then the platform 400b can send data back to the application program 400a, allowing the document model 402 and / or the Analysis Context object 404 to rebuild its data structure based on the processing and results of the parser, as briefly depicted arrow 410 in figure 4.

Figure 5 in General terms shows an example of a data structure that can be used to store data from an electronic document in at least some examples of the present invention, for example, as part of an Analysis Context object 404, as a document model 402, or as a data structure, generated and / or output by parser 408. Specifically, FIG. 5 shows an exemplary electronic document 500. When stored as an Analysis Context object 502 or parsed, the data structure may include various elements of the document, classified or located in different groupings or “nodes”. For example, as shown in FIG. 5, an electronic document 500 includes three paragraphs 504, 506, and 508 of electronic text; one drawing 510 made by electronic ink, and one annotation 512 made by electronic ink. An exemplary Analysis Context data structure 502 for an electronic document 500 is shown in FIG. 5 as a list of top-level elements. Specifically, node element 504 (a) corresponds to paragraph 504 in electronic document 500, node element 506 (a) corresponds to paragraph 506 in electronic document 500, node element 508 (a) corresponds to paragraph 508 in electronic document 500, node element 510 (a) corresponds to ink drawing 510 in electronic document 500 and node element 512 (a) corresponds to ink annotation 512 in electronic document 500. Additional nodes can be stored in the data structure (for example, as a hierarchical diagram) as parent nodes and / or child nodes rel regarding the nodes 504a, 506a, 508a, 510a, and 512a of the upper level shown in FIG. 5, as described in detail below.

B. Context nodes and hierarchical data structures

6A-6I illustrate various examples of context nodes and data structures that can be used in constructing, analyzing, and parsing data from an electronic document, such as an Analysis Context object, in accordance with some examples of the present invention. FIG. 6A shows an exemplary data structure 600 that can be used to store data of unclassified electronic inks (eg, ink newly introduced by a user that has not previously been parsed). Said exemplary data structure 600 includes a root node 602 (which may correspond to an entire electronic document in an application program, a specific document page, or any suitable or desired data grouping, such as the grouping used by the application program). Any suitable or required data may be included in the root node 602, such as page number, pointers to the previous page and next page, location or size of fields, etc. Each root node 602 may include any number of ungraded ink nodes 604 as child nodes, and each ungraded ink node 604 may further include any number of individual strokes 606 (the last node in the hierarchical structure (stroke node 606 in this example) may also be referred to as an “end node” in the present description of the invention). (The letter "n" in various figures may represent any number, including zero.) The various nodes of the data structure 600 may include any required or suitable data corresponding to the node. For example, stroke units 606 may include data representing: points of a digital converter or other data identifying a stroke of electronic ink; data on the position or orientation of the stroke; bar color data; data on pressing when performing a stroke; data on the time characteristics of the stroke input; line identifier data and / or any other standard or useful data used to store information about the stroke in systems and methods capable of receiving electronic ink. Similarly, the ungraded ink node (s) 604 may include any required or suitable data associated with the ink or node, such as data representing: the size of the bounding box of all the strokes contained in the node; location of the bounding box; information about the timing of input; geometric information for the ink (such as, for example, information about the convex hull) and / or any other standard or useful data related to the strokes of unclassified ink included in the node. Alternatively, if required, stroke data can basically be saved as a property of another node (for example, a word node, table node, etc.), and not as a separate node in the document tree.

Once unclassified ink is sent to a classification tool, layout analyzer, recognition tool, and / or other parsing system, at least some parts of the ink can be associated together and / or classified into all sorts of different types, and the data structure associated with the ink , can be modified in such a way as to correspond to the numerous associations and types into which it was classified. FIG. 6B depicts an exemplary electronic ink data structure 610 that has been classified as ink text. Using parsing technology including conventional parsing tools known and used in the art, parsing of electronic ink strokes can be parsed and stored hierarchically (for example, electronic ink data collected between a pen lowering event and a subsequent pen-raising event or in some other way), in which the associated individual ink strokes can be grouped together and saved as ink words linearly arranged across bold words can be grouped together and saved as ink lines, linked ink lines can be grouped together and saved as ink paragraphs, and linked paragraphs can be saved as electronic documents (for example, in separate parts of a page, as whole documents or in any suitable root groupings). An exemplary hierarchical data structure 610 corresponding to parsing electronic ink data and stored as shown in FIG. 6B may include a root node 612 that may correspond to an entire electronic document, page, or other grouping used by the application program. Each root node 612 may contain any number of nodes 614 corresponding to paragraphs or similar groupings of strokes of electronic ink. Paragraph nodes 614 can contain any number of separate line nodes 616, which, in turn, can include any number of individual ink word nodes 618, which can further comprise any number of separate strokes 620 corresponding to individual strokes of electronic ink input data. Optionally, as noted above, the stroke data can be remembered as a “property” of the associated word node, if required. The various nodes 612, 614, 616, 618, and 620 may store any suitable data relating to various individual strokes or collections of strokes contained in the node, like spatial orientation or location data and / or other data similar to those described above.

Other types of data and organization of data structures are possible, corresponding to the entered strokes of electronic ink, without departing from the invention. For example, a classification technology, layout analysis, or recognition tool may determine that the introduced electronic ink forms an ink pattern. Figure 3 shows an exemplary ink drawing 304, which contains several separate strokes of ink. 6C and 6D illustrate exemplary hierarchical data structures that can be used to group and store electronic ink data defined in relation to ink drawings. As shown in FIG. 6C, the data structure 630 includes a root node 632 (which may correspond to an entire electronic document, electronic page, or some other data grouping, as described above). Each root node 632 may include any number of nodes 636 of individual ink drawings (for example, so that each individual figure has a separate node that stores data relating, for example, to the width of the picture, the height of the picture, the position of the picture, the size of the bounding box, and t .p.), and each ink drawing unit 636 can include any number of nodes 638 of individual ink strokes (corresponding to the individual ink strokes making up the pattern). Optionally, ink stroke data may be stored as a “property” of their corresponding ink drawing unit (or other unit).

6D depicts an alternative data structure 630a for electronic documents containing one or more ink drawings. In this example, an electronic document, or a page, or another data grouping (represented by the root node 632a) may include one or more groups (represented by the node 634a) of drawings made by electronic ink, and their data is associatively linked together and stored in a hierarchical manner, described in connection with figs (for example, an entire electronic document (represented by the root node 632a) may include several separate pages (each represented by a group node 634a), and each individual page A ink can have one or more separate ink drawings (each represented by an ink drawing unit 636a), and each individual ink drawing can contain one or more separate ink strokes (each is represented by a stroke unit 638a or in the property of an ink drawing unit)). It should be noted that both groups and ink drawings can be child nodes of the same root node or any group node. In addition, if required, the group node may contain additional group nodes. Of course, other suitable data structures may be used to store electronic ink patterns without departing from the present invention.

Hierarchical data structures, however, are not limited to the use of electronic ink with the data entered. Typed text (for example, from the keyboard) can also be associated into various groups and stored in a hierarchical data structure. 6E shows an example of such a data structure 640. Said data structure 640 may include a root node 642 (for example, similarly to the root nodes described above). Each root node 642 may include any number of nodes 644 of a paragraph of text (corresponding to paragraphs of text), and each node 644 of a paragraph may include any number of nodes 646 of a line of text (or, alternatively, nodes of text sentences), and each node 646 of a line further may include any number of nodes 648 individual words of text. If required, the words of the text can be further divided into nodes of individual signs of the text and still further can be divided into nodes of individual signs of the sign (relating, for example, to baseline, serif, superscript, subscript of each character) without departing from the invention . Different nodes can store any suitable data related to the text, such as spatial position, content, paragraph size, line or word, etc .; page margins, sizes or placement of margins; number of pages, etc.

The various data structures described above in connection with FIGS. 6A-6E depict data structures that contain exclusively electronic ink data or electronic typewriter data only. Parsing technology can be used to analyze, combine, associate and group the various data types mentioned into a single data structure without departing from the invention. FIG. 6F shows an exemplary data structure 650 in which an electronic document or part thereof (represented by the root node 652) contains one or more separate paragraphs (represented by the paragraph node 654) and one or more separate lines (represented by the line node 656). The lines can potentially contain both typewritten text (represented by a text word unit 658) and text made by electronic ink (represented by an ink word unit 660 and an ink stroke unit 662). Each individual node of a paragraph, line and word (both typewritten text and ink words) contains the corresponding independent data, depending on the contents of the paragraph, line and word. Additionally or alternatively, individual words may further include both ink marks and typewritten characters without departing from the invention.

Other electronic data may also be included in the data structure of an electronic document (including hierarchical data structures) without departing from the invention. For example, image data (such as digital photography, graphic information, etc.) may be included in the data structure for an electronic document, for example, as shown in the data structure 664 of FIG. 6G. As depicted in this example, the data structure 664 includes an electronic document or part thereof (represented by the root node 666) that contains one or more separate data groups (represented by the group node 668). Each data grouping in this example may additionally include an electronic image (represented by the image unit 670) as well as typewritten text (represented by the paragraph unit 672, line unit 674 and text word unit 676), as described in general form above in connection with FIG. .6E and 6F. Of course, a data structure similar to that shown in FIG. 6G may also contain electronic ink data (for example, similar to the data structures shown in FIGS. 6A-6D) or other required data types without departing from the invention.

FIG. 6H shows an example of another potential data structure 680 that may be used in accordance with at least some examples of the present invention. More specifically, the data structure 680 of FIG. 6H corresponds to an electronic document or part thereof (represented by the root node 682) containing one or more lists (represented by the list node 684). The list in the electronic document may contain any number of individual list items (represented by the list item node 686), and each list item may contain all kinds of various kinds of electronic information. For example, some list items, optionally, may include a list marker, which may consist of a typewriter or electronic ink marker (ink marker assembly 688, including one or more corresponding ink stroke nodes 690, is shown in the example of FIG. 6H). In addition, each list item, optionally, may include one or more paragraphs of electronic ink (represented by a chain of 692 nodes), one or more ink drawings (represented by a chain of 694 nodes) and / or one or more typewritten paragraphs (represented by a chain 696 knots). Additionally (or alternatively), each list item, optionally, may include one or more images (represented by the image node 698) and / or one or more data groups, such as the groups described above in connection with FIG. 6D and 6G, (represented by a chain of 700 group nodes). Since the various nodes and / or chains 692, 694, 696, 698, 700 nodes can, in general, correspond to the exemplary data structures depicted in FIGS. 6A-6G, as well as subsets and / or groups of the mentioned data structures, then further explanation is omitted. If desired, at least in some cases, the node 686 of an individual list item may itself also contain a list having a data structure similar to that shown in Fig.6H.

6I shows another exemplary data structure 710 that can be used in at least some examples of the present invention. Said exemplary data structure 710 corresponds to electronic data analyzed, associatively linked together, grouped and stored in a table. In this illustrated example, each electronic document or part thereof (represented by the root node 712) may include one or more tables (represented by the table node 714). The table in this example may consist of one or more rows of information (represented by the node 716 rows of the table), and each row of the table may consist of one or more separate cells (represented by the node 718 cells). Individual cells may contain one or more data structures, including, for example, any of the various various data structures described above. In the illustrated example, the individual cells of the table are shown as potentially containing any number of images (represented by the image node 720), any number of typescript paragraphs (represented by a chain of 722 nodes), any number of groupings (represented by a chain of 724 nodes), any number of ink drawings (represented by a chain 726 nodes), any number of additional tables (represented by a chain of 728 nodes) and any number of lists (represented by a chain of 730 nodes). Of course, any other suitable or required data structures may be included in the table without departing from the present invention, including various specific data structures described above and / or combinations and subsets of data structures, for example, similar to those described above.

Any suitable or required data may be stored in various various nodes, including the various types of data described above. For use with processing data related to annotations, it may be useful in at least some examples of the invention that various nodes store data related to the spatial position, orientation or placement of the node in an electronic document, optionally, since the spatial information referred to or linked to other nodes in the data structure of an electronic document.

Also, other types of parsing programs, classification tools and / or recognition tools and data types may be used without departing from the invention. For example, recognition tools can be used to analyze, recognize, group, and / or associate electrical or electronic symbols (eg, resistors, voltage sources, capacitors, etc.); musical characters; mathematical symbols; block diagram elements; pie chart elements, etc. without departing from the invention.

The data structures depicted in FIGS. 6A-6I are merely examples of all kinds of data structures that can be used in accordance with aspects of the present invention. It will be apparent to those skilled in the art that the particular data structures used can vary widely without departing from the invention. For example, various nodes representing different layout groups can be used, additional types of nodes can be added, or some of the described types of nodes can be excluded without departing from the invention (for example, column nodes can be used instead of table row nodes in the table data structure) . Additionally, the above description is optimized for use with languages and text representations in which characters are read from left to right and then from top to bottom on the page. Data structures designed to be used and consistent with other languages and character placement can be used, for example, data structures to allow reading and / or writing in languages from right to left, top to bottom, bottom to top, and combinations thereof without departing from the invention. In addition, if required (in particular for some languages, such as Asian languages), strokes can be grouped appropriately as characters, which can then be grouped into lines without using an intermediate “word” node, as described above. As another alternative, the use of line nodes may be eliminated, and the word nodes may be grouped together as paragraphs and / or other required groupings. Data and other varieties in the data structure may be used, for example, to harmonize with the characteristics of a particular language without departing from the invention.

C. Application of Aspects of the Invention to Annotations

Aspects of the present invention, including the use of context nodes and hierarchical data structures, as described above, can be used to obtain “smart” annotations made by electronic ink in electronic documents. As described above, one desirable and advantageous use of pen-based computing systems is their use in annotating electronic documents. However, in order for them to be particularly useful and effective, annotations made with electronic ink in documents on a pen-based computing system must be flexible enough so that if and when the characteristics associated with various elements in the underlying document are for some reason are changed (for example, fields are changed, the display size is changed, the font is changed, information is added, information is deleted, etc.), then annotations are able to precisely change the position and / or other characteristics, Based on changes to annotated elements in the base document. This can be accomplished, for example, by linking at least one context node associated with the annotation to one or more context nodes associated with the base document. In at least some cases, data and / or characteristics associated and stored related to the underlying document, such as data relating to spatial attributes, position and / or placement of one or more elements of the document, can be used to control the placement, appearance and / or other annotation features.

On figa and 7B depicts one example of an annotation that can be applied to an electronic document using electronic ink. As shown in FIG. 7A, electronic document 750 includes the sentence “Today the sky is green”. A pen-based computing system user annotated the aforementioned document 750 using electronic ink, underlining the word “green” (underlining the annotation is shown at 752). FIG. 7B shows an exemplary hierarchical data structure 760 corresponding to said composite electronic document 750 (an entire electronic document 750 is represented in the data structure 760 by the root node 762). In full parsing, the data structure 760 includes context nodes showing the structure of the complete document. More specifically, information related to the typescript in electronic document 750 is analyzed and associatively linked together to indicate that the electronic document 750 contains one paragraph (paragraph node 764), the paragraph contains a line (line node 766), and the line contains five words (one node 768a-768e of the word text for each of the words "Today", "the", "sky", "is" and "green", respectively). In addition, the parser notes the presence of data in the electronic document 750 related to annotation 752, and it classifies the data as a pattern made by electronic ink (ink pattern unit 770). Furthermore, based on the placement and position of the ink stroke (s) relative to the typescript in the base document 750, the parser recognizes and classifies the annotation as an underline annotation and generates an underline unit 772. In addition, the parser stores the data of all strokes corresponding to the actual ink stroke (s) of the annotation ink underscore in one or more end nodes 774 of the stroke (optionally, the stroke data may be stored as a property of the underline node 772 or another corresponding node) .

Additionally, in order to save the electronic ink underline stroke 752 with this separate word “green”, wherever it appears in the electronic document 750 (for example, if the word is moved for any reason), the data is stored in the data structure 760 of the above example , meaning that the underscore context node 772 (for example, the “source” context node in this binding example) is associated with the text word node 768e for the word “green” (the “destination context” node in this example), as shown in FIG. 7B arrow 776, and data sah hurt, meaning that the text word node 768e is connected through the underline node 772, as indicated by arrow 778. The referenced binding “rigidly ties” the underline node 772 to the text word node 768 (for example, a “hard link” type relationship). Therefore, if for some reason the data stored in the text word context node 768e means that the position of the word “green” has changed in the electronic document, then the connections 776 and 778 allow the application program to detect that, if possible, it should move the playback submission of annotation by underlining the location and / or spatial position of the text word context node 768e of the text in the modified electronic document. As another example, if the data is a “bounding box”, word width data, or other data stored in context node 768e, it means that for some reason the size or position of the word has changed (for example, due to a change in font size, adding characters, characters, changes in characters, etc.), then the application program can then change the size of the underscore being played (for example, by lengthening or shortening), if possible, so that it matches the new size associated with the text word, etc. dstavlennym node 768e context. If applicable, systems and methods in accordance with at least some examples of the invention may break the bar (s) of the annotation so that it is present on more than one line if changes in the base document cause the annotated element of the document to appear on several lines (for example, for added words, a hyphen, or the like). Also, any suitable linking scheme between nodes or any suitable data representing a link can be used without departing from the invention.

On figa and 8B depicts another example of annotation. In this example, as shown in FIG. 8A, the electronic document 800 again includes the sentence “Today the sky is green”, but in this case, the user of the pen computing system annotated the document 800 using electronic ink, striking out the word “green” (strikethrough shown at 802). FIG. 8B shows an exemplary hierarchical data structure 810 corresponding to said composite electronic document 800 when it is fully syntactically parsed (since the specific context nodes for most of FIG. 8B correspond to the context nodes present in FIG. 7B, these same positions, as those present in FIG. 7B are used in FIG. 8B, and a duplicate explanation is omitted). In this case, the parser notes the presence of data in the electronic document 800 related to annotation 802, and it classifies the data as a pattern made by electronic ink (ink pattern unit 812). In addition, based on the location and position of the ink stroke (s) relative to the contents of the base typescript in document 800 (for example, strikethrough covers the horizontal word of the text), the parser recognizes and classifies the annotation as a strikethrough annotation and generates a strikethrough 814 . In addition, the parsing program stores all the stroke data corresponding to the actual stroke (s) of the annotation ink in strikethrough at one or more stroke end nodes 816 (or alternatively, as a property in strikethrough node 814 or another corresponding node).

As with the underscore annotation described with respect to FIGS. 7A and 7B, a strikethrough annotation can be saved with this single word “green” if for any reason the word is moved. Again, this can be accomplished, for example, by storing data in the data structure 810, meaning that the strikeout context node 814 (in this example, the “source” context node) is associated with the text word node 768e for the word “green” (in this example “destination” context node), as shown in FIG. 8B by arrow 818, and by storing data in the data structure, meaning that the text word node 768e is connected through the strikeout node 814, which is indicated by arrow 820. The communication type in this example is type of connection "with horizontal coverage ", which binds tightly node 814 to the node 768e strikeout words of the text. Therefore, if for some reason the data stored in the text word context node 768 e means that the position and / or size of the word “green” (or another word (s) associated with the node 768 e) has changed in the electronic document 800, then links 818 and 820 allow the application program to detect that, if possible, it should adjust the reproduced strikethrough representation and / or adjust its size (if possible) to the location, spatial position and / or size of the word stored in the context node 768e in the changed email electronic document. Again, any suitable linking scheme between nodes or any suitable data representing the link can be used without departing from the invention.

9A and 9B depict another example of an annotation type, and further associate or associate an annotation with more than one word of text. In this example shown in FIGS. 9A and 9B, the electronic document 900 again includes the sentence “Today the sky is green”, but in this case, the pen-computer user annotated the document 900 using electronic ink, circling the words “is green ”(a circle or annotation of the“ container ”type is shown at 902 in FIG. 9A). FIG. 9B shows an exemplary hierarchical data structure 910 corresponding to said composite electronic document 900 when it is fully syntactically parsed (since the specific context nodes for most of FIG. 9B correspond to the context nodes present in FIG. 7B, these same positions, as those present in FIG. 7B are used in FIG. 9B, and a duplicate explanation is omitted). In this case, the parser again notes the presence of data in the electronic document 900 related to annotation 902, and it classifies the data as a drawing made by electronic ink (ink drawing unit 912). In addition, based on the location and position of the ink stroke (s) relative to the contents of the base typescript in document 900, the parser recognizes and classifies the annotation as a “container” annotation and generates an ink container assembly 914 in the data structure 910. In addition, the parser stores all the stroke data corresponding to the actual ink stroke (s) of the container annotation in one or more stroke end nodes 916 (or alternatively, as a property in the ink container node or other corresponding node).

As with the underlining and strikethrough annotations described with respect to FIGS. 7A-8B, the container annotation can be saved with the words “is green” if for some reason the words are moved. Again, this can be accomplished, for example, by storing data in the data structure 910, meaning that the ink container context node 914 (in this example, the “source” context node) is associated with each of the text word nodes 768d and 768e for the words “is green "(in this example, the nodes of the context of the" destination "), as shown in figv arrows 918a and 918b. In addition, data can be stored in the data structure 910, meaning that the text word nodes 768d and 768e are connected by the container node 914, as indicated by arrows 920a and 920b in FIG. 9B. This relationship is a container type relationship. Therefore, if for some reason the data stored in the text word context nodes 768d and 768e means that the position and / or size of the words “is green” (or another word (s) associated with these nodes has changed in the electronic document 900 context), then links 918a, 918b, 920a and 920b allow the application program to find that, if possible, it should adjust the reproduced representation of the annotation container and / or its size to the location, spatial position and / or size of words stored in nodes 768d and context 768e in treason Mr electronic document. In addition, if the user adds words between the words “is” and “green,” or if the words or characters are deleted or changed in any way, the application program can resize and / or place the annotation container to accommodate the changes. Again, another communication scheme or data indicating communication can be stored without departing from the invention.

On figa and 10B depicts another type of commonly used annotations, namely annotations such as "comment on the margin." In this example, as shown in FIG. 10A, the user annotated an electronic document 1000 containing the sentence “Today the sky is green”, with a comment 1002 on the margins made by electronic ink stating: “it's not green!” (It is not green) . In this case, it should be noted that, as shown in FIG. 10B, the data structure 1010 associated with the typewritten text has been changed since the text is present on two lines in the electronic document 1000 (and not on one line shown in FIG. 7A 8A and 9A). Therefore, the data structure 1010 has two line nodes 1012 and 1014, and the first line node 1012 includes text word nodes 768a and 768b (associated with the words “Today” and “the”), and the second line node 1014 includes the nodes 768c , 768d and 768e are words of the text (associated with the words “sky”, “is” and “green”).

If the composite electronic document 1000 is fully syntactically parsed, the parser recognizes the annotation 1002 as containing text made by electronic ink, and it classifies the annotation (for example, due to its placement on the margins of the document 1000) as an annotation of the "marginal comment" type. Therefore, the parser generates a corresponding context node 1016 for commenting on the margins. Since this marginal comment 1002 contains only electronic ink text, the parser further generates the paragraph node 1018, two line nodes 1020 and 1022, and the corresponding ink word nodes 1024, 1026, 1028 and the stroke nodes 1030, 1032 and 1034 associated with hierarchical structure of electronic ink (as previously noted, optionally and alternatively, ink stroke data can be stored as properties of an ink word node (s) and / or line nodes can be excluded from data structure 1010).

Another binding relationship is depicted in the example of FIGS. 10A and 10B. Specifically, in this example, based on the placement and orientation of the marginal comment 1002 relative to the page margins and the contents of the base document, the parser stores the data in the data structure 1010, meaning that the paragraph node 1018 associated with the marginal comment 1002 (context node "Source" in this example) is associated with the paragraph node 764 associated with the typewritten text (the "destination" context node in this example), as shown in FIG. 10B by arrow 1036. In addition, the parser stores the data in the data structure 1010, meaning that the paragraph unit 764 associated with the typewritten text is connected by the field unit 1018 of the comment paragraph 1002 in the margin, as shown by arrow 1038. The above is a “hard link” type communication scheme in which the ink paragraph rigidly attached to a paragraph of text. Thus, wherever a paragraph of typewritten text in electronic document 1000 moves, links 1036 and 1038 allow the application to detect that, if possible, it should move the comment 1002 in the margins so that it remains in the margins next to the associated typewritten paragraph, even if optionally, additional lines and / or words will be added to the paragraph, and / or if the lines and / or words are deleted from the paragraph, and / or if other changes are made to the paragraph or electronic document 1000. Of course, there can be Aneny other linking scheme or data associated with communications, without departing from the invention. For example, the marginal comment node 1016 may be associated with one or more nodes of the individual typescript text words, such as the first text word node 768a.

On figa and 11B depicts to some extent a more complex example of an annotation recognized by systems and methods in accordance with at least some examples of the present invention. In the illustrated example, the electronic document 1100 again includes the phrase “Today the sky is green”, but in this case, the user annotated the electronic document 1100 using a combination of various types of annotations. Specifically, the user drew a container type annotation 1102 encircling the word “green”, a marginal comment type annotation 1104 containing the electronic ink word “Blue!” (Blue!), And a connector type annotation 1106 (for example, arrows in this example) directed from annotation 1102 of the type of container to annotation 1104 of the type of marginal comment.

Due to the relative position and spatial orientation of the various types of annotations with respect to the base document, typewritten text and document fields, the parsing system in accordance with at least some examples of the present invention recognizes various types of annotations, their contents and their relationship with each other and with typewritten text electronic document, as outlined above. 11B shows an exemplary hierarchical data structure 1110 for an electronic document 1100 generated by a parser. Since the hierarchical structure for typewritten text is similar to the structure described above in FIG. 7B, the same reference numbers are used for the structure in FIG. 11B, and a detailed explanation is omitted. Also, as shown in FIG. 11B, the node chain 1112 for the ink container type annotation 1102 is similar to the chain shown in FIG. 9B, and the node chain 1114 for the annotation type annotation 1104 is similar to the chain shown in FIG. 10B. Therefore, a detailed description of the given chains of nodes is not included.

In order to preserve the spatial position of the annotations with the corresponding part and the spatial arrangement of the base text, various relationships between the annotation nodes and the corresponding text node are provided by the parsing program and stored in the data structure 1110. In the illustrated example, the ink container assembly 1116 (in this case, the “source” context node) is associated with the text word node 768e for the word that it contains (ie, the “destination” node corresponding to the word “green” in this example), as 11B is shown by arrow 1118. Similarly, the data is stored in the data structure 1110, meaning that the text word node 768e is connected by the ink container assembly 1116, as indicated in FIG. 11B by the arrow 1120. The above is a “container” type coupling scheme .

As noted above, the parser in accordance with this example of the invention further recognizes that the electronic document 1100 includes an ink drawing 1106 corresponding to an ink connector (i.e., the arrow between the annotation container 1102 and the annotation type annotation 1104 in the margins in this example). This annotation connector 1106 is stored in the data structure 1110 as an ink drawing (represented by the ink drawing node 1122), which contains a node indicating the type of annotation (ink connector node 1124 in this example), which further includes data indicating a stroke or strokes constituting the connector (for example, in the block (s) 1126 of the stroke or in the property associated with the node 1124 of the ink connector (or other suitable node)). In addition, information is stored in the data structure 1110, meaning that the ink connector assembly 1124 (the “destination” node in this example) points from the ink container assembly 1116 (the “source” node in this example). This relationship (“pointing from” type relationship) is represented in FIG. 11B by arrow 1128. In addition, the data structure 1110 further includes data meaning that the ink connector assembly 1124 (the “source” context node in this example) is associated with the ink word node 1132 (the “destination” context node in this example). Said link (“pointing to” type link) is shown in FIG. 11B by arrow 1134. Alternatively, the ink connector assembly 1124 may be connected to any one node from the ink line assembly, the ink paragraph assembly, or the commentary node in the margins of the chain 1114 nodes without deviations from the invention. With the specific relationships described above, the data structure 1110 means that the ink connector 1106 points out from the outline 1102 and points to an ink word, as shown in FIG. 11A. Mentioned various relationships allow the application program to detect the association between the underlying document and the various components of the annotation. Then, if possible, the application program can move the annotation elements so that they correspond appropriately to the movements in the underlying document. Any suitable linking scheme or data associated with the link can be used without departing from the invention.

12A and 12B depict another example of a commonly used annotation feature. In this example, as shown in FIG. 12A, the electronic document 1200 again includes the “Today the sky is green” sentence, but in this case, the pen-computer user annotated the document 1200 using electronic ink to place a star or asterisk around the first word of the sentence (annotation shown at 1202). FIG. 12B depicts an exemplary hierarchical data structure 1210 corresponding to said composite electronic document 1200 when it is fully parsed (since the context nodes for most of FIG. 12B correspond to the context nodes present in FIG. 7B, the positions are the same with those which are present in FIG. 7B are used in FIG. 12B, and a duplicate explanation is excluded). In this case, the parser notes the presence of electronic ink data related to the annotation, and it classifies the data as an ink drawing (ink drawing unit 1212). In addition, based on the shape of the ink stroke, as well as the location and position of the ink stroke (s) of the annotation 1202 relative to the contents of the base document 1200 (for example, along a typewriter line), the parser recognizes the annotation as a “star”, “marker” or hard link annotations and generates an asterisk node 1214. In addition, the parsing program saves all the stroke data corresponding to the actual stroke (s) of the asterisk ink in one or more stroke end nodes 1216 or in the property of the asterisk node (or some other node).

In this example, an asterisk-type annotation 1202 can be saved with the entire line of typescript wherever the line appears in electronic document 1200 if for some reason the typescript is moved. This can be done, for example, by storing data in the data structure 1210, which means that the asterisk context node 1214 (the “source” context node in this example) is associated with the text line node 766 (the “destination” context node in this example), as shown in FIG. 12B by arrow 1218, and storing data, meaning that the text line node 766 is connected by an asterisk node 1214, as indicated by arrow 1220. Therefore, if for some reason the data stored in the text line context node 766 is mean that in electronic document 1200 if the position or placement of the line has changed, then the links 1218 and 1220 allow the application program to detect that, if possible, it should adjust the reproduced representation of the asterisk to the placement and / or spatial position corresponding to the reproduction of the data associated with the context node 766 in the modified electronic document. Alternatively, if required, the parser may associate an asterisk node 1214 (or any suitable node associated with an asterisk annotation) with a paragraph node 764 or one of the individual word nodes (eg, word node 768a) without departing from the invention. Also, if desired, the ink drawing assembly 1212 may be a source assembly mapped to a string assembly 766 (or another suitable assembly corresponding to the typescript in this example). Any suitable binding scheme or data representing the link may be used without departing from the invention.

A related annotation of the “block diagram” type is depicted in FIGS. 13A and 13B. In this example, the electronic document 1300 includes a block diagram, which, optionally, may be included as part of the annotation in the entire larger electronic document. As shown in FIG. 13A, the annotation or block diagram includes a first ink element 1302 (letter “A” in this example) circled or covered by the first annotation container 1304, and a second ink element 1306 circled or covered by the second annotation - container 1308. In addition, connector annotation 1310 (arrow in this example) extends from the first annotation-container 1304 to the second annotation-container 1308. It must be understood, however, that the relationships between elements of a document can be represented in an alternative way, different from t connections. For example, the mentioned relationships can be represented in the structure of the data tree itself by, for example, using container nodes that contain the associated elements of the document.

FIG. 13B illustrates an exemplary data structure 1320 that may be associated with a block diagram type annotation of the electronic document 1300 in FIG. 13A if it has been parsed completely. In said exemplary data structure 1320, a root node 1322, which may correspond to the entire electronic document 1300 or some part thereof, is provided as the parent node for the nodes in a block diagram type annotation. The first context node related to specific annotation structures is the ink drawing node 1324 corresponding to the ink container 1304. An ink container assembly 1326 is provided as a child node for the ink drawing assembly 1324, and the particular stroke (s) associated with the ink container annotation 1304 is stored in the end assembly (s) 1328 or as properties for the ink container assembly 1326 (or another node).

As noted above with respect to FIG. 13A, the annotation further includes electronic ink text corresponding to the letter “A” 1302. Like other electronic ink text described above, electronic ink associated with said text executed by ink, stored in the hierarchical data structure 1320 as a paragraph node 1330, which contains a line node 1332, which contains an ink word node 1334, which optionally contains one or ultiple end nodes 1336 corresponding to the individual strokes in text annotations made by means of ink or property comprising stroke data. The information stored in the data structure 1320 indicates that the ink container assembly 1326 of the annotation container 1304 (the “source” node) is associated with the paragraph “A” paragraph node 1330 (the “destination” context node), as shown in FIG. 13B by arrow 1338 Alternatively, if desired, the ink word node 1334 or line node 1332 may serve as a destination node. In addition, the information stored in the data structure 1320 also indicates that the paragraph assembly 1330 is connected through the ink container assembly 1326, as indicated by arrow 1340 in FIG. 13B. Since the data structure for the ink container 1308 and the letter “B” 1306 annotations in In this example, the same general structure is shared as for the ink container 1304 and the letter “A” 1302, including the same general binding structure, the different nodes in the data structure 1320 for the letter “B” 1306 and the associated container 1308 share the same positions as ass of dissociated with the letter "A" and its container, but the letter "b" follows the reference numerals for the container 1308 and ink word 1306 ink 13B. Of course, various hierarchical binding patterns and structures or binding data may differ from those specifically described above without departing from the invention.

The abstract further includes a connector annotation 1310 represented in the data structure 1320 by the ink drawing unit 1342. As described above in connection with FIG. 11B, the data structure for an ink pattern representing an ink container annotation may further include an ink connector assembly 1344, which may further include one or more properties or an end assembly (s) 1346, which includes data associated with the particular stroke (s) that make up the connector 1310. To complete the annotation data structure of the above example, the data associated with the ink annotation container 1310 (source node) is associated with yes GOVERNMENTAL associated with two annotations-containers 1304 and 1308 (destination nodes). Specifically, as shown in the example of FIG. 13B, the information is stored in the data structure 1320, meaning that the ink container assembly 1326 is connected to the ink connector assembly 1344 (represented by arrow 1348) as an “tip off” annotation. In addition, information is stored in the data structure 1320, meaning that the ink connector assembly 1344 is associated with the ink container assembly 1326b (represented by arrow 1352) as an “pointing to” annotation. Thus, the stored data related to the links means that the connector 1310 points from the container 1304 encirclement container 1308 encirclement, as shown in figa. Thus, the existence of various relationships allows the other parts of the annotation to track the movement of one part of the annotation in the electronic document 1300, thereby preserving the annotation intact, even if the electronic document 1300 changes in some way. Of course, this entire structure can also be associated with some element of the underlying document, as described above, without departing from the invention.

Similarly, the annotation may include two captors connected via a connector to a written comment, such as, for example, “change them!”. If one part of the annotation moves, then it may be necessary to independently move the other part of the annotation, for example, when words are entered between two captors. Links between context nodes representing annotations of electronic ink and nodes representing non-ink content can be used by the software application to move parts of the annotations and then link them back together when the placement operation is completed.

On figa and 14B depicts another example of an electronic document 1400 having an annotation or ink structure such as a block diagram. In this example, the annotation made by electronic ink for the letter “A” 1402 is connected to the annotations made by electronic ink for the letters “B” 1404 and “C” 1406, with the connectors 1408 and 1410 made by electronic ink being located between letters "A" and "B", and "A" and "C", respectively.

FIG. 14B shows an exemplary data structure 1420 for an electronic document 1400 after parsing. Data related to the entire electronic document or some part of it can be stored in the parent root node 1422. Data corresponding to the words “A”, “B” and “C” of electronic ink have the same node structure (1424a, 1424b, 1424c) paragraph, node (1426a, 1426b, 1426c) of the line, node (1428a, 1428b, 1428c) of the ink word and the node (s) (1430a, 1430b, 1430c) of the stroke, as used in several of the figures described above. A repeated description of the nodes is omitted. Similarly, ink connectors have the structure of an ink drawing assembly (1432a, 1432b), ink container assembly (1434a, 1434b), and a stroke assembly (s) (1436a, 1436b) used in the above-described FIGS. 11B and 13B, so that a repeated description is omitted. Of course, varieties can be used in the aforementioned specific data structures, including the possible varieties described above, without departing from the invention.

In order to complete the annotation data structure of the above example, the data associated with the ink annotation connector 1408 is associated with the data associated with the two ink annotations 1402 and 1404 that it connects and the data associated with the ink annotation connector 1410 , are associated with data associated with the two annotations 1402 and 1406 in the form of ink words that it connects. Specifically, as shown in the example of FIG. 14B, the information is stored in the data structure 1420, meaning that the ink drawing unit 1432a (the “source” node) “points to” the ink word nodes 1428a and 1428b (represented by arrows 1438 and 1440, respectively ) Similarly, information is stored in the data structure 1420, meaning that the ink drawing unit 1432b (the “source” node) “points to” the ink word nodes 1428a and 1428c (represented by arrows 1442 and 1444, respectively). The ink word nodes 1428a, 1428b, and 1428c serve as “destination” nodes. Due to various relationships, the movement of one part of the annotation in the electronic document 1400 can be tracked by the other parts of the annotation, thereby keeping the annotation intact or related, even if the electronic document 1400 is modified in some way. Also, this annotation may be associated with one or more other elements in an electronic document, such as typewritten text, drawings, images, etc., without departing from the invention.

FIGS. 15A-15C illustrate examples of an electronic document 1500 and exemplary data structures that include electronic table annotation in the form of a table having a plurality of rows and columns. Specifically, as shown in the example of FIG. 15A, the electronic document 1500 includes a table 1502 with two rows and two columns (forming four cells as a result), and each cell of the table is composed of borders made by electronic ink and contains input data in the form of text made by electronic ink.

FIG. 15B shows an exemplary data structure 1510 for the electronic ink table 1502 shown in FIG. 15A after parsing. In this example, the data structure 1510 includes a root node 1512, which may correspond to all or part of an electronic document 1500. Under the root node 1512, the corresponding table related data is stored in the table node 1514. They may include, for example, data determining the size of the table, the position of the table, the number of rows of the table, the number of columns, the size of the bounding box, and the like. Under the table node 1514, nodes of individual table rows are provided for each table row (in this example, as shown in FIG. 15B, two table row nodes 1516 and 1518 are provided). Each table row node of the exemplary data structure further includes one or more cell nodes (in the illustrated example, the table row node 1516 includes two cell nodes 1520 and 1522, while the table row node 1518 includes two cell nodes 1524 and 1526). Since each cell of this example contains electronic ink text data, the rest of said exemplary hierarchical data structure 1510 has a regular paragraph node, line node, ink word node and stroke node (s) described in detail above. Of course, the cells of the table may contain data of one or more other types in addition to and / or instead of electronic ink data, including, for example, typescript data and / or any of the various types of data described in detail above with respect to FIG. from the present invention. As another alternative, instead of table row nodes 1516 and 1518, the data structure may include column nodes, which then may include cell nodes, paragraph nodes, etc., as shown in FIG. 15B. The ink strokes that make up the actual table can be saved, for example, as part of a table node 1514 (for example, as properties associated with this node), as an ink drawing node that depends on table node 1514 (with associated strokes saved below it) and / or at any other suitable location in the data structure 1510.

FIG. 15C shows an alternative exemplary data structure 1530 for an electronic document 1500 including table 1502. In this example, the root node 1512; table node 1514; table row nodes 1516 and 1518 and cell nodes 1520, 1522, 1524 and 1526 are the same as those shown in FIG. However, instead of including text data generated by electronic ink as child nodes under various cell nodes 1520, 1522, 1524, and 1526, nodes 1528d, 1528b, 1528c, 1528d of a paragraph of text data made by electronic ink, ( or any other suitable or desired data structure) are associated with their respective cell nodes 1520, 1522, 1524 and 1526, as shown in FIG. 15C with arrows 1530a, 1530b, 1530c and 1530d, respectively. The cell nodes in this example can serve as “source” nodes of a container type for the ink words contained in them, and paragraph nodes, lines, or ink words serve as “destination” context nodes. Due to various relationships, the movement of one part of the annotation table in the electronic document 1500 can be tracked by the other parts of the annotation, thereby keeping the annotation table whole or linked, even if the electronic document 1500 is modified in some way. The table may also be associated with some other nodes present based on a basic electronic document. Again, as an alternative, table row nodes 1516 and 1518 can be replaced with column nodes that contain different cell nodes, without departing from the invention. Also, the ink strokes that make up the actual table can be stored at any location without departing from the invention, for example, as part of a table node 1514 (for example, as properties associated with a given node), as an ink drawing unit, which depends on a table node 1514 (with associated strokes stored underneath) and / or at any other suitable location in the data structure 1510.

Of course, various figures relating to the various exemplary types of annotations described above describe simply examples of possible types of annotations, the use of annotations, and information that may be included in annotations. It will be apparent to those skilled in the art that numerous modifications and changes can be made without departing from the invention. For example, annotations may contain numerous different types of data (such as electronic ink text, drawings, typescript and images, etc.) in many different combinations and permutations without departing from the invention.

In addition, various binding schemes are described in connection with the various data structures described above. Such binding schemes or techniques are merely examples of how different data nodes can be linked together. Any suitable linking scheme or data associated with the link may be stored without departing from the invention. For example, it is possible to represent each node as a separate identifier (such as a “global unique identifier”, or GUID) and to represent links using two (or more) GUIDs. As another alternative, a single bond can be used to bind the nodes, rather than the double binding scheme described in relation to some of the above examples. As another possible alternative, if necessary or if required, the node can be connected to itself (self-binding, for example, when the source node and the destination node are the same node). It will be apparent to those skilled in the art that any suitable method of linking or associating different data sets with each other can be used without departing from the invention.

Ink analysis

The following describes various exemplary methods for analyzing electronic ink in an electronic document in accordance with examples of the invention. In particular, FIGS. 16A-16E are a flowchart showing steps of analyzing a document in accordance with various examples of the invention. On Fig-26 depicts the relationship between the various components used during the analysis process.

17 depicts a software application 1701. The software application 1701 supports a document 1703, which may include both electronic ink data 1705 and non-ink data 1707, such as typewritten characters or images. As described in detail above, the properties of both electronic ink data 1705 and non-ink data 1707 can be described by a hierarchical data structure such as a tree. In order to begin the analysis of electronic ink, at step 1601, the software application 1701 creates such a data structure identified as the analysis context object 1709 in FIG.

On Fig shows an exemplary tree 1801 type, which can be included in the object 1709 context analysis. Tree 1801 includes a root node 1803 and a paragraph node 1805. Paragraph assembly 1805 corresponds to a paragraph of electronic ink text that, for example, may have been previously identified in an earlier analysis. The paragraph node 1805 is connected to two line nodes 1807 and 1809, which represent two lines in a paragraph of electronic ink text. The line node 1807, in turn, is associated with two word nodes 1811 and 1813. Each word node 1811 and 1813 corresponds to a word in a line of text made by electronic ink and represented by a line node 1807. The word node 1811 contains ink stroke data 1815 and ink stroke data 1817. Thus, the stroke data 1815 and 1817 correspond to strokes of text made by electronic ink constituting a word represented by a word unit 1811. Similarly, the word node 1813 contains ink stroke data 1819 and ink stroke data 1821 that correspond to strokes of text made by electronic ink constituting a word represented by the word node 1813. Tree 1801 also includes stroke data 1823-1827 that is associated only with root node 1803. Said ink data 1823-1827 represents new ink strokes that have not been classified or associated with another ink stroke or other document element.

Thus, instead of storing strokes in nodes of individual strokes, different nodes can have an associated property of strokes, which stores data corresponding to strokes associated with a node. For example: (a) nodes of an unclassified context may contain one or more properties of “strokes” having one or more strokes that need to be analyzed; (b) the nodes of the ink word may contain one or more properties of “strokes” containing one or more strokes that make up the ink word; (c) the nodes of the drawing can contain one or more properties of the "strokes" containing one or more strokes that make up the picture; and (d) the nodes of the marker can contain one or more properties of the "strokes" containing one or more strokes that make up the marker in list item. Of course, in accordance with alternative embodiments of the invention, nodes of individual strokes in the data tree can be used to represent individual strokes of electronic ink instead of associating ink strokes, for example, with a word node or a drawing node.

Although the tree 1801 shown in FIG. 18 includes ink strokes that are already organized into words, lines, and paragraphs, it should be understood that the tree 1801 can only contain new ink strokes that have not been classified or associated with another ink stroke or element of the document. For example, when a user initially enters electronic ink into a document 1705, said initial ink strokes are not analyzed. You must also understand that the tree 1801 is only indicative, and it is simplified to facilitate understanding of various aspects of the invention. For example, a line node is usually associated with a plurality of word nodes, and each word node may contain stroke data for several ink strokes. It should also be noted that, although the tree 1801 includes only ink-related nodes, the analysis context object 1709 may include nodes representing non-ink elements of the document, such as images and typewritten text, as described in detail above.

Returning now to FIG. 17, in accordance with some examples of the invention, software application 1701 creates and maintains its own analysis context object 1709. For said software applications 1701, software application 1701 may simply provide a reference to an analysis object 1709. For the mentioned software applications 1701, however, the application 1701 must include the mechanisms necessary to create, populate and maintain the object of analysis. Some software developers, however, do not want to take the trouble to create such mechanisms in application 1701.

Therefore, in accordance with other examples of the invention, software application 1701 may implement another object to create an analysis context object 1709. For example, software application 1701 may use an ink analysis tool or other object to create and / or maintain an analysis context object 1709. For example, a software application 1701 can identify or provide unprocessed ink data to an ink analysis tool. Depending on the instructions from the software application 1701, the ink analysis tool can then create an analysis context object 1709 representing the entire document 1703, or, alternatively, only a specific area (or areas) of the document that contains unanalyzed ink identified by the software application 1701. In accordance with some software applications 1701, the software application 1701 can then maintain and update the analysis context object 1709 either by itself or using services, both sintered by an ink analysis tool or other object. According to other software applications 1701, the software application 1701 may not support the analysis document object 1709, but instead the ink analysis tool or other object creates a new analysis document object 1709 when necessary.

Typically, the analysis context object 1709 contains information about the document elements for the entire document 1703. It should also be noted, however, that in accordance with some examples of the invention, the analysis context object 1709 can contain information for the document elements only in part of the document 1703. That is, an analysis context object 1709 can only represent document elements in the spatial zone of a document containing new or “dry” ink or other data. If all elements of the document in the document area, including electronic ink, have already been analyzed, then previously analyzed elements of the document cannot be included in the analysis context object 1709. As another alternative, the application program may support a separate analysis context object 1709 for each page or other subset of data related to the entire electronic document 1703.

If the software application 1701 created the analysis context object 1709, then in step 1603, the ink analysis tool 1901 copies at least a portion of the analysis context object 1709, as shown in FIG. More specifically, if the analysis context object 1709 is not limited to areas containing new ink and / or other data, as described above, then software application 1701 designates areas of a document 1703 that contain new ink and / or other data that needs to be analyzed. The software application 1701 then calls the ink analysis tool 1901 to copy a portion of the analysis context object 1709 corresponding to the designated area of the document 1703. (Of course, if the analysis context object 1709 is already limited to describing areas that contain new ink, then the ink analysis tool 1901 may copy the entire analysis context object 1709.)

Although the application program 1701 knows which ink has not been previously analyzed, it may not know which part (s) of the electronic document 1703 mentioned above (for example, previously analyzed ink, basic data in the document structure, etc.) influences new ink (and therefore which part (s) is to be presented to the ink analysis tool 1901). Therefore, in at least some examples of the invention, the application program 1701 makes available large sections of the electronic document 1703 for the ink analysis tool 1901 using the analysis context object 1709. The application program 1701 leaves the ink analysis tool 1901 as part of the responsibility of determining the information actually needed from the application program 1701 using the analysis context object 1709.

In response to a challenge to analyze new data, the ink analysis tool 1901 performs various callbacks of the analysis context object 1709, if necessary, to obtain information from the analysis context object 1709 necessary to analyze new ink and / or other data in the area indicated by the software application 1701. For example, in addition to new ink and / or other data, the ink analysis tool 1901 may request information from the analysis context object 1709 regarding ink and / or other data about The values of the areas that have already been analyzed, or information concerning nechernilnyh elements of the document in the designated area. A “designated area” may, in at least some examples of the invention, correspond to data in a spatial zone and / or near a zone containing new ink or other data to be analyzed.

Both previously analyzed ink and non-ink elements of a document, in particular those elements that are located near newly entered data to be analyzed, can provide a context that will improve the analysis of unanalyzed ink or other data. Since the ink analysis tool 1901 receives the required information from the analysis context object 1709, the ink analysis tool 1901 creates a document-independent analysis context object 1903 using the obtained information. Thus, the document-independent analysis context object 1903 contains at least a subset of the information contained in the analysis context object 1709, but it does not depend on the document 1703. Using this “callback” technique in the query-based spatial abstraction method described above, makes it possible for the ink analysis tool 1901 to efficiently obtain and analyze the necessary data, even for large documents. It should be noted, however, that application 1701 can still restrict content to a document-independent analysis context object 1903 by simply not opening the content to the analysis tool when prompted.

It should also be noted that although the ink analysis tool 1901 creates a document-independent analysis context object 1903, the software application 1701 must not modify the analysis context object 1709 (and it cannot allow other threads in the application program and / or other programs to change the object 1709 analysis context for a given period of time). Those. software application 1701 should not enter new data into document 1703 within a given period of time, and it should not allow other threads and / or programs of the application to do so. The process of creating an object of analysis context object 1903 independent of the document is relatively quick, but usually it does not significantly affect the operation of software application 1701. If you need to, however, attempts to enter data into document 1703 can be cached and entered into the system after creating the document independent object 1903 context analysis.

Once a document-independent analysis context object 1903 has been created, all subsequent analysis of the unanalyzed ink and / or other data can be performed with respect to the document-independent analysis context object 1903, and not with the analysis context object 1709, allowing the software application 1701 to continue its normal work without delay or stop as a result of the analysis of unanalyzed ink and / or other data. Software application 1701 can even enter new electronic ink data 1705 into a document 1703 (and into an analysis context object 1709) without interfering with the analysis of unanalyzed ink in a document-independent analysis context object 1903.

Therefore, after creating the document-independent analysis context object 1903 in step 1605, the ink analysis tool 1901 creates a separate analysis flow for analyzing ink in the document-independent analysis context object 1903. Therefore, at step 1607, the software application 1701 can resume control of the main processing flow and can continue its normal operation. Then, in step 1609, the ink analysis tool 1901 provides a document-independent analysis context object 1903 to the first analysis process for analysis. For example, as shown in FIG. 20, the ink analysis tool 1901 may transfer a document-independent parser object 1903 to the parsing process 2001 for parsing using the parsing stream. The parsing process may additionally include one or more classification processes without departing from the invention (for example, for classifying ink and / or other data into all kinds of different types, such as ink text, ink drawings, tables, charts, graphs, images, music , mathematics, drawings containing special characters (such as electrical circuits with resistors, capacitors, etc.), etc.). It should be noted that in accordance with some examples of the invention, the ink analysis tool 1901 can make a copy of the document-independent analysis context object 1903. As described in detail below, a copy of the original form of the analysis context-independent object 1903 can be used to reconcile the results of the analysis of the analysis process with the current state of the document 1703.

After the first analysis process analyzes the ink and / or other data in a document-independent analysis context object 1903, the first analysis process returns the analysis results to the ink analysis tool 1901, which, in turn, informs the software application 1701 in step 1611 of the analysis results . For example, as shown in FIG. 21, if the parsing process is a parsing process 2001, the parsing process returns the parsing results 2101 to the ink analysis tool 1901, which can then pass a link to the parsing results 2101 to the software application 1701 (e.g., by activation of the event).

As described in detail below, with some examples of the invention, the analysis result 2101 may be a document-independent analysis context object 1903, initially presented to the analysis process, but modified to include new information generated by the first analysis process (for example, parsing results in this example). According to other examples of the invention, however, the analysis results 2101 may be a modified copy of the document-independent analysis context object 1903. By including new information created by the first analysis process in a copy of the document-independent analysis context object 1903, the original document-independent analysis context object 1903 can be stored for use, for example, by means of reconciliation and / or other analysis processes, as noted earlier.

FIG. 22 shows an example of how a parsing process, such as a parsing process 2001, can modify a data tree 1801 to show layout changes created by a parsing operation. As can be seen in this figure, the parsing process 2001 determined that the stroke of the unparsed ink represented by the stroke data 1823 is part of the word represented by the word node 1813. Therefore, the parsing process disconnected the stroke data 1823 from the root node 1803 and instead associated it with the word node 1813.

The parsing process also determined that the ink strokes represented by the unparsed strokes data 1825 and 1827 are part of a new word in a line represented by a line node 1809 that was not previously identified. Therefore, parsing process 2001 created a new word node 2201 and associated this word node 2201 with line node 1809. The parsing process 2001 then associated the data 1825 and 1827 strokes with the new node 2201 words. Thus, parsing results 2101 describe the relationships between strokes of previously unanalyzed ink and other strokes of ink (or other elements of a document) identified by the parsing process 2001. Also, in addition to depicting the dependency changes determined by the parsing process 2001, the parsing results 2101 may also include classification information determined by the parsing process 2001. For example, each instance of ink stroke data 1823-1827 can be modified to classify its corresponding ink stroke as a stroke of text ink, rather than a stroke of ink of a drawing. In at least some examples of the invention, if required, parsing results 2101 can be presented in a document-independent analysis context object 1903.

As noted earlier, the software application 1701 can introduce new data into the document 1703, including both new electronic ink data 1705 and new non-ink data 1707, while the first analysis process analyzes the analysis context-independent object 1903. Therefore, the results of the first analysis process can no longer be applied to the current state of document 1703. For example, while the parsing process can determine that an ink stroke is associated with a word, the user can completely remove the ink stroke from document 1703 (for example, together with all associated words or to a larger extent). Therefore, the results of the analysis 2101 must be consistent with the current state of the document 1703 in step 1613. An exemplary approval process is depicted in a general way in FIG. 23, and is described in detail below.

In accordance with some examples of the invention, the software application 1701 can automatically match the results of the analysis 2101 with the current state of the document 1703. That is, software application 1701 can sort using analysis results 2101 to determine which of the results are related to document elements currently in document 1703. According to other examples of the invention, however, ink analysis tool 1901 can provide a matching function to reconcile the results of the analysis 2101 with the current analysis context object 1709 (i.e., the analysis context object 1709 reflecting the current state of the document 1703). As described in detail below, the matching function of various examples of the invention identifies conflicts or “collisions” between the analysis results of the first analysis process 2101 and the analysis context object 1709 for the current state of the document 1703. The ink analysis tool 1901 then updates the analysis context object 1709 based on the analysis results , in step 1615, and creates a new document-independent analysis context object 1903 from the agreed analysis results in the analysis context object 1709 in step 1617.

It should be noted that while the ink analysis tool 1901 matches the results of the analysis 2101 with the current analysis context object 1709, the software application 1701 should not modify the analysis context object 1709 (and it should not allow the analysis context object 1709 to change through other threads in application program and / or any other application program). Those. software application 1701 should not enter new data into document 1703, and / or it should not allow other threads and / or application program to enter new data into document 1703 until negotiation is complete. Like the process of initially creating the document context object 1903 independent of the document, the reconciliation process is relatively fast, and usually does not significantly affect the operation of the software application 1701. If required, however, user input can be cached and entered after the completion of the reconciliation process.

After the ink analysis tool 1901 matches the analysis results 2101 with the analysis context object 1709 for the current state of the document 1703, the ink analysis tool 1901 optionally can provide consistent analysis results to the second analysis process for further analysis. For example, as shown in FIG. 24, the ink analysis tool 1901 can provide consistent parsing results 2401 for the ink recognition process 2003 for recognition (for example, for recognizing handwritten text, music, mathematical information, or other types of special data). In particular, the ink analysis tool 1901 again creates a separate analysis flow for performing the second analysis process in step 1619. Since the ink analysis tool 1901 creates this separate flow for performing the second analysis process, the software application 1701 can resume its normal operation again step 1621, and it may even receive new input data, including new electronic ink data 1705.

Then, in step 1623, the ink analysis tool 1901 provides consistent results 2401 of the first analysis process to the second analysis process. After the second analysis process has begun, application 1701 may start another execution of the first analysis process for new unanalyzed ink. Thus, multiple analysis processes can be performed simultaneously, so that while the second analysis process is working on the results of the first analysis process, application 1701 can restart the first analysis process to prepare the next set of analysis results.

It is beneficial that parallel execution of the analysis processes can improve the performance of the 1701 software application. For example, the parsing process is usually quicker than the recognition process, and the results of the parsing can be used to implement the correct selection behavior and insertion behavior without using recognition analysis results. Thus, since various analysis processes can be performed in parallel, the new inks that are added to document 1703 will not have to wait until the recognition analysis process for the data of earlier inks is completed before they can be correctly selected or spaces are entered correctly in them . Further, various examples of the invention perform asynchronous ink analysis to protect application developers as well as developers of algorithms for parsing and recognition from multithreaded processing problems, improving maintainability and simplifying the development process for both groups, and allow changes in the interaction between analysis processes without changes to the application.

As noted above, the ink analysis tool 1901 can provide consistent analysis results by creating a new document-independent analysis context object 1903 from the analysis document object 1703 after it has been updated to include relevant parts of the analysis results 2101. According to other examples of the invention, however, the ink analysis tool 1901 can simply update the original document-independent analysis object 1903 with analysis results 2101 to reflect the current state of the document 1703. Further, the ink analysis tool 1901 can simply provide the corresponding ink data. subject to analysis, additionally to the second analysis process.

Thus, various examples of the invention align the results of the previous analysis process with the current state of document 1703 before performing the subsequent analysis process. You must understand, however, that the mentioned intermediate step of coordination may be excluded in some examples of the invention. More specifically, in accordance with some examples of the invention, the results of an earlier analysis process can be directly provided to the subsequent analysis process without reconciling the results with the current state of document 1703. For example, if the ink classification tool process is provided separately from the ink composition analysis tool, then ink classified as handwritten text can be provided to the ink composition analysis tool process without intermediate step of approval.

In any design, after the second analysis process 2003 completes the analysis of the agreed results 2401 of the first analysis process, the second analysis process 2003 returns its results 2501 to the ink analysis tool 1901, as shown in FIG. 25. The ink analysis tool 1901 then informs the software application 1701 of the results of the second analysis 2501 in step 1625.

As previously noted, the software application 1701 can introduce new data into the document 1703, including new electronic ink data 1705, while the second analysis process analyzes the consistent results 2401 of the first analysis process. The results 2501 of the second analysis process, therefore, can no longer be applied to the current state of document 1703. Therefore, the results 2501 of the second analysis process 2003 must also be consistent with the current state of document 1703 in step 1627. An exemplary approval process is depicted mainly in Fig.26, and it is described in detail below.

Again, in accordance with some examples of the invention, the software application 1701 may automatically reconcile the results of the second analysis 2501 with the current state of the document 1703. Alternatively, in accordance with other examples of the invention, the ink analysis tool 1901 may provide a matching function for reconciling the results of the 2101 analysis with the current object 1709 context analysis. Said matching function identifies conflicts or “collisions” between the analysis results of the second analysis process 2501 and the analysis context object 1709 for the current state of the document 1703, and the ink analysis tool 1901 then updates the analysis context object 1709 based on the analysis results in step 1629. This process it can then be repeated as necessary to analyze new ink or ink input that has been changed during analysis processes.

From the foregoing description, it is apparent that ink analysis techniques in accordance with various examples of the invention allow electronic ink analysis to be performed asynchronously with respect to software application 1701, allowing software application 1701 to continue to receive input data, including new electronic ink, during analysis processes. It should be noted that although only two analysis processes are described above, any number of parallel analysis processes can be performed simultaneously. For example, various embodiments of the invention may use a first parsing process, a second parsing process, and a recognition process. Thus, if the process of each analysis is completed, its results can be transferred to the process of the next analysis, and the process of the first analysis can again be performed simultaneously and in parallel with respect to the newly introduced ink.

Further, in accordance with various examples of the invention, one or more analysis processes may be performed sequentially. For example, in accordance with some software applications, a user may define a specific language for a ink stroke or a collection of strokes, such as English or Japanese. This language definition can then be included in the analysis context object 1709 and the document-independent analysis context object 1903. Based on the definition of the language, the ink analysis tool 1901 can direct the electronic ink defined as the first language to a recognition process designed for the first language and subsequently direct the electronic ink defined as the second language to another recognition process designed for the second language. In the same way, entered data, classified and / or defined as other specific types, can be sent to other specialized recognition tools, such as recognition tools for music, electrical symbols, mathematical symbols, features of flowcharts, graphic elements, etc.

In addition, since analysis processes do not have to be part of a software application 1701, any required analysis process can be used to analyze electronic ink in a software application 1701. For example, a software application developer can develop and use a parsing process or a recognition process that specifically suitable for analysis of electronic ink input as proposed by 1701 software application.

Still other varieties of these techniques may be used in various embodiments of the invention. For example, in some situations, it may be undesirable for the software application 1701 to maintain an internal document tree that reflects the relationships between the various elements of the document in the document 1703. Instead, the software application 1701 can only be used to use simple status information regarding electronic ink input. According to the mentioned software applications 1701, the software application 1701 can exclude the creation of an analysis context object 1709 that reflects the current state of the entire document 1703. Rather, the software application 1701 can create an analysis context object 1709 with a specific purpose that contains information relevant only to a particular input the ink that the 1701 software application is about to analyze. By using the analysis context object 1709 for a specific purpose, the software application 1701 can eliminate the complexity of supporting the internal data structure corresponding to the state of the document 1703, while at the same time being able to use the required analysis processes to analyze the ink in the document.

Also, although asynchronous analysis of electronic ink is described above, it should be noted that various examples of the invention may enable the ink analysis tool 1901 to perform synchronous ink analysis. For example, some embodiments of the invention may enable software application 1701 to instruct the ink analysis tool 1901 to perform an immediate and synchronous analysis of electronic ink. Further, synchronous analysis can be performed only for ink in a specific area of document 1703 or for the entire document 1703.

Ink Analysis Tool

As described above, the ink analysis tool 1901 performs a variety of functions for processing electronic ink. For example, in accordance with various embodiments of the invention, the ink analysis tool 1901 receives information from a software application 1701 indicating one or more areas in a document 1703 with ink data to be analyzed, requests the software application 1701 to obtain information regarding these areas, and then constitutes an analysis context object 1903 independent of the document based on the information mentioned. Further, the ink analysis tool 1901 provides a document-independent analysis context object 1903 to one or more analysis processes for analysis and then reconciles the results of the analysis process with the current state of the document 1703. Various processes can be performed using separate streams (as described above) using a pool of threads, in the form of separate processes, using separate machines in a cluster or in any other suitable or necessary way without departing from the invention.

In accordance with various examples of the invention, a variety of software objects and tools can be used to implement the ink analysis tool 1901 and perform these functions. In accordance with some examples of the invention, however, the ink analysis tool 1901 is implemented as an application programming interface (API). More specifically, in accordance with some examples of the invention, the ink analysis tool 1901 may be implemented as a group of program object procedures and related information that may, if necessary, be invoked by the ink analysis software application 1701 in document 1703.

For a person skilled in the art it is obvious that the application programming interface can be organized in the form of categories or "classes" related to program objects. An “object” is a collection of memorized state values and executable behaviors. More specifically, an object can support various types of state values, referred to as “fields” and “properties,” which can be associated with or retrieved by the object. In relation to external software applications or another program object, these properties can only be read (identified using the term "{get;}"), only written (identified using the term "{set;}") or read and written (identified as "{get ; set} "). An object can also perform tasks or “methods”, often at the request or “call” of a software application or other object. Thus, the application programming interface can include various objects that can be instructed by another software application to perform specific tasks or provide specific information upon request.

It should be noted that various examples of the invention can be implemented so as to require the software application 1701 to provide only the minimum amount of information necessary to complete the required analysis process. For this reason, some of the properties described in detail below can only be described as read-only properties (ie, "{get;}") or write-only properties (ie, "{set;}"), but can be both readable and writable for various alternative embodiments of the invention. For example, a property that defines “hints” that an analysis process should use to analyze electronic ink can be identified as a read-only property for situations where the 1701 software application does not intend to use hints. This allows the object created by Ink Analyzer to at least read this property and determine that the value of this property is the default value of zero (that is, "hints" are not required by the 1701 software application). If, however, the software application 1701 intends to use prompts for ink analysis, then this property will be both readable and writable, allowing the software application 1701 to change the value of this property from zero to specify the desired prompts.

In accordance with various examples of the invention, an API embodying an implementation of an ink analysis tool 1901 (referred to below as an ink analysis API) may comprise two main classes. The first class is referred to as the Analysis Context class, and the second class is the Ink Analyzer class. The components of the Analysis Context class are used to create the analysis context object 1709. Therefore, in accordance with various embodiments of the invention, for which the software application 1701 creates and maintains its own analysis context object 1701, one or more components in this class can be excluded from the ink analysis tool. Instead, one or more of these components may be implemented by the 1701 software application itself.

The components of the Ink Analyzer class are then used to create and use an object that provides a document-independent context object 1903 for analyzing the analysis process, determining when the analysis results were generated, and matching the analysis results with the current state of the document 1703. These and details are described below. other classes that may be included in the ink analysis API, in accordance with various examples of the invention.

Turning first to the Analysis Context class, this class is implemented by the host application 1701 to create an analysis context object 1703, which serves as an intermediary view of the internal document tree of the software application 1701. Alternatively, as noted earlier, if the application implements its own analysis context object, then one or more components in this class can be excluded from the ink analysis tool 1901. As described previously, the analysis context object 1703 contains all the unanalyzed ink data, and the analysis context object 1703 is used to identify which unanalyzed ink data should be analyzed. The analysis context object 1703 also contains information on previously analyzed ink. Mentioned previously analyzed ink can be used to decide on how to analyze currently unanalyzed ink, and themselves can be modified during the analysis of unanalyzed ink. Further, the analysis content object 1703 contains information about the non-ink content of the document 1703, which is used to correctly classify ink as annotations for the non-ink content.

In accordance with various examples of the invention, the Analysis Context class includes a constructor that, when called by software application 1701, creates an analysis context object 1709. This class may also include a variety of properties for the analysis context object 1709, including a property named “Dirty Region {get;}”. The Dirty Region property determines the part of the document (and, thus, the part of the analysis context object 1709) that contains the data of unanalyzed ink. In accordance with various examples of the invention, the Dirty Region may define an incompatible zone. The Dirty Region is defined as AnalysisRegion, which is described in detail below, but which simply can be a collection of rectangular zones in a document. If the Dirty Region is empty, then no analysis occurs. This class may further include a property for the analysis context object 1703 named “Margins {get;}”, which declares page zones in document 1703 that are considered fields. An analysis process that, for example, analyzes the layout and determines the classification of electronic ink, can use this property to help determine the classification of electronic ink that annotate non-ink content (for example, marginal ink).

Further, the Analysis Context class may include a property called “Rootnode {get;}” that identifies the topmost or root context node in the analysis context object 1709. As described in detail above, this context root node contains, as child context nodes, all other context node objects for this analysis context object 1709. It should be noted that in accordance with various examples of the invention, the root context node must be a “Root” type context node. It should also be noted that in accordance with examples of the invention, where the application 1701 implements its own analysis context object 1709, the analysis context object 1709 may have other context nodes as sister nodes of the root context node, but the components of the Ink Analyzer class may be limited to considering context nodes, which contains the context root node.

The Analysis Context class may further include the “Analysis Hints {get;}” property, which returns an array of analysis hint objects set by the 1701 software application. As described in more detail below, analysis hint objects can contain any type of information that can facilitate the analysis process. Mentioned information may include, for example, factoids, exhortations, or lists of words. It may also include information establishing the language that you want to use for analysis, information denoting unanalyzed ink as handwritten text or just a picture, or providing any kind of instruction for parsing, such as identifying ink as lists, tables , forms, block diagrams, connectors, containers, etc.

In addition to these properties, the Analysis Context class can also include a variety of methods that can be invoked, for example, by a software application 1701 to perform a task by an analysis context object 1709. For example, the Analysis Context class may include a method called "FindNode (Guid id)." Each node in the analysis context object 1709 has a globally unique identifier (or GUID), and this method determines the location of the node specified in the call, anywhere in the analysis context object 1709. This search method should be implemented in the most efficient way, since this method can be called from many time-critical operations.

Like the Analysis Context class, the Ink Analyzer class also defines an open constructor that allows the 1701 application to instantiate the class (i.e., an ink analyzer object) along with a variety of properties. For example, it may contain a property called “User Interface Context {get; set;} ", which defines the processing flow to which the results of the analysis process return. This property allows the results to be synchronized with another object. For example, if it is installed on the main form, then the results of the parsing program will be activated along the main flow of the application. It may also contain the property “AnalysisOptions AnalysisOptions {get; set;} ”, which sets various criteria that can be used for the analysis process. These criteria may include, for example, permission to recognize text, permission to use tables, permission to use lists, permission to use annotations, and permission to use connectors and containers.

The Ink Analyzer class includes various methods. For example, a given class may include a method called “AnalysisRegion Analyze ().” This method starts the synchronous analysis process. The data of the document elements is transmitted to this method, which describes the current state of the document 1703 and indicates which ink in the document 1703 should be analyzed. In accordance with some embodiments of the invention, data from document elements can be represented as an analysis context object 1709 (i.e., AnalysisRegion Analyze (Analysis Context)), as noted above. Alternatively, individual ink strokes can be passed to the analysis process, either by using a reference to the strokes (ie AnalysisRegion Analyze (Strokes)) or by referencing as a property of an Ink Analyzer object (eg, InkAnalyzer.Strokes {get; set}) without passing properties to the Analyze method.

After the analysis process is completed, this method returns a link to the document-independent object of the analysis context, which has been modified to contain the results of the analysis process. The method also returns an AnalysisRegion value (described in detail below) that describes the area in the document where the results were calculated.

The Ink Analyzer class can also include a method called "AnalysisRegion Analyze (AnalysisContext, waitRegion)." This method is similar to the synchronous Analysis Region Analyze () method described above, but it analyzes ink only if results in the specified waitRegion zone are required. More specifically, a call to this method identifies the analysis context object 1709 for the document 1703 and the region of the analysis context object 1709 (referred to as “waitRegion”) for which the analysis process must perform the analysis synchronously. In accordance with various examples of the invention, all other areas of the analysis context object 1709 will be ignored unless the analysis process needs to analyze the contents in these areas in order to complete its waitRegion analysis. As described above, the analysis context object 1709 passed to this method contains a property named “DirtyRegion” that describes areas of the document 1703 that require analysis. By setting a specific waitRegion, software application 1701 can obtain analysis results faster for one given area of interest, rather than analyzing all ink data in document 1703.

When any of these Analyze methods is called, each available analysis process will be executed. Also, since these Analyze methods are synchronous calls, there is no need to carry out the coordination process at their completion and the event will not be triggered if it is completed.

The Ink Analyzer class can also include a method called "BackgroundAnalyze (AnalysisContext)." This method starts the specified analysis operation, but does it in a separate background analysis stream. Thus, this method returns control to the main processing thread almost instantly, while the actual analysis operation completes in the background thread. In particular, this method returns true if the analysis process has been successfully started. Again, the AnalysisContext value passed to the method identifies the analysis context object 1709 for the document 1703 and indicates which ink in the document 1703 needs analysis. As soon as the analysis operation is completed in the background thread, the Results event is set, allowing the 1701 application to access the results. The event contains the results and a matching method that is used to embed the results back into the analysis context object 1709 for the current state of the document 1703 when the results are returned.

It should be noted that each of these three Analyze methods, in turn, calls the Clone method in the Analysis Region class, which is described in detail below. Using the Clone method, these Analyze methods create a document-independent analysis context object, which is subsequently modified by the analysis process to show the results of the analysis.

The Ink Analyzer class can also include a method called “Reconcile (AnalysisContext current, AnalysisResultsEventArgs resultArgs),” which the 1701 application calls after receiving a result event that was caused by a call to the BackgroundAnalyze (AnalysisContext) method. The Reconcile method compares the analysis results contained in a document-independent analysis context object with the current version of the analysis context object 1709 supported by the software application 1701. This method identifies nodes that need to be added and removed from the current version of the analysis context object 1709, and identifies changed whether any of the following properties of an existing node: its recognition results, its placement, ink strokes associated with the node, or any other data associated with res ltatami analysis operation. This method also writes the identified identified changes to the current version of the analysis context object 1709. This method is sensitive to the order of the location of the context nodes, such as the order of the word context nodes on the line context node.

Analysis results (i.e., the value of the AnalysisResultsEventArgs property) are passed back using this method, since they contain an open result structure and a closed result structure. An open structure is returned, so that the software application 1701 can preview the changes that will take place during the reconciliation phase. A closed structure is included in order to prevent software application 1701 from changing any analysis result before the approval process.

The Ink Analyzer class can also include methods with the names "Recognizers RecognizersPriority ()" and "SetHighestPriorityRecognizer (recognizer)". When ink needs to be recognized, an appropriate recognition tool is used based on language and capabilities. Therefore, the Recognizers RecognizersPriority () method returns the recognition processes in the order in which they are evaluated by the Ink Analyzer. The order is determined according to the system, depending on the recognition processes available, but can be redefined for the 1701 application by calling SetHighestPriorityRecognizer (recognizer) in relation to the Ink Analyzer object. InkAnalyzer lists this ordered list until the appropriate recognition tool is found. The SetHighestPriorityRecognizer (recognizer) method sets the priority of the recognition process. By prioritizing a particular recognition process, this recognition process will be used if it matches the desired language and capabilities of the current recognition operation. Essentially, the SetHighestPriorityRecognizer (recognizer) pushes the designated recognition process to the top of the list returned by the RecognizersPriority method.

The Ink Analyzer class may also contain a method called “AnalysisRegion Abort (), which can use the analysis context object as a parameter. This method provides the possibility of early completion of the operation of high priority or background analysis. This method returns the area of analysis that describes the area that was analyzed before premature termination. Thus, if the software application 1701 intends to continue the analysis operation later, then this area can be combined with the DirtyRegion of the analysis context object 1709 for the current state of the document 1703. Further, the Ink Analyzer class may include an event called “AnalysisResultsEventHandler” that is fired for InkAnalyzer objects as often as appropriate. More specifically, this event can be fired between analysis processes and at least every 5 seconds. This event can be used to provide application 1701 with updates regarding the status of a passing asynchronous analysis process (or processes).

The ink analysis API can also include classes in addition to the Analysis Context class and Ink Analyzer class. For example, an ink analysis API in accordance with various examples of the invention may include a Context Node class. This class may include various components related to the context nodes that make up the analysis context object 1709 and a document-independent analysis context object, such as a property named “ContextNodeType Type {get;}”. For a person skilled in the art it is obvious that each context node has a type, and there is a given set of rules that each type must adhere to. They include rules, such as, for example, what types of child context nodes are allowed and whether or not strokes can directly be associated with a context node or only through its child context nodes.

Possible types of context nodes can be defined in the ContextNodeTypes enumeration and can include (but not in a restrictive sense), for example, the following types: InkAnnotation node, which represents ink data annotating non-text data; an InkDrawing node that represents ink data forming a pattern; an InkWord node that represents ink data forming a word; Line node, which contains one or more InkWord nodes and / or TextWord nodes for words forming a line of text; ListItem node, which may contain Paragraph, Image nodes, or the like, assumed in the list; and a List node that contains one or more ListItem nodes, each describing an item in a list. Types of nodes may also include a NonInkDrawing node representing an image of a non-ink drawing; An Object node representing data not covered by other values of the ContextNodeType enumeration; a Paragraph node that contains one or more Line nodes corresponding to the lines forming the paragraph; a Picture or Image node representing the image of the drawing; Root node, which serves as the topmost node in the analysis context object; a Table node that contains nodes representing the elements that make up the table; TextBox node representing the text box; a TextWord node and an UnclassifiedInk node corresponding to ink data that has not yet been classified. Types of nodes can also include a Group node for groups of other nodes, an InkBullet node for marker elements, a Row node for document elements presented in a table row, and a Cell node for document elements presented in a table cell.

The Context Node class can also include a property called "GUID Id {get;}", which is a globally unique identifier for the current context node. In order to access any desired context node, each context node in a separate analysis context object must always have a unique identifier. This class may also include a property with the name “AnalysisRegion Location {get;}”, which determines the location in the document space where the corresponding context node is actually located. As noted earlier, AnalysisRegion is a two-dimensional structure that groups together one or more possibly disjoint structures like a rectangle. This class may also include a property called "StrokeCollection Strokes {get;}" that identifies ink strokes associated with the corresponding context node. In accordance with various examples of the invention, only context end nodes (such as Word, Drawing, and Bullet nodes) are allowed to have ink analysis APIs with strokes. Software application 1701 can use this property to refer to strokes at the end node level by all ancestor nodes of the context (for example, the root node contains a link to strokes to all strokes in the corresponding analysis context object).

Further, this class may include a property with the name “ContextNode ParentNode {get;}”, which identifies the parent context node containing the corresponding context node. In accordance with various examples of the invention, context nodes are always created depending on the parent context node, wherein the Root context node is a static member of the analysis context object. This class may also include the “ContextNode [] SubNodes {get;}” property, which identifies all context nodes that are immediate child nodes of the corresponding context node. Those. this property identifies only those child context nodes that are one level lower in the analysis context object. For example, the value of this property for a Paragraph context node identifies only the line context nodes contained in the Paragraph node, and not the word context nodes, which are child nodes of the line context node.

This class may also include a property called “RecognitionResult RecognitionResult {get;}”, which provides the recognition result calculated by the corresponding recognition analysis process or processes, since RecognitionResult can represent more than one line of text from more than one language. The Recognition Result is available for each context node in a document-independent context object for analysis, even if the RecognitionData property (described in detail below), which is set by the recognition analysis process and used to create the RecognitionResult object, can be set only on one level of the context node tree to avoid duplication data. If the node does not have the RecognitionData associated with it, then it either combines the recognition results of all its subnodes or extracts the recognition result from its parent node. This class may also include a property called Stream RecognitionData {get; set;} ”, which is a constant form of the value of RecognitionResult. Again, the recognition analysis process creates a Stream RecognitionData value that is set on the corresponding context node. The RecognitionResult object is then constructed based on the value mentioned.

The Context Node class can additionally include a property called “ContextLink [] Links {get;}”, which provides an array of ContextLink objects. The ContextLink object describes an alternative relationship between two context nodes. Although context nodes typically have a parent-child relationship with other context nodes, ContextLink allows for an alternative relationship between context nodes. For example, ContextLink can take into account the connection between two context nodes, tight binding of one context node to another context node, embedding of one context node in another context node, or the required type of communication defined by the 1701 application. ContextLinks can be added to this array by calling the AddLink method described in detail below. Similarly, ContextLinks can be removed from this array by calling the DeleteLink method, which is also described in detail below.

Further, this class may include the properties "IsContainer {get;}" and "IsInkLeaf {get;}". The IsContainer {get;} property is true if the corresponding context node is not a context end node (that is, if the corresponding context node contains child context nodes and is thus considered to be a container context node) and false " otherwise. The IsInkLeaf {get;} property is true if the current context node is not a container context node and false otherwise. That is, if the current context node does not contain any child context nodes, then it is considered the end context node. It should also be noted that with various examples of the invention, it is expected that the InkLeaf context node contains references to stroke data, while container context nodes do not have the above limitation. Container context nodes may or may not refer to stroke data, as determined by software application 1701.

The Context Node class may also contain the property "Rect RotatedBoundingBox {get; set;} ". The value of this property is calculated by the process of layout analysis and classification. If the ink data associated with the corresponding context node is recorded at an angle, then the boundaries of the context node will still be aligned horizontally. The value of the RotatedBoundingBox property, however, will be aligned with the angle at which the ink data associated with the corresponding context node was recorded. Further, this class may include the “ReClassifiable {get;}” property, which informs InkAnalyzer whether it is allowed to modify the values of the corresponding context node.

In addition to these properties, the Context Node class can also include various methods. For example, this class may include a method called “ContextNode CreateSubNode (ContextNodeType type)”. This method allows you to create a child context node of a specific type. In accordance with various examples of the invention, this method may allow the creation of only valid types of child nodes of the corresponding context node, thereby preventing the creation of distorted data structures. For example, this method may allow the Line context node to create only child InkWord and TextWord context nodes. This class may also contain a method called "void DeleteSubNode (ContextNode node)", which removes the child context node to which the link is made from the corresponding analysis context object. It should be noted, however, that in accordance with various examples of the invention, if the context node to which the reference is made still contains strokes or child context nodes, then this method should fail. Also, if the referenced context node is not a direct child of the corresponding context node, then this method should fail. It should be noted that if a software application 1701 implements its own analysis context object 1709 and, in turn, uses this method, it does not delete nonempty context nodes or context nodes that are not immediate child nodes of the corresponding context node to prevent distorted data structures in object 1709 context analysis.

Further, this class may include the “ContextNode [] HitTestSubNodes (AnalysisRegion region)” method, which returns an array of context nodes that are located in a given area. It should be noted, however, that only the immediate child nodes of this element are returned, and not all descendant nodes. A region is defined by an AnalysisRegion object, which, as previously noted, can be a collection of one or more rectangles. In accordance with various examples of the invention, if any part of the location of the context node intersects a given area, then this context node will be returned in the array. This method is used, for example, to create an analysis context object independent of the document and to coordinate the analysis results with the analysis context object corresponding to the current state of document 1703. Thus, this method is often called and should be optimized for quick multiple access from the InkAnalyzer object.

The Context Node class may also contain a method named "MoveStroke (Stroke stroke, ContextNode destination)." This method moves the stroke association from one end node of the context to another. In accordance with various examples of the invention, this method is used only between the end nodes of the context. It may also include a method called “MoveSubNodeToPosition (int OldIndex, int NewIndex)”, which reorders the corresponding context node relative to its sister context nodes. For example, if document 1703 has three words per line, for example, word 1, word 2, and word 3, then their order is the result of an array of subnodes returned from the parent context node. This method allows you to change their order, so that relative to the corresponding parent context node, word 1 is set by the last word in the line by moving the context node for word 1 from position one to position three.

Further, this class may include a method called “AddLink (ContextLink link)”, which adds a new ContextLink object to the current context node. In accordance with various examples of the invention, the ContextLink object must contain a reference to the corresponding context node so that the ContextLink is successfully added to the ContextLinks array associated with the corresponding context node. It may also contain a method called “DeleteLink (ContextLink link)”. This method eliminates or removes the specified ContextLink from the ContextLinks array for the corresponding context node. It should be noted that in accordance with various examples of the invention, this method call always succeeds, even if ContextLink does not exist in the ContextLinks array associated with the corresponding context node.

The ink analysis API may also include the Analysis Hint class. As with many of the previously described classes, the Analysis Hint class may include a constructor named “AnalysisHint ()” that initializes the Analysis Hint to an empty state. This class can also include a number of properties, including a property named “AnalysisRegion Location {get;}”. This property sets the location in document 1703 (as AnalysisRegion) to which AnalysisHint applies. For example, if document 1703 is a free form node with a title section at the top of the page, then application 1701 can establish that AnalysisHint for the title area specifies that a horizontal ink line is assumed in said area. This Analysis Hint will help improve the accuracy of the analysis process.

This class may also include a property called “string Factoid {get; set;} ", which specifies the specific" factoroid "that should be used for placement in document 1703, to which AnalysisHint applies. As a person skilled in the art knows, factoids provide clues to the recognition process regarding the expected use of ink data (for example, as plain text, numbers, postal codes, file names and Unified Resource Locators (URLs) of the World Wide Web (Web)). This class may also include properties with the names "RecognizerGuide Guide {get; set;} "and" OverrideLanguageId {get; set;} ". RecognizerGuide Guide Property {get; set;} specifies the record guidance to apply to the location in document 1703 to which AnalysisHint applies. Recording instructions, for example, can help improve the accuracy of the recognition tool analysis process by indicating to the user and informing the recognizer analysis process where the user will record lines or characters. Property OverrideLanguageId {get; set;} sets the Language Hint for document 1703 to which AnalysisHint applies. Setting the Language Hint causes the InkAnalyzer object to use the designated language instead of the language specified by the context node.

This class may also include a property named “PrefixText {get; set;} ", which sets the text that is written or printed before the line of ink that should be recognized. Further, this class may include a property named “RecognitionModes RecognitionFlags {get; set;} ”, which sets the specific type of modes that the recognition process must observe in the placement in document 1703, to which AnalysisHint applies. Further, this class may include a property named “SuffixText {get; set;} ", which sets the text that is written or printed after the ink line to be recognized, and a property named" WordList WordList {get; set;} ”, which defines a specific set of words that should be used by the recognition analysis process. Word lists can be used when the expected recognition results are known before the user actually records the input data, such as a list of medical terms that are expected to be recorded on the medical record.

Further, this class may include a property named “WordMode {get; set;} ". If the said value is “truth,” then the analysis process adjusts itself to return one word for the entire field of analysis. It may also include a property called “Coerce {get; set;} ”, which, if its value is“ true ”, forces the analysis process to limit its result to any factoid or word list value set in the corresponding prompt. This class may also include a property named “AllowPartialDictionaryTerms {get; set;} ". If the value of this property is “true,” then the recognition analysis process is allowed to return parts of the word from its recognition dictionary.

In accordance with various examples of the invention, the Ink Analysis API may further include an Analysis Region class. This class may include, for example, many constructors for constructing an AnalysisRegion object. For example, it may contain a first constructor for constructing an AnalysisRegion object that has any area, a second constructor for constructing an AnalysisRegion object based on parameters for a two-dimensional rectangle, and a third constructor for constructing an AnalysisRegion object based on four spatial coordinates. The default constructor, for example, can create an empty area. This class may also include a number of properties. For example, this class may include a property named “Rectangle Bounds {get;}” that retrieves the bounding box for AnalysisRegion, a property named “IsEmpty {get;}” that indicates whether the corresponding AnalysisRegion has an empty inner part, and a property called "IsInfinite {get;}" that indicates whether AnalysisRegion is set to infinity or not.

This class may also include a number of methods, such as a method called “AnalysisRegion Clone (),” which clones the corresponding AnalysisRegion object. This class may also include a method called “Equals (AnalysisRegion otherRegion)”, which checks if the specified AnalysisRegion object (referred to as otherRegion) is identical to the corresponding AnalysisRegion object. This method returns true if the inside of the specified Analysis Region is identical to the inside of the corresponding Analysis Region, and otherwise returns false.

This class can optionally include the Intersect (AnalysisRegion regionToIntersect) method, which truncates the corresponding AnalysisRegion object to the specified analysis area. Thus, the resulting AnalysisRegion object includes only zones that overlap or intersect the specified analysis area. This class may also include a method called "Intersect (Rectangle rectangle)" that trims the corresponding AnalysisRegion to the specified rectangle. Again, the resulting AnalysisRegion will only include zones that overlap or intersect the specified rectangle. It can also include a method called “MakeEmpty ()” that initializes the corresponding AnalysisRegion to an empty part, and a method called “MakeInfinite ()” that sets the zone occupied by the corresponding AnalysisRegion to infinity. It may additionally include various methods for combining or disconnecting variously defined zones, such as a method named “Union (AnalysisRegion regionToUnion)”, which sets the AnalysisRegion object to be merged or added to the corresponding AnalysisRegion object, and a method named “Union (Rectangle rectangle ) ”, Which combines the specified rectangle with the corresponding Analysis Region object. Using this method, a rectangle can be specified in the coordinate space values for the corresponding AnalysisRegion object. Of course, this class can include many other methods for combining zones or extracting one zone from another, based on any desired definition for the zones.

The ink analysis API can also have a Recognition Result class. As with the many classes previously described, the Recognition Result class can include one or more constructors. For example, this class may include a constructor named “RecognitionResult (Stream lattice)”, which constructs a RecognitionResults object from a given recognition lattice. In accordance with various examples of the invention, the recognition lattice is a sequential form of the results of the recognition process. This method, for example, can set the recognition lattice as a byte array to be used to construct the corresponding RecognitionResult object. It can also include a constructor named “RecognitionResult (ContextNode node)”, which constructs a RecognitionResults object from a given context node. It can also include a constructor named “RecognitionResult (string Text, int StrokeCount)”, which builds a RecognitionResults object from a given text value, which, in turn, is associated with a given number of strokes and can be used for correction if the recognition process does not will generate an alternate recognition value corresponding to the actual ink data. Further, this class may include a constructor named “RecognitionResult (RecognitionResult leftRecognitionResult, RecognitionResult rightRecognitionResult)”, which constructs a RecognitionResults object by combining two existing Recognition Results objects together.

The Recognition Result class can also include one or more properties, such as a property named "StrokeCollection StrokeCollection {get;}", which provides an array of stroke indices representing a collection of strokes that are contained in a separate ink object, and a property called "RecognitionAlternate TopAlternate {get;} ”, which provides the best alternative to a recognition result. This class may also include a property with the name “RecognitionConfidence RecognitionConfidence {get;}”, which provides a level of confidence (for example, strong, intermediate or weak) in the selected best alternative for the current results of the recognition analysis process, and a property with the name “string TopString {get;} ”, which returns the string of the best result from the analysis of the recognition analysis process.

The Recognition Results class can also include a number of methods, such as a method called “public RecognitionAlternateCollection GetAlternateCollectionFromSelection (selectionStart, selectionLength, maximumAlternates)”, which sets up a collection of alternatives from the selected fragment within the best result row from the analysis results of the recognition analysis process, where each alternative matches only one ink segment. Input parameters for this method may include, for example, a value that sets the beginning of the selected text fragment from which the collection of alternatives is returned, a value that sets the length of the selected text fragment from which the collection of alternatives is returned, and a value that sets the maximum number of alternatives for refund. This method can then return a collection of RecognitionAlternateCollection of alternatives from the selected fragment within the row of the best result from the recognition result.

The Recognition Results class may additionally include a method called “RecognitionResult Merge (RecognitionResult left, string separator, RecognitionResult right)”. This method can be used to create a new RecognitionResult object from a single row leading to a flat grid, attach one line to the beginning or end of an existing RecognitionResult object, or concatenate one line between two existing RecognitionResult objects. This class may also include a method called “ModifyTopAlternate (RecognitionAlternate alternate), which sets the modification of the recognition result using a well-known alternative. According to some embodiments of the invention, by default, the line of best result from the results of the recognition analysis process corresponds to the best alternative. However, this method can be used to specify that alternatives other than the best alternative are used in the results of the recognition analysis process. If the new best alternative leads to segmentation different from the previous one, then the ModifyTopAlternate method will automatically update the context nodes to reflect the changes. It should be noted that in order to extract alternatives that can be used to modify the recognition result, this method calls the GetAlternatesFromSelection method, described in detail below. This class may also have a method called “Stream Save ()”, which constantly supports the corresponding RecognitionResults object in the form of a recognition lattice. The recognition lattice is a serialized format used to express the results of the recognition process.

The ink analysis API can also be an enumerated type of Analysis Options. This type may contain one or more fields that specify how the ink data will be analyzed by the analysis process, such as a field called “Const AnalysisOptions Default”, which allows all available options for the analysis process. The mentioned field, for example, can allow text recognition, the use of tables, the use of lists, the use of annotations, the use of connectors and containers, and the use of intermediate results. The type may also include a field called “const AnalysisOptions EnableAnnotations” that enables and disables the detection of annotations, a field called “const AnalysisOptions EnableConnectorsAndContainers” that allows and disables the detection of connectors and containers, and a field called “const AnalysisOptions EnableIntermediateResults” enables and disables the return of analysis results to software application 1701 between the use of various sequential analysis processes (for example, between the parsing process and the subsequent recognition process). The type mentioned may also have a field called const AnalysisOptions EnableLists, which enables and disables the discovery of lists, and a field called const AnalysisOptions EnableTables, which enables and disables table discovery. Said enumerated type may further include a field called “Const AnalysisOptions EnableTextRecognition”, which enables and disables the text recognition analysis process. It should be noted, however, that if additional analysis processes are available (or different versions of the same analysis process), then this type may include, therefore, additional AnalysisOptions.

Further, the ink analysis API may include an AnalysisResultsEventArgs class. This class may have a constructor named “public AnalysisResultsEventArgs ()” that creates a data structure that contains the results of the analysis and returns to the 1701 application when the AnalysisResults event is set. This class may also include a property named "InkAnalyzer InkAnalyzer {get;}", which identifies the InkAnalyzer object that performed the analysis process.

The API can also have a Line class that can be used with some types of operating systems that recognize the use of a Line object that represents a geometric line. This class may include a constructor, such as a constructor named "public Line (Point beginPoint, Point endPoint)", which creates the Line object. This class may also include various properties, such as a property called "public Point BeginPoint {get; set;} ", which represents the starting point of the line object, and a property named" public Point EndPoint {get; set;} ”, which represents the endpoint of a line object.

In addition to these classes, the Ink Analysis API may also contain the Recognition Alternate class. This class may include elements representing possible word matches for ink segments that are compared with a dictionary of recognition tools. For example, this class may include a property with the name “Line Ascender {get;}”, which provides the line of the superscript element of the RecognitionAlternate object that exists on one line (moreover, the line is represented as two dots), a property with the name “public Line Baseline {get;} ”, which provides the Baseline of the RecognitionAlternate object that exists on the same line, and a property called“ Line Descender {get;} ”, which provides the line of the subscript of the RecognitionAlternate object that exists on the same line. This class may also include a property called "RecognitionResult Extract {get;}", which provides a RecognitionResults object for the current RecognitionAlternate object. This property can be used, for example, to retrieve the RecognitionResult object for a word from the RecognitionResult object for a string containing the given word.

It can also include a property called “Line Midline {get;}” that provides a midline for a RecognitionAlternate object that exists on the same line, a property called “StrokeCollection Stroke {get;}” that provides a collection of strokes contained in the ink object (that is, it provides a StrokeCollection representing the strokes that are associated with the RecognitionResult) and a property named "StrokeCollection [] StrokesArray {get;}" that provides a collection of strokes contained in one or more ink objects representing strokes that are associated with RecognitionResult. This class may also include a property called “RecognitionConfidence RecognitionConfidence {get;}” that provides the level of confidence (for example, strong, intermediate, or weak) that the recognition analysis process determines when the RecognitionAlternate object or gesture is recognized. For non-line nodes, the weakest RecognitionConfidence of the child nodes of the corresponding context nodes is returned. It can also contain a property called "string RecognizedString {get;}" that sets the string of the alternative result. Thus, for any context node above the word context node, the result string is concatenated together using this method. For example, a row node contains a result row, which in turn contains the results of all its child nodes or word nodes. The paragraph node then contains a result row that contains the results of all its child nodes or line nodes.

The Recognition Alternate class can also contain one or more methods, including, for example, a method called "StrokeCollection [] GetStrokesArrayFromTextRange (int selectionstart, selectionlength)", which sets the StrokeCollection from each ink object that corresponds to a known region of text. This class may also contain a method called "StrokeCollection [] GetStrokesFromStrokesArrayRanges (StrokeCollection [] strokesArray)", which defines the smallest stroke collection that contains the known stroke collection and for which the recognition tool can provide alternatives. More specifically, strokes are returned by an array of ink objects, each containing an array of stroke indices for the collection. It should be noted that the collection of ink strokes returned by this method may correspond to the entered collection, or it may be larger if the entered collection matches only part of the smallest recognition result, which includes all entered strokes. This class may additionally include a method called "StrokeCollection GetStrokesFromStrokesRanges (StrokeCollection strokes)", which defines the smallest stroke collection that contains the known stroke collection and for which the recognition tool can provide alternatives, and a method called "StrokeCollection GetStrokesFromTextRange int selection , int selectionlength) ”, which sets the StrokeCollection that matches the known area of the text.

This class may additionally include a method called “void GetTextRangeFromStrokes (ref int selectionstart, ref int selectionend, StrokeCollection strokes)”, which sets the smallest area of recognizable text for which the recognition tool can return an alternative that contains a known set of strokes, and a method named "void GetTextRangeFromStrokesArray (ref int selectionstart, ref int selectionend, StrokeCollection [] strokesarray)", which sets the smallest area of recognizable text for which the recognition tool can return an alternative that contains a known set of strokes. It may also have a method called “RecognitionAlternateCollection SplitWithConstantPropertyValue (GUID propertyType)”, which returns a collection of alternatives that represent the section of the alternative for which this method is called. Each alternative in the collection contains adjacent recognition segments that have the same property value for the property passed to the method. For example, this method can be used to obtain alternatives that divide the initial alternative by the level of confidence boundaries (strong, intermediate, or weak) as a result of recognition, line boundaries, or segment boundaries. It can additionally include a method called “byte [] GetPropertyValue (GUID propertyType)”, which sets the value of a known alternative property, such as recognizer confidence in the alternative. Not all recognition analysis processes, however, provide value for all types of properties. Thus, this method provides data for types supported by the corresponding recognition analysis process.

The ink analysis API can also include the Recognition Alternate Collection class. Like many of the classes described above, this class may include a constructor named “RecognitionAlternateCollection ()” to create the RecognitionAlternateCollection object. This class may also include a number of properties, such as a property with the name “Count {get;}” that provides the number of objects or collections contained in the collection of alternative recognition values, a property with the name “IsSynchronized {get;}” that provides a value indicating whether access to the collection of alternate recognition values is synchronized with software application 1701 (ie, “thread reliability”), and a property named “SyncRoot {get;}” that provides an object that can be used to synchronize access to count projection alternatives recognition values.

This class may also contain one or more methods, such as a method called “virtual void CopyTo (Array array, int index)”, which copies all the elements of the current collection of alternative recognition values to a given one-dimensional array, starting at a given index of the destination array, and a method called "IEnumerator IEnumerable.GetEnumerator ()", which is a standard implementation of IEnumerable that allows the caller to use for each construction to list each RecognitionAlternate in the collection of alternate recognition values. This class may also include a method called “RecognitionAlternateCollectionEnumerator GetEnumerator ()”, which returns a RecognitionAlternateCollectionEnumerator that contains all the objects inside the collection of alternative recognition values. This method can be used, for example, to retrieve each object in a collection of alternative recognition values.

The ink analysis API may further include a Recognition Confidence enumeration and a Recognition Mode enumeration, each of which may contain one or more fields related to the recognition analysis process. For example, the Recognition Confidence class may contain many fields, such as a field named "Intermediate", indicating that the recognition analysis process is sure that the correct result is in the list of provided alternate recognition values, a field called "Poor", which indicates that recognition analysis is not sure that the result is in the list of provided alternate recognition values, and a field with the name “Strong”, which indicates that the recognition analysis process is sure that the best alternative is recognition of alternatives is correct.

Similarly, the Recognition Mode class may include fields that specify how the recognition analysis process interprets electronic ink data and thus defines a string of recognition results. For example, this class may include a field called “Coerce”, which specifies that the recognition analysis process requires the recognition result to be based on the factoid that was specified for the context, and a field called “Line”, which specifies that the analysis process recognition recognizes electronic ink data as a single line. This class may also include a field with the name "None", which specifies that the recognition analysis process does not apply the recognition mode, and a field with the name "Segment", which determines that the recognition analysis process considers the electronic ink data as forming one word or sign. Further, this class may include a field named "TopInkBreaksOnly", which prohibits multiple segmentation.

Further, the ink analysis API may include a Context Link class that defines how two context nodes can be linked together. The ContextLink object itself represents which two context nodes are connected, the direction of communication and the type of communication. This class may include a property with the name “ContextNode SourceNode {get;}”, which defines the source context node that is associated with another context node, a property with the name “ContextLinkType LinkType {get;}”, which sets the type of communication relationship that exists between the source and destination context nodes, and a property named “CustomLinkType {get;}” that specifies that a non-standard relationship is used. This situation may occur when the application decides to use the ink analyzer API binding system to represent the relationships specific to the application, in addition to what the API can recognize.

This class may also include a property called “ContextNode DestinationNode {get;}”, which defines the destination context node that is bound from another context node. There can be two constructors available for a given class that create a relationship between existing source and destination context nodes.

This class may also include an enumeration with the name “ContextLinkType enum”, which defines the type of relationship shared by two context nodes. Mentioned various types of communication may include, for example, the type of "AnchorsTo", which describes that one node is rigidly attached to another node. Both nodes can use the SourceNode or DestinationNode property, based on the parsing situation. Communication types may also include the “Contains” type, which describes that one node contains another node. According to the relationship, a container node may be referred to as a SourceNode, while a container content node may be referred to as a DestinationNode. Communication types may further include a PointsTo type, which describes that one node points to another node. For the relationship, the pointing node may be referred to as a SourceNode, while the node being referred to may be referred to as a DestinationNode. Further, communication types can be of the type “PointsFrom”, which describes what one node points from another node. With this relationship, a node pointing from another node may be referred to as a SourceNode, while a node pointed to may be referred to as a DestinationNode.

Types of links may additionally include a “SpansHorizontally” type, which describes that one node runs along the length in the horizontal direction of another node, and a “SpansVertically” type, which describes that one node runs along the length in the vertical direction of another node. For these types of nodes, the node covering (strikethrough, underlining, dash line) another node, usually written last, may be referred to as SourceNode, while a node that is overlapping may be referred to as DestinationNode. Link types can also include a “Custom” type, which describes that a custom link type is being used. When the said value is used, the “CustomLinkType” property of the ContextLink object can provide more details regarding the purpose of the association.

Therefore, the Ink Analyzer API provides a variety of functions and services for asynchronously analyzing electronic ink in a document and then subsequently matching the results of the analysis process with the current state of the document, as described in detail above. Further, you need to understand that the various classes described above can be applied to a variety of operating systems and environments, such as the Microsoft Windows operating environment, the Microsoft COM operating environment, the Unix or Linux operating environment, or any other suitable computer operating environment. In addition, it should be understood that the application programming interface in accordance with various embodiments of the invention may exclude one or more of the components of the class described above, or may include additional components to obtain the desired service or functionality.

Matching

As described in detail above, various examples of the invention allow software application 1701 to continue to operate, while unanalyzed electronic ink in document 1703 is analyzed through a background analysis process. As a result, software application 1701 may modify document 1703 in various ways that will conflict with the results of the analysis process. For example, software application 1701 can enter new electronic ink data 1705 into document 1703 or delete existing electronic ink data 1705 from document 1703. Further, software application 1701 can edit existing electronic ink data 1705, for example, by moving the position of existing electronic ink data 1705 or by changing properties of existing 1705 electronic ink data. Further, software application 1701 can add, remove, or modify non-ink elements 1707 of a document in a manner that affects existing electronic ink data 1705. For example, software application 1701 may remove typewritten text that has been annotated with electronic ink data 1705.

Software application 1701 can optionally “pin” existing electronic ink data 1705 in order to prevent it from being altered by the analysis process. For example, if the user manually sets the layout or classification for a group of ink strokes, then the software application 1701 may indicate that these ink strokes remain in the specified layout or with the classification, regardless of the results obtained by the parsing process. Similarly, the user can set a specific recognition result for a group of ink strokes, regardless of the results obtained by the recognition process.

Various types of fastening can be used in accordance with various embodiments of the invention. For example, the ink analysis tool 1901 may allow the software application 1701 to "fix" the ink. In accordance with the above scheme, ink strokes cannot be added to any of the end nodes under the fixed node, strokes cannot be removed from any end nodes under the fixed node, neither the addition nor deletion of child nodes is allowed, and re-permission is not allowed the formation of parent nodes for any node under the attached node. Alternatively or additionally, the ink analysis tool 1901 can enable “hard” pinning with later more strokes, which allows the addition of later strokes under specified conditions, prohibits the removal of strokes from any end node under the fixed node, prohibits the addition or removal of child nodes, and prohibits the re-formation of parent nodes for any node under the attached node. Further, the ink analysis tool 1901 can enable the software application 1701, alternatively or additionally, to “gently” fix the ink. In accordance with the above scheme, strokes cannot be removed from any end node under a fixed node, there are predefined rules allowing the addition of strokes (adding late strokes is allowed), and rearrangement, addition and removal of child nodes are allowed. It should be noted that the pin can be removed, and if the pin was removed, then previously pinned nodes and their child nodes can then be considered as “changed” or requiring re-analysis.

Further, an ink processing process in accordance with various examples of the invention may utilize numerous analysis processes, as described in detail above. Therefore, the results of an earlier analysis process may modify the electronic ink data 1705, while the subsequent second analysis process is being performed. Therefore, the results of the analysis process must be consistent with the current state of the document 1703, so that only those results that are valid for the current state of the document 1703 are applied to its analysis context object 1709. Those. the current analysis context object 1709 (and, in some cases, the results of the analysis process) is modified to eliminate discrepancies or “conflicts” between the results of the analysis process and the current state of the document 1703.

It should be noted that in order for the results of the approval process to be valid, the status of document 1703 should not change during the approval process. The matching process may thus be performed using the primary thread in which the software application 1701 is executed, and the matching process may temporarily stop the operation of the software application 1701. Alternatively, other techniques, such as blocking the data structure, may be used to ensure that the state of document 1703 does not change during the approval process. Therefore, it is desirable that the approval process be performed as quickly as possible in order to eliminate user dissatisfaction with the performance of software application 1701. Another issue in the approval process that needs to be considered is its effect on the performance of the analysis process running in a separate background analysis flow. If the discrepancies between the current analysis context object 1709 (i.e., the analysis context object 1709 reflecting the current state of the document 1703) and the analysis results are too broadly defined, then large quantities of electronic ink will be re-analyzed without need. Of course, other ways of protecting the integrity of the document during approval can be used without departing from the invention.

In accordance with various examples of the invention, the analysis process and the matching process may adhere to one or more of the following conventions in order to increase the efficiency and convenience of the matching process. First, the analysis process can, as far as possible, reuse nodes for a document element in a document-independent analysis context object 1903. Those. Collisions should not be canceled by creating new unrelated nodes for document elements. In addition, the matching process must adhere to the “pinning” indicated by the software application 1701. Although this analysis process usually follows the pinning labels of the software application 1701, the software application 1701 can pin the electronic ink data 1705 during the analysis process. Further, the approval process should ensure that there are no empty nodes in the current analysis context object 1709 after the approval process is completed. It should be noted, however, that one or more of the above agreements may be omitted and may not be performed according to alternative embodiments of the invention. For example, some embodiments of the invention may allow the analysis object 1709 to contain empty nodes.

In addition to these conventions, the reconciliation process should typically comply with the interface rules dictated by the analysis context object 1709. For example, in accordance with some embodiments of the invention, the analysis context object 1709 may not allow the removal of a node for a document element unless it has no child nodes.

As previously noted, collisions occur when the analysis process makes a change in the document-independent analysis context object 1903, which conflicts in some way with the change in the analysis context object 1709 that was made after the initiation of the analysis process. Collisions can be divided into two types: mandatory collisions and discretionary collisions.

Mandatory collisions occur when you cannot apply the change made by the analysis process in the document independent of the analysis context object 1903 to the analysis context object 1709 for the current state of the document 1703. Mandatory collision occurs when, for example, the software application 1701 “fixed” or fixed the node in the analysis context object 1709, and the analysis process changed the corresponding node in the document-independent analysis context object 1903. Mandatory collision also occurs when the analysis process made a change to any kind of node in the document context analysis object 1903, but the software application 1701 removed the corresponding node from the analysis context object 1709, and when the software application 1701 added strokes or child nodes to the node in the analysis context object 1709 when the analysis process deleted the corresponding node in the document-independent analysis context object 1903. Further, a mandatory collision occurs when the software application 1701 deleted the node in the analysis context object 1709, when the analysis process reordered or created a connection with the corresponding node in the document-independent analysis context object 1903.

Discretionary conflict occurs when the software application 1701 changes the value in the analysis context object 1709, which is associated with the value changed by the analysis process in the analysis context object 1903 independent of the document, but the constraints of pinning, restrictions on reuse of the element and the inherent limitations of the interface for object 1709 the analysis context can still allow the application of the change made by the analysis process in the analysis context object 1709. Discretionary conflict can still be applied as a change to the analysis context object 1709 or canceled. Further, the reconciliation process may simply completely ignore some types of discretionary collisions.

One graphic example of discretionary collision occurs when the source node, both in the analysis context object 1709 and in the analysis context object 1903 independent of the document, has child nodes A and B for ink strokes A and B. Software application 1701 can then add a third child node C for ink stroke C in the analysis context object 1709, while the analysis process adds a third child node D for ink stroke D to the document-independent analysis context object 1903. In accordance with various embodiments of the invention, the reconciliation process may add a child node D to the parent node in the analysis context object 1709 based on the analysis results. Although this change of the parent node in the analysis context object 1709 is not prohibited, it nevertheless changes the characteristics of the parent node in a way that may not be desirable for the 1701 software application. For example, the ink associated with the parent node may subsequently be considered analyzed and may not be reanalyzed to take into account the effect of ink C on a group of ink strokes. Further, the recognition results for the parent node can now refer to incorrect child nodes or strokes, and this can never be corrected again.

Therefore, various examples of the invention do not apply changes to discretionary conflict when updating the analysis context object 1709 based on the results of the analysis process. Although the mentioned criteria may require additional processing to identify and block changes based on discretionary conflicts, the mentioned criterion is relatively simple to implement and simple to maintain. Of course, other examples of the invention may make changes corresponding to discretionary collisions according to other criteria. More specifically, these alternative examples may implement changes from discretionary collisions that do not create permanent logically incompatible relationships in the analysis context object 1709.

It should also be noted that collisions can have a transitive effect, namely that one collision can create another. For example, the analysis process can create a node L for a string and then create a node W for the word as a child node for node L. If the creation of node L was not applied to the analysis context object 1709 due to a collision of any kind, then the creation of node W will be mandatory collision.

Various examples of the invention may use a logging-based approach to align the results of the analysis process with the current state of document 1703. In this logging-based approach, the document-independent analysis context object 1903 includes a protocol of changes to the document-independent analysis context object 1903 made by the analysis process . The protocol may be in the form of, for example, a list of change records. Each change record, then, may include the type of change that was made (by identifying, for example, the method that was called to change the analysis context object 1903 independent of the document), the element of the document in which the change was made (by identifying, for example, a node in an analysis context object 1903 independent of the document for which the method was called), and any information needed to recreate the arguments to invoke the method. It is advantageous because the document-independent context object 1903 is performed by the ink analysis tool 1901 and the analysis processes so that the change protocol can be invisible to the software application 1701 (although the protocol can alternatively be opened to application 1701 if necessary).

In order to complete the reconciliation process using the change logging approach, the ink analysis tool 1901 examines each change record in a chronological order of changes. Those. the ink analysis tool 1901 identifies each change made to a document-independent analysis context object 1903. For each change, the ink analysis tool 1901 can perform the process depicted in FIG. First, in step 2701, the ink analysis tool 1901 attempts to access the corresponding nodes in the current analysis context object 1709 to which the change needs to be applied. It should be noted that this extraction step may not succeed if the software application 1701 deleted one or more necessary nodes from the analysis context object 1709, leading to a mandatory collision.

Then, at step 2703, the ink analysis tool 1901 determines whether the change creates a mandatory or discretionary collision. The procedure for performing this definition is explained in detail below. In step 2705, the ink analysis tool 1901 either makes a change or prohibits the change if it creates a mandatory conflict or discretionary conflict that is prohibited by the criterion for the matching process. For example, if the change creates a mandatory conflict or discretionary conflict, prohibited by the criterion for the approval process, then the ink analysis tool 1901 can block changes to nodes in the analysis context object 1709 representing the corresponding area of the document 1703 in which the change was to be made. If, on the other hand, the change is applied, then the ink analysis tool 1901 can invoke the appropriate method to make the desired change to the necessary nodes in the analysis context object 1709.

If the analysis process adds a new element node to the analysis context object 1709, but then cannot move the stroke nodes to a new node, then the analysis process will also not attempt to delete the element node on the assumption that the stroke nodes have been successfully moved to new nodes. Thus, in the document-independent analysis context object 1903, there will be an element node that corresponds to an empty node in the current analysis context object 1709. Therefore, if all changes to the analysis context object 1709 were processed, then at step 2707, the ink analysis tool 1901 scans the document-independent analysis context object 1903 to remove any “empty” nodes in the analysis context object 1709 that correspond to nodes in the document-independent object 1903 context analysis. This step of removing empty nodes is optional and may be omitted without departing from the invention.

It should be noted that the document-independent analysis context object 1903 should not contain any empty nodes, and the reconciliation process should prohibit the creation of empty nodes in the document-independent analysis context object 1903. It should also be noted that this step, as an exception, does not attempt to delete the root node of the analysis context object 1709, even if it is empty. Finally, in step 2709, the ink analysis tool 1901 identifies any collisions between the analysis results in the document-independent analysis context object 1903 and the current analysis context object 1709 for the software application 1701, so that the software application 1701 can include areas of the document 1703 affected by the collisions in subsequent analysis process.

Returning now to collision detection in the aforementioned step 2703, if all nodes in the analysis context object 1709 are available that correspond to changes in the document of the analysis context object 1903, then all other possible collisions for each node in the analysis context object 1709 can be detected statically ( or alternatively, the absence of a mandatory collision can be detected for each node). More specifically, mandatory collisions can be detected based on the rules indicated by the interface for the analysis context object 1709.

Discretionary conflicts, however, usually cannot be identified without additional status information regarding document 1703, since these changes are not authorized by the interface of the analysis context object 1709, but, instead, are based on a discretionary decision about the authorization of changes made by software application 1701, regarding the changes made by the analysis process when both acted on the same node in the analysis context object 1709. Instead, these collisions can be detected by comparing the current analysis context object 1709 with the original version of the document-independent analysis context object 1903 to determine which nodes in the analysis context object 1709 have been changed by software application 1701.

For example, a matching criterion may determine that the addition or removal of a stroke child node from a parent node by software application 1701 constitutes a discretionary conflict for a given parent node. To determine whether such a change has occurred, the ink analysis tool 1901 can look at the child nodes of the stroke, depending on the node in the current object 1709 of the analysis context, and compare their number and GUID (or other unique identifier) with the child nodes of the stroke, depending on the corresponding node in the original version of the document-independent object 1903 analysis context. If the child nodes of the stroke are consistent in these aspects (and in any other required aspects), then the ink analysis tool 1901 can apply the changes made by the analysis process to the parent node in the analysis context object 1709.

Continuing this example, the next change in the protocol entry may consist of another stroke node being moved to depend on the parent node in the document-independent analysis context object 1903. If there are no mandatory collisions due to this change, then the ink analysis tool 1901 is allowed to make a corresponding change to the analysis context object 1709. When the ink analysis tool 1901 checks the original version of the document-independent analysis context object 1903 to determine if there is a discretionary conflict, however, it concludes that the parent node in the current analysis context object 1709 has one more dash node more than the parent node in the original version of the document-independent object 1903 of the analysis context, so that it will erroneously determine that there is a discretionary conflict (i.e. it will determine that there is a conflict based on the changes it made).

Therefore, the logging-based reconciliation process should exclude previous changes made as a result of the reconciliation from the comparison between the current analysis context object 1709 and the original version of the analysis context object 1903 independent of the document. In accordance with various examples of the invention, this exception can be made by comparing all nodes in the current analysis context object 1709 with their corresponding nodes in the original version of the document-independent analysis context object 1903 before examining any elements of the change list and saving the list of nodes that have collisions.

Alternatively or additionally, various examples of the invention may maintain a cache of elements in the current analysis context object 1709 corresponding to elements in a document-independent analysis context object, with analysis results that are used whenever appropriate elements are retrieved. There is no preliminary transfer, but the cache is filled the first time when it is determined that the corresponding element is free from discretionary collisions. Optionally, the cache can also track which elements in the results have discretionary collisions. This approach has better performance than the approach with the transfer of finding discretionary conflicts, when the change protocol is short, but can have worse performance if the document-independent analysis context object 1903 with the analysis results contains many nodes. It is also no more complicated than another approach.

It should be noted that the parent nodes of the context of elements in the cache must also be cached. More specifically, a change in a child context node is considered as a change in the properties of its parent context node. Thus, the parent context node of the context node stored in the cache also needs to be evaluated with respect to the changes made by the 1701 software application at the moment. Caching the parent nodes for each context node in the cache requires repeating up the tree (for example, from the parent node to the grandfather node and to the great-grandfather node) until the root node is traversed or changes are detected. In many situations, however, this repetition is only a few nodes, since the tree usually does not get very tall.

It should also be noted that, in accordance with various examples of the invention, the original version of the document-independent object 1903 of the analysis context is used only to detect discretionary collisions. If detection of discretionary collisions is optional in accordance with an example of the invention, then both the original version of the document-independent analysis context object 1903 and the cache of the corresponding elements can be excluded. This “logging-based” approach may be easier to support than the “logging-based” approach described below, since it requires less search to determine what changes were made by the analysis process and operations on a document-independent object 1903 analysis contexts (such as “delete node” or “create node”) are processed separately, so adding a new type of operation does not tend to affect other operations.

Instead of a “protocol-based” methodology for matching analysis results with the current state of document 1703, various examples of the invention may use a “comparison-based” approach to harmonize. The main distinguishing feature of the comparison-based approach is that there is no protocol for changes made to the document-independent analysis context object 1903, so the technique does not collect information about what the analysis process did otherwise than by comparing the document-independent context object 1903 analysis containing the results of the analysis, with the original version of the document-independent object 1903 of the analysis context. Therefore, in accordance with the mentioned approach, the initial version of the document-independent object 1903 of the analysis context is always required regardless of any estimates of discretionary collisions.

Using the above approach, the ink analysis tool 1901 first builds a stroke map. Those. for each ink end node in the original version of the document-independent analysis context object 1903, if there is a corresponding node in the current analysis context object 1709, then the end node is added to the hash table (or other suitable data structure). Thus, the hash table displays the GUIDs (or other unique node identifiers) of the stroke in the original version of the document-independent analysis context object 1903 for links to end nodes in the current analysis context object 1709.

Then, the ink analysis tool 1901 identifies all nodes that the software application 1701 has changed in the analysis context object 1709. This can be done using techniques for determining discretionary collisions, described in detail above. For each node in the original version of the document-independent analysis context object 1903, if there is a corresponding node in the current analysis context object 1709, and it differs to some extent significantly from the node in the original version of the document-independent analysis context object 1903, then the tool The 1901 ink analysis determines that a given node has been modified by the application and represents a potential collision. Links to such nodes are stored in the first list corresponding to the nodes of the analysis results, which should not be added to the analysis context object 1709 (hereinafter referred to as the list of resultsNodesNotToAdd ), and in the second list of nodes that create discretionary collisions.

For each node in the original version of the document-independent analysis context object 1903, the ink analysis tool 1901 searches for the corresponding nodes both in the document-independent analysis context object 1903 with the analysis results and in the current analysis context object 1709. For each node in the analysis context object 1709, the ink analysis tool 1901 determines whether the software application 1701 has deleted the node. Such nodes exist in the original version of the analysis context object 1903 independent of the document, but do not have a corresponding node in the analysis context object 1709. The corresponding node in the document-independent object 1903 of the analysis context with the analysis results is added to the list of resultsNodesNotToAdd , if it exists.

Then, the ink analysis tool 1901 determines whether the analysis process has removed a node from the document-independent analysis context object 1903. This is detected when a corresponding node exists in the current analysis context object 1709, and it has not changed, but no corresponding node exists in the document-independent analysis context object 1903 with the analysis results. The nodes deleted by the analysis process can be stored in the list of remote nodes. Such a node is not added to the current analysis context object 1709, but it is not included in the resultsNodesNotToAdd list because there is no corresponding node in the document-independent analysis context object 1903 with the analysis results.

If the software application 1701 neither deleted nor changed the node, and the analysis process did not delete the node, then the ink analysis tool 1901 can determine if the analysis process has changed the node. The same techniques as those described above for use in detecting changes made by software application 1701 can be used to detect changes made by the analysis process. The nodes that have been modified by the analysis process are added to the resultsNodesNotToAdd list, and the changes are transferred by calling the appropriate methods for the nodes in the analysis context object 1709. If neither the software application 1701 nor the analysis process changed or deleted the node, then nothing happened with this node, and the ink analysis tool 1901 adds the node to the list of resultsNodesNotToAdd, since the node exists in the analysis context object 1903 with analysis results and should not be added to the analysis context object 1709.

Then, for each node in the document-independent analysis context object 1903 with the analysis results, if the node is not in the resultsNodesNotToAdd list, then the ink analysis tool 1901 can transfer any changes to the node using the analysis process by creating the corresponding node in the analysis context object 1709. Crawl nodes in the analysis results can be performed using a crawl in the direct order from top to bottom, so that it is guaranteed that the parent nodes are created in front of the child nodes.

When creating a new end node in the analysis context object 1709, any ink strokes associated with the node contained in the analysis results can also be moved to a new node from other nodes in the analysis context object 1709. The stroke map described above can then be used to identify source elements for moving strokes. Ink strokes do not move if the source element does not exist (creating a mandatory collision), or if the source element is on the discretionary collisions list also noted above. By storing optional calls to the analysis context object 1709, this ink analysis tool 1901 can avoid invoking a method to create a child context node if at least one non-conflicting stroke source node does not exist. The same stroke source check is also performed on non-ink end nodes, so that a node for the line is not created until at least one of the words making up the line, for example, is going to contain at least one successfully moved stroke.

Then other information is also applied to the newly created node, such as a recognition grid. The recognition lattice is nevertheless applied, even if certain strokes cannot be moved, creating an applicable (i.e., non-blocking) discretionary collision, which leads to temporary logical incompatibility. However, since unsuccessful strokes are reflected in the designation of the part of the analysis context object 1709 corresponding to the region of the document for which the changes are blocked, the incompatibility is temporary, because the software application 1701 is notified of the reanalysis of the blocked region.

For each node to be removed from the analysis context object 1709, the list of the ink analysis tool 1901 checks whether the node can actually be deleted now that all other edits and additions to the analysis context object 1709 have been completed. The nodes that should be deleted and which are actually free of all child nodes and strokes are removed from the analysis context object 1709. If such a node is deleted, then all the ancestor nodes (up to, but not including, the root node) are deleted sequentially until the ancestor node meets the other remaining child nodes.

Finally, the ink analysis tool 1901 performs a global comparison of the ordering of the nodes between the document-independent analysis context object 1903 with the analysis results and the analysis context object 1709 in order to detect and transmit changes in the analysis process to the analysis context object 1709. It should be noted, however, that this comparison ignores the original version of the analysis context object 1903, which is independent of the document, so it is not possible to detect changes in the ordering made by software application 1701. Then, for each container node in the analysis results, which has a corresponding node in object 1709 analysis context, compares the ordering of child nodes for container nodes. If both container nodes have the same set of child nodes, then the ordering in the analysis results is passed to the analysis context object 1709 by organizing loops on the list of container child nodes in the analysis results until a node in the same position is found in the container list in the object 1709 analysis context that does not have the same GUID (or other type of identifier) as the child node in the analysis results. When such a mismatch is detected, the corresponding node should be found below in the list of child nodes in the analysis context object 1709. The ink analysis tool 1901 then searches for the node in the rest of the list of child nodes in the analysis context object 1709 and calls a method to move the child node to correct the order for that node in the node context object 1709. Then, the ink analysis tool 1901 continues organizing the nodes in the analysis results.

Of course, the assumption of identical lists is not fair, since either the analysis list or the 1701 software application can delete or insert nodes into any list. The "child node list renderer" object is used to simulate a valid assumption by presenting "truncated" lists that contain only elements that are contained together. The procedure described above is then performed on the displayed lists of child nodes, and the movements are converted into real indices and calls by the object - the list display tool.

Thus, matching techniques in accordance with the various examples of the invention described above allow the results of the analysis process to be applied to the current elements of the document in the document, even if the contents of the document have changed since the analysis process was initiated. These matching techniques, therefore, allow the analysis of electronic ink in a document asynchronously with respect to the operation of a software application that is leading with respect to the document. Further, these matching techniques can be advantageously used by a variety of different software applications, including existing multi-threaded software applications with existing unique locking or other synchronization strategies.

Event driven system

As described above, various examples of the invention create a “snapshot” of the state of the document by copying a document-independent analysis context object and then asynchronously analyze the document-independent analysis context object, while the software application that is leading to the document continues to work. Alternatively, various examples of the invention may refuse to use a document-independent analysis context object in favor of asynchronous ink analysis. Instead, these examples of the invention may use an isolated component to store all ink and semantic data for a document. More specifically, these examples of the invention recognize two types of modifications that can be made in a document: ink events, such as adding, deleting, or modifying strokes; and structural events, such as grouping strokes into words, adding semantic nodes, or associating text recognition results with strokes. Each event contains all the data necessary to fully describe the event for an external listener. For example, the “add stroke” event includes all stroke data. With the help of event data with “wide possibilities”, the listener can keep an exact duplicate of the ink object by applying the events in the order in which they were received.

On Fig shows an example of how this scheme can be used, in accordance with various examples of the invention. As can be seen in this figure, application 2801 uses the ink analysis tool 2803. The ink analysis tool 2803 supports ink data 2805 and a document structure 2807 (for example, such as a tree structure) for application 2801. The application also uses an event queue 2809, parsing process 2811, and recognition process 2817. The parsing process 2811 supports the clone 2813 of the ink data 2805 and the clone 2815 of the document structure 2807. Similarly, the recognition process 2817 supports the ink data clone 2819 2819 and the clone 2821 of the document structure 2807.

When an application generates ink data, it provides ink data to the ink analysis tool 2803 by method 2823. In response, the ink analysis tool 2803 generates an event corresponding to a change in ink data, adds a label (tag) to each event, and defines the desired process or processes analysis as components that listen for events with specified labels. For example, all user changes in ink can be marked with a “UserChange” label, and event 2825 with this tag is sent synchronously to the 2809 event queue in response to the change. At some point in the future, the parsing process 2811 and the recognition process 2817 retrieve the tagged event from the event queue 2809.

The first analysis process (eg, parsing process 2811) then listens for all events emitted by the ink analysis tool 2803 and responds to those marked with a UserChange tag. The first analysis process then applies the changes specified by the event to its internal copy of the ink data 2813 and the document structure 2815, analyzes the data described in the event, and then creates and marks the events 2827 generated by their changes with the “Parser1Change” label. These events 2827 are sent by the ink analysis tool 2803 back to the event queue 2809. The second analysis process (eg, handwriting recognition process 2817) then listens for events tagged with Parser1Change. In response, he applies the changes specified by the event to his internal copy of the ink data 2819 and the document structure 2821 and, in response, analyzes the data described in the events. The second analyzer then creates and marks event 2829 with its analysis results with the “HandwritingRecognitionChange” tag. Since the ink analysis tool 2803 receives events from the parsing process 2811 and the recognition process 2817, it sends events 2831 to the application 2801 to indicate a change in the document structure 2807.

These embodiments of the invention can also support each analysis process by listening to events with more than one event, and mark events based on their changes with various labels based on internal processing. Therefore, these alternative embodiments of the invention may also include an analysis of electronic ink in a document that is asynchronous to the operation of the software application that is leading to the document.

Conclusion

Although the invention has been described in relation to specific examples, including currently preferred embodiments of the invention, it is obvious to a person skilled in the art that there are numerous variations and variations of the above systems and methods that are within the spirit and scope of the invention set forth in the appended claims inventions.

Claims (19)

1. A method for analyzing electronic ink, comprising the steps of: receive from a software application running in a first processing stream a first data object corresponding to a first state of a document containing electronic ink content; using a first processing stream to provide a first data object to an electronic ink analysis process for analysis in a second processing stream; accept the results of the analysis process from the second processing stream; after receiving the results of the analysis process, a second data object corresponding to the second state of the document is received from the software application; and directly coordinate the results of the analysis process with the second data object. 2. The method according to claim 1, additionally containing stages in which: accept consistent analysis results from a software application running in a first processing stream; using a first processing stream to provide consistent analysis results to a second electronic ink analysis process for analysis in a third processing stream; accept the results of the second analysis process from the third processing stream; after receiving the results of the second analysis, a third data object corresponding to the third state of the document is received from the software application; and coordinate the results of the second analysis process with the third data object. 3. The method according to claim 2, in which the first analysis process is a process for analyzing the layout and classification of electronic ink and the second analysis process is a recognition process. 4. The method according to claim 2, in which the third processing stream is the same stream as the second processing stream. 5. The method according to claim 2, further comprising stages in which: accept the consistent results of the second analysis from a software application running in the first processing stream; using the first processing stream to provide consistent results of the second analysis to the third electronic ink analysis process for analysis in the fourth processing stream; accept the results of the third analysis process from the fourth processing stream; after receiving the results of the third analysis process, a fourth data object corresponding to the fourth state of the document is received from the software application; and coordinate the results of the third analysis process with the fourth data object. 6. The method according to claim 2, in which the second analysis process is a recognition process with a first electronic ink data recognition stage for the first language and a second electronic ink data recognition stage for the second language. 7. The method of claim 1, wherein the document includes electronic ink content and non-ink document content. 8. The method according to claim 7, in which at least part of the content of electronic ink annotates non-ink content. 9. The method according to claim 1, additionally containing stages in which: creating a data structure based on a first document data object received from a software application; and provide this data structure to the first analysis process. 10. The method according to claim 9, in which upon agreement provide the data structure to the software application. 11. The method according to claim 1, in which, upon agreement, it is determined based on the first data object and the second data object that the state of the document has changed during the analysis performed by the electronic ink analysis process. 12. The method according to claim 1, in which the first processing stream and the second processing stream operate asynchronously, so that the software application continues to work when the electronic ink analysis process analyzes the document. 13. The method according to claim 1, in which, upon agreement, the second data object is updated based on the results of the analysis process, and the updated second data object is returned to the software application. 14. The method according to claim 1, in which, upon agreement, the second data object is updated to associate annotations in the form of electronic ink with one or more non-ink elements of the document. 15. Machine-readable medium, including computer-executable instructions stored on it, to perform the method according to claim 1. 16. A computer-readable medium storing computer-executable components, which, when executed in a computer system, perform an electronic ink analysis method, said computer-executable components including: a software application that supports a document containing document data corresponding to electronic ink data, the software application being executed in a first processing stream; ink analysis process for electronic ink analysis; wherein the ink analysis process is carried out in a second processing stream; and an ink analysis tool that receives document data containing electronic ink data from a software application, provides document data to the electronic ink analysis process for analysis, receives results from the ink analysis process and returns the results from the ink analysis process, which are subject to direct approval by the software application, wherein the first processing stream and the second processing stream operate asynchronously, so that the software application continues to work when the ink analysis process analyzes document data corresponding to electronic ink data. 17. The machine-readable medium of claim 16, wherein the ink analysis tool receives updated document data from the software application after receiving results from the ink analysis process and matches the results from the ink analysis process with updated document data received from the software application. 18. The computer-readable medium of claim 16, wherein the computer-executable components further include: a second analysis process that analyzes electronic ink; wherein the ink analysis tool provides results from the ink analysis process to the second analysis process for analysis; takes the results from the second analysis process and returns the results from the second analysis process programmatically application. 19. The computer readable medium of claim 18, wherein ink analysis tool receives updated document data from a software application after receiving the results from the second analysis process and aligns the results from the second analysis process with updated document data received from a software application.

Images