JP2017514225A - Smart optical input / output (I / O) extension for context-sensitive workflows - Google Patents

Abstract

Disclosed are systems, methods and computer program products for smart and automatic capture of text information using an optical sensor of a mobile device. The text information is provided to the mobile application or workflow without the need for user intervention such as a copy / paste operation and without the user having to manually enter or transfer data. Capture and delivery is context-aware and can normalize or confirm captured text information prior to entry in a workflow or mobile application. Other information required by the workflow and available to the mobile device light sensor can also be captured and provided in a single automated process. As a result, the overall process of capturing information from optical input using a mobile device is significantly simplified and improved in terms of data transfer / input accuracy, workflow speed and efficiency, and user experience.

Description

RELATED APPLICATION This application claims US application Ser. No. 14 / 686,644, filed Apr. 14, 2015, which claims the benefit of US Provisional Patent Application No. 61 / 979,949, filed Apr. 15, 2014. Claims the advantages of the issue, which is incorporated herein by reference in its entirety.

The present disclosure relates to input / output (I / O) using optical components of mobile devices. More specifically, the concept relates to integrating the light input function of a mobile device with the output function of the mobile device, and even more specifically, the mobile workflow or application of light input from the mobile device camera. Performing context-sensitive integration into text output for

Background Mobile devices occupy an increasingly prominent gap market in an ever-growing market, and function as access points at various stages of performing countless activities on the surface. As this trend continues, mobile devices and the mobile network capabilities provided by them are being utilized in an increasing number of broader scenarios. Recent examples include the expansion of mobile technology to provide numerous financial services such as check deposit, bill bill payment, account management and the like. In addition, location data collected by mobile devices is being used in an increasing number of applications, for example, to provide targeted advertising, situational awareness, and the like.

  When the mobile development community discovers a new utility for a device, to provide input that is required by or advantageous to the underlying process that the mobile device is being utilized for execution In addition, the user is given more complex and specific opportunities. In addition, the context of situations in which a user may interact with or provide input to the process continues to diversify.

  This diversification inherently includes an expansion into the gap market where the implemented technology may not always be optimal or acceptable from the user's point of view. In a culture that instantly determines whether a solution to a given challenge is acceptable or unacceptable, developers seek all possible performance benefits to achieve superior technology doing.

  For example, some inefficiencies are known for user input received via a mobile device. The first inefficiency is the small screen size typical of mobile devices, especially mobile phones. A traditional “smartphone” eliminates the physical keyboard and pointer device and instead relies on touch screen technology, so given a virtual “keyboard” displayed on the mobile device screen The amount of physical space allocated to the keys is much less than the range that can be activated accurately and reliably with a human finger. As a result, typographical errors frequently occur in light of textual user input received via the mobile device.

  To eliminate such limitations, typical mobile devices employ powerful predictive analysis and dictionaries to “learn” the input behavior of a given user. If the user's actual input corresponds to text that does not conform to the specified quota, pattern, etc., the mobile device may predict the user's intended input text based on the developed prediction model. it can. The most prominent example of utilizing such predictive analysis and dictionaries is embodied in the traditional “autocorrect” feature available on most typical mobile devices.

  However, these “autocorrect” approaches generate inaccurate (or inadequate) predictions and are therefore not well received in the mobile community. In some contexts, these inaccuracies can be humorous, but widespread mispredictions can result in miscommunication and errors, frustrating the underlying processes and users, and ultimately Makes it impossible to employ mobile devices in a wide variety of contexts where they can be leveraged to gain significant benefits, making their usefulness invalid.

  As a result, some developers have relied on alternative input sources and techniques for aggregating input via mobile devices. For example, most solutions have focused on utilizing speech input as an alternative or supplement to text input (ie, haptic input received by a virtual keyboard shown on a mobile device display). In practice, this technology has traditionally been integrated with the voice recognition capabilities of mobile devices (eg, by “virtual assistants” such as “Siri” installed in APPLE mobile devices (iOS 5.0 and above)). Have been materialized).

  A specific embodiment of this voice input extension added to the mobile keyboard is shown in the accompanying drawings. Although this figure shows the interface generated using APPLE's iOS Mobile operating system, similar functionality is found on other platforms such as ANDROID®, MICROSOFT SURFACE RT as well. There is a possibility.

  Voice input can be received by integrating an extension into the mobile virtual keyboard so that the user can easily make input other than typical haptic input received via the display of the mobile device. To. In one approach, the audio extension is displayed as a button indicating a microphone icon or symbol immediately adjacent to the space bar (on the left). A user may interact with a field configured to accept text input (eg, fields in online form, PDF, etc.). The mobile device utilizes an operating system to invoke the user interface of the mobile virtual keyboard in response to detecting user interaction with the field. In addition, the user optionally makes haptic input to enter the desired text or interacts with the voice extension to invoke the voice input interface. In the art, this technique is generally known as a “speech-to-text” function that accepts speech input and converts the received speech input into text information.

  When the voice input interface is invoked, optionally additional input from the user is received via the display of the mobile device (eg, tapping the voice extension twice to indicate the start of voice input). In response, the user makes speech input, which is analyzed by the mobile device speech recognition component, converted to text, entered into the field, and the user interacts with it to invoke the mobile virtual keyboard.

  By integrating voice input into the text input / output capabilities of mobile devices, users can enter text information in a hands-free approach, which makes the device's available usefulness in other cases. Extends to the entire host of unrealizable contexts. For example, a user may generate a text message using only voice input according to these approaches. However, these approaches are also plagued by inaccuracies and inconsistencies that similarly cause frustration and degrade performance, which are well known in existing speech recognition technology. As a result, current speech recognition approaches that supplement or replace text input are insufficient.

  It is well known that currently available speech recognition is disturbed, and often speech recognition software may simply not be able to recognize the unique utterances played by a particular individual. Similarly, speech recognition is prone to “audiographical” errors (ie, errors similar to “wrong” errors in speech input such as “recognizing” a spoken word incorrectly). .

  Furthermore, speech recognition is inherently limited by the existence of a predetermined set of rules (eg, a set of assumptions or conditions that can be defined based on the language being spoken). In addition, even in the same language, usage is often significantly different between spoken and written sentences, so it may not be possible to use speech input as a supplement or alternative to text input. unknown. For example, speech input is haptic in environments where the expected representation and / or form of use (which often prescribes the “rules” on which speech recognition relies) corresponds to the written form of the language. Often this is an infeasible alternative to input.

  Speech recognition is also a subordinate tool that should be utilized to obtain or confirm user input corresponding to information that is not common or cannot be verbalized. A basic example of these restrictions can be demonstrated in terms of user input including symbols that are often used to label units. Even if these units are generally accepted pronunciations (eg, a passing unit known as “US dollar” corresponds to the symbol “$”), these pronunciations are not necessarily the corresponding terms. (For example, “pound” may correspond to either a unit of weight measurement, ie “lbs.” Or a currency unit, eg “£”, depending on the context).

  Speech recognition also includes text input that includes grammatical symbols (eg, one or more “symbols” used to convey grammatical information (eg, comma “,” semicolon “;”, period “.”, Etc.)); or Suitable for accepting and processing format inputs that contain symbols (eg, carriage returns, tabs, spaces, specific string alignments, etc.) that do not necessarily have a corresponding physical representation in the expressed language Not.

  Other existing approaches include the use of light input as a supplement to text input, but these techniques merely provide the ability to combine text input with images or video clips, and favor user communication. The format (ie, SMS text message, email, video chat, etc.) distributes this combined input. These conventional approaches typically include a combined input interface, which includes tactile input via the mobile device's virtual keyboard and light input via a separate button located on the input interface. (But not necessarily included in the virtual keyboard as in the case of the voice input function described above).

  When the user interacts with this separate button, the device can easily include a pre-captured light input, or alternatively can easily call the capture interface to capture a new light input It becomes easier for the user to include pre-captured or newly captured light input in addition to text information input by providing tactile input to the mobile virtual keyboard.

  As a result of the above, the integration of existing optical and voice inputs via mobile devices is strictly limited as a supplemental or alternative approach to receiving and processing user inputs via mobile devices. End up. With the existing strategy, there is a possibility that the input of speech for speech recognition becomes complicated or the input of an image for supplementing the text input becomes complicated. However, these technologies cannot integrate these various input capabilities in a context sensitive manner that provides an intelligent alternative and / or supplement to text input by mobile devices.

  Ensuring additional input capabilities is implemented in a productive manner that supports rather than degrades device performance, and user interaction with the device is a complex task. This necessitates careful consideration of the various contexts in which light input may be useful, as well as appropriate conditions for capturing and / or analyzing the light input, which can The context-dependent benefits provided by intelligently integrating the mobile device's camera as a source of input for receiving text information may be realized.

  Thus, user input is received and output (especially output determined based in whole or in part on the received input and the context for which the input was received or the purpose for which the input was provided). It would be very useful to provide new methods, systems and / or computer program product techniques configured to supplement and / or replace haptic and auditory inputs as mechanisms.

FIG. 3 illustrates a mobile device user interface configured to receive user input according to one embodiment. FIG. 3 illustrates a mobile device user interface configured to receive user input according to one embodiment. 2 is a flowchart of a method according to one embodiment. 2 is a flowchart of a method according to one embodiment.

SUMMARY OF THE INVENTION In one embodiment, a method calls a user input interface on a mobile device; calls a light input extension of the user input interface; and transmits light via one or more light sensors of the mobile device. Capturing input, determining text information from the captured light input, and providing the determined text information to a user input interface.

  In another embodiment, a method receives light input via one or more light sensors of a mobile device and analyzes the light input using a processor of the mobile device to determine the context of the light input. And automatically calling a contextally appropriate workflow based on the context of the light input.

  In yet another embodiment, a computer program product includes a computer readable storage medium having program code embedded therein. The program code causes the processor to call a user input interface on the mobile device, call a light input extension of the user input interface, capture the light input via one or more light sensors of the mobile device, and capture the captured light. It is readable / executable by determining text information from the input and providing the determined text information to a user input interface.

  Other aspects and embodiments of the present invention will become apparent from the following detailed description, taken in conjunction with the accompanying drawings, illustrating by way of example the principles of the invention.

DETAILED DESCRIPTION The following description is made for the purpose of illustrating the general principles of the invention and is not intended to limit the inventive concepts claimed herein. Furthermore, the particular features described in this specification can be used in combination with other possible features described in each of the various possible combinations and alternatives.

  Unless otherwise defined in this specification, all terms are as broad as possible, including those implied by the specification and meanings understood by those of ordinary skill in the art and / or defined in dictionaries, complete text, etc. Interpretation is given.

  Also, as used in the specification and the appended claims, the singular forms “a”, “an” and “the” include plural referents unless the context clearly dictates otherwise. Don't be.

  This application describes image processing of images (eg, photographs, drawings, graphs, movies, videos, films, single frames of clips, etc.) captured by cameras, particularly mobile device cameras. As understood herein, a mobile device is capable of receiving data without being supplied with power through a physical connection (eg, wire, cord, cable, etc.) and a physical data connection (eg, wire Any device that can receive data without a cord, cable, etc.). Mobile devices within the scope of this disclosure include exemplary devices such as mobile phones, smartphones, tablets, personal digital assistants, iPod®, iPad®, BLACKBERRY® devices, and the like.

  Of course, the various embodiments described herein may be implemented utilizing hardware, software, or any desired combination thereof. In that regard, any type of logic capable of implementing the various functions described herein may be utilized.

  One advantage of using mobile devices is that using data plans, image processing and information processing based on captured images is done in a much more convenient, streamlined and integrated way than previous methods that rely on the presence of a scanner. Be able to. However, the use of mobile devices as document capture and / or processing devices has been previously considered impossible for a variety of reasons.

  In one approach, the image may be captured by a mobile device camera. The term “camera” should be broadly interpreted to include any type of device that can capture images of physical objects external to the device, such as paper. The term “camera” does not include a peripheral scanner or multifunction device. Any type of camera can be used. In a preferred embodiment, a camera with a higher resolution may be used, for example 8MP or higher, ideally 12MP or higher. The image can be captured in color, grayscale, monochrome, or with any other known optical effect. The term “image” referred to herein is intended to include any type of data corresponding to the output of the camera, including raw data, processed data, and the like.

  As described herein, the term “voice recognition” is equivalent to or equivalent to the so-called “speech to text” function (eg, “Siri”) provided on some mobile devices. This feature allows the voice input to be converted to text output. In contrast, the inventive techniques described in this specification are referred to as “image-to-text” or “video-to-text” functions. May be.

  As will be appreciated by one skilled in the art, aspects of the present invention may be embodied as a system, method or computer program product. Accordingly, aspects of the invention may take the form of an entirely hardware embodiment, an entirely software embodiment (including firmware, resident software, microcode, etc.), or software It can take the form of an embodiment combining aspects and hardware aspects. All of these may be generally referred to herein as “logic”, “circuit”, “module”, or “system”. Further, aspects of the invention may take the form of a computer program product embodied in one or more computer readable media incorporating computer readable program code.

  A combination of one or more computer readable media may be used. The computer readable medium may be a computer readable signal medium or a computer readable storage medium. The computer-readable storage medium may be, for example, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, and any suitable combination thereof, It is not limited. A more specific example (non-exhaustive list) of computer readable storage media would include: Portable computer diskette, hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), portable A compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or an appropriate combination thereof. In the context of this document, a computer-readable storage medium is a tangible medium that can contain or store a program for use by or in connection with an instruction execution system, apparatus, processor, or device. There may be.

  A computer readable signal medium may include a propagated data signal incorporating computer readable program code as part of a carrier wave, for example in baseband, and an electrical connection having one or more wires, An optical fiber or the like. Such a propagated signal may take any of a variety of forms including, but not limited to, electromagnetic, light, or a suitable combination thereof. A computer-readable signal medium is not a computer-readable storage medium and is a computer capable of transmitting, propagating, or carrying a program for use by or in connection with an instruction execution system, apparatus, or device Any medium that can be read may be used.

  Program code implemented on a computer readable medium may be transmitted using any suitable medium including, but not limited to, wireless, wireline, fiber optic cable, RF, etc., or any suitable combination thereof.

  Computer program code for performing operations of aspects of the present invention includes object-oriented programming languages such as Java (registered trademark), Smalltalk (registered trademark), C ++, and conventional procedural programming such as "C" programming language. It may be written in a language or a combination of one or more programming languages, including similar programming languages. The program code may be executed entirely on the user's computer, partly on the user's computer, or as a stand-alone software package, partly on the user's computer and partly a remote computer It may be executed on the whole, or it may be executed entirely on a remote computer or server. In the latter scenario, the remote computer may be connected to the user's computer through some type of network, including a local area network (LAN) or a wide area network (WAN), or This connection may be made to an external computer (eg, through the Internet using an Internet server provider).

  Aspects of the present invention are described below with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems) and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions are provided to a general purpose computer, special purpose computer, or processor of another programmable data processing device to form a machine and are executed via the processor of the computer or other programmable data processing device. Instructions may create a means for implementing the specified function / operation in one or more blocks of the flowcharts and / or block diagrams.

  These computer program instructions may also be stored on a computer readable medium capable of directing a computer, other programmable data processing device, or other device to function in a particular manner. The instructions stored on the computer-readable medium may thereby constitute a product that includes instructions that implement the functions / operations specified in one or more blocks of the flowcharts and / or block diagrams. May be.

  Computer program instructions are also loaded into a computer, other programmable data processing device, or other device to cause a series of operational steps to be performed on the computer, other programmable device, or other device. May generate a computer-implemented process whereby instructions to be executed on the computer or other programmable device are specified in one or more blocks of the flowcharts and / or block diagrams. Processes for realizing the functions / operations may be provided.

  The flowchart and block diagrams in the Figures illustrate the architecture, functionality, and operation of possible implementations of systems, methods and computer program products according to various embodiments of the present invention. In this regard, each block in the flowchart or block diagram may represent a module, segment, or portion of code that includes one or more executable instructions for implementing a specified logical function. Note that in some alternative implementations, the functions shown in the blocks may occur in an order different from the order shown in the drawings. For example, depending on the function involved, two blocks shown in succession may actually be executed substantially simultaneously, or they may be executed in reverse order. Also, each block in the block diagram and / or flowchart diagram, and combinations of blocks in the block diagram and / or flowchart diagram, combine specified functions or operations, or dedicated hardware and computer instructions It can be realized by a system based on dedicated hardware that executes things.

  The application may be installed on the mobile device, for example, stored in a non-volatile memory of the device. In one approach, the application includes instructions that cause image processing to be performed on the mobile device. In another approach, the application includes instructions that cause the image to be transmitted to a remote server, such as a network server. In yet another approach, the application may include instructions that determine whether to perform some or all of the processing on the mobile device and / or to send the image to a remote site.

  In one general embodiment, a method calls a user input interface on a mobile device, calls a light input extension of the user input interface, and light via one or more light sensors of the mobile device. Capturing input, determining text information from the captured light input, and providing the determined text information to a user input interface.

  In another schematic embodiment, a method receives light input via one or more light sensors of a mobile device and uses a mobile device processor to determine a light input context. Analyzing the input and automatically invoking a contextually appropriate workflow based on the context of the light input.

  In yet another schematic embodiment, a computer program product includes a computer readable storage medium having program code embedded therein. The program code causes the processor to call a user input interface on the mobile device, call a light input extension of the user input interface, capture the light input via one or more light sensors of the mobile device, and capture the captured light. It is readable / executable by determining text information from the input and providing the determined text information to a user input interface.

  In various embodiments, the methods, systems and / or computer program products disclosed herein are optionally related US Pat. No. 8,855,375 filed Jan. 11, 2013; US Patent No. 13 / 948,046 filed on July 22, 2013; US Patent Publication No. 2014/0270349 filed on March 13, 2013; US filed on March 13, 2014 Patent Publication No. 2014/0270536; U.S. Patent No. 8,885,229 filed May 2, 2014; and / or U.S. Patent Application No. 14/220, filed Mar. 19, 2014. Any of the functions disclosed in 029 may be utilized and / or included. Each of the aforementioned patent applications is incorporated herein by reference. For example, in some specific embodiments, the document from which text information is to be obtained is classified, data extraction is performed on the document, the document or information extracted from the document is verified, and before the capture operation, It may be advantageous to perform additional processing on the image data during or after operation (eg, to improve image quality). This can be understood by one of ordinary skill in the art upon reading this description.

  Digital images suitable for processing according to the algorithms disclosed herein include page detection, rectangularization, non-uniform illumination detection, illumination normalization, resolution estimation, blur detection, classification, data extraction, document Any image processing operation disclosed in the above-mentioned patent application such as confirmation may be performed.

  In a further approach, the methods, systems, and / or computer program products disclosed herein may include, in embodiments, an image processing mobile application, a case management application, a classification application, and / or a data extraction application. It may be utilized with one or more user interfaces configured to facilitate performing any function disclosed in the specification and / or in this above-mentioned related patent application. It may be implemented in one or more user interfaces and / or may include one or more user interfaces.

  In yet other approaches, the systems, methods and / or computer program products disclosed herein are disclosed in the above-mentioned related patent applications and may be recognized by those skilled in the art, especially by reading these descriptions. It may be advantageously applied to one or more of the possible methodologies and / or scenarios.

  It will be further appreciated that the embodiments presented in this specification can be provided in the form of services that are deployed for customers to provide services on demand.

  The inventive concepts disclosed herein relate to integrating optical input into the I / O capabilities of mobile devices in an intelligent manner that facilitates accurate and easy input of text information. Exemplary scenarios where these concepts are most applicable include entering text information into documents, forms, web pages, etc., as can be understood by those skilled in the art by reading this specification. Advantageously, the techniques disclosed herein may have inherent disadvantages of utilizing speech input (eg, less accurate speech recognition) or tactile input (eg, a small “key” size via a virtual mobile keyboard). Input of text information without incurring inaccuracies of input, inadequate “correction” using a predictive dictionary or “autocorrect” function, etc.).

  In particular, this technology provides users with superior performance and convenience. The superior performance is improved accuracy and input time when providing text input via a mobile device (especially when the light input shows suitable information for use in multiple contexts or fields). Includes features such as shortening. In part, the performance advantage is that the inventive approach disclosed herein captures, analyzes, and provides text information from light input without relying on tactile feedback from the user. It is because it is configured to do. As a result, these techniques do not have the disadvantages common to input interfaces that utilize a miniaturized virtual keyboard as described above.

  At the same time, this technology offers better performance than the existing integration of light input to be used in combination with text input. For example, referring to the conventional scenario described above that requires the construction and origination of a message that includes both text input and optical input, this technique advantageously increases the optical input capability of a mobile device to textual I / O. Therefore, it is not necessary for the user to perform tactile input to convey text information.

  Furthermore, the light input can be captured, analyzed and converted to text information in a context-dependent manner. Context-sensitive invocation, capture and analysis for optical input is described in further detail below.

Optical Input Extension for Mobile Virtual Keyboard User Interface (UI) The optical input function disclosed herein captures optical input and haptic input according to the specific mobile operating system in which the function is included. In order to be provided by leveraging unique tools, procedures, calls, components, libraries and so on. In this aspect, the technology represents a seamless integration of light input into the context, typically limited to capturing text information via haptic input or voice input.

  This seamless integration offers advantages over the existing optical and haptic input capture capabilities inherent in mobile operating systems. This is because these existing capabilities are not intended to use light input as an alternative or supplement to tactile input for the purpose of capturing and providing text information.

  Most notably, text information is received, determined and / or utilized via a mobile device, even though traditional mobile operating systems may provide independent optical and haptic input capture capabilities There are currently no known techniques for integrating optical inputs as supplemental and / or alternative techniques for doing so.

  Although rare, some mobile operating systems have also been captured (eg, by optical character recognition (OCR) or other similar functionality as may be recognized by one skilled in the art). The ability to analyze image data and identify, locate, and / or translate text information presented therein may be provided. However, these rare embodiments allow the user to take advantage of the combined ability of capturing and analyzing light input to effectively achieve input of text information by capturing light input. Does not provide any integration of inherent OS capabilities.

  For example, currently known techniques directly capture optical input indicating an identifier that contains desired text information or that includes other information that can be utilized to determine or obtain the desired text information. , It cannot allow the user to enter text information into, for example, a field of the form. “Other” information is any type that can be understood by those skilled in the art as being suitable or useful for obtaining or determining the desired text information by reading this specification. Information may also be included.

  In general, identifiers suitable for extraction in the context of this optical input extension and contextual invocation application are business workflows such as insurance claims or applications; payment account processes such as invoice creation and delivery; navigation processes, communication processes, tracking It may include any type of identifying information (preferably textual information) that may be useful in a process, a financial transaction or workflow such as a tax return or statement survey; a browsing process; an entry or customer boarding process, etc. This can be understood by one of ordinary skill in the art upon reading this description. While a suitable identifier may include any type of identification information suitable for a process such as the exemplary embodiment described above, some type of information may be selected application (eg, accessing or identifying a particular resource). It should be understood that it is particularly useful for unique identifiers that may be required to complete the current workflow.

  Thus, in various embodiments, the extracted identifier is preferably a telephone number, a complete or partial address, a universal resource locator (URL), a query number; a vehicle identification Vehicle identification number (VIN), vehicle model / model and / or year, social security number (SSN), (universal product code (UPC)) or stock keeping unit (stock keeping unit: Product name or code) (such as SKU) or other similar textual information typically found on the invoice; insurance group number and / or policy number, insurance provider name, personal name, date (eg , Date of birth or maturity), (preferably handwritten) signature, etc. Including one or more. This can be understood by one of ordinary skill in the art upon reading this description.

  Similarly, “other information” may be obtained or determined using any suitable technique, including known techniques such as verification operations, reverse verification, authentication, and the like. This can be understood by one of ordinary skill in the art upon reading this specification.

  Rather, in order to achieve this result using currently available technology, the user is unique (eg, as described below with respect to the 12-step conventional procedure that requires using the current technology). It would be necessary to perform a series of separate steps by manually calling separate functions of the OS.

  By “extension”, this disclosure refers to functionality included in features that are otherwise existing in the mobile device. Further, referring to an exemplary scenario for receiving voice input with or instead of tactile input, the microphone “button” shown in the above figure is considered a voice extension of a mobile virtual keyboard user interface. Also good. In contrast, a stand-alone application, function or feature that requires an independent call by the user (eg, calling an application, function or feature without interacting with one of the standard user interfaces with a mobile operating system) ) Should not be considered an “extension” of an existing capability.

  In a preferred embodiment, the light input extension is seamless to the user across multiple fields presented via the user interface (eg, web page, application, form, field, etc.) in the middle of capturing light input. Configured to facilitate navigating. In some approaches, this functionality may be embodied as a “next” or “finished” button, gesture, symbol, option, etc. included in the light input capture interface.

  In fact, according to one exemplary scenario, if the user wishes to capture data corresponding to text information intended to be entered into multiple different fields such as forms, web pages, etc. There is also. For example, the user's “focus” is present on the user interface as indicated by clicking, tapping, hovering, selecting, and tabging to interact with the data entry field. A unique user input / virtual keyboard interface including an optical input extension is invoked when it is detected on the data entry field (which may be the first data entry field of a plurality of such data entry fields). Is done.

  The user may interact with the first data entry field to invoke the light input extension, for example, by tapping a “camera” button displayed on the virtual keyboard. In response to invoking the light input extension, the user may be presented with a capture interface that includes a “preview” of the light input being captured (eg, substantially on a camera or other light input device). "Viewfinder" is displayed). Preferably, the “preview” and capture capabilities of the light input extension may be utilized without switching the focus of the mobile device from a browser, application, etc. that displays the data entry field with which the user is interacting.

  In other words, the optical input extension of the virtual keyboard interface described in this specification is preferably a seamless integration of functions so that the user can locate the data entry field and It is possible to call the input extension, capture the optical input via the optical input extension, and populate the data entry field with text information determined from the captured optical input. Preferably, the entire process described above is “seamless”. That is, the user is configured to "copy and paste" data between independent applications that can run on, for example, the mobile device's multitasking capabilities or on a mobile device This is because since the clipboard is used, all configuration functions that do not need to be used can be terminated. This can be understood by one of ordinary skill in the art upon reading this description.

  In a scenario where the user is interacting with a browser page, application, etc. or includes multiple data entry fields, the user preferably takes advantage of additional functionality provided by the light input extension to You may move between data entry fields. In this aspect, the user may enter text information by selectively utilizing the light input extension to capture light input for a desired subset of the total number of data fields presented. Similarly, the user may sequentially enter text information in some of the plurality of data entry fields using the optical input extension.

  Preferably, the movement of the user between the data entry fields is achieved by a button or gesture in which the light input interface is configured. Exemplary embodiments may employ, for example, “next” and / or “previous” buttons, or translate one or more swipes or multi-touch gestures. It may be configured to move between multiple data entry fields. Even more preferably, the optical input interface also includes functionality for allowing a user to terminate or indicate completion of the optical input capture process. For example, in some embodiments, the optical input interface may include a “last” button, a “finished” or “done” button, etc., that allows the user to input an optical input. It may be possible to end the capture process and preferably resume interaction with the browser page, application interface, etc.

  Thus, in at least some embodiments, an important feature of the presently disclosed inventive concept is that the optical input capability is directly integrated into an existing interface with a mobile operating system. The light input capability is specifically integrated into a unique virtual keyboard user interface with a mobile operating system as an extension of the virtual keyboard user interface.

  Thus, this technology inefficiently “stitches” with existing capabilities as carried by separate (ie, non-integrated) mobile device camera and virtual keyboard user interface components. It should be distinguished from the approach that may be required. For example, a technology that simply utilizes a combination of haptic input and light input, such as received via a totally separate interface, function, application, etc., does not promote ease and accuracy of input, rather It becomes complicated.

  For example, a stand-alone application or function configured to capture and analyze light input to determine the presence of text information (and optionally determine and / or output the indicated text) Such capture and / or analysis of the light input cannot be performed in a dependent manner. For example, a stand-alone application, function, feature, etc. is the context of a particular field or form displayed on a website that is not configured to be set as the first stage, eg, the stand-alone application, feature, feature, etc. Is not configured to provide the desired text information.

  As a result, the user will have to call several independent processes and be involved in the full course of performing the processes that each needs. For example, if the light input and haptic input disclosed herein are not integrated, users attempting a similar process using the prior art can be individually installed, configured, invoked, and manipulated by the user. Would have to be involved in processes that are overly cumbersome and significantly inferior, including separate operations performed using multiple independent processes.

An exemplary conventional process refers to the procedure described above substantially as follows:
(1) Call a mobile web browser application (eg, Safari for iOS);
(2) Use a mobile device web browser to navigate to a web page that requires text information;
(3) close or pause the mobile browser application;
(4) Invoking a separate light input function (eg, a “camera” application);
(5) capture light input containing the desired text information by a separate light input application;
(6) Close or pause the light input application;
(7) Invoking a separate optical analysis application (eg, OCR application);
(8) Analyzing the captured light input and determining the text information shown there using a separate optical analysis application;
(7) specifying the position of desired text information from the determined text information;
(8) Select desired text information from the determined text information (or similarly deselect, delete or discard undesired text information);
(9) Copy the desired text information (eg, to the “clipboard” of the mobile device, or similarly simply through a user storing the desired text information);
(10) close or pause the optical analysis application;
(11) Call or resume a closed / paused web browser application (if the web browser is closed rather than paused, navigate to the web page as in step (2) above) The gate must also be repeated); and
(12) Paste (or replay) the desired text information from step (2) above into the appropriate field of the web page.

  For example, if a particular mobile device does not support the required multitasking capabilities, or if there are insufficient system resources to effectively “switch” between independent applications needed to achieve the desired result, The above scenario involving the use of multiple independent processes may not be feasible.

  For comparison, an exemplary process utilizing the integrated optical and haptic input functionality by the optical input extension to the virtual keyboard user interface is illustrated according to one embodiment in the method 200 of FIG. As such, it will be substantially more efficient (in terms of system resource consumption, as well as in terms of user convenience and time).

  The method 200 may be performed in any suitable environment, including those shown in FIGS. 1A and 1B, as well as any other suitable environment that will be recognized by those skilled in the art reading this description.

  In operation 202, a user input user interface (UI) is invoked on the mobile device.

In operation 204, the optical input extension of the user input UI is invoked.
In operation 206, light input is captured by one or more light sensors of the mobile device.

In act 208, text information is determined from the captured light input.
In operation 210, the determined text information is provided to the user input UI.

  The method 200 may also include any one or more additional or alternative features disclosed herein. In various approaches, the method 200 may additionally and / or alternatively include features such as selective identification, normalization, confirmation, and providing text information from the light input to the user input UI.

  The user input interface is preferably invoked in response to detecting a user interaction with a user interface element configured to receive text information. In such an approach, the method may advantageously include analyzing the light input to determine text information. Accordingly, the analyzing step comprises performing optical character recognition (OCR); identifying desired text information from the determined text information based on the OCR; and the desired text information in the user input interface One or more of the steps of selectively providing to.

  Preferably, the desired text information includes a plurality of identifiers, each identifier corresponding to one of a plurality of user interface elements configured to receive the text information. In some embodiments, some or all of the identifiers include text information required by one of the user interface elements. Thus, it is advantageous to determine which identifiers contain such required text information and selectively provide each corresponding identifier to the appropriate user interface element, preferably in an appropriate format.

  Provide functionality for automatically correcting OCR errors for the entire user experience, for example ensuring correct playback of captured content and proper formatting of information in the manner expected by the workflow It is very beneficial to make sure. Thus, in some approaches, the method can be used to match the identifier format to match one or more of the expected format of the desired text information and the expected range of values for the desired text information. It is advantageous to include one or more of identifying and normalizing at least one of them.

  In various approaches, the step of confirming may include determining one or more of business rules applicable to at least one of the identifiers and reference content from the supplemental document. This determining step is preferably based on an element corresponding to the identifier, and the confirming step is based on one or more of the reference content and business rules. Similarly, normalizing may include determining formatting from supplemental documents, business rules and / or elements invoked by the user.

  In a further embodiment, the method also includes the desired text information to match either or both the expected format of the desired text information and the expected range of values for the desired text information. (I.e. check content and / or format accuracy against reference content, for example) and normalize (i.e. change format or display to match expected format or other business rules, etc.) May include one or more of the steps. This facilitates correcting OCR errors to ensure that the captured content can be accurately played and the information is properly formatted in a manner expected by the workflow. In some embodiments, the verifying and normalizing steps are based on one or more of business rules and reference content from supplemental documents. Thus, the method may also include determining one or more of supplemental documents and business rules based on the elements with which the user interacted.

  In some approaches, the light input extension is presented at the same time as the called user input interface is presented. Preferably, the user input interface includes a virtual keyboard displayed on the mobile device, which includes a camera button displayed on the virtual keyboard.

  The method additionally and / or alternatively includes automatically invoking the optical input capture interface in response to detecting an optical input extension invocation.

  In various embodiments, the method may additionally and / or alternatively include pre-analyzing the light input before capturing the light input. The pre-analyzing step includes detecting an object indicated at the light input; determining one or more characteristics of the object indicated at the light input; and 1 based at least in part on the determined characteristics. Determining one or more analysis parameters; and the like. The one or more analysis parameters preferably include OCR parameters.

  Although the above examples have been described in the context of a user interacting with a mobile browser, those skilled in the art will in principle understand that the inventive concepts disclosed herein can be used in various embodiments in mobile browsers, mobile device operating systems. It will be appreciated that it can be applied to any user interaction with any data entry field, whether presented via system functions, third party applications, native OS applications, etc.

  As demonstrated according to the exemplary scenario described above, the techniques disclosed herein can reduce the number of individual operations required to achieve good results by at least a factor of two. In view of the additional benefits described below with respect to contextual invocation, capture and analysis of optical input, one of ordinary skill in the art can, on the other hand, achieve one or more configuration operations described in this disclosure. On the other hand, using conventional techniques that are totally impossible to carry the performance benefits achieved by integrating these functions into an integrated procedure with superior capabilities and performance characteristics In contrast, one will recognize the significant benefits afforded by the inventive techniques described herein.

Context-sensitive invocation, capture and analysis of light input In a preferred approach, the light input technology of the invention disclosed herein includes optical or text information, data entry operations, forms into which data is to be entered, fields, etc. Context information may be used. This can be understood by one of ordinary skill in the art upon reading this description.

  It may be particularly advantageous to capture certain types of text information using light input as opposed to text input. If the light input is advantageously selected contextually as the preferred form of input data, it may include preferentially capturing the light input whenever conventional tactile or audio input is a problem.

  For example, if the text information does not follow any established convention or rule set (eg, as typically used by a predictive dictionary), tactile input or voice input (eg, a virtual keyboard on a mobile device) Attempting to enter text information by “typing” on the interface, reading or enumerating text information, etc. is error prone. A predictive dictionary or speech recognition function may improperly "correct" or translate input provided by a user who is trying to implement one or more of the inappropriate rules or rules. is there.

  In further embodiments, light input may be preferred when large amounts of text information and / or complex collections of text information are required. For example, a user involved in working through their mobile device wants to complete a form with several fields that require different types of text information, and some or all of the text information Rather than requiring the user to manually enter each of the desired text information, if shown on one or more documents, capture optical input that includes an image of the document showing that text information It may be advantageous to determine or obtain text information by means of

  Similarly, it may be advantageous to analyze the light input according to its context. For example, in one approach, a user may be provided via optical input, utilizing a document as a source of text information. Documents may take any form and have unique characteristics that indicate documents belonging to a predetermined class of documents (eg credit cards, credit reports, driver's licenses, financial statements, tax forms, etc.) It may be presented. This can be understood by one of ordinary skill in the art upon reading this description. In addition, pre-determined analysis parameters, settings, techniques, assumptions, etc. that have been found to result in ideal analysis results for that class of documents, in whole or in part, due to these unique features It may be advantageous to use to analyze the light input indicating documents belonging to a predetermined class.

  For example, for documents that show text information with a specific color profile, especially a background that has a specific color profile if such color profile is a non-standard (eg non-black and white) color profile. It may be advantageous to analyze the light input using a predetermined setting that has been found to yield favorable results.

  Similarly, if a document class is characterized by known dimensions, known orientation, known layout or organization, etc. for text information, it will produce excellent analysis results for its layout, organization, orientation, etc. It may be advantageous to utilize configured analysis parameters, settings, etc.

  In addition, utilizing unique analysis parameters configured to analyze text information represented in a particular font or style can be achieved by, for example, average character width, high for each feasible character. Now, it may be advantageous to use known features of the font, such as the expected size. This can be understood by one of ordinary skill in the art upon reading this disclosure.

  In various embodiments, the predetermined analysis parameters, settings, techniques, etc. employed preferably include one or more OCR parameters, settings, techniques, etc.

  Thus, in some scenarios, it may be further preferred to include a function configured to pre-analyze the light input presented to the mobile device light sensor before capturing the light input. For example, in a preferred embodiment, when a capture interface is invoked, the mobile device includes an object whose light input is identifiable (whether automatic or not in response to user input commanding the capture interface). Whether, or more ideally, the characteristics of the light input may be determined, including but not limited to, including any such detected object identity or classification.

  In a further approach, the light input is a web page, application, form, field with which the user interacts to invoke a user input interface (eg, in various embodiments, a virtual keyboard and / or its light input extension). For example, or may be analyzed based on context information determined based on these. For example, as can be appreciated by those skilled in the art, existing technology, for example, has a limited input interface (eg, a numeric interface for entering a date of birth or social security number; a “name”, etc. The user interface restricts the input that the user can give to the user interface.

  Similarly, the light input extension described herein affects, determines, or determines the analysis parameters used to analyze the light input captured using the extension, or May limit analysis parameters. In an exemplary scenario where the analysis includes optical character recognition, for example, the analysis parameters used for a field that accepts only numbers are OCR alphabets limited to numbers, or conversely, fields that accept only alphabet characters. May contain OCR alphabets limited to letters. In a preferred approach, the optical input extension may define analysis parameters automatically and transparently based on acceptable input types, formats, etc. for a given data entry field. May be executed directly in response to the user receiving an instruction identifying an acceptable type of input for a particular field interacting with the data entry field. For example, illustrate the scenario of a user interacting with a fillable data entry field that expects a phone number as input. According to a conventional example, a user interacting with this data entry field is presented with a keyboard consisting of the numbers 0-9, while according to the inventive concept disclosed herein, it is described in this specification. Users who interact with the same data entry field and use the optical input extension are limited to the numbers 0-9 and may employ analysis parameters including the OCR alphabet.

  In an exemplary scenario according to the above-described embodiments, a user may navigate to a web page, form, mobile application, etc. using a mobile device. The one or more fillable fields with which the user can interact are on the web page, on the navigation bar of the web browser, or on any other element of the medium that the user is interacting with and accepts text information as suitable input. Can also be presented. In response to detecting this interaction and / or in response to input from the user, the mobile device is typically included, for example, in a unique OS function provided in a conventional mobile device. An optical capture interface that substantially represents a “camera” application may be invoked.

  When the light capture interface is invoked, the display of the mobile device displays a “view finder” that shows the field of view of the mobile device light sensor, preferably in real time or near real time. The mobile device is responsive to user input or (preferably) light input received by the mobile device light sensor in an automatic manner that is transparent to the user (eg, light utilized to generate a viewfinder display). Input) may be used to perform a prior analysis as described above.

  In a particularly preferred approach, the prior analysis displays a step of identifying any text information shown in a portion of the optical sensor's field of view (eg, bounding box) and a preview of any text information identified. Steps. Even more preferably, the identified text may be displayed in a data entry field with which the user has interacted to invoke the user input interface and / or its optical input extension.

  In a further approach, the methods, systems, and / or computer program products disclosed herein may include one or more user interfaces (UIs) configured to facilitate receiving user input and generating corresponding output. ), May be implemented in the one or more user interfaces (UI), and / or may include the one or more user interfaces (UI). The user input UI may be in the form of a standard UI included with a mobile device operating system, for example, a keyboard interface such as that used with standard SMS messaging functions and applications, browser applications, etc .; A numeric keypad interface as used with functions and applications; or other standard operating system UIs configured to receive user input, particularly input that includes or corresponds to text information (ie, a screen User input including taps on various locations, or speech that would be converted to textual information).

  For example, as shown in FIG. 1A, the user input UI 100 includes a navigation UI 110, a form or page 120, and a keyboard UI 130. Each UI 110, UI 120, UI 130 may be a standard UI provided by a mobile device operating system, standard browser or mobile application included with the mobile device operating system, or a stand-alone installed separately May be provided by the application. Stand-alone application embodiments are preferred because of their ability to efficiently integrate contextual and capture / extraction functions in a seamless workflow and user experience.

  With continued reference to FIG. 1A, in the context of the present application where the workflow is facilitated by a user input UI 100, the navigation UI 110 includes a mobile browser address bar, forward to assist navigation through various stages such as the workflow. It includes a navigation component 112 such as a button and / or a backward button (not shown). This can be understood by one of ordinary skill in the art upon reading this description.

  The workflow form / page 120 preferably inputs multiple identifiers (optionally normalized and / or verified as described herein) output from the workflow capture and extraction operations. Includes a plurality of fields 122-128 configured to receive as. As shown in FIG. 1A, the fields include a city field 122, a postal code field 124, a telephone number field 126, and a state field 128. Of course, additional fields may be included in the form / page 120 and the user can select the various fields using any suitable technique that would be recognized by one of ordinary skill in the art upon reading this description. May be moved within the form / page 120 for visual display.

  Further, each field may be associated with an expected format and / or value or range of values for text information received as input thereto. For example, the city field 122 begins with an uppercase letter followed by multiple lowercase letters, and optionally includes one or more spaces or hyphen characters, but expects a series of alphabetic characters excluding numbers and other special characters. Can be done. Conversely, the zip code field 124 can be expected to be a series of 5 numbers or 10 characters including numbers and any hyphens or spaces. In the zip code field 124, ten character strings according to a specific format such as “####-####” can be further expected. Similarly, the phone number field 126 can be expected to have seven numbers, and optionally one or more spaces, parentheses, periods, commas and / or hyphens. The phone number field 126 may also be used by a device according to a mask corresponding to one of several standard phone number formats such as “(XXX) ##-####” in the United States. Depending on the region being served, the text information entered therein can be expected according to other corresponding known conventions. In the state field 128, two uppercase strings can be expected. Of course, other fields are similarly associated with the expected format and / or value or range of values according to known conventions, criteria, etc. associated with the information intended to be received as input therein. Also good.

  The user may interact with one of the fields 122-128 using any suitable means, for example by tapping on the area of the mobile device display corresponding to the field, and accordingly Then, the keyboard interface 130 may be called. Alternatively, in the case of a drop-down menu field such as state field 128, the keyboard interface may not be invoked if the field does not accept user-defined text information. For fields that accept user-defined text information, user interaction with the field may be indicated by the presence of the cursor 121. User interaction with specific fields, as described in more detail in this document, applies contextual components of the workflow (eg, applies specific business rules, performs verification, and performs document classification) A component configured in such a manner may be called or scheduled.

  The keyboard interface 130 may be based on the context of the field in which the user was interacting (eg, an expected value or range of values for text information entered into the field) (eg, as shown in FIG. 1A, Select alphabetic character set (responsive to user interaction with city field 122) or number / symbol character set (eg, responding to user interaction with postal code field 124, as shown in FIG. 1B) May be included. Preferably, the keyboard interface 130 includes a plurality of keys 132 configured to facilitate the user “typing” text information into the field, as well as a mobile device such as a microphone and / or camera of the mobile device. The function buttons 134 are configured to perform one or more operations using the I / O components of the.

  When the keyboard interface 130 is invoked, as shown in FIG. 1A, a function button 134 of the keyboard interface 130 (eg, generally associated with a voice capture or speech to text function, as shown in FIG. 1B). Button) may interact with the user to invoke the light input extension of the mobile application or workflow. In practice, the optical input extension calls the capture interface and initiates capture and extraction operations (optionally including confirmation, classification, etc.) as described in more detail below.

  Additionally and / or alternatively, the light input extension is displayed separately from the keyboard interface 130 as a separate button 136 within the form / page 120, for example, as schematically illustrated in FIG. 1B. May be.

  In one approach, an image of a document may be captured or received by a mobile device, and image processing operations such as optical character recognition (OCR) may be performed on the image. In a further approach, the user can move the mobile device over the document so that the identifier is extracted directly from the feed video via OCR without having to call the capture operation separately. Based in whole or in part on the OCR results, an identifier, preferably a unique identifier, can be extracted from the image.

The normalized, confirmed and extracted identifier may be compared to the reference content or analyzed in view of one or more business rules. Reference content and / or business rules are preferably stored locally on the mobile device to facilitate efficient comparison and / or analysis and may be provided in any suitable form.

  In a myriad of approaches, the reference content may incorporate a supplemental document form for the document from which the identifier is to be extracted. Supplementary documents may include textual information documents, files, or any other suitable source that are matched when a simple comparison of extracted identifiers can be performed. For example, in a preferred approach, the mobile application includes data in which one or more supplementary documents are stored, each supplementary document corresponding to at least one identifier or one of the mobile applications. It corresponds to the identifier of the type used in the above workflow.

  The supplemental document may include an identifier that may be obtained and stored in a data store based on prior capture and extraction operations, for example, using a mobile application. Advantageously, the supplemental document may include a processed image for the document indicating the identifier, and the above processing may be carried out according to a specialized second (for example, based on color profile, correction of projection effects, orientation correction, etc. It may be configured to improve the quality of the image for data extraction purposes (by valuation). The document image may serve as a verification tool to ensure the accuracy of identifiers extracted from the imaged document during subsequent calls of the mobile application or its particular workflow. Of course, a similar function may be achieved if the supplemental document simply includes a confirmed identifier (eg, a series of characters, symbols for an identifier that is known to be accurate).

  In additional and / or alternative embodiments, the business rule may indicate the expected format of the extracted identifier, and further (eg, using OCR parameters based on a specific color profile of the document) May include rules on how to selectively extract identifiers (using OCR parameters that limit the position in the document where the identifier is being searched) and / or using, for example, a mask or regular expression Thus, the OCR parameters may be changed, such as by changing the extracted identifier and changing the OCR alphabet to exclude some symbols or character sets to match the expected format. This can be understood by one of ordinary skill in the art upon reading this specification.

  In one approach, business rules may indicate that only the portion of the information that is considered appropriately an identifier within the scope of this disclosure is needed or desired in the context of a particular workflow. There is also sex. For example, the workflow can be just the postal code of the address, only the last four digits of the social security number or credit card number, only the month and year of the date, only the portion of the account on the invoice, eg price or product code Only one or the other may be required. This can be understood by one of ordinary skill in the art upon reading this specification.

  A particular advantage of utilizing business rules that include the inventive concepts disclosed herein is that the specific business rules that apply to a particular extraction operation are in a confidential context, and therefore are subject to extraction attempts. It is possible to automatically determine the appropriate business rules to be applied.

  In some approaches, the extracted identifier may be corrected to account for and automatically correct the cause of the OCR error. For example, preferably the extracted identifier is corrected using textual information from supplemental documents and / or defined business rules.

  Predefined business rules preferably in this context are business-oriented criteria / conditions for processing data, such as setting thresholds for acceptable inconsistencies to which corrections can be applied. May be included. (For example, the correction may be applied to inconsistencies that are smaller than the maximum threshold number of characters, the maximum percentage of characters, etc., and the correction is within a defined set of “acceptable” errors. It may only apply to matching inconsistencies, for example the number “1” instead of the letter “l”, or vice versa, and the dash instead of the hyphen “-”. It may be “-” and may only apply to other similar business-oriented criteria / conditions, which can be understood by one of ordinary skill in the art upon reading this description.

  Additionally and / or alternatively, the extracted identifier may be changed. For example, inconsistencies resulting from OCR errors may be handled automatically using this technique. In one embodiment, according to business rules, the identifier is expected to be in a predetermined format. For example, in the context of a document that requires attention to handling, such as a credit card, the identifier is typically "####-####-##" as found on conventional credit / debit cards. It may be an account number expected in a 16-digit numeric format that substantially conforms to “##-####”, or an expiration date in “MM / YY” format, or the like. This can be understood by one of ordinary skill in the art upon reading this description.

  In a further embodiment, the extracted identifier is. May be extracted accurately, but may still be presented in a different format than expected (e.g. identifiers are expected symbols such as spaces, dashes or unacceptable characters or May include or exclude formatting) (e.g. "Jan" and "January" including alphabetic characters if the expected format is strictly a number such as "01") The month is specified for the date).

  This property mismatch may be resolved automatically by leveraging the data normalization function. For example, in some approaches where the extracted identifier includes a date, January 1, 2001 (01 January, 2001; January 01, 2001), 01/01/01, January 1, 2001 ( There is a finite set of preferred formats in which date data can be represented, such as Jan. 1, 01,). This can be understood by one of ordinary skill in the art upon reading this description. Other types of identifier data may be account numbers (eg, format ####-####-####-####, #### #### ######## , ######## ######## etc., conventional 16-digit account number), cardholder name, (for example, last name, first name; last name, first name, middle name) Middle Initial (MI); first name, last name; first name, MI, last name; etc.), PIN (for example, a 3 or 4 digit number, an alphanumeric string containing both letters and numbers) Etc.) may be expressed in a similar manner. This can be understood by one of ordinary skill in the art upon reading this description.

  Based on business rules that stipulate a finite set of expected or feasible formats for identifier data, the techniques disclosed herein enable data obtained from financial documents, for example, textual information in supplemental documents. Configured to automatically normalize data obtained from an imaged financial document (eg, by extraction) in a manner consistent with the expected format of the corresponding data contained / presented in May be. For example, if the extracted data such as date is determined to be in a specific format (eg Jan. 01, 2001) other than the expected format (eg MM / YY), the extracted data is identified It is advantageous to convert from one format to the expected format, making it possible to easily and accurately match the identifier data derived from the image with the corresponding text information from the supplementary document.

  In other cases, it may be advantageous to utilize an iterative iterative approach to achieve data normalization. For example, in one embodiment, the first iteration is performed substantially as described above. That is, extract an identifier from a document image and compare the extracted identifier with corresponding data from one or more data sources (eg, supplementary documents, database records, text information from predefined business rules, etc.) To do. However, the comparison in the first iteration does not provide any consistency between the extracted identifier and the corresponding data from the data source. In some approaches, this inconsistency is the result of an OCR error rather than a true inconsistency between the identifier on the imaged document and the corresponding data from one or more data sources. There is a possibility.

  This nature of OCR error may be corrected in some approaches by determining one or more characteristics of the data corresponding to the identifier. In one embodiment, the first OCR iteration may extract identifiers in an unacceptable format (eg, the data is not properly normalized) and / or one or more extracted identifiers. OCR may be performed in such a manner as to include OCR errors. As a result, the extracted identifier is one or more, despite the fact that a “true” identifier as shown on the document actually matches at least some of the corresponding data. Does not match any corresponding data in the data source. This false negative result for diversity can be mitigated or avoided by changing the parameters, rules and / or assumptions underlying OCR operation based on the identifier feature.

  For example, in one embodiment, identifiers are extracted and compared with corresponding data from one or more data sources. The character string including the extracted identifier does not match any account number in the corresponding data. In response to not identifying any corresponding data in the data source, the extracted identifier is further analyzed to determine its characteristics.

  In one approach, the extracted identifier is compared to multiple predefined identifier types (eg, “first name”, “last name”, “account number”, “expiration date”, “PIN”, etc.). Thus, it may be determined whether the extracted identifier exhibits any feature corresponding to one of the defined identifier types. For example, the extracted identifier and a defined identifier type may be compared to determine if there is a similarity regarding data formatting and / or data values.

  Exemplary identifier features suitable for such comparisons include, in some approaches, character lengths that can be formed with identifiers such as string length, alphabetic string (ie, “alphabet”, “numeric”, “alphanumeric”) Set), the presence of one or more recognizable patterns common to a particular type of identifier, or other features as may be recognized by one of ordinary skill in the art having read these descriptions. In a preferred approach, the identifier feature can include any pattern that can be recognized using known pattern matching tools (eg, regular expressions).

  Additionally and / or alternatively, the identifier type may be one or more document features, such as a location in the document from which the identifier is extracted; (published on September 18, 2014 as US Patent Publication 2014/0270349) Of the document from which identifiers are extracted (as disclosed in the related US patent application Ser. No. 13 / 802,226 filed Mar. 13, 2013, incorporated herein by reference). The determination may be based in whole or in part on classification and / or characteristics of data located next to, above, below, or in spatial proximity to an identifier on a document. This can be understood by those skilled in the art who have read the description. For example, in a preferred embodiment, the identifier feature is the location from which the identifier located below the data indicating the relevant information is extracted, eg, typically the city, state and It may be determined based on the location where an identifier or the like located below the line of the street address corresponding to the postal code is extracted. In another preferred embodiment, the identifier feature is horizontally adjacent to the associated data, for example, as the expiration date or account number, respectively, is described on typical credit card and debit card documents. It may be determined based on an identifier extracted from the position to be.

  Once the identifier feature is determined, the extracted identifier is used to determine whether any convention or rule describing the identifier feature has been violated (in various approaches, such as OCR errors, inappropriate data normalization or May be analyzed) to determine (which may indicate an extracted identifier, including both). In one example, the extracted identifier does not match any of the corresponding data in one or more data sources based on the initial comparison. In response to not matching, the extracted identifier is analyzed and determined to be of the identifier type “account number” based at least in part on the extracted string length being 16 characters. Is done. The extracted identifier is further analyzed and determined to violate the “account number” feature. As will be apparent from this analysis, the account number string is composed of numeric characters, and the extracted identifier contains non-numeric characters. For example, one character in the extracted identifier string is not the number “8” but the character “B”, not the number “1” but the character “l”, and not the number “0” but the character “O”. This is because it was determined inappropriately. This can be understood by one of ordinary skill in the art upon reading this description.

  The OCR error may be corrected using a second OCR iteration based at least in part on the establishment of the identifier feature. In the above example of account numbers that incorrectly include alphabetic characters rather than numbers, the OCR engine may be limited to alphabetic candidate characters consisting of the entire number digits. The decision to restrict the OCR alphabet is based on predefined business rules that apply to the account number format (ie, the account number consists of number digits). Thus, the second iteration properly recognizes the number “8” in the identifier rather than the letter “B” that was erroneously determined from the first iteration. Preferably, the identifier conforms to at least one business rule as described above. More preferably, the business rules may be expressed as at least one logical expression (eg, rules, formulas, patterns, conventions, structures, organizations, etc., or any number or combination thereof).

  Those skilled in the art will be aware that similar business rules can inform the OCR process regarding how to define an extracted identifier string in a variety of situations different from the number / character distinction illustrated above. You will recognize.

  For example, in one embodiment, a business rule may indicate that a particular alphabet symbol should be used, for example, as opposed to more comprehensive or various alphabet symbols. The business rule is that the account number contains a hyphen character (ie “-”) but excludes a dash character (ie “—”), an underscore character (ie “_”) and an interval character (ie “”). Indicates that you are following a convention. Thus, if the first iteration cannot successfully extract an identifier that matches the corresponding data, for example in the supplemental document, the second iteration will normalize the extraction results according to the predictions reflected in the business rules. May be performed using a more restricted alphabet.

Example context-sensitive workflow use cases For example, a mobile application or a user working within a workflow may interact with fields such as applications, web pages, etc. May be applied to subsequent capture and extraction tasks. For example, a field requesting a zip code (eg, field 124 in FIG. 1A) may indicate or call a business rule that the extracted identifier should have a 5 (or 9) digit format, and all characters Should be a number (or include a hyphen) and alphabetic characters adjacent to a number string of 5 (or 9) should not be included in the extracted identifier. Thus, user interaction with a particular field can then make a context-sensitive determination of the appropriate business rules to be applied in capturing and extracting identifiers from the document.

  In this aspect, the user can selectively capture only the zip code from a document showing all street addresses and populate the zip code field of the corresponding mobile application or workflow, but any instructions in the mobile application or workflow There is no need to provide any text information in the field.

  Similarly, business rules may be based in part or in whole on the context of a document in view of a mobile application or workflow. For example, in a situation similar to that described above, the user may interact with a form or web page field that expects a zip code. However, the form or page also includes other fields that request various information such as telephone number, city and state of the address, name, social security number, expiration date, credit card number and the like. The field that the user interacted with is part of a form / page that requests other information that is likely to be listed on a single document (eg, driver's license, utility bill, credit card, etc.) Because of this, business rules may be invoked so that subsequent capture and extraction operations extract multiple identifiers even though the user may not have interacted with other fields. Trying to populate multiple fields in a form in a single process. More specifically, this constitutes the context of the workflow as in the above example, not the context of the document.

  When the capture interface is invoked to determine the document context, the document in the viewfinder can be analyzed to determine the document type in a single approach. Based on this determination, multiple identifier extraction and field population processes are performed (for example, if the document type is a type of document that is likely to include multiple identifiers corresponding to multiple fields). Or (for example, because the document type typically does not indicate information corresponding to other fields on the form / page, the document is of the appropriate type to attempt multiple extractions. If not) may be avoided.

  In this aspect, the mobile application / workflow context can be exploited, for example, as indicated by user interaction with the field as well as the context of the document from which the identifier is being extracted. Advantageously, this dual context approach allows an optical input based autofill function without depending on any prior data input. Autofill can be performed during the first capture in near real time.

  In a preferred approach, the user may capture images of one or more documents. The image is preferably captured by the capture component of the mobile device (eg, as described above) by invoking the capture interface by an optical I / O extension (eg, the respective extension 134 or 136 of FIGS. 1A and 1B, respectively). "Camera"). The captured image may optionally be stored in a memory, eg, a mobile device memory, to allow for future use and / or reuse, as described herein. In particular, other embodiments of this disclosure also encompass scenarios where the document image is not captured, but in other cases the information shown on or associated with the document (eg, , Corresponding identifiers shown on various documents) are received at a device (preferably a device having a processor such as a mobile device) for later use in extracting and / or verifying.

  The document image is analyzed by performing OCR on it. OCR may be utilized substantially as described above to identify and / or extract characters, particularly text characters, from images. Even more preferably, the extracted characters include an identifier that uniquely identifies the document. The identifier may take any suitable form known in the art, and in some approaches an alphanumeric string of characters (e.g., account number of a document that requires careful handling); (typically credits) 16-digit account number associated with card / debit card account), PIN (CCV code on debit card / credit card, scratch card verification code, personal identification number (PIN), etc.), expiration date (Eg, format “MM / YY”), etc. This can be understood by one of ordinary skill in the art upon reading this description.

  The technology disclosed herein may provide document owners with useful information and / or services regarding their documents by utilizing several advantageous features. For example, optionally considering the context information, such as a mobile application running on a mobile device, the data can be automatically provided to the mobile application without the need for any text information to be entered by the user. Such problems, user errors, predictive dictionary preconceptions, and other problems common to traditional user-based text entry for mobile devices can be avoided.

  For example, in one embodiment, a mobile application, which may be a standard browser that displays a particular web page, stand-alone application, etc., includes a workflow configured to facilitate a user to apply for car insurance. The workflow may include fields requesting information such as the applicant's name, driver's license number, vehicle type, model and / or year, state of residence, etc.

  A keyboard or other user input interface (eg, UI 110, as shown in FIG. 1A) displayed based on the user calling one of the fields of the mobile application and / or via the mobile device. A capture interface such as a viewfinder based on the user calling an optical input extension of UI 120, UI 130) (eg, extension 134 as shown in FIG. 1A or extension 136 as shown in FIG. 1B). Is called on the mobile device.

  The capture interface may immediately guide the user to capture captured images of one or more documents that show some or all of the required information about the workflow field, such as a driver's license and vehicle registration. Preferably, the capture interface automatically detects the document shown in the viewfinder and captures the image when optimal capture conditions (eg, illumination, field of view and zoom / resolution) are achieved. Composed. The viewfinder is arranged in a rectangle so that it is easy to capture the image of the entire document, that is, a rectangular box shape, in order to easily capture the text information lines, fields, etc. shown on the document. A reticle such as one corner may be included. This can be understood by one of ordinary skill in the art upon reading this description. The reticle is preferably configured to assist the user in directing the device and / or document to achieve optimal capture conditions.

  More preferably, the capture operation is recognized by the context to facilitate the accurate and thorough extraction of the identifier from the document and to facilitate the accurate and thorough output of the corresponding text information in the workflow field. The corresponding text information that is being performed may be the same as the extracted identifier in various approaches, or may be normalized according to the expected format and / or exact OCR error. In a further approach, the identifier may be verified against reference content or business rules described in more detail herein to facilitate in-depth and accurate extraction and output.

  In some approaches, the document, as further described in this specification, to determine the context of the document and / or to determine whether a multi-field extraction operation should be attempted, It may be analyzed and classified.

Context-Dependent Process Invocation In further embodiments, it may be advantageous to automatically invoke one or more contextually appropriate processes based on light input received via a mobile device light sensor.

  In general, such a process may be represented graphically as shown in the method 300 of FIG. 3 in accordance with embodiments. Method 300 may be performed in any suitable environment, including those shown in FIGS. 1A and 1B, and in any other suitable environment that can be recognized by one of ordinary skill in the art upon reading this description.

  As shown in FIG. 3, the method 300 includes acts 302-306. In operation 302, light input is received via one or more light sensors of the mobile device, as occurs when the viewfinder interface is invoked and a video feed showing the field of view of the mobile device light sensor is displayed.

  In operation 304, the light input is analyzed using the processor of the mobile device to determine the context of the light input.

  In operation 306, a context-appropriate workflow is invoked based on the context of the light input.

  The context may contain any suitable information related to performing the actions in the corresponding workflow, preferably the type of document represented at the light input; and the document represented at the light input Contains one or more of the content.

  If the context includes a document type, preferably the document type is selected from the group consisting of contracts, sensitive documents, identification documents, insurance documents, certificates, quotations, and vehicle registrations. . If the context includes document content, preferably the content is a phone number, social security number, signature, invoice account, partial or complete address, universal resource locator, insurance group number, credit card number, Selected from query number, photo, and distribution of fields shown on the document.

  In one approach, the user may locate a document that indicates the signature, such as a driver's license, personal check, business check, contract, etc., within the optical sensor of the mobile device. The mobile device is preferably paired with one or more other identifying features of the document (eg, a photo on a driver's license, a specific font such as a magnetic ink character recognition font on a check, a distribution of fields on a form, etc.). The presence of the combined signature may be detected and the appropriate mobile application on the mobile device may be called automatically or semi-automatically. Additionally and / or alternatively, context-sensitive business processes or workflows may be invoked as well within a particular mobile application.

  Various pieces of information that may indicate the appropriate workflow to call are insurance estimates, health care authorization processes, signature rituals, deposits, or any combination thereof. The driver's license number and vehicle identification number may indicate the validity of the car insurance quote. The health insurance provider name, policyholder (patient name) and / or group number may alternatively indicate the adequacy of the health care authorization workflow or health insurance quote workflow. A document that includes text information common to a loan agreement such as a mortgage or loan application along with a signature or signature line may indicate the validity of the signature ritual workflow. A document that includes a signature and an account number or deposit amount may indicate the validity of the deposit workflow. Of course, the inventive concepts disclosed herein may be applied to other workflows as will be understood by those of ordinary skill in the art having read this disclosure without departing from the scope of the description.

  For example, in response to detection of a signature and a photo, the mobile application may call an insurance quote workflow to help the user obtain car insurance. In response to detection of the signature and a particular font, a mobile check deposit workflow may be invoked. A mortgage application process or document signature ritual process may be invoked in response to detection of signature and field distributions. Similarly, if the operation has not yet taken place from within the mobile application, the mobile device may call an application configured to facilitate contextually appropriate operation as described above in various embodiments. Good.

  Other examples of context sensitive process invocations may include any one or more of the following. In response to detecting that the document shown taking into account the optical sensor of the mobile device is an invoice (for example, the term “invoice”, invoice number, known service provider entity name, address, etc. Call system, applications, products (SAP) or other similar enterprise applications (by detecting presence) to automatically display invoice status.

  In response to detecting that the text information shown in view of the light sensor of the mobile device is a phone number, the phone application of the mobile device operating system may be called and the number is automatically entered and / or It may be dialed.

  In response to detecting that the text information shown in view of the mobile device's light sensor is a universal resource locator, the mobile device's web browser application may be invoked and the URL is displayed in the navigation or address bar. It may be entered and / or the browser may be automatically directed to the resource indicated by the URL.

  In response to detecting that the text information shown taking into account the optical sensor of the mobile device is a credit card number, the financial service application or credit card company website (if the website is called up the browser And a bill of accounts, balance, maturity date, etc. may be displayed to the user.

  In response to detecting that the text information shown in view of the optical sensor of the mobile device is a social security number, a tax finalization application or website may be invoked.

  Of course, as those skilled in the art will appreciate upon reading this specification, the inventive concepts disclosed herein provide any suitable scenario, implementation that requires the use of optical input as a source of text information. For example, it may be used for an application. In a particularly preferred approach, the workflow user input UI may be invoked by the context based on the light input in the mobile device view, and any appropriate information in the mobile device view is automatically captured and invoked. When output to the appropriate field of the configured UI, it is properly formatted and any OCR error is already corrected.

  Although several exemplary scenarios have been described above to illustrate the concepts and features of the inventive subject matter disclosed in this specification, those skilled in the art will recognize a number of scenarios where these concepts are similar, You will recognize that it is equally applicable to realizations and practical applications. For example, some examples described in this specification are from the perspective of a user who wants to interact with a web page and enter the text information shown on the document into a fillable field on that web page. Although presented, the subject matter of the invention described above is equally applicable to any similar or equivalent scenario that can be recognized by one of ordinary skill in the art reading these disclosures. For example, the subject matter can be used in any situation where a user enters text information via a virtual keyboard user interface, eg, the user creates an email, the user interacts with an application, etc. It can be applied as well.

  Although this description has been made primarily in connection with methods, those skilled in the art will recognize that the inventive concepts described herein are in a system and / or computer program product or as a system and / or computer program product. Will recognize that they can be equally realized.

  For example, a system within the scope of this description may include a processor and may be logic-in in the processor by causing the processor to perform the method steps described herein, And / or may be executed by the processor.

  Similarly, a computer program product within the scope of this description may be a computer-readable storage medium having program code embedded therein, which program code performs the steps of the methods described herein on a processor. As such, it is readable / executable by the processor.

  While various embodiments have been described above, it should be understood that these have been presented for purposes of illustration only and are not intended to be limiting. For example, any equivalents of the embodiments disclosed in this specification that may be recognized by one of ordinary skill in the art upon reading this disclosure are understood to be included within the scope of the inventive concepts described in this specification. Should be. Similarly, the disclosures of these inventions may be combined in any suitable manner, and their substitutions, synthesis, and modifications will be recognized by those skilled in the art after reading these descriptions.

  Thus, the breadth and scope of embodiments of the present invention should not be limited by any of the above-described specific embodiments, but is defined only in accordance with the appended claims and their equivalents. Should.

Claims (20)

A method,
Invoking a user input interface on the mobile device;
Calling an optical input extension of the user input interface;
Capturing light input via one or more light sensors of the mobile device;
Determining text information from the captured light input;
Providing the determined text information to the user input interface.   The method of claim 1, wherein the user input interface is invoked in response to detecting user interaction with a user interface element configured to receive text information. Analyzing the light input to determine the text information, the analyzing step comprising:
Performing optical character recognition (OCR);
Identifying desired text information among the determined text information based on the OCR;
And selectively providing the desired text information to the user input interface. The desired text information includes a plurality of identifiers, each identifier corresponding to one of a plurality of user interface elements configured to receive the text information;
The method of claim 3, wherein the providing step further comprises selectively providing each of the plurality of identifiers to the corresponding user interface element.   Checking and normalizing at least one of the identifiers to match one or more of an expected format of the desired text information and an expected range of values for the desired text information The method of claim 4, further comprising one or more of the steps of: Determining one or more of business rules applicable to at least one of the identifiers and reference content from supplemental documents;
The step of determining is based on the element corresponding to the identifier;
6. The method of claim 5, wherein the step of verifying and the step of normalizing are based on one or more of the reference content and the business rules.   Confirming and normalizing the desired text information to match one or more of an expected format of the desired text information and an expected range of values for the desired text information; The method of claim 3, further comprising one or more of the steps.   8. The method of claim 7, wherein the step of verifying and the step of normalizing are based on one or more of business rules and reference content from supplemental documents.   The method of claim 8, further comprising determining one or more of the supplemental document and the business rules based on the elements with which the user interacted.   The method of claim 1, wherein the optical input extension is provided simultaneously with providing the called user input interface. The user input interface includes a virtual keyboard displayed on the mobile device;
The method of claim 1, wherein the light input extension includes a camera button displayed on the virtual keyboard.   The method of claim 1, further comprising automatically calling an optical input capture interface in response to detecting a call to the optical input extension. Further comprising pre-analyzing the light input prior to capturing the light input, wherein the pre-analyzing comprises:
Detecting an object indicated in the light input;
Determining one or more characteristics of the object indicated in the light input;
And determining one or more analysis parameters based at least in part on the determined characteristics.   The method of claim 13, wherein the one or more analysis parameters include OCR parameters. A method,
Receiving light input via one or more light sensors of the mobile device;
Analyzing the light input using a processor of the mobile device to determine a context of the light input;
Automatically calling a contextually appropriate workflow based on the context of the light input. The context is
The type of document represented in the optical input;
The method of claim 15, comprising one or more of the content of the document represented at the optical input.   The method of claim 16, wherein the document type is selected from the group consisting of a contract, a sensitive document, an identification document, an insurance document, a title, a quote, and a vehicle registration.   The content is shown on the phone number, social security number, signature, invoice account, partial or complete address, universal resource locator, insurance group number, credit card number, inquiry number, photo, and document The method of claim 16, wherein the method is selected from a distribution of measured fields.   The method of claim 15, wherein the workflow includes one or more of insurance amount estimation, health care authorization process, signature ritual, and deposit. A computer program product comprising a computer readable storage medium having program code embedded therein, wherein the program code is
Call the user input interface on the mobile device,
Calling the optical input extension of the user input interface;
Capturing light input via one or more light sensors of the mobile device;
Determining text information from the captured light input;
A computer program product that is readable / executable by providing the determined text information to the user input interface.

Images