RU2765237C2 - Method for entering information and device for its implementation - Google Patents

Abstract

FIELD: computer technology.SUBSTANCE: invention relates to computer technology and can be used to enter information. The claimed result is achieved by the fact that the movement of the finger relative to the selected element of the information matrix is identified by its location in relation to other elements of the information matrix located in the field of view of the camera, while the act of interaction of the finger with the element of the information matrix is programmatically calculated relative to the elements of the information matrix visible by the camera, respectively, a command is generated for comparison, while using algorithms for comparison with the standard or by criteria.EFFECT: realization of the purpose of the claimed invention specified by the applicant, which consists in ensuring the input of information.1 cl, 9 dwg, 1 tbl

Description

Technical field

The invention relates to computer technology and can be used to enter information.

Prior Art

One of the main tasks in creating input devices is to improve their ergonomics.

Known designs of ergonomic keyboards, for example, known ergonomic keyboard (US patent 5612691) containing at least two sections with the ability to rotate and tilt relative to each other using a hinge.

Also known ergonomic keyboard (US patent 7338224) containing ergonomic keys, optimally positioned for user input.

Also known is a device for inputting information, made in the form of a bracelet, using wrist movements (application US 20160299570). Wrist gestures are recognized using sensors in the bracelet. The specific gesture can be determined from the database based on the analysis of the sensor signals.

It is also known a device in the form of a three-point belt for fixing the camera (application of Japan 2013-134682t), located on the hand.

There is also known a method and system for a virtual information input device (US application 20130016070), according to which a laser projector installed in the user's glasses projects a virtual keyboard onto the user's hand, and a camera located in the user's glasses captures his gestures. The camera recognizes gestures such as pressing virtual keys and moving your hand.

There is also known a method for entering information and a device for its implementation (RF patent 2141685) in which the finger is brought closer to the tactile element, pressed on it and the finger is returned to its original state, while one tactile element is placed using a base, for example flexible or elastic, on one phalanx of the finger. Then it is fixed in a position that provides accurate identification of the tactile element, which is placed in the access field of the terminal phalanx of the finger of the same hand, two fingers of one hand are brought together and the tactile element of the other is pressed with the terminal phalanx of one finger.

There is also known a method and device for optical input of commands or data (patent EP0554492). The patent describes a user input method that allows analyzing the movements of a person's hand and fingers. The method allows emulating various types of manually controlled input devices (mouse, keyboard, etc.). According to the method, movements of the user's hand and fingers are observed using a system that includes a TV sensor, means for digitizing and processing signals from the specified TV sensor, while the positions and active movements of the user's hand and/or fingers are monitored, which are further analyzed and interpreted as appropriate inputs or commands.

Closest to the claimed technical solution is a mobile device and its control method (US application 20160127624). The device includes an image projector with a virtual user interface, a screen, a camera for capturing an image, and a processor that is configured to detect a specified area of an image captured by the camera. Information is entered using a virtual screen (prototype). The disadvantages of known devices are:

- inconvenience in work (low ergonomics);

- large weight and dimensions;

- low reliability of information processing;

- inconvenience when used for people with physical disabilities.

Disclosure of invention

The aim of the invention is to expand the functionality and areas of application, increase ergonomics, reduce weight and dimensions, as well as the possibility of using for people with physical disabilities. It is also possible to increase the amount of transmitted information per unit of time and increase the versatility of using the device by manufacturing it in the form of a separate removable module.

This goal is achieved by the fact that in the method of inputting information, in which the camera captures images of the movement of the user's fingers relative to the elements of the information matrix, which are subsequently identified and recognized with the issuance of control commands to the terminal device, the camera is placed on the palmar surface of the hand with the possibility of capturing finger movements or fingers relative to each other and/or relative to the elements of the information matrix by the camera and placed on a closed or open base, located on the palmar and back surface of the hand.

This goal is also achieved by the fact that a projector is placed on the palmar surface of the hand, forming a virtual information matrix with elements on any surface, while the camera is placed with the ability to view (fix) the movements of the finger or fingers relative to each other and / or relative to the elements of the virtual information matrices.

This goal is also achieved by the fact that the movement of the finger relative to the selected element of the information matrix is identified by its location relative to other elements of the information matrix in the field of view of the camera. This goal is also achieved by the fact that the information matrix with elements is made taking into account the anatomical characteristics of the user, while varying the size, geometry, location of the elements of the information matrix. This goal is also achieved by the fact that the movements of the finger or fingers relative to the information matrix are interpreted as movements of the computer mouse with pressing the control keys of the computer mouse with the issuance of control commands to the terminal device.

This goal is also achieved by the fact that in the information input device containing a camera placed on the base and associated with the terminal device, the camera on the base is placed on the side of the palmar surface of the hand with the ability to track the movements of the finger or fingers relative to each other and / or relative to the information elements. matrices.

This goal is also achieved by the fact that the camera, battery, transmitter and microcontroller are made in the form of a separate removable module.

This goal is also achieved by the fact that the camera is equipped with an ultra wide-angle lens.

This goal is also achieved by the fact that the base is made in the form of a bracelet or in the form of a watch, and the camera is located on the side of the bracelet or watch, facing the lens towards the palmar surface of the hand, with the possibility of capturing the movements of the fingers relative to each other and / or relative to the elements of the information matrix .

This goal is also achieved by the fact that the base with the camera is placed on the proximal phalanx of the fingers.

Brief description of the figures of the drawings

The method and device are illustrated by drawings, where in Fig. 1 shows a general view of the device, Fig. 2 and 3 show possible layouts of the device on the side of the palmar surface of the hand, in Fig. 4 and 5 show the viewing angles of the camera of the device, FIG. 6 shows a device with a removable module, FIG. 7 shows a device with a projector and a virtual information matrix, FIG. 8 shows a possible block diagram of the device, and FIG. 9 is a processed image captured by a camera located on the palmar surface of the hand. The table shows the possible versions of the device.

The device for inputting information contains a camera 1 (Fig. 1) placed on the base 2 and connected by a communication line with the terminal device (not shown), while the camera 1 on the base 2 is placed on the palmar surface of the hand 3 with the ability to track (monitor) finger movements hand relative to each other and relative to the elements 4 of the information matrix 5. The camera 1 has a viewing angle that allows you to track the movements of the fingers of the hand relative to each other and relative to the elements 4 of the information matrix 5. This is achieved using, for example, an ultra-wide-angle camera or a fisheye camera, or appropriate attachments (lenses) on the camera. The information matrix 5 is located on some surface (not shown), for example, on the surface of a table. The camera 1 can be connected via a communication line with various terminal devices, for example, with mobile devices - a mobile phone, a tablet, also a game console, a laptop, a computer, a remote control, etc. The communication line can be made in the form of a wired or wireless connection. The camera is powered from a battery (accumulator or battery) located on base 2 (not shown in the figure) or via a wired connection, for example, from a USB connection from a terminal device.

The location of the camera 1 and the base 2 allows the camera 1 to capture the interaction and relative position of the finger or fingers, mainly the terminal phalanges of the fingers with the element 4 of the information matrix 5. The base 2 can be made flexible, for example, from plastic or elastic, for example, from latex or leather and can be both a closed shell with fixation and an open shell. Also, the base 2 can be made transparent, for example, from transparent plastic. The device can be placed on one or both hands of the user.

The base can be made in the form of a glove made with or without fingers (not shown) or in the form of a closed or open shell (tape) 6 (Fig. 2), partly located on the side of the palmar surface of the hand 3, and partly on the back side. surface of the hand (not indicated) with a camera 7, which is located on the palmar surface of the hand 3, with a cutout for the thumb. The base 8 can also be made in the form of a closed or open shell located on the wrist on the palmar 3 and back sides of the surface of the hand, while the camera 9 is located and oriented to track (monitor) the movements of the fingers. The base 8 can be made in the form of a bracelet, strap, fitness bracelet, in the form of a watch, etc., and the camera is located on the side of the bracelet or watch, facing the lens towards the fingers, with the ability to track (monitor) the movements of the fingers relative to each other. friend and relative to the elements 4 of the information matrix 5. The camera can be located on a special elevation of the base (not shown) 8 and/or moved forward to the fingers in order to better view the movements of the fingers.

The base can also be made in the form of a closed or open shell 10 located on the thumb with a camera, while the camera 11 is located and oriented to track the movements of the fingers relative to each other and relative to the elements 4 of the information matrix 5.

The base 12 can also be made in the form of a closed or open shell in the form of a fingertip located on the phalanx of a finger, for example, on the proximal phalanx, with a camera 13. There can be more than one base 12 with cameras 13 - to increase the accuracy and reliability of identification of finger movements. The base (Fig. 3) can also be made in the form of a closed or open shell 14 covering the hand from the palmar and back sides, and a camera 15 installed on the side of the palmar surface of the hand 3. The closed shell 14 can be made in the form of a strap fixed with fasteners (not shown). The open sheath may be made of a flexible material such as copper. The base can also be made in the form of a closed or open shell 16 covering several phalanges of the fingers with a camera 17 installed with the possibility of capturing an image on the side of the palmar surface of the hand 3.

The base can also be made in the form of a closed or open shell 18 covering the proximal phalanx of the thumb, covering part of the palm with a camera 19 installed on the side of the palmar surface of the hand 3.

The base can also be made in the form of a closed or open shell 20 covering the middle (not shown) or distal phalanx of the finger with the camera 21. The camera 15 (Fig. 4) on the base 14 is placed on the palmar surface of the hand 3 and has a wide viewing angle A of the lens, which is achieved by using an ultra-wide-angle lens of the camera 15, for example, a fisheye or appropriate attachments on the camera, i.e. the camera has the ability to detect movement on the palmar hemisphere. The viewing angle of the camera 15 may be less than or greater than 180 degrees.

The camera 9 (Fig. 5) on the base 8 is placed on the wrist from the side of the palmar surface of the hand 3 and is made in the form of a bracelet or a wristwatch, and has a wide viewing angle B of the lens, which is achieved using a wide-angle lens of the camera 9 or the corresponding optical attachments on the camera 9 If necessary, the optical attachments on the camera 9 can be made removable and have different viewing angles.

The base 2 can be made in the form of a tape (Fig. 6), at both ends of which there are clamps 22 and 23, made, for example, in the form of a hook-and-loop tape. The latches 22 consist of hooks and the latches 23 of loops. The camera 1 is placed with the ability to view part of the palmar surface of the hand and is connected (not shown) to the battery 24, such as a battery, a microcontroller 25 and a transmitter 26, such as a WiFi transmitter. In this case, all elements - 1, 24, 25 and 26 are located in a separate removable module 27, fastened to the base 2. The removable module 27 is made with the possibility of its fixation with the base 2.

Based on 2 (Fig. 7), a projector 28 can be placed, for example, a laser projector that projects a virtual information matrix 29 with virtual elements 30 onto a surface (not shown), while the camera 1 on the base 2 is located with the possibility of fixing the movement of the fingers and their relative position relative to the information matrix 29 with virtual elements 30. As a surface, for example, the surface of a table, a sheet of paper, the back of a chair, the surface of a door, the back surface of the other hand, etc. can be used. The information matrix 29 with virtual elements 30 can be made in the form of a keyboard, a set of code lock elements, remote control elements, etc.

The block diagram of the proposed technical solution is shown in Fig. 8 and includes an image capture block 31, an image pre-processing block 32, and an information recognition block 33. The solution may include one or more operations, functions, or actions, as shown in one or more blocks 31,32 and 33. Although the blocks are shown in serial order, these blocks may also be executed in parallel and/or in a different order than those described here. In addition, various blocks may be combined into fewer blocks, divided into additional blocks, and/or removed based on the desired device implementation. Communication lines between units can be either wired or wireless, such as Wi-Fi, infrared or Bluetooth.

The block diagram shows the functionality and operation of one of the possible implementations of device variants. In this regard, each block may represent a module, segment, or part of the program code, which includes one or more instructions executed by the processor to implement certain logical functions. The program code may be stored on any type of readable medium, such as a storage device such as a flash drive, hard drive, microcontroller or computer memory.

Embodiments of the invention

The information input method can be implemented as follows (Fig. 1). The information matrix 5, in the form of, for example, a keyboard, is made by graphic printing of elements 4 on a sheet of paper by a laser printer. The location and size of the elements 4 can be made in the form of keys of a standard QWERTY keyboard. Also, the location and size of the elements 4 can be selected taking into account the anatomical features of a particular user. In the future, the information matrix 5 is placed on a flat surface, such as a table surface, and the user fixes the device on the hand.

After receiving a nerve impulse from the brain, the terminal phalanx of the user's finger approaches the selected element 4 of the information matrix 5. Camera 1 located on the base 2 from the side of the palmar surface of the hand 3 captures the movement of the fingers relative to the elements 4 of the information matrix 5, as well as pressing the finger on the a certain element 4 of the information matrix 5 and transmits the video image to the terminal device or to the microcontroller (not shown) via a wired or wireless communication line (not shown). The image captured by the camera 1 can be transmitted as a raw video signal or pre-processed by software in a microcontroller (not indicated) in order to optimize the transfer of digital video files.

If necessary, the image is processed, the information image is recognized and compared with the basic symbols stored in the memory of the terminal device or microcontroller. Since the symbol on the pressed element 4 can be hidden by the finger itself from the camera 1, the act of pressing the element 4 is calculated by software relative to the nearby elements 4 of the information matrix 5 visible by the camera 1. The act of pressing itself is identified by software in the processor of the terminal device or microcontroller and, accordingly, a command is generated for comparisons, while using algorithms for comparing with a standard or by criteria. Further, a control command is generated - for example, dialing a number when dialing a mobile phone number, letters when typing text, or remote control commands. One image or a series of images is analyzed, which makes it possible to consider the dynamics of finger movement, as well as to predict the sequence of finger movement relative to the elements 4 of the information matrix 5. The image is captured by the optical matrix of camera 1 in block 31 (Fig. 8). Before processing the image with recognition algorithms, it may be pre-processed in block 32. For example, image compression and/or processing aimed at improving image quality.

It may include filtering the image from noise, increasing the sharpness and contrast of the image, leveling and converting to the format used by the system (Fig. 9). Subsequently, the image enters the input of the character recognition block 33 (Fig. 8), where the image is segmented. The task of this block is to identify structural units, such as letters and symbols, or track the movements of the user's fingers relative to the information matrix and / or relative to each other. In some cases, complex heuristic algorithms can be used to solve the letter-level segmentation problem.

The information input method can also be implemented as follows. The base 8 is made in the form of a bracelet that covers the wrist (Fig. 2). The camera 9 is fastened to the base 8, while the lens (not shown) of the camera 1 is located on the end part of the bracelet facing the fingers (Fig. 5) with the possibility of review, finger movements relative to each other and/or relative to the elements of the information matrix. The camera lens captures an image with viewing angle B, which includes the movements of the fingers. An information matrix 5 with elements 4 is made using a 3D printer (Fig. 1), i.e. are made volumetric, taking into account the ergonomic features of the user or group of users. The user attaches the device to the wrist. The camera 8 (Fig. 5) captures the movement of the user's fingers relative to each other and relative to the elements 4 of the information matrix 5 and transmits a video image via a wired USB line to a terminal device, such as a tablet. The input signal is processed programmatically and the pressed element 4 of the information matrix 5 is recognized with the subsequent generation of a command to control the terminal device, for example, a tablet cursor or character input.

The information input method can also be implemented as follows. The user puts the device on a finger with a base 2 in the form of a tape and a removable module 27 fastened to the tape (Fig. 6). At the moment of contact of the halves of the tape 2, the latches 22 and 23 - loops are hooked on the hooks, and thus the device is fixed on the finger. Hook-and-loop fastening allows you to repeatedly connect the tape, with virtually no loss of adhesion. The removable module 27 with the camera 1 is placed on the phalanx of the finger (from the side of the fingertip) with the possibility of viewing the palmar surface of the hand and is connected to the microcontroller 25, which is connected to the transmitter 26, such as WiFi. transmitter. The camera 1, the microcontroller 25 and the transmitter 26 are powered by the battery 24, such as a battery. In this case, all elements - 1, 24, 25 and 26 are located in a separate removable module 27. The image captured by the camera 1 is processed by the software processor of the microcontroller 25, where the image (or sequence of images) is processed, recognized and transmitted using the transmitter 26 wirelessly to the terminal device , such as a display or a mobile phone.

The information input method can also be implemented as follows. Based on 2 (Fig. 7), a camera 1 and a projector 28 are placed. The projector 28 projects, for example, using a laser, a virtual information matrix 29 with virtual elements 30 on some surface (not shown), for example, on a table top. The camera 1 is placed on the same base 2 and captures the image of the movement of the fingers, their relative position relative to each other and relative to the virtual elements 30 of the virtual information matrix 29. The user moves the finger relative to the laser pattern of the information matrix 29 with virtual elements 30 and selects one of the virtual elements 30. The camera 1 captures images of the finger as the finger moves to identify the selection of one of the objects of the laser pattern. The image from the camera 1 is transmitted to the microcontroller or to the terminal device, in which it is digitized and processed. Further, the position and active movements of the user's hand and/or fingers are programmatically recognized and analyzed, which are further interpreted to generate appropriate commands on the terminal device.

The information input method can also be implemented as follows (Fig. 1). The information matrix 5 is made in the form of, for example, a grid printed on a printer or a natural, for example, wooden surface with elements 4 is used. The elements 4 can be, for example, a wood texture, or, for example, a texture or graphic images on a fabric. The movements of the finger or fingers of the hand relative to the information matrix 5 captured by the camera 1 are programmatically recognized and interpreted as movements of the computer mouse, they are also recognized as pressing the control keys of the computer mouse with subsequent issuance of control commands to the terminal device. Finger movements parallel to the surface of the information matrix 5 are interpreted, for example, as mouse movement, and movements of the finger or fingers perpendicular to the surface of the information matrix 5 are interpreted as pressing the corresponding mouse button. The very act of pressing is recognized by software in the processor of the terminal device or microcontroller with the generation of a control command.

By "camera" is meant devices in the form of a video camera equipped with lenses, such as ultra-wide-angle lenses or interchangeable optical attachments - interchangeable lenses with different focal lengths. Both one camera and several cameras can be used for more accurate identification of the information matrix elements and the movement of a finger or fingers. By "information matrix" is meant a surface made of artificial or natural materials. On the surface of the information matrix, alphanumeric images of elements can be applied. The surface texture of natural or artificial materials of the information matrix can also be used, which in this case represents a surface, for example the surface of a wooden table or the surface of a fabric. The information matrix can be made of paper, plastic, leather, wood, metal, etc. The "information matrix" also refers to a virtual information matrix and a virtual image created on a surface using a projector. By "information matrix elements" are meant elements with graphic images printed on them in the form of character-alphabetic keys, numeric keys, control keys, function keys, cursor keys, Braille code keys, in the form of a grid, natural surfaces, etc. Also, the surface texture can be considered as elements, which is advisable to use when the device is used as a computer mouse.

By "base" is meant a flexible or elastic base (for example, made of rubber), which, depending on the implementation of the technical solution, has various designs - a fingertip for one phalanx of a finger, a glove, a ribbon, a strap, a bracelet, etc. Depending on the size of the hand, the bases can be individually selected for a particular hand. The base may completely cover the hand or part of the hand, or fingers, or wrist, or be open. The open base can be made of metal in the form of an open annular element, which is installed and fixed by bending it, and removed by unbending it. By "palmar surface of the hand" is meant areas of the surface of the hand such as the fingers, the palm of the hand, or the palm side of the wrist. Similarly for the back of the hand.

By "microcontroller" is meant a computing device that combines the functions of a processor and peripheral devices, contains RAM and/or ROM. By "link" is meant both wired and radio or infrared communications. By "terminal device" is meant, for example, various devices such as mobile phones, tablets, computers, wristwatches, wristbands, remote controls, displays, various types of locks, and the like.

The device can also be used as a mouse or joystick using data on parallel and perpendicular movements of the finger or fingers relative to the information matrix to enter movement information. The device can also be used as a remote control using data on the movement and relative position of the fingers relative to the information matrix or other fingers (parts of the palm) to enter information. The device can also be used to recognize gestures when communicating with deaf people.

Industrial Applicability

When analyzing the invention for compliance with the "novelty" criterion, it was revealed that some of the features of the claimed combination are new, therefore, the invention meets the "novelty" criterion.

When analyzing the invention for compliance with the "inventive step" criterion, it was revealed that the technical solution of the analyzed object is new, therefore, the features correspond to the "inventive step" criterion, since it is a new set of features, as a combination of known features and a new technical property, and also represents new location, number and relationships of elements. In addition, by means of the proposed method and device, a result has been achieved that satisfies long-standing needs (speeding up the input of information, improving the ergonomics of work).

The invention can be used in computing, can be replicated and, therefore, meets the criterion of "industrial applicability". The advantages of the proposed method and information input device:

- small weight and dimensions;

- reduction of tunnel syndrome during work;

- ease of use;

- adaptability;

- the possibility of use for people with physical disabilities;

- ergonomics, the ability to take into account the anatomical features of a particular user;

- versatility achieved by using a removable module,

- solution of the problem of recognition of gestures.

Claims (1)

A method for inputting information, in which the camera captures the movements of the user's fingers relative to the elements of the information matrix, which are subsequently identified and recognized with the issuance of control commands to the terminal device, while the camera is placed using the base on the side of the palmar surface of the hand with the ability to view the movements of the finger or fingers hands relative to each other and the matrix by the camera, characterized in that the movement of the finger relative to the selected element of the information matrix is identified by its location in relation to other elements of the information matrix that are in the field of view of the camera, while the act of interaction of the finger with the element of the information matrix is programmatically calculated relative to the visible the camera elements of the information matrix, respectively, a command is generated for comparison, while using algorithms for comparison with a standard or criteria.

Images