RU2755606C2 - Method and system for classifying data for identifying confidential information in the text - Google Patents

Abstract

FIELD: computer technology.

SUBSTANCE: invention relates to the field of computer technology for identifying confidential information. A computer-implemented method contains stages in which: data presented in a text format is obtained; obtained data is processed using machine learning algorithms, during which each word in a text is assigned with a tag corresponding to a given type of confidential information, wherein a classification matrix is formed for each machine learning algorithm, based on which an F-measure is calculated for each data type; the classification of each word in the text is performed based on texts with tags received from each machine learning algorithm and the F-measure matrix corresponding to machine learning algorithms, and the final version of the text with tags is formed; the classification of the text with tags attached to each word by privacy classes is performed based on the comparison of the combination of available tags in the text with specified tags of confidential information.

EFFECT: increased accuracy of classification of confidential information.

4 cl, 8 dwg

Description

FIELD OF TECHNOLOGY

[0001] The present technical solution, in General, relates to the field of computational data processing, and in particular, to methods of classifying data to identify confidential information.

LEVEL OF TECHNOLOGY

[0002] Currently, the identification of confidential information from a large amount of data and its subsequent classification is a priority task for many industries. The most widespread use of these technologies is observed in the financial sector, where, among large amounts of various data, it is necessary to separately identify and classify confidential information. For this, various tools and technologies are used that allow one way or another to identify confidential information from large amounts of common data. A key feature in the work of such tools is data analysis using machine learning algorithms.

[0003] The data is stored and processed in various automated systems and file resources with different levels of confidentiality, access methods, and attribute composition. Checking for the presence of sensitive data is carried out by various tools. In this regard, it became necessary to create a single technical solution that allows using neural networks to automatically process a large amount of data and identify confidential information. To do this, it is necessary to train artificial intelligence to recognize the content of documents that may contain confidential information. At the moment, prior art solutions are known for storing and classifying data according to user-defined criteria.

[0004] Amazon Marie is a data monitoring service that uses several methods to automatically classify content to identify and prioritize sensitive data and accurately determine the business value of the data. The service recognizes information such as personal information or intellectual property. One of the classification methods is regular expression classification. Object classification using regular expressions is based on specific data or data patterns that Amazon Marie looks for when validating the contents of data objects. Amazon Marie offers a set of managed regular expressions, each with a specific risk level of 1 to 10. Amazon Marie also classifies objects using support vector machines.

[0005] The disadvantages of this solution are: the inability to modify existing or add new regular expressions, the ability to only enable or disable the search for any existing regular expressions, the service identifies only those objects that match the rules. The disadvantages of using regular expressions are that for each type of confidential information, it is necessary to write several regular expressions that do not take into account rare features of the data or can be more general, for example, contain extra data.

[0006] A Google Cloud DLP solution is known to provide fast, scalable classification and editing for sensitive data such as credit card numbers, names, social security numbers, selected international identifiers, phone numbers, and GCP credentials. The DLP cloud classifies this data using over 90 predefined detectors to identify patterns, formats and checksums.

[0007] The disadvantage of this solution is the use of only regular expressions, for each type of confidential information it is necessary to write several regular expressions that do not take into account rare features of the data or may be more general, for example, contain unnecessary data.

ESSENCE OF THE TECHNICAL SOLUTION

[0008] The claimed technical solution proposes a new approach to identifying and classifying confidential information by creating machine learning models for processing large amounts of data.

[0009] The technical problem or technical problem to be solved is the creation of a new method for classifying data with a high degree of accuracy and high speed of recognition of confidential information.

[0010] The main technical result achieved when solving the above technical problem is to improve the accuracy of the classification of confidential information.

[0011] An additional technical result achieved by solving the above technical problem is to increase the speed of classification of confidential information.

[0012] The claimed results are achieved by a computer-implemented method for classifying data for identifying confidential information, performed using at least one processor and containing the steps at which:

• receive data presented in text format;

• processing the received data using machine learning algorithms, during which each word in the text is assigned a tag corresponding to a given type of confidential information, and for each machine learning algorithm a classification matrix is formed, on the basis of which the F-measure is calculated for each type of data;

• perform the classification of each word in the text based on the tagged texts received from each machine learning algorithm and the matrix of F-measures corresponding to the machine learning algorithms and form the final version of the tagged text;

• carry out the classification of the text with tags affixed to each word according to confidentiality classes based on a comparison of the set of available tags in the text with the specified tags of confidential information.

[0013] In one particular embodiment of the method, for each machine learning algorithm, F-measures are calculated for each data type.

[0014] In another particular embodiment of the method, the confidential information is represented at least in the form of text data and / or numerical data.

[0015] Also, these technical results are achieved by implementing a data classification system for identifying confidential information, which contains at least one processor; at least one memory connected to the processor, which contains machine-readable instructions that, when executed by at least one processor, enable the above method to be performed.

DESCRIPTION OF DRAWINGS

[0016] Features and advantages of the present invention will become apparent from the following detailed description of the invention and the accompanying drawings, in which:

[0017] FIG. 1 illustrates a block diagram of the implementation of the claimed method.

[0018] FIG. 2 illustrates an example of retrieved named entities.

[0019] FIG. 3 illustrates an example of a CRF architecture with Bi-LSTM.

[0020] FIG. 4 illustrates an example of tagged data for training models.

[0021] FIG. 5 illustrates the results of training models.

[0022] FIG. 6 illustrates the results of training models.

[0023] FIG. 7 illustrates an example of test results on a test sample.

[0024] FIG. 8 illustrates a general view of the claimed system.

CARRYING OUT THE INVENTION

[0025] In this technical solution can be used for clarity of understanding of the work of terms and abbreviations, which will be deciphered later in this application materials.

[0026] A model in machine learning is a set of artificial intelligence methods, a characteristic feature of which is not a direct solution to a problem, but learning in the process of applying solutions to many similar problems.

[0027] AI (Artificial Intelligence) - artificial intelligence.

[0028] Token - an element of a sequence of letters or words, or a punctuation mark.

[0029] Tag is a value assigned to a token.

[0030] The task of tagging a sequence (sequence labeling problem) is to assign each element of the sequence (token) a corresponding tag.

[0031] A named entity is a word or phrase that designates an object or phenomenon of a specific category.

[0032] Named entity recognition, NER - extraction of named entities - extraction of objects from the text, a set of objects and assigning to these objects a category that determines the meaning of these objects (for example, full name, names of organizations, locations).

[0033] RNN is short for Recurrent neural network, recurrent neural networks.

[0034] LSTM is short for Long ShortTerm Memory, long short term memory is a recurrent neural network architecture.

[0035] Bi-LSTM is short for Bidirectional Long ShortTerm Memory, Bidirectional Long ShortTerm Memory is a recurrent neural network architecture.

[0036] NLP is short for Natural Language Processing, natural language processing.

[0037] CRF stands for Conditional Random Field, conditional random fields.

[0038] Word embeddings - vector representation of words - matching of objects supplied to the model input to vectors.

[0039] The claimed method (100) for classifying data for revealing confidential information, as shown in Fig. 1, consists in performing a series of sequential steps carried out by a processor of a computing device.

[0040] The initial step (101) is to obtain a data array in text format. Text data contains information that can be bank card numbers, SNILS, OKPO, OGRN, TIN, date, passport number, phone number, last name, first name, patronymic, e-mail, address, position, site address, etc., not limited to.

[0041] The next step (102) is the processing of the obtained data using machine learning algorithms, during which, each word in the text is assigned a tag corresponding to a given type of confidential information, and for each neural network, a classification matrix is formed, on the basis of which F- measure for each data type.

[0042] In order for the models to process the data supplied to them as input, it is necessary to present the text in a form understandable to neural networks. To do this, it is necessary to match all the input objects to vectors. In the claimed method, a combination of vector representation of words and symbols is used for this. A combination of methods is introduced to improve the quality of the model. The letters of each word in a sentence are fed into the Bi-LSTM network in order to reveal the characteristics of words at the symbolic level. A separate vector representation of words (token embedding) is created. Then the vector representations of words and symbols are concatenated and then fed into the Bi-LSTM + CRF model. Pre-trained vector word representations are a standard component of a neural network for solving natural language processing problems.

[0043] Training of neural networks occurs on pre-labeled data. Each token in the sequence is assigned a tag (a short string that one-to-one corresponds to the types of confidential information) from a predefined set of tags. Tags are selected so that the user can intuitively understand what this tag means, for example, CARD - card number, NAME - name, etc. Tags are written in Latin, so that they have a common look in all encodings. The types of confidential information fall into one of the categories of legally regulated data, for example, personal data, bank secrets, trade secrets, etc.

[0044] There are several general entity types for the NER task, which are essentially tags. To determine the confidentiality of a document, it is necessary to be able to extract the following entities: name, date, position, postal code, etc., but not limited to. An example of retrieved named entities is shown in FIG. 2.

[0045] The original text is tokenized and tagged. Each token has a separate tag with markup. Tags are separated from tokens by spaces. Sentences are separated by blank lines. A dataset is a text file or a collection of text files. The dataset should be split into three sections: training, test, and validation. Trainers are used to train the network, namely to adjust the weights with a gradient descent. Validation is used to monitor learning progress and stop earlier.

[0046] The method for training neural networks will be disclosed further in this application materials.

[0047] The classification matrix is a standard tool for evaluating statistical models, it displays the probabilities of recognizing the actual value as predicted, for each given predicted option.

[0048] Based on the classification of the test data, F-measures are calculated. The F-score or (F1-score) is a joint score for accuracy and completeness. This metric is calculated using the following formula: F-measure = 2 * Precision * Completeness / (Precision + Completeness). The F-measure is calculated in each algorithm for each kind of data.

[0049] In the next step (103), each word in the text is classified based on the tagged texts received from each neural network and the matrix of F-measures corresponding to the neural networks, and the final version of the tagged text is generated.

[0050] The set of tags is predefined. This set is formed to designate all categories of data that need to be checked for classification as confidential, in accordance with legislative, regulatory, internal or other standards. Data preprocessing (vectorization) is aimed at speeding up and facilitating further processing, and the combination of several neural networks is aimed at increasing the accuracy of tagging. The tag is affixed to which at least one model most likely points to. Also, models should conduct contextual sequential analysis, since addresses, full names and some other types of data, in the general case, consist of several words. To prepare the text for classification, it is tokenized and tagged. Each token has a separate tag with markup. Tags are separated from tokens by spaces. Sentences are separated by blank lines. The resulting dataset is a tagged text file or a set of text files.

[0051] This solution uses a Yargy parser to retrieve named entities. The markup parser is a ready-made mechanism that can extract names, dates, locations, organizations, etc. To improve the work of the parser, an existing library named "Natasha" is used. Some ready-made parsing rules are already available in the Natasha library. For the current solution, the rules for extracting entities are described using context-free grammars and dictionaries based on the requirements set by the normative documents. For example, if there are several levels of information criticality, then it is listed which entities should belong to each of the levels.

[0052] As a result, a breakdown by proposals is made, each proposal is assigned a corresponding number. All available characters are matched against words in sentences. Tags are converted to categorical type.

[0053] The first layer of the model (Embedding layer) turns sequences of numbers (words mapped to numbers) into dense vectors of a fixed size. Next, the TimeDistributed wrapper is used to apply the Embedding layer to each sequence of characters and get a vector representation of the words. Next, the vector representations of words and symbols are concatenated.

[0054] The resulting vector representations are fed into the main layer of the model (Bidirectional). This layer calculates tag probabilities for each word in a sentence. These probabilities are then fed into the CRF layer, which calculates the probability distribution of the transition from one tag to another.

[0055] At step (104), the text is classified according to the privacy class of the tagged text for each word based on the comparison of the set of available tags in the text with the specified tags of confidential information.

[0056] The data prepared in step (103) is used to analyze and assign a classification level to the entire text. For this, prepared tables of combinations of tags are used that determine the total level of confidentiality of the entire text. The level of confidentiality depends on the combination of tags, not just on their presence.

[0057] Example: just the address in the entire text does not represent criticality, and the address mentioning the circulating funds should already be classified with increased severity. Also, the name classifies a whole group of people and cannot be considered critical. But the name with the phone number is already personal data, which should be classified at the appropriate level of severity. If the document contains only the full name, this is one level, but if, in addition to the full name, there is a phone number and date of birth, then the level of confidentiality of the document is much higher.

[0058] For correct classification, it is important to keep track of the context of the tagged word: that is, evaluate the entities on the left and right. Tags receive prefixes: "B-" if this is the first occurrence of a tag of this type, and "Ι-" if it is a continuation. Example: "April [B-Date] 2019 [I-Date] of the year [I-Date]" or "117 [B-Money] million [I-Money]".

[0059] Recurrent neural networks (RNNs) are used to solve various problems, including natural language processing problems due to their ability to use previous information from the sequence to calculate the current output.

[0060] To correctly process the current word in the text (assign a tag), it is necessary that the network builds on the understanding of the previous context. This means that she must remember what the text was to the left of the current word. Traditional neural networks do not have this property; they cannot be trained for long-term dependencies. Recurrent neural networks help solve this problem. They contain feedbacks, thanks to which they can transfer information from one step of the network to another.

[0061] However, despite the possibility of learning long-term dependencies, in practice, RNN models do not work as expected and suffer from a vanishing gradient problem. This problem occurs because backtracking signals quickly become very small (or, conversely, excessively large). In practice, they decrease exponentially with the number of layers in the network. For this reason, a special RNN architecture called Long ShortTerm Memory (LSTM) was developed to cope with the fading gradient. One repeating LSTM network module consists of four layers. LSTM replaces hidden blocks in the RNN architecture with blocks called memory blocks, which contain four components: three kinds of filters (input filter, forget filter, output filter, and memory cell).

[0062] Correct recognition of a named object in a sentence is context dependent. The preceding and following words are relevant for tag prediction. Bidirectional recurrent neural networks have been developed to encode each element into a sequence given the left and right contexts, making them one of the best choices for the NER problem. The bi-directional computation model consists of two steps: the forward layer computes the representation of the left context, the inverse layer computes the representation of the right context. The outputs of these steps are combined to provide a complete representation of an element of the input sequence.

[0063] Conditional Random Field (CRF) is an undirected probabilistic graphical model for structured prediction of conditional probabilities of events corresponding to the vertices of a certain graph, given the observed data. The CRF architecture with Bi-LSTM used to implement the method (100) is shown in FIG. 3.

[0064] In the combined model, vector representations of words (a method for obtaining a vector representation is described in the practical part) are fed to a bi-directional neural network Bi-LSTM. This network calculates tag probabilities for each word in a sentence. Let for the input sequence of words (sentences) X = (x ₀ , x ₁ ,…, x _n ) Ρ - the matrix of probabilities, which is produced by the Bi-LSTM network. This is an n * k matrix, where k is the number of distinct tags and n is the length of the input sequence. Ρ _{i, j} is the probability that the i-th word in the sentence has the tag j. For a sequence of answers y = {y ₀ , y ₁ ,…, y _N } the score is calculated using the following formula:

[0065]

- обозначает вероятность, которая представляет собой оценку перехода от тега i к тегу j, то есть того, что на позиции j=i+1 будет именно тег y_i+1 при условии, что предыдущий тег у_i.[0066] where

- denotes the probability, which is an estimate of the transition from tag i to tag j, that is, that at the position j = i + 1 there will be just tag y _{i + 1} , provided that the previous tag is in _i .

[0067] To improve prediction accuracy, the CRF layer is trained to enforce constraints based on the order of tags. For example, in an IOB schema (I -Inside, O-Other, B-Beginning), the I tag never appears at the beginning of a sentence, or O I IN O is an invalid tag sequence. The complete set of parameters for this model consists of the Bi-LSTM layer parameters (weight matrices, biases, vector representation matrix of words) and the transition matrix of the CRF layer. All of these parameters are tuned during training of the stochastic gradient descent backpropagation algorithm.

[0068] Next, the principle of training neural networks and evaluating the quality of models for the purposes of implementing the claimed method will be presented.

[0069] FIG. 4 shows an example of labeled data for training models.

[0070] To train models, you need to prepare a tagged dataset with text and tags (Fig. 4). A breakdown is made by offers, each offer is assigned a corresponding number.

Example: in the prepared text there are 47959 sentences containing 35179 different words. It turns out 847657 lines in the dataset.

[0071] At the first stage of training, a breakdown is made by sentences, each sentence is assigned a corresponding number. At the next stage, the tokens (each value of the Word column in Fig. 4) are converted to a vector form. For this, dictionaries are introduced for words, for symbols and for tags. Words in sentences are matched against a sequence of numbers, and then pad_sequensec () is applied to the number sequences to convert the sequences to the same size. Further, all available characters are compared with words in sentences, and the tags are converted to a categorical type.

[0072] At the next stage, the sample is divided into training, validation and test samples. Proportions of 80% to 10% to 10% are used.

Example: training 38846 sentences, validation 4317 sentences, test 4796 sentences.

[0073] The next stage of training is the construction of the model. The first layer is the Embedding layer, the task of which is to translate the sequences of numbers (to which the words of the marked-up text were mapped) into dense vectors of a fixed size. Thus, we get a vector representation of words.

[0074] Next, the TimeDistributed wrapper is used to apply the Embedding layer to each character sequence. After the above processing, a vector representation of symbols is obtained. Next, the vector representations of words and symbols are concatenated.

[0075] The next step involves the main layer of the model - Bidirectional. The vector representations obtained at the previous stage are fed to the Bidirectional layer. This layer calculates tag probabilities for each word in a sentence. These probabilities are then fed into the CRF layer, which calculates the probability distribution of the transition from one tag to another. All model parameters (weight matrices, biases, vector word representation matrix, and transition matrix of the CRF layer) are adjusted during training of the backpropagation algorithm with stochastic gradient descent.

[0076] Next, the model is trained / trained.

[0077] FIG. 5, 6 show the results of training the Bi-LSTM + CRF model. It shows an increase in the accuracy of training (accuracy) and an increase in the accuracy on the validation data (validation accuracy) with an increase in the number of epochs, and also shows how the losses decreased.

[0078] FIG. 7 shows the results of testing on a test sample. Matches the text that was marked up by a human with the markup results from the output of the model. The quality of the model is determined by how closely the model predicted the value of the tag, or in other words, how much less deviations in the result of the model's operation from the pre-marked tag values. To assess the quality of a particular model, generally accepted methods and metrics for assessing the quality of models are used.

[0079] FIG. 8 shows an example of a general view of a computing system (300) that implements the claimed method (100) or is part of a computer system, for example, a server, a personal computer, a part of a computing cluster that processes the necessary data to implement the claimed technical solution.

[0080] In the General case, the system (300) contains one or more processors (301) united by a common bus of information exchange, memory means such as RAM (302) and ROM (303), input / output interfaces (304), input devices / output (1105), and a device for networking (306).

[0081] The processor (301) (or multiple processors, multi-core processor, etc.) can be selected from a range of devices currently widely used, for example, such manufacturers as: Intel ™, AMD ™, Apple ™, Samsung Exynos ™, MediaTEK ™, Qualcomm Snapdragon ™, etc. Under the processor or one of the processors used in the system (300), it is also necessary to take into account the graphics processor, for example, NVIDIA GPU or Graphcore, the type of which is also suitable for full or partial execution of the method (100), and can also be used for training and applying machine models. training in various information systems.

[0082] RAM (302) is a random access memory and is intended for storing machine-readable instructions executed by the processor (301) for performing the necessary operations for logical data processing. RAM (302), as a rule, contains executable instructions of the operating system and corresponding software components (applications, software modules, etc.). In this case, the available memory of the graphics card or graphics processor can act as RAM (302).

[0083] ROM (303) is one or more persistent storage devices, for example, hard disk drive (HDD), solid state data storage device (SSD), flash memory (EEPROM, NAND, etc.), optical storage media ( CD-R / RW, DVD-R / RW, BlueRay Disc, MD), etc.

[0084] Various types of I / O interfaces (304) are used to organize the operation of system components (300) and to organize the operation of external connected devices. The choice of the appropriate interfaces depends on the specific version of the computing device, which can be, but are not limited to: PCI, AGP, PS / 2, IrDa, FireWire, LPT, COM, SATA, IDE, Lightning, USB (2.0, 3.0, 3.1, micro, mini, type C), TRS / Audio jack (2.5, 3.5, 6.35), HDMI, DVI, VGA, Display Port, RJ45, RS232, etc.

[0085] To ensure user interaction with the computing system (300), various means (305) I / O information are used, for example, a keyboard, display (monitor), touch display, touch-pad, joystick, mouse manipulator, light pen, stylus, touch panel, trackball, speakers, microphone, augmented reality, optical sensors, tablet, light indicators, projector, camera, biometric identification (retina scanner, fingerprint scanner, voice recognition module), etc.

[0086] The means of networking (306) provides data transmission via an internal or external computer network, for example, Intranet, Internet, LAN, etc. One or more means (306) may be used, but not limited to: Ethernet card, GSM modem, GPRS modem, LTE modem, 5G modem, satellite communication module, NFC module, Bluetooth and / or BLE module, Wi-Fi module, etc.

[0087] The presented application materials disclose preferred examples of the implementation of the technical solution and should not be construed as limiting other, particular examples of its implementation, not going beyond the scope of the claimed legal protection, which are obvious to specialists in the relevant field of technology.

Claims (10)

1. A computer-implemented method for classifying data for identifying confidential information, performed using at least one processor and containing the stages at which: - receive data presented in text format; - carry out the processing of the obtained data using machine learning algorithms, during which a vector representation of words and symbols is created, each token in the text is assigned a tag corresponding to a given type of confidential information, and for each machine learning algorithm a classification matrix is formed, on the basis of which F- measure for each data type; - perform the classification of each word in the text based on the tagged texts received from each machine learning algorithm and the matrix of F-measures corresponding to the machine learning algorithms and form the final version of the tagged text; - carry out the classification of the text with tags affixed to each word according to confidentiality classes based on a comparison of the set of available tags in the text with the specified tags of confidential information. 2. The method according to claim 1, characterized in that for each machine learning algorithm, the F-measure indicators are calculated for each data type. 3. The method according to claim 1, characterized in that the confidential information is presented at least in the form of text data and / or numerical data. 4. A data classification system for identifying confidential information, containing - at least one processor; - at least one memory connected to the processor, which contains machine-readable instructions, which, when executed by at least one processor, ensure the execution of the method according to any one of claims. 1-3.

Images