Improving fraud detection with semi-supervised topic modeling and keyword integration

GuidedLDA

GuidedLDA or SeededLDA implements LDA and can be guided by setting some seed words per topic, which will cause topics to converge in that direction (Singh, 2022a). In the study by Andrzejewski, Zhu & Craven (2009), they used words that belong to specific topics and are limited to appearing in some subset of all possible topics. A second model proposed by Andrzejewski & Zhu (2009) uses relationships between words to break up confusing topics. While in Jagarlamudi, Daumé III & Udupa (2012), they propose SeededLDA, an extension of semi-supervised LDA, and use seed words to influence both topic-word distributions and document-topic distributions; it is a model that guides but does not force these topics into seed words. Specifically, the generative process of estimating these distributions is guided by initial word-level information, a set of user-defined words characteristic of the topics in the study corpus. This approach allows the user to provide N sets of representative seed words from the corpus to guide the topic discovery process. These “seed sets” correspond to the word sets preliminarily obtained by the LDA (Zhou et al., 2023) model. To obtain contextually relevant topics, such as the impacts of fraud, strategies, and initiatives for its prevention and mitigation; some initial keywords highlighted by topic must be established, allowing us to obtain topics that help us understand the content of the dataset we are analyzing.

(Anchored) correlation explanation

CorEx is a topic model based on total correlation explanation, which allows identifying topics in a corpus and explaining their structure through the dependency found on the data (Steeg & Galstyan, 2014). In addition, it is a seeded technique with several advantages over the seeded LDA variant (SeededLDA), such as a better consistency of the derived topics and good algorithmic performance (Gallagher et al., 2017). Unlike LDA, CorEx makes no assumptions about the data generation process but instead approaches topic modeling in an information-theoretic way. This model allows the incorporation of domain knowledge through user-specific anchor words that guide the model to topics of interest; this allows the model to represent topics that do not arise naturally and provides the ability to separate keywords that allow topics to be identified differently (Sockin, 2022). Anchored CorEx optimizes the following in Eq. (1) (John et al., 2019): (1)${\displaystyle \underset{X;Y}{\text{Maximize}}TC\left(X;Y\right)+\beta \sum I\left(x;y\right)}$

where X and Y are random variables, TC and I represent the total correlation and mutual information, respectively, and x is an anchor word.

Initial strategy—application of the LDA model

Of the topic modeling algorithms reviewed in Sánchez-Aguayo, Urquiza-Aguiar & Estrada-Jiménez (2022), it was determined that LDA has the best behavior when analyzing data related to fraud since it more consistently groups words by topic. After carrying out different tests in the experimentation, it was validated that the adequate number of topics is four. With this value, the LDA algorithm is applied to the study dataset, obtaining a distribution of words categorized into four topics according to their context and problem of study. The present work relates to the vertices of the FTT, “pressure, opportunity, and rationalization,” and another topic they call others. This distribution of words can be seen in Table 3, which is ordered by topic and prevalence. In addition, the words related to fraud are formatted to identify that they belong to a specific vertex of the fraud triangle. Words unrelated to fraud that belong to said vertices were not formatted. The top 20 terms are manually analyzed and filtered to use only the most significant ones (for each topic).

Table 3 presents inductive labels, presenting the main terms identified by the class and the most significant to be used as seed or anchor words for the semi-supervised models, which are identified by a text format. For example, in Topic 2 (T2), words like “life” or “word” are related to a different approach to fraud, and, therefore, we did not choose them as meaningful representations for one of the vertices of the fraud triangle. As can be seen, the words related to fraud are distributed through the topics without distinguishing groups in the different topics; this indicates that the topics obtained through LDA cannot be directly associated with the vertices of the fraud triangle. However, due to the presence of a high number of words with a high degree of repetition in the different topics, the existence of behavior related to fraud follows.

Proposal—explore topic modeling using semi-supervised learning

Classical topic modeling methods are algorithms that generate various topics from a study dataset. However, due to their unsupervised nature, these methods can impede the comprehension of the analyzed texts. They are prone to create less essential topics, leaving aside several others that may interest them Steeg (2017). Each word is randomly assigned to a topic in LDA, controlled by Dirichlet priorities through the Alpha parameter. Then it is required to find out which term belongs to which topic. LDA uses a straightforward approach to finding the topic for one term at a time. Suppose we want to find the topic of the word “problem” related to fraud. The algorithm distributes each word evenly across all the topics found and assumes it is the right topic for those words. Then, find out what other words the word “problem” is associated with most often. In this context, it is determined what the most common topic among those terms is. Therefore, the word “problem” is assigned to that topic. The word “problem” is close to any topic where words like “debt” and ”need’ are found. These three words are closer to each other before this step. Finally, the model moves on to the next word and repeats the process as many times as necessary to converge. Semi-supervised topic modeling allows the introduction of prior knowledge by incorporating words called “seed” or “anchor” into the algorithm that stimulates or encourages (but does not force) the model to build topics around these anchor words. This alternative of adding keywords gives the flexibility to generate relevant topics while allowing the discovery of unknown topics. GuidedLDA and CorEx use tag seed words to make their training converge around these words. That is, a set of specific words relevant to a tag related to the same topic is used, and the weight of these particular words is increased during training to capture other strongly related words. In other words, these seed words function as anchors.

The coherence score aims to measure the similarity between words and how interpretable the topics obtained by the model are. Starting from the premise that we have the reference coherence score obtained for the LDA model, in which several sensitivity tests were carried out to determine the adequate number of themes, they established C_v as a metric for performance comparison. Since the coherence score gradually increased with the number of topics, the model with the highest C_v was chosen. In this case, K = 4. For the semi-supervised models that we will use in this proposal and considering that we will use the same study corpus, we will use this value of K-topics to perform the respective tests.

As mentioned, semi-supervised topic models require a list or set of keywords called seed or anchor related to each topic for modeling. These words are used to identify specific topics; in this sense, they are related to the three vertices of the FTT for the present work. In these models, a force or push parameter defines the bias of the generated topics toward the seed or anchor keywords. This value can vary between models; for the case of GuidedLDA, it can range between 0 and 1. A 0.1 can bias the seed words by 10% more toward the seed topics. On the other hand, in CorEx, it should always be above 1, and higher values indicate a more substantial bias toward anchor keywords. In this context, the list of anchor keywords for the models was provided, those that were generated in the initial strategy applying LDA, represented in Table 3, and those words with the greatest representativeness related to the three vertices were chosen from the different topics “pressure, opportunity, and rationalization.”

Words are initialized by setting tags as keys and a list of initial words (relative to critical topics) as values.

Table 4 shows the results of applying GuidedLDA and CorEx topic modeling and the top 20 terms identified by topic. It can be observed how the words of the study corpus are distributed in the four established topics. These words are organized by topic and prevalence to identify those that the model considers most relevant. Using the same procedure as Sánchez-Aguayo, Urquiza-Aguiar & Estrada-Jiménez (2022), we format the words to identify their belonging to each vertex of the fraud triangle. Those not related to the vertices were not formatted. We can observe that the model obtained by GuidedLDA has a behavior similar to that of regular LDA, which does not reflect a relationship between the topics obtained with each of the vertices of the fraud triangle since the words within each topic contain words associated with different vertices of the triangle. As in regular LDA, the model does not group words into topics related to each vertex of the fraud triangle. Still, the probability that the corpus documents belong to each topic provided by the model is helpful for feed classification algorithms to detect whether or not a phrase is related to fraud. On the other hand, the results obtained by CorEx are more interpretable than their predecessor since each topic obtained can be linked directly with the knowledge of the domain established in the list of initial words or anchors. A clear relationship can be seen between the resulting topics and the vertices of the fraud triangle; for example, in topic 1 (T1-CorEx), the words (bold) related to pressure are grouped in order of importance; in topic 2 (T2-CorEx), the words (italic) related with opportunity, topic 3 (T3-CorEx) the words (underline) related to rationalization and finally topic four those that are not related to fraud. CorEx allows the obtained model to converge to link the seed or anchor words to a given topic.

keywords = [

[‘economic’, ‘problem’, ‘deadline’, ‘review’, ‘debt’, ‘exploitation’,

‘lose’, ‘job’, ‘scared’],

[‘earning’, ‘insufficient’, ‘inadequate’, ‘evacuation’, ‘supervision’, ‘weakness’,

‘error’, ‘failure’, ‘support’, ‘easily’],

[‘deserve’, ‘abuse’, ‘fair’, ‘temporary’, ‘unethical’, ‘poor’,

‘steal’, ‘care’, ‘fix’, ‘later’],

[‘love’, ‘study’, ‘think’, ‘time’, ‘people’, ‘write’, ‘play’, ‘game’, ‘passion’]

]

Once the models are obtained, the four resulting topics are manually labeled concerning the three vertices of the fraud triangle fraud pressure, opportunity, rationalization, and others. This categorization of topics is essential since it allows for interpreting the corpus and identifying the implicit topics in a dataset. The interpretation of a topic can be achieved by examining a ranked list of terms in each topic (Merino & Atzmueller, 2019).

With the defined models, we can obtain the probability that a particular document in our corpus belongs to a specific topic; by entering the document into the model, it analyzes it and calculates the possibility of belonging to each one of the topics, establishing a percentage of probability per topic. One approach to classifying a document as belonging to a particular topic is to analyze which topic contributed the most to that document and assign it to that topic. Table 5 shows the percentages of belonging to a document associated with each topic. In this case, the one with the highest value corresponds to the most related or dominant for GuidedLDA.

Applying the same procedure to CorEx, it can be seen in Table 5 that the algorithm returns boolean values (True or False) to determine if a topic contributed more to a document and if this document is more related to that topic. In general, this approach could inform about a document belonging to a specific topic without specifying the weights to which each topic contributed to that document.

Given that the metrics corresponding to the probabilities obtained by the models are necessary to feed classification models and their subsequent fraud prediction, it is essential to obtain the required values and be able to process them using machine learning algorithms.

In this context, the operation of the “corextopic.py” module, developed in Python, which contains the functions associated with transforming the data according to the previously defined model, was analyzed. The transform() algorithm ?? takes a matrix X consisting of (n_samples, n_visible), where n_samples is the number of data points, and n_visible is the dimensionality of each data point. The input data samples are preprocessed by applying a normalization or standardization, which is done by calling the preprocessing method; the preprocessed data is then stored in X. The latent_calculation method is then called to calculate the latent variables p(y—x) and the log-likelihood of the data log(z) for the preprocessed data samples X and the model parameters (self.theta). The resulting values are stored in p_y_give_x and log_z, respectively. Finally, the label() method is called to assign a label to each data sample; this algorithm ?? is inside the same “corextopic.py” class, which takes the matrix p_y_given_x of the form [n_samples, n_hidden] that represents the distribution over the hidden variables given the observed variables and returns binary labels for each sample based on the estimate of maximum likelihood. Additionally, it applies a threshold of 0.5 to the probabilities at p_y_given_x. If the probability of the hidden variable is more significant than 0.5, the method assigns it a true label; otherwise, it assigns a false label. The output is a boolean array of the form [n_samples, n_hidden] representing the labels of each sample. The resulting labels are stored in the labels variable.

Because it is required that the estimation of the document-topic distributions be obtained as a return value, it is necessary to update the “transform” method, for which its counterpart of the GuidedLDA algorithm was taken as a reference in the “guidedlda.py” module, this method applies topic modeling using latent Dirichlet assignment (LDA) on a document term matrix X. In this case, the transform() algorithm ?? takes the document term matrix X as a numpy array and the parameters “max_iter” and “tol” to control the convergence of the model. Stores the topic distribution for each document in the corresponding row of the doc_topic array and returns this array containing the probability values corresponding to the topic distribution for each document.

Once the changes have been made, the module is imported again. It generates the results with the probabilities by topic and dominant topic, as seen in Table 6.

In addition to the probabilities obtained, we label the first 7,113 records with 1 to indicate that these documents are fraud-related and the remaining 7,113 with 0 to indicate otherwise. A filter by dominant topic is applied to this new representation of the dataset 6, obtaining four datasets per topic (pressure, opportunity, rationalization, and others) that served as input to train classification algorithms.

Classifier performance

In the present work, the ROC curve was used to represent the performance of different machine learning models when classifying documents as related or unrelated to fraud. Several metrics, including recall, accuracy, and precision, can be used to assess the performance of a classification model. One of the main weaknesses of these metrics is that they are susceptible to changes in class distribution. When the ratio of positive to negative occurrences in a test set changes, a model’s performance may no longer be optimal or acceptable. However, the ROC curve is independent of the class distribution changes (Wu, Flach & Ferri, 2007), so for this type of analysis, it is a frequently used technique (Trifonova, Lokhov & Archakov, 2014; Mallett et al., 2014). The ROC curve will not change even if there is a change in the class distribution of a test set; This is because the ROC curve is based on the underlying class conditional distributions from which the data is drawn. It plots a model’s true positive rate on the y-axis against its false positive rate on the x-axis. It provides a general measure of model performance, regardless of the various thresholds used. The results can be seen in Fig. 2 but are also presented in Table 7.

Based on these findings, random forest and gradient boosting perform the best, with a mean area under the curve (AUC) of 0.87 and 0.88, respectively. These findings imply that our method to identify fraudulent actions based on topic identification using semi-supervised models would be feasible when developing machine learning models.

Comparison of classification algorithms

When comparing the performance of the classification methods, it was observed that RF and GB showed similar performances, with an average AUC of 87% and 88%, respectively, as shown in Table 7. Furthermore, GB exhibited a slight superiority of 1% about RF. These findings align with the results reported in Sánchez-Aguayo, Urquiza-Aguiar & Estrada-Jiménez (2022), where RF and GB were identified as the most efficient classification algorithms, achieving an average AUC of 81%. By using semi-supervised methods for topic modeling, a notable improvement of 7% was observed in the performance of the classification methods to predict behaviors suspected of fraud; this suggests that incorporating the semi-supervised approach improves obtaining document probabilities by topic, increasing the accuracy and efficiency of fraud prediction models that use RF and GB algorithms.

Validation

The validation of a model consists of evaluating its performance using a dataset that has yet to be used during the training process. The main goal of validation is to estimate a model’s performance and get an idea of how well it will work with new data. When building a machine learning model, it is necessary to guarantee its performance through a proper validation process. A standard model validation method uses learning curves and graphs showing the relationship between model performance on training and validation sets as a function of the training data. Observing the relationship between model performance and the amount of training data is possible by analyzing the learning curves. Through cross-validation, it is possible to use k-folds to create a learning curve, train the model on different subsets of data, and evaluate its performance on the validation set. Cross-validation is a technique used to assess the performance of a model by dividing data into k-folds or k-subsets; This allows the model to be trained and evaluated k times, each time on a different subset of data. On the other hand, ROC (receiver operating characteristics) curves provide a way to assess the trade-off between model sensitivity and specificity, so they can help determine the optimal threshold for classification tasks. Together, these metrics provide a comprehensive approach to assess and validate the performance of a machine learning model.

Using multiple datasets to validate a model contributes to a more robust estimate of its performance. In this context, four datasets WebScraping, Students, NN, and ChatGPT, were used to perform the tests. Through the application of learning curves, the classifiers (RF and GB) were trained with the four datasets individually. For their validation, the three remaining sets were concatenated, all for each of the four study topics. In other words, four training-validation rounds were carried out, one per dataset; for example, for the first set of tests, the model was trained with WebScraping and validated with (Students+NN+ChatGPT), for each topic, for the three rounds. The remaining datasets were exchanged until all possible combinations were covered. As can be seen in Fig. 3, a recurring behavior was identified because of applying this technique, observing in the different test rounds that GB has a low bias and acceptable variance in the four topics, which suggests that the model adequately works both in the training set and the test set. Therefore, it can capture the relationship between the characteristics and the objective variable. That is, it does not make assumptions. Furthermore, the model is not sensitive to variations in the training set and can generalize new data well. In the case of RF, it can be mentioned that, in contrast, it has a high bias and a high variance, which means that the model cannot efficiently capture the relationship between the characteristics and the target variable in the dataset and is sensitive to variations in the dataset so it cannot generalize well to new data.

Each of the four datasets was used to train GB once it was verified that its performance was superior to RF. In contrast, the remaining three sets were used to test the model’s performance. This process was repeated using each of the four training datasets and testing the performance with the remaining three until all possible training-test combinations were covered in the four established topics. ROC curves were generated for each training-test combination to assess the model’s performance; This made it possible to compare the classifier’s performance on different datasets and determine which dataset reported superior performance.

To identify the behavior the different combinations provide, we can look at the AUC scores obtained for each topic using GB in the four assessments. The higher the AUC score, the better the performance of the classifier. As can be seen in the Tables 8, 9, 10 and 11, the combinations that present the best performance are those where the dataset used for training is the same used for testing in all evaluations, obtaining consistently high AUC scores in all the topics, values represented by the main diagonal of each matrix, highlighted in bold. In those tests where the datasets with which the model was tested differ from those with which the model was trained, we observed that the metrics obtained fluctuate in the combinations made; some have higher AUC values for specific topics and classifiers, while others have lower scores, this suggests that the performance of the classifiers depends on the dataset used for training and testing. Additionally, the values of the four tests were averaged, as seen in Table 12, observing the same behavior. In addition, we use the cross-validation (CV) technique to contrast the data obtained with the external validation. The dataset was divided into five different “folds,” allowing us to train and test the model iteratively. Each iteration used a different fold as a test set, while the remaining folds were used for training. Once all the iterations were completed, the results were averaged to derive a comprehensive performance measure for the model, as seen in Table 13. In this context, it is possible to affirm that the model is generalizable since it has been externally validated and cross-validated using the study datasets and the best performance classifier, in addition to the scores in a general way in the different phases of training and test, per topic are consistently high.

Discussion

This section compares different topic modeling approaches to capture fraud-related phrases and their computational complexity. The distribution of the main terms and topics obtained from the classic LDA is presented in Table 3, with words related to fraud labeled in a specific format. However, fraud-related words are randomly distributed in the topics without any specific clustering, which prevents tagging the topics with the vertices of the FTT; this suggests that the modeling approach cannot determine the relationship between fraud and the FTT. As a result, it cannot be applied on this initial attempt.

Like its unsupervised LDA predecessor, guided LDA does not show any visible grouping by topic and cannot be associated with the FTT. However, the CorEx algorithm performs highly satisfactorily with grouping words by themes. Table 4 shows how words are arranged by a specific format related to the vertex of a fraud triangle, allowing for labeling based on their theme of “pressure, opportunity, and rationalization”; this allows a connection between the FTT and the results obtained by the CorEx model, suggesting that fraud-related phrases within the same individual and corresponding to topics related to the fraud triangle indicate potential fraudsters requiring further investigation.

The dataset is balanced between fraud and non-fraud classes. It is mentioned that analyzing the results with balanced precision or the area under the curve (AUC-ROC) is preferable when dealing with imbalanced data. CorEx shows higher recall, meaning it finds more true positives but has a lower precision or higher false positive rate. Additionally, CorEx outperforms normal and GuidedLDA in terms of balanced accuracy. The semi-supervised approach is considered an alternative strategy to the classic unsupervised model, as it avoids challenges in determining the nature of topics and their labels. Although the topics identified by CorEx do not cover all fraud theories, they align with factors in the FTT, such as “pressure, opportunity, and rationalization.” This approach, incorporating semi-supervised topic modeling techniques and pre-obtained keywords from LDA, is beneficial for identifying relevant topics.

To analyze the computational complexity of LDA, GuidedLDA, and CorEx approaches in discovering latent themes and structures in data, it is essential to understand the practical implications and considerations that researchers should consider when choosing one of these approaches. In the case of LDA, its complexity is influenced by critical factors such as the number of documents (N), the size of the vocabulary (V), and the number of topics (K), with an approximate complexity of O (I * N * K * V) (Ihou & Bouguila, 2019), where I represents the number of iterations that an algorithm requires to converge or reach a steady state. The Expectation-Maximization (EM) algorithm used in LDA refers to the number of times the expectation and maximization steps are performed to fit the model to the data. The complexity grows as the number of documents, the size of the vocabulary, and the number of topics increase. This behavior can limit the scalability of LDA on massive data sets or in situations where a high level of thematic granularity is sought. On the other hand, GuidedLDA, by incorporating external information to guide topic assignments, can present additional complexity due to the extra computations required to integrate these guides. However, it follows a similar structure to the LDA in terms of complexity. The benefits of the guide can be remarkable, especially in cases where the interpretability and quality of the topics are a priority. However, an increase in training time can accompany this improvement. In contrast, CorEx differs from other techniques by addressing the correlation and dependency between variables, which impacts its complexity depending on the dimensionality and the number of samples. Since CorEx operates differently from the probabilistic approach of LDA and GuidedLDA, its complexity is influenced by the number of samples without a fixed number of topic parameters. In summary, the choice between these approaches must consider not only the quality of the results but also the computational complexity and characteristics of the data in question.