CN110781409A - Article recommendation method based on collaborative filtering - Google Patents

Abstract

The present invention provides an item recommendation method based on collaborative filtering, which relates to the technical field of recommendation systems. The present invention introduces a special dynamic weight to better predict the user u's preference for item i. This dynamic weight uses an attention mechanism to Estimation, the recommendation performance is evaluated by the recall rate and precision rate, which improves the effectiveness and recommendation quality of the recommendation system, and confirms that the attention mechanism is helpful for estimating the contribution of the historical items interacted by the user to the user's preference representation , making the personalized recommendation more accurate. Using dot product attention and self-attention to calculate the attention score separately, and achieved remarkable results. At the same time, the transformer model is combined with the recommendation algorithm and compared with the conventional embedding model, showing the improvement of the recommendation effect.

Description

An item recommendation method based on collaborative filtering

technical field

The invention relates to the technical field of recommendation systems, in particular to an item recommendation method based on collaborative filtering.

Background technique

Collaborative Filtering (CF) is the earliest and most famous recommendation algorithm. The main function is prediction and recommendation, which is not only deeply studied in academia, but also widely used in industry. The algorithm discovers the user's preference by mining the user's historical behavior data, and recommends items with similar tastes to the user based on different preferences. Collaborative filtering recommendation algorithms are mainly divided into two categories, namely User-based Collaborative Filtering (UserCF) and Item-based Collaborative Filtering (ItemCF). To put it simply: people gather in groups, and things divide in groups. The user-based collaborative filtering algorithm is to discover the user's likes of goods or content (such as commodity purchases, collections, content comments or sharing) through the user's historical behavior data, and to measure and score these preferences. Calculate the relationship between users according to the attitudes and preferences of different users to the same product or content, and recommend products among users who have the same preference. Simply put, if both users A and B have purchased three books x, y, and z, and gave 5-star praise, then A and B belong to the same category of users, and they can read the books A read w. Also recommended to User B. UserCF is used in some websites (such as Digg), but the algorithm has some drawbacks. First of all, as the number of users of the website increases, it will become more and more difficult to calculate the similarity matrix of user interests, and the increase of the computational time complexity and space complexity and the increase of the number of users are approximated by a square relationship. Second, user-based collaborative filtering is difficult to interpret the recommendation results. Therefore, the famous e-commerce company Amazon has proposed another item-based collaborative filtering algorithm.

Item-based collaborative filtering (ICF) recommends items to users that are similar to items they liked before. For example, the algorithm will recommend "Machine Learning" to you because you have purchased "Introduction to Data Mining". However, the ICF algorithm does not use the content attributes of the items to calculate the similarity between items. It mainly calculates the similarity between items by analyzing the user's behavior records. ICF not only provides persuasive explanations for prediction results in many recommendation scenarios, but also makes real-time personalization easier. Specifically, the main computation of estimating the similarity between items can be done offline, while the online recommendation module only needs to perform a series of lookups for similar items, which is easily done in real-time.

The earliest item-based collaborative filtering ItemCF method is to determine whether to add the target item to the user's recommendation list by calculating the similarity between the items that the user has contacted in the past and the current target item, that is, the predicted user u for a specific item i. The score is equal to the final accumulated value of the similarity s _ij of all items j interacted by user u with item i multiplied by the user's score r _uj for j. Calculated as follows:

Early ItemCF methods used statistical measures to calculate the similarity between user's historical items and target items, such as methods such as Pearson coefficient and Cosine similarity. The method is simple, but this heuristic-based approach for estimating item similarity lacks optimization tailored for recommendation and thus yields sub-optimal performance. Secondly, in the case of sparse data, it is assumed that the user-adjusted cosine similarity for unevaluated items is 0, and the co-related items that users evaluate together in the Pearson coefficient may be small. Therefore, these methods need to be adjusted and optimized to adapt different datasets to recommendation schemes. With the development of machine learning, a learning-based method was proposed, called SLIM. The method mainly customizes a recommendation objective function to learn the similarity between items by optimizing it adaptively from the data. That is to minimize the loss between the original user-item interaction matrix and the interaction matrix reconstructed by the item-based CF model. Although SLIM can achieve better recommendation accuracy, it has two inherent limitations. First, the offline training process may be very time-consuming for large-scale data, because the time complexity is in the order of O(I ² ) to directly learn the similarity matrix S. Second, it can only estimate the similarity between two items that are purchased or rated jointly, and cannot estimate the similarity between unrelated items, so it cannot capture the transitive relationship between items. In practical recommendation tasks, especially when the data is sparse, the recommendation effect of SLIM will drop.

FISM addresses these limitations nicely. This method mainly represents items as low-dimensional embedding vectors, so the similarity s _ij between items is parameterized as the inner product of the embedding vectors of items i and j. As the number of users and items increases, the entire interaction matrix becomes sparse, and the effectiveness of existing Top-K recommendation methods decreases. In the FISM algorithm, an item-based method is proposed to generate Top-K recommendations. The recommendation algorithm The learning of the item similarity matrix is set as the product of two low-dimensional latent factor matrices. A full set of experiments on multiple datasets with several different sparsity levels show that the proposed method in the FISM algorithm can effectively handle sparse datasets. Due to this, the recommendation accuracy of FISM is superior to other very popular Top-K recommendation algorithms, especially its performance is greatly improved as the dataset becomes sparse. Although it has superior performance, it is obviously unreasonable to assume that all the historically interacted items of the user have the same contribution to the representation of the user's preferences. For example, basketball and daily necessities have different effects on real-time recommendation of basketball shoes. So we introduce a special dynamic weight to better predict the user u's preference for item i, and this dynamic weight will be estimated using the attention mechanism.

SUMMARY OF THE INVENTION

In view of the problems existing in the prior art, the present invention provides an item recommendation method based on collaborative filtering.

An item recommendation method based on collaborative filtering, the specific steps are as follows:

1. A method for recommending items based on collaborative filtering, characterized in that: comprising the following steps:

Step 1: Calculate user u's prediction score for target item i, and use one-hot encoding to predict target item i to obtain its embedding vectors p and q through the embedding layer, where p indicates that the item is a predicted item, and q indicates that it is a historical interaction Items, get their evaluation indicators, and define the attention-based ItemCF formula as follows:

a _ij =f(pi , _{q j )}

表示去掉正例集中的物品i，

为系数；Among them, i is the predicted target item, j is the user's historical interactive item, a _ij is the weight of the historical interactive item calculated by the attention network to the user's preference representation, p _i and q _j represent the embedding vector of the predicted item set, respectively Embedding vector of items that have interacted with the user, R represents the positive example set of user u,

means to remove the item i in the positive example set,

is the coefficient;

将拼接向量作为点乘注意力模型的输入，将注意力机制的第一次尝试命名为Dot；Step 1.1: Splicing the embedding vector p _i of the query item set and the embedding vector q _j of the item set interacted by the user to obtain the splicing vector c,

The spliced vector is used as the input of the dot product attention model, and the first attempt of the attention mechanism is named Dot;

Step 1.1.1: Perform three linear transformations on the splicing vector c independently, and the coefficient matrices are W _Q , W _K , W _V respectively, thereby obtaining the input Query, Key, Value (Q, K, V) of the attention network;

Step 1.1.2: Use highly optimized matrix multiplication to achieve the dot product of Q and K transpose, after softmax, multiply with V to get the weight matrix, denote the attention function as Attention(Q, K, V), The calculation formula is as follows:

Among them, d _k represents the dimension of K, and the softmax function converts the value into a probability distribution. If the dimensions of Q, K, and V are the same, the dimensions of the output attention weight matrix are also the same as them;

作为输入放入网络中，将前面的单个点乘注意力重复h次，将h次的结果矩阵拼接起来，最后通过线性变换转为需要的维度，即将注意力函数设置为自注意模型来计算历史物品j对用户u预测目标物品i的得分所贡献的权重，并将其命名为Self；Step 1.2: Put the stitching vector

Put it into the network as input, repeat the previous single point multiplication attention h times, splicing the result matrix of h times, and finally convert it to the required dimension through linear transformation, that is, set the attention function to the self-attention model to calculate the history The weight that item j contributes to user u's prediction of the target item i's score, and it is named Self;

Step 1.3: Use the main framework of the Transformer model, which is mainly divided into two modules: encoder and decoder, and set the input of the first sub-module of the encoder module as the embedding vector p _i of the target item to be predicted, and the rest of each sub-module. The input is the output of the previous sub-module. Each encoder sub-module consists of two layers. The first layer is the self-attention model layer, and the second layer is the feedforward layer. After the Attention operation, the encoder and decoder will both contain a fully connected front. To the network, including two linear transformations and a Relu activation output, the formula is as follows:

FFN(x)=max(0, xW ₁ +b ₁ )W ₂ +b ₂

Input the first sub-module of the decoder module to set the set q _j of historical items of user interaction, the input of each remaining sub-module is the output of the previous sub-module, each decoder sub-module consists of three layers, the first layer and The second layer is a self-attention layer, but the input Q of the second layer is the output of the previous layer, K and V are the output of the encoder, and the third layer is the feedforward layer. After each layer, add an "Add&Normalize" layer to Prevent the gradient from disappearing or exploding, while preventing overfitting; the output of the model is transformed into the required size through a fully connected layer and the softmax function to obtain the attention weight a _ij and the following work is performed, defining the model as Trans;

Step 1.4: Customize the objective function to treat observed user-item interactions as positive instances, extract negative instances from the remaining unobserved interactions, use R ⁺ and R- ^to represent the set of positive and negative instances, and use log as the loss term , and use the L2 norm to penalize the embedding vector and the coefficients and bias terms of each network. Then the loss function is as follows:

where N represents the total number of training instances, σ represents the sigmoid method to convert the predicted value into a probability value, and the strength of the L2 paradigm controlled by the hyperparameter λ is used to prevent overfitting, θ={{p _i },{q _j },W ,b,h} represents all trainable parameters, where W, b, h and all the linear transformation parameters used have regular penalties; a variant of the stochastic gradient descent algorithm is called Adagrad optimization objective function , which applies an adaptive learning rate to each parameter, draws random samples from all training instances, and updates the relevant parameters in the negative direction of the gradient. Using a mini-batch approach, a user is randomly selected and the set of all items it has interacted with is used as a mini-batch.

Step 2: Experiment on the real item dataset on the evaluation index, the performance is judged by the recommendation results, and the experimental results are compared with other recommendation methods.

Beneficial effects of the present invention:

In the present invention, the attention mechanism transformer of machine translation in natural language processing is applied to the recommendation model, and the method proposed in this paper is tested on two real data sets of movies and pictures, and two commonly used recall rate and precision rate are used. Recommend model evaluation metrics for evaluation. Based on the recall rate, the method achieves a relative improvement of 3.2%, and based on the precision rate, it achieves a relative improvement of 4.3%, so this method can generate a more accurate personalized recommendation list for users. An efficient recommendation system can provide users with an efficient and intelligent information filtering technology when users lack experience in relevant fields or cannot process massive data, mine users' potential consumption tendencies, and provide personalized services for many users. . By accurately recommending items to users, it can enhance users' interest, increase website page views, click-through rate and purchase rate, bring revenue to the website and bring great convenience to users' life and leisure. A better recommendation method can bring business value to enterprise entities, optimize sales boundaries and profits, help products expand boundaries, provide more diverse and more intimate experiences through scenario construction, and ultimately increase profits, etc.

Description of drawings

Figure 1 shows the basic framework of the attention-based Item CF model;

Figure 2 shows the structure of the dot product attention model;

Figure 3 shows the basic framework of the Transformer model;

Figure 4 shows the performance comparison of models FISM, Dot, Self, and Trans when the embedding size is 16;

Figure (a) is ML-1M-HR, Figure (b) is ML-1M-NDCG, Figure (c) is Pinterest-20-HR, and Figure (d) is Pinterest-20-NDCG.

Detailed ways

In order to make the objectives, technical solutions and advantages of the present invention clearer, the present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments. The specific embodiments described herein are only used to explain the present invention, and are not intended to limit the present invention.

An item recommendation method based on collaborative filtering, comprising the following steps:

Step 1: As shown in Figure 1, Multi-hot encoding is used to represent users, that is, all items that user u has interacted with in the case of implicit feedback are used to represent u. In this case, the user's Multi-hot encoding is embedded by embedding layer and produce a set of vectors, where each vector represents a historical item associated with the user, and the target item to be predicted uses one-hot encoding to obtain its embedding vector through the embedding layer. Calculate the prediction score of user u for target item i, and use one-hot encoding to predict target item i to obtain its embedding vectors p and q through the embedding layer, where p indicates that the item is a predicted item, and q indicates that it is a historical interactive item, such as As shown in Figure 1, the formula for defining the attention-based ItemCF is as follows:

a _ij =f(pi , _{q j )}

表示去掉正例集中的物品i，

为系数；Among them, i is the predicted target item, j is the user's historical interactive item, a _ij is the weight of the historical interactive item calculated by the attention network to the user's preference representation, p _i and q _j represent the embedding vector of the predicted item set, respectively Embedding vector of items that have interacted with the user, R represents the positive example set of user u,

means to remove the item i in the positive example set,

is the coefficient;

Step 1.1: Splicing the embedding vector p _i of the query item set and the embedding vector q _j of the item set interacted by the user to obtain the splicing vector c to learn the interaction weight, The spliced vector is used as the input of the dot product attention model, as shown in Figure 2, and the first attempt of the attention mechanism is named Dot;

Step 1.1.1: Perform three linear transformations on the splicing vector c independently, and the coefficient matrices are W _Q , W _K , W _V respectively, thereby obtaining the input Query, Key, Value (Q, K, V) of the attention network;

起调节作用，使得内积不至于太大，否则会导致softmax层的值非0即1，引起梯度消失或者爆炸问题，这样做可以使值保持在梯度较大的位置，softmax函数是将值转换为概率分布，这对梯度计算非常友好，在softmax之后，与V相乘以得到权重矩阵，将注意力函数表示为Attention(Q，K，V)，计算公式如下所示：Step 1.1.2: Use highly optimized matrix multiplication to implement the dot product of the Q and K transposes, as the dot product is faster, more space efficient, and can be implemented using highly optimized matrix multiplication, where the factor

It plays a regulating role so that the inner product is not too large, otherwise the value of the softmax layer will be either 0 or 1, causing the gradient to disappear or explode. This can keep the value at a position with a large gradient. The softmax function converts the value It is a probability distribution, which is very friendly to gradient calculation. After softmax, it is multiplied by V to get the weight matrix, and the attention function is expressed as Attention(Q, K, V). The calculation formula is as follows:

Among them, d _k represents the dimension of K, and the softmax function converts the value into a probability distribution. If the dimensions of Q, K, and V are the same, the dimensions of the output attention weight matrix are also the same as them;

作为输入放入网络中，将前面的单个点乘注意力重复h次，将h次的结果矩阵拼接起来，最后通过线性变换转为需要的维度，即将注意力函数设置为自注意模型来计算历史物品j对用户u预测目标物品i的得分所贡献的权重，并将其命名为Self。Step 1.2: Put the stitching vector

Put it into the network as input, repeat the previous single point multiplication attention h times, splicing the result matrix of h times, and finally convert it to the required dimension through linear transformation, that is, set the attention function to the self-attention model to calculate the history The weight that item j contributes to user u's prediction of target item i's score is named Self.

Step 1.3: Use the main framework of the Transformer model, as shown in Figure 3, which is mainly divided into two modules: encoder and decoder, and set the input of the first sub-module of the encoder module as the embedding vector p _i of the target item to be predicted, The input of each remaining sub-module is the output of the previous sub-module. Each encoder sub-module consists of two layers. The first layer is the self-attention model layer, and the second layer is the feedforward layer. After the Attention operation, the encoder and decoder will be Contains a fully connected forward network consisting of two linear transformations and a Relu activation output with the following formula:

FFN(x)=max(0, xW ₁ +b ₁ )W ₂ +b ₂

The first submodule input of the decoder module sets the set q _j of historical items of user interaction, which greatly enhances the interpretability of the model, the input of each remaining submodule is the output of the previous submodule, and each decoder The sub-module consists of three layers. The first layer and the second layer are self-attention layers, but the input Q of the second layer is the output of the previous layer, K and V are the output of the encoder, and the third layer is the feedforward layer. Add "Add&Normalize" layer after each layer to prevent the gradient from disappearing or exploding, while preventing overfitting; for the output of the model, it will be transformed to the desired size through a fully connected layer and softmax function to obtain the attention weight α _ij and do the following work, define the model as Trans;

Step 1.4: Customize the objective function to treat observed user-item interactions as positive instances, extract negative instances from the remaining unobserved interactions, use R ⁺ and R- ^to represent the set of positive and negative instances, use a crossover loss function As the objective function, the embedding vector and the coefficients and bias terms of each network are penalized with L2 norm. Then the objective function is as follows:

where N represents the total number of training instances, σ represents the sigmoid method to convert the predicted value into a probability value representing the possibility that user u will interact with item i, and the hyperparameter λ controls the strength of the L2 paradigm to prevent overfitting, θ={{ p _i },{q _j },W,b,h} represent all trainable parameters, where W, b, h and all the linear transformation parameters used have regular penalties; the stochastic gradient descent algorithm is used. A variant called Adagrad optimizes the objective function, which applies an adaptive learning rate to each parameter, draws random samples from all training instances, and updates the relevant parameters in the negative direction of the gradient.

This example uses TensorFlow to implement all models, and TensorFlow requires that all training instances of a batch must have the same length, because some active users may interact with thousands of items, making the sampled mini-batch training set still very large. In order to solve this problem, this embodiment adopts a mini-batch method, randomly selects a user, and then uses all the interacted item sets as a mini-batch, instead of randomly selecting a fixed number of training instances as mini-batch training set. This approach has two advantages: 1) it doesn't have to use masking tricks, so it's faster; 2) it doesn't need to specify the batch size, which avoids batch resizing. If the attention network and the item embedding vector are trained at the same time, since the output of the attention network will make changes to the item embedding, joint training will easily lead to an adaptive effect, thereby reducing the convergence speed. In order to solve the practical problem in model training, this embodiment uses the FISM algorithm proposed by Kabbur et al. to pre-train the model, and uses the item embedding vector learned by the FISM algorithm to initialize the model. Since the FISM algorithm itself has no self-adaptation problem, it can better learn the similarity of the embedding-encoded items. Therefore, using the FISM algorithm to initialize the model can greatly promote the learning of the attention network, resulting in better performance and faster convergence.

Step 2: Experiment on the real item dataset on the evaluation index, the performance is judged by the recommendation results, and the experimental results are compared with other recommendation methods.

This embodiment assigns a weight to each item that the user has interacted with in the history, so that the user's historical item set can more accurately represent the user's preference when the user predicts and scores the target item, the recommendation effect is improved, and the personalized recommendation is also more accurate. , we attribute these improvements to the effective introduction of an attention mechanism to distinguish the importance of historical items in user representations. We conduct comprehensive experiments on two real-item datasets, ML-1M and Pinterest-20, on the evaluation metrics HR and NDCG to evaluate Top-K. The performance is judged by the Hit Ratio (HR) and Normalized Discounted Cumulative Gain (NDCG) of the top 10 recommended results. These two metrics have been widely used in search systems evaluating Top-K recommendation and information retrieval literature. HR@10 can be interpreted as a recall-based metric representing the percentage of successfully recommended users (i.e., positive instances appearing in the top 10), while NDCG@10 is the predicted position considering the positive instances. Bigger is better.

We compare the experimental results with some popular recommendation methods. For these embedding-based methods (MF, MLP, FISM, and our model), the embedding size controls their modeling power; therefore, we set all methods to 16. The results are shown in Table 1. It can be seen that the attention-based models have achieved better results in the end, and the final results are similar. They achieved the highest NDCG and HR scores in all datasets. On the ML-1M dataset, the performance of the three models has been improved to a certain extent compared with FISM, among which the Self model has a relative improvement of 3.1% and 4.3% in HR and NDCG compared with FISM. This may be that a model with a relatively simple structure can more comprehensively capture user characteristics and describe user preferences on a less sparse dataset. On Pinterest-20, Trans outperformed the other two and got the highest score and improved by 3.2% on NDCG relative to FISM, which may be because deeper networks can better capture the characteristics of sparse data. Learning-based collaborative filtering methods generally perform much better than heuristic-based methods such as Pop and ItemKNN, especially FISM is much higher than ItemKNN. Considering that the two methods use the same prediction model, but the similarity estimation method for items is different, the positive impact of customized optimization on recommendation can be clearly seen.

Table 1 Effect comparison chart

As shown in Figure 4, when the item embedding size is 16, the performance of FISM and Dot, Self and Trans proposed in this application in each epoch, these three models have achieved the highest HR and NDCG on both datasets. points, they achieve the same level of performance and achieve a significant improvement over FISM. We attribute these advantages to the efficient design of attention networks when learning item-to-item interactions. Especially in the first epoch, the performance of our model significantly exceeds that of FISM, and as the training time increases, the experimental effect will become better until convergence.

Based on the above discussion, this paper conducts research on item-based collaborative filtering algorithms, tries to access a variety of attention models to improve the learning of dynamic weight coefficients, implements and experiments, and compares with other models, and achieves better results. Effect. The main contributions are: (1) It is confirmed that the attention mechanism helps to capture the dynamic weight of the contribution of new items to the similarity calculation between the set of historical items that the user has been exposed to. Make personalized recommendation more accurate. (2) Using the dot product attention and self-attention to calculate the attention score separately, and achieved good results. (3) The transformer model is combined with the recommendation algorithm and compared with the conventional embedding model, showing the improvement of the recommendation effect.

Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, but not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those of ordinary skill in the art should understand; The technical solutions described in the foregoing embodiments are modified, or some or all of the technical features thereof are equivalently replaced; therefore, these modifications or replacements do not make the essence of the corresponding technical solutions depart from the scope defined by the claims of the present invention.

Claims (2)

1. an article recommendation method based on collaborative filtering, is characterized in that: comprise the following steps: Step 1: Calculate user u's prediction score for target item i, and use one-hot encoding to predict target item i to obtain its embedding vectors p and q through the embedding layer, where p indicates that the item is a predicted item, and q indicates that it is a historical interaction Items, get their evaluation indicators, and define the attention-based ItemCF formula as follows:

a _ij =f(pi , _{q j )} Among them, i is the predicted target item, j is the user's historical interactive item, a _ij is the weight of the historical interactive item calculated by the attention network to the user's preference representation, p _i and q _j represent the embedding vector of the predicted item set, respectively Embedding vector of items that have interacted with the user, R represents the positive example set of user u,

means to remove the item i in the positive example set,

is the coefficient; Step 2: Experiment on the real item dataset on the evaluation index, the performance is judged by the recommendation results, and the experimental results are compared with other recommendation methods. 2. A collaborative filtering-based item recommendation method according to claim 1, wherein the specific steps of the step 1 are: Step 1.1: Splicing the embedding vector p _i of the predicted item set and the embedding vector q _j of the item set interacted by the user to obtain the splicing vector c,

The spliced vector is used as the input of the dot product attention model, and the first attempt of the attention mechanism is named Dot; Step 1.1.1: Perform three linear transformations on the splicing vector c independently, and the coefficient matrices are W _Q , W _K , W _V respectively, thereby obtaining the input Query, Key, Value (Q, K, V) of the attention network; Step 1.1.2: Use highly optimized matrix multiplication to achieve the dot product of Q and K transpose, after softmax, multiply with V to get the weight matrix, denote the attention function as Attention(Q, K, V), The calculation formula is as follows:

Among them, d _k represents the dimension of K, and the softmax function converts the value into a probability distribution. If the dimensions of Q, K, and V are the same, the dimensions of the output attention weight matrix are also the same as them; Step 1.2: Put the stitching vector Put it into the network as input, repeat the previous single point multiplication attention h times, splicing the result matrix of h times, and finally convert it to the required dimension through linear transformation, that is, set the attention function to the self-attention model to calculate the history The weight that item j contributes to user u's prediction of the target item i's score, and it is named Self; Step 1.3: Use the main framework of the Transformer model, which is mainly divided into two modules: encoder and decoder, and set the input of the first sub-module of the encoder module as the embedding vector p _i of the target item to be predicted, and the rest of each sub-module. The input is the output of the previous sub-module. Each encoder sub-module consists of two layers. The first layer is the self-attention model layer, and the second layer is the feedforward layer. After the Attention operation, the encoder and decoder will both contain a fully connected front. To the network, including two linear transformations and a Relu activation output, the formula is as follows: FFN(x)=max(0, xW ₁ +b ₁ )W ₂ +b ₂ Input the first sub-module of the decoder module to set the set q _j of historical items of user interaction, the input of each remaining sub-module is the output of the previous sub-module, each decoder sub-module consists of three layers, the first layer and The second layer is a self-attention layer, but the input Q of the second layer is the output of the previous layer, K and V are the output of the encoder, and the third layer is the feedforward layer. After each layer, add an "Add&Normalize" layer to Prevent the gradient from disappearing or exploding, and at the same time prevent overfitting; the output of the model is converted to the required size through a fully connected layer and the softmax function to obtain the attention weight a _ij and the following work is performed, and the model is defined as Trans; Step 1.4: Customize the objective function to treat observed user-item interactions as positive instances, extract negative instances from the remaining unobserved interactions, use R ⁺ and R- ^to represent the set of positive and negative instances, and use log as the loss term , and use the L2 norm to penalize the embedding vector and the coefficients and bias terms of each network, then the loss function is as follows:

where N represents the total number of training instances, σ represents the sigmoid method to convert the predicted value into a probability value, and the strength of the L2 paradigm controlled by the hyperparameter λ is used to prevent overfitting, θ={{p _i },{q _j },W ,b,h} represents all trainable parameters, where W, b, h and all the linear transformation parameters used have regular penalties; a variant of the stochastic gradient descent algorithm is called Adagrad optimization objective function , which applies an adaptive learning rate to each parameter, draws random samples from all training instances, updates the relevant parameters in the negative direction of the gradient, uses a mini-batch method that randomly picks a user, and then uses it for all interactions The set of items passed as a mini-batch.

Images