CN116933049A - Feature selection method, device, electronic equipment and storage medium - Google Patents

Abstract

The application provides a feature selection method, a feature selection device, an electronic device, a computer readable storage medium and a computer program product; can be applied to an in-vehicle scene, the method comprising: invoking a feature selection model to read weights corresponding to a plurality of features of the training sample from the controller model, and fusing the weights to each feature correspondingly; based on each feature fused with the weight, calling a feature information mining model to execute a test task to obtain a test result; performing back propagation processing based on the difference between the test result and the marking result of the training sample to update the characteristic information mining model and the controller model; and calling a feature selection model to read updated weights from the controller model, and screening partial features from the plurality of features based on the updated weights to combine to obtain combined features. According to the application, the combined characteristics for the model to execute the prediction task can be accurately and efficiently screened from the large-scale data set.

Description

Feature selection method, device, electronic device and storage medium

Technical Field

The present application relates to the field of artificial intelligence technology, and in particular to a feature selection method, device, electronic device and computer-readable storage medium.

Background Art

Artificial Intelligence (AI) is a comprehensive technology in computer science. By studying the design principles and implementation methods of various intelligent machines, machines are given the functions of perception, reasoning and decision-making. Artificial Intelligence technology is a comprehensive discipline that covers a wide range of fields, such as natural language processing technology and machine learning/deep learning. With the development of technology, artificial intelligence technology will be applied in more fields and play an increasingly important role.

When training machine learning models, the feature selection schemes provided by related technologies often only apply to small-scale data sets when selecting features for training samples. That is, for large-scale data sets, the efficiency of the selected features and the prediction accuracy of the model cannot be guaranteed.

Summary of the invention

The embodiments of the present application provide a feature selection method, device, electronic device, computer-readable storage medium and computer program product, which can accurately and efficiently screen out combined features for a model to perform prediction tasks from a large-scale data set.

The technical solution of the embodiment of the present application is implemented as follows:

The present application provides a feature selection method, including:

Calling the feature selection model to read the weights corresponding to the multiple features of the training sample from the controller model, and integrating the weights to each of the features;

Based on each of the features fused with the weight, calling the feature information mining model to perform the test task and obtain the test result;

Performing back propagation processing based on the difference between the test result and the labeling result of the training sample to update the feature information mining model and the controller model;

Calling the feature selection model to read the updated weights from the controller model, and selecting some features from the multiple features based on the updated weights for combination to obtain combined features;

The combined features are used by the updated feature information mining model to perform prediction tasks.

The present application provides a feature selection device, including:

A calling module is used to call the feature selection model to read the weights corresponding to multiple features of the training sample from the controller model;

A fusion module, used for fusing the weights to each of the features;

The calling module is further used to call the feature information mining model to perform the test task based on each feature fused with the weight, and obtain the test result;

An updating module, configured to perform back propagation processing based on the difference between the test result and the labeling result of the training sample, so as to update the feature information mining model and the controller model;

The calling module is further used to call the feature selection model to read the updated weight from the controller model;

A combination module, used for selecting some features from the multiple features based on the updated weights for combination to obtain combined features;

The combined features are used by the updated feature information mining model to perform prediction tasks.

An embodiment of the present application provides an electronic device, including:

A memory for storing executable instructions;

The processor is used to implement the feature selection method provided in the embodiment of the present application when executing the executable instructions stored in the memory.

An embodiment of the present application provides a computer-readable storage medium storing executable instructions for implementing the feature selection method provided in the embodiment of the present application when executed by a processor.

An embodiment of the present application provides a computer program product, including a computer program or instructions, which are used to implement the feature selection method provided in the embodiment of the present application when executed by a processor.

The embodiments of the present application have the following beneficial effects:

The feature selection model first determines the weighted features based on the weights of multiple features of the training samples in the controller model. Then the feature information mining model performs the test task based on the weighted features to optimize the weights in the controller model, and then selects some of the best features for combination for the feature information mining model to perform subsequent prediction tasks. Compared with the existing technology, it avoids the large demand for expert knowledge and manpower, can adapt to training sets of different sizes, reduce computing requirements, and select the best combination of features for the updated feature information mining model to perform prediction tasks, thereby ensuring prediction accuracy.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG1 is a schematic diagram of the architecture of a feature selection system 100 provided in an embodiment of the present application;

FIG2 is a schematic diagram of the structure of a server 200 provided in an embodiment of the present application;

FIG3 is a schematic diagram of the principle of the feature selection method provided in an embodiment of the present application;

FIG4 is a schematic diagram of a flow chart of a feature selection method provided in an embodiment of the present application;

FIG5A is a schematic diagram of a flow chart of a feature selection method provided in an embodiment of the present application;

FIG5B is a schematic diagram of a flow chart of a feature selection method provided in an embodiment of the present application;

FIG6 is a schematic diagram showing the principle of a feature selection method provided in an embodiment of the present application;

FIG7A is a schematic diagram of a controller network in an initialization phase provided by an embodiment of the present application;

FIG7B is a schematic diagram of a controller network in a search phase according to an embodiment of the present application;

FIG7C is a schematic diagram of a controller network in a retraining phase according to an embodiment of the present application;

FIG8 is a schematic diagram of the structure of a feature information mining model provided in an embodiment of the present application.

DETAILED DESCRIPTION

In order to make the purpose, technical solutions and advantages of the present application clearer, the present application will be further described in detail below in conjunction with the accompanying drawings. The described embodiments should not be regarded as limiting the present application. All other embodiments obtained by ordinary technicians in the field without making creative work are within the scope of protection of this application.

In the following description, reference is made to “some embodiments”, which describe a subset of all possible embodiments, but it will be understood that “some embodiments” may be the same subset or different subsets of all possible embodiments and may be combined with each other without conflict.

It is understandable that in the embodiments of the present application, data related to the user's gender, age, preferences, etc., when the embodiments of the present application are applied to specific products or technologies, user permission or consent is required, and the collection, use and processing of relevant data need to comply with relevant laws, regulations and standards of relevant countries and regions.

In the following description, the terms "first\second\..." involved are merely used to distinguish similar objects and do not represent a specific ordering of the objects. It can be understood that "first\second\..." can be interchanged with a specific order or sequence where permitted, so that the embodiments of the present application described herein can be implemented in an order other than that illustrated or described herein.

In the following description, the term "plurality" referred to means at least two.

Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by those skilled in the art to which this application belongs. The terms used herein are only for the purpose of describing the embodiments of this application and are not intended to limit this application.

Before further describing the embodiments of the present application in detail, the nouns and terms involved in the embodiments of the present application are explained. The nouns and terms involved in the embodiments of the present application are subject to the following interpretations.

1) Features: including numerical features and categorical features, where numerical data has actual measurement meaning, such as a person's height, weight, blood pressure, etc., or counts, such as the number of times a website is visited, the number of times an item is purchased, etc. The quantity represented by categorical data can be a person's gender, marital status, hometown, or the type of movies they like. The value of categorical data can be numerical (for example, "1" represents male and "0" represents female), but the numerical values have no mathematical meaning and they cannot be used for mathematical operations. Categorical features can not only be extracted directly from the original data, but also be obtained by discretizing numerical features.

2) Feature selection: Also known as attribute selection or variable selection, it refers to selecting only some effective features from all available features and excluding invalid and harmful features. Feature selection is very important for machine learning applications, which can simplify the model, improve performance and versatility, and reduce the risk of overfitting.

3) Combined features: synthetic features formed by combining two or more individual features (such as multiplying or finding Cartesian products), such as the combination of user and category features (for example, the number of clicks by users in different categories on an e-commerce platform), the combination of different age groups of users and category features (for example, on a news platform, young users will have a higher number of clicks on entertainment news, while young users will have a higher number of clicks on social news), and the combination of user identity and time features (for example, students will have more diverse behaviors in a certain game client on weekends).

4) Multi-layer Perceptron (MLP): Also known as artificial neural network, in addition to the input layer and output layer, it can contain multiple hidden layers in the middle to map multiple input data sets to a single output data set.

Feature selection is an important task in the training phase of machine learning models applied to various scenarios. Taking the information recommendation scenario as an example, the quality of features can significantly affect the performance of the recommendation system. In related technologies, the method of manually selecting features for combination using expert knowledge is actually a trial-and-error method. Because in the recommendation system, even experts cannot predict whether the combined features obtained by combining the selected features can achieve a good model effect. Once the result is not as expected, the user needs to invest a lot of manpower again to analyze the reasons and make new selections. This method inevitably requires expert knowledge and a lot of labor, which is difficult for many users of recommendation systems to support.

In addition, the solution provided by the related technology using search algorithms to solve the feature selection problem also requires a large amount of computing resources. Even if a carefully designed search algorithm is used, it is impossible to avoid the need to fully train the model each time a search is performed in order to obtain the performance of the combined features obtained by combining the selected features in the recommendation system, which will bring a large amount of computing requirements.

Furthermore, the feature selection methods designed based on statistical theory provided by related technologies, whether packaging, filtering, or embedding, are not designed for large-scale data sets. Simple methods, such as filtering using chi-square scores, can be used on large-scale data sets, but their accuracy is affected by the size of the data and the limitations of the algorithm itself, and they cannot give good feature selection results; complex methods, such as the Fast Correlation-Based Filter (FCBF) and the Minimum Redundancy Maximum Relevance (MRMR) method, cannot give feature selection results within a limited time when applied to large-scale data sets due to the complexity of the algorithm.

Finally, the feature selection methods based on reinforcement learning provided by related technologies also have similar problems. For example, in order to calculate the "reward", it is inevitable to train the model to convergence; at the same time, the reinforcement learning feature selection methods provided by related technologies are designed for small-scale data sets. When facing large-scale data sets, the efficiency of feature selection and the prediction accuracy of the model will be greatly reduced.

In summary, the feature selection schemes of related technologies are not suitable for large-scale data sets, and the feature selection efficiency is low, resulting in a large consumption of computing resources; at the same time, the accuracy of the selected feature combination is not enough, which affects the prediction accuracy of the model.

In view of this, the embodiments of the present application provide a feature selection method, device, electronic device, computer-readable storage medium and computer program product, which can accurately and efficiently determine the combined features for the model to perform prediction tasks from a large-scale data set. The following describes an exemplary application of the electronic device provided by the embodiments of the present application. The electronic device provided by the embodiments of the present application can be implemented as a server, or can be implemented by a server and a terminal device in collaboration.

For example, see Figure 1, which is a schematic diagram of the architecture of a feature selection system 100 provided in an embodiment of the present application. In order to support an application that can accurately and efficiently screen out combined features for a model to perform prediction tasks from a large-scale data set, a terminal device 400 is connected to a server 200 via a network 300. The network 300 can be a wide area network or a local area network, or a combination of the two.

The models involved in the embodiments of the present application include a controller model, a feature selection model, and a feature information mining model. These models can be run uniformly by the server 200, or can be run collaboratively by the server 200 and the terminal device 400, as described in detail below.

In some embodiments, the feature selection method provided in the embodiments of the present application can be implemented by the server alone, that is, the server runs all the above-mentioned models. For example, the server 200 may call the feature selection model to read the weights corresponding to the multiple features of the training sample from the controller model, and determine each feature fused with the weight, wherein the training sample may be obtained by the server 200 from a database (not shown in FIG. 1 ); then the server 200 may call the feature information mining model to perform the test task based on each feature fused with the weight, and obtain the test result; then the server 200 may perform back propagation processing based on the difference between the test result and the labeling result of the training sample to update the feature information mining model and the controller model; finally, the server 200 may call the feature selection model to read the updated weight from the controller model, and select some features from the multiple features based on the updated weight to combine them to obtain the combined feature; wherein the combined feature may be used for the feature information mining model to perform the prediction task, for example, when the prediction task is a recommendation task, the server 200 may send the information whose click rate is greater than the click rate threshold to the terminal device 400 based on the prediction result output by the feature information mining model (such as the click rate of the user for different information), so as to be displayed in the human-computer interaction interface of the client 410 running on the terminal device 400.

In other embodiments, the feature selection method provided in the embodiment of the present application can also be implemented in coordination by the terminal device and the server, that is, the server runs the controller model, the feature selection model and the feature information mining model, and the terminal device runs the feature information mining model (that is, the server sends the trained feature information mining model to the terminal device). For example, after the server 200 calls the feature selection method provided in the embodiment of the present application to select some features from multiple features of the training sample for combination, obtains the combined features, and retrains the feature information mining model based on the combined features, the trained feature information mining model can be sent to the terminal device 400, so that when the terminal device 400 receives a new sample, it can call the trained feature information mining model sent by the server 200 based on the combined features corresponding to the new sample to perform the prediction task and obtain the prediction result.

The following describes the application scenarios of the feature information mining model provided in the embodiments of the present application.

The feature information mining model provided in the embodiment of the present application can be applied to various fields, for example, it can be applied to information recommendation scenarios (such as news recommendation, advertising recommendation, etc.), and it can also be applied to classification scenarios (such as text classification, image classification, etc.).

In one implementation scenario, the feature information mining model may be a recommendation model. In order to improve the accuracy of content recommendation (e.g., news recommendation), when performing content recommendation, the feature selection method provided in the embodiment of the present application may be called to select some of the best features (e.g., the user's gender and content producer) from multiple features of a training sample (e.g., historical behavior data of a sample user) (e.g., including object features, content features, and context features, etc., wherein object features include but are not limited to the user's gender, age, registration time, delivery address, frequently used areas, etc., content features include but are not limited to commodities, title segmentation of content, content source, content producer, etc., and context features are features that represent the user's current spatiotemporal state and the behavior abstraction in the recent period, such as the user's current coordinates, recently browsed content, recently purchased commodities, etc.) and combine them to obtain a combined feature (e.g., a synthetic feature formed by combining the user's gender and content producer). Then, the recommendation model may perform a prediction task based on the obtained combined feature (e.g., predicting the user's click-through rate for other content published by the same content producer, and finally recommending content with a click-through rate greater than a click-through rate threshold to the user), thereby improving the accuracy of content recommendation.

In another implementation scenario, the feature information mining model can be a text classification model. In order to improve the accuracy of text classification (for example, identification of spam), the feature selection method provided in the embodiment of the present application can be called to filter out some of the optimal features (for example, email name and sending time) from multiple features (for example, including email name, sender, recipient, sending time, email size, etc.) of a training sample (for example, multiple historical emails) and combine them to obtain a combined feature (for example, a synthetic feature formed by combining the email name and sending time). Then the text classification model can perform a prediction task based on the obtained combined feature (for example, when a new email is received, the text classification model can perform a classification task based on the combined feature consisting of the email name and sending time of the new email to determine whether the newly received email is spam), thereby improving the accuracy of spam prediction.

In some embodiments, the embodiments of the present application can be implemented with the help of cloud technology (Cloud Technology). Cloud technology refers to a hosting technology that unifies a series of resources such as hardware, software, and network within a wide area network or a local area network to achieve data calculation, storage, processing, and sharing.

Cloud technology is a general term for network technology, information technology, integration technology, management platform technology, and application technology based on the cloud computing business model. It can form a resource pool that can be used on demand and is flexible and convenient. Cloud computing technology will become an important support. The background services of the technical network system require a large amount of computing and storage resources.

For example, the server 200 shown in FIG1 can be an independent physical server, or a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, content distribution networks (CDN, Content Delivery Network), and big data and artificial intelligence platforms, wherein the cloud service can be a feature selection service for the terminal device 400 to call. The terminal device 400 can be a smart phone, a tablet computer, a laptop computer, a desktop computer, a smart TV, a smart watch, a car terminal, etc., but is not limited thereto. The terminal device 400 and the server 200 can be directly or indirectly connected by wired or wireless communication, which is not limited in the embodiments of the present application.

The structure of the server 200 shown in Figure 1 is described below. Referring to Figure 2, Figure 2 is a schematic diagram of the structure of the server 200 provided in an embodiment of the present application. The server 200 shown in Figure 2 includes: at least one processor 210, a memory 240, and at least one network interface 220. The various components in the server 200 are coupled together through a bus system 230. It can be understood that the bus system 230 is used to achieve connection and communication between these components. In addition to the data bus, the bus system 230 also includes a power bus, a control bus, and a status signal bus. However, for the sake of clarity, various buses are labeled as bus systems 230 in Figure 2.

The processor 210 can be an integrated circuit chip with signal processing capabilities, such as a general-purpose processor, a digital signal processor (DSP), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc., where the general-purpose processor can be a microprocessor or any conventional processor, etc.

The memory 240 may be removable, non-removable, or a combination thereof. Exemplary hardware devices include solid-state memory, hard disk drives, optical disk drives, etc. The memory 240 may optionally include one or more storage devices that are physically remote from the processor 210.

The memory 240 includes a volatile memory or a non-volatile memory, and may also include both volatile and non-volatile memories. The non-volatile memory may be a read-only memory (ROM), and the volatile memory may be a random access memory (RAM). The memory 240 described in the embodiments of the present application is intended to include any suitable type of memory.

In some embodiments, memory 240 can store data to support various operations, examples of which include programs, modules, and data structures, or a subset or superset thereof, as exemplarily described below.

Operating system 241, including system programs for processing various basic system services and performing hardware-related tasks, such as a framework layer, a core library layer, a driver layer, etc., for implementing various basic services and processing hardware-based tasks;

A network communication module 242, used to reach other electronic devices via one or more (wired or wireless) network interfaces 220, exemplary network interfaces 220 include: Bluetooth, Wireless Compatibility Certification (WiFi), and Universal Serial Bus (USB), etc.;

In some embodiments, the feature selection device provided in the embodiments of the present application can be implemented in software. FIG. 2 shows a feature selection device 243 stored in a memory 240, which can be software in the form of a program and a plug-in, including the following software modules: a calling module 2431, a fusion module 2432, an updating module 2433, a combining module 2434, a determining module 2435, a dividing module 2436, and an adjusting module 2437. These modules are logical, and therefore can be arbitrarily combined or further split according to the functions implemented. It should be pointed out that in FIG. 2, for the sake of convenience of expression, all the above modules are shown at once. In actual applications, it is not excluded that the feature selection device 243 only includes the calling module 2431, the fusion module 2432, the updating module 2433, and the combining module 2434. The functions of each module will be explained below.

The feature selection method provided in the embodiment of the present application will be described below in combination with the exemplary application and implementation of the electronic device provided above in the embodiment of the present application. As mentioned above, the feature selection method described below can be implemented by the server alone, or by the terminal device and the server in collaboration, and will not be repeated.

See FIG3 , which is a schematic diagram of the principle of the feature selection method provided by the embodiment of the present application. As shown in FIG3 , the feature selection method provided by the embodiment of the present application consists of two stages: a search stage and a retraining stage. In the search stage, the parameters of the controller model are first initialized (i.e., the weights corresponding to multiple features are initialized). After that, all the features to be selected (e.g., multiple features of the training sample after embedding processing) are input into the feature selection model, so that the feature selection model reads the weight corresponding to each feature from the controller model (e.g., for feature e ₁ , the feature selection model reads the corresponding weight from the controller model as in, represents the probability of discarding feature e ₁ , represents the probability of selecting feature e ₁ when the controller model is initialized. That is, the probability of selecting feature _e1 is the same as the probability of discarding feature _e1 ), and each feature fused with weights is determined. Next, each feature fused with weights can be input into the feature information mining model so that the feature information mining model performs the test task (for example, the feature information mining model can use these weighted features to predict user preferences) and obtain the test results. Subsequently, back propagation processing can be performed based on the difference between the test results (that is, the predicted values) and the labeled results of the training samples (that is, the true values) to update the feature information mining model and the controller model. At the end of this stage, the parameters of the controller model are fully trained (that is, the weights corresponding to multiple features in the controller model are updated. For example, for feature 1, when the controller model is updated, The value of is updated to 0.9, The value of is updated to 0.2, that is, the probability of selecting feature e ₁ is greater than the probability of discarding feature e ₁ ).

Continuing to refer to Figure 3, after the search phase ends, the retraining phase begins. At the beginning of the retraining phase, the feature selection model can read the updated weights from the controller model, and select some features from multiple features based on the updated weights for combination (for example, the feature selection model selects features _e1 and _e4 from features { _e1 , e2, _e3 , _e4 } based on the updated weights read from the controller model) to obtain combined features; wherein the combined features can be used for the feature information mining model to perform prediction tasks _.

The feature selection method provided in the embodiment of the present application will be specifically described below in combination with the controller model, feature selection model and feature information mining model shown in Figure 3.

For example, see Figure 4, which is a flowchart of the feature selection method provided in an embodiment of the present application, which will be described in conjunction with the steps shown in Figure 4.

In step 101, the feature selection model is called to read the weights corresponding to multiple features of the training sample from the controller model, and the weights are correspondingly integrated into each feature.

In some embodiments, as shown in Figure 7A, the feature search space of the controller model can be a directed complete graph, wherein the directed complete graph includes: multiple input nodes corresponding one-to-one to multiple features (for example, three input nodes corresponding to feature 1, feature 2, and feature 3, respectively), and an output node representing the feature selection results; there are two edges between each input node and the output node, which respectively represent the weights of the features corresponding to the input nodes, wherein the weights include a first weight for representing the probability that the feature selection model selects the feature, and a second weight for representing the probability that the feature selection model discards the feature.

For example, assuming that the training sample has a total of N different features (for example, including object features, content features, and context features, etc., wherein the object features include but are not limited to the user's gender, age, registration time, delivery address, common areas, etc., the content features include but are not limited to the goods, the title segmentation of the content, the content source, the content producer, etc., the context feature is a feature representing the user's current spatiotemporal state and the behavior abstraction of the recent period of time, such as the user's current coordinates, the most recently browsed content, the most recently purchased goods, etc.), then all possible feature combinations have a total of 2^N. If all feature combinations are directly represented as feature search spaces using the encoding method, then as the number of features to be selected increases, the feature search space will tend to infinity. In view of this, the embodiment of the present application defines the feature search space as a directed complete graph, takes each feature as an input node in the graph, and takes the result of feature selection as the output node in the graph. There are two edges between each input node and the output node, representing the selection or discarding of the feature corresponding to the node (for example, a solid line can be used to represent selection, and a dotted line can represent discarding; of course, different colors can also be used to distinguish, such as red for selection and green for discarding). Under such a design, for N different features, the embodiment of the present application only needs to determine the status of the edges between 2N input nodes and output nodes, which can significantly reduce the feature search space (i.e., from 2^N to 2N).

In some embodiments, the weight corresponding to each feature includes a first weight and a second weight, wherein the first weight is used to characterize the probability that the feature selection model selects the feature, and the second weight is used to characterize the probability that the feature selection model discards the feature. The weights can be fused to each feature in the following manner: the following processing is performed for each feature: binarization processing (for example, one-hot encoding processing) is performed on the feature whose initial format is a discrete format to obtain a binarized feature, and the binarized feature is embedded to obtain an embedded feature, wherein the dimension of the feature space of the embedded feature is smaller than the dimension of the feature space of the binarized feature; a first multiplication result (for example, a first coefficient (set to 1)) of the first weight corresponding to the embedded feature is determined. in, represents the first weight corresponding to the nth feature), and the second multiplication result (for example, the second weight corresponding to the embedded feature and the second coefficient (for example, set to 0) corresponding to the second feature in, represents the second weight corresponding to the nth feature), and the first multiplication result and the second multiplication result are summed to obtain a first summation result (for example ); the multiplication result of the first summation result and the embedded feature (for example, the nth embedded feature e _n ) (for example ), determined as a fusion of weighted features.

For example, take the feature "gender" as an example. The input is "male" or "female". After binary processing, (1, 0) can be used to represent "male" and (0, 1) can be used to represent "female". Assuming that the multiple features of the training sample are N features, the N features after binary processing can be expressed as:

x＝[x ₁ ,x ₂ ,…,x _N ]

Among them, _xn represents the result of binary processing corresponding to the nth feature. After obtaining the result of binary processing, the binary features can also be projected into a low-dimensional continuous space: _en = A _n x _n , and finally the embedded N features are obtained:

E＝[e ₁ ,e ₂ ,…,e _N ]

Among them, _en represents the result of embedding processing corresponding to the nth feature, is the parameter to be learned, and d is determined by the user (different features can use the same value or different values). In this way, by further embedding the binary features, the spatial dimension of the features is compressed, which facilitates the subsequent use of the features.

After obtaining the embedded N features, the weight corresponding to each feature can be read from the controller model. For example, taking the embedded feature _e2 as an example, the weight corresponding to feature _e2 read by the feature selection model from the controller model is Among them, the first weight represents the probability of selecting feature e ₂ , the second weight represents the probability of discarding feature e _2. Subsequently, the first weight can be The first multiplication result with the first coefficient (assuming it is 1), and the second weight The second multiplication result with the second coefficient (assumed to be 0) is summed, and the summation result is multiplied with the embedded feature e ₂ to determine the weighted feature e′ ₂ , that is,

In other embodiments, the weight corresponding to each feature includes a first weight and a second weight, wherein the first weight is used to characterize the probability that the feature selection model selects the feature, and the second weight is used to characterize the probability that the feature selection model discards the feature. The weights can be fused to each feature in the following manner: performing the following processing on each feature: binarizing the feature in an initial discrete format to obtain a binarized feature, embedding the binarized feature to obtain an embedded feature, wherein the dimension of the feature space of the embedded feature is smaller than the dimension of the feature space of the binarized feature; determining a first intermediate parameter (e.g. ) and a second intermediate parameter (e.g. in, and According to the first weight corresponding to the nth embedded feature _en and the second weight obtain, which will be described in detail below); determine a third multiplication result (for example, ), and a fourth multiplication result (eg, ), and sum the third multiplication result and the fourth multiplication result to obtain a second summation result (for example ); multiply the second summation result and the embedded feature (e.g. ), determined as a fusion of weighted features.

By way of example, the above-mentioned determination of the first intermediate parameter and the second intermediate parameter based on the first weight and the second weight corresponding to the embedded features can be implemented in the following manner: based on the first weight and the second weight corresponding to the embedded features, a normalized exponential function is called to determine the first intermediate parameter and the second intermediate parameter; wherein the normalized exponential function satisfies the following conditions: the values of the first intermediate parameter and the second intermediate parameter are marginalized (i.e., one is close to 0 and the other is close to 1); and the differentiability of the controller model is maintained.

For example, taking the normalized exponential function as the Gumbel-Softmax function, the first weight and the second weight corresponding to the embedded features can be implemented in the following manner: calling the normalized exponential function to determine the first intermediate parameter and the second intermediate parameter: calling the normalized exponential function based on the first weight and the second weight corresponding to the embedded features to perform the following processing: taking the logarithm of the first weight (for example ) and the first noise coefficient (eg, g ₁ ), and is divided by the annealing parameter (eg, τ) to obtain a first division result (eg, ); Take the logarithm of the second weight (for example ) and the second noise coefficient (eg, g ₀ ), and is divided by the annealing parameter to obtain a second division result (eg, ); sum the exponential result of the first division result and the exponential result of the second division result to obtain a third summation result (for example ); Exponentiate the result of the first division (e.g. ) and the third summation result, and determine it as the first intermediate parameter (for example in, represents the probability of the new selected embedded feature e _n ); the exponential result of the second division result (e.g. ) and the third summation result, and determine it as the second intermediate parameter (for example in, represents the probability of the new feature e _n after discarding the embedding).

In step 102, based on each feature fused with weights, a feature information mining model is called to execute the test task and obtain the test result.

As an example, the test task can be a recommendation task or a classification task. When the test task is a recommendation task, the test result can be the user's preference (such as the user's click rate on a certain item); when the test task is a classification task, the test result can be the classification result output by the feature information mining model based on each feature with a fusion weight (such as determining whether a certain email is spam).

In some embodiments, the feature information mining model may include an embedding unit and a perceptron unit, and the above-mentioned step 102 may be implemented in the following manner: based on each embedded feature fused with a weight, calling the perceptron unit to execute a test task to obtain a test result; wherein the embedded feature is obtained by calling the embedding unit to perform the following processing on the feature whose initial format is a discrete format: binarizing the feature whose initial format is a discrete format to obtain a binarized feature, embedding the binarized feature to obtain an embedded feature, wherein the dimension of the feature space of the embedded feature is smaller than the dimension of the feature space of the binarized feature.

For example, as shown in FIG8 , the perceptron unit may be a multilayer perceptron, which includes multiple layers (e.g., 4 layers), each layer including a corresponding weight matrix and a bias vector; the above-mentioned method based on each embedded feature fused with weights may be implemented in the following manner, calling the perceptron unit to perform the test task and obtain the test result: taking each embedded feature fused with weights as the input of the first layer (i.e., the input layer) of the multilayer perceptron, and iterating m to perform the following processing: summing the multiplication result of the weight matrix of the mth layer and the input of the mth layer with the bias vector of the mth layer to obtain a fourth summation result; performing a linear transformation on the fourth summation result (e.g., using a ReLU function for linear transformation); The output of the mth layer is obtained by performing a conversion process, and the output of the mth layer is used as the input of the m+1th layer (that is, the output of the previous layer will be used as the input of the next layer, for example, the output of the 1st layer is input into the 2nd layer, the output of the 2nd layer is input into the 3rd layer, and so on); wherein, the value of m increases gradually and satisfies 1≤m≤M-1, and M is the total number of multiple layers; the multiplication result of the weight matrix of the Mth layer (that is, the output layer) and the input of the Mth layer is summed with the bias vector of the Mth layer to obtain a fifth summation result; a nonlinear activation function (such as a Sigmoid function) corresponding to the test task is determined; a nonlinear activation process is performed on the fifth summation result based on the nonlinear activation function to obtain a test result.

In step 103, back propagation processing is performed based on the difference between the test result and the labeled result of the training sample to update the feature information mining model and the controller model.

In some embodiments, step 103 can be implemented in the following manner: taking the loss of the feature information mining model and the controller model as factors, and based on different optimization objectives, respectively constructing the first objective function and the second objective function, wherein the optimization objective of the first objective function can be to directly minimize the loss function, and the optimization objective of the second objective function can be to minimize the loss function by using a function that seeks a set of functions (e.g., an arg function) (i.e., the optimization objective of the first objective function and the optimization objective of the second objective function are different). Subsequently, in each round of training, the difference between the test result and the labeled result of the training sample can be used as an error signal, substituted into any one of the first objective function and the second objective function, and combined with the gradient descent algorithm, gradient back propagation is performed in the feature information mining model and the controller model in turn, thereby updating the parameters of the feature information mining model and the weights of the controller model in turn.

In other embodiments, referring to FIG. 5A , FIG. 5A is a flow chart of a feature selection method provided in an embodiment of the present application. As shown in FIG. 5A , step 103 shown in FIG. 4 can be implemented through steps 1031 to 1032 shown in FIG. 5A , which will be described in conjunction with the steps shown in FIG. 5A .

In step 1031, a plurality of different training samples are divided into a training set and a validation set.

In some embodiments, multiple different training samples can be divided into training sets and validation sets according to a set ratio (for example, 8:2), wherein the training set is used to update the parameters of the feature information mining model, and the validation set is used to update the parameters (i.e., weights) of the controller model. In this way, by using different batches of data to respectively train the feature information mining model and the controller model, the occurrence of overfitting problems can be avoided.

In step 1032, the test results corresponding to the training set and the labeled results of the training set are substituted into the first objective function to determine the corresponding first difference, determine the first gradient of the feature information mining model according to the first difference, and update the parameters of the feature information mining model according to the first gradient.

In some embodiments, the loss function used by the first objective function may be a binary cross entropy (BCE) loss function. The above-mentioned step of substituting the test result corresponding to the training set and the labeled result of the training set into the first objective function to determine the corresponding first difference may be implemented in the following manner: the following processing is performed by the binary cross entropy loss function: the labeled result of the training set (i.e., the true value, such as y1) and the test result corresponding to the training set (i.e., the predicted value output by the feature information mining model, such as ) are multiplied to obtain a fifth multiplication result (e.g. ); determine the first difference between 1 and the labeled result of the training set (e.g. (1-y1)), and the logarithm result of the second difference between 1 and the test result corresponding to the training set (e.g. ), and multiply the logarithmic results of the first difference and the second difference to obtain a sixth multiplication result (for example ); The sum of the fifth multiplication result and the sixth multiplication result is determined as the first difference.

It should be noted that since the embodiment of the present application is mainly aimed at the binary classification problem, the BCE loss function is used here; it can be understood that other loss functions can also be used for other categories of recommendation tasks, such as hinge loss function, mean square error loss function, etc., and the embodiment of the present application does not make specific limitations on this.

In step 1033, the test results corresponding to the validation set and the labeled results of the validation set are substituted into the second objective function to determine the corresponding second difference, the second gradient of the controller model is determined according to the second difference, and the weight of the controller model is updated according to the second gradient.

In some embodiments, the weight corresponding to each feature includes a first weight and a second weight, the first weight is used to characterize the probability that the feature selection model selects the feature, and the second weight is used to characterize the probability that the feature selection model discards the feature; wherein, when the controller model is initialized, the first weight and the second weight corresponding to each feature have the same value (for example, taking the embedded feature _e2 as an example, when the controller model is initialized, the feature selection model reads the first weight corresponding to the embedded feature _e2 from the controller model and the second weight The values of are the same, both are 0.5, that is, when the controller model is initialized, the probability of selecting the embedded feature e ₂ is the same as the probability of discarding the embedded feature e ₂ ); when the controller model is updated, when the feature is a valid feature, the first weight corresponding to the feature is greater than the second weight (for example, still taking the embedded feature e ₂ as an example, when the controller model is updated and the embedded feature e ₂ is a valid feature, the first weight corresponding to the embedded feature e ₂ is greater than the second weight The value will be greater than the second weight For example, assuming that the controller model is trained in the search phase, the first weight The value of is updated from 0.5 to 0.9, and the second weight The value of is updated from 0.5 to 0.2, that is, for a valid feature, when the controller model is updated, the probability of selecting the feature will be greater than the probability of discarding the feature. Therefore, in the retraining stage, the feature selection model can select the feature from multiple features according to the updated weight); when the feature is an invalid feature, the first weight corresponding to the feature is less than the second weight (for example, still taking the embedded feature _e2 as an example, when the controller model is updated and the embedded feature _e2 is an invalid feature, the first weight corresponding to the embedded feature _e2 is The value will be less than the second weight For example, assuming that the controller model is trained in the search phase, the first weight The value of is updated from 0.5 to 0.3, and the second weight The value of is updated from 0.5 to 0.8, that is, for invalid features, when the controller model is updated, the probability of selecting the feature will be less than the probability of discarding the feature. Therefore, in the retraining stage, the feature selection model can discard the feature from multiple features according to the updated weight).

In some other embodiments, the loss function used by the second objective function may also be a binary cross entropy loss function. The above-mentioned substitution of the test results corresponding to the validation set and the labeled results of the validation set into the second objective function to determine the corresponding second difference may be achieved in the following manner: the following processing is performed by the binary cross entropy loss function: the labeled results of the validation set (i.e., the true value, such as y2) and the test results corresponding to the validation set (i.e., the predicted value output by the feature information mining model, such as ) and multiply the logarithmic result to obtain the seventh multiplication result (for example ); determine the third difference between 1 and the labeled result of the validation set (e.g., (1-y2)), and the logarithm result of the fourth difference between 1 and the test result corresponding to the validation set (e.g., ), and multiply the logarithmic results of the third difference and the fourth difference to obtain an eighth multiplication result (for example ); The sum of the seventh multiplication result and the eighth multiplication result is determined as the second difference.

It should be noted that since the embodiment of the present application is mainly aimed at the binary classification problem, the BCE loss function is used. It can be understood that other types of loss functions can also be used for other types of recommendation tasks or classification tasks, such as mean square error loss function, hinge loss function, cross entropy loss function, etc. The embodiment of the present application does not make specific limitations on this.

In step 104, the feature selection model is called to read the updated weights from the controller model, and some features are selected from multiple features based on the updated weights to combine and obtain combined features.

Here, the combined features are used for the updated feature information mining model to perform prediction tasks, where the prediction tasks can be recommendation tasks, such as recommendation tasks for news, movies, music and other information; they can also be classification tasks, such as text classification tasks (such as spam identification), image classification tasks (such as face recognition), etc.

It should be noted that the test tasks and prediction tasks involved in the embodiments of the present application are for the training phase/application phase of the feature information mining model, but the goals of the test tasks and prediction tasks are the same (for example, both are to predict the click rate of users for a certain item, or to determine whether a certain email is spam). Taking the feature information mining model as a classification model as an example, the classification task performed by the feature information mining model in the training phase is called a test task, and the classification task performed in the application phase after the feature information mining model is trained is called a prediction task. In other words, in these two phases, the features used by the feature information mining model when performing the classification task are different. In the training phase, the feature information mining model performs the classification task based on each feature fused with a weight; and in the application phase, the feature information mining model performs the classification task based on the combined features (i.e., a subset of multiple features) determined from multiple features.

In some embodiments, referring to FIG. 5B , FIG. 5B is a flow chart of a feature selection method provided in an embodiment of the present application. As shown in FIG. 5B , step 104 shown in FIG. 4 can be implemented through steps 1041 to 1042 shown in FIG. 5B , which will be described in conjunction with the steps shown in FIG. 5B .

In step 1041, the feature selection model is called to read the updated weights from the controller model, and based on the updated weights, the scores corresponding to the plurality of features are determined.

In some embodiments, the updated weight corresponding to each feature includes an updated first weight and an updated second weight, wherein the updated first weight is used to characterize the probability that the feature selection model selects the feature, and the updated second weight is used to characterize the probability that the feature selection model discards the feature. The above-mentioned updated weight-based determination of scores corresponding to the plurality of features can be implemented in the following manner: For each feature, the following processing is performed: the exponential result of the updated first weight corresponding to the feature (for example in, represents the updated first weight corresponding to the nth embedded feature e _n . For example, assuming ), the updated second weight corresponding to the feature (e.g. Assumptions ) is summed to obtain the sixth summation result (e.g. ); Exponentially update the first weight corresponding to the feature (e.g. ) and the sixth summation result are divided to determine the score corresponding to the feature (for example, π _n , i.e., the score corresponding to the nth embedded feature _en ).

In step 1042, at least some of the features having scores greater than a score threshold are selected from the multiple features and combined to obtain a combined feature.

In some embodiments, taking the multiple features as N features as an example, after obtaining the scores corresponding to the N features respectively, the N features can be sorted in descending order from high to low according to the scores, and the K features ranked in the top K positions in the descending sorting results are screened out and combined to obtain combined features, where K and N are both positive integers, and K＜N.

For example, take the multiple features of the training sample as 4 features, namely {e ₁ , e ₂ , e ₃ , e ₄ }. At the same time, assuming that the score corresponding to feature e ₁ is 80, the score corresponding to feature e ₂ is 60, the score corresponding to e ₃ is 90, and the score corresponding to e ₄ is 70, then the sorting result of sorting these 4 features in descending order from high to low is: e ₃ , e ₁ , e ₄ , e _2. Then, the top 2 features (i.e., features e ₃ and e ₁ ) can be selected from the descending sorting result and combined to obtain the combined feature.

In other embodiments, the number of features included in the combined feature is less than the number of multiple features, and the feature information mining model is constructed based on multiple features; then, after some features are screened out from the multiple features based on the updated weights for combination to obtain the combined feature, the following processing can also be performed: the size of the input layer of the feature information mining model (for example, the feature information mining model after the difference update) is adjusted from the size corresponding to the number of multiple features to the size corresponding to the number of features included in the combined feature to obtain a new feature information mining model.

For example, take the case where the multiple features are N features, that is, the feature information mining model is constructed based on N features; then after at least some features are screened out from the N features based on the updated weights (for example, K features are screened out from the N features, where K is less than N) and combined to obtain the combined features, the following processing can also be performed: the size of the input layer of the feature information mining model is adjusted from the size corresponding to the number of N features to the size corresponding to the number of features included in the combined features (that is, K) (for example, the input size of the first MLP layer included in the feature information mining model is adjusted from N×d to K×d, where d is the dimension of the feature), to obtain a new feature information mining model.

The feature selection method provided in the embodiment of the present application is that the feature selection model first determines the weighted features based on the weights of multiple features of the training samples in the controller model, and then the feature information mining model performs the test task based on the weighted features to optimize the weights in the controller model, and then screens out some of the best features for combination for the updated feature information mining model to perform subsequent prediction tasks. Compared with the existing technology, it avoids the large demand for expert knowledge and manpower, can adapt to training sets of different sizes, reduce computing requirements, and can screen out the best combination of features for the feature information mining model to perform prediction tasks, thereby ensuring prediction accuracy.

Below, the exemplary application of the embodiments of the present application in the recommendation scenario is explained by taking the content recommendation scenario as an example.

The quality of features can significantly affect the performance of the feature information mining model used by the recommendation system. Therefore, in the feature information mining model designed based on deep learning technology, feature selection is a very important module. The embodiment of the present application provides an automated machine learning framework that can automatically and adaptively select key features for the feature information mining model. Specifically, the embodiment of the present application uses the relevant technology of neural network search to design a differentiable controller network (corresponding to the above-mentioned controller model), combined with the other two models provided by the embodiment of the present application, namely the feature selection model and the feature information mining model, which can automatically adjust the probability of a specific feature being selected; after that, the embodiment of the present application also provides a specific method for selecting features according to the probability given by the controller network; finally, only the selected features are input into the corresponding feature information mining model for retraining. At the same time, the feature selection method provided in the embodiment of the present application has good generalization and can be applied to different recommendation systems.

For example, the feature selection method provided in the embodiment of the present application can be applied to various recommendation systems, and more generally, it can also be applied to classification problems. Specifically, it is only necessary to process the data to be used into a tabular form, which can be used as the input of the feature selection method provided in the embodiment of the present application, so that the selected features can be given for combination. Finally, it is only necessary to input the combined features obtained by combining the selected features into the recommendation system or classification system to be used, and a good recommendation effect or classification result can be obtained.

The feature selection method provided in the embodiment of the present application is described in detail below.

For example, see Figure 6, which is a schematic diagram of the principle of the feature selection method provided by the embodiment of the present application. As shown in Figure 6, the feature selection method provided by the embodiment of the present application consists of two stages: a search stage (Search Stage) and a retraining stage (Retraining Stage). In order to find the optimal features for combination, the embodiment of the present application updates the parameters of the controller network (Controller) (that is, the weights corresponding to each feature) in the search stage; thereafter, in the retraining stage, according to the parameters finally obtained by the controller network in the previous stage, the embodiment of the present application selects the optimal features for combination, and inputs the combined features obtained by combining the selected features into the recommendation system to be used. As can be seen from Figure 6, both stages are composed of three modules: a controller network, a feature selection model (Feature Selection Modle), and a feature information mining model (Feature Utilizing Model).

In the search phase, the embodiment of the present application first initializes the parameters of the controller network (for example, the probability of all features being selected is set to 0.5). After that, all the features to be selected (Input Fields) are input into the feature selection model, so that the feature selection model gives each feature a weight according to the parameters of the controller network, and all features are multiplied by the corresponding weights and input into the feature information mining model. The feature information mining model uses these weighted features to predict user preferences (corresponding to the above-mentioned test tasks, such as predicting the user's click rate for a certain item). The embodiment of the present application updates the parameters of the feature information mining model by predicting the preferences of users in the training set; at the same time, the parameters of the controller network are updated according to the prediction results of the users in the verification set. At the end of this stage, the parameters in the controller network are fully trained.

After the search results are completed, the retraining phase begins. Specifically, at the beginning of the retraining phase, the feature selection model selects the best features from all the features to be selected for combination based on the parameters of the controller network that has been fully trained in the search phase, for example, the best features _e1 and _e4 are selected from all the features to be selected { _e1 , _e2 , _e3 , _e4 }. It should be noted that the role of the feature selection model in the two phases is different (specifically, in the search phase, the feature selection model will give each feature an initial weight, and this selection will not really filter out the features; while in the retraining phase, the feature selection model will filter out bad features from the features to be selected based on the updated weights). Subsequently, the feature information mining model is adapted and optimized based on the selected feature combination (for example, the selected features _e1 and _e4 ).

First, the controller network shown in FIG. 6 will be described below.

For example, assuming that there are a total of N different features, there are a total of 2^N for all possible feature combinations. If all feature combinations are directly represented as search space using the method of encoding, then as the number of features to be selected increases, the search space will tend to infinity. In view of this, as shown in Figure 7A, the embodiment of the present application defines the search space as a directed complete graph, using each feature as a node in the graph, and using the result of feature selection as an output node; There are two edges between each node and the output node, representing the selection and discarding of the features represented by the node (for example, a solid line represents selection, and a dotted line represents discarding). Under such a design, assuming that there are a total of N different features, the embodiment of the present application only needs to determine the state of the edge between 2N nodes and the output node, which significantly reduces the search space.

In the search phase, each edge is first initialized with a uniform weight (for example, for feature 1, its corresponding weight is and The value of is 0.5, among which, represents the probability of discarding feature 1, Represents the probability of selecting feature 1), these weights will be input into the feature selection model to determine the selection effect of the feature selection model. As shown in Figure 7B, as the search proceeds, the uniform initial weights gradually become unbalanced (for example, the thickness of the line can be used to represent the numerical value of the corresponding weight, the thicker the line, the larger the numerical value of the corresponding weight), the weight of the edge representing the selection of valid features gradually increases, and the weight of the edge representing the selection of invalid features gradually decreases. Finally, as shown in Figure 7C, in the retraining stage, only the thickest of the two edges of each node will be retained, thereby completing the feature selection task.

The feature information mining model shown in FIG. 6 will be described below.

For example, see Figure 8, which is a structural diagram of the feature information mining model provided by an embodiment of the present application. As shown in Figure 8, the feature information mining model consists of two basic units, namely an embedding unit (Embedding Layer) and a perceptron unit (MLP Layers). Among them, the embedding unit is used to embed discrete inputs into a low-dimensional continuous space. First, the original input features are binarized (binarization), that is, only 0 and 1 are used to represent them. Specifically, assuming that a feature has M different values, M vectors of length M, each containing only 1 "1" and the rest are "0", can be used to represent each possible value (that is, the feature is uniquely encoded, so that each feature value corresponds to a one-dimensional feature, and the unique hot encoding obtains a sparse feature matrix). Taking the feature "gender" as an example, assuming that the input is "male" or "female", after binarization, (1, 0) can be used to represent "male" and (0, 1) can be used to represent "female". Assuming that there are N features in total, the N features after binarization can be expressed as:

x＝[x ₁ ,x ₂ ,…,x _N ]

Among them, _xn is the result of binary processing corresponding to the nth feature, and Among them, _Dn is the number of different values of the corresponding feature (i.e., M in the above text). Afterwards, since such features are very sparse and some features have too many values, resulting in a very large dimension, which is not conducive to the subsequent use of features, it is also necessary to project the binary processed features into a low-dimensional continuous space:

e _n =A _n x _n

Where _en is the result of the projection of the nth binary feature (i.e., the nth embedded feature), and is the parameter to be learned, and d is determined by the user (different features can use the same value). Finally, N features can be obtained after embedding:

E＝[e ₁ ,e ₂ ,…,e _N ]

Wherein, _en is the nth embedded feature. After obtaining the embedded N features, the embedded N features can also be concatenated (Concatenate), and the concatenated features obtained after the concatenation are input into the perceptron unit.

In some embodiments, the perceptron unit may be a multi-layer perceptron (MLP), which is composed of multiple layers of linear transformation (Linear Transformation) and nonlinear activation functions (such as Sigmoid). The perceptron unit is used to mine information in the splicing features and give a prediction result. The specific formula is as follows:

h ₁ ＝E,

h _m+1 =ReLU(W _m h _m +b _m ),1≤m≤M-1,

Where E is the output of the embedding unit mentioned above, _Wm , _bm , and _hm represent the linear transformation matrix (also known as the weight matrix), bias vector, and input of the mth layer, respectively, hm ₊₁ represents the output of the mth layer (as the input of the m+1th layer), and ReLU represents a linear activation function, such as a maximum value function. M is the number of layers included in the multilayer perceptron, _WM , _bM represent the linear transformation matrix and bias vector of the output layer (i.e., the Mth layer), respectively, and σ(·) represents the nonlinear activation function of the output layer (e.g., Sigmoid function), and its type depends on the recommendation task.

The feature selection model shown in FIG6 is further described below.

In some embodiments, the function of the feature selection model is to apply the weights given by the controller network to the embedded features E:

E′＝[e′ ₁ ,e′ ₂ ,…,e′ _N ]

Among them, _en is the result of embedding the nth feature mentioned above, and _e′n is the corresponding feature after the feature selection model is applied. is the weight given by the controller network Got it.

For example, the feature selection model can be based on the weights given by the controller network get In order to The value of is more marginalized (i.e., one is close to 0 and the other is close to 1), while maintaining the end-to-end differentiability of the network structure. In the embodiment of the present application, a normalized exponential function (such as the Gumbel-Softmax function) can be used to obtain The details are as follows:

g _j = -log(-log(u _j ))

u _j ～Uinform(0,1)

in, represents the probability of selecting (j=1) or discarding (j=0) the nth feature, _gj represents independent and identically distributed gumbel noise, the value range of j is 1 or 0, τ represents the annealing parameter (also known as temperature parameter), and Uinform(0,1) represents generating a real number between 0 and 1 according to the uniform distribution.

The optimization algorithm provided in the embodiments of the present application is described below.

In some embodiments, the Newton gradient descent method may be used to alternately update the parameters of the feature information mining model and the parameters of the controller network.

For example, in the search phase, the parameters that need to be optimized are the parameters of the controller network And the parameter W of the feature information mining model. Since the parameters of these two parts are highly correlated, if a batch of data is used to train the parameters of these two parts at the same time, it will lead to serious overfitting problems. To this end, the embodiment of the present application solves the overfitting problem by designing the following two objective functions:

in, Corresponding to the first objective function mentioned above, Corresponding to the second objective function mentioned above; W represents the parameter of the feature information mining model, are the parameters (i.e. weights) of the controller network, W ^* represents the optimal parameters under the corresponding constraints, Respectively represent the loss on the training set and the validation set, and the training set and the validation set can be randomly divided before the algorithm runs. The loss function used by the two objective functions can be the binary cross entropy (BCE) loss function. Specifically, the loss function can be defined as:

Among them, y is the true value, is the model's predicted value.

However, at each step of optimization, in order to obtain It is inevitable to train the feature information mining model to convergence, and the amount of calculation required in this case cannot be met. Therefore, in the embodiment of the present application, a first-order approximation can be used to reduce the amount of calculation:

Right now

Therefore, the optimization algorithm finally used in the embodiment of the present application can be:

Among them, δ _W , Are the learning rates corresponding to the two parts of the parameters, and the updates of these two parts of the parameters are performed alternately at a certain update frequency.

The following is a description of the retraining process.

In some embodiments, after training in the search phase, the parameters in the controller network are It has been well trained and can give accurate feature selection results based on it.

For example, in the retraining phase, the feature selection model can first be based on the parameters of the controller network trained Select the best partial features (for example, K features) from all the features to be selected. In this stage, the feature selection model can use the Softmax algorithm to calculate the corresponding score of each feature:

After obtaining the score corresponding to each feature, the K features with the highest scores can be selected as the final feature combination.

In addition, after selecting the features, the embodiment of the present application can also re-establish a new feature information mining model based on the number K of selected features (because the original feature information mining model is designed for N features). For example, the input size of the first layer included in the feature information mining model can be adjusted from N×d to K×d, thereby obtaining a new feature information mining model.

It should be noted that, in addition to using the perceptron unit mentioned above, the embodiment of the present application can also replace this module with a recommendation system of other structures, such as a factorization machine (FM). In other words, the feature selection result given by the embodiment of the present application has good generalization.

The beneficial effects of the feature selection method provided in the embodiment of the present application will be further described below in combination with experimental data.

In order to evaluate the effectiveness of the feature selection method provided in the embodiment of the present application, the embodiment of the present application conducted a wide range of comparative experiments based on different feature selection methods, wherein the feature selection methods involved in the comparison include: principal component analysis (PCA, Principal Component Analysis), regression model (LASSO, Least Absolute Shrinkageand Selection Operator), gradient boosting regression (GBR, Gradient Boosting Regression), gradient boosting decision tree (GBDT, Gradient Boosting Decision Tree) and IDARTS, wherein the feature selection method provided in the embodiment of the present application discarded 11 features from the Avazu data set and 6 features from the Criteo data set. In order to ensure the fairness of the experiment, other feature selection methods also discarded the same number or fewer invalid features from the two data sets. The specific results can be seen in Table 1.

It can be seen from Table 1 that the feature selection method provided in the embodiment of the present application can achieve good performance while discarding more features (for example, 50% of the features are discarded in the Avazu data set). For example, in the Avazu data set, the AUC (i.e., the area under the receiver operating characteristic curve (ROC), which can quantitatively reflect the model performance measured based on the ROC curve. The larger the AUC, the better the effect of the model) corresponding to the feature selection method provided in the embodiment of the present application is 0.7773, which is greater than the AUC corresponding to other feature selection methods. In addition, the Logloss (an indicator used to evaluate whether it is accurate, the smaller the value, the better) corresponding to the feature selection method provided in the embodiment of the present application is also smaller than the Logloss corresponding to other feature selection methods.

Table 1 Overall performance comparison table

In addition, in order to further study the transferability of the feature selection results given in the embodiments of the present application, the embodiments of the present application also apply the selected features to six advanced deep recommendation models, including the Factorization Machine (FM) on the Avazu dataset, DeepFM (an algorithm derived from FM, combining Deep with FM, using FM for low-order combinations between features, and using the Deep NN part for high-order combinations between features, combining the two methods in parallel), xDeepFM (extended by Deep and Cross, which inherits the characteristics of Deep and Cross that can control the interaction of high-order features), IPNN, Wide&Deep (WD) and DeepCrossNet (DCN). For specific results, see Table 2. In Table 2, the first row of each model is the performance corresponding to all the features to be selected (including AUC, Logloss, average inference time (Infer)), and the second row is the performance corresponding to the features selected by the feature selection method provided in the embodiments of the present application.

As can be seen from Table 2, for all the above models, the feature selection results given in the embodiments of the present application can not only improve the performance of the model, but also reduce the time of reasoning, thereby improving the efficiency of online reasoning. In other words, the feature selection method provided in the embodiments of the present application has good generalization and can be applied to various recommendation systems.

Table 2 Transferability analysis on the Avazu dataset

The following continues to describe an exemplary structure of the feature selection device 243 provided in an embodiment of the present application implemented as a software module. In some embodiments, as shown in FIG. 2 , the software module stored in the feature selection device 243 in the memory 240 may include: a calling module 2431, a fusion module 2432, an updating module 2433 and a combining module 2434.

The calling module 2431 is used to call the feature selection model to read the weights corresponding to multiple features of the training sample from the controller model; the fusion module 2432 is used to fuse the weights to each feature; the calling module 2431 is also used to call the feature information mining model to perform the test task and obtain the test result based on each feature fused with the weight; the updating module 2433 is used to perform back propagation processing based on the difference between the test result and the marking result of the training sample to update the feature information mining model and the controller model; the calling module 2431 is also used to call the feature selection model to read the updated weights from the controller model; the combining module 2434 is used to select some features from multiple features based on the updated weights for combination to obtain the combined feature; wherein the combined feature is used for the updated feature information mining model to perform the prediction task.

In some embodiments, the feature search space of the controller model is a directed complete graph, and the directed complete graph includes: multiple input nodes corresponding one-to-one to multiple features, and an output node representing the feature selection results; wherein there are two edges between each input node and the output node, respectively representing the weights of the features corresponding to the input nodes, wherein the weights include a first weight for representing the probability of the feature selection model selecting the feature, and a second weight for representing the probability of the feature selection model discarding the feature.

In some embodiments, the weight corresponding to each feature includes a first weight and a second weight, the first weight is used to characterize the probability of the feature selection model selecting the feature, and the second weight is used to characterize the probability of the feature selection model discarding the feature; the feature selection device 243 also includes a determination module 2435, which is also used to perform the following processing for each feature: binarizing the features whose initial format is a discrete format to obtain binarized features, embedding the binarized features to obtain embedded features, wherein the dimension of the feature space of the embedded features is smaller than the dimension of the feature space of the binarized features; determining a first multiplication result of the first weight corresponding to the embedded feature and the first coefficient, and a second multiplication result of the second weight corresponding to the embedded feature and the second coefficient, and summing the first multiplication result and the second multiplication result to obtain a first summation result; determining the first summation result and the multiplication result of the embedded feature as a feature fused with weights.

In some embodiments, the weight corresponding to each feature includes a first weight and a second weight, the first weight is used to characterize the probability of the feature selection model selecting the feature, and the second weight is used to characterize the probability of the feature selection model discarding the feature; the determination module 2435 is also used to perform the following processing for each feature: binarize the features whose initial format is a discrete format to obtain binarized features, embed the binarized features to obtain embedded features, wherein the dimension of the feature space of the embedded features is smaller than the dimension of the feature space of the binarized features; determine the first intermediate parameter and the second intermediate parameter based on the first weight and the second weight corresponding to the embedded features; determine the third multiplication result of the first intermediate parameter and the first coefficient, and the fourth multiplication result of the second intermediate parameter and the second coefficient, and sum the third multiplication result and the fourth multiplication result to obtain a second summation result; determine the second summation result and the multiplication result of the embedded features as a weighted feature.

In some embodiments, the determination module 2435 is also used to call a normalized exponential function based on the first weight and the second weight corresponding to the embedded features to obtain a first intermediate parameter and a second intermediate parameter; wherein the normalized exponential function satisfies the following conditions: marginalizing the values of the first intermediate parameter and the second intermediate parameter; and maintaining the differentiability of the controller model.

In some embodiments, the determination module 2435 is also used to call the normalized exponential function to perform the following processing based on the first weight and the second weight corresponding to the embedded feature: divide the sum of the logarithm result of the first weight and the first noise coefficient by the annealing parameter to obtain a first division result; divide the sum of the logarithm result of the second weight and the second noise coefficient by the annealing parameter to obtain a second division result; sum the exponential result of the first division result and the exponential result of the second division result to obtain a third summation result; determine the division result of the exponential result of the first division result and the third summation result as the first intermediate parameter; determine the division result of the exponential result of the second division result and the third summation result as the second intermediate parameter.

In some embodiments, the feature information mining model includes an embedding unit and a perceptron unit; the calling module 4651 is also used to call the perceptron unit to perform a test task based on each embedded feature fused with a weight, and obtain a test result; wherein the embedded feature is obtained by calling the embedding unit to perform the following processing on the feature whose initial format is a discrete format: binarizing the feature whose initial format is a discrete format to obtain the binarized feature, embedding the binarized feature to obtain the embedded feature, wherein the dimension of the feature space of the embedded feature is smaller than the dimension of the feature space of the binarized feature.

In some embodiments, the perceptron unit is a multilayer perceptron, which includes multiple layers, each layer including a corresponding weight matrix and a bias vector; calling module 2431 is also used to use each embedded feature fused with weights as the input of the first layer of the multilayer perceptron, and iterate m to perform the following processing: summing the multiplication result of the weight matrix of the mth layer and the input of the mth layer with the bias vector of the mth layer to obtain a fourth summation result; performing linear transformation processing on the fourth summation result to obtain the output of the mth layer, and using the output of the mth layer as the input of the m+1th layer; wherein the value of m gradually increases and satisfies 1≤m≤M-1, and M is the total number of multiple layers; summing the multiplication result of the weight matrix of the Mth layer and the input of the Mth layer with the bias vector of the Mth layer to obtain a fifth summation result; determining a nonlinear activation function corresponding to the test task; performing nonlinear activation processing on the fifth summation result based on the nonlinear activation function to obtain a test result.

In some embodiments, the feature selection device 243 also includes a division module 2436, which is used to divide multiple different training samples into a training set and a validation set; the update module 2433 is also used to substitute the test results corresponding to the training set and the labeled results of the training set into the first objective function to determine the corresponding first difference, determine the first gradient of the feature information mining model according to the first difference, and update the parameters of the feature information mining model according to the first gradient; and to substitute the test results corresponding to the validation set and the labeled results of the validation set into the second objective function to determine the corresponding second difference, determine the second gradient of the controller model according to the second difference, and update the weight of the controller model according to the second gradient; wherein the first objective function uses the loss of the controller model and the feature information mining model as a factor, the second objective function uses the loss of the controller model and the feature information mining model as a factor, and the optimization target of the first objective function is different from the optimization target of the second objective function.

In some embodiments, the weight corresponding to each feature includes a first weight and a second weight, the first weight is used to characterize the probability of the feature selection model selecting the feature, and the second weight is used to characterize the probability of the feature selection model discarding the feature; wherein, when the controller model is initialized, the first weight and the second weight corresponding to the feature have the same value; when the controller model is updated, when the feature is a valid feature, the first weight is greater than the second weight; when the feature is an invalid feature, the first weight is less than the second weight.

In some embodiments, the loss functions used by the first objective function and the second objective function are both binary cross entropy loss functions; the determination module 2435 is further used to perform the following processing through the binary cross entropy loss function: multiplying the labeled result of the training set by the logarithm result of the test result corresponding to the training set to obtain a fifth multiplication result; determining the first difference between 1 and the labeled result of the training set, and the logarithm result of the second difference between 1 and the test result corresponding to the training set, and multiplying the logarithm result of the first difference by the second difference to obtain a sixth multiplication result; multiplying the fifth The sum of the multiplication result and the sixth multiplication result is determined as the first difference; and is used to perform the following processing through the binary cross entropy loss function: multiply the logarithm result of the labeled result of the validation set by the test result corresponding to the validation set to obtain a seventh multiplication result; determine the third difference between 1 and the labeled result of the validation set, and the logarithm result of the fourth difference between 1 and the test result corresponding to the validation set, and multiply the logarithm result of the third difference by the fourth difference to obtain an eighth multiplication result; and determine the sum of the seventh multiplication result and the eighth multiplication result as the second difference.

In some embodiments, the calling module 2431 is also used to call the feature selection model to read the updated weights from the controller model; the determining module 2435 is also used to determine the scores corresponding to multiple features based on the updated weights; the combining module 2434 is also used to filter out at least some features whose scores are greater than the score threshold from the multiple features for combination to obtain combined features.

In some embodiments, the updated weight corresponding to each feature includes an updated first weight and an updated second weight, wherein the updated first weight is used to characterize the probability of the feature selection model selecting the feature, and the updated second weight is used to characterize the probability of the feature selection model discarding the feature; the determination module 2435 is also used to perform the following processing for each feature: summing the exponential result of the updated first weight corresponding to the feature with the exponential result of the updated second weight corresponding to the feature to obtain a sixth summation result; and determining the result of dividing the exponential result of the updated first weight corresponding to the feature by the sixth summation result as the score corresponding to the feature.

In some embodiments, the number of features included in the combined feature is less than the number of multiple features, and the feature information mining model is constructed based on multiple features; the feature selection device 243 also includes an adjustment module 2437, which is used to adjust the size of the input layer of the feature information mining model from the size corresponding to the number of multiple features to the size corresponding to the number of features included in the combined feature, so as to obtain a new feature information mining model.

It should be noted that the description of the device of the embodiment of the present application is similar to the description of the above-mentioned method embodiment, and has similar beneficial effects as the method embodiment, so it is not repeated. The unfinished technical details of the feature selection device provided in the embodiment of the present application can be understood according to the description of any of Figures 4, 5A, or 5B.

The embodiment of the present application provides a computer program product or a computer program, which includes computer instructions (i.e., executable instructions), which are stored in a computer-readable storage medium. A processor of an electronic device reads the computer instructions from the computer-readable storage medium, and the processor executes the computer instructions, so that the electronic device performs the feature selection method described in the embodiment of the present application.

An embodiment of the present application provides a computer-readable storage medium storing executable instructions, wherein executable instructions are stored. When the executable instructions are executed by a processor, the processor will execute the method provided by the embodiment of the present application, for example, the feature selection method shown in any of Figures 4, 5A, or 5B.

In some embodiments, the computer-readable storage medium may be a memory such as FRAM, ROM, PROM, EPROM, EEPROM, flash memory, magnetic surface storage, optical disk, or CD-ROM; or it may be various devices including one or any combination of the above memories.

In some embodiments, executable instructions may be in the form of a program, software, software module, script or code, written in any form of programming language (including compiled or interpreted languages, or declarative or procedural languages), and may be deployed in any form, including as a stand-alone program or as a module, component, subroutine or other unit suitable for use in a computing environment.

As an example, executable instructions may, but need not, correspond to a file in a file system, may be stored as part of a file that stores other programs or data, such as in one or more scripts in a HyperText Markup Language (HTML) document, in a single file dedicated to the program in question, or in multiple coordinated files (e.g., files storing one or more modules, subroutines, or code portions).

As an example, the executable instructions may be deployed to be executed on one electronic device, or on multiple electronic devices located at one site, or on multiple electronic devices distributed at multiple sites and interconnected by a communication network.

The above is only an embodiment of the present application and is not intended to limit the protection scope of the present application. Any modifications, equivalent substitutions and improvements made within the spirit and scope of the present application are included in the protection scope of the present application.

Claims (18)

1. A feature selection method, characterized in that the method includes: Call the feature selection model to read the weights corresponding to multiple features of the training sample from the controller model, and fuse the weights to each of the features; Based on each of the features fused with the weight, call the feature information mining model to perform the test task and obtain the test results; Perform backpropagation processing based on the difference between the test results and the labeling results of the training samples to update the feature information mining model and the controller model; Call the feature selection model to read the updated weights from the controller model, select some features from the multiple features based on the updated weights and combine them to obtain combined features; Wherein, the combined features are used for the updated feature information mining model to perform prediction tasks. 2. The method according to claim 1, characterized in that, The feature search space of the controller model is a directed complete graph, and the directed complete graph includes: multiple input nodes corresponding to the multiple features one-to-one, and output nodes representing the feature selection results; wherein, each There are two edges between each of the input nodes and the output node, respectively representing the weights of the features corresponding to the input nodes, and the weights include the weights used to represent the features selected by the feature selection model. a first weight of probability, and a second weight characterizing the probability that the feature selection model discards the feature. 3. The method according to claim 1, characterized in that, The weight corresponding to each feature includes a first weight and a second weight. The first weight is used to characterize the probability that the feature selection model selects the feature. The second weight is used to characterize the feature. The probability that the selection model discards said feature; The corresponding fusion of the weights to each of the features includes: The following processing is performed for each described feature: The features whose initial format is a discrete format are binarized to obtain the binarized features, and the binarized features are embedding to obtain the embedded features, where embedding The dimension of the feature space of the feature after binaryization is smaller than the dimension of the feature space of the feature after binaryization; Determining the first multiplication result of the first weight and the first coefficient corresponding to the embedded feature, and the second multiplication result of the second weight and the second coefficient corresponding to the embedded feature, and performing a summation process on the first multiplication result and the second multiplication result to obtain a first summation result; The multiplication result of the first summation result and the embedded feature is determined as the feature fused with the weight. 4. The method according to claim 1, characterized in that, The weight corresponding to each feature includes a first weight and a second weight. The first weight is used to characterize the probability that the feature selection model selects the feature. The second weight is used to characterize the feature. The probability that the selection model discards said feature; The corresponding fusion of the weights to each of the features includes: The following processing is performed for each described feature: The features whose initial format is a discrete format are binarized to obtain the binarized features, and the binarized features are embedding to obtain the embedded features, where embedding The dimension of the feature space of the feature after binaryization is smaller than the dimension of the feature space of the feature after binaryization; Based on the first weight and the second weight corresponding to the embedded feature, determine a first intermediate parameter and a second intermediate parameter; Determine the third multiplication result of the first intermediate parameter and the first coefficient, and the fourth multiplication result of the second intermediate parameter and the second coefficient, and compare the third multiplication result and the fourth The multiplication results are summed to obtain the second summation result; The multiplication result of the second summation result and the embedded feature is determined as the feature fused with the weight. 5. The method of claim 4, wherein determining the first intermediate parameter and the second intermediate parameter based on the first weight and the second weight corresponding to the embedded feature includes: : Based on the first weight and the second weight corresponding to the embedded feature, a normalized exponential function is called for processing to obtain the first intermediate parameter and the second intermediate parameter; Wherein, the normalized exponential function satisfies the following conditions: marginalizing the values of the first intermediate parameter and the second intermediate parameter; maintaining the differentiability of the controller model. 6. The method according to claim 5, characterized in that the first weight and the second weight corresponding to the embedded feature are processed by calling a normalized exponential function to obtain the first An intermediate parameter and the second intermediate parameter include: Based on the first weight and the second weight corresponding to the embedded feature, a normalized exponential function is called to perform the following processing: Divide the logarithmic result of the first weight, the summation result of the first noise coefficient and the annealing parameter to obtain a first division result; Divide the summation result of the logarithm of the second weight and the second noise coefficient with the annealing parameter to obtain a second division result; Perform a summation process on the exponential result of the first division result and the exponential result of the second division result to obtain a third summation result; Determine the division result of the exponential result of the first division result and the division result of the third summation result as the first intermediate parameter; The exponential result of the second division result and the division result of the third summation result are determined as the second intermediate parameters. 7. The method according to claim 1, characterized in that, The feature information mining model includes an embedding unit and a perceptron unit; Based on each of the features fused with the weight, the feature information mining model is called to perform the test task and the test results are obtained, including: Based on each embedded feature fused with the weight, call the perceptron unit to perform a test task and obtain a test result; The embedded features are obtained by calling the embedding unit to perform the following processing on the features whose initial format is a discrete format: The features whose initial format is a discrete format are binarized to obtain the binarized features, and the binarized features are embedding to obtain the embedded features, where embedding The dimension of the feature space of the feature after the feature is smaller than the dimension of the feature space of the feature after the binary feature. 8. The method according to claim 7, characterized in that, The perceptron unit is a multi-layer perceptron, and the multi-layer perceptron includes multiple layers, and each layer includes a corresponding weight matrix and bias vector; The perceptron unit is called to perform a test task based on each embedded feature fused with the weight, and the test results are obtained, including: Each embedded feature fused with the weight is used as the input of the first layer of the multi-layer perceptron, and the following processing is performed iteratively m: The multiplication result of the weight matrix of the mth layer and the input of the mth layer is summed with the bias vector of the mth layer to obtain the fourth summation result; Perform linear transformation processing on the fourth summation result to obtain the output of the m-th layer, and use the output of the m-th layer as the input of the m+1-th layer; Wherein, the value of m gradually increases and satisfies 1≤m≤M-1, and M is the total number of the multiple layers; The multiplication result of the weight matrix of the Mth layer and the input of the Mth layer is summed with the bias vector of the Mth layer to obtain a fifth summation result; Determine a nonlinear activation function corresponding to the test task; Perform nonlinear activation processing on the fifth summation result based on the nonlinear activation function to obtain the test result. 9. The method according to claim 1, characterized in that the back propagation process is performed based on the difference between the test result and the marking result of the training sample to update the feature information mining model and the control Server models include: Divide multiple different training samples into a training set and a verification set; Substitute the test results corresponding to the training set and the labeling results of the training set into the first objective function to determine the corresponding first difference, determine the first gradient of the feature information mining model based on the first difference, and Update parameters of the feature information mining model according to the first gradient; The test results corresponding to the verification set and the marking results of the verification set are substituted into the second objective function to determine the corresponding second difference, the second gradient of the controller model is determined according to the second difference, and according to the second gradient updates the weights of the controller model; Wherein, the first objective function takes the loss of the controller model and the feature information mining model as a factor, and the second objective function takes the loss of the controller model and the feature information mining model as a factor, And the optimization objective of the first objective function is different from the optimization objective of the second objective function. 10. The method of claim 9, wherein, The weight corresponding to each feature includes a first weight and a second weight. The first weight is used to characterize the probability that the feature selection model selects the feature. The second weight is used to characterize the feature. The probability that the selection model discards said feature; Wherein, when the controller model is initialized, the first weight and the second weight corresponding to the feature have the same value; after the controller model is updated, when the feature is a valid feature, The first weight is greater than the second weight; when the feature is an invalid feature, the first weight is less than the second weight. 11. The method according to claim 9, characterized in that the loss functions used by the first objective function and the second objective function are binary cross-entropy loss functions; Substituting the test results corresponding to the training set and the marking results of the training set into the first objective function to determine the corresponding first difference includes: The following processing is performed by the binary cross-entropy loss function: Multiply the logarithm of the labeling result of the training set and the test result corresponding to the training set to obtain a fifth multiplication result; Determine the first difference between 1 and the labeling result of the training set, and the logarithm result of the second difference between 1 and the test result corresponding to the training set, and compare the first difference Multiply the logarithm result of the second difference to obtain a sixth multiplication result; Determine the summation result of the fifth multiplication result and the sixth multiplication result as the first difference; Substituting the test results corresponding to the verification set and the marking results of the verification set into the second objective function to determine the corresponding second difference includes: The following processing is performed by the binary cross-entropy loss function: Multiply the logarithm of the marking results of the verification set and the test results corresponding to the verification set to obtain a seventh multiplication result; Determine the third difference between 1 and the marking result of the verification set, and the logarithm result of the fourth difference between 1 and the test result corresponding to the verification set, and compare the third difference Multiply the logarithm result of the fourth difference to obtain an eighth multiplication result; The summation result of the seventh multiplication result and the eighth multiplication result is determined as the second difference. 12. The method of claim 1, wherein the calling the feature selection model reads the updated weight from the controller model, and selects the updated weight from the plurality of features based on the updated weight. Select some features and combine them to obtain combined features, including: Call the feature selection model to read the updated weights from the controller model, and determine the scores corresponding to the multiple features based on the updated weights; At least some features with a score greater than a score threshold are selected from the plurality of features and combined to obtain a combined feature. 13. The method according to claim 12, characterized in that, The updated weight corresponding to each feature includes an updated first weight and an updated second weight, wherein the updated first weight is used to represent the probability of the feature selection model selecting the feature, so The updated second weight is used to represent the probability that the feature selection model discards the feature; Determining the scores corresponding to the multiple features based on the updated weights includes: The following processing is performed for each described feature: Perform a summation process on the exponential results of the updated first weight corresponding to the feature and the exponential results of the updated second weight corresponding to the feature to obtain a sixth summation result; The division result of the exponential result of the updated first weight corresponding to the feature and the sixth summation result is determined as the score corresponding to the feature. 14. The method according to claim 1, characterized in that, The number of features included in the combined feature is less than the number of the multiple features, and the feature information mining model is constructed based on the multiple features; After selecting some features from the multiple features based on the updated weights and combining them to obtain the combined features, the method further includes: Adjust the size of the input layer of the feature information mining model from a size corresponding to the number of the plurality of features to a size corresponding to the number of features included in the combined feature to obtain the new feature information mining model. . 15. A feature selection device, characterized in that the device includes: The calling module is used to call the feature selection model to read the weights corresponding to multiple features of the training sample from the controller model; A fusion module, used to fuse the weight corresponding to each of the features; The calling module is also used to call the feature information mining model to perform the test task based on each feature fused with the weight, and obtain test results; An update module, configured to perform backpropagation processing based on the difference between the test results and the marking results of the training samples to update the feature information mining model and the controller model; The calling module is also used to call the feature selection model to read the updated weights from the controller model; A combination module, configured to select some features from the multiple features based on the updated weights and combine them to obtain combined features; Wherein, the combined features are used for the updated feature information mining model to perform prediction tasks. 16. An electronic device, characterized in that the electronic device includes: Memory, used to store executable instructions; A processor configured to implement the feature selection method described in any one of claims 1 to 14 when executing executable instructions stored in the memory. 17. A computer-readable storage medium, characterized in that it stores executable instructions for implementing the feature selection method according to any one of claims 1 to 14 when executed by a processor. 18. A computer program product, comprising a computer program or instructions, characterized in that when the computer program or instructions are executed by a processor, the feature selection method according to any one of claims 1 to 14 is implemented.