WO2021139078A1

WO2021139078A1 - Artificial intelligence system risk detection method and apparatus, and computer device and medium

Info

Publication number: WO2021139078A1
Application number: PCT/CN2020/093555
Authority: WO
Inventors: 李洋
Original assignee: 平安科技（深圳）有限公司
Priority date: 2020-01-07
Filing date: 2020-05-30
Publication date: 2021-07-15
Also published as: CN111240975B; CN111240975A

Abstract

The present application relates to the field of artificial intelligence model deployment, and in particular to an artificial intelligence system risk detection method and apparatus, and a computer device and a storage medium. The method comprises: running an artificial intelligence system to be subjected to detection; acquiring index parameters of various risk evaluation indexes of the artificial intelligence system to be subjected to detection during running; performing multi-dimensional quantitative detection on the artificial intelligence system according to the index parameters of the risk evaluation indexes; by means of a source code of the system and the use thereof, determining a weight parameter corresponding to the system; comprehensively taking the index parameters and corresponding parameters into consideration to obtain a running risk parameter of the artificial intelligence system to be subjected to detection; and detecting a running risk of the artificial intelligence system by means of the running risk parameter, such that the running artificial intelligence system is ensured to be secure and controllable, risks are mastered and avoided in an application process of the artificial intelligence system, and the occurrence of security problems or events is reduced.

Description

Artificial intelligence system risk detection method, device, computer equipment and medium

This application requires the priority of a Chinese patent application filed with the Chinese Patent Office on January 7, 2020 with the application number CN 202010014256.3 and the invention title "Artificial intelligence system risk detection methods, devices, computer equipment and media", and its entire contents Incorporated in this application by reference.

Technical field

This application relates to the field of computer technology, in particular to an artificial intelligence system risk detection method, device, computer equipment and storage medium.

Background technique

With the development of computer science, artificial intelligence technology is constantly updated. Artificial intelligence is a branch of computer science. It attempts to understand the essence of intelligence and produce a new kind of response that can respond in a similar way to human intelligence. Intelligent machines. Research in this field includes robotics, language recognition, image recognition, natural language processing, and expert systems. Since the birth of artificial intelligence, the theory and technology have become increasingly mature, and the field of application has continued to expand. It is conceivable that the technological products brought by artificial intelligence in the future will be the "containers" of human wisdom. Artificial intelligence can simulate the information process of human consciousness and thinking. Artificial intelligence is not human intelligence, but it can think like humans and may exceed human intelligence.

While artificial intelligence helps people’s lives more convenient and comfortable, it also brings huge security risks. These security risks may not only cause property losses, but even endanger human life. The inventor realizes that the industry has The risks of artificial intelligence algorithms/models, data, etc., have not yet had enough cognitive and scientific detection methods to detect their risks.

Summary of the invention

Based on this, it is necessary to have sufficient knowledge and scientific detection methods for the risks of artificial intelligence algorithms/models, data, etc., and cannot detect the technical problems of their risks. Provide a manual that can effectively detect the risks of artificial intelligence. Intelligent system risk detection method, device, computer equipment and storage medium.

An artificial intelligence system risk detection method, the method includes:

Start the artificial intelligence system to be tested;

Run the artificial intelligence system to be tested to obtain the index parameters of each risk evaluation index of the artificial intelligence system to be tested in operation;

Identify the source code and application field of the artificial intelligence system to be tested, and obtain the detection weight parameters corresponding to each risk evaluation index of the artificial intelligence system to be tested according to the source code and application field of the artificial intelligence system to be tested;

According to the index parameters of each risk evaluation index and the detection weight parameter corresponding to each risk evaluation index, obtain the current operating risk parameters of the artificial intelligence system to be tested;

When the operating risk parameter is greater than a preset risk threshold, it is determined that the artificial intelligence system to be detected has a security risk.

An artificial intelligence system risk detection device, the device comprising:

The model start module is used to start the artificial intelligence system to be tested;

An index acquisition module, configured to run the artificial intelligence system to be tested, and acquire the index parameters of each risk evaluation index of the artificial intelligence system to be tested in operation;

The weight acquisition module is used to identify the source code and application field of the artificial intelligence system to be tested, and obtain the detection weight corresponding to each risk evaluation index of the artificial intelligence system to be tested according to the source code and application field of the artificial intelligence system to be tested parameter;

The risk prediction module is used to obtain the current operational risk parameters of the artificial intelligence system to be tested according to the index parameters of each risk evaluation index and the detection weight parameters corresponding to each risk evaluation index;

The risk determination module is configured to determine that the artificial intelligence system to be tested has a security risk when the operating risk parameter is greater than a preset risk threshold.

A computer device includes a memory and a processor, the memory stores a computer program, and the processor implements the following steps when the processor executes the computer program:

Start the artificial intelligence system to be tested;

A computer-readable storage medium having a computer program stored thereon, and when the computer program is executed by a processor, the following steps are implemented:

Start the artificial intelligence system to be tested;

The above-mentioned artificial intelligence system risk detection method, device, computer equipment and storage medium, by running the artificial intelligence system to be tested, obtain the index parameters of each risk evaluation index of the artificial intelligence system to be tested in operation, and according to the index parameters of each risk evaluation index Perform multi-angle quantitative detection on the artificial intelligence system, and then determine its corresponding weight parameters through the source code of the system and its application fields, comprehensively consider the index parameters and corresponding parameters to obtain the operational risk parameters of the artificial intelligence system to be tested, and use this Operating risk parameters are used to detect the operating risk of the artificial intelligence system to ensure that the operating artificial intelligence system is safe and controllable, grasp and avoid risks during the application of the artificial intelligence system, and reduce the occurrence of safety problems or incidents.

Description of the drawings

Figure 1 is an application environment diagram of an artificial intelligence system risk detection method in an embodiment;

FIG. 2 is a schematic flowchart of a risk detection method for an artificial intelligence system in an embodiment;

FIG. 3 is a schematic flowchart of a risk detection method for an artificial intelligence system in another embodiment;

Figure 4 is a structural block diagram of an artificial intelligence system risk detection device in an embodiment;

Fig. 5 is an internal structure diagram of a computer device in an embodiment.

Detailed ways

In order to make the purpose, technical solutions, and advantages of this application clearer and clearer, the following further describes the application in detail with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application, and are not used to limit the present application.

The artificial intelligence system risk detection method provided in this application can be applied to the application environment as shown in FIG. 1. Among them, the artificial intelligence server 102 communicates with the detection server 104 via the network. The artificial intelligence server 102 is equipped with an artificial intelligence system to be tested. The detection server 104 first connects to the artificial intelligence server 102 to start the artificial intelligence system to be tested; runs the artificial intelligence system to be tested, and obtains the index parameters of each risk evaluation index of the artificial intelligence system to be tested in operation; identifies the source of the artificial intelligence system to be tested Code and application fields, according to the source code and application fields of the artificial intelligence system to be tested, obtain the detection weight parameters corresponding to each risk evaluation index of the artificial intelligence system to be tested; according to the index parameters of each risk evaluation index and the corresponding detection of each risk evaluation index The weight parameter obtains the current operational risk parameters of the artificial intelligence system to be tested; when the operational risk parameter is greater than the preset risk threshold, it is determined that the artificial intelligence system to be tested has a security risk. Among them, the artificial intelligence server 102 and the detection server 104 can be implemented by independent servers or a server cluster composed of multiple servers.

In one embodiment, as shown in FIG. 2, a risk detection method for an artificial intelligence system is provided. Taking the method applied to the detection server 104 in FIG. 1 as an example for description, the method includes the following steps:

S100, the detection server starts the artificial intelligence system to be detected.

Among them, the detection server specifically refers to the server used to perform endogenous security detection of the artificial intelligence system, and the endogenous security specifically refers to the security problems caused by the mechanism of the artificial intelligence system itself. The endogenous security of native artificial intelligence systems is often caused by artificial intelligence system code security, data integrity, model confidentiality, and model robustness. The detection server can determine the risk of the artificial intelligence system to be detected from the above-mentioned aspects to detect whether it has a security risk. Corresponding to endogenous security is exogenous security, which refers to factors outside the mechanism of the artificial intelligence system itself---such as the environment that the artificial intelligence system depends on in the application process, and the artificial intelligence system needs in the application process. Or security incidents caused by the security of the data generated and whether the application of the artificial intelligence system complies with the corresponding laws and regulations. Specifically, the detection server may be connected to the artificial intelligence server equipped with the artificial intelligence system to be tested through a network, and then the artificial intelligence system to be tested is activated to start the detection process.

S300: Run the artificial intelligence system to be tested, and obtain index parameters of various risk evaluation indicators of the artificial intelligence system to be tested in operation.

The risk detection of the artificial intelligence system specifically includes 5 risk evaluation indicators, including: code security risk, which refers to the security caused by vulnerabilities in the artificial intelligence system. The accuracy of the model refers to the correctness of the model's judgment on the current input in the current environment. Model interpretability refers to the operation mechanism of the model and the feedback of input data that can convince everyone. When the model gives wrong feedback, it can locate the specific reason. Model sensitivity refers to the degree of impact on the actual business of the artificial intelligence system when the model is misjudged or deviates from the artificial intelligence system design expectations during the application process. The aggressiveness of the input data. The aggressiveness of the input data refers to the degree to which the input data makes the model wrong or deviates from the design expectations. While running the artificial intelligence system to be tested, the detection server can obtain the index parameters of the above-mentioned risk evaluation indicators by detecting all aspects of the artificial intelligence system, and express them in specific digital form.

S500: Identify the source code and application field of the artificial intelligence system to be tested, and obtain the detection weight parameters corresponding to each risk evaluation index of the artificial intelligence system to be tested according to the source code and application field of the artificial intelligence system to be tested.

Among them, the detection weight parameter refers to the referenceability of a risk evaluation index in the current artificial intelligence system to be tested. Because the focus, use and composition of artificial intelligence systems are not the same, so for different artificial intelligence systems to be tested, The weight of detection is also different. While obtaining each corresponding risk evaluation index, the server also needs to identify the source code and application field of the artificial intelligence system to be tested, and obtain each risk evaluation index based on the attributes of the artificial intelligence system to be tested. The corresponding detection weight parameter.

S700, according to the index parameters of each risk evaluation index and the detection weight parameter corresponding to each risk evaluation index, obtain the current operating risk parameter of the artificial intelligence system to be detected.

The operational risk parameters of the artificial intelligence system to be tested can be used

Representation refers to the probability that the artificial intelligence system's expression or judgment of things during the operation process will affect the business. The calculation formula of operational risk parameters can be specifically as follows:

Among them, W1-W5 are the weights of various indicators. For the sake of simplification, the weight of each indicator and the detection of each indicator can use the percentile system, and then the product is normalized and converted into a percentile system. code _risk is the code security risk parameter,

Is the accuracy of the model, Inter is the interpretability of the model, Sen is the sensitivity of the model, and data _attack is the aggressiveness of the input data of the model. The operational risk parameters in this application can be tested in terms of code security risk parameters, model accuracy, model interpretability, model sensitivity, and input data aggressiveness from the endogenous security dimension to ensure the safety of artificial intelligence itself. The artificial intelligence system is safe and controllable. At the same time, it can reveal the security risks of the artificial intelligence system to users, enterprises, etc., so as to grasp and avoid risks in the application process, and reduce the occurrence of security problems or incidents.

S900: When the operating risk parameter is greater than the preset risk threshold, it is determined that the artificial intelligence system to be detected has a security risk.

When the calculated operational risk parameter is greater than the preset risk threshold, it can be judged that the current artificial intelligence system to be tested has a certain security risk. At this time, it can be determined that the artificial intelligence system to be tested has certain operational defects, which may be in actual conditions. There are security risks in the use of.

As shown in FIG. 3, in one of the embodiments, before step S900, the method further includes:

S810: Obtain historical operation risk parameters corresponding to the artificial intelligence system with security risks in the historical records.

S830: Update a preset risk threshold according to historical operating risk parameters.

At the initial moment, the detection server can determine an initial preset risk threshold, and then during the operation of the method, according to the operating risk parameters of the artificial intelligence system that has been detected in the history and that have operating errors during the testing process, Continuously update the current preset risk threshold. The update may specifically be to obtain the operating risk parameters of the artificial intelligence system in the detection process where an operating error occurs, and then calculate the lowest value of all operating risk parameters, and use the lowest value as the new preset risk threshold. By continuously updating the preset risk thresholds, the accuracy of risk detection in artificial intelligence systems can be effectively improved.

The above-mentioned risk detection method of artificial intelligence system, by running the artificial intelligence system to be tested, obtains the index parameters of each risk evaluation index of the artificial intelligence system to be tested in operation, and performs multi-angle analysis on the artificial intelligence system according to the index parameters of each risk evaluation index. Quantitative detection, and then determine its corresponding weight parameters through the source code of the system and its application fields, comprehensively consider the index parameters and corresponding parameters to obtain the operating risk parameters of the artificial intelligence system to be tested, and use the operating risk parameters to control the artificial intelligence system Operational risk detection to ensure that the operating artificial intelligence system is safe and controllable, grasp and avoid risks in the application process of artificial intelligence system, and reduce the occurrence of safety problems or incidents.

In one of the embodiments, the risk evaluation indicators include code security risk parameters, model accuracy, model interpretability, model sensitivity, and model input data offensiveness, and obtain various preset detections of the artificial intelligence system to be tested during operation. The corresponding risk evaluation indicators at all times include:

Obtain the source code corresponding to the artificial intelligence system to be tested, and obtain the code security risk parameters according to the source code.

Obtain the labeled preset test sample data, input the test sample data into the artificial intelligence system to be tested, obtain the output data corresponding to the labeled preset test sample data, and obtain the accuracy of the model based on the output data and the corresponding predicted test sample data. .

According to the output data and the corresponding prediction test sample data, the error data in the output data is obtained, the error explanation information corresponding to the error data is obtained, and the model interpretability is obtained according to the error explanation information.

According to the error data of the artificial intelligence system to be tested and the application field of the artificial intelligence system to be tested, the sensitivity of the model is obtained.

Obtain the corresponding dependent data set of the artificial intelligence system to be detected, and obtain the aggressiveness of the input data of the model according to the dependent data set.

Some risk evaluation indicators of the model can be obtained based on the composition of the model itself, but other indicators need to be tested and run on the model to obtain. In this application, the specific process of obtaining each risk evaluation indicator includes: obtaining the current artificial intelligence to be tested The code (such as open source or commercial), framework (such as Tensorflow, Pytorch, etc.) used by the system, through source code auditing, Fuzzing and other methods to identify the risk of the code framework, and obtain the code security risk corresponding to the artificial intelligence system to be tested, that is, the code Security risk parameters. The accuracy of the model can be tested by using a certain amount of test samples, and the audit score corresponding to the accuracy of the model can be obtained. Obtain data instances where the artificial intelligence system predicts or recognizes errors in the test data, and judges whether the current artificial intelligence system to be tested can make corresponding error explanations for each data instance with errors in prediction or recognition (such as a deep learning framework for classification errors) Explain, point out that its convolutional layer may affect the recognition effect, etc.), and obtain the corresponding model interpretability score. Obtain data instances where the artificial intelligence system predicts or identify errors in the test data, determine the degree of influence of the data instance on the actual business of the artificial intelligence system to be tested, and obtain the model sensitivity score according to the degree of influence. Through the data set that the artificial intelligence system relies on, detecting the degree to which the input data makes the model error or deviating from the design expectations, the risk value of the risk indicator is obtained, and the input data attack score of the artificial intelligence system to be tested is obtained according to the risk. The model is evaluated from multiple perspectives through different methods, so as to obtain risk evaluation indicators from different perspectives of the artificial intelligence system to be tested.

In one of the embodiments, step S500 includes:

Identify the source code and application field of the artificial intelligence system to be tested.

Determine the risk parameter category corresponding to the source code, search the preset risk parameter category weight parameter table according to the risk parameter category, obtain the weight parameter corresponding to the risk parameter category, and use the weight parameter corresponding to the risk parameter category as the corresponding code security risk parameter The first detection weight parameter.

According to the application field, the preset model accuracy category weight distribution table is searched to obtain the first weight parameter corresponding to the application field, and the first weight parameter corresponding to the application field is used as the second detection weight parameter corresponding to the model accuracy.

Determine the composition algorithm corresponding to the source code, search the preset model interpretability weight distribution table according to the composition algorithm, obtain the weight parameter corresponding to the composition algorithm, and use the weight parameter corresponding to the composition algorithm as the third test corresponding to the model interpretability Weight parameter.

Determine the business impact level according to the application field, look up the preset model sensitivity weight distribution table according to the business impact level, and obtain the weight parameter corresponding to the business impact level, and use the weight parameter corresponding to the business impact level as the fourth corresponding to the model sensitivity Detection weight parameter. Specifically, according to the application field of the artificial intelligence system to be tested, the business impact assessment of the artificial intelligence system to be tested can be performed, and the business impact level corresponding to the artificial intelligence system to be tested can be obtained.

According to the application field, look up the preset model input data aggressiveness weight distribution table, obtain the second weight parameter corresponding to the application field, and use the second weight parameter corresponding to the application field as the fifth detection weight parameter corresponding to the aggressiveness of the model input data .

In one of the embodiments, the business impact level is determined according to the application field, and the preset model sensitivity weight distribution table is searched according to the business impact level, and the weight parameter corresponding to the business impact level is obtained. The weight parameter corresponding to the business impact level is taken as Before the fourth detection weight parameter corresponding to the model sensitivity, it also includes:

The server can identify the source code and application field of the artificial intelligence system to be tested, and then obtain the detection weight parameter corresponding to each risk evaluation index according to the source code and application field. The preset weight distribution table can be obtained by summarizing historical empirical data or through expert experience. Generally, for each type of model, the corresponding weight distribution includes the following laws, such as the detection weight parameters corresponding to the code security risk parameters, which have a significant impact on the risk of code execution, such as buffer overflow, information leakage, unauthorized viewing, SQL (Structured Query Language, Structured Query Language) injection and other risks are assigned with larger detection weight parameters. For risks that have a general impact on code execution, such as code logic errors, low-efficiency codes, and other code risks, a lower detection weight parameter is assigned. The selection of the detection weight parameters for the accuracy of the model mainly depends on the application corresponding to the accuracy of the model. For the more critical applications, a higher weight interval is assigned, and vice versa, a lower interval is assigned. For example, artificial intelligence systems used in image recognition and speech recognition will be assigned a lower weight, while artificial intelligence systems in key aspects such as unmanned vehicles and airplanes will be assigned a higher weight. The selection of the detection weight parameter of the model interpretability mainly depends on the interpretability of the corresponding algorithm/model, and is related to the corresponding artificial intelligence classification. Algorithms/models with better interpretability are assigned a lower risk weight interval, and vice versa, a higher interval is assigned. For example, artificial intelligence systems that apply rule-based machine learning such as D3, C4.5/C5.0 decision tree algorithms will be assigned a higher weight, and artificial intelligence systems that use deep learning models such as Tensorflow, Pytorch, and Caffe will be assigned a higher weight. Is assigned a lower weight. The selection of the detection weight parameter of model sensitivity mainly depends on the importance of the corresponding application level. Specifically, according to the application field of the artificial intelligence system to be tested, the business impact assessment of the artificial intelligence system to be tested can be performed to obtain the artificial intelligence system to be tested. Corresponding business impact level, and then look up the table according to the business impact level to obtain the fourth detection weight parameter corresponding to the model sensitivity. This weight is determined according to different business impact levels. For example, artificial intelligence systems such as unmanned vehicles, airplanes, and medical operations will be assigned a higher weight, while artificial intelligence systems that use AI for attendance and check-in and other low-impact artificial intelligence systems will be assigned a lower weight. The selection of the aggressive detection weight parameters of the input data mainly depends on the detection of possible pollution/bait injection and other risks in the data set on which the AI algorithm/model/supported application depends. The more easily the data set is damaged or misleading, the higher its weight, and vice versa, the lower it is. Artificial intelligence systems such as image recognition and voice will be assigned a higher weight, while artificial intelligence systems such as human-machine question answering and unmanned driving will be assigned a lower weight.

In one of the embodiments, after step S700, the method further includes:

Generate model evaluation reports based on risk evaluation indicators and operational risk parameters.

After obtaining the risk evaluation index and obtaining the corresponding operation risk parameters according to the risk evaluation index, all the risk evaluation indexes and operation risk parameters can be sorted out, and then the corresponding model evaluation report can be generated according to the obtained indexes and operation risk parameters. The model evaluation report can be used to determine which aspects of the artificial intelligence system to be tested have operational risks, and at the same time it is convenient to grasp and avoid risks in the application process, and reduce the occurrence of security problems or incidents. By generating a model evaluation report, the test results of the artificial intelligence system to be tested can be displayed more intuitively to users.

It should be understood that, although the various steps in the flowchart of FIGS. 2-3 are displayed in sequence as indicated by the arrows, these steps are not necessarily performed in sequence in the order indicated by the arrows. Unless specifically stated in this article, the execution of these steps is not strictly limited in order, and these steps can be executed in other orders. Moreover, at least some of the steps in Figure 2-3 may include multiple sub-steps or multiple stages. These sub-steps or stages are not necessarily executed at the same time, but can be executed at different times. These sub-steps or stages The execution order of is not necessarily performed sequentially, but may be performed alternately or alternately with at least a part of other steps or sub-steps or stages of other steps.

In one embodiment, as shown in FIG. 4, an artificial intelligence system risk detection device is provided, including:

The model starting module 100 is used to start the artificial intelligence system to be tested;

The index acquisition module 300 is used to run the artificial intelligence system to be tested, and acquire the index parameters of each risk evaluation index of the artificial intelligence system to be tested in operation;

The weight acquisition module 500 is used to identify the source code and application field of the artificial intelligence system to be tested, and obtain the detection weight parameters corresponding to each risk evaluation index of the artificial intelligence system to be tested according to the source code and application field of the artificial intelligence system to be tested;

The risk prediction module 700 is used to obtain the current operating risk parameters of the artificial intelligence system to be tested according to the index parameters of each risk evaluation index and the detection weight parameter corresponding to each risk evaluation index;

The risk determination module 900 is configured to determine that the artificial intelligence system to be tested has a security risk when the operating risk parameter is greater than the preset risk threshold.

In one of the embodiments, the risk evaluation indicators include code security risk parameters, model accuracy, model interpretability, model sensitivity, and model input data aggressiveness. The indicator acquisition module 300 is used to: acquire the corresponding source of the artificial intelligence system to be detected Code, obtain code security risk parameters according to the source code; obtain the marked preset test sample data, input the test sample data into the artificial intelligence system to be tested, obtain the corresponding output data of the marked preset test sample data, and according to the output data and the corresponding prediction The mark of the test sample data is used to obtain the accuracy of the model; according to the mark of the output data and the corresponding predicted test sample data, the error data in the output data is obtained, the error explanation information corresponding to the error data is obtained, and the model can be obtained according to the error explanation information. Interpretation; obtain model sensitivity according to the error data of the artificial intelligence system to be detected and the application field of the artificial intelligence system to be tested; obtain the corresponding dependent data set of the artificial intelligence system to be tested, and obtain the aggressiveness of the model input data according to the dependent data set.

In one of the embodiments, the weight acquisition module 500 is used to identify the source code and application field of the artificial intelligence system to be detected; determine the risk parameter category corresponding to the source code, and search the preset risk parameter category weight parameter table according to the risk parameter category to obtain The weight parameter corresponding to the risk parameter category will be the weight parameter corresponding to the risk parameter category as the first detection weight parameter corresponding to the code security risk parameter; according to the application field, look up the preset model accuracy category weight distribution table, and obtain and apply The first weight parameter corresponding to the domain, the first weight parameter corresponding to the application domain, is used as the second detection weight parameter corresponding to the accuracy of the model; the composition algorithm corresponding to the source code is determined, and the interpretability of the preset model is found according to the composition algorithm The weight distribution table obtains the weight parameters corresponding to the constituent algorithm, and uses the weight parameter corresponding to the constituent algorithm as the third detection weight parameter corresponding to the interpretability of the model; determines the business impact level according to the application field, and finds the preset according to the business impact level The model sensitivity weight distribution table is used to obtain the weight parameter corresponding to the business impact level, and the weight parameter corresponding to the business impact level is used as the fourth detection weight parameter corresponding to the model sensitivity; according to the application field, search for preset model input data aggressiveness The weight distribution table obtains the second weight parameter corresponding to the application field, and uses the second weight parameter corresponding to the application field as the fifth detection weight parameter corresponding to the aggressiveness of the model input data.

In one of the embodiments, the weight obtaining module 500 is further configured to perform business impact assessment on the artificial intelligence system to be tested according to the application field of the artificial intelligence system to be tested, and obtain the business impact level corresponding to the artificial intelligence system to be tested.

In one of the embodiments, a threshold update module is also reported to obtain historical operating risk parameters corresponding to the artificial intelligence system with security risks in the historical records; and to update the preset risk threshold according to the historical operating risk parameters.

In one of the embodiments, a report generation module is further included, which is used to generate a model evaluation report according to the risk evaluation index and the operational risk parameters.

For the specific limitation of the artificial intelligence system risk detection device, please refer to the above limitation on the artificial intelligence system risk detection method, which will not be repeated here. Each module in the above-mentioned artificial intelligence system risk detection device can be implemented in whole or in part by software, hardware and a combination thereof. The above-mentioned modules may be embedded in the form of hardware or independent of the processor in the computer equipment, or may be stored in the memory of the computer equipment in the form of software, so that the processor can call and execute the operations corresponding to the above-mentioned modules.

In one embodiment, a computer device is provided. The computer device may be a server, and its internal structure diagram may be as shown in FIG. 5. The computer equipment includes a processor, a memory, a network interface, and a database connected through a system bus. Among them, the processor of the computer device is used to provide calculation and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program, and a database. The internal memory provides an environment for the operation of the operating system and computer programs in the non-volatile storage medium. The database of the computer equipment is used to store data related to the operational risk parameters of the historical artificial intelligence system. The network interface of the computer device is used to communicate with an external terminal through a network connection. The computer program is executed by the processor to realize an artificial intelligence system risk detection method.

Those skilled in the art can understand that the structure shown in FIG. 5 is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may Including more or fewer parts than shown in the figure, or combining some parts, or having a different arrangement of parts.

In one embodiment, a computer device is provided, including a memory and a processor, a computer program is stored in the memory, and the processor implements the following steps when the processor executes the computer program:

Start the artificial intelligence system to be tested;

When the operating risk parameter is greater than the preset risk threshold, it is determined that the artificial intelligence system to be tested has a security risk.

In one embodiment, the processor further implements the following steps when executing the computer program: obtain the source code corresponding to the artificial intelligence system to be tested, obtain the code security risk parameters according to the source code; obtain the marked preset test sample data, and input the test sample data The artificial intelligence system to be tested obtains the output data corresponding to the labeled preset test sample data, and obtains the model accuracy according to the label of the output data and the corresponding predicted test sample data; according to the output data and the label of the corresponding predicted test sample data , Obtain the error data in the output data, obtain the error interpretation information corresponding to the error data, and obtain the interpretability of the model according to the error interpretation information; obtain the model sensitivity according to the error data of the artificial intelligence system to be tested and the application field of the artificial intelligence system to be tested Degree; Obtain the dependent data set corresponding to the artificial intelligence system to be detected, and obtain the aggressiveness of the model input data according to the dependent data set.

In one embodiment, the processor further implements the following steps when executing the computer program: identifying the source code and application field of the artificial intelligence system to be tested; determining the risk parameter category corresponding to the source code, and searching for the preset risk parameter category weight according to the risk parameter category Parameter table to obtain the weight parameter corresponding to the risk parameter category, and use the weight parameter corresponding to the risk parameter category as the first detection weight parameter corresponding to the code security risk parameter; search for the preset model accuracy category weight distribution table according to the application field , To obtain the first weight parameter corresponding to the application field, and use the first weight parameter corresponding to the application field as the second detection weight parameter corresponding to the accuracy of the model; determine the composition algorithm corresponding to the source code, and search for the preset according to the composition algorithm Model interpretability weight distribution table, obtain the weight parameters corresponding to the constituent algorithm, and use the weight parameter corresponding to the constituent algorithm as the third detection weight parameter corresponding to the model interpretability; determine the business impact level according to the application field, and determine the business impact level according to the business impact Level search the preset model sensitivity weight distribution table, obtain the weight parameter corresponding to the business impact level, and use the weight parameter corresponding to the business impact level as the fourth detection weight parameter corresponding to the model sensitivity; search for the preset model according to the application field Enter the data aggressiveness weight distribution table to obtain the second weight parameter corresponding to the application field, and use the second weight parameter corresponding to the application field as the fifth detection weight parameter corresponding to the aggressiveness of the model input data.

In one embodiment, the processor further implements the following steps when executing the computer program: according to the application field of the artificial intelligence system to be tested, the artificial intelligence system to be tested performs business impact assessment, and the business impact level corresponding to the artificial intelligence system to be tested is obtained.

In one embodiment, the processor further implements the following steps when executing the computer program: acquiring historical operating risk parameters corresponding to the artificial intelligence system with security risks in the historical record; updating the preset risk threshold according to the historical operating risk parameters.

In an embodiment, the processor further implements the following steps when executing the computer program: generating a model evaluation report according to the risk evaluation index and the operating risk parameter.

In one embodiment, a computer-readable storage medium is provided. The computer-readable storage medium may be non-volatile or volatile. The computer-readable storage medium stores a computer program. When executed by the processor, the following steps are implemented:

Start the artificial intelligence system to be tested;

In one embodiment, when the computer program is executed by the processor, the following steps are also implemented: obtain the source code corresponding to the artificial intelligence system to be tested, obtain the code security risk parameters according to the source code; obtain the marked preset test sample data, and transfer the test sample data Input the artificial intelligence system to be tested, obtain the output data corresponding to the labeled preset test sample data, obtain the model accuracy according to the output data and the corresponding prediction test sample data, and obtain the model accuracy; according to the output data and the corresponding prediction test sample data Mark, obtain the error data in the output data, obtain the error explanation information corresponding to the error data, obtain the interpretability of the model according to the error explanation information; obtain the model according to the error data of the artificial intelligence system to be tested and the application field of the artificial intelligence system to be tested Sensitivity: Obtain the dependent data set corresponding to the artificial intelligence system to be detected, and obtain the aggressiveness of the input data of the model according to the dependent data set.

In one embodiment, when the computer program is executed by the processor, the following steps are also implemented: identify the source code and application field of the artificial intelligence system to be tested; determine the risk parameter category corresponding to the source code, and search for the preset risk parameter category according to the risk parameter category Weight parameter table, get the weight parameter corresponding to the risk parameter category, and use the weight parameter corresponding to the risk parameter category as the first detection weight parameter corresponding to the code security risk parameter; find the preset model accuracy category weight distribution according to the application field Table to obtain the first weight parameter corresponding to the application field, and use the first weight parameter corresponding to the application field as the second detection weight parameter corresponding to the accuracy of the model; determine the composition algorithm corresponding to the source code, and look up the forecast according to the composition algorithm Set the model interpretability weight distribution table, obtain the weight parameters corresponding to the constituent algorithm, and use the weight parameter corresponding to the constituent algorithm as the third detection weight parameter corresponding to the model interpretability; determine the business impact level according to the application domain, and determine the business impact level according to the business The influence level searches the preset model sensitivity weight distribution table to obtain the weight parameter corresponding to the business influence level, and uses the weight parameter corresponding to the business influence level as the fourth detection weight parameter corresponding to the model sensitivity; find the preset according to the application field The model input data aggressiveness weight distribution table is used to obtain the second weight parameter corresponding to the application domain, and the second weight parameter corresponding to the application domain is used as the fifth detection weight parameter corresponding to the aggressiveness of the model input data.

In one embodiment, when the computer program is executed by the processor, the following steps are further implemented: according to the application field of the artificial intelligence system to be tested, the business impact assessment of the artificial intelligence system to be tested is performed, and the business impact level corresponding to the artificial intelligence system to be tested is obtained.

In one embodiment, when the computer program is executed by the processor, the following steps are further implemented: obtaining historical operating risk parameters corresponding to the artificial intelligence system with security risks in the historical records; updating the preset risk threshold according to the historical operating risk parameters.

In one embodiment, when the computer program is executed by the processor, the following steps are further implemented: generating a model evaluation report according to the risk evaluation index and the operating risk parameters.

A person of ordinary skill in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be implemented by instructing relevant hardware through a computer program. The computer program can be stored in a non-volatile computer readable storage medium. When the computer program is executed, it may include the processes of the above-mentioned method embodiments. Wherein, any reference to memory, storage, database, or other media used in the embodiments provided in this application may include non-volatile and/or volatile memory. Non-volatile memory may include read only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), or flash memory. Volatile memory may include random access memory (RAM) or external cache memory. As an illustration and not a limitation, RAM is available in many forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous chain Channel (Synchlink) DRAM (SLDRAM), memory bus (Rambus) direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM), etc.

The technical features of the above embodiments can be combined arbitrarily. In order to make the description concise, all possible combinations of the technical features in the above embodiments are not described. However, as long as there is no contradiction in the combination of these technical features, they should be It is considered as the range described in this specification.

The above examples only express several implementation manners of the present application, and the description is relatively specific and detailed, but it should not be understood as a limitation on the scope of the invention patent. It should be pointed out that for those of ordinary skill in the art, without departing from the concept of this application, several modifications and improvements can be made, and these all fall within the protection scope of this application. Therefore, the scope of protection of the patent of this application shall be subject to the appended claims.

Claims

An artificial intelligence system risk detection method, wherein the method includes:

Start the artificial intelligence system to be tested;

Run the artificial intelligence system to be tested to obtain the index parameters of each risk evaluation index of the artificial intelligence system to be tested in operation;

Identify the source code and application field of the artificial intelligence system to be tested, and obtain the detection weight parameters corresponding to each risk evaluation index of the artificial intelligence system to be tested according to the source code and application field of the artificial intelligence system to be tested;

According to the index parameters of each risk evaluation index and the detection weight parameter corresponding to each risk evaluation index, obtain the current operating risk parameters of the artificial intelligence system to be tested;

When the operating risk parameter is greater than a preset risk threshold, it is determined that the artificial intelligence system to be detected has a security risk.
The method according to claim 1, wherein the risk evaluation indicators include code security risk parameters, model accuracy, model interpretability, model sensitivity, and model input data aggressiveness, and said running the artificial intelligence to be detected The system, acquiring each risk evaluation index corresponding to each preset detection time in the running of the artificial intelligence system test to be detected includes:

Obtain the source code corresponding to the artificial intelligence system to be tested, and obtain code security risk parameters according to the source code;

Obtain labeled preset test sample data, input the test sample data into the artificial intelligence system to be tested, obtain output data corresponding to the labeled preset test sample data, and based on the output data and corresponding predicted test sample data With the mark, get the accuracy of the model;

Acquiring the error data in the output data according to the marks carried by the output data and the corresponding predicted test sample data, acquiring the error explanation information corresponding to the error data, and acquiring the model interpretability according to the error explanation information;

Acquiring the model sensitivity according to the error data of the artificial intelligence system to be tested and the application field of the artificial intelligence system to be tested;

Obtain the dependent data set corresponding to the artificial intelligence system to be detected, and obtain the aggressiveness of the model input data according to the dependent data set.
The method according to claim 2, wherein said identifying the source code and application field of the artificial intelligence system to be tested, and acquiring each of the artificial intelligence system to be tested according to the source code and application field of the artificial intelligence system to be tested The detection weight parameters corresponding to the risk evaluation indicators include:

Identify the source code and application field of the artificial intelligence system to be tested;

Determine the risk parameter category corresponding to the source code, search the preset risk parameter category weight parameter table according to the risk parameter category, obtain the weight parameter corresponding to the risk parameter category, and convert the risk parameter category to the risk parameter category. The weight parameter is used as the first detection weight parameter corresponding to the code security risk parameter;

Search for a preset model accuracy category weight distribution table according to the application field to obtain a first weight parameter corresponding to the application field, and use the first weight parameter corresponding to the application field as the accuracy of the model The second detection weight parameter corresponding to the degree;

Determine the composition algorithm corresponding to the source code, search the preset model interpretability weight distribution table according to the composition algorithm, obtain the weight parameter corresponding to the composition algorithm, and use the weight parameter corresponding to the composition algorithm as the The third detection weight parameter corresponding to the interpretability of the model;

Determine the business impact level according to the application field, search the preset model sensitivity weight distribution table according to the business impact level, obtain the weight parameter corresponding to the business impact level, and convert the weight corresponding to the business impact level Parameter as the fourth detection weight parameter corresponding to the model sensitivity;

According to the application field, search for a preset model input data aggressive weight distribution table to obtain a second weight parameter corresponding to the application field, and use the second weight parameter corresponding to the application field as the model input The fifth detection weight parameter corresponding to the data aggressiveness.
The method according to claim 3, wherein the determining the business impact level according to the application field, searching a preset model sensitivity weight distribution table according to the business impact level, and obtaining a weight parameter corresponding to the business impact level , Before using the weight parameter corresponding to the service impact level as the fourth detection weight parameter corresponding to the model sensitivity, the method further includes:

According to the application field of the artificial intelligence system to be tested, a business impact assessment is performed on the artificial intelligence system to be tested, and the business impact level corresponding to the artificial intelligence system to be tested is obtained.
The method according to claim 1, wherein when the operating risk parameter is greater than a preset risk threshold, before determining that the artificial intelligence system to be detected has a security risk, the method further comprises:

Obtain historical operating risk parameters corresponding to artificial intelligence systems with security risks in historical records;

The preset risk threshold is updated according to the historical operating risk parameter.
The method according to claim 1, wherein, after acquiring the operating risk parameters of the artificial intelligence system to be tested currently according to each risk evaluation index and the detection weight parameter corresponding to each index, the method further comprises:

A model evaluation report is generated according to the risk evaluation index and the operation risk parameter.
An artificial intelligence system risk detection device, wherein the device includes:

Startup module, used to start the artificial intelligence system to be tested;

An index acquisition module, configured to run the artificial intelligence system to be tested, and acquire the index parameters of each risk evaluation index of the artificial intelligence system to be tested in operation;

The weight acquisition module is used to identify the source code and application field of the artificial intelligence system to be tested, and obtain the detection weight corresponding to each risk evaluation index of the artificial intelligence system to be tested according to the source code and application field of the artificial intelligence system to be tested parameter;

The risk prediction module is used to obtain the current operational risk parameters of the artificial intelligence system to be tested according to the index parameters of each risk evaluation index and the detection weight parameters corresponding to each risk evaluation index;

The risk determination module is configured to determine that the artificial intelligence system to be tested has a security risk when the operating risk parameter is greater than a preset risk threshold.
A computer device includes a memory and a processor, the memory stores a computer program, and the processor implements the following steps when the processor executes the computer program:

Start the artificial intelligence system to be tested;

Run the artificial intelligence system to be tested to obtain the index parameters of each risk evaluation index of the artificial intelligence system to be tested in operation;

Identify the source code and application field of the artificial intelligence system to be tested, and obtain the detection weight parameters corresponding to each risk evaluation index of the artificial intelligence system to be tested according to the source code and application field of the artificial intelligence system to be tested;

According to the index parameters of each risk evaluation index and the detection weight parameter corresponding to each risk evaluation index, obtain the current operating risk parameters of the artificial intelligence system to be tested;

When the operating risk parameter is greater than a preset risk threshold, it is determined that the artificial intelligence system to be detected has a security risk.
8. The computer device according to claim 8, wherein the risk evaluation index includes code security risk parameters, model accuracy, model interpretability, model sensitivity, and model input data offensiveness.
The computer device according to claim 9, wherein said running the artificial intelligence system to be tested and acquiring each risk evaluation index corresponding to each preset detection time during the running of the artificial intelligence system to be tested test comprises:

Obtain the source code corresponding to the artificial intelligence system to be tested, and obtain code security risk parameters according to the source code;

Obtain labeled preset test sample data, input the test sample data into the artificial intelligence system to be tested, obtain output data corresponding to the labeled preset test sample data, and based on the output data and corresponding predicted test sample data With the mark, get the accuracy of the model;

Acquiring the error data in the output data according to the marks carried by the output data and the corresponding predicted test sample data, acquiring the error explanation information corresponding to the error data, and acquiring the model interpretability according to the error explanation information;

Acquiring the model sensitivity according to the error data of the artificial intelligence system to be tested and the application field of the artificial intelligence system to be tested;

Obtain the dependent data set corresponding to the artificial intelligence system to be detected, and obtain the aggressiveness of the model input data according to the dependent data set.
The computer device according to claim 10, wherein the source code and application field of the artificial intelligence system to be tested are identified, and the artificial intelligence system to be tested is obtained according to the source code and application field of the artificial intelligence system to be tested The detection weight parameters corresponding to each risk evaluation index include:

Identify the source code and application field of the artificial intelligence system to be tested;

Determine the risk parameter category corresponding to the source code, search the preset risk parameter category weight parameter table according to the risk parameter category, obtain the weight parameter corresponding to the risk parameter category, and convert the risk parameter category to the risk parameter category. The weight parameter is used as the first detection weight parameter corresponding to the code security risk parameter;

Search for a preset model accuracy category weight distribution table according to the application field to obtain a first weight parameter corresponding to the application field, and use the first weight parameter corresponding to the application field as the accuracy of the model The second detection weight parameter corresponding to the degree;

Determine the composition algorithm corresponding to the source code, search the preset model interpretability weight distribution table according to the composition algorithm, obtain the weight parameter corresponding to the composition algorithm, and use the weight parameter corresponding to the composition algorithm as the The third detection weight parameter corresponding to the interpretability of the model;

Determine the business impact level according to the application field, search the preset model sensitivity weight distribution table according to the business impact level, obtain the weight parameter corresponding to the business impact level, and convert the weight corresponding to the business impact level Parameter as the fourth detection weight parameter corresponding to the model sensitivity;

According to the application field, search for a preset model input data aggressive weight distribution table to obtain a second weight parameter corresponding to the application field, and use the second weight parameter corresponding to the application field as the model input The fifth detection weight parameter corresponding to the aggressiveness of the data.
The computer device according to claim 11, wherein the business impact level is determined according to the application field, and a preset model sensitivity weight distribution table is searched according to the business impact level to obtain the weight corresponding to the business impact level Parameter, before using the weight parameter corresponding to the service impact level as the fourth detection weight parameter corresponding to the model sensitivity, the method further includes:

According to the application field of the artificial intelligence system to be tested, a business impact assessment is performed on the artificial intelligence system to be tested, and the business impact level corresponding to the artificial intelligence system to be tested is obtained.
8. The computer device according to claim 8, wherein when the operating risk parameter is greater than a preset risk threshold, before determining that the artificial intelligence system to be detected has a security risk, the method further comprises:

Obtain historical operating risk parameters corresponding to artificial intelligence systems with security risks in historical records;

The preset risk threshold is updated according to the historical operating risk parameter.
8. The computer device according to claim 8, wherein, after acquiring the operating risk parameters of the artificial intelligence system to be tested currently according to each risk evaluation index and the detection weight parameter corresponding to each index, the method further comprises:

A model evaluation report is generated according to the risk evaluation index and the operation risk parameter.
A computer-readable storage medium having a computer program stored thereon, wherein the computer program is executed by a processor to implement the following steps:

Start the artificial intelligence system to be tested;

Run the artificial intelligence system to be tested to obtain the index parameters of each risk evaluation index of the artificial intelligence system to be tested in operation;

Identify the source code and application field of the artificial intelligence system to be tested, and obtain the detection weight parameters corresponding to each risk evaluation index of the artificial intelligence system to be tested according to the source code and application field of the artificial intelligence system to be tested;

According to the index parameters of each risk evaluation index and the detection weight parameter corresponding to each risk evaluation index, obtain the current operating risk parameters of the artificial intelligence system to be tested;

When the operating risk parameter is greater than a preset risk threshold, it is determined that the artificial intelligence system to be detected has a security risk.
The computer-readable storage medium according to claim 15, wherein the risk evaluation indicators include code security risk parameters, model accuracy, model interpretability, model sensitivity, and model input data offensiveness, and the operation of the The artificial intelligence system to be tested, acquiring each risk evaluation index corresponding to each preset detection time in the running of the artificial intelligence system test to be tested includes:

Obtain the source code corresponding to the artificial intelligence system to be tested, and obtain code security risk parameters according to the source code;

Obtain labeled preset test sample data, input the test sample data into the artificial intelligence system to be tested, obtain output data corresponding to the labeled preset test sample data, and based on the output data and corresponding predicted test sample data With the mark, get the accuracy of the model;

Acquiring the error data in the output data according to the marks carried by the output data and the corresponding predicted test sample data, acquiring the error explanation information corresponding to the error data, and acquiring the model interpretability according to the error explanation information;

Acquiring the model sensitivity according to the error data of the artificial intelligence system to be tested and the application field of the artificial intelligence system to be tested;

Obtain the dependent data set corresponding to the artificial intelligence system to be detected, and obtain the aggressiveness of the model input data according to the dependent data set.
The computer-readable storage medium according to claim 16, wherein the source code and application field of the artificial intelligence system to be tested are identified, and the source code and application field of the artificial intelligence system to be tested are obtained according to the source code and application field of the artificial intelligence system to be tested. The detection weight parameters corresponding to each risk evaluation index of the artificial intelligence system include:

Identify the source code and application field of the artificial intelligence system to be tested;

Determine the risk parameter category corresponding to the source code, search the preset risk parameter category weight parameter table according to the risk parameter category, obtain the weight parameter corresponding to the risk parameter category, and convert the risk parameter category to the risk parameter category. The weight parameter is used as the first detection weight parameter corresponding to the code security risk parameter;

Search for a preset model accuracy category weight distribution table according to the application field to obtain a first weight parameter corresponding to the application field, and use the first weight parameter corresponding to the application field as the accuracy of the model The second detection weight parameter corresponding to the degree;

Determine the composition algorithm corresponding to the source code, search the preset model interpretability weight distribution table according to the composition algorithm, obtain the weight parameter corresponding to the composition algorithm, and use the weight parameter corresponding to the composition algorithm as the The third detection weight parameter corresponding to the interpretability of the model;

Determine the business impact level according to the application field, search the preset model sensitivity weight distribution table according to the business impact level, obtain the weight parameter corresponding to the business impact level, and convert the weight corresponding to the business impact level Parameter as the fourth detection weight parameter corresponding to the model sensitivity;

According to the application field, search for a preset model input data aggressive weight distribution table to obtain a second weight parameter corresponding to the application field, and use the second weight parameter corresponding to the application field as the model input The fifth detection weight parameter corresponding to the aggressiveness of the data.
The computer-readable storage medium according to claim 17, wherein the business impact level is determined according to the application field, and a preset model sensitivity weight distribution table is searched according to the business impact level to obtain the business impact level The corresponding weight parameter, before the weight parameter corresponding to the business impact level is used as the fourth detection weight parameter corresponding to the model sensitivity, further includes:

According to the application field of the artificial intelligence system to be tested, a business impact assessment is performed on the artificial intelligence system to be tested, and the business impact level corresponding to the artificial intelligence system to be tested is obtained.
15. The computer-readable storage medium according to claim 15, wherein when the operating risk parameter is greater than a preset risk threshold, before determining that the artificial intelligence system to be detected has a security risk, the method further comprises:

Obtain historical operating risk parameters corresponding to artificial intelligence systems with security risks in historical records;

The preset risk threshold is updated according to the historical operating risk parameter.
15. The computer-readable storage medium according to claim 15, wherein, after acquiring the operating risk parameters of the artificial intelligence system to be tested currently according to each risk evaluation index and the detection weight parameter corresponding to each index, the method further comprises:

A model evaluation report is generated according to the risk evaluation index and the operation risk parameter.