CN115757075A

CN115757075A - Task abnormity detection method and device, computer equipment and storage medium

Info

Publication number: CN115757075A
Application number: CN202211568702.0A
Authority: CN
Inventors: 钟辉
Original assignee: Ping An Property and Casualty Insurance Company of China Ltd
Current assignee: Ping An Property and Casualty Insurance Company of China Ltd
Priority date: 2022-12-08
Filing date: 2022-12-08
Publication date: 2023-03-07

Abstract

The embodiment of the application belongs to the field of artificial intelligence and relates to a task abnormity detection method, which comprises the following steps: acquiring target running log data of a target task in a service system; loading the running log data into a preset target database table; extracting first data of a first preset event type from a target database table; acquiring a file statistic corresponding to a first target field from the first data; if the file statistics value is larger than a preset value threshold value, generating abnormal alarm information corresponding to the target task; and pushing the abnormal alarm information to a target user. The application also provides a task abnormity detection device, computer equipment and a storage medium. In addition, the present application also relates to a block chain technique, and file statistics may be stored in the block chain. According to the method and the device, whether the number of the files in the task is abnormal or not is accurately detected, the processing efficiency of abnormal detection on the target task is improved, and the use experience of a target user is improved.

Description

Task abnormity detection method and device, computer equipment and storage medium

Technical Field

The present application relates to the field of artificial intelligence technologies, and in particular, to a method and an apparatus for task exception detection, a computer device, and a storage medium.

Background

In the wave of big data, the data volume of enterprises in various industries, such as financial enterprises, is getting larger and larger, and during the data processing process of the business system, scenes with excessive files exist in individual tasks, and if the scenes are not found and corrected in time, production problems can be caused. However, the current business system can only process some tasks and provide the processing results of the tasks to the user, but cannot determine which tasks are subtasks with abnormal file number, so how to determine whether an abnormal task with abnormal file number exists in a plurality of tasks is a problem to be solved urgently.

Disclosure of Invention

An embodiment of the application aims to provide a task abnormity detection method, a task abnormity detection device, a computer device and a storage medium, so as to solve the technical problem that the prior art cannot determine whether an abnormal task with an abnormal file number exists in a plurality of tasks.

In order to solve the above technical problem, an embodiment of the present application provides a task anomaly detection method, which adopts the following technical solutions:

acquiring target running log data of a target task in a service system;

loading the running log data into a preset target database table;

extracting first data of a first preset event type from the target database table;

acquiring file statistics corresponding to a first target field from the first data;

if the file statistics is larger than a preset value threshold, generating abnormal alarm information corresponding to the target task;

and pushing the abnormal alarm information to a target user.

Further, the step of obtaining target operation log data of a target task in the service system specifically includes:

acquiring a target task identifier of the target task;

calling a preset task log acquisition component;

acquiring appointed running log data corresponding to the target task identifier from a preset storage directory based on the task log acquisition component;

and taking the specified operation log data as the target operation log data.

Further, after the step of loading the running log data into a preset target database table, the method further includes:

extracting second data corresponding to a second preset event type from the target database table;

acquiring input quantity statistical data corresponding to a second target field from the second data;

acquiring the processing data volume of each target processing unit corresponding to the target task based on the input volume statistical data;

calculating the median of all the processed data quantities;

and performing data inclination detection processing on the target task based on the processing data amount and the median to generate a corresponding inclination detection result.

Further, the step of performing data skew detection processing on the target task based on the processed data amount and the median to generate a corresponding skew detection result specifically includes:

acquiring the designated processing data volume with the maximum value from all the processing data volumes;

calculating the ratio of the specified processing data amount to the median;

judging whether the ratio is larger than a preset ratio threshold value or not;

if the value is larger than the ratio threshold, generating a first inclination detection result of the target task with data inclination risk, otherwise, generating a second inclination detection result of the target task without data inclination risk.

Further, after the step of obtaining the target running log data of the target task in the service system, the method further includes:

acquiring target operation data corresponding to a preset data type from the target operation log data;

inputting the target operation data into a pre-trained anomaly analysis model;

performing anomaly analysis on the target operation data through the anomaly identification model to generate an anomaly analysis result corresponding to the target task;

and pushing the analysis result to the target user.

Further, before the step of inputting the target operation data into a pre-trained anomaly analysis model, the method further includes:

acquiring historical running log data in a preset time period;

constructing a training sample and a testing sample based on the historical running log data; the training sample comprises training sample data and a training sample label corresponding to the training sample data, and the test sample comprises test sample data and a test sample label corresponding to the test sample data;

calling a pre-constructed neural network model;

taking training sample data in the training samples as input of the neural network model, taking training sample labels in the training samples as output of the neural network model, training the neural network model, continuously adjusting parameters of the neural network model until evaluation indexes of the trained neural network model meet expected conditions, and taking the trained neural network model as the anomaly analysis model; wherein the evaluation index is obtained by checking the trained neural network model using the test sample.

Further, after the step of generating the abnormal warning information corresponding to the target task if the file statistics is greater than a preset value threshold, the method further includes:

generating a termination instruction corresponding to the target task;

and sending the termination instruction to the service system so as to control the service system to execute task termination processing on the target task based on the termination instruction.

In order to solve the above technical problem, an embodiment of the present application further provides a task anomaly detection apparatus, which adopts the following technical solutions:

the first acquisition module is used for acquiring target running log data of a target task in a service system;

the loading module is used for loading the running log data into a preset target database table;

the first extraction module is used for extracting first data of a first preset event type from the target database table;

the second acquisition module is used for acquiring the file statistics corresponding to the first target field from the first data;

the first generation module is used for generating abnormal alarm information corresponding to the target task if the file statistics is larger than a preset value threshold;

and the first pushing module is used for pushing the abnormal alarm information to a target user.

In order to solve the above technical problem, an embodiment of the present application further provides a computer device, which adopts the following technical solutions:

acquiring target running log data of a target task in a service system;

loading the running log data into a preset target database table;

acquiring a file statistic corresponding to a first target field from the first data;

if the file statistics value is larger than a preset value threshold value, generating abnormal alarm information corresponding to the target task;

and pushing the abnormal alarm information to a target user.

In order to solve the above technical problem, an embodiment of the present application further provides a computer-readable storage medium, which adopts the following technical solutions:

acquiring target running log data of a target task in a service system;

loading the running log data into a preset target database table;

and pushing the abnormal alarm information to a target user.

Compared with the prior art, the embodiment of the application mainly has the following beneficial effects:

the method comprises the steps of firstly, obtaining target running log data of a target task in a service system; then loading the running log data into a preset target database table; then extracting first data of a first preset event type from the target database table; subsequently acquiring file statistics corresponding to the first target field from the first data; and if the file statistics is larger than a preset value threshold, generating abnormal alarm information corresponding to the target task, and pushing the abnormal alarm information to a target user. According to the method and the device, the file statistic corresponding to the first preset event type and the first target field is extracted from the target running log data of the target task in the service system, and the obtained file statistic is compared with the preset value threshold, so that the file abnormal result corresponding to the target task is rapidly and accurately generated according to the obtained comparison result, whether the task has the file number abnormality is accurately detected, and the processing efficiency of abnormality detection on the target task is improved. And the generated abnormal alarm information corresponding to the target task is pushed to the target user, so that the target user can perform corresponding processing measures on the target task based on the abnormal alarm information, and the use experience of the target user is improved.

Drawings

In order to more clearly illustrate the solution of the present application, the drawings needed for describing the embodiments of the present application will be briefly described below, and it is obvious that the drawings in the following description are some embodiments of the present application, and that other drawings can be obtained by those skilled in the art without inventive effort.

FIG. 1 is an exemplary system architecture diagram in which the present application may be applied;

FIG. 2 is a flow diagram of one embodiment of a task exception detection method according to the present application;

FIG. 3 is a schematic diagram of the structure of one embodiment of a task anomaly detection apparatus according to the present application;

FIG. 4 is a block diagram of one embodiment of a computer device according to the present application.

Detailed Description

Unless defined otherwise, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application belongs; the terminology used in the description of the application herein is for the purpose of describing particular embodiments only and is not intended to be limiting of the application; the terms "including" and "having," and any variations thereof, in the description and claims of this application and the description of the above figures are intended to cover non-exclusive inclusions. The terms "first," "second," and the like in the description and claims of this application or in the foregoing drawings are used for distinguishing between different objects and not for describing a particular sequential order.

Reference herein to "an embodiment" means that a particular feature, structure, or characteristic described in connection with the embodiment can be included in at least one embodiment of the application. The appearances of the phrase in various places in the specification are not necessarily all referring to the same embodiment, nor are separate or alternative embodiments mutually exclusive of other embodiments. It is explicitly and implicitly understood by one skilled in the art that the embodiments described herein can be combined with other embodiments.

In order to make the technical solutions better understood by those skilled in the art, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the accompanying drawings.

As shown in fig. 1, the system architecture 100 may include

terminal devices

101, 102, 103, a network 104, and a server 105. Network 104 is the medium used to provide communication links between

terminal devices

101, 102, 103 and server 105. Network 104 may include various connection types, such as wired, wireless communication links, or fiber optic cables, to name a few.

The user may use the

terminal devices

101, 102, 103 to interact with the server 105 via the network 104 to receive or send messages or the like. The

terminal devices

101, 102, 103 may have various communication client applications installed thereon, such as a web browser application, a shopping application, a search application, an instant messaging tool, a mailbox client, social platform software, and the like.

The

terminal devices

101, 102, 103 may be various electronic devices having a display screen and supporting web browsing, including but not limited to smart phones, tablet computers, e-book readers, MP3 players (Mov I picture Experts G roup Aud I o Layer I, mpeg compression standard audio Layer 3), MP4 players (Mov I ng P I ctu Experts G roup Aud I o Layer I V, mpeg compression standard audio Layer 4), laptop portable computers, desktop computers, and the like.

The server 105 may be a server providing various services, such as a background server providing support for pages displayed on the

terminal devices

101, 102, 103.

It should be noted that the task abnormality detection method provided in the embodiments of the present application is generally executed by a server/terminal device, and accordingly, the task abnormality detection apparatus is generally disposed in the server/terminal device.

It should be understood that the number of terminal devices, networks, and servers in fig. 1 is merely illustrative. There may be any number of terminal devices, networks, and servers, as desired for implementation.

With continuing reference to FIG. 2, a flowchart of one embodiment of a task anomaly detection method according to the present application is shown. The task abnormity detection method comprises the following steps:

step S201, obtaining target operation log data of a target task in a service system.

In this embodiment, the electronic device (for example, the server/terminal device shown in fig. 1) on which the task abnormality detection method operates may acquire the target operation log data in a wired connection manner or a wireless connection manner. It should be noted that the wireless connection means may include, but is not limited to, a 3G/4G/5G connection, a wifi connection, a bluetooth connection, a wimax connection, a Z i gbee connection, a UWB (u l t ra W i deband) connection, and other wireless connection means now known or developed in the future. The business system can be a Hadoop distributed system, and the target task can be a MapReduce task. MapReduce is a programming model for parallel operation of large-scale data sets (greater than 1 TB). The concepts "Map" and "Reduce" are their main ideas, both borrowed from functional programming languages, and features borrowed from vector programming languages. The method greatly facilitates programmers to operate own programs on the distributed system under the condition of no distributed parallel programming. Current software implementations specify a Map function to Map a set of key-value pairs into a new set of key-value pairs, and a concurrent Reduce function to ensure that each of all mapped key-value pairs share the same key-set. In addition, in the above specific implementation process of acquiring the target operation log data of the target task in the service system, the detailed description will be further described in the following specific embodiments, and will not be set forth herein.

Step S202, loading the running log data into a preset target database table.

In this embodiment, the target database table may be a database table in an h i ve database in a business system, i.e., a Hadoop distributed system. The running log data can be loaded into a preset target database table through an l oad instruction.

Step S203, extracting first data of a first preset event type from the target database table.

In this embodiment, the first preset event type is specifically a JOB _ fn I _ SHED event. The first data belongs to j son file data and represents a data source of an interface of a counter r of a target task in j obj i store.

Step S204, obtaining the file statistics corresponding to the first target field from the first data.

In this embodiment, the first target field is specifically a CREATE _ F I LES field, and the statistical data corresponding to the CREATE _ F I LES field represents how many hdfs files are created after the target task is completed. By analyzing the CREATE _ F I LES field, the number of created files corresponding to the target task can be obtained.

Step S205, if the file statistics is larger than a preset value threshold, generating abnormal alarm information corresponding to the target task.

In this embodiment, the value of the numerical threshold is not particularly limited, and may be determined after an actual data test, and may be set to 8w, for example. The method comprises the steps of obtaining a task name of a target task, obtaining a preset abnormal alarm information template, and filling the task name of the target task into the abnormal alarm information template to generate abnormal alarm information corresponding to the target task. The abnormal alarm information template can be generated by writing of developers according to actual business requirements.

And step S206, pushing the abnormal alarm information to a target user.

In this embodiment, the target user may be an operation and maintenance person related to task maintenance work of the business system. The abnormal alarm information can be pushed to the target user based on the communication information by acquiring the communication information of the target user. The communication information may include a mobile phone number or a mail address.

Firstly, acquiring target running log data of a target task in a service system; then loading the running log data into a preset target database table; then extracting first data of a first preset event type from the target database table; subsequently acquiring file statistics corresponding to the first target field from the first data; and if the file statistics value is larger than a preset value threshold value, generating abnormal alarm information corresponding to the target task, and pushing the abnormal alarm information to a target user. According to the method and the device, the file statistic corresponding to the first preset event type and the first target field is extracted from the target running log data of the target task in the service system, and the obtained file statistic is compared with the preset value threshold, so that the file abnormal result corresponding to the target task is rapidly and accurately generated according to the obtained comparison result, whether the task has the file number abnormality is accurately detected, and the processing efficiency of abnormality detection on the target task is improved. And the generated abnormal alarm information corresponding to the target task is pushed to the target user, so that the target user can perform corresponding processing measures on the target task based on the abnormal alarm information, and the use experience of the target user is improved.

In some optional implementations, step S201 includes the following steps:

and acquiring a target task identifier of the target task.

In this embodiment, the target task identifier may include a task name or task I D of the target task.

And calling a preset task log collection component.

In this embodiment, the task log collection component may be a built-in log collection component in a Hadoop distributed system.

And acquiring appointed running log data corresponding to the target task identifier from a preset storage directory based on the task log acquisition component.

In this embodiment, the storage directory may be a map reduce.

And taking the specified operation log data as the target operation log data.

The method comprises the steps of obtaining a target task identifier of a target task; then calling a preset task log acquisition component; subsequently, based on the task log collection component, acquiring specified running log data corresponding to the target task identifier from a preset storage directory; and taking the specified operation log data as the target operation log data. According to the method and the device, based on the use of the task log collection component, the target operation log data corresponding to the target task can be quickly extracted from the preset storage directory, and the acquisition efficiency of the target operation log data is improved.

In some optional implementation manners of this embodiment, after step S202, the electronic device may further perform the following steps:

and extracting second data corresponding to a second preset event type from the target database table.

In this embodiment, the second predetermined event type is a REDUCE _ ATTEMPT _ fn I _ SHED event. The second data belongs to j son file data and represents a data source of a counter interface of r reducer attempt in j ob i st rory.

And acquiring input quantity statistical data corresponding to a second target field from the second data.

In this embodiment, the second target field is specifically a Reduce i nput receiver ds field. The input quantity statistical data refers to the statistical data of the input quantities of all the reduce processing units corresponding to the target task. The input amount of reduce represents the amount of data that the reduce processing unit needs to process.

And acquiring the processing data amount of each target processing unit corresponding to the target task based on the input amount statistical data.

In this embodiment, the target processing unit is specifically referred to as a reduce processing unit.

Calculating the median of all the processed data quantities.

In this embodiment, the process of calculating the median of all the processed data amounts may refer to the conventional way of calculating the median, and will not be described herein too much.

In this embodiment, the specific implementation process of performing data skew detection processing on the target task based on the processed data amount and the median to generate a corresponding skew detection result is described in further detail in the following specific embodiments, and will not be described herein too much. Wherein the inclination detection result comprises a first inclination detection result of the target task with data inclination risk, or the target task does not have data inclination risk. In addition, after the inclination detection result is generated, the inclination detection result can be pushed to the target user, so that the target user can know whether the target task has the risk of data inclination in time, corresponding processing measures can be carried out on the target task according to the obtained inclination detection result, and the use experience of the target user is improved.

Second data corresponding to a second preset event type is extracted from the target database table; then acquiring input quantity statistical data corresponding to a second target field from the second data; then, based on the input quantity statistical data, acquiring the processing data quantity of each target processing unit corresponding to the target task; subsequently calculating the median of all the processed data; and further performing data skew detection processing on the target task based on the processing data amount and the median to generate a corresponding skew detection result. According to the method and the device, the input quantity statistical data corresponding to the second preset event type and the second target field are extracted from the target database table, and then the target task is subjected to data tilt detection processing quickly and accurately based on the input quantity statistical data, so that the processing efficiency of the data tilt detection of the target task is improved, and the accuracy of the generated tilt detection result is ensured.

In some optional implementations, the performing data skew detection processing on the target task based on the processing data amount and the median to generate a corresponding skew detection result includes:

and acquiring the designated processing data volume with the largest value from all the processing data volumes.

In the present embodiment, the designated processing data amount having the largest value can be obtained from all the processing data amounts by performing numerical comparison on all the processing data amounts.

Calculating a ratio of the specified processing data amount to the median.

And judging whether the ratio is greater than a preset ratio threshold value or not.

In this embodiment, the value of the ratio threshold is not specifically limited, and may be determined after an actual data test, and may be set to 10, for example.

The method comprises the steps of obtaining a designated processing data volume with the largest numerical value from all the processing data volumes; then calculating the ratio of the specified processing data volume to the median; subsequently judging whether the ratio is larger than a preset ratio threshold value or not; if the value is larger than the ratio threshold, generating a first inclination detection result of the target task with data inclination risk, otherwise, generating a second inclination detection result of the target task without data inclination risk. According to the method and the device, the specific processing data volume with the largest numerical value in the processing data volumes and the specific value of the median are calculated, and then the obtained specific value is compared with the preset specific value threshold, so that the inclination detection result corresponding to the target task is rapidly and accurately generated according to the obtained comparison result, the processing efficiency of data inclination detection of the target task is improved, and the accuracy of the generated inclination detection result is ensured.

In some optional implementations, after step S201, the electronic device may further perform the following steps:

and acquiring target operation data corresponding to a preset data type from the target operation log data.

In this embodiment, the data type refers to type information of data capable of reflecting the operation state of the target task, and may include, for example, task processing time and the like.

And inputting the target operation data into a pre-trained anomaly analysis model.

In this embodiment, for the training and generating process of the anomaly analysis model, further details will be described in the following specific embodiments, which are not set forth herein too much.

And performing anomaly analysis on the target operation data through the anomaly identification model to generate an anomaly analysis result corresponding to the target task.

In this embodiment, the abnormality analysis result may include that the target task is operating normally or the target task is operating abnormally.

And pushing the analysis result to the target user.

After target operation log data are obtained, target operation data corresponding to a preset data type are obtained from the target operation log data; and then inputting the target operation data into a pre-trained anomaly analysis model, performing anomaly analysis on the target operation data through the anomaly identification model to generate an anomaly analysis result corresponding to the target task, and pushing the analysis result to the target user. According to the method and the device, the target operation data of the target task can be subjected to abnormal analysis by using the abnormal analysis model, so that the operation state of the target task can be accurately determined in real time, and the accuracy of the generated abnormal analysis result is ensured.

In some optional implementation manners of this embodiment, before the step of inputting the target operation data into the pre-trained anomaly analysis model, the electronic device may further perform the following steps:

and acquiring historical running log data in a preset time period.

In this embodiment, the value of the preset time period is not specifically limited, and may be set according to actual use requirements, for example, may be set within half a year of the current time.

And constructing a training sample and a testing sample based on the historical running log data. The training sample comprises training sample data and a training sample label corresponding to the training sample data, and the test sample comprises test sample data and a test sample label corresponding to the test sample data.

In this embodiment, corresponding data may be randomly screened from the historical operation log data according to a preset division ratio to serve as a training sample and a test sample. Wherein, do not do specifically to the value of above-mentioned partition ratio and do not limit, can set up according to actual user demand, for example can set up to training sample and test sample's ratio and be 8:2. the generation mode of the sample label can adopt a manual marking mode or a machine marking mode, and the sample label can comprise normal task operation or abnormal task operation.

And calling a pre-constructed neural network model.

In this embodiment, the selection of the neural network model is not particularly limited, and may be any one of a random forest network model, a recurrent neural network model, a convolutional neural network model, and a recurrent neural network model, for example.

And taking training sample data in the training samples as the input of the neural network model, taking training sample labels in the training samples as the output of the neural network model, training the neural network model, continuously adjusting parameters of the neural network model until the evaluation index of the trained neural network model meets the expected condition, and taking the trained neural network model as the anomaly analysis model. Wherein the evaluation index is obtained by testing the trained neural network model using the test sample.

In this embodiment, the evaluation index may include a classification accuracy, and if the classification accuracy obtained by checking the trained neural network model using the test sample is greater than a preset accuracy threshold, it is determined that the trained neural network model passes verification. The classification accuracy rate corresponds to the expected condition, and the value of the classification accuracy rate can be set according to actual use requirements. The specific training and generating process of the anomaly analysis model may refer to a training process of a corresponding existing neural network model, which is not described herein too much.

The method comprises the steps of obtaining historical running log data in a preset time period, and constructing a training sample and a testing sample based on the historical running log data; then calling a pre-constructed neural network model; and then training the neural network model by taking training sample data in the training sample as input of the neural network model and taking a training sample label in the training sample as output of the neural network model, continuously adjusting parameters of the neural network model until evaluation indexes of the trained neural network model meet expected conditions, and taking the trained neural network model as the anomaly analysis model. According to the method and the device, the pre-constructed neural network model is trained by using the historical operation log data in the preset time period as sample data, and the required anomaly analysis model can be generated quickly, so that whether the operation state of the task is abnormal or not can be predicted quickly and accurately by using the anomaly analysis model subsequently, namely, the target operation data of the target task can be subjected to anomaly analysis by using the anomaly analysis model, the operation state of the target task can be determined accurately in real time, and the accuracy of the generated anomaly analysis result is ensured.

In some optional implementation manners of this embodiment, after step S205, the electronic device may further perform the following steps:

and generating a termination instruction corresponding to the target task.

In this embodiment, the termination instruction is an instruction for task termination of the target task.

In this embodiment, after receiving the termination instruction, the service system responds to the termination instruction to perform task termination processing on the target task.

According to the method and the device, after the abnormal warning information corresponding to the target task is generated, the fact that the target task has too many files is determined, the termination instruction corresponding to the target task is generated intelligently, the termination instruction is sent to the business system, the business system is controlled based on the termination instruction to carry out task termination processing on the target task, so that the situation that the target task is mistakenly reported in the executing process to cause production accidents can be effectively avoided, the operating speed of the business system is favorably improved, and the normal operation of the business system is guaranteed.

It is emphasized that the file statistics may also be stored in nodes of a block chain in order to further ensure privacy and security of the file statistics.

The block chain referred by the application is a novel application mode of computer technologies such as distributed data storage, point-to-point transmission, a consensus mechanism, an encryption algorithm and the like. The blockchain (B l ockcha i n), which is essentially a decentralized database, is a string of data blocks associated by using a cryptographic method, and each data block contains information of a batch of network transactions for verifying the validity (anti-counterfeiting) of the information and generating a next block. The blockchain may include a blockchain underlying platform, a platform product service layer, an application service layer, and the like.

The embodiment of the application can acquire and process related data based on an artificial intelligence technology. The artificial intelligence (Art I f I c I a l I nte l I gene, A I) is a theory, method, technology and application system for simulating, extending and expanding human intelligence, sensing environment, acquiring knowledge and using knowledge to obtain optimal results by using a digital computer or a machine controlled by the digital computer.

The artificial intelligence infrastructure generally includes technologies such as sensors, dedicated artificial intelligence chips, cloud computing, distributed storage, big data processing technologies, operation/interaction systems, mechatronics, and the like. The artificial intelligence software technology mainly comprises a computer vision technology, a robot technology, a biological recognition technology, a voice processing technology, a natural language processing technology, machine learning/deep learning and the like.

It will be understood by those skilled in the art that all or part of the processes of the methods of the embodiments described above can be implemented by hardware associated with computer readable instructions, which can be stored in a computer readable storage medium, and when executed, the processes of the embodiments of the methods described above can be included. The storage medium may be a non-volatile storage medium such as a magnetic disk, an optical disk, a Read-only Memory (ROM), or a Random Access Memory (RAM).

It should be understood that, although the steps in the flowcharts of the figures are shown in order as indicated by the arrows, the steps are not necessarily performed in order as indicated by the arrows. The steps are not performed in the exact order shown and may be performed in other orders unless explicitly stated herein. Moreover, at least a portion of the steps in the flow chart of the figure may include multiple sub-steps or multiple stages, which are not necessarily performed at the same time, but may be performed at different times, which are not necessarily performed in sequence, but may be performed alternately or alternately with other steps or at least a portion of the sub-steps or stages of other steps.

With further reference to fig. 3, as an implementation of the method shown in fig. 2, the present application provides an embodiment of a task anomaly detection apparatus, where the embodiment of the apparatus corresponds to the embodiment of the method shown in fig. 2, and the apparatus may be specifically applied to various electronic devices.

As shown in fig. 3, the task abnormality detection apparatus 300 according to the present embodiment includes: a first obtaining module 301, a loading module 302, a first extracting module 303, a second obtaining module 304, a first generating module 305, and a first pushing module 306. Wherein:

a first obtaining module 301, configured to obtain target operation log data of a target task in a service system;

a loading module 302, configured to load the running log data into a preset target database table;

a first extraction module 303, configured to extract first data corresponding to a first preset event type from the target database table;

a second obtaining module 304, configured to obtain a file statistic corresponding to the first target field from the first data;

a first generating module 305, configured to generate abnormal warning information corresponding to the target task if the file statistics is greater than a preset value threshold;

the first pushing module 306 is configured to push the abnormal alarm information to a target user.

In this embodiment, the operations performed by the modules or units respectively correspond to the steps of the task abnormality detection method in the foregoing embodiment one by one, and are not described herein again.

In some optional implementations of this embodiment, the first obtaining module 301 includes:

the first obtaining submodule is used for obtaining a target task identifier of the target task;

the calling submodule is used for calling a preset task log acquisition component;

the second acquisition sub-module is used for acquiring specified running log data corresponding to the target task identifier from a preset storage directory based on the task log acquisition component;

and the determining submodule is used for taking the specified operation log data as the target operation log data.

In this embodiment, the operations executed by the modules or units respectively correspond to the steps of the task abnormality detection method in the foregoing embodiment one by one, and are not described herein again.

In some optional implementations of this embodiment, the task exception detecting device further includes:

the second extraction module is used for extracting second data corresponding to a second preset event type from the target database table;

the third acquisition module is used for acquiring input quantity statistical data corresponding to a second target field from the second data;

a fourth obtaining module, configured to obtain, based on the input amount statistical data, a processing data amount of each target processing unit corresponding to the target task;

the calculation module is used for calculating the median of all the processing data volumes;

and the detection module is used for carrying out data inclination detection processing on the target task based on the processing data amount and the median and generating a corresponding inclination detection result.

In this embodiment, the operations that the modules or units are respectively used to execute correspond to the steps of the task exception detection method in the foregoing embodiment one to one, and are not described herein again.

In some optional implementations of this embodiment, the detecting module includes:

the third acquisition submodule is used for acquiring the designated processing data volume with the maximum numerical value from all the processing data volumes;

the calculation submodule is used for calculating the ratio of the specified processing data quantity to the median;

the judgment submodule is used for judging whether the ratio is larger than a preset ratio threshold value or not;

and the generation submodule is used for generating a first inclination detection result of the target task with the data inclination risk if the ratio is larger than the ratio threshold, and otherwise, generating a second inclination detection result of the target task without the data inclination risk.

a fifth obtaining module, configured to obtain target operation data corresponding to a preset data type from the target operation log data;

the input module is used for inputting the target operation data into a pre-trained anomaly analysis model;

the second generation module is used for performing anomaly analysis on the target operation data through the anomaly identification model to generate an anomaly analysis result corresponding to the target task;

and the second pushing module is used for pushing the analysis result to the target user.

the sixth acquisition module is used for acquiring historical running log data in a preset time period;

the construction module is used for constructing a training sample and a testing sample based on the historical running log data; the training sample comprises training sample data and a training sample label corresponding to the training sample data, and the test sample comprises test sample data and a test sample label corresponding to the test sample data;

the calling module is used for calling a pre-constructed neural network model;

a training module, configured to use training sample data in the training sample as input of the neural network model, use a training sample label in the training sample as output of the neural network model, train the neural network model, continuously adjust parameters of the neural network model until an evaluation index of the trained neural network model meets an expected condition, and use the trained neural network model as the anomaly analysis model; wherein the evaluation index is obtained by checking the trained neural network model using the test sample.

a third generation module, configured to generate a termination instruction corresponding to the target task;

and the sending module is used for sending the termination instruction to the service system so as to control the service system to execute task termination processing on the target task based on the termination instruction.

In order to solve the technical problem, an embodiment of the present application further provides a computer device. Referring to fig. 4, fig. 4 is a block diagram of a basic structure of a computer device according to the present embodiment.

The computer device 4 comprises a memory 41, a processor 42, a network interface 43 communicatively connected to each other via a system bus. It is noted that only computer device 4 having components 41-43 is shown, but it is understood that not all of the shown components are required to be implemented, and that more or fewer components may be implemented instead. AS will be understood by those skilled in the art, the computer device herein is a device capable of automatically performing numerical calculation and/or information processing according to a preset or stored instruction, and the hardware includes, but is not limited to, a microprocessor, an application specific integrated circuit (App I cat I on Spec I C I integrated C I rcu I, AS ic), a programmable Gate array (F I l D-programmable ab l Gate Ar ray, FPGA), a digital Processor (D I ta l S I gna l Processor, DSP), an embedded device, and the like.

The computer device can be a desktop computer, a notebook, a palm computer, a cloud server and other computing devices. The computer equipment can carry out man-machine interaction with a user in a keyboard mode, a mouse mode, a remote controller mode, a touch panel mode or a voice control equipment mode.

The memory 41 includes at least one type of readable storage medium including a flash memory, a hard disk, a multimedia card, a card type memory (e.g., SD or DX memory, etc.), a Random Access Memory (RAM), a Static Random Access Memory (SRAM), a Read Only Memory (ROM), an Electrically Erasable Programmable Read Only Memory (EEPROM), a Programmable Read Only Memory (PROM), a magnetic memory, a magnetic disk, an optical disk, etc. In some embodiments, the memory 41 may be an internal storage unit of the computer device 4, such as a hard disk or a memory of the computer device 4. In other embodiments, the memory 41 may also be an external storage device of the computer device 4, such as a plug-in hard disk, a Smart Memory Card (SMC), a secure digital (Secu re D i g i ta l, SD) Card, a flash memory Card (F l ash Card), and the like, which are provided on the computer device 4. Of course, the memory 41 may also include both internal and external storage devices of the computer device 4. In this embodiment, the memory 41 is generally used for storing an operating system installed in the computer device 4 and various types of application software, such as computer readable instructions of a task exception detection method. Further, the memory 41 may also be used to temporarily store various types of data that have been output or are to be output.

The processor 42 may be a Central Processing Unit (CPU), controller, microcontroller, microprocessor, or other data processing chip in some embodiments. The processor 42 is typically used to control the overall operation of the computer device 4. In this embodiment, the processor 42 is configured to execute computer readable instructions stored in the memory 41 or process data, for example, execute computer readable instructions of the task exception detection method.

The network interface 43 may comprise a wireless network interface or a wired network interface, and the network interface 43 is generally used for establishing communication connection between the computer device 4 and other electronic devices.

in the embodiment of the application, target running log data of a target task in a service system is obtained firstly; then loading the running log data into a preset target database table; then extracting first data of a first preset event type from the target database table; subsequently acquiring file statistics corresponding to the first target field from the first data; and if the file statistics is larger than a preset value threshold, generating abnormal alarm information corresponding to the target task, and pushing the abnormal alarm information to a target user. According to the method and the device, the file statistic corresponding to the first preset event type and the first target field is extracted from the target running log data of the target task in the service system, and the obtained file statistic is compared with the preset value threshold, so that the file abnormal result corresponding to the target task is rapidly and accurately generated according to the obtained comparison result, whether the task has the file number abnormality is accurately detected, and the processing efficiency of abnormality detection on the target task is improved. And the generated abnormal alarm information corresponding to the target task is pushed to the target user, so that the target user can perform corresponding processing measures on the target task based on the abnormal alarm information, and the use experience of the target user is improved.

The present application further provides another embodiment, which is to provide a computer-readable storage medium storing computer-readable instructions executable by at least one processor to cause the at least one processor to perform the steps of the task exception detection method as described above.

in the embodiment of the application, target running log data of a target task in a service system is obtained firstly; then loading the running log data into a preset target database table; then extracting first data of a first preset event type from the target database table; subsequently acquiring a file statistic corresponding to a first target field from the first data; and if the file statistics is larger than a preset value threshold, generating abnormal alarm information corresponding to the target task, and pushing the abnormal alarm information to a target user. According to the method and the device, the file statistic corresponding to the first preset event type and the first target field is extracted from the target running log data of the target task in the service system, and the obtained file statistic is compared with the preset value threshold, so that the file abnormal result corresponding to the target task is rapidly and accurately generated according to the obtained comparison result, whether the task has the file number abnormality is accurately detected, and the processing efficiency of abnormality detection on the target task is improved. And the generated abnormal alarm information corresponding to the target task is pushed to the target user, so that the target user can perform corresponding processing measures on the target task based on the abnormal alarm information, and the use experience of the target user is improved.

Through the description of the foregoing embodiments, it is clear to those skilled in the art that the method of the foregoing embodiments may be implemented by software plus a necessary general hardware platform, and certainly may also be implemented by hardware, but in many cases, the former is a better implementation. Based on such understanding, the technical solutions of the present application may be embodied in the form of a software product, which is stored in a storage medium (such as ROM/RAM, magnetic disk, optical disk) and includes instructions for enabling a terminal device (such as a mobile phone, a computer, a server, an air conditioner, or a network device) to execute the method according to the embodiments of the present application.

It is to be understood that the above-described embodiments are merely illustrative of some, but not restrictive, of the broad invention, and that the appended drawings illustrate preferred embodiments of the invention and do not limit the scope of the invention. This application is capable of embodiments in many different forms and is provided for the purpose of enabling a thorough understanding of the disclosure of the application. Although the present application has been described in detail with reference to the foregoing embodiments, it will be apparent to one skilled in the art that the present application may be practiced without modification or with equivalents of some of the features described in the foregoing embodiments. All equivalent structures made by using the contents of the specification and the drawings of the present application are directly or indirectly applied to other related technical fields and are within the protection scope of the present application.

Claims

1. A task abnormity detection method is characterized by comprising the following steps:

acquiring target running log data of a target task in a service system;

loading the running log data into a preset target database table;

and pushing the abnormal alarm information to a target user.

2. The method for detecting task exception according to claim 1, wherein the step of obtaining target running log data of a target task in a business system specifically includes:

acquiring a target task identifier of the target task;

calling a preset task log acquisition component;

and taking the specified operation log data as the target operation log data.

3. The task anomaly detection method according to claim 1, wherein after the step of loading the running log data into a preset target database table, the method further comprises:

calculating the median of all the processed data quantities;

and performing data tilt detection processing on the target task based on the processing data amount and the median to generate a corresponding tilt detection result.

4. The task anomaly detection method according to claim 3, wherein the step of performing data skew detection processing on the target task based on the processing data amount and the median to generate a corresponding skew detection result specifically comprises:

calculating the ratio of the specified processing data amount to the median;

judging whether the ratio is larger than a preset ratio threshold value or not;

if the target task is larger than the ratio threshold, generating a first inclination detection result of the target task with data inclination risk, otherwise, generating a second inclination detection result of the target task without data inclination risk.

5. The method for detecting task anomaly according to claim 1, wherein after the step of obtaining target running log data of a target task in a business system, the method further comprises:

inputting the target operation data into a pre-trained anomaly analysis model;

and pushing the analysis result to the target user.

6. The task abnormality detection method according to claim 5, characterized by further comprising, before the step of inputting the target operation data into a pre-trained abnormality analysis model:

acquiring historical running log data in a preset time period;

calling a pre-constructed neural network model;

taking training sample data in the training samples as input of the neural network model, taking training sample labels in the training samples as output of the neural network model, training the neural network model, continuously adjusting parameters of the neural network model until evaluation indexes of the trained neural network model meet expected conditions, and taking the trained neural network model as the abnormal analysis model; wherein the evaluation index is obtained by checking the trained neural network model using the test sample.

7. The method according to claim 1, wherein after the step of generating the abnormal warning information corresponding to the target task if the file statistics is greater than a preset value threshold, the method further comprises:

generating a termination instruction corresponding to the target task;

8. A task abnormality detection apparatus characterized by comprising:

9. A computer device comprising a memory having computer readable instructions stored therein and a processor which when executed implements the steps of the task exception detection method of any one of claims 1 to 7.

10. A computer readable storage medium having computer readable instructions stored thereon which, when executed by a processor, implement the steps of the task anomaly detection method of any one of claims 1 to 7.