CN113536381A - Big data analysis processing method and system based on terminal - Google Patents
Big data analysis processing method and system based on terminal Download PDFInfo
- Publication number
- CN113536381A CN113536381A CN202110884930.8A CN202110884930A CN113536381A CN 113536381 A CN113536381 A CN 113536381A CN 202110884930 A CN202110884930 A CN 202110884930A CN 113536381 A CN113536381 A CN 113536381A
- Authority
- CN
- China
- Prior art keywords
- data
- module
- information
- terminal
- big data
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Withdrawn
Links
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F21/00—Security arrangements for protecting computers, components thereof, programs or data against unauthorised activity
- G06F21/60—Protecting data
- G06F21/62—Protecting access to data via a platform, e.g. using keys or access control rules
- G06F21/6218—Protecting access to data via a platform, e.g. using keys or access control rules to a system of files or objects, e.g. local or distributed file system or database
- G06F21/6245—Protecting personal data, e.g. for financial or medical purposes
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F21/00—Security arrangements for protecting computers, components thereof, programs or data against unauthorised activity
- G06F21/60—Protecting data
- G06F21/606—Protecting data by securing the transmission between two devices or processes
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F21/00—Security arrangements for protecting computers, components thereof, programs or data against unauthorised activity
- G06F21/60—Protecting data
- G06F21/62—Protecting access to data via a platform, e.g. using keys or access control rules
- G06F21/6218—Protecting access to data via a platform, e.g. using keys or access control rules to a system of files or objects, e.g. local or distributed file system or database
- G06F21/6227—Protecting access to data via a platform, e.g. using keys or access control rules to a system of files or objects, e.g. local or distributed file system or database where protection concerns the structure of data, e.g. records, types, queries
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Health & Medical Sciences (AREA)
- Bioethics (AREA)
- General Health & Medical Sciences (AREA)
- Software Systems (AREA)
- Computer Security & Cryptography (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Computer Hardware Design (AREA)
- Databases & Information Systems (AREA)
- Medical Informatics (AREA)
- Telephonic Communication Services (AREA)
Abstract
The invention belongs to the technical field of data processing, and discloses a big data analysis processing method and a big data analysis processing system based on a terminal, wherein the big data analysis processing system based on the terminal comprises: the system comprises a data acquisition module, a data preprocessing module, a permission management module, a wireless signal transmission module, a central processing and control module, a data interaction module, a network server module, a safety detection module, a warning module, a local storage module and a man-machine interaction module. According to the invention, the data acquisition module is used for acquiring data and transmitting the acquired data information to the data analysis processing module, and the safety detection module is used for detecting the safety of the data, so that the intrusion of dangerous data can be effectively avoided; the corresponding relation between each step of the preprocessing flow and the program is established by correspondingly setting the configuration files according to the attributes of the data to be processed in advance, wherein the configuration files corresponding to the data to be processed with the same attribute are the same, so that the workload and the error rate are reduced, and the analysis processing efficiency of the data is improved.
Description
Technical Field
The invention belongs to the technical field of data processing, and particularly relates to a big data analysis processing method and system based on a terminal.
Background
Currently, with the rapid development of information technology, mobile terminals and high-speed mobile networks provide users with rich information and resources, and users need to download a large amount of applications to the mobile terminals via wireless mobile networks while working, living and entertaining with the information and resources. Various application programs for improving user experience exist in the application market of the intelligent mobile terminal, and a series of safety problems are brought while users enjoy convenience. Firstly, the network gradually becomes a way for spreading malicious applications, and after the applications are downloaded, stored and installed in a local terminal to run, some files in the local terminal can be modified maliciously, so that the system is paralyzed or the running is slowed down, and secondly, the risk of revealing individual privacy is brought, wherein the individual privacy comprises personal identity, bank account, financial status information, behavior preference, health condition, social status, social records and other private information of a user. By mining specific data of a single user, a large amount of diversified information intersection of the application program and the associated malicious network resources or analysis tools can finally accurately depict the user outline, such as personal age, economic condition, consumption behavior and level, social status, social circle and the like, and further bring forth some new privacy risks and ethical safety problems to be solved urgently. Therefore, the installed application needs to be detected, and if malicious attempts exist, the detection and killing needs to be performed, however, the detection and killing in the prior art have a series of problems. For the searching and killing of the malicious application program, the malicious program is generally deleted after being detected out so as to avoid the malicious program from executing malicious behaviors, but the source of the malicious program cannot be traced, so that the source of the malicious program cannot be thoroughly searched and killed, and the source of the malicious program cannot be cut off. Moreover, analysis of malicious applications includes both static analysis and dynamic analysis. Static analysis is simple and fast, but requires knowledge of information of known malicious applications, such as signatures, behavior patterns, permission applications, etc., prior to scanning. Dynamic analysis runs and monitors applications in a closed environment and analyzes behavioral characteristics of the applications, such as file permission changes, process and thread running conditions, system call conditions, network access conditions, and the like. However, the analysis efficiency of the method is not ideal whether static analysis or dynamic analysis is adopted. In addition, malicious newly installed applications often attempt to access the privacy trust of the terminal; although some application programs have legal authority to legally access privacy information of a user, such as incoming short message service, the prior art lacks effective file protection on the privacy of the airborne existing user and reasonable management on the access of the privacy information, so that the installed application programs steal the privacy information of the user, further the assets and the privacy of the user are leaked, and irreparable loss is caused.
Through the above analysis, the problems and defects of the prior art are as follows: the existing data processing mode has too low analysis efficiency, cannot meet the existing use requirements, cannot ensure the safety of data information in the existing program, is easy to cause the leakage of assets and privacy of users, and causes irreparable loss.
Disclosure of Invention
Aiming at the problems in the prior art, the invention provides a big data analysis and processing method and a big data analysis and processing system based on a terminal.
The invention is realized in such a way that a big data analysis processing method based on a terminal comprises the following steps:
inputting preset parameters through a human-computer interaction module, and selecting the type of data to be analyzed and processed according to the demand information;
reading the category information of the information to be collected contained in the control instruction through a data acquisition module according to a preset instruction, and collecting the category information in local storage and network storage according to the category information;
step three, storing the collected data into a data cache server to generate original data, classifying the data, storing the classified data into the data cache server again according to different categories to generate classified data;
step four, calling the classification data by using terminal equipment, comparing the original data with the classification data called by the data calling signal, outputting the two groups of data to a central processing and control module after the two groups of data are compared, and realizing the extraction and collection of locally stored data or network receiving data;
acquiring data to be processed through a data preprocessing module, and acquiring a configuration file matched with the data to be processed according to the attribute of the data to be processed;
step six, determining a current preprocessing mode corresponding to data preprocessing, and controlling each operating node to process the obtained processing result according to a next preprocessing mode of the current preprocessing mode after processing the data to be processed according to the current preprocessing mode;
step seven, respectively acquiring programs corresponding to the steps according to the steps of the pre-processing flow, and processing the data to be processed at each operation node according to the current pre-processing mode; executing the programs corresponding to the steps according to the execution sequence of the steps to realize the pretreatment of the data to be treated;
step eight, providing different authority levels for different users through the authority management module so as to ensure the safety of the architecture; data transmission is carried out between the wireless signal transmitter and a remote server through a wireless signal transmission module; performing data interaction with a remote server through a data interaction module according to the control instruction;
processing and analyzing the acquired data by using a central processing unit through a central processing and controlling module, and performing coordination control on each controlled module of the terminal-based big data analyzing and processing system according to a processing result and preset parameters;
step ten, storing the big data information and remotely analyzing and processing the acquired information by using a remote server through a network server module; the safety of the collected big data information is detected through a safety detection module, and when the safety information is detected to exceed a threshold value preset by an administrator, a warning message is sent to the administrator through a warning module;
eleven, storing the information to be processed or the network receiving information, the data preprocessing result, the authority level, the remote analysis processing result, the safety detection result and the warning message by using a storage hard disk through a local storage module; and updating and displaying the working state of the system, the information to be processed or the network receiving information, the data preprocessing result, the authority level, the remote analysis processing result, the safety detection result and the real-time data of the warning message through the man-machine interaction module.
Further, in step three, the classification method for classifying data includes:
(1) preprocessing a data text, and respectively generating a word vector training set and a BERT language training set;
(2) training each of a plurality of neural network models through the word vector training set respectively, and training a BERT classification model through the BERT language training set;
(3) and determining a fusion model according to the training result, and performing labeling classification on the target data according to the fusion model.
Further, in the fifth step, the configuration file matched with the data to be processed includes the identifier and the operation parameters of the preprocessing operation included in the step of the preprocessing flow.
Further, in the tenth step, the security of the collected big data information is detected by a security detection module, which includes:
(1) when the acquired big data information is used for carrying out security detection on a terminal, acquiring operation parameters of data to be detected and analyzing the operation parameters;
(2) acquiring a security authentication result of big data information; the terminal equipment determines whether to keep the information in the terminal or delete the information according to the security authentication result of the big data information;
(3) this information is sent to the network server while retained to update the database for big data analysis, decision making and validation.
Further, the obtaining of the security authentication result of the big data information includes:
1) the system sends a preset authentication mode of the big data information to the terminal to prompt the user to select multiple authentication modes from the preset authentication mode to authenticate the big data information;
2) acquiring a plurality of authentication modes selected by a user and authentication information of big data information respectively input in the plurality of authentication modes;
3) and obtaining a security authentication result of the big data information based on the preset weight corresponding to each authentication mode and based on each authentication information authentication.
Further, in the eleventh step, storing the information to be processed or the network reception information, the data preprocessing result, the permission level, the remote analysis processing result, the security detection result, and the warning message by using the storage hard disk through the local storage module includes:
(1) calculating the size of the data information to be stored to obtain the data volume of the information to be stored;
(2) if the data volume of the information to be stored is larger than the maximum data volume of unit information which can be stored and is preset in a storage hard disk, splitting the information to be stored into at least two pieces of segment information, so that the data volume of each piece of segment information is smaller than or equal to the maximum data volume;
(3) and storing each piece of segment information and a segment identifier corresponding to each piece of segment information in the storage hard disk.
Further, the data volume includes a data source identifier, a time identifier and a data stream size of the data to be processed.
Another object of the present invention is to provide a terminal-based big data analysis processing system using the terminal-based big data analysis processing method, the terminal-based big data analysis processing system including:
the system comprises a data acquisition module, a data preprocessing module, a permission management module, a wireless signal transmission module, a central processing and control module, a data interaction module, a network server module, a safety detection module, a warning module, a local storage module and a man-machine interaction module.
The data acquisition module is connected with the central processing and control module and used for extracting and collecting locally stored data or data received by a network through terminal equipment;
the data preprocessing module is connected with the central processing and control module and is used for preprocessing data to be processed;
the authority management module is connected with the central processing and control module and is used for providing different authority levels for different users so as to ensure the safety of the architecture;
the wireless signal transmission module is connected with the central processing and control module and is used for carrying out data transmission with the remote server through the wireless signal transmitter;
the central processing and control module is connected with the data acquisition module, the data preprocessing module, the authority management module, the wireless signal transmission module, the data interaction module, the network server module, the safety detection module, the warning module, the local storage module and the man-machine interaction module, and is used for processing and analyzing the acquired data through the central processing unit and carrying out coordination control on each controlled module of the terminal-based big data analysis and processing system according to a processing result and preset parameters;
the data interaction module is connected with the central processing and control module and is used for carrying out data interaction with the remote server according to the control instruction;
the network server module is connected with the central processing and control module and used for storing the big data information and remotely analyzing and processing the acquired information through a remote server;
the safety detection module is connected with the central processing and control module and is used for detecting the safety of the acquired big data information;
the warning module is connected with the central processing and control module and is used for sending warning information to an administrator when detecting that the safety information exceeds a threshold value preset by the administrator;
the local storage module is connected with the central processing and control module and used for storing the information to be processed or the network receiving information, the data preprocessing result, the authority level, the remote analysis processing result, the safety detection result and the warning message through the storage hard disk;
and the human-computer interaction module is connected with the central processing and control module and used for inputting preset parameters and updating and displaying the working state of the system, the information to be processed or the network receiving information, the data preprocessing result, the authority level, the remote analysis processing result, the safety detection result and the real-time data of the warning message.
Another object of the present invention is to provide a computer program product stored on a computer readable medium, comprising a computer readable program for providing a user input interface to implement the terminal-based big data analysis processing method when executed on an electronic device.
Another object of the present invention is to provide a computer-readable storage medium storing instructions that, when executed on a computer, cause the computer to perform the terminal-based big data analysis processing method.
By combining all the technical schemes, the invention has the advantages and positive effects that: according to the big data analysis processing method based on the terminal, data acquisition is carried out through the data acquisition module, acquired data information is transmitted into the data analysis processing module, the safety of the data is detected through the safety detection module, and intrusion of dangerous data can be effectively avoided; the configuration files are respectively and correspondingly set in advance according to the attributes of the data to be processed, the configuration files corresponding to the data to be processed with the same attribute are the same, the corresponding relation between each step of the preprocessing flow and the program is established, the workload and the error rate are reduced, and the data analysis processing efficiency is effectively improved.
Meanwhile, the data collection module can collect all behavior data of the user on the user terminal in a unified way, then the data collected in the data collection module in a unified way are classified according to different indexes through the data classification module, the behavior of primarily selecting and calling required data from all disordered data is effectively saved, the data calling time is shortened, before the data are transmitted to the central processing through the data output module, the called classified data are compared with the original data stored in the data cache server through data comparison, the data damage is avoided, the lossy data are prevented from being called, and the analysis result error caused by the mistake of the data calling is avoided.
Drawings
In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings needed to be used in the embodiments of the present application will be briefly described below, and it is obvious that the drawings described below are only some embodiments of the present application, and it is obvious for those skilled in the art that other drawings can be obtained from the drawings without creative efforts.
Fig. 1 is a flowchart of a terminal-based big data analysis processing method according to an embodiment of the present invention.
Fig. 2 is a block diagram of a terminal-based big data analysis processing system according to an embodiment of the present invention;
in the figure: 1. a data acquisition module; 2. a data preprocessing module; 3. a rights management module; 4. a wireless signal transmission module; 5. a central processing and control module; 6. a data interaction module; 7. a network server module; 8. a security detection module; 9. a warning module; 10. a local storage module; 11. and a man-machine interaction module.
Fig. 3 is a flowchart of a method for extracting and collecting locally stored data or data received by a network by using a terminal device through a data acquisition module according to a preset instruction according to an embodiment of the present invention.
Fig. 4 is a flowchart of a method for preprocessing data to be processed by a data preprocessing module according to an embodiment of the present invention.
Fig. 5 is a flowchart of a method for storing information to be processed or network reception information, a data preprocessing result, an authority level, a remote analysis processing result, a security detection result, and an alert message by using a storage hard disk through a local storage module according to an embodiment of the present invention.
Detailed Description
In order to make the objects, technical solutions and advantages of the present invention more apparent, the present invention is further described in detail with reference to the following embodiments. It should be understood that the specific embodiments described herein are merely illustrative of the invention and are not intended to limit the invention.
In view of the problems in the prior art, the present invention provides a method and a system for analyzing and processing big data based on a terminal, and the present invention is described in detail below with reference to the accompanying drawings.
As shown in fig. 1, the method for analyzing and processing big data based on a terminal according to the embodiment of the present invention includes the following steps:
s101, inputting preset parameters through a human-computer interaction module, and selecting the type of data to be analyzed according to the demand information;
s102, extracting and collecting locally stored data or data received by a network by using terminal equipment through a data acquisition module according to a preset instruction;
s103, preprocessing the data to be processed through a data preprocessing module, and storing the information to be processed or the network receiving information through a local storage module;
s104, providing different authority levels for different users through the authority management module to ensure the safety of the architecture; data transmission is carried out between the wireless signal transmitter and a remote server through a wireless signal transmission module; performing data interaction with a remote server through a data interaction module according to the control instruction;
s105, processing and analyzing the acquired data by using a central processing unit through a central processing and controlling module, and performing coordination control on each controlled module of the terminal-based big data analysis and processing system according to a processing result and preset parameters;
s106, storing big data information and remotely analyzing and processing acquired information by using a remote server through a network server module;
s107, detecting the safety of the acquired big data information through a safety detection module, and sending a warning message to an administrator through a warning module when the safety information is detected to exceed a threshold value preset by the administrator;
s108, storing the information to be processed or the network receiving information, the data preprocessing result, the authority level, the remote analysis processing result, the safety detection result and the warning message by using the storage hard disk through the local storage module;
and S109, updating and displaying the working state of the system, the information to be processed or the network receiving information, the data preprocessing result, the authority level, the remote analysis processing result, the safety detection result and the real-time data of the warning message through the human-computer interaction module.
In step S107 provided in the embodiment of the present invention, detecting the security of the collected big data information by using a security detection module includes:
(1) when the acquired big data information is used for carrying out security detection on a terminal, acquiring operation parameters of data to be detected and analyzing the operation parameters;
(2) acquiring a security authentication result of big data information; the terminal equipment determines whether to keep the information in the terminal or delete the information according to the security authentication result of the big data information;
(3) this information is sent to the network server while retained to update the database for big data analysis, decision making and validation.
The security authentication result for acquiring the big data information provided by the embodiment of the invention comprises the following steps:
1) the system sends a preset authentication mode of the big data information to the terminal to prompt the user to select multiple authentication modes from the preset authentication mode to authenticate the big data information;
2) acquiring a plurality of authentication modes selected by a user and authentication information of big data information respectively input in the plurality of authentication modes;
3) and obtaining a security authentication result of the big data information based on the preset weight corresponding to each authentication mode and based on each authentication information authentication.
As shown in fig. 2, the terminal-based big data analysis processing system provided in the embodiment of the present invention includes: the system comprises a data acquisition module 1, a data preprocessing module 2, a right management module 3, a wireless signal transmission module 4, a central processing and control module 5, a data interaction module 6, a network server module 7, a safety detection module 8, a warning module 9, a local storage module 10 and a human-computer interaction module 11.
The data acquisition module 1 is connected with the central processing and control module 5 and used for extracting and collecting locally stored data or data received by a network through terminal equipment;
the data preprocessing module 2 is connected with the central processing and control module 5 and is used for preprocessing data to be processed;
the authority management module 3 is connected with the central processing and control module 5 and is used for providing different authority levels for different users so as to ensure the safety of the architecture;
the wireless signal transmission module 4 is connected with the central processing and control module 5 and is used for carrying out data transmission with a remote server through a wireless signal transmitter;
the central processing and control module 5 is connected with the data acquisition module 1, the data preprocessing module 2, the authority management module 3, the wireless signal transmission module 4, the data interaction module 6, the network server module 7, the safety detection module 8, the warning module 9, the local storage module 10 and the man-machine interaction module 11, and is used for processing and analyzing the acquired data through the central processing unit and carrying out coordination control on each controlled module of the terminal-based big data analysis and processing system according to a processing result and preset parameters;
the data interaction module 6 is connected with the central processing and control module 5 and is used for carrying out data interaction with the remote server according to the control instruction;
the network server module 7 is connected with the central processing and control module 5 and used for storing the big data information and remotely analyzing and processing the acquired information through a remote server;
the safety detection module 8 is connected with the central processing and control module 5 and is used for detecting the safety of the acquired big data information;
the warning module 9 is connected with the central processing and control module 5 and is used for sending a warning message to an administrator when detecting that the safety information exceeds a threshold value preset by the administrator;
the local storage module 10 is connected with the central processing and control module 5 and used for storing the information to be processed or the network receiving information, the data preprocessing result, the authority level, the remote analysis processing result, the safety detection result and the warning message through a storage hard disk;
and the human-computer interaction module 11 is connected with the central processing and control module 5 and is used for inputting preset parameters and updating and displaying the working state of the system, information to be processed or network receiving information, data preprocessing results, authority levels, remote analysis processing results, safety detection results and real-time data of the warning messages.
The invention is further described with reference to specific examples.
Example 1
As shown in fig. 1 and fig. 3, the method for analyzing and processing big data based on a terminal according to an embodiment of the present invention, as a preferred embodiment, includes:
s201, reading the category information of the information required to be collected contained in the control instruction, and collecting the category information in local storage and network storage according to the category information;
s202, storing the collected data into a data cache server to generate original data, classifying the data, storing the classified data into the data cache server again according to different categories to generate classified data;
and S203, calling the classification data, comparing the original data with the classification data called by the data calling signal, and outputting the two groups of data to a central processing and control module after the two groups of data are compared.
The classification method for classifying data in the embodiment of the invention comprises the following steps:
(1) preprocessing a data text, and respectively generating a word vector training set and a BERT language training set;
(2) training each of a plurality of neural network models through the word vector training set respectively, and training a BERT classification model through the BERT language training set;
(3) and determining a fusion model according to the training result, and performing labeling classification on the target data according to the fusion model.
Example 2
As shown in fig. 1 and fig. 4, the method for analyzing and processing big data based on a terminal according to an embodiment of the present invention includes, as a preferred embodiment, a method for preprocessing data to be processed by a data preprocessing module, including:
s301, acquiring data to be processed, and acquiring a configuration file matched with the data to be processed according to the attribute of the data to be processed;
s302, determining a current preprocessing mode corresponding to data preprocessing, and controlling each operating node to process an obtained processing result according to a next preprocessing mode of the current preprocessing mode after processing data to be processed according to the current preprocessing mode;
s303, respectively acquiring programs corresponding to the steps according to the steps of the pre-processing flow, and processing the data to be processed at each operation node according to the current pre-processing mode;
s304, executing the programs corresponding to the steps according to the execution sequence of the steps, and realizing the pretreatment of the data to be treated.
The configuration file matched with the data to be processed in the embodiment of the invention comprises the identification and the operation parameters of the preprocessing operation contained in the step of the preprocessing flow.
Example 3
Fig. 1 shows a method for analyzing and processing big data based on a terminal according to an embodiment of the present invention, and fig. 5 shows a method for storing information to be processed or network reception information, a data preprocessing result, an authority level, a remote analysis processing result, a security detection result, and an alert message, which are to be processed, by using a storage hard disk through a local storage module according to an embodiment of the present invention, the method including:
s401, calculating the size of the data information to be stored to obtain the data volume of the information to be stored;
s402, if the data volume of the information to be stored is larger than the maximum data volume of unit information which can be stored and is preset in a storage hard disk, splitting the information to be stored into at least two pieces of segment information, so that the data volume of each piece of segment information is smaller than or equal to the maximum data volume;
s403, storing each piece of segment information and the segment identifier corresponding to each piece of segment information in the storage hard disk.
The data volume in the embodiment of the invention comprises the data source identification, the time identification and the data flow size of the data to be processed.
In the above embodiments, the implementation may be wholly or partially realized by software, hardware, firmware, or any combination thereof. When used in whole or in part, can be implemented in a computer program product that includes one or more computer instructions. When loaded or executed on a computer, cause the flow or functions according to embodiments of the invention to occur, in whole or in part. The computer may be a general purpose computer, a special purpose computer, a network of computers, or other programmable device. The computer instructions may be stored in a computer readable storage medium or transmitted from one computer readable storage medium to another, for example, the computer instructions may be transmitted from one website site, computer, server, or data center to another website site, computer, server, or data center via wire (e.g., coaxial cable, fiber optic, Digital Subscriber Line (DSL), or wireless (e.g., infrared, wireless, microwave, etc.)). The computer-readable storage medium can be any available medium that can be accessed by a computer or a data storage device, such as a server, a data center, etc., that includes one or more of the available media. The usable medium may be a magnetic medium (e.g., floppy Disk, hard Disk, magnetic tape), an optical medium (e.g., DVD), or a semiconductor medium (e.g., Solid State Disk (SSD)), among others.
The above description is only for the purpose of illustrating the preferred embodiments of the present invention, and the scope of the present invention is not limited thereto, and any modification, equivalent replacement, and improvement made by those skilled in the art within the technical scope of the present invention disclosed herein, which is within the spirit and principle of the present invention, should be covered by the present invention.
Claims (10)
1. A big data analysis processing method based on a terminal is characterized by comprising the following steps:
inputting preset parameters through a human-computer interaction module, and selecting the type of data to be analyzed and processed according to the demand information;
reading the category information of the information to be collected contained in the control instruction through a data acquisition module according to a preset instruction, and collecting the category information in local storage and network storage according to the category information;
step three, storing the collected data into a data cache server to generate original data, classifying the data, storing the classified data into the data cache server again according to different categories to generate classified data;
step four, calling the classification data by using terminal equipment, comparing the original data with the classification data called by the data calling signal, outputting the two groups of data to a central processing and control module after the two groups of data are compared, and realizing the extraction and collection of locally stored data or network receiving data;
acquiring data to be processed through a data preprocessing module, and acquiring a configuration file matched with the data to be processed according to the attribute of the data to be processed;
step six, determining a current preprocessing mode corresponding to data preprocessing, and controlling each operating node to process the obtained processing result according to a next preprocessing mode of the current preprocessing mode after processing the data to be processed according to the current preprocessing mode;
step seven, respectively acquiring programs corresponding to the steps according to the steps of the pre-processing flow, and processing the data to be processed at each operation node according to the current pre-processing mode; executing the programs corresponding to the steps according to the execution sequence of the steps to realize the pretreatment of the data to be treated;
step eight, providing different authority levels for different users through the authority management module so as to ensure the safety of the architecture; data transmission is carried out between the wireless signal transmitter and a remote server through a wireless signal transmission module; performing data interaction with a remote server through a data interaction module according to the control instruction;
processing and analyzing the acquired data by using a central processing unit through a central processing and controlling module, and performing coordination control on each controlled module of the terminal-based big data analyzing and processing system according to a processing result and preset parameters;
step ten, storing the big data information and remotely analyzing and processing the acquired information by using a remote server through a network server module; the safety of the collected big data information is detected through a safety detection module, and when the safety information is detected to exceed a threshold value preset by an administrator, a warning message is sent to the administrator through a warning module;
eleven, storing the information to be processed or the network receiving information, the data preprocessing result, the authority level, the remote analysis processing result, the safety detection result and the warning message by using a storage hard disk through a local storage module; and updating and displaying the working state of the system, the information to be processed or the network receiving information, the data preprocessing result, the authority level, the remote analysis processing result, the safety detection result and the real-time data of the warning message through the man-machine interaction module.
2. The method for analyzing and processing big data based on terminal as claimed in claim 1, wherein in step three, the classifying method for classifying data includes:
(1) preprocessing a data text, and respectively generating a word vector training set and a BERT language training set;
(2) training each of a plurality of neural network models through the word vector training set respectively, and training a BERT classification model through the BERT language training set;
(3) and determining a fusion model according to the training result, and performing labeling classification on the target data according to the fusion model.
3. The terminal-based big data analysis processing method according to claim 1, wherein in step five, the configuration file matched with the data to be processed includes an identifier and operation parameters of the preprocessing operation included in the step of preprocessing flow.
4. The big data analyzing and processing method based on the terminal as claimed in claim 1, wherein in the tenth step, the detecting the security of the collected big data information by the security detecting module includes:
(1) when the acquired big data information is used for carrying out security detection on a terminal, acquiring operation parameters of data to be detected and analyzing the operation parameters;
(2) acquiring a security authentication result of big data information; the terminal equipment determines whether to keep the information in the terminal or delete the information according to the security authentication result of the big data information;
(3) this information is sent to the network server while retained to update the database for big data analysis, decision making and validation.
5. The terminal-based big data analysis processing method of claim 4, wherein the obtaining of the security authentication result of the big data information comprises:
1) the system sends a preset authentication mode of the big data information to the terminal to prompt the user to select multiple authentication modes from the preset authentication mode to authenticate the big data information;
2) acquiring a plurality of authentication modes selected by a user and authentication information of big data information respectively input in the plurality of authentication modes;
3) and obtaining a security authentication result of the big data information based on the preset weight corresponding to each authentication mode and based on each authentication information authentication.
6. The big data analyzing and processing method based on the terminal as claimed in claim 1, wherein in the eleventh step, storing the information to be processed or the network receiving information, the data preprocessing result, the authority level, the remote analyzing and processing result, the security detection result and the warning message by using the storage hard disk through the local storage module comprises:
(1) calculating the size of the data information to be stored to obtain the data volume of the information to be stored;
(2) if the data volume of the information to be stored is larger than the maximum data volume of unit information which can be stored and is preset in a storage hard disk, splitting the information to be stored into at least two pieces of segment information, so that the data volume of each piece of segment information is smaller than or equal to the maximum data volume;
(3) and storing each piece of segment information and a segment identifier corresponding to each piece of segment information in the storage hard disk.
7. The terminal-based big data analysis processing method of claim 6, wherein the data volume comprises a data source identifier, a time identifier and a data stream size of the data to be processed.
8. A terminal-based big data analysis processing system applying the terminal-based big data analysis processing method according to any one of claims 1 to 7, the terminal-based big data analysis processing system comprising:
the system comprises a data acquisition module, a data preprocessing module, a right management module, a wireless signal transmission module, a central processing and control module, a data interaction module, a network server module, a safety detection module, a warning module, a local storage module and a man-machine interaction module;
the data acquisition module is connected with the central processing and control module and used for extracting and collecting locally stored data or data received by a network through terminal equipment;
the data preprocessing module is connected with the central processing and control module and is used for preprocessing data to be processed;
the authority management module is connected with the central processing and control module and is used for providing different authority levels for different users so as to ensure the safety of the architecture;
the wireless signal transmission module is connected with the central processing and control module and is used for carrying out data transmission with the remote server through the wireless signal transmitter;
the central processing and control module is connected with the data acquisition module, the data preprocessing module, the authority management module, the wireless signal transmission module, the data interaction module, the network server module, the safety detection module, the warning module, the local storage module and the man-machine interaction module, and is used for processing and analyzing the acquired data through the central processing unit and carrying out coordination control on each controlled module of the terminal-based big data analysis and processing system according to a processing result and preset parameters;
the data interaction module is connected with the central processing and control module and is used for carrying out data interaction with the remote server according to the control instruction;
the network server module is connected with the central processing and control module and used for storing the big data information and remotely analyzing and processing the acquired information through a remote server;
the safety detection module is connected with the central processing and control module and is used for detecting the safety of the acquired big data information;
the warning module is connected with the central processing and control module and is used for sending warning information to an administrator when detecting that the safety information exceeds a threshold value preset by the administrator;
the local storage module is connected with the central processing and control module and used for storing the information to be processed or the network receiving information, the data preprocessing result, the authority level, the remote analysis processing result, the safety detection result and the warning message through the storage hard disk;
and the human-computer interaction module is connected with the central processing and control module and used for inputting preset parameters and updating and displaying the working state of the system, the information to be processed or the network receiving information, the data preprocessing result, the authority level, the remote analysis processing result, the safety detection result and the real-time data of the warning message.
9. A computer program product stored on a computer readable medium, comprising a computer readable program for providing a user input interface to implement the terminal-based big data analysis processing method of any of claims 1 to 7 when executed on an electronic device.
10. A computer-readable storage medium storing instructions which, when executed on a computer, cause the computer to perform the terminal-based big data analysis processing method according to any one of claims 1 to 7.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202110884930.8A CN113536381A (en) | 2021-08-03 | 2021-08-03 | Big data analysis processing method and system based on terminal |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202110884930.8A CN113536381A (en) | 2021-08-03 | 2021-08-03 | Big data analysis processing method and system based on terminal |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN113536381A true CN113536381A (en) | 2021-10-22 |
Family
ID=78121889
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202110884930.8A Withdrawn CN113536381A (en) | 2021-08-03 | 2021-08-03 | Big data analysis processing method and system based on terminal |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN113536381A (en) |
Cited By (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN114338221A (en) * | 2022-01-06 | 2022-04-12 | 北京为准智能科技有限公司 | Network detection system based on big data analysis |
-
2021
- 2021-08-03 CN CN202110884930.8A patent/CN113536381A/en not_active Withdrawn
Cited By (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN114338221A (en) * | 2022-01-06 | 2022-04-12 | 北京为准智能科技有限公司 | Network detection system based on big data analysis |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN111092852B (en) | Network security monitoring method, device, equipment and storage medium based on big data | |
| CN110958220B (en) | Network space security threat detection method and system based on heterogeneous graph embedding | |
| US20220210200A1 (en) | Ai-driven defensive cybersecurity strategy analysis and recommendation system | |
| US20220201042A1 (en) | Ai-driven defensive penetration test analysis and recommendation system | |
| CN111274583A (en) | Big data computer network safety protection device and control method thereof | |
| CN107733863B (en) | Log debugging method and device under distributed hadoop environment | |
| CN110020687B (en) | Abnormal behavior analysis method and device based on operator situation perception portrait | |
| US20230050771A1 (en) | Method for determining risk level of instance on cloud server, and electronic device | |
| CN114338064B (en) | Method, device, system, equipment and storage medium for identifying network traffic type | |
| CN108234426B (en) | APT attack warning method and APT attack warning device | |
| CN112926048A (en) | Abnormal information detection method and device | |
| CN111314063A (en) | Big data information management method, system and device based on Internet of things | |
| CN111724069A (en) | Method, apparatus, device and storage medium for processing data | |
| US20230396635A1 (en) | Adaptive system for network and security management | |
| CN110798353B (en) | Network behavior risk perception and defense method based on behavior characteristic big data analysis | |
| CN113536381A (en) | Big data analysis processing method and system based on terminal | |
| CN114356712A (en) | Data processing method, device, equipment, readable storage medium and program product | |
| CN116980159A (en) | Abnormal behavior detection method, device, electronic equipment and storage medium | |
| CN114282223A (en) | Method, defense system, device and medium for improving security of terminal equipment | |
| [Retracted] Design of a Network Security Audit System Based on Log Data Mining | ||
| CN109902831B (en) | Service decision processing method and device | |
| CN113992371A (en) | Method and device for generating threat tag of flow log and electronic equipment | |
| CN112839029A (en) | Botnet activity degree analysis method and system | |
| Bockermann et al. | On the automated creation of understandable positive security models for web applications | |
| CN115696341A (en) | Identification and analysis method and device for harmful events based on consistency and consensus |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| WW01 | Invention patent application withdrawn after publication |
Application publication date: 20211022 |
|
| WW01 | Invention patent application withdrawn after publication |