KR20150057322A - System and Method for Deducting about Weak Signal Using Big Data Analysis - Google Patents

Abstract

The present invention relates to a system and a method for extracting a weak signal based on a big data analysis. The system includes: a collection managing module which sets at least one collection criterion from a collection data source module, a collection key word, and a collection period; a data collection module which collects big data from the data source module according to the collection criterion set by the collection managing module and stores the big data in a database module; a morpheme analyzing module which analyzes morphemes of the big data, stored in the database, sorts the big data by the morphemes, and then generates morpheme analysis data; and a data analysis module which calculates a statistical value by processing the morpheme analysis data with an algorithm based on a distributed and parallel process, and outputs the weak signal based on the statistical data.

Description

TECHNICAL FIELD The present invention relates to a system and method for deriving a week signal based on a big data analysis,

[0001] The present invention relates to a system and method for deriving a week signal based on big data analysis, and more particularly, to a system and method for deriving a week signal based on big data analysis, The present invention relates to a system and method for deriving a week signal based on a big data analysis capable of quantitatively analyzing a signal.

In recent years, as the interest in the future industry has increased, there is a growing demand for forecasting the future of science and technology, especially the signs of future science and technology change and the search for 'weaksignal'.

Week signals are now uncertain and unusual signals, but they are a preliminary indication that will lead to future changes, providing an important clue to future predictions.

Definitions of weak signals vary from scholar to expert, and the terms 'emerging issues', 'seeds of change', 'wild cards', 'early indicators' or 'early warning signals' have.

Research on week signals can be important in that it provides a crucial first step in studying the future of technology, but it is a field with a long history ranging from its concept to its exploration methods and applications. Deep research is required.

However, since the search of the week signal according to the prior art mainly depends on the expert's insight, reliability and objectivity are a problem. When using the metric information analysis technique based on the data, There was a limitation that only analysis was possible.

In addition, there has been a problem in that it does not support the technique of analyzing the week signal with the unstructured data whose explosive increase has been increasing recently.

Korean Patent Laid-Open Publication No. 10-2013-0077751: Data processing method, data processing apparatus, data collection method, and information providing method

SUMMARY OF THE INVENTION The present invention has been made in order to solve the above-mentioned problems, and it is an object of the present invention to provide a method and apparatus for automatically collecting, storing and analyzing large data including not only regular data but also atypical data, A system and method for deriving a week signal based on data analysis are provided.

According to an aspect of the present invention, there is provided a data collection system including a collection management module for setting collection criteria of at least one of a collection target data source module, a collection target keyword and a collection target period; A data collection module for collecting big data from the data source module according to the data collection criteria set by the collection management module and storing the collected big data in the database module; A morpheme analysis module for analyzing morphemes of big data stored in the database module and dividing them by morphemes and generating morpheme analysis data; And a data analysis module for processing the morpheme analysis data using a statistical analysis algorithm based on a distributed parallel processing to calculate a statistical value and output a weak signal according to the statistical value, .

Here, the data analysis module may include: a distributed file system for storing the morpheme analysis data; A data processing unit for providing a virtualization database interface to process morpheme analysis data existing in the distributed file system based on a structured query language (SQL); And a statistical analysis algorithm through the virtualization database interface provided by the data processing unit to calculate statistical analysis algorithms based on the morphological analysis data, such as instance frequencies, number of simultaneous appearances between words and words per year, And a statistical analysis unit for calculating at least one of the processed values from the number of simultaneous occurrences of word-words.

In addition, the statistical analysis unit may calculate the rate of increase in the number of occurrences of words based on the instance frequencies and the number of simultaneous occurrences between the words.

In addition, the statistical analysis unit may standardize the frequency of appearances of the words per document by dividing the instance frequencies of each year by the number of documents per year to calculate the standardized frequency of occurrence rate of the words with respect to the words .

Also, the statistical analysis unit may calculate the connection degree-of-centration increase rate by using the word-word coincidence count matrix, and may further include a word or a word having the connection degree degree increase rate and the connection degree- The words below the lower part of the schedule can be derived as week signals.

Meanwhile, the data collection module may collect big data including at least one of image, moving picture, voice, sensor, GPS, GIS, and M2M data.

According to another aspect of the present invention for solving the above-mentioned problems, there is provided a method of collecting and storing big data through a network according to a predetermined collection criterion, comprising the steps of: (A) (B) analyzing a morpheme of the big data, dividing the morpheme by morphemes, and generating morpheme analysis data; (C) calculating a statistical value by processing the morpheme analysis data with a statistical analysis algorithm based on a distributed parallel processing, by a data analysis module; And (D) the data analysis module outputting a week signal according to the statistical value.

The step (A) may include the step of setting the collection criteria of at least one of the collection target data source module, the collection target keyword, and the collection target period.

In the step (A), the data collection module may collect big data including at least one of at least one of image, moving picture, voice, sensor, GPS, GIS, and M2M data.

The step (C) may further include extracting, from the morphological analysis data, the number of simultaneous appearances between year-by-year instance frequencies, the number of simultaneous occurrences of word-words by year, the frequency of occurrence by year, And calculating at least one of the values.

The system and method for deriving the Big Signal based on Big Data analysis of the present invention can quantitatively analyze the Week signal by automatically collecting, storing and analyzing Big Data.

In addition, the system and method for deriving a Big Signal based on Big Data Analysis of the present invention can quantify a Great Signal from Big Data and predict a possible change in technology and market by detecting a possible anomaly in the future.

FIG. 1 is a diagram illustrating a network connection state diagram of a big signal analysis based weak signal analysis system according to an embodiment of the present invention.
2 is a block diagram of a Big Signal analysis system based on Big Data Analysis according to an embodiment of the present invention.
3 is a flowchart of a method of deriving a week signal based on a big data analysis according to an embodiment of the present invention.
FIG. 4 is a diagram showing the number of simultaneous occurrences of word-words using a big signal analysis-based week signal derivation system according to an embodiment of the present invention in a matrix form
FIG. 5 is a graph showing the rate of increase in the degree of center-of-connection of words using the Big Signal analysis-based week signal derivation system according to the embodiment of the present invention
6 is a diagram showing a relationship between a rate of increase in the number of occurrences of a word and a degree of centrality of connection degree of centration in a week signal derivation system based on a big data analysis according to an embodiment of the present invention

Hereinafter, preferred embodiments of the present invention will be described in detail with reference to the accompanying drawings. In the following description with reference to the accompanying drawings, the same or corresponding components will be denoted by the same reference numerals, and redundant description thereof will be omitted.

FIG. 1 is a network connection diagram of a big signal analysis based weak signal derivation system 200 according to an embodiment of the present invention.

1, a user terminal 10 accesses a service server 100 through a network to create document data, and the service server 100 stores and publishes document data provided by the user terminal 10, do. Here, the service server 100 may apply various service servers 100 capable of storing and posting data such as SNS, blog, cafe, and news service.

The weak signal derivation system 200 can collect big data including fixed and unstructured data using each service server 100 as a data source, analyze and process the collected data, and derive a weak signal.

Here, the Big Data is data of a size exceeding the range that can be processed and analyzed by existing database technology, and can be used for transaction information in the financial / distribution field, viewership and log information in the communication / broadcasting field, Various types of data such as sensors and operational data in the manufacturing field can be included in the big data.

An important criterion for distinguishing Big Data is structuralization of data. First, structured data refers to numeric or symbol data specified in a fixed field, which is general data managed by relational databases and spreadsheets. Second, semi-structured data is not specified in fixed fields, but XML or HTML text data is data managed by metadata and schema. Lastly, unstructured data may include plain text documents and images, moving images, voice, GPS, GIS, sensors, M2M data, etc., without any fixed fields or schemas.

FIG. 2 is a control block diagram of a Big Data analysis based weak signal derivation system 200 according to an embodiment of the present invention.

2, the weak signal derivation system 200 includes a data source module 210, a data acquisition module 220, a collection management module 230, a storage adapter module 240, a database module 250, A morpheme analysis module 260, and a data analysis module 270.

The data source module 210 includes various service servers 100 capable of storing and posting data such as SNS, blog, cafe, news service, and the like.

The collection management module 230 sets a data collection criterion on the data collection module 220 side. The collection management module 230 provides collection criteria such as a collection target data source module 210, a collection target keyword, and a collection target period of the data collection module 220.

In addition, the collection management module 230 may direct the operation of the data collection module 220 according to a request from the user. The collection management module 230 can set various criteria such as a target module, a keyword, and a period required for data collection, and can instruct the data collection module 220 about the execution timing and method. The data collected in parallel by the data collection module 220 may be stored in a specific area of the database module 250 through the storage adapter module 240. [

The data collection module 220 collects data from the data source module 210 according to the collection criteria provided by the collection management module 230 and stores the collected data in a specific area of the database module 250.

The data collection module 220 may collect data of a predetermined area according to the structure of each data source module 210 such as SNS, news, cafe, blog, and the like. The data collection module 220 distributes various kinds of news data such as news, tweets, blogs, and cafes of the data source module 210 corresponding to the collection criteria received from the collection management module 230, It is possible to collect. The data collected through the data collection module 220 may include unstructured data such as plain text documents and images, moving images, voice, GPS, GIS, sensors, M2M data, and the like.

The data collection module 220 can collect data in parallel processing method in cooperation with a plurality of systems and can collect data on an online basis using a crawler, Can be collected. The data source module 210 transmits data such as social media data, online news, portal site cafes, and blog posts collected from the data source module 210 to the database module 250 through the storage adapter module 240 The storage adapter module 240 transmits data received from the data collection module 220 to the database module 250 using a connection function to the database module 250.

In the database module 250, data collected by the data collection module 220 is stored. The database module 250 is a distributed parallel processing environment that can be operated by a plurality of servers in order to secure processing performance. The database module 250 may distribute and store data received from the storage adapter module 240 to each database.

The morpheme analysis module 260 generates morpheme analysis data by dividing the original data stored in the database module 250 into morphemes by using morpheme analysis programs and then transmits the morpheme analysis data to the database module 250 . The original data stored in the database module 250 may include social media data, online news, portal site cafes, and blog posts.

The data analysis module 270 includes a statistical analysis unit 272, a data processing unit 274, and a distributed file system 276.

The distributed file system 276 reads the morpheme analysis data stored in the database module 250 and stores the morpheme analysis data in the distributed file system 276 capable of processing large-capacity data.

The data processing unit 274 provides a virtualization database interface so that the morphological analysis data existing in the distributed file system 276 can be processed based on a structured query language (SQL). SQL is a common language that allows you to manipulate or access data in a database.

The statistical analysis unit 272 implements various statistical analysis algorithms based on distributed parallel processing on the morphological analysis data stored in the distributed file system 276 through the virtualization database interface provided by the data processing unit 274, do. Here, the applied statistical analysis algorithm is to calculate the index used in the model for technology and market prediction. The technology and market prediction model is a model of existing Fisher-Pry model, Bass model, TAM (technology acceptance model) Can be applied, and can be linked by using a model that is improved, supplemented or developed depending on the situation.

The statistical analysis unit 272 extracts at least one of the processed frequencies from the number of simultaneous appearances among the number of simultaneous appearances between year-by-year instance frequency, year-to-year word-word simultaneous occurrence frequency, One or more can be used as a variable index.

The statistical analysis unit 272 can analyze the rate of increase in the frequency of occurrence of each word from the instance frequencies of the extracted words by year. In addition, the statistical analysis unit 272 standardizes the frequency of occurrence of the corresponding words per document by dividing the frequency of occurrence of each instance frequency by the number of documents per year, and analyzes the rate of increase in the frequency of appearance of standardized words can do. By measuring the rate of increase in the number of occurrences per word, it is possible to detect a change that deviates from the rate of increase in the average number of occurrences.

In addition, the statistical analysis unit 272 can analyze the rate of increase of the degree of connection degree of the word by using the number of simultaneous word-word emergence matrices.

FIG. 4 is a diagram showing the number of simultaneous occurrences of word-words in a matrix form, and FIG.

As shown in FIG. 4 and FIG. 5, by analyzing the relationship between words and words in time series, it is possible to know which words are related to other words over time. If the difference is larger than the average value, It can be judged as a symptom.

In order to evaluate the rate of increase of the degree of centrality of words, we can measure the change of centrality index by performing SNA analysis between words.

Using social network analysis, a node corresponds to a measurement target word, an event can be viewed as an individual article, a link is a relationship between words constituting an article and an event Can be seen as. In other words, in a social network analysis method, a relationship between a certain node (word) and another node (word) is considered to be related if two words exist in the same article at the same time.

The centrality analysis is a measure of the degree to which a particular node is centrally located in the network. In particular, degree centrality analysis measures in-degree centrality and out-of-degreentality by measuring how closely the network is located in a network by measuring the connection lines connected to a particular node. The higher the frequency with which nodes in a network are connected to a particular node, the more centralized it is.

The statistical analysis unit 272 may determine a word having a higher rate of increase in the frequency of appearance of the standardized word and a rate of increase of the degree of connection degree centroid or a word having a lower or equal lower level portion as a week signal.

FIG. 6 is a diagram illustrating a relationship between a rate of increase in the number of occurrences of a word and a degree of centrality of connection degree in a week signal derivation system 200 based on a big data analysis according to an embodiment of the present invention.

As shown in FIG. 6, a group of words distributed at the center indicates macroscopic trends, and a group of words having a higher degree of deviation from the center indicates week signals indicating signs of change. Accordingly, the statistical analyzer 272 can determine a word having a higher rate of occurrence rate and a connection degree-centrecy rate higher than a certain level or a word having a lower level level as a week signal. That is, the statistical analysis unit 272 performs a quantitative analysis by analyzing at least one of the values of the appearance frequencies of the extracted words by year and the appearance frequency of each year as variables, It can be judged as a weak signal.

3 is a flowchart of a method of deriving a week signal based on a big data analysis according to an embodiment of the present invention.

The data collection module 220 collects data on the network according to a predetermined criterion and stores the collected data in the database module 250 (S110). The data collection module 220 may divide and collect publication data such as news, tweets, blogs, and cafes by period, media, and keywords according to the collection criteria provided by the collection management module 230.

The morpheme analysis module 260 analyzes the morpheme data of the collected data and stores morpheme analysis data in the database module 250 (S112). The morpheme analysis module 260 divides data of social media data, online news, portal site cafe, and blog posting stored in the database module 250 into morphemes using a morpheme analysis program, .

The morpheme analysis data is stored in the distributed file system 276 capable of processing large-capacity data (S114). The data analysis module 270 reads the morpheme analysis result stored in the database module 250 into the distributed file system 276 and stores the morpheme analysis result.

The morphometric analysis data stored in the distributed file system 276 is analyzed statistically by the statistical analysis unit 272 (S116). The statistical analysis unit 272 of the data analysis module 270 analyzes the morphological analysis data by the number of instances frequencies per year, the number of simultaneous occurrences of word-words by year, the frequency of occurrence by year, and the word- At least one of the processed values can be used as a parameter. In addition, the statistical analysis unit 272 can analyze the increase rate of occurrence frequency by word from the instance frequencies of the extracted words by year.

In addition, the statistical analysis unit 272 standardizes the frequency of occurrence of the corresponding words per document by dividing the frequency of occurrence of each instance frequency by the number of documents per year, and analyzes the rate of increase in the frequency of appearance of standardized words can do. By measuring the rate of increase in the number of occurrences per word, it is possible to detect a change that deviates from the rate of increase in the average number of occurrences. In addition, the statistical analysis unit 272 can analyze the rate of increase of the degree of connection degree of the word by using the number of simultaneous word-word emergence matrices.

The statistical analysis unit 272 derives a week signal according to the analysis result (S118). The statistical analysis unit 272 may determine a word having a higher rate of increase in the frequency of appearance of the standardized word and a rate of increase of the degree of connection degree centroid or a word having a lower or equal lower level portion as a week signal.

Thus, those skilled in the art will appreciate that the present invention may be embodied in other specific forms without departing from the spirit or essential characteristics thereof. It is therefore to be understood that the embodiments described above are to be considered in all respects only as illustrative and not restrictive. The scope of the present invention is defined by the appended claims rather than the detailed description and all changes or modifications derived from the meaning and scope of the claims and their equivalents are to be construed as being included within the scope of the present invention do.

The present invention relates to a system and method for deriving a week signal based on a big data analysis capable of quantitatively analyzing a week signal by automatically collecting, storing and analyzing large data including not only regular data but also atypical data Lt; / RTI >

10: User terminal 100: Service server
200: Week signal derivation system 210: Data source module
220: data collection module 230: collection management module
240: storage adapter module 250: database module
260: Morphological analysis module 270: Data analysis module
272: statistical data analysis unit 274: data processing unit
276: Distributed File System

Claims (10)

A collection management module that sets collection criteria of at least one of a collection target data source module, a collection target keyword, and a collection target period;
A data collection module for collecting big data from the data source module according to the data collection criteria set by the collection management module and storing the collected big data in the database module;
A morpheme analysis module for analyzing morphemes of big data stored in the database module and dividing them by morphemes and generating morpheme analysis data; And
And a data analysis module that processes the morpheme analysis data using a statistical analysis algorithm based on a distributed parallel processing to calculate a statistical value and outputs a weak signal according to the statistical value.
The method according to claim 1,
The data analysis module comprising:
A distributed file system for storing the morpheme analysis data;
A data processing unit for providing a virtualization database interface to process morpheme analysis data existing in the distributed file system based on a structured query language (SQL); And
A statistical analysis algorithm is executed through a virtualization database interface provided by the data processing unit, and statistical analysis algorithms are executed to calculate statistical analysis frequencies based on the instance frequencies for each year, the number of simultaneous appearances between words and words per year, - a statistical analysis unit for calculating at least one of the processed values from the number of simultaneous occurrences between words.
3. The method of claim 2,
Wherein the statistical analysis unit comprises:
Wherein the statistical analysis unit calculates the number of occurrences per word based on the number of instances frequencies and the number of simultaneous occurrences between words and words by year.
The method of claim 3,
Wherein the statistical analysis unit comprises:
Wherein the number of occurrence frequencies for each word is standardized by dividing the instance frequencies by the number of documents per year to standardize the appearance frequency of the words per document to calculate the standardized word- Based weak signal derivation system.
In the fourth aspect,
Wherein the statistical analysis unit comprises:
Wherein the connection degree centricity increase rate for each word is calculated using the word-word coincidence count matrix, and the words having the connection degree centricity increase rate and the word having the connection degree centricity increase rate equal to or higher than a certain upper part, Wherein the system is derived from a big data analysis based on a week signal.
The method according to claim 1,
Wherein the data collection module comprises:
And collects big data including at least one of at least one of image, moving picture, voice, sensor, GPS, GIS, and M2M data.
(A) collecting and storing big data through a network according to a predetermined collection criterion;
(B) analyzing a morpheme of the big data, dividing the morpheme by morphemes, and generating morpheme analysis data;
(C) calculating a statistical value by processing the morpheme analysis data with a statistical analysis algorithm based on a distributed parallel processing, by a data analysis module; And
(D) the data analysis module outputting a week signal according to the statistical value.
8. The method of claim 7,
The step (A)
Wherein the collection management module includes a step of setting collection criteria of at least one of a collection target data source module, a collection target keyword, and a collection target period.
8. The method of claim 7,
The step (A)
Wherein the data collection module collects big data including at least one of at least one of image, moving picture, voice, sensor, GPS, GIS, and M2M data.
8. The method of claim 7,
The step (C)
Calculating at least one or more of the processed values from the number of simultaneous occurrences among the number of simultaneous appearances between year-by-year word-word occurrences, word-by-word occurrences, year-by-year occurrence frequency, And generating a Big Signal based on Big Data Analysis.

Images