KR101810180B1 - Method and apparatus for distributed processing of big data based on user equipment - Google Patents

Abstract

Disclosed are a method and a device of a big data distribution processing based on a user device. The method of a big data distribution processing based on a user device comprises the following steps of: dividing and loading, by a collection server, collection target data; transmitting, by the collection server, a storage state of the collection target data to a master node; requesting, by the master node, a processing of the collection target data to a plurality of selection data nodes among a plurality of data nodes based on state information of the plurality of data nodes; and distributing and processing, by the plurality of selection data nodes, the collection target data.

Description

BACKGROUND OF THE INVENTION 1. Field of the Invention [0001] The present invention relates to a method, apparatus,

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to a large data processing, and more particularly, to a large data distribution processing method and apparatus based on a user apparatus.

Gartner, a US research firm in 2012, defined Big Data as "data with a lot of volume, variety of data formats, and fast data rates." In wikipedia, Big Data is defined as a large amount of regular or unstructured data sets that go beyond data collection, storage, management and analysis as existing database management tools.

According to mobile spokesman Tech Spartan, in 2014, the amount of data produced per minute on the Internet is 600,000 logins on Facebook, 433,000 messages tweet on Twitter, and 419,000 searches on Google , 306 hours of video uploads on YouTube, 50,200 apps on Apple iTunes, 80,000 transactions on Amazon, and 136,319,444 emails. Today, in the age of this massive data age, large IT vendors such as IBM, HP, SAP, MS, EMC and Oracle are developing Big Data solutions.

With the advent of the Big Data era, a new processing / analysis paradigm is required, and the statistical engine, R, is centering on the US as an analytical platform for enterprises. R has already been used as an analytics platform for dot-com companies such as Google, Facebook, Yahoo and Amazon. In particular, IT companies seeking high-performance analysis of Big Data, such as Oracle, IBM, and Teradata, can use in-memory analytics or in-database analytics to perform real- (R) as a basic analytical platform for high performance computing (HPC) systems capable of real-time database analysis and database analytics. R has the disadvantage that scalability is low despite the many uses as above.

Therefore, only a limited data size is processed and operated with the basic package of R.

R users have presented various methods for processing large amounts of data. Among them, using ff package and bigmemory package is a way to load and process only the data structure in memory instead of loading all existing data into memory It is too slow and has limitations on large data processing due to limit of physical memory expansion.

Hadoop is an open source platform that can distribute large amounts of data. R and Hadoop integration environments include Rhipe and Rhadoop.

10-2013-0169007

One aspect of the present invention provides a method of processing a large data distribution based on a user device.

Another aspect of the present invention provides an apparatus for performing a method of processing a large data distribution based on a user apparatus.

A method for processing a large data distribution based on a user device according to an aspect of the present invention includes the steps of: a collection server loading and storing data to be collected; the collection server transmitting a storage state of the collection object data to a master node; The master node requesting the processing of the collection object data to a plurality of selection data nodes among the plurality of data nodes on the basis of the status information of the plurality of data nodes, .

Meanwhile, in the method of processing large data distributed based on a user device, the plurality of data nodes may further include transmitting data processable information to the master node.

In addition, the data processable information may include information on a communication state of the data node, a central processing unit (CPU) usage rate, and a memory usage rate.

The selection data node may receive allocation data from the collection server based on a data processing instruction included in the push notification information received from the master node, process the received allocation data, and transmit the processing result to the service database.

In addition, the master node may identify an unprocessed data node to which the allocation data has not been processed until a specific time, and may request the other data node to perform the task again after requesting the unprocessed data node as a push notification or canceling the mission of the unprocessed data node You can tell.

According to another aspect of the present invention, there is provided a system for distributing large data based on a user device, comprising: A collection server configured to transmit a storage state of the collection target data to a master node, a processing server configured to request processing of the collection target data to a plurality of selection data nodes of the plurality of data nodes based on status information of the plurality of data nodes And a plurality of selected data nodes for distributing the collection target data.

Meanwhile, the plurality of data nodes may be configured to transmit data processable information to the master node.

In addition, the data processable information may include information on a communication state of the data node, a central processing unit (CPU) usage rate, and a memory usage rate.

The selection data node may receive allocation data from the collection server based on a data processing instruction included in the push notification information received from the master node, process the received allocation data, and transmit the processing result to the service database.

In addition, the master node may identify an unprocessed data node to which the allocation data has not been processed until a specific time, and may request the other data node to perform the task again after requesting the unprocessed data node as a push notification or canceling the mission of the unprocessed data node You can tell.

According to the method and apparatus for distributing big data distributed on the basis of the user apparatus according to the embodiment of the present invention, a computing resource (CPU) of a personal mobile smart device (such as a smart phone) which is already popular and is equipped with PC (personal computer) (central processing unit), a memory (mempry), etc.), it is possible to replace the introduction of a data node (server) for processing large data of several tens or hundreds of units. Thus, on the enterprise side, massive cost savings can be achieved, and barriers to entry of big data processing systems can be reduced.

In addition, in terms of individuals who provide mobile resources to companies, it is desirable to provide resources of smart devices of individuals who are idle for the majority of the time, receive a variety of content services free of charge from companies, Revenue may be provided. In other words, in the field of large-scale data distribution processing, both individuals and companies can become another new IT-related solution that can provide interactive benefits.

1 is a conceptual diagram illustrating a big data distribution processing system based on a user apparatus according to an embodiment of the present invention.
2 is a flowchart illustrating a method of distributing big data based on a user apparatus according to an embodiment of the present invention.
3 is a conceptual diagram illustrating a big data distribution processing scenario according to an embodiment of the present invention.
4 is a conceptual diagram illustrating a data distribution processing method according to an embodiment of the present invention.

The following detailed description of the invention refers to the accompanying drawings, which illustrate, by way of illustration, specific embodiments in which the invention may be practiced. These embodiments are described in sufficient detail to enable those skilled in the art to practice the invention. It should be understood that the various embodiments of the present invention are different, but need not be mutually exclusive. For example, certain features, structures, and characteristics described herein may be implemented in other embodiments without departing from the spirit and scope of the invention in connection with an embodiment. It is also to be understood that the position or arrangement of the individual components within each disclosed embodiment may be varied without departing from the spirit and scope of the invention. The following detailed description is, therefore, not to be taken in a limiting sense, and the scope of the present invention is to be limited only by the appended claims, along with the full scope of equivalents to which such claims are entitled, if properly explained. In the drawings, like reference numerals refer to the same or similar functions throughout the several views.

Hereinafter, preferred embodiments of the present invention will be described in more detail with reference to the drawings.

The HADOOP Big Data Distributed Storage and Processing System, or similar modern big data processing solutions developed by a foreign IT (Information Technology) company, which is currently used by enterprises and national organizations, has many nodes (server computers) The company directly purchases or leases the method is used.

This method not only has a huge amount of money invested for the initial construction and maintenance of the system for the big data processing, but also the power supply and equipment space are leased to the fixed cost even when idle because there is no data to process There are various entry barriers to introduce the system.

A big data processing method using a mobile smart device is proposed in a big data distribution processing method based on a user apparatus according to an embodiment of the present invention. By sharing computing resources (CPU (central processing unit), memory (mempry), etc.) of personal mobile smart devices (smart phones, etc.) that are already popular and are equipped with PC (personal computer) It is possible to replace the introduction of a data node (server) for processing a large amount of data. Thus, on the enterprise side, massive cost savings can be achieved, and barriers to entry of big data processing systems can be reduced.

In addition, in terms of individuals who provide mobile resources to companies, it is desirable to provide resources of smart devices of individuals who are idle for the majority of the time, receive a variety of content services free of charge from companies, Revenue may be provided.

The use of the large data distribution processing method based on the user device according to the embodiment of the present invention enables the individual and the corporation to become a new IT related solution that can achieve two- .

1 is a conceptual diagram illustrating a big data distribution processing system based on a user apparatus according to an embodiment of the present invention.

1, a big data distribution processing system for distributing big data based on a user apparatus is disclosed.

The big data distribution processing system may include a data node 100, a master node 120, a service database 160, and a collection server.

The data node 100 may be a user device. The user device may be a device such as a smart phone carried by the user. The data node 100 may be a device registered as the data node 100 by installing a smart application capable of communicating with the collection server 120 and the master node 140 as a user device that agrees with the big data distribution process.

The data node 100 may periodically transmit the device status while communicating with the master node 140 and the master node 140 may periodically check the status of the data node 100 and update the related information.

The master node 140 may be a node for managing the collection server 120 and the data node 100. Further, the master node 140 can manage the analysis processing algorithm and manage the analysis processing schedule.

The master node 140 can instruct the data processing of the collection server information, the data set information, and the processing algorithm necessary for the big data processing to the node having a low resource utilization rate among the registered data nodes 100 by push notifications. The data node 100 receives and processes a small amount of data allocated to itself from the collection server 120 based on the data processing instruction included in the push notification information received from the master node 140 and outputs the result to the service database 160 ). ≪ / RTI >

The collection server 120 can collect internal large-capacity data and external collection target data. The collection server 120 may be plural. Internal mass data may be collected based on various communication protocols (e.g., file transfer protocol (FTP) / secure FTP (SFTP), NFS). For example, the internal large-capacity data may include various cadastral data, crawled unstructured data, communication information, card usage information, photographic image data, large-volume big data, and the like. The external collection target data can be collected based on communication protocols (eg FTP / SFTP, SMTP, HTTP / HTTPS, OFF LINE collection). For example, the external collection target data may include real-time driving record information, IoT (internet of things) sensor event information, and other large-capacity data.

The collection server 120 may transmit the storage state to the master node 140 after the data is divided and loaded into the collection server 120 through various communication protocols (FTP / HTTP / NFS and the like).

The service database 160 may receive data processing results of the data node 100 from the master node 140 and the received data processing results may be used as statistical analysis data, visualization data, and decision data.

2 is a flowchart illustrating a method of distributing big data based on a user apparatus according to an embodiment of the present invention.

FIG. 2 illustrates a method for distributing large data based on a user apparatus of a user set as a data node.

Referring to FIG. 2, the user device may install a distributed processing application for distributed processing of big data.

The user device installs the distributed processing application, agrees with the distributed processing of the big data, and can communicate with the collection server and the master node (step S200).

The user device may be loaded with a distributed processing application and registered as a data node in the master node.

The collection target data is divided and loaded into the collection server, and the collection server transmits the storage status to the master node (step S210).

The internal / external collection target data may be stored in a storage state to the master node after the divided data is loaded to the collection server based on various communication protocols such as FTP / HTTP / NFS (step S220).

For example, the internal large-capacity data may include various cadastral data, crawled unstructured data, communication information, card usage information, photographic image data, large-volume big data, and the like. The external collection target data can be collected based on communication protocols (eg FTP / SFTP, SMTP, HTTP / HTTPS, OFF LINE collection). For example, the external collection target data may include real-time driving record information, IoT (internet of things) sensor event information, and other large-capacity data.

The data node periodically communicates with the master node and transmits the device status, and the master node can periodically check the status of the data node and update related information (step S230).

The master node can instruct the data processing by transmitting push notification to the data node having a low resource utilization rate among the registered data nodes, the collection server information, the data set information and the processing algorithm necessary for the big data processing (step S240).

Based on the data processing instruction included in the push notification information received from the master node, the data node receives and processes a small amount of data allocated to the data node from the collection server, and stores the result in the service database (step S250).

That is, a method of distributing large data on the basis of a user device includes a step in which the collection server divides the collection object data, the collection server transmits the storage state of the collection object data to the master node, Requesting a plurality of selection data nodes among the plurality of data nodes to process the collection object data based on the status information, and distributing the collection object data to the plurality of selection data nodes.

The selection data node can receive the allocation data from the collection server based on the data processing instruction included in the push notification information received from the master node, process the received allocation data, and transmit the processing result to the service database.

The master node can identify unprocessed data nodes that have not been processed for assignment data until a certain time, re-request them as push notifications to the unprocessed data nodes, or indicate the task to another data node after canceling the mission of the unprocessed data node.

3 is a conceptual diagram illustrating a big data distribution processing scenario according to an embodiment of the present invention.

In Fig. 3, a big data distribution processing scenario is disclosed.

Referring to FIG. 3, a preparation step of a data node (user device) may be performed (step S300).

For example, a distributed processing application for managing or calculating data resources in a plurality of smartphones (e.g., 100) may be installed.

The data node (smartphone) can periodically transmit data processable information to the master node based on various setting information included in the app. The data processable information may be data processing availability status information such as terminal communication status (Wi-Fi, wireless communication), CPU, memory utilization rate, and the like.

The master node can manage the information on the available data node information, the number and the capacity by analyzing the state of the data node included in the data processable information.

A data collection step may be performed (step S310).

For example, from external Web crawls over the last decade to collection servers, textual data from newspapers can be collected and data names can be assumed to be newspaper data sets (eg, 10GB in capacity).

The collection server may transmit collection-related information to the master node that it has normally received and stored the newspaper data set currently collected. The collection-related information may include information about the collection server name, file name, path, capacity, collection path, and the like.

The master node may store or update its contents in a data management frame residing in the memory and physically store and update the node in the collection node management database.

A task assignment step for distributed processing may be performed (step S320).

For example, a user (a programmer) can instruct the master node to count the number of 'Korean' words in a newspaper dataset for fast data distribution processing for a specific purpose. The master node can calculate the data node that is determined to be able to use the resource in the currently managed data node (smart phone).

Data collection information about how much data is to be allocated per data node in the newspaper data set 10G, and the contents of the work to be performed by each data node (indicating a Korean word count) are included in the array To be included in the push notification content and transmitted to the data node.

A big data distribution processing step may be performed (step S330).

The data node receiving the job content as a push notification can request the specific interval data of the newspaper data set allocated to the data node by communicating with the collection server based on the data processing allocation information of the data node included in the push notification.

The collection server may divide the information of the newspaper data set based on the information requested from the data node and transmit the divided information to the respective data nodes.

100 data nodes can receive 100 megabyte of their quota from the collection server and perform Korean word count on the corresponding data fragments.

Each data node may store the resultant value (e.g., 100 counts in Korea) in the execution result information and send the success or failure value to the master node.

The data node receiving the information that the result value has been normally received from the master node can secure the capacity by deleting the data piece held by the data node.

The master node can identify and manage the data nodes for which the success or failure result has not arrived by a certain time. For example, the master node can either re-request a push notification, or delegate that task to another node after canceling the mission.

When a result value is received from all the data nodes, a result value (for example, 10000 counts in Korea) that arrives from all the data nodes can be collected. The collected results can be stored in the service database corresponding mission results and returned to the user (programmer).

4 is a conceptual diagram illustrating a data distribution processing method according to an embodiment of the present invention.

FIG. 4 illustrates a method for determining whether or not to process a big data distribution of a user apparatus.

Referring to FIG. 4, the user device must consume its computing resources for the big data distribution processing. Accordingly, the user apparatus can determine whether the current state of the user apparatus is the current state, and determine whether to lend the computing resource for the big data distribution processing.

It can be assumed that the user is currently using more than a threshold computing resource via the user device. For example, it can be assumed that a user performs data communication with a threshold value or higher within a predetermined period, and uses a processor of a user apparatus of a threshold value or more.

In this case, the user equipment can determine the amount of data throughput in consideration of the amount of data communication and the amount of processor used within a predetermined period when transmitting the data processable information.

The user device may estimate the data usage of the user device / the usage of the processor to be performed within a predetermined period. When the first application is running on the user device, the first application may determine the predicted data amount and the predictive processing amount to be used by the first application on the user device. When the first application continuously uses the amount of data / the amount of processing in the threshold range, the predicted amount of data 400 / the amount of predicted processing 420 is predicted in a predetermined amount based on the existing data amount and the average of the processing amount You can decide. Conversely, when the first application uses a data amount / processing amount with a large amount of change, the predicted value can be determined by predicting based on the existing data amount and the maximum value of the processing amount.

The user equipment may determine the transmission period of the data processable information based on the amount of data used in the user equipment and the change in the amount of processing. When the first application continuously uses a constant data amount / constant processing amount, the transmission period of the data processable information can be set to be relatively long. On the other hand, when the first application uses a data amount / processing amount with a large amount of change, the transmission period of the data processable information can be set relatively short so that the change in the data amount / processing amount used in the first application can be transmitted more frequently .

Also, according to an embodiment of the present invention, the data processable information may be generated by further considering data / processor usage characteristics for each user. The distributed processing application may determine the user device usage characteristics of the user and may generate data processable information. For example, in the case of a specific user, a usage pattern of using various applications (or applications using a lot of processing performance) at the same time on the user device may be used for another user, Only applications using processing power can be used. Such user device usage characteristic information of the user may be recorded in the distributed processing application, and the distributed processing application may generate the data processable information based on the usage characteristic of the user.

The above-described methods may be implemented in an application or may be implemented in the form of program instructions that may be executed through various computer components and recorded on a computer-readable recording medium. The computer-readable recording medium may include program commands, data files, data structures, and the like, alone or in combination.

The program instructions recorded on the computer-readable recording medium may be those specially designed and configured for the present invention and may be those known and used by those skilled in the computer software arts.

Examples of computer-readable recording media include magnetic media such as hard disks, floppy disks and magnetic tape, optical recording media such as CD-ROMs and DVDs, magneto-optical media such as floptical disks, media, and hardware devices specifically configured to store and execute program instructions such as ROM, RAM, flash memory, and the like.

Examples of program instructions include machine language code such as those generated by a compiler, as well as high-level language code that can be executed by a computer using an interpreter or the like. The hardware device may be configured to operate as one or more software modules for performing the processing according to the present invention, and vice versa.

While the present invention has been particularly shown and described with reference to exemplary embodiments thereof, it will be understood by those skilled in the art that various changes and modifications may be made therein without departing from the spirit and scope of the invention as defined in the appended claims. It will be possible.

Claims (10)

A method for processing a large data distribution based on a user device,
A step in which the collection server divides and stores the data to be collected;
The collection server transmitting to the master node a storage state of the collection object data;
Requesting, by the master node, processing of the collection target data to a plurality of selected data nodes among the plurality of data nodes based on state information of the plurality of data nodes;
Wherein the plurality of selected data nodes distribute the collection object data,
The user apparatus operating as each of the plurality of data nodes transmits the data processable information in consideration of the predicted data traffic amount and the prediction processor usage amount,
Wherein the predicted data communication amount is a size of data predicted to be used in an application driven by the user apparatus,
Wherein the prediction processor usage is an amount of processor usage predicted to be used in the application,
Wherein the transmission period of the data processable information is set based on a change in the predicted data traffic amount and the prediction processor usage amount. The method according to claim 1,
Wherein the plurality of data nodes further comprise transmitting data processable information to the master node,
Wherein the data processable information further comprises information about a user's usage characteristics for the user device. 3. The method of claim 2,
Wherein the data processable information includes information on a communication state of the data node, a central processing unit (CPU) usage rate, and a memory usage rate. The method of claim 3,
Wherein the selection data node receives and processes allocation data from the collection server based on a data processing instruction included in the push notification information received from the master node, and transmits the processing result to the service database. 5. The method of claim 4,
The master node determines an unprocessed data node to which the allocation data has not been processed until a predetermined time, requests the unprocessed data node for a push notification again, or indicates the corresponding task to another data node after canceling the mission of the unprocessed data node ≪ / RTI > A big data distribution processing system based on a user device,
Dividing the data to be collected into segments; A collection server configured to send a storage state of the collection target data to a master node;
A master node configured to request processing of the collection target data to a plurality of selected data nodes among the plurality of data nodes based on status information of the plurality of data nodes; And
And a plurality of selection data nodes for distributing the collection target data,
The user apparatus operating as each of the plurality of data nodes transmits the data processable information in consideration of the predicted data traffic amount and the prediction processor usage amount,
Wherein the predicted data communication amount is a size of data predicted to be used in an application driven by the user apparatus,
Wherein the prediction processor usage is an amount of processor usage predicted to be used in the application,
Wherein the transmission period of the data processable information is set based on a change in the predicted data communication amount and the predictive processor usage amount. The method according to claim 6,
Wherein the plurality of data nodes are implemented to transmit data processable information to the master node,
Wherein the data processable information further includes information on a user's usage characteristic with respect to the user device. 8. The method of claim 7,
Wherein the data processable information includes information on a communication state of the data node, a central processing unit (CPU) usage rate, and a memory usage rate. 9. The method of claim 8,
Wherein the selection data node receives and processes allocation data from the collection server based on a data processing instruction included in the push notification information received from the master node, and transmits the processing result to the service database. Big data distribution processing system. 10. The method of claim 9,
The master node determines an unprocessed data node to which the allocation data has not been processed until a predetermined time, requests the unprocessed data node for a push notification again, or indicates the corresponding task to another data node after canceling the mission of the unprocessed data node And a user data processing unit for processing the data.

Images