CN103299298B - The method and system of process business - Google Patents
The method and system of process business Download PDFInfo
- Publication number
- CN103299298B CN103299298B CN201180003695.3A CN201180003695A CN103299298B CN 103299298 B CN103299298 B CN 103299298B CN 201180003695 A CN201180003695 A CN 201180003695A CN 103299298 B CN103299298 B CN 103299298B
- Authority
- CN
- China
- Prior art keywords
- tupe
- attribute
- processing unit
- business
- processing
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
- 238000000034 method Methods 0.000 title claims abstract description 163
- 238000000638 solvent extraction Methods 0.000 claims description 21
- 238000007726 management method Methods 0.000 claims description 8
- 238000013316 zoning Methods 0.000 claims description 2
- 238000010586 diagram Methods 0.000 description 8
- 230000006870 function Effects 0.000 description 8
- 230000004044 response Effects 0.000 description 6
- 238000005192 partition Methods 0.000 description 4
- 238000006243 chemical reaction Methods 0.000 description 3
- 238000010168 coupling process Methods 0.000 description 3
- 238000005859 coupling reaction Methods 0.000 description 3
- 238000005516 engineering process Methods 0.000 description 3
- 238000004891 communication Methods 0.000 description 2
- 230000008878 coupling Effects 0.000 description 2
- 239000004744 fabric Substances 0.000 description 1
- 238000011176 pooling Methods 0.000 description 1
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/24—Querying
- G06F16/245—Query processing
- G06F16/2455—Query execution
- G06F16/24568—Data stream processing; Continuous queries
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Computational Linguistics (AREA)
- Data Mining & Analysis (AREA)
- Databases & Information Systems (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Stored Programmes (AREA)
- Management, Administration, Business Operations System, And Electronic Commerce (AREA)
Abstract
The invention discloses a kind of method and system processing business.The method comprises determining that the tupe attribute of the first step for processing business, and the tupe attribute of this first step includes batch mode or stream tupe；Tupe attribute according to this first step, creates first processing unit corresponding with the tupe attribute of this first step, to process this first step.This system includes scheduler module and node administration module, and this scheduler module includes that first determines unit, and this node administration module includes the first creating unit.The method and system of the process business of the embodiment of the present invention, by the tupe attribute according to each step for processing business, create with this corresponding processing unit of tupe attribute to perform each step, the processing mode making business is more flexible, it is thus possible to process business in the way of batch processing and stream process and combine, it is possible to the treatment effeciency of raising system, improve the performance of system, and the experience of user can be improved.
Description
Technical field
The present invention relates to areas of information technology, particularly relate to areas of information technology process business method and
System.
Background technology
Parallel transcoding is generally made up of burst, transcoding, three steps (STEP) of merging, wherein burst
Step, for splitting the file of user's input, generates small pieces video file；Transcoding step is for right
After burst, the small video file of output carries out form conversion；Combining step is for same video literary composition
All bursts after transcoding of part merge.
Map at current cloud and simplify under (Cloud Map Reduce, referred to as " CMR ") framework,
Parallel transcoding is made up of burst, transcoding, three steps of merging.The input file of user first passes around burst
Step, generates key-value pair (KEY, VALUE) and injects transcoding processing unit (WORKER)
Input rank (INPUT QUEUE), wherein key (KEY) is filename, and value (VALUE) is
The storage position of the small pieces video file after burst；Transcoding processing unit obtains data from input rank, and
Key-value pair output is generated after completing transcoding；After CMR framework collects this key-value pair, according to dividing of setting
This key-value pair is injected into the simplification queue (REDUCE QUEUE) of correspondence by district's algorithm.
After the transcoding processing unit of all bursts has been carried out form conversion, start merging treatment unit.
This merging treatment unit obtains from main simplification queue (MASTER REDUCE QUEUE) and simplifies queue,
Obtain the key-value pair (KEY, VALUE) simplified in queue again, to perform pooling function.
According to current CMR framework, it is necessary to after all transcoding processing units complete form conversion
Could start merging process, the processing mode of business is single, dumb, causes the treatment effeciency of system relatively
Low.Therefore when user have submitted multiple video file transcoding simultaneously, owing to transcoding process is the longest,
The treatment effeciency of system and the performance of system are had a strong impact on.On the other hand, CMR pattern is used to need
Store data in queue, then read from queue, for merging process, need team
Row carry out twi-read and just can get data, thus also have a strong impact on the performance of system.
Accordingly, it would be desirable to a kind of suitably scheme, it is possible to increase process the performance of the system of business.
Summary of the invention
Embodiments provide a kind of method and system processing business, it is possible to increase the process of system
Efficiency.
On the one hand, embodiments providing a kind of method processing business, the method comprises determining that
For processing the tupe attribute of the first step of business, the tupe attribute of this first step includes
Batch mode or stream tupe, this first step is any one in the step processing this business
Individual step；Tupe attribute according to this first step, creates the tupe with this first step and belongs to
Corresponding first processing unit of property, to process this first step.
On the other hand, embodiments providing a kind of system processing business, this system includes scheduling
Module and node administration module, this scheduler module includes: first determines unit, is used for determining for processing
The tupe attribute of the first step of business, the tupe attribute of this first step includes batch processing mould
Formula or stream tupe, this first step is any one step in the step processing this business；
This node administration module includes: the first creating unit, for the tupe attribute according to this first step,
Create first processing unit corresponding with the tupe attribute of this first step, to process this first step
Suddenly.
Based on technique scheme, the method and system of the process business of the embodiment of the present invention, by basis
For processing the tupe attribute of each step of business, create with this that tupe attribute is corresponding processes
Unit is to perform each step so that the processing mode of business is more flexible, more diversified such that it is able to
In the way of batch processing and stream process and combine, process business, it is possible to the treatment effeciency of raising system,
The performance of raising system, and the experience of user can be improved.
Accompanying drawing explanation
In order to be illustrated more clearly that the technical scheme of the embodiment of the present invention, below will be in the embodiment of the present invention
The required accompanying drawing used is briefly described, it should be apparent that, drawings described below is only this
Some embodiments of invention, for those of ordinary skill in the art, are not paying creative work
Under premise, it is also possible to obtain other accompanying drawing according to these accompanying drawings.
Fig. 1 is the indicative flowchart of the method for process business according to embodiments of the present invention.
Fig. 2 is another indicative flowchart of the method for process business according to embodiments of the present invention.
Fig. 3 is the indicative flowchart of the method creating processing unit according to embodiments of the present invention.
Fig. 4 is another indicative flowchart of the method for process business according to embodiments of the present invention.
Fig. 5 is another indicative flowchart of the method for process business according to embodiments of the present invention.
Fig. 6 is another indicative flowchart of the method for process business according to embodiments of the present invention.
Fig. 7 is another indicative flowchart of the method for process business according to embodiments of the present invention.
Fig. 8 is the schematic block diagram of the system of process business according to embodiments of the present invention.
Fig. 9 is the schematic block diagram of scheduler module according to embodiments of the present invention.
Figure 10 is the schematic block diagram of the first creating unit according to embodiments of the present invention.
Figure 11 is another schematic block diagram of the system of process business according to embodiments of the present invention.
Figure 12 is another schematic block diagram of the system of process business according to embodiments of the present invention.
Figure 13 is another schematic block diagram of the system of process business according to embodiments of the present invention.
Figure 14 is another schematic block diagram of the system of process business according to embodiments of the present invention.
Detailed description of the invention
Below in conjunction with the accompanying drawing in the embodiment of the present invention, the technical scheme in the embodiment of the present invention is carried out
Clearly and completely describe, it is clear that described embodiment is a part of embodiment of the present invention, and not
It is whole embodiments.Based on the embodiment in the present invention, those of ordinary skill in the art are not making wound
The every other embodiment obtained on the premise of the property made work, all should belong to the scope of protection of the invention.
Fig. 1 shows the indicative flowchart of the method 100 of process business according to embodiments of the present invention.
As it is shown in figure 1, the method 100 includes:
S110, determines the tupe attribute of first step for processing business, the place of this first step
Reason mode attribute includes batch mode or stream tupe, and this first step is for processing this business
Any one step in step；
S120, according to the tupe attribute of this first step, creates the tupe with this first step
Corresponding first processing unit of attribute, to process this first step.
The system of process business is for pending business, it may be determined that for processing one of them of business
The tupe attribute of step, thus the system processing business can belong to according to the tupe of this step
Property, create processing unit (Worker) corresponding with the tupe of this step.Wherein, this process mould
Formula attribute includes that batch processing (Batch) pattern or stream process (Streaming) pattern, thus processes industry
The system of business can process this business in the way of batch processing and stream process and combine.
Therefore, the method for the process business of the embodiment of the present invention, by according to each step for processing business
Rapid tupe attribute, creates with this corresponding processing unit of tupe attribute to perform each step,
The processing mode making business is more flexible, more diversified such that it is able to process phase with batch processing and stream
In conjunction with mode process business, it is possible to the treatment effeciency of raising system, improve the performance of system, and
The experience of user can be improved.
Such as, for parallel transcoding business, if the tupe attribute of combining step is set to stream
Tupe, then system can create for processing at the stream of combining step according to this stream tupe
Reason unit (Streaming Worker).Thus, the stream processing unit of this combining step can use at stream
Business is processed by reason pattern, and the key-value pair sent at the processing unit receiving previous step
Time (Key, Value), it is possible to start the process of business, and without waiting for the process list of previous step
Unit could perform after being fully completed.Therefore, the method for the embodiment of the present invention makes the processing mode of business
More flexible, more diversified, it is possible to increase the treatment effeciency of system such that it is able to improve the property of system
Can, and it is possible to improve the experience of user.
In S110, the system processing business determines the tupe attribute of first step.
Should be understood that this first step is any one step in the step for processing business, such as,
This first step could be for the initial step of process business, or last step, this first step
Can also be that the embodiment of the present invention is not limited to this for processing business intermediate steps.
In embodiments of the present invention, the service request that the system of business can send is processed according to user, really
Determine the tupe attribute of first step；The system of process business can also be according to before first step
The way of output of individual step, determines the tupe attribute of this first step.Below in conjunction with Fig. 2, right
Above-mentioned two situations are described respectively.
As in figure 2 it is shown, the method 100 of process business according to embodiments of the present invention can also include:
S130, receives the service request for asking process business that user sends, and this service request includes
For processing the tupe attribute of each step of this business；
Now, S110 includes: according to this service request, determines first step for processing this business
Tupe attribute.
Such as, user can send service request by service template to system, and this business module can show
Formula instruction is for processing the tupe attribute of each step of this business, and thus system please according to this business
Ask, it may be determined that the tupe attribute of each step.
On the other hand, the system processing business can also be according to a step before immediately this first step
The way of output whether use subregion (Partition) way of output, determine the process mould of this first step
Formula attribute.Specifically, when the way of output of the previous step of this first step is subregion output, can
With the tupe attribute that determines this first step for stream tupe；Previous step when this first step
Way of output when not using partitioned mode, it may be determined that the tupe attribute of this first step for batch at
Reason pattern.
In S120, the system creation processing business is corresponding with the tupe attribute of this first step
First processing unit.
In embodiments of the present invention, tupe attribute includes batch mode and stream tupe, accordingly
Ground, the system processing business can create at batch unit or stream according to the tupe attribute of step
Reason unit.It is described below in conjunction with the method 120 creating processing unit shown in Fig. 3.
As it is shown on figure 3, alternatively, the method 120 includes:
S121, when the tupe attribute of this first step is this batch mode, creates this at first
First processing unit that reason unit includes, the key-value pair in input rank is entered by this first processing unit
Row processes；
S122, when the tupe attribute at this first step is this stream tupe, creates this at first
The first-class processing unit that reason unit includes, at this first-class processing unit key-value pair to receiving
Reason.
Therefore, the method for the process business of the embodiment of the present invention, by according to each step for processing business
Rapid tupe attribute, creates with this corresponding processing unit of tupe attribute to perform each step,
The processing mode making business is more flexible, more diversified such that it is able to process phase with batch processing and stream
In conjunction with mode process business, it is possible to the treatment effeciency of raising system, improve the performance of system, and
The experience of user can be improved.
In embodiments of the present invention, process business system at the tupe attribute according to first step,
After creating corresponding first processing unit, or meanwhile, it is also possible to according to following this first step closely
The tupe attribute of second step afterwards, processes accordingly, to improve the treatment effeciency of system,
The performance of raising system.
Below in conjunction with Fig. 4 to Fig. 6, it is respectively directed to following three situation and the embodiment of the present invention is carried out in detail
Thin describe, i.e. the tupe attribute of second step be batch mode, stream tupe and this first
Step is last step of process business.Should be understood that this second step is to follow closely for processing business
First step after a step, this first step is any one in the step for processing business
Individual step.
Fig. 4 shows the indicative flowchart of the method 200 of process business according to embodiments of the present invention.
Alternatively, as shown in Figure 4, the method 200 includes:
S210, is this batch processing mould at the tupe attribute following the second step after this first step closely
During formula, create the input rank of this second step；
S220, is set to the ground of the input rank of this second step by the OPADD of this first processing unit
Location, the input rank of this second step is for defeated for processing the second batch processing unit offer of this second step
Enter.
The service request that the system of process business can send according to user, determines the process mould of second step
Formula attribute；This system according to the way of output of first step, can also determine the process mould of this second step
Formula attribute, specifically: when the way of output at this first step is subregion output, determine this second step
Rapid tupe attribute is this stream tupe；The way of output at this first step does not uses square partition
During formula, determine that the tupe attribute of this second step is this batch mode.
When the tupe attribute that the system of the business of process determines second step is batch mode, system
Can create the input rank of this second step, such as request Distributed Queuing System creates this second step
Input rank.When this input rank creates successfully, system can be by the output of this first processing unit
Address is set to the address of the input rank of this second step, and the input rank of this second step is for being used for locating
The second batch processing unit managing this second step provides input.
Should be understood that, after the system of process business has performed S210, S220, this system can circulate
Perform S110, S120, create the second processing unit for processing this second step in a similar fashion,
And also can carry out corresponding according to the tupe attribute of the third step followed closely after this second step
Process, to improve the treatment effeciency of system, improve system performance.For sake of simplicity, it is the most superfluous at this
State.
Should also be understood that in various embodiments of the present invention, the size of the sequence number of above-mentioned each process unawareness
Taste the priority of execution sequence, and the execution sequence of each process should determine with its function and internal logic, and not
The implementation process of the reply embodiment of the present invention constitutes any restriction.
Fig. 5 shows the indicative flowchart of the method 300 of process business according to embodiments of the present invention.
Alternatively, as it is shown in figure 5, the method 300 includes:
S310, is that this stream processes mould at the tupe attribute following the second step after this first step closely
During formula, perform partitioning algorithm that is that set or that give tacit consent to；
S320, the subregion number set according to this partitioning algorithm, determine for process this second step
The quantity of two stream processing units, in order to create second processing unit corresponding with this quantity；
S330, creates the division route information table according to this partitioning algorithm, in order to according to this subregion route letter
Breath table, is sent to this second processing unit corresponding by the output result of this first processing unit.
When the system of the business of process determines the tupe attribute of second step for stream tupe, system
Can perform partitioning algorithm, this partitioning algorithm can be specified by user in advance, it would however also be possible to employ system default
Partitioning algorithm, this partitioning algorithm e.g. Hash (HASH) algorithm, sample partitioning algorithm etc..Place
The subregion number that the system of reason business sets according to this partitioning algorithm, it may be determined that be used for processing this second step
The quantity of rapid second processing unit, in order to create second processing unit corresponding with this quantity.
The system of process business can also create the division route information table according to this partitioning algorithm, in order to
The output result of this first processing unit, according to this subregion route information table, is sent to phase by one processing unit
This second processing unit answered.This subregion routing table such as can include partition identification, such as key Key
Deng, it is also possible to include processing the positional information of the processing unit of task, such as IP address etc. in this subregion.
This subregion routing table can be saved in distributed coordination system or the server of such as CHUBBY.This
Inventive embodiments only illustrates as example, but the embodiment of the present invention is not limited to this.
In embodiments of the present invention, when the tupe attribute of second step is for stream tupe, it is used for
Output result is directly sent to for processing second step phase by the first processing unit processing first step
The second processing unit answered, need not store output result in queue；Meanwhile, second processes single
When business is processed by unit, can directly obtain the output result that the first processing unit sends, without
Frequently read queue and obtain the data of input.Therefore, the method for the embodiment of the present invention can be prevented effectively from many
The problem of secondary reading queue such that it is able to improve the treatment effeciency of system further, improves the performance of system,
And the experience of user can be improved.
In embodiments of the present invention, when this first step is last step processing this business, can
Selection of land, as shown in Figure 6, method 400 according to embodiments of the present invention includes:
S410, creates the output queue of this first step；
S420, is set to the ground of the output queue of this first step by the OPADD of this first processing unit
Location.
When this first step is last step processing this business, the system processing business such as please
Distributed Queuing System is asked to create the output queue of this first step, and by the output of this first processing unit
Address is set to the address of the output queue of this first step.
Therefore, the method for the process business of the embodiment of the present invention, by according to each step for processing business
Rapid tupe attribute, creates with this corresponding processing unit of tupe attribute to perform each step,
The processing mode making business is more flexible, more diversified such that it is able to process phase with batch processing and stream
In conjunction with mode process business, it is possible to the treatment effeciency of raising system, improve the performance of system, and
The experience of user can be improved.
As a example by below the system to process business being included scheduler module and node administration module, and combine figure
The method 500 of 7 pairs of process business according to embodiments of the present invention is described in detail.
As it is shown in fig. 7, the method 500 includes:
S501, what scheduler module reception user sent is used for asking to process the business mould of parallel transcoding business
Plate, this service template indicates this parallel transcoding business to include segmenting step, transcoding step and merging step successively
Suddenly, wherein the tupe attribute of this segmenting step is batch mode (Batch), this transcoding step
Tupe attribute is batch mode (Batch), and the tupe attribute of this combining step is that stream processes
Pattern (Streaming), and the partitioning algorithm that this service template indicates this combining step to use is Hash
Algorithm；Scheduler module is by resolving this service template, it may be determined that first process the segmenting step of this business,
And the tupe attribute that may determine that segmenting step is batch mode；
S502, scheduler module creates the input rank of segmenting step, this point to Distributed Queuing System request
The simple queue service of cloth queue system for example, Amazon (Amazon) (Simple Queue Service,
Referred to as " SQS ") etc.；
S503, Distributed Queuing System returns to the address of the input rank created to scheduler module；
S504, scheduler module sends request to node administration module, and requesting node management module 1 is burst
Step creates batch unit (Batch Worker), and the input rank arranging this batch unit is S503
The address of the input rank of middle return, and load user's processing routine；
S505, node administration module 1 creates successfully backward scheduler module and returns success response；
S506, scheduler module is by resolving service template, it may be determined that need to process after segmenting step
Transcoding step, and determine that the tupe attribute of this transcoding step is batch mode；
S507, scheduler module creates the input rank of transcoding step to Distributed Queuing System request；
S508, Distributed Queuing System returns to the address creating queue to scheduler module；
S509, scheduler module requesting node management module 1 arranges the defeated of the output module of burst processing unit
Going out address is the queue address returned in S507；
S510, node administration module 1 returns and arranges success response；
S511, scheduler module requesting node management module 2 creates batch unit (Batch for transcoding step
Worker), the input rank arranging this Batch Worker is the queue address returned in S508, and adds
Carry user's processing routine；
S512, after node administration module 2 creates successfully, returns success response；
S513, scheduler module, by resolving service template, determines and finally processes combining step, and determine this
The tupe attribute of step is stream tupe (Streaming Worker)；
S514, scheduler module determines the processing unit of combining step according to the subregion number that partitioning algorithm sets
(Worker) number；
S515, scheduler module requesting node management module 3 creates stream processing unit for combining step
(Streaming Worker)；
S516, after node administration module 3 creates successfully, returns success response；
S517, scheduler module creates the division routing table, and this subregion routing table is stored in distributed coordination
In coordination；
S518, scheduler module requesting node management module 2 boot partition module, to load the subregion set
Algorithm, and set the reference address of subregion routing table；
S519, node administration module 2 returns and starts success response；
S520, scheduler module continues to resolve service template, and determines when not having subsequent step, scheduler module
Queue is created to Distributed Queuing System request；
S521, Distributed Queuing System returns to the address of the queue created；
S522, scheduler module requesting node management module 3 arranges the output ground of the output module of combining step
Location is the queue address returned in S521；
S523, node administration module 3 returns and arranges success response.
Should be understood that in various embodiments of the present invention, the size of the sequence number of above-mentioned each process is not intended to
The priority of execution sequence, the execution sequence of each process should determine with its function and internal logic, and should not
The implementation process of the embodiment of the present invention is constituted any restriction.
Therefore, the method for the process business of the embodiment of the present invention, by according to each step for processing business
Rapid tupe attribute, creates with this corresponding processing unit of tupe attribute to perform each step,
The processing mode making business is more flexible, more diversified such that it is able to process phase with batch processing and stream
In conjunction with mode process business, it is possible to the treatment effeciency of raising system, improve the performance of system, and
The experience of user can be improved.
Above in conjunction with Fig. 1 to Fig. 7, describe the side of process business according to embodiments of the present invention in detail
Method, below in conjunction with Fig. 8 to Figure 14, the system of description process business according to embodiments of the present invention.
Fig. 8 shows the schematic block diagram of the system 600 of process business according to embodiments of the present invention.As
Shown in Fig. 8, this system 600 includes scheduler module 700 and node administration module 800, this scheduler module
700 include:
First determines unit 710, for determining the tupe attribute of the first step for processing business,
The tupe attribute of this first step includes batch mode or stream tupe, and this first step is for using
Any one step in the step processing this business；
This node administration module 800 includes:
First creating unit 810, for according to the tupe attribute of this first step, create with this
Corresponding first processing unit of tupe attribute of one step, to process this first step.
Therefore, the system of the process business of the embodiment of the present invention, by according to each step for processing business
Rapid tupe attribute, creates with this corresponding processing unit of tupe attribute to perform each step,
The processing mode making business is more flexible, more diversified such that it is able to process phase with batch processing and stream
In conjunction with mode process business, it is possible to the treatment effeciency of raising system, improve the performance of system, and
The experience of user can be improved.
In embodiments of the present invention, alternatively, as it is shown in figure 9, this scheduler module 700 also includes:
Receive unit 720, for receiving the service request for asking to process this business that user sends,
This service request includes the tupe attribute of each step for processing this business；
This first determines that unit 710 is additionally operable to: according to this service request, determine the process of this first step
Mode attribute.
In embodiments of the present invention, alternatively, as shown in Figure 10, this first creating unit 810 includes:
First creates subelement 811, is this batch processing mould for the tupe attribute at this first step
During formula, creating first processing unit that this first processing unit includes, this first processing unit is to defeated
Key-value pair in enqueue processes；
Second creates subelement 812, and being used at the tupe attribute of this first step is that this stream processes mould
During formula, creating the first-class processing unit that this first processing unit includes, this first-class processing unit docks
The key-value pair received processes.
Alternatively, as shown in figure 11, this scheduler module 700 also includes:
Second creating unit 730, for the tupe at the second step followed closely after this first step
When attribute is this batch mode, create the input rank of this second step；
This node administration module 800 also includes:
First arranges unit 820, for the OPADD of this first processing unit is set to this second step
The address of rapid input rank, the input rank of this second step is for processing the second of this second step
Batch unit provides input.
Alternatively, as shown in figure 12, this node administration module 800 also includes:
Zoning unit 830, for following the tupe attribute of the second step after this first step closely
During for this stream tupe, perform partitioning algorithm that is that set or that give tacit consent to；
This scheduler module 700 also includes:
Second determines unit 740, for the subregion number set according to this partitioning algorithm, determines for locating
Manage the quantity of the second processing unit of this second step, in order to create second corresponding with this quantity
Processing unit；
3rd creating unit 750, for creating the division route information table according to this partitioning algorithm, in order to
According to this subregion route information table, the output result of this first processing unit is sent to corresponding this second
Stream processing unit.
Alternatively, as described in Figure 13, this scheduler module 700 also includes:
4th creating unit 760, is used for when this first step is last step processing this business,
Create the output queue of this first step；
This node administration module 800 also includes:
Second arranges unit 840, for the OPADD of this first processing unit is set to this first step
The address of rapid output queue.
In embodiments of the present invention, alternatively, as shown in figure 14, this scheduler module 700 also includes:
3rd determines unit 770, for when the way of output of this first step is subregion output, determining
The tupe attribute of this second step is this stream tupe；
4th determines unit 780, is used for when the way of output of this first step does not uses partitioned mode,
The tupe attribute determining this second step is this batch mode.
Should be understood that the process that system 600 according to embodiments of the present invention may correspond in the embodiment of the present invention
Above and other operation of the modules in the system of business, and system 600 and/or function are respectively
Realize the corresponding flow process of each method 100 to 500 in Fig. 1 to Fig. 7, for sake of simplicity, at this not
Repeat again.
Therefore, the system of the process business of the embodiment of the present invention, by according to each step for processing business
Rapid tupe attribute, creates with this corresponding processing unit of tupe attribute to perform each step,
The processing mode making business is more flexible, more diversified such that it is able to process phase with batch processing and stream
In conjunction with mode process business, it is possible to the treatment effeciency of raising system, improve the performance of system, and
The experience of user can be improved.
Those of ordinary skill in the art are it is to be appreciated that combine each of the embodiments described herein description
The unit of example and algorithm steps, it is possible to come real with electronic hardware, computer software or the combination of the two
Existing, in order to clearly demonstrate the interchangeability of hardware and software, the most according to function one
As property describe composition and the step of each example.These functions are held with hardware or software mode actually
OK, application-specific and the design constraint of technical scheme are depended on.Professional and technical personnel can be to each
Specifically should be used for using different methods to realize described function, but this realization is it is not considered that surpass
Go out the scope of the present invention.
Those skilled in the art is it can be understood that arrive, for convenience of description and succinctly, above-mentioned
The specific works process of the system, device and the unit that describe, be referred in preceding method embodiment is right
Answer process, do not repeat them here.
In several embodiments provided herein, it should be understood that disclosed system, device and
Method, can realize by another way.Such as, device embodiment described above is only shown
Meaning property, such as, the division of described unit, be only a kind of logic function and divide, actual can when realizing
There to be other dividing mode, the most multiple unit or assembly can in conjunction with or be desirably integrated into another
System, or some features can ignore, or do not perform.It addition, shown or discussed each other
Coupling or direct-coupling or communication connection can be the INDIRECT COUPLING by some interfaces, device or unit or
Communication connection, it is also possible to be electric, machinery or other form connect.
The described unit illustrated as separating component can be or may not be physically separate, makees
The parts shown for unit can be or may not be physical location, i.e. may be located at a place,
Or can also be distributed on multiple NE.Can select according to the actual needs part therein or
The whole unit of person realizes the purpose of embodiment of the present invention scheme.
It addition, each functional unit in each embodiment of the present invention can be integrated in a processing unit
In, it is also possible to it is that unit is individually physically present, it is also possible to be that two or more unit are integrated in
In one unit.Above-mentioned integrated unit both can realize to use the form of hardware, it would however also be possible to employ software
The form of functional unit realizes.
If described integrated unit realizes and as independent product pin using the form of SFU software functional unit
When selling or use, can be stored in a computer read/write memory medium.Based on such understanding,
The part that prior art is contributed by technical scheme the most in other words, or this technical side
Completely or partially can embodying with the form of software product of case, this computer software product is stored in
In one storage medium, including some instructions with so that a computer equipment (can be individual calculus
Machine, server, or the network equipment etc.) perform the whole or portion of method described in each embodiment of the present invention
Step by step.And aforesaid storage medium includes: USB flash disk, portable hard drive, read only memory (ROM,
Read-Only Memory), random access memory (RAM, Random Access Memory),
The various medium that can store program code such as magnetic disc or CD.
The above, the only detailed description of the invention of the present invention, but protection scope of the present invention is not limited to
In this, any those familiar with the art, can be easily in the technical scope that the invention discloses
Expecting amendment or the replacement of various equivalence, these amendments or replacement all should be contained in protection scope of the present invention
Within.Therefore, protection scope of the present invention should be as the criterion with scope of the claims.
Claims (10)
1. the method processing business, it is characterised in that including:
Determine the tupe attribute of first step for processing business, the process mould of described first step
Formula attribute includes batch mode or stream tupe, and described first step is for processing described business
Any one step in step；
Tupe attribute according to described first step, creates the tupe with described first step and belongs to
Corresponding first processing unit of property, to process described first step；
When the way of output at described first step is subregion output, determine after following described first step closely
The tupe attribute of second step be described stream tupe；
When the way of output of described first step does not uses partitioned mode, determine the place of described second step
Reason mode attribute is described batch mode.
Method the most according to claim 1, it is characterised in that described method also includes:
What reception user sent is used for asking to process the service request of described business, and described service request includes
For processing the tupe attribute of each step of described business；
The tupe attribute of the described first step determined for processing business, including:
According to described service request, determine the tupe attribute of described first step.
Method the most according to claim 1, it is characterised in that described establishment and the described first step
Corresponding first processing unit of rapid tupe attribute, including:
When the tupe attribute of described first step is described batch mode, create at described first
First processing unit that reason unit includes, first processing unit described is to the key-value pair in input rank
Process；
When the tupe attribute of described first step is described stream tupe, create at described first
The first-class processing unit that reason unit includes, at the described first-class processing unit key-value pair to receiving
Reason.
The most according to the method in any one of claims 1 to 3, it is characterised in that described method
Also include:
When the tupe attribute of described second step is described batch mode, create described second step
Rapid input rank；
The OPADD of described first processing unit is set to the ground of the input rank of described second step
Location, the input rank of described second step is to carry for processing the second batch processing unit of described second step
For input.
The most according to the method in any one of claims 1 to 3, it is characterised in that described method
Also include:
When the tupe attribute of described second step is described stream tupe, perform that set or silent
The partitioning algorithm recognized；
The subregion number set according to described partitioning algorithm, determine for process described second step second
The quantity of stream processing unit, in order to create second processing unit corresponding with described quantity；
Create the division route information table according to described partitioning algorithm, in order to according to described subregion routing iinformation
Table, is sent to corresponding described second processing unit by the output result of described first processing unit.
6. the system processing business, it is characterised in that described system includes scheduler module and node
Management module, described scheduler module includes:
First determines unit, for determining the tupe attribute of the first step for processing business, institute
The tupe attribute stating first step includes batch mode or stream tupe, and described first step is
Any one step in the step processing described business；
Described node administration module includes:
First creating unit, for according to the tupe attribute of described first step, creates and described the
Corresponding first processing unit of tupe attribute of one step, to process described first step；
Described scheduler module also includes:
3rd determines unit, for when the way of output of described first step is subregion output, determining tight
It is described stream tupe with the tupe attribute of the second step after described first step；
4th determines unit, is used for when the way of output of described first step does not uses partitioned mode, really
The tupe attribute of fixed described second step is described batch mode.
System the most according to claim 6, it is characterised in that described scheduler module also includes:
Receive unit, for receiving the service request for asking to process described business that user sends, institute
State service request and include the tupe attribute of each step for processing described business；
Described first determines that unit is additionally operable to: according to described service request, determine the place of described first step
Reason mode attribute.
System the most according to claim 6, it is characterised in that described first creating unit includes:
First creates subelement, is described batch processing mould for the tupe attribute at described first step
During formula, create first processing unit that described first processing unit includes, first processing unit described
Key-value pair in input rank is processed；
Second creates subelement, and being used at the tupe attribute of described first step is that described stream processes mould
During formula, create the first-class processing unit that described first processing unit includes, described first-class processing unit
The key-value pair received is processed.
9. according to the system according to any one of claim 6 to 8, it is characterised in that described scheduling
Module also includes:
Second creating unit, is described batch mode for the tupe attribute at described second step
Time, create the input rank of described second step；
Described node administration module also includes:
First arranges unit, for the OPADD of described first processing unit is set to described second step
The address of rapid input rank, the input rank of described second step is for processing described second step
Second batch processing unit provides input.
10. according to the system according to any one of claim 6 to 8, it is characterised in that described node
Management module also includes:
Zoning unit, is used for when the tupe attribute of described second step is described stream tupe,
Perform partitioning algorithm that is that set or that give tacit consent to；
Described scheduler module also includes:
Second determines unit, for the subregion number set according to described partitioning algorithm, determines for processing
The quantity of the second processing unit of described second step, in order to create corresponding with described quantity second
Stream processing unit；
3rd creating unit, for creating the division route information table according to described partitioning algorithm, in order to root
According to described subregion route information table, the output result of described first processing unit is sent to corresponding described
Second processing unit.
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| PCT/CN2011/085176 WO2013097234A1 (en) | 2011-12-31 | 2011-12-31 | Service processing method and system |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN103299298A CN103299298A (en) | 2013-09-11 |
| CN103299298B true CN103299298B (en) | 2016-11-02 |
Family
ID=48696287
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201180003695.3A Active CN103299298B (en) | 2011-12-31 | 2011-12-31 | The method and system of process business |
Country Status (2)
| Country | Link |
|---|---|
| CN (1) | CN103299298B (en) |
| WO (1) | WO2013097234A1 (en) |
Families Citing this family (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN106873945A (en) * | 2016-12-29 | 2017-06-20 | 中山大学 | Data processing architecture and data processing method based on batch processing and Stream Processing |
| CN110110170B (en) * | 2019-04-30 | 2021-12-07 | 北京字节跳动网络技术有限公司 | Data processing method, device, medium and electronic equipment |
| WO2021035517A1 (en) * | 2019-08-27 | 2021-03-04 | Microsoft Technology Licensing, Llc | Workflow-based scheduling and batching in multi-tenant distributed systems |
| CN112822125B (en) * | 2020-04-08 | 2023-08-01 | 中兴通讯股份有限公司 | Method, device, equipment and storage medium for transmitting service flow |
| CN112150248B (en) * | 2020-09-30 | 2021-08-03 | 欧冶云商股份有限公司 | Method, system and device for counting hung goods amount based on batch flow fusion |
Citations (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US5878196A (en) * | 1996-03-27 | 1999-03-02 | Fuji Xerox Co., Ltd. | Printer control system |
| CN101197827A (en) * | 2007-12-14 | 2008-06-11 | 华为技术有限公司 | Document management method, system and correlated equipment |
| CN101841476A (en) * | 2010-04-22 | 2010-09-22 | 北京星网锐捷网络技术有限公司 | Message processing method, device and network equipment |
Family Cites Families (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN101459848B (en) * | 2008-12-26 | 2010-11-10 | 西安交通大学 | Distributed multi-format digital video transcoding construction design method |
| JP4967014B2 (en) * | 2009-12-16 | 2012-07-04 | 株式会社日立製作所 | Stream data processing apparatus and method |
-
2011
- 2011-12-31 CN CN201180003695.3A patent/CN103299298B/en active Active
- 2011-12-31 WO PCT/CN2011/085176 patent/WO2013097234A1/en active Application Filing
Patent Citations (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US5878196A (en) * | 1996-03-27 | 1999-03-02 | Fuji Xerox Co., Ltd. | Printer control system |
| CN101197827A (en) * | 2007-12-14 | 2008-06-11 | 华为技术有限公司 | Document management method, system and correlated equipment |
| CN101841476A (en) * | 2010-04-22 | 2010-09-22 | 北京星网锐捷网络技术有限公司 | Message processing method, device and network equipment |
Also Published As
| Publication number | Publication date |
|---|---|
| CN103299298A (en) | 2013-09-11 |
| WO2013097234A1 (en) | 2013-07-04 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN102469033B (en) | Message subscription system and message sending method | |
| US9742667B2 (en) | Packet processing method, device and system | |
| CN103299298B (en) | The method and system of process business | |
| CN102932472B (en) | Data processing method and communication system | |
| CN101163117B (en) | Packet management method, packet resource sharing method and instant communication equipment | |
| CN103797463A (en) | Method and apparatus for assignment of virtual resources within a cloud environment | |
| CN107332814B (en) | Request message transmission method and device | |
| CN112583931A (en) | Message processing method, message middleware, electronic device and storage medium | |
| CN109413202B (en) | System and method for sorting block chain transaction information | |
| CN107562803B (en) | Data supply system and method and terminal | |
| CN108418752A (en) | A kind of creation method and device of aggregation group | |
| CN104158860A (en) | Job scheduling method and job scheduling system | |
| CN108933675B (en) | Conference resource allocation method, device and storage medium | |
| CN111835809B (en) | Work order message distribution method, work order message distribution device, server and storage medium | |
| CN105656769A (en) | Service data processing method, apparatus and system | |
| CN116582173A (en) | Method, device and storage medium for processing data by satellite-based distributed network | |
| CN104022896A (en) | Configuration management method and device and network equipment | |
| CN110086722A (en) | A kind of routed path determines method and device, routing device | |
| CN111309467B (en) | Task distribution method and device, electronic equipment and storage medium | |
| CN107710793A (en) | A kind of localization method and corresponding intrument | |
| CN106487858B (en) | Information method for uploading and device | |
| CN104852858B (en) | A kind of flow forwarding method and equipment | |
| CN115529276A (en) | Data transmission method and related equipment thereof | |
| CN107851058A (en) | Storage system and apparatus scanning method | |
| CN111340413A (en) | Distribution method, distribution device, server and storage medium of distribution tasks |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| C06 | Publication | ||
| PB01 | Publication | ||
| C10 | Entry into substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| C14 | Grant of patent or utility model | ||
| GR01 | Patent grant | ||
| TR01 | Transfer of patent right |
Effective date of registration: 20220211 Address after: 550025 Huawei cloud data center, jiaoxinggong Road, Qianzhong Avenue, Gui'an New District, Guiyang City, Guizhou Province Patentee after: Huawei Cloud Computing Technologies Co.,Ltd. Address before: 518129 Bantian HUAWEI headquarters office building, Longgang District, Guangdong, Shenzhen Patentee before: HUAWEI TECHNOLOGIES Co.,Ltd. |
|
| TR01 | Transfer of patent right |