CN112866136B - Service data processing method and device - Google Patents
Service data processing method and device Download PDFInfo
- Publication number
- CN112866136B CN112866136B CN202110352525.1A CN202110352525A CN112866136B CN 112866136 B CN112866136 B CN 112866136B CN 202110352525 A CN202110352525 A CN 202110352525A CN 112866136 B CN112866136 B CN 112866136B
- Authority
- CN
- China
- Prior art keywords
- data processing
- transaction
- service data
- request
- token
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
Images
Classifications
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L47/00—Traffic control in data switching networks
- H04L47/10—Flow control; Congestion control
- H04L47/215—Flow control; Congestion control using token-bucket
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L47/00—Traffic control in data switching networks
- H04L47/50—Queue scheduling
- H04L47/62—Queue scheduling characterised by scheduling criteria
- H04L47/6215—Individual queue per QOS, rate or priority
-
- Y—GENERAL TAGGING OF NEW TECHNOLOGICAL DEVELOPMENTS; GENERAL TAGGING OF CROSS-SECTIONAL TECHNOLOGIES SPANNING OVER SEVERAL SECTIONS OF THE IPC; TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
- Y02—TECHNOLOGIES OR APPLICATIONS FOR MITIGATION OR ADAPTATION AGAINST CLIMATE CHANGE
- Y02D—CLIMATE CHANGE MITIGATION TECHNOLOGIES IN INFORMATION AND COMMUNICATION TECHNOLOGIES [ICT], I.E. INFORMATION AND COMMUNICATION TECHNOLOGIES AIMING AT THE REDUCTION OF THEIR OWN ENERGY USE
- Y02D10/00—Energy efficient computing, e.g. low power processors, power management or thermal management
Landscapes
- Engineering & Computer Science (AREA)
- Computer Networks & Wireless Communication (AREA)
- Signal Processing (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
The application provides a service data processing method and device, wherein the method comprises the following steps: receiving a service data processing request; determining the priority of the service data processing request according to the service type identifier in the service data processing request; storing the business data processing requests into corresponding transaction queues based on the priorities of the business data processing requests, storing the business data processing requests with high priorities into a first transaction queue, and storing the business data processing requests with low priorities into a second transaction queue; reading a business data processing request from a first transaction queue by utilizing a first transaction processing module and executing corresponding business data processing; reading a business data processing request from a second transaction queue by utilizing a second transaction processing module and executing corresponding business data processing; the first transaction processing module and the second transaction processing module are different transaction processing modules. The scheme can realize transaction diversion, relieve transaction processing blockage and improve processing efficiency.
Description
Technical Field
The present invention relates to the field of service data processing technologies, and in particular, to a service data processing method and device.
Background
As traffic increases, so does the amount of transaction calls for existing business systems. In the face of large numbers of cyclically invoked transactions by some customers in a short period of time, system performance pressures proliferate, resulting in transaction processing blocking. More seriously, during transaction blocking, repeated and adjusted transactions with multiple increases are brought about, and finally avalanche effect is formed, so that large-area system paralysis is caused.
In view of the above problems, no effective solution has been proposed at present.
Disclosure of Invention
The embodiment of the application provides a business data processing method and a business data processing device, which are used for solving the problem of system transaction processing blocking caused by continuous increase of business volume in the prior art.
The embodiment of the application provides a service data processing method, which is applied to a service data processing system and comprises the following steps: receiving a service data processing request, wherein the service data processing request carries a service type identifier; determining the priority of a service data processing request according to the service type identifier; storing the business data processing requests into corresponding transaction queues based on the priorities of the business data processing requests, wherein the transaction queues comprise a first transaction queue and a second transaction queue, the business data processing requests with high priorities are stored into the first transaction queue, and the business data processing requests with low priorities are stored into the second transaction queue; reading a business data processing request from a first transaction queue by utilizing a first transaction processing module and executing corresponding business data processing; reading a business data processing request from a second transaction queue by utilizing a second transaction processing module and executing corresponding business data processing; the first transaction processing module and the second transaction processing module are different transaction processing modules in the business data processing system.
In one embodiment, transaction processing modules in a business data processing system are deployed in a containerized manner; correspondingly, the method further comprises the steps of: collecting transaction processing information generated when the first transaction processing module and the second transaction processing module execute business data processing, and the quantity of business data processing requests to be processed in the first transaction queue and the second transaction queue; calculating a load factor of the service data processing system by utilizing the transaction processing information and the number of the service data processing requests to be processed, wherein the load factor is used for representing the load degree of the service data processing system; and carrying out elastic transverse expansion and contraction on the service data processing system based on the load coefficient of the service data processing system and the number of container instances of the service data processing system.
In one embodiment, depositing the business data processing request into a corresponding transaction queue based on the priority of the business data processing request includes: determining whether the number of tokens remaining in the token container is less than a preset number; under the condition that the number of the tokens remained in the token container is not less than the preset number, acquiring the preset number of tokens from the token container, and storing the business data processing request into a corresponding transaction queue based on the priority of the business data processing request; and under the condition that the number of the tokens remained in the token container is less than the preset number, carrying out flow limiting processing on the business data processing request.
In one embodiment, the method further comprises: acquiring a load coefficient of a service data processing system; calculating the token replenishment rate in a token container of the business data processing system according to the load coefficient; tokens are added to the token container based on the token replenishment rate.
In one embodiment, after calculating the token replenishment rate in the token container of the business data processing system based on the load factor, further comprising: determining whether the token replenishment rate is less than a preset rate; and under the condition that the token supplement rate is smaller than the preset rate, transversely expanding the service data processing system to reduce the load coefficient of the service data processing system.
In one embodiment, calculating a token replenishment rate in a token container of a business data processing system based on a load factor comprises: the token replenishment rate in the token container is calculated according to the following formula:
R＝i/L；
wherein R is the token replenishment rate, i is the preset replenishment rate, and L is the load factor.
In one embodiment, the service data processing request further carries a request parameter, and the method further includes: the method comprises the steps of storing request parameters of a service data processing request and processing results corresponding to the service data processing request in a preset cache in an associated mode; after receiving the target service data processing request, inquiring whether a request parameter in the target service data processing request exists in a preset cache; and under the condition that the request parameters in the target business data processing request are inquired, determining the processing result corresponding to the searched request parameters as the processing result corresponding to the target business data processing request.
The embodiment of the application also provides a service data processing device, which is applied to a service data processing system and comprises: the request receiving module is used for receiving a service data processing request, wherein the service data processing request carries a service type identifier; the priority determining module is used for determining the priority of the service data processing request according to the service type identifier; the request storage module is used for storing the business data processing requests into corresponding transaction queues based on the priorities of the business data processing requests, wherein the transaction queues comprise a first transaction queue and a second transaction queue, the business data processing requests with high priorities are stored into the first transaction queue, and the business data processing requests with low priorities are stored into the second transaction queue; the data processing module is used for reading the business data processing request from the first transaction queue by utilizing the first transaction processing module and executing corresponding business data processing; reading a business data processing request from a second transaction queue by utilizing a second transaction processing module and executing corresponding business data processing; the first transaction processing module and the second transaction processing module are different transaction processing modules in the business data processing system.
The embodiment of the application also provides a computer device, which comprises a processor and a memory for storing instructions executable by the processor, wherein the steps of the service data processing method in any embodiment are realized when the instructions are executed by the processor.
The embodiments of the present application further provide a computer readable storage medium having stored thereon computer instructions that when executed implement the steps of the service data processing method described in any of the above embodiments.
In this embodiment of the present application, a service data processing method is provided, which may receive a service data processing request, determine a priority of the service data processing request according to a service type identifier carried in the service data processing request, store a service data processing request with a high priority in a first transaction queue, store a service data processing request with a low priority in a second transaction queue, and read the service data processing request from the first transaction queue by using a first transaction processing module and execute corresponding service data processing, and read the service data processing request from the second transaction queue by using a second transaction processing module and execute corresponding service data processing, where the first transaction processing module and the second transaction processing module are different transaction processing modules in the service data processing system. In the scheme, the priority of the business data processing request is determined according to the business type, the requests with different priorities are stored in different transaction queues, the different transaction processing modules are utilized to read the requests from the corresponding transaction queues and execute business data processing, for the classification processing of the high-low priority transactions, each transaction processing module only processes the transactions with high priority or low priority and does not process the transactions with different priority levels at the same time, so that the transactions with low priority can be ensured not to block the transaction processing with high priority, the transaction diversion is realized, the transaction processing blocking is relieved, and the transaction processing efficiency can be effectively improved.
Drawings
The accompanying drawings are included to provide a further understanding of the application, and are incorporated in and constitute a part of this application. In the drawings:
FIG. 1 is a flow chart of a business data processing method according to an embodiment of the present application;
FIG. 2 is a schematic diagram illustrating a method of transaction throttling through a token container in one embodiment of the present application;
FIG. 3 illustrates a flow chart of transaction outcome caching in an embodiment of the present application;
FIG. 4 shows a schematic diagram of a traffic data processing apparatus in an embodiment of the present application;
fig. 5 shows a schematic diagram of a computer device in an embodiment of the present application.
Detailed Description
The principles and spirit of the present application will be described below with reference to several exemplary embodiments. It should be understood that these embodiments are presented merely to enable one skilled in the art to better understand and practice the present application and are not intended to limit the scope of the present application in any way. Rather, these embodiments are provided so that this disclosure will be thorough and complete, and will fully convey the scope of the disclosure to those skilled in the art.
Those skilled in the art will appreciate that embodiments of the present application may be implemented as a system, apparatus device, method, or computer program product. Accordingly, the present disclosure may be embodied in the following forms, namely: complete hardware, complete software (including firmware, resident software, micro-code, etc.), or a combination of hardware and software.
The embodiment of the application provides a business data processing method which is applied to a business data processing system. Fig. 1 shows a flowchart of a service data processing method in an embodiment of the present application. Although the present application provides a method operation step or apparatus structure as shown in the following examples or figures, more or fewer operation steps or module units may be included in the method or apparatus based on routine or non-inventive labor. In the steps or structures where there is no necessary causal relationship logically, the execution order of the steps or the module structure of the apparatus is not limited to the execution order or the module structure shown in the drawings and described in the embodiments of the present application. The described methods or module structures may be implemented sequentially or in parallel (e.g., in a parallel processor or multithreaded environment, or even in a distributed processing environment) in accordance with the embodiments or the method or module structure connection illustrated in the figures when implemented in a practical device or end product application.
Specifically, as shown in fig. 1, the service data processing method provided in an embodiment of the present application may include the following steps.
Step S101, a service data processing request is received.
The service data processing method in the embodiment can be applied to a service data processing system. The business data may include various banking data, among others. The service data processing system may receive a service data processing request sent by the client. The service data processing request carries a service type identifier. The service type identifier may represent a type of data processing corresponding to the service data processing request. For example, the service type identification may include a query class service identification, a financial processing class service identification, and the like.
Step S102, determining the priority of the service data processing request according to the service type identifier.
After receiving the service data processing request, the priority of the service data processing request can be determined according to the service type identifier in the service data processing request. For example, the priority of the service data processing request carrying the query class service identification may be determined as high priority, and the priority of the service data processing request carrying the financial processing class service identification may be determined as low priority.
For another example, a priority relation table is stored in the service data processing system, and a correspondence between the service type identifier and the priority is stored in the priority relation table. The priority of the traffic data processing request may be determined by querying a priority relationship table.
Step S103, storing the business data processing request in a corresponding transaction queue based on the priority of the business data processing request.
After determining the priority of the business data processing request, the business data processing request may be deposited into a corresponding transaction queue based on the priority of the business data processing request. Wherein the transaction queues include a first transaction queue and a second transaction queue. The high priority traffic data processing requests may be deposited to a first transaction queue and the low priority traffic data processing requests may be deposited to a second transaction queue. The first transaction queue and the second transaction queue are two different transaction queues.
Step S104, the first transaction processing module is utilized to read the business data processing request from the first transaction queue and execute corresponding business data processing; and reading the business data processing request from the second transaction queue by utilizing the second transaction processing module and executing corresponding business data processing.
The first transaction processing module may be utilized to read the high priority transaction data processing request from the first transaction queue and execute the transaction data processing corresponding to the transaction data processor request. The second transaction processing module may be used to read the low-priority service data processing request from the second transaction queue and execute the service data processing corresponding to the service data processing request. The first transaction processing module and the second transaction processing module are different transaction processing modules in the business data processing system. That is, the first transaction queue may correspond to one or more first transaction processing modules and the second transaction queue may correspond to one or more second transaction processing modules. The first transaction processing module is only used for processing the service data processing request with high priority. The second transaction processing module is only used for processing the business data processing request with low priority.
In the above embodiment, the priority of the service data processing request is determined according to the service type, the requests with different priorities are stored in different transaction queues, and different transaction processing modules are utilized to read the requests from the corresponding transaction queues and execute the service data processing, for the classification processing of the transactions with high and low priorities, each transaction processing module only processes the transactions with high priority or low priority, and does not process the transactions with different priority levels at the same time, so that the transactions with low priority can be ensured not to block the transaction processing with high priority, the transaction diversion is realized, the transaction processing blocking is relieved, and the transaction processing efficiency can be effectively improved.
In some embodiments of the present application, transaction processing modules in a business data processing system may be deployed in a containerized manner; accordingly, the method may further include: collecting transaction processing information generated when the first transaction processing module and the second transaction processing module execute business data processing, and the quantity of business data processing requests to be processed in the first transaction queue and the second transaction queue; calculating a load factor of the service data processing system by utilizing the transaction processing information and the number of the service data processing requests to be processed, wherein the load factor is used for representing the load degree of the service data processing system; and carrying out elastic transverse expansion and contraction on the service data processing system based on the load coefficient of the service data processing system and the number of container instances of the service data processing system.
In particular, the business data processing system includes a plurality of transaction processing modules. The plurality of transaction processing modules may be deployed in a containerized manner and may be managed in a clustered manner. Transaction processing information generated when each transaction processing module executes the service data processing can be collected, and the number of the service data processing requests to be processed in each transaction queue can be collected. The transaction processing information may include information such as transaction processing timeliness and throughput generated by the transaction processing module. The load factor of the business data processing system can then be calculated using the transaction processing information and the number of business data processing requests to be processed. The load factor is used for representing the load degree of the service data processing system, and the larger the load factor is, the higher the load of the service data processing system is. The load factor is inversely proportional to the number of traffic data processing requests to be processed. After the load coefficient of the system is calculated, the current system load condition can be obtained according to the system load coefficient, and the service data processing system is elastically and transversely stretched, namely elastically and transversely expanded or contracted by comparing the current system load condition with the number of the current system container examples.
In one embodiment, system load information such as transaction processing information, transaction queue accumulation, etc. may be stored in the cache module. The transaction processing information acquisition module can be utilized to acquire information such as transaction processing timeliness, throughput and the like generated by the transaction processing module at regular time. The transaction amount to be processed in each transaction queue can be collected at fixed time by utilizing the transaction queue accumulation amount collection module. The system load condition analysis module can be triggered at fixed time, the system load coefficient L is calculated quantitatively by comprehensively analyzing transaction processing information and transaction queue accumulation amount, and the system load coefficient L is stored in the cache module. The transaction processing modules may be deployed in a containerized manner and managed in a clustered manner using a transaction processing cluster control module. And (3) by reading the load coefficient L in the cache module, corresponding to the number n of the current container examples, carrying out elastic capacity expansion or capacity contraction according to the upper threshold T and the lower threshold B of the preset load coefficient. When L > nT, the number n of the current instances is considered to be insufficient for processing the transaction request with the load coefficient of L, the cluster needs to be expanded, and k instances are newly added to meet L < = (n+k) T. When L < nB, the current system load factor L is considered to have performance redundancy for the current n instance processing, the cluster needs to be scaled, and k instances are destroyed to satisfy L > = (n-k) B. By the method, service availability can be guaranteed by expanding capacity and improving transaction processing capacity under a high-load scene; meanwhile, under the common load scene, unnecessary computing resources can be avoided being wasted through capacity shrinkage.
In some embodiments of the present application, depositing the business data processing request into the corresponding transaction queue based on the priority of the business data processing request includes: determining whether the number of tokens remaining in the token container is less than a preset number; under the condition that the number of the tokens remained in the token container is not less than the preset number, acquiring the preset number of tokens from the token container, and storing the business data processing request into a corresponding transaction queue based on the priority of the business data processing request; and under the condition that the number of the tokens remained in the token container is less than the preset number, carrying out flow limiting processing on the business data processing request.
In particular, a token container may be provided for storing a quantity of tokens, having a maximum capacity N. When the number of tokens in a container is equal to its capacity N, no more tokens can be added to it. Before entering the system, the service data processing request firstly tries to acquire c tokens from the token container, if the token allowance r in the token container is greater than or equal to c, the c tokens are removed from the container, and the service data processing request is stored in a corresponding transaction queue based on the priority of the service data processing request. If the token margin r in the token container is less than c, the request is restricted, and the number of tokens in the container is unchanged. In the mode, when the number of tokens in the token container is small, the transaction request is subjected to flow limiting processing, the processing pressure of the system is reduced, and the transaction processing is prevented from being blocked.
In some embodiments of the present application, the method may further include: acquiring a load coefficient of a service data processing system; calculating the token replenishment rate in a token container of the business data processing system according to the load coefficient; tokens are added to the token container based on the token replenishment rate.
Specifically, the load factor of the current system stored in the cache is read, the token replenishment rate in the token container is determined according to the load factor, and the token is added to the token container based on the token replenishment rate. The token replenishment rate may be inversely related to the current load factor of the system, i.e., the larger the load, the smaller the token replenishment rate, and the more restrictive the traffic data processing request. Compared with the traditional token container, the token recovery rate is constant, the token replenishment rate can be dynamically adjusted according to the dynamic load condition of the system, so that the business data transaction request can be dynamically limited, and the overlarge data processing pressure of the system is avoided.
In some embodiments of the present application, after calculating the token replenishment rate in the token container of the business data processing system according to the load factor, the method may further comprise: determining whether the token replenishment rate is less than a preset rate; and under the condition that the token supplement rate is smaller than the preset rate, transversely expanding the service data processing system to reduce the load coefficient of the service data processing system.
In particular, as the load factor increases, the token replenishment factor decreases continuously, in which case a minimum rate, i.e., a preset rate, may be set. Under the condition that the token supplementing rate is smaller than the preset rate, the processing capacity can be improved by transversely expanding the service data processing system, and then the load is reduced. By the method, under the condition that the token supplementing rate is smaller than the preset rate, the processing capacity is improved by expanding the service data processing system, so that the normal service of the system can be ensured.
In some embodiments of the present application, calculating a token replenishment rate in a token container of a business data processing system from a load factor may comprise: the token replenishment rate in the token container is calculated according to the following formula:
R＝i/L；
wherein R is the token replenishment rate, i is the preset replenishment rate, and L is the load factor. In this way, the token replenishment rate can be calculated.
Referring to fig. 2, a schematic diagram of a method for limiting flow of transactions through a token container in an embodiment of the present application is shown. The self-adaptive token generator can add the token to the token container in a timing self-adaptive manner according to the current load condition of the business data processing system. The adaptive token generator may read the current lineage stored in the cacheAnd under the condition of uniform load L, according to the preset token quantity i supplemented in unit time, calculating the rate R & lt alpha & gt i/L of supplementing the tokens, namely supplementing R tokens into the token container in unit time. Note that when the system load L increases, the R value decreases, and to ensure normal service of the system, the minimum acceptable value R of the R value is preset min When the R value touches R min And will not continue to decrease. In this case, the system may laterally expand the service data processing system based on the load factor of the service data processing system and the number of container instances of the service data processing system, so as to improve the processing capability and reduce the load. Upon receipt of the transaction request, a token may be obtained from the token container. When the token is sufficient, the token is consumed and the transaction request is cleared. In the event of insufficient tokens, transaction throttling is performed.
In some embodiments of the present application, the service data processing request further carries a request parameter, and the method may further include: the method comprises the steps of storing request parameters of a service data processing request and processing results corresponding to the service data processing request in a preset cache in an associated mode; after receiving the target service data processing request, inquiring whether a request parameter in the target service data processing request exists in a preset cache; and under the condition that the request parameters in the target business data processing request are inquired, determining the processing result corresponding to the searched request parameters as the processing result corresponding to the target business data processing request.
Specifically, after each transaction processing module executes one service data processing, the request parameter of the corresponding service data processing request and the processing result corresponding to the request may be stored in a preset cache in an associated manner. After receiving a new target business data processing request, the request parameters of the target business data processing request can be acquired. And then, inquiring whether the request parameters of the target business data processing request exist in a preset buffer memory. And under the condition that the request parameters of the target business data processing request are inquired, determining the processing result corresponding to the searched request parameters as the processing result corresponding to the target business data processing request, and returning the processing result to the client.
Referring to fig. 3, a flowchart of a transaction result cache provided in an embodiment of the present application is shown. The transaction request parameters may be obtained before a new transaction request enters the transaction throttling module. The corresponding value may be queried in the cache using the transaction request parameter as a key. If the corresponding value can be queried, the user is directly returned with the value, and the transaction is directly closed-loop. If the corresponding value of the transaction parameter cannot be queried in the cache, continuing to enter the current limiting module, and performing subsequent normal operation. After the transaction processing module processes a transaction, the transaction processing module can store the parameters of the transaction as keys and the processing results as values into the caching module, and set a shorter caching expiration time. By utilizing the caching module, the transaction request with the same parameters can be ensured to be actually processed once within the expiration time of the caching, and the rest transaction can be directly obtained from the caching and returned, so that the load of the transaction processing module in a transaction scene of a large number of cyclic calls in a short time is further reduced.
Based on the same inventive concept, the embodiment of the application also provides a service data processing device, which is applied to a service data processing system, as described in the following embodiment. Since the principle of the service data processing device for solving the problem is similar to that of the service data processing method, the implementation of the service data processing device can refer to the implementation of the service data processing method, and the repetition is omitted. As used below, the term "unit" or "module" may be a combination of software and/or hardware that implements the intended function. While the means described in the following embodiments are preferably implemented in software, implementation in hardware, or a combination of software and hardware, is also possible and contemplated. Fig. 4 is a block diagram of a service data processing apparatus according to an embodiment of the present application, as shown in fig. 4, including: the structure of the request receiving module 401, the priority determining module 402, the request depositing module 403, and the data processing module 404 will be described below.
The request receiving module 401 is configured to receive a service data processing request, where the service data processing request carries a service type identifier.
The priority determining module 402 is configured to determine a priority of the service data processing request according to the service type identifier.
The request depositing module 403 is configured to deposit the service data processing request into a corresponding transaction queue based on the priority of the service data processing request, where the transaction queue includes a first transaction queue and a second transaction queue, the service data processing request with high priority is deposited into the first transaction queue, and the service data processing request with low priority is deposited into the second transaction queue.
The data processing module 404 is configured to read, by using the first transaction processing module, a service data processing request from the first transaction queue and perform corresponding service data processing; reading a business data processing request from a second transaction queue by utilizing a second transaction processing module and executing corresponding business data processing; the first transaction processing module and the second transaction processing module are different transaction processing modules in the business data processing system.
In some embodiments of the present application, transaction processing modules in a business data processing system are deployed in a containerized manner; correspondingly, the method further comprises the steps of: collecting transaction processing information generated when the first transaction processing module and the second transaction processing module execute business data processing, and the quantity of business data processing requests to be processed in the first transaction queue and the second transaction queue; calculating a load factor of the service data processing system by utilizing the transaction processing information and the number of the service data processing requests to be processed, wherein the load factor is used for representing the load degree of the service data processing system; and carrying out elastic transverse expansion and contraction on the service data processing system based on the load coefficient of the service data processing system and the number of container instances of the service data processing system.
In some embodiments of the present application, the request deposit module may be specifically configured to: determining whether the number of tokens remaining in the token container is less than a preset number; under the condition that the number of the tokens remained in the token container is not less than the preset number, acquiring the preset number of tokens from the token container, and storing the business data processing request into a corresponding transaction queue based on the priority of the business data processing request; and under the condition that the number of the tokens remained in the token container is less than the preset number, carrying out flow limiting processing on the business data processing request.
In some embodiments of the present application, the apparatus may further include a token replenishment module, which may be configured to: acquiring a load coefficient of a service data processing system; calculating the token replenishment rate in a token container of the business data processing system according to the load coefficient; tokens are added to the token container based on the token replenishment rate.
In some embodiments of the present application, after calculating the token replenishment rate in the token container of the business data processing system according to the load factor, further comprising: determining whether the token replenishment rate is less than a preset rate; and under the condition that the token supplement rate is smaller than the preset rate, transversely expanding the service data processing system to reduce the load coefficient of the service data processing system.
In some embodiments of the present application, calculating a token replenishment rate in a token container of a business data processing system from a load factor comprises: the token replenishment rate in the token container is calculated according to the following formula:
R＝i/L；
wherein R is the token replenishment rate, i is the preset replenishment rate, and L is the load factor.
In some embodiments of the present application, the service data processing request further carries a request parameter, and the apparatus further includes: the result caching module is specifically used for: the method comprises the steps of storing request parameters of a service data processing request and processing results corresponding to the service data processing request in a preset cache in an associated mode; after receiving the target service data processing request, inquiring whether a request parameter in the target service data processing request exists in a preset cache; and under the condition that the request parameters in the target business data processing request are inquired, determining the processing result corresponding to the searched request parameters as the processing result corresponding to the target business data processing request.
From the above description, it can be seen that the following technical effects are achieved in the embodiments of the present application: the priority of the business data processing request is determined according to the business type, the requests with different priorities are stored in different transaction queues, the different transaction processing modules are utilized to read the requests from the corresponding transaction queues and execute business data processing, for the classified processing of the high-priority and low-priority transactions, each transaction processing module only processes the transactions with high priority or low priority and does not process the transactions with different priority levels at the same time, so that the transactions with low priority can be ensured not to block the transaction processing with high priority, the transaction diversion is realized, the transaction processing blocking is relieved, and the transaction processing efficiency can be effectively improved.
The embodiment of the application further provides a computer device, and in particular, referring to a schematic diagram of a composition structure of the computer device based on the service data processing method provided by the embodiment of the application shown in fig. 5, the computer device may specifically include an input device 51, a processor 52, and a memory 53. Wherein the memory 53 is configured to store processor-executable instructions. The processor 52, when executing the instructions, implements the steps of the traffic data processing method described in any of the embodiments above.
In this embodiment, the input device may specifically be one of the main apparatuses for exchanging information between the user and the computer system. The input device may include a keyboard, mouse, camera, scanner, light pen, handwriting input board, voice input device, etc.; the input device is used to input the original service data and the program for processing these numbers into the computer. The input device can also acquire and receive service data transmitted by other modules, units and devices. The processor may be implemented in any suitable manner. For example, the processor may take the form of, for example, a microprocessor or processor, and a computer-readable medium storing computer-readable program code (e.g., software or firmware) executable by the (micro) processor, logic gates, switches, an application specific integrated circuit (Application Specific Integrated Circuit, ASIC), a programmable logic controller, and an embedded microcontroller, among others. The memory may in particular be a memory device for storing information in modern information technology. The memory may comprise a plurality of layers, and in a digital system, the memory may be any memory as long as the memory can store binary service data; in an integrated circuit, a circuit with a memory function without a physical form is also called a memory, such as a RAM, a FIFO, etc.; in the system, the storage device in physical form is also called a memory, such as a memory bank, a TF card, and the like.
In this embodiment, the specific functions and effects of the computer device may be explained in comparison with other embodiments, and will not be described herein.
The present application further provides a computer storage medium based on a service data processing method, where the computer storage medium stores computer program instructions, and when the computer program instructions are executed, the steps of the service data processing method in any embodiment are implemented.
In the present embodiment, the storage medium includes, but is not limited to, a Random access Memory (Random AccessMemory, RAM), a Read-Only Memory (ROM), a Cache (Cache), a hard disk (HDD), or a Memory Card (Memory Card). The memory may be used to store computer program instructions. The network communication unit may be an interface for performing network connection communication, which is set in accordance with a standard prescribed by a communication protocol.
In this embodiment, the functions and effects of the program instructions stored in the computer storage medium may be explained in comparison with other embodiments, and are not described herein.
It will be apparent to those skilled in the art that the modules or steps of the embodiments of the application described above may be implemented in a general purpose computing device, they may be concentrated on a single computing device, or distributed across a network of computing devices, they may alternatively be implemented in program code executable by computing devices, so that they may be stored in a storage device for execution by computing devices, and in some cases, the steps shown or described may be performed in a different order than what is shown or described, or they may be separately fabricated into individual integrated circuit modules, or multiple modules or steps of them may be fabricated into a single integrated circuit module. Thus, embodiments of the present application are not limited to any specific combination of hardware and software.
It is to be understood that the above description is intended to be illustrative, and not restrictive. Many embodiments and many applications other than the examples provided will be apparent to those of skill in the art upon reading the above description. The scope of the application should, therefore, be determined not with reference to the above description, but instead should be determined with reference to the appended claims, along with the full scope of equivalents to which such claims are entitled.
The foregoing description is only a preferred embodiment of the present application and is not intended to limit the present application, and various modifications and variations may be made to the embodiment of the present application by those skilled in the art. Any modification, equivalent replacement, improvement, etc. made within the spirit and principles of the present application should be included in the protection scope of the present application.
Claims (8)
1. A method for processing service data, applied to a service data processing system, the method comprising:
receiving a service data processing request, wherein the service data processing request carries a service type identifier;
determining the priority of the service data processing request according to the service type identifier;
storing the business data processing requests into corresponding transaction queues based on the priorities of the business data processing requests, wherein the transaction queues comprise a first transaction queue and a second transaction queue, the business data processing requests with high priorities are stored into the first transaction queue, and the business data processing requests with low priorities are stored into the second transaction queue;
reading a business data processing request from the first transaction queue by using a first transaction processing module and executing corresponding business data processing; reading a business data processing request from the second transaction queue by using a second transaction processing module and executing corresponding business data processing; wherein the first transaction processing module and the second transaction processing module are different transaction processing modules in the business data processing system;
wherein the method further comprises:
acquiring a load coefficient of the service data processing system;
calculating a token replenishment rate in a token container of the business data processing system according to the load coefficient; the token replenishment rate is inversely related to the load factor;
adding tokens to the token container based on the token replenishment rate;
after calculating the token replenishment rate in a token container of the business data processing system according to the load factor, further comprising:
determining whether the token replenishment rate is less than a preset rate;
and under the condition that the token supplement rate is smaller than the preset rate, transversely expanding the service data processing system to reduce the load coefficient of the service data processing system.
2. The method of claim 1, wherein transaction processing modules in the business data processing system are deployed in a containerized manner; correspondingly, the method further comprises the steps of:
collecting transaction processing information generated when the first transaction processing module and the second transaction processing module execute service data processing and the quantity of service data processing requests to be processed in the first transaction queue and the second transaction queue;
calculating a load factor of the service data processing system by using the transaction processing information and the number of the service data processing requests to be processed, wherein the load factor is used for representing the load degree of the service data processing system;
and carrying out elastic transverse expansion and contraction on the service data processing system based on the load coefficient of the service data processing system and the number of container instances of the service data processing system.
3. The method of claim 1, wherein depositing the business data processing request into a corresponding transaction queue based on the priority of the business data processing request comprises:
determining whether the number of tokens remaining in the token container is less than a preset number;
under the condition that the number of the tokens remained in the token container is not less than the preset number, acquiring the preset number of tokens from the token container, and storing the business data processing request into a corresponding transaction queue based on the priority of the business data processing request;
and under the condition that the number of the tokens remained in the token container is less than the preset number, carrying out flow limiting processing on the business data processing request.
4. The method of claim 1, wherein calculating a token replenishment rate in a token container of the business data processing system from the load factor comprises:
calculating a token replenishment rate in the token container according to the following formula:
R＝i/L；
wherein R is the token replenishment rate, i is the preset replenishment rate, and L is the load factor.
5. The method of claim 1, wherein the service data processing request further carries a request parameter, and wherein the method further comprises:
the request parameters of the service data processing request and the processing results corresponding to the service data processing request are stored in a preset cache in an associated mode;
after receiving a target service data processing request, inquiring whether a request parameter in the target service data processing request exists in the preset cache;
and under the condition that the request parameters in the target service data processing request are inquired, determining the processing result corresponding to the searched request parameters as the processing result corresponding to the target service data processing request.
6. A business data processing apparatus for use in a business data processing system, said apparatus comprising:
the request receiving module is used for receiving a service data processing request, wherein the service data processing request carries a service type identifier;
the priority determining module is used for determining the priority of the service data processing request according to the service type identifier;
a request storage module, configured to store the service data processing request into a corresponding transaction queue based on a priority of the service data processing request, where the transaction queue includes a first transaction queue into which a service data processing request with a high priority is stored and a second transaction queue into which a service data processing request with a low priority is stored;
the data processing module is used for reading the business data processing request from the first transaction queue by utilizing the first transaction processing module and executing corresponding business data processing; reading a business data processing request from the second transaction queue by using a second transaction processing module and executing corresponding business data processing; wherein the first transaction processing module and the second transaction processing module are different transaction processing modules in the business data processing system;
the device further comprises a token supplementing module, wherein the token supplementing module is specifically used for:
acquiring a load coefficient of the service data processing system;
calculating a token replenishment rate in a token container of the business data processing system according to the load coefficient; the token replenishment rate is inversely related to the load factor;
adding tokens to the token container based on the token replenishment rate;
the token replenishing module is also specifically used for: after calculating a token replenishment rate in a token container of the business data processing system according to the load factor, determining whether the token replenishment rate is less than a preset rate;
and under the condition that the token supplement rate is smaller than the preset rate, transversely expanding the service data processing system to reduce the load coefficient of the service data processing system.
7. A computer device comprising a processor and a memory for storing processor-executable instructions which when executed by the processor implement the steps of the method of any one of claims 1 to 5.
8. A computer readable storage medium having stored thereon computer instructions, which when executed, implement the steps of the method of any of claims 1 to 5.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202110352525.1A CN112866136B (en) | 2021-03-31 | 2021-03-31 | Service data processing method and device |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202110352525.1A CN112866136B (en) | 2021-03-31 | 2021-03-31 | Service data processing method and device |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN112866136A CN112866136A (en) | 2021-05-28 |
| CN112866136B true CN112866136B (en) | 2023-05-02 |
Family
ID=75992029
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202110352525.1A Active CN112866136B (en) | 2021-03-31 | 2021-03-31 | Service data processing method and device |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN112866136B (en) |
Families Citing this family (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN114553937A (en) * | 2022-01-12 | 2022-05-27 | 北京达佳互联信息技术有限公司 | Data acquisition method, device, equipment and storage medium |
| CN114721823A (en) * | 2022-03-29 | 2022-07-08 | 中信百信银行股份有限公司 | Information processing method, device, electronic equipment and storage medium |
| CN114862500A (en) * | 2022-04-14 | 2022-08-05 | 中国联合网络通信集团有限公司 | Order processing method and device, electronic equipment and storage medium |
| CN115278786B (en) * | 2022-06-08 | 2024-09-17 | 深圳华云信息系统科技股份有限公司 | Service access method, device, electronic equipment and storage medium |
| CN115190078B (en) * | 2022-06-28 | 2024-06-14 | 上海砾阳软件有限公司 | Access flow control method, device, equipment and storage medium |
| CN115225577B (en) * | 2022-09-20 | 2022-12-27 | 深圳市明源云科技有限公司 | Data processing control method and device, electronic equipment and readable storage medium |
Citations (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN108848037A (en) * | 2018-05-31 | 2018-11-20 | 平安医疗科技有限公司 | Service request processing method, device, computer equipment and storage medium |
| CN109729013A (en) * | 2017-10-30 | 2019-05-07 | 深圳市中兴微电子技术有限公司 | The method, apparatus and computer readable storage medium of token are added in a kind of traffic shaping |
| CN109802895A (en) * | 2017-11-16 | 2019-05-24 | 阿里巴巴集团控股有限公司 | Data processing system, method and token management method |
Family Cites Families (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US8990534B2 (en) * | 2012-05-31 | 2015-03-24 | Apple Inc. | Adaptive resource management of a data processing system |
| CN113282659B (en) * | 2017-03-28 | 2024-08-20 | 创新先进技术有限公司 | Data processing method and device based on block chain |
| CN109412972A (en) * | 2017-08-17 | 2019-03-01 | 阿里巴巴集团控股有限公司 | A kind of data reordering method, device and node server |
| CN110674064B (en) * | 2019-09-05 | 2021-06-29 | 苏州浪潮智能科技有限公司 | Data transmission method, device, equipment and computer readable storage medium |
| CN111930486B (en) * | 2020-07-30 | 2023-11-17 | 中国工商银行股份有限公司 | Task selection data processing method, device, equipment and storage medium |
| CN112099975B (en) * | 2020-09-25 | 2024-03-26 | Oppo广东移动通信有限公司 | Message processing method and system and storage medium |
-
2021
- 2021-03-31 CN CN202110352525.1A patent/CN112866136B/en active Active
Patent Citations (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN109729013A (en) * | 2017-10-30 | 2019-05-07 | 深圳市中兴微电子技术有限公司 | The method, apparatus and computer readable storage medium of token are added in a kind of traffic shaping |
| CN109802895A (en) * | 2017-11-16 | 2019-05-24 | 阿里巴巴集团控股有限公司 | Data processing system, method and token management method |
| CN108848037A (en) * | 2018-05-31 | 2018-11-20 | 平安医疗科技有限公司 | Service request processing method, device, computer equipment and storage medium |
Also Published As
| Publication number | Publication date |
|---|---|
| CN112866136A (en) | 2021-05-28 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN112866136B (en) | Service data processing method and device | |
| CN110489447B (en) | Data query method and device, computer equipment and storage medium | |
| CN107592345B (en) | Transaction current limiting device, method and transaction system | |
| US10884667B2 (en) | Storage controller and IO request processing method | |
| US9003084B2 (en) | Sorting | |
| Xie et al. | Pandas: robust locality-aware scheduling with stochastic delay optimality | |
| CN111563093A (en) | Detection and avoidance system and method for union block chain conflict transaction | |
| US20220050712A1 (en) | Distributed streaming system supporting real-time sliding windows | |
| CN110413978B (en) | Data paging export method, device, computer equipment and storage medium | |
| AU2004285241C1 (en) | Tracking space usage in a database | |
| CN110597879B (en) | Method and device for processing time series data | |
| CN115774605A (en) | Kubernetes prediction type elastic expansion method and system | |
| US10747472B2 (en) | Method, device, and computer program product for operation control in storage system | |
| US10728186B2 (en) | Preventing reader starvation during order preserving data stream consumption | |
| CN112073327B (en) | Anti-congestion software distribution method, device and storage medium | |
| CN114553959A (en) | Situation awareness-based cloud native service grid configuration on-demand issuing method and application | |
| US9063773B2 (en) | Automatic parallelism tuning for apply processes | |
| CN114157717B (en) | System and method for dynamic current limiting of micro-service | |
| US10540341B1 (en) | System and method for dedupe aware storage quality of service | |
| CN115499513A (en) | Data request processing method and device, computer equipment and storage medium | |
| CN115904246A (en) | Data reading method and device based on multi-path DDR memory | |
| CN109445934B (en) | Query request distribution method and system | |
| CN115442432B (en) | Control method, device, equipment and storage medium | |
| CN115599838B (en) | Data processing method, device, equipment and storage medium based on artificial intelligence | |
| CN116756058A (en) | Memory cache processing method, device, terminal and storage medium |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |