CN114969256A - Data query method and unit - Google Patents
Data query method and unit Download PDFInfo
- Publication number
- CN114969256A CN114969256A CN202210579957.0A CN202210579957A CN114969256A CN 114969256 A CN114969256 A CN 114969256A CN 202210579957 A CN202210579957 A CN 202210579957A CN 114969256 A CN114969256 A CN 114969256A
- Authority
- CN
- China
- Prior art keywords
- query
- task
- database
- query task
- new
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 238000000034 method Methods 0.000 title claims abstract description 60
- 238000012545 processing Methods 0.000 claims abstract description 35
- 238000012544 monitoring process Methods 0.000 claims abstract description 8
- 239000003999 initiator Substances 0.000 claims description 24
- 230000008569 process Effects 0.000 description 13
- 238000010586 diagram Methods 0.000 description 12
- 238000005516 engineering process Methods 0.000 description 5
- 230000006870 function Effects 0.000 description 5
- 238000004590 computer program Methods 0.000 description 3
- 230000004044 response Effects 0.000 description 3
- 230000005540 biological transmission Effects 0.000 description 2
- 238000011161 development Methods 0.000 description 2
- OKTJSMMVPCPJKN-UHFFFAOYSA-N Carbon Chemical compound [C] OKTJSMMVPCPJKN-UHFFFAOYSA-N 0.000 description 1
- 230000001413 cellular effect Effects 0.000 description 1
- 230000008859 change Effects 0.000 description 1
- 229910021389 graphene Inorganic materials 0.000 description 1
- 230000006872 improvement Effects 0.000 description 1
- 230000004048 modification Effects 0.000 description 1
- 238000012986 modification Methods 0.000 description 1
- 230000003287 optical effect Effects 0.000 description 1
- 230000003068 static effect Effects 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/30—Information retrieval; Database structures therefor; File system structures therefor of unstructured textual data
- G06F16/33—Querying
- G06F16/3331—Query processing
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/30—Information retrieval; Database structures therefor; File system structures therefor of unstructured textual data
- G06F16/33—Querying
- G06F16/332—Query formulation
- G06F16/3325—Reformulation based on results of preceding query
Landscapes
- Engineering & Computer Science (AREA)
- Physics & Mathematics (AREA)
- Theoretical Computer Science (AREA)
- Mathematical Physics (AREA)
- Computational Linguistics (AREA)
- Data Mining & Analysis (AREA)
- Databases & Information Systems (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
The application discloses a data query method and unit. The method comprises the following steps: monitoring a task queue of a message system, and acquiring a query task from the task queue, wherein the query task comprises a query condition; performing data query in a database based on the query task, and acquiring a cursor identifier generated by the database after the query task is executed; after the cursor identification is obtained, a new query task corresponding to the cursor identification is generated based on the cursor identification and the query condition, and the new query task is sent to a message system so that the message system can add the new query task to the task queue. According to the method and the system, data query can be cooperatively realized by combining a plurality of database query clients, and the data query does not need to wait for the queried data to be processed, so that the decoupling of the data query and the data processing is realized, and the efficiency of the data query is greatly improved.
Description
Technical Field
The present application relates to the field of data processing technologies, and in particular, to a data query method and unit.
Background
With the rapid development of internet technology, more and more data are deposited and stored in a database, and how to rapidly query the data in the database becomes a technical problem to be solved urgently.
Disclosure of Invention
In view of the above, the present application provides a data query method and unit.
Specifically, the method is realized through the following technical scheme:
a data query method is applied to a database query client, and comprises the following steps:
monitoring a task queue of a message system, and acquiring a query task from the task queue, wherein the query task comprises a query condition;
performing data query in a database based on the query task, and acquiring a cursor identifier generated by the database after the query task is executed;
after the cursor identification is obtained, a new query task corresponding to the cursor identification is generated based on the cursor identification and the query condition, and the new query task is sent to a message system so that the message system can add the new query task to the task queue.
Optionally, the querying the data in the database based on the query task includes:
under the condition that the cursor identification corresponding to the query task is not included in the query task, performing data query in a database based on the query condition;
and under the condition that the query task also comprises a cursor identifier corresponding to the query task, carrying out data query in a database based on the cursor identifier and the query condition.
Optionally, the method further includes:
judging whether the query task carries a current limiting label or not;
and under the condition that the query task carries a current limiting label, limiting the flow of the new query task based on a preset current limiting measure.
Optionally, the flow limiting the new query task based on the preset flow limiting measure includes:
starting timing after the new query task is generated;
and after the preset delay time is reached, sending the new query task to a message system.
Optionally, the flow limiting the new query task based on the preset flow limiting measure includes:
starting timing after the cursor identification is obtained;
and after the preset delay time is reached, executing a new query task which is generated based on the cursor identification and the query condition and corresponds to the cursor identification, and sending the new query task to a message system.
Optionally, the method further includes:
processing the data queried from the database;
and after the data processing is finished, updating the processed data to the database in batch.
A data query method is applied to a message system and comprises the following steps:
responding to a query task sent by a query initiator, adding the query task into a task queue, wherein the query task comprises a query condition, so that a database query client performs data query in a database after acquiring the query task from the task queue, acquires a cursor identifier generated by the database after the query task is executed, and generates a first new query task corresponding to the cursor identifier based on the cursor identifier and the query condition;
responding to a first new query task sent by a database query client, adding the first new query task into the task queue, so that the database query client can perform data query in the database after acquiring the first new query task from the task queue, and executing generation of a second new query task based on a cursor identifier generated by the database after the first query task is executed and the query condition.
Optionally, the adding the first new query task to the task queue includes:
adding the first new query task to a delay queue;
and after the preset delay time is reached, adding a first new query task in the delay queue to the task queue.
Optionally, the adding the first new query task to the delay queue includes:
judging whether the query task carries a current limiting label or not;
and under the condition that the current limiting label is carried in the query task, adding the first new query task into a delay queue.
A data query unit is applied to a database query client, and comprises:
the task monitoring module monitors a task queue of the message system and acquires a query task from the task queue, wherein the query task comprises a query condition;
the data query module is used for performing data query in a database based on the query task and acquiring a cursor identifier generated by the database after the query task is executed;
and the task generation module is used for generating a new query task corresponding to the cursor identification based on the cursor identification and the query condition after the cursor identification is obtained, and sending the new query task to a message system so that the message system can add the new query task to the task queue.
A data query unit applied to a message system comprises:
the first adding module is used for responding to a query task sent by a query initiator, adding the query task into a task queue, wherein the query task comprises a query condition, so that a database query client side can perform data query in a database after acquiring the query task from the task queue, acquire a cursor identifier generated by the database after the query task is executed, and generate a first new query task corresponding to the cursor identifier based on the cursor identifier and the query condition;
and the second adding module is used for responding to a first new query task sent by the database query client, adding the first new query task into the task queue so that the database query client can perform data query in the database after acquiring the first new query task from the task queue, and executing generation of a second new query task based on the cursor identifier generated by the database after the first query task is executed and the query condition.
An electronic device, comprising:
a processor;
a memory for storing processor-executable instructions;
wherein, the processor executes the executable instruction to realize the data query method.
A computer readable storage medium having stored thereon computer instructions, wherein the instructions, when executed by a processor, implement the steps of the aforementioned data query method.
By adopting the embodiment, the database query client can monitor the task queue of the message system, acquire the query task from the task queue, execute the data query, and acquire the cursor identifier generated by the database after the query task is executed. On the basis, the database query client can also generate a new query task corresponding to the cursor identification based on the cursor identification and the query condition, then the new query task is sent to the message system, and the message system further adds the new query task to the task queue so that other database query clients can acquire and execute data query.
By adopting the technical scheme provided by the application, the database query client can generate a new query task and send the new query task to the message system, so that other database query clients can acquire the new query task from the task queue of the message system and execute the new query task, the data query is cooperatively realized by combining a plurality of database query clients, and the data query does not need to wait for the queried data to be processed completely, so that the decoupling of the data query and the data processing is realized, and the efficiency of the data query is greatly improved.
Drawings
Fig. 1 is a schematic diagram of an architecture of a data query system in the related art.
Fig. 2 is a flowchart illustrating a data query method according to an exemplary embodiment of the present application.
Fig. 3 is a flowchart illustrating another data query method according to an exemplary embodiment of the present application.
FIG. 4 is a block diagram illustrating an architecture of a data query system according to an exemplary embodiment of the present application.
Fig. 5 is a diagram illustrating a new query task throttling method according to an exemplary embodiment of the present application.
FIG. 6 is a diagram illustrating another method for throttling new query tasks according to an exemplary embodiment of the present application.
Fig. 7 is a hardware structure diagram of an electronic device in which a data query unit is located according to an exemplary embodiment of the present application.
FIG. 8 is a block diagram of a data query unit, shown in an exemplary embodiment of the present application.
FIG. 9 is a block diagram of another data query unit shown in an exemplary embodiment of the present application.
Detailed Description
Reference will now be made in detail to the exemplary embodiments, examples of which are illustrated in the accompanying drawings. When the following description refers to the accompanying drawings, like numbers in different drawings represent the same or similar elements unless otherwise indicated. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the present application. Rather, they are merely examples of units and methods consistent with aspects of the present application, as detailed in the appended claims.
The terminology used herein is for the purpose of describing particular embodiments only and is not intended to be limiting of the application. As used in this specification and the appended claims, the singular forms "a", "an", and "the" are intended to include the plural forms as well, unless the context clearly indicates otherwise. It should also be understood that the term "and/or" as used herein refers to and encompasses any and all possible combinations of one or more of the associated listed items.
It is to be understood that although the terms first, second, third, etc. may be used herein to describe various information, such information should not be limited to these terms. These terms are only used to distinguish one type of information from another. For example, first information may also be referred to as second information, and similarly, second information may also be referred to as first information, without departing from the scope of the present application. The word "if" as used herein may be interpreted as "at … …" or "when … …" or "in response to a determination", depending on the context.
With the rapid development of internet technology, more and more data are deposited and stored in a database, and how to rapidly query the data in the database becomes a technical problem to be urgently solved.
Fig. 1 is a schematic diagram of an architecture of a data query system in the related art.
Referring to fig. 1, the data query system may include a database query client and a database.
The database query client is usually provided by a database, can receive a data query task sent from the outside, and can query corresponding data in the database based on query conditions in the query task. For example, the database query client may receive a data query task from a user or from other business systems, and the data query task may include query conditions such as keywords, time, and the like. The database query client can query corresponding data in the database based on the query conditions, and the database query client can be integrated in the database or can be independently arranged. One database can be configured with one database query client, and can also be configured with a plurality of database query clients.
Taking ES (elastic search, a search server providing a distributed full-text search engine) database as an example, in order to avoid the Deep paging problem (Deep paging) generated in the query process, the cursor query (Scroll query) is generated. The database query client can query data from the database in batches according to a certain step length, a cursor identifier (Scroll ID) required by the next query can be returned after the query is finished every time, and the cursor identifier returned in the last query is carried during the next query so as to continue data query.
In the above process, the database query client often needs to process the queried data, for example, desensitize the data, mark the data, and the like. The data processing and query are serial, that is, the database query client performs data processing after querying a batch of data, and performs next query after the batch of data is processed, thereby resulting in low data query efficiency. Moreover, a database is usually configured with a plurality of database query clients, and one query task can be executed by only one database query client, which fails to fully utilize the processing resources of other database query clients.
The application provides a data query scheme, which can decouple data query and data processing, and collaborate with a plurality of database query clients to perform data query, so that the efficiency of data query is improved.
Fig. 2 is a flowchart illustrating a data query method according to an exemplary embodiment of the present application.
Referring to fig. 2, the data query method can be applied to a database query client, and includes the following steps:
step 202, monitoring a task queue of a message system, and obtaining a query task from the task queue, wherein the query task comprises a query condition.
In the present application, the message system is an "intermediate system" and is located between a query task initiator (e.g., a user, a business system, etc.) and a database query client during a query task flow process. The physical carrier of the message system may be an independent server cluster, or may integrate its relay function into a service system or a database, which is not particularly limited in this application.
In the application, a task queue for storing query tasks is arranged in a message system, and the received query tasks are added to the task queue for a database query client to obtain. At least one of the query tasks includes a query condition, such as a keyword, a time, and the like, which can be specified by a query task initiator.
In one aspect, a message system may receive a query task initiated by a query task initiator and then add the query task to a task queue.
On the other hand, the message system can also receive the query task sent by the database query client and add the query task to the task queue. After the database query client performs data query, the query task sent by the database query client is generated based on a new cursor identifier and a query condition generated by the database after the query task is executed, which will be described in detail in the following embodiments.
In the application, each database query client can monitor the task queue of the message system, acquire the query task when the query task exists in the task queue, and perform data query in the database based on the query task.
And 204, performing data query in a database based on the query task, and acquiring a cursor identifier generated by the database after the query task is executed.
Based on the foregoing step 202, after obtaining the query task, the database client may perform data query in the database.
In an example of the present application, the query task includes a cursor identifier that is needed to be used in the query of this time and corresponds to the query task, and the database client can perform data query in the database based on the cursor identifier and the query condition. For example, the data query is performed from the position corresponding to the cursor identifier.
When the obtained query task includes the cursor identifier corresponding to the query task, it can be stated that the query task is not the original query task initiated by the query task initiator, and the original query task has been subjected to data query by the database query client, so that the cursor identifier is included.
In another example of the present application, the query task does not include a cursor identifier corresponding to the query task, which may indicate that the query task is an original query task initiated by a task initiator, and the database client may perform data query in the database based on a query condition in the query task.
In the present application, after the data query is completed, that is, after the data search task is completed, if there is data that has not been queried yet in the database, the database may generate a cursor identifier. If the data in the database is completely queried after the query is finished, the completion of the execution of the query task can be determined, and the database does not need to generate a cursor identifier.
In the present application, the database query client may obtain the cursor identifier generated by the database after the query task is completed, and for convenience of distinguishing, the cursor identifier generated by the database after the query is completed is subsequently referred to as a new cursor identifier.
Step 206, after the cursor identifier is obtained, generating a new query task corresponding to the cursor identifier based on the cursor identifier and the query condition, and sending the new query task to a message system, so that the message system adds the new query task to the task queue.
In the method, after the database query client acquires the new cursor identifier generated by the database after the query task is executed, on one hand, a new query task corresponding to the new cursor identifier can be generated based on the new cursor identifier and the query condition, and then the new query task is sent to the message system. The message system can further add the new query task to the task queue for other database query clients to obtain and execute. On the other hand, the database query client can process the queried data, for example, desensitize processing and marking processing are performed on the data.
In the application, after acquiring the new query task from the task queue of the message system, other database query clients can perform data query in the database based on the new cursor identifier and the query condition in the new query task, so that it is unnecessary to wait for the data query corresponding to the new query task to finish the processing of the data queried last time, the decoupling of the data query and the data processing is realized, and the efficiency of the data query is improved.
As can be seen from the above description, the database query client according to the present application may monitor the task queue of the message system, obtain the query task from the task queue, perform data query, and obtain the cursor identifier generated by the database after the query task is executed. On the basis, the database query client can also generate a new query task corresponding to the cursor identification based on the cursor identification and the query condition, then the new query task is sent to the message system, and the message system further adds the new query task to the task queue so that other database query clients can acquire and execute data query.
By adopting the technical scheme provided by the application, the database query client can generate a new query task and send the new query task to the message system so that other database query clients can acquire the new query task from the task queue of the message system and execute the new query task, the data query is cooperatively realized by combining a plurality of database query clients, and the data query does not need to wait for the completion of the processing of the queried data, so that the decoupling of the data query and the data processing is realized, and the efficiency of the data query is greatly improved.
Fig. 3 is a flowchart illustrating another data query method according to an exemplary embodiment of the present application.
Referring to fig. 3, the data query method can be applied to a message system, which is an "intermediate system" located between a query task initiator (e.g., a user, a business system, etc.) and a database query client during a query task flow. The physical carrier of the message system may be an independent server cluster, or may integrate its relay function into a service system or a database, which is not particularly limited in this application. The data query method can comprise the following steps:
step 302, responding to a query task sent by a query initiator, adding the query task into a task queue, wherein the query task comprises a query condition, so that a database query client performs data query in a database after acquiring the query task from the task queue, acquires a cursor identifier generated by the database after the query task is executed, and generates a first new query task corresponding to the cursor identifier based on the cursor identifier and the query condition.
In the application, a task queue for storing query tasks is arranged in a message system, and the received query tasks are added to the task queue for being acquired by a database query client. At least one of the query tasks includes a query condition, such as a keyword, a time, and the like, which can be specified by a query task initiator.
Step 304, responding to a first new query task sent by the database query client, adding the first new query task to the task queue, so that the database query client performs data query in the database after acquiring the first new query task from the task queue, and generates a second new query task based on the cursor identifier generated by the database after the first query task is executed and the query condition.
In the present application, the query task received by the message system generally has the following two cases.
In one aspect, a message system may receive a query task initiated by a query task initiator and then add the query task to a task queue. The query task initiated by the query task initiator is an original query task, wherein the original query task does not carry a cursor identifier. When the database query client executes the query task, the database query client can perform data query in the database based on the query condition.
On the other hand, the message system can also receive the query task sent by the database query client, and then the query task is also added to the task queue. The query task generated by the database client is called a new query task. The new query task sent by the database query client is generally generated based on a new cursor identifier and a query condition after the database query client finishes executing the last data query task and acquires the new cursor identifier generated by the database. The new query task sent by the database query client comprises the cursor identifier, and the database query client which acquires the new query task comprising the cursor identifier can perform data query in the database based on the cursor identifier and the query condition.
As can be seen from the above description, the database query client according to the present application may monitor the task queue of the message system, obtain the query task from the task queue, perform data query, and obtain the cursor identifier generated by the database after the query task is executed. On the basis, the database query client can also generate a new query task corresponding to the cursor identification based on the cursor identification and the query condition, then the new query task is sent to the message system, and the message system further adds the new query task to the task queue so that other database query clients can acquire and execute data query.
By adopting the technical scheme provided by the application, the database query client can generate a new query task and send the new query task to the message system so that other database query clients can acquire the new query task from the task queue of the message system and execute the new query task, the data query is cooperatively realized by combining a plurality of database query clients, and the data query does not need to wait for the completion of the processing of the queried data, so that the decoupling of the data query and the data processing is realized, and the efficiency of the data query is greatly improved.
The data query method of the present application is described below in conjunction with a system architecture.
FIG. 4 is a block diagram illustrating an architecture of a data query system according to an exemplary embodiment of the present application.
Referring to fig. 4, the data query system includes an initiator of a data query task, a messaging system, a plurality of database query clients, and a database.
The initiator of the data query task may be a user, or may be any business system having a database query authority, which is not limited in this application. The messaging system and database querying client may refer to the foregoing description.
In this embodiment, taking the database as the ES database as an example for description, the process of data query may include the following steps:
firstly, a query task initiator initiates an original query task to a message system.
In this embodiment, the query task initiator may encapsulate a query task (referred to as an original query task for convenience of distinction) as an object and then send the object to the message system, and the message system may add the original query task to a task queue for storing the query task after receiving the original query task.
The original query task comprises query conditions such as keywords, time and the like.
Each database query client of the database can monitor the task queue of the message system and can obtain the query task from the task queue.
Secondly, the database query client 1 acquires an original query task from a task queue of the message system.
Fourthly, the database query client 1 queries data in the database based on the original query task, and can obtain the queried original data and the cursor identifier 1 generated by the database after the original query task is executed.
And the database query client 1 generates a new query task 1 corresponding to the cursor identifier 1 based on the cursor identifier 1 and the query condition, sends the new query task 1 to a message system, and the message system stores the new query task 1 into the task queue.
In addition, the database query client 1 can also process the queried raw data.
For example, the raw data may be subjected to privacy removal processing, and sensitive information such as a user account number and balance may be hidden.
For another example, the original data may be subjected to a marking process to add a predetermined label.
In the present application, the processing on the original data may be performed by the database query client, or may be performed by the database query client invoking another system, for example, the database query client 1 invokes the marking system to mark the original data, and the like, which is not limited in this application.
Sixthly, the database query client 2 monitors the task queue of the message system and obtains the new query task 1.
In this embodiment, the new query task 1 includes a cursor identifier 1 and a query condition.
And (c) carrying out data query in the database by the database query client 2 based on the cursor identifier 1 and the query condition.
In this embodiment, the database query client 2 may also query a batch of original data from the database, and obtain the cursor identifier 2 generated by the database after the new query task 1 is executed.
In this embodiment, on one hand, the database query client 2 can generate a new query task 2 corresponding to the cursor identifier 2 based on the cursor identifier 2 and the query condition, and can send the new query task 2 to a message system (not shown), and the message system stores the new query task into the task queue.
In this embodiment, on the other hand, the database query client 2 may further process the queried original data, where the processing of the original data may refer to the processing of the original data by the database query client 1, and details of this embodiment are not repeated herein.
In this embodiment, the other database query clients may monitor the task queue, and obtain and execute the new query task 2. For example, the database query client 3 acquires the new query task 2 and executes the database query. Similarly, after the new query task 2 is executed, the database query client 3 can obtain a new cursor identifier 3 generated by the database, and on one hand, the new query task 3 corresponding to the cursor identifier 3 can be generated based on the new cursor identifier 3 and the query condition and sent to the message system, and the message system stores the new query task 3 into the task queue; on the other hand, the database query client 3 may also process the queried raw data.
Therefore, the new query task 1 executed by the database query client 2 does not need to wait for the database query client 1 to finish processing the original data acquired in the step (iv), and decoupling of data query and data processing is further realized. Moreover, the query task initiated by the initiator can be cooperatively used by a plurality of database query clients to realize data query, so that the processing resources of the database query clients can be fully utilized, and the efficiency of data query is greatly improved.
Optionally, by using the data query method provided by the present application, a plurality of database query clients may be coordinated to perform a data query task together, however, too fast data query may affect the stability of the database, and pressure is brought to the database. The method and the device also provide a flow limiting measure of the query task, and can limit the flow of a new query task based on a preset flow limiting measure, so that the query speed of the database is reduced, and the stability of the database is ensured.
In one example, the above-described current limiting measure may be implemented by a database query client. The predetermined current limiting measure may include suspending generation of the new query task or suspending transmission of the new query task to the messaging system after generation thereof.
Referring to fig. 5, limiting the flow of the new query task based on the preset flow limiting measure may include the following steps:
step 502, start timing after generating a new query task.
And step 504, after the preset delay time is reached, sending the new query task to a message system.
In this example, after the database query client queries the original data and acquires the new cursor identifier generated by the database, a new query task corresponding to the new cursor identifier is generated based on the new cursor identifier and the query condition, then timing is started, and after a preset delay time is reached, the generated new query task is sent to the message system.
The delay time may be a preset fixed time, for example, 1 second, and each database query client waits for 1 second before performing subsequent operations.
Still taking the data query shown in fig. 4 as an example, after the database query client 2 generates the new query task 1, the new query task 1 may be sent to the message system after waiting for a delay time of 1 second, and the message system adds the new query task to the task queue.
Referring to fig. 6, limiting the flow of the new query task based on the preset flow limiting measure may further include the following steps:
in step 602, timing is started after a cursor identifier generated by the database after the query task is executed is obtained.
And step 604, after a preset delay time is reached, generating a new query task corresponding to the cursor identifier based on the cursor identifier and the query condition, and sending the new query task to a message system.
In this example, the database query client starts timing after querying the original data and acquiring the new cursor identifier generated by the data, and generates a new query task according to the new cursor identifier and the query condition after reaching a preset delay time.
In this example, after the new query task is generated, the generated new query task can be directly sent to the message system without waiting. Or, after the new query task is generated, a wait for a certain preset time may be performed, and then the new query task is sent to the message system, which is not particularly limited in the present application.
In another example, the above-described current limiting measure may be implemented by a messaging system. After receiving the query task from the database query client, the message system may add the query task to the delay queue, and then add the query task in the delay queue to the task queue after a preset delay time is reached.
The delay time may also be a preset fixed time, which is not particularly limited in this application.
In this example, for the query task from the initiator, the message system does not need to adopt a current limiting measure, and the query task sent by the initiator can be added to the task queue after being received.
Still taking the data query shown in fig. 4 as an example, after generating the new query task 1, the database query client may send the new query task 1 to the messaging system, and after receiving the new query task 1, the messaging system adds the new query task 1 to the delay queue, and may add the new query task 1 to the task queue after waiting for the delay time.
In the present application, the aforementioned query task throttling measures described in the present application may be performed for all types of query tasks. The current limiting measures can also be implemented for some of the query tasks with larger query quantity. For example, an identification bit for carrying a current-limiting tag may be set in the query task, the current-limiting tag may be added in the query task for a service scenario with a large query volume, and the non-current-limiting tag may be added in the query task for a service scenario with a small query volume, so that after receiving the query task, the message queue or the database query client may determine whether the current-limiting tag is carried therein, and perform traffic limitation on a new query task under the condition of carrying the current-limiting tag.
It should be noted that, when the original query task initiated by the query initiator carries a current-limiting tag, the database query client may also add the current-limiting tag to a new query task when generating the new query task.
Of course, in other examples, the partial query task flow limitation and the partial query task non-flow limitation may be implemented in other manners, which are not limited in this application.
Therefore, the method and the device can limit the flow of the query task generated by the database query client, further reduce the query speed of the database and ensure the stability of the database.
Optionally, in another example of the present application, after the database query client finishes processing the queried data, the database query client may also update the processed data into the database in batch.
Still taking the data query shown in fig. 4 as an example, assuming that the database query client 2 performs marking processing on the obtained original data, after marking of the batch of data is completed, the database query client 2 may update the batch of marked data to the database in batch, and compared with updating data one by one, the efficiency of data updating may be greatly improved.
In this embodiment, the update operation of the processed data may be performed by the database query client, or may be performed by the database query client invoking other processing resources, for example, the database query client 2 may invoke a processor dedicated to update the database to perform the batch update, which is not limited in this application.
Corresponding to the embodiment of the data query method, the application also provides an embodiment of a data query unit.
The embodiment of the data query unit can be respectively applied to a database query client and a message system. The unit embodiments may be implemented by software, or by hardware, or by a combination of hardware and software. Taking a software implementation as an example, as a logical unit, the unit is formed by reading a corresponding computer program instruction in the nonvolatile memory into the memory for operation through the processor of the electronic device where the unit is located. From a hardware aspect, as shown in fig. 7, the present application is a hardware structure diagram of an electronic device where a data query unit is located, where the electronic device where the unit is located in the embodiment may further include other hardware according to an actual function of the electronic device, in addition to the processor, the memory, the network interface, and the nonvolatile memory shown in fig. 7, which is not described again.
FIG. 8 is a block diagram of a data query unit, shown in an exemplary embodiment of the present application.
Referring to fig. 8, the data query unit 800 may be applied to the electronic device shown in fig. 7, where the electronic device may be an electronic device loaded with a database query client, such as a server, and includes: a task listening module 801, a data query module 802 and a task generating module 803.
The task monitoring module 801 is configured to monitor a task queue of a message system, and obtain a query task from the task queue, where the query task includes a query condition. The data query module 802 is configured to perform data query in a database based on the query task, and obtain a cursor identifier generated by the database after the query task is executed. The task generating module 803 is configured to generate a new query task corresponding to the cursor identifier based on the cursor identifier and the query condition after the cursor identifier is obtained, and send the new query task to the message system, so that the message system adds the new query task to the task queue.
Optionally, the step of the data query module 802 performing data query in the database based on the query task includes: under the condition that the cursor identification corresponding to the query task is not included in the query task, performing data query in a database based on the query condition; and under the condition that the query task also comprises a cursor identifier corresponding to the query task, carrying out data query in a database based on the cursor identifier and the query condition.
Optionally, the step of querying the data further includes: judging whether the query task carries a current limiting label or not; and limiting the flow of the new query task based on a preset flow limiting measure under the condition that the query task carries a flow limiting label.
Optionally, the step of limiting the flow of the new query task based on a preset flow limiting measure includes: starting timing after the new query task is generated; and after the preset delay time is reached, sending the new query task to a message system.
Optionally, the step of limiting the flow of the new query task based on a preset flow limiting measure includes: starting timing after the cursor identification is obtained; and after the preset delay time is reached, executing a new query task which is generated based on the cursor identification and the query condition and corresponds to the cursor identification, and sending the new query task to a message system.
Optionally, the step of querying data further includes: processing the data inquired from the database; and after the data processing is finished, updating the processed data to the database in batch.
FIG. 9 is a block diagram of another data query unit, shown in an exemplary embodiment of the present application.
Referring to fig. 9, the data query unit 900 may be applied to the electronic device shown in fig. 7, where the electronic device may be a server of a message system, and includes: a first adding module 901 and a second adding module 902.
The first adding module 901 is configured to add a query task to a task queue in response to the query task sent by a query initiator, where the query task includes a query condition, so that a database query client performs data query in a database after obtaining the query task from the task queue, obtains a cursor identifier generated by the database after the query task is completed, and generates a first new query task corresponding to the cursor identifier based on the cursor identifier and the query condition. The second adding module 902 is configured to add, in response to a first new query task sent by a database query client, the first new query task to the task queue, so that the database query client performs data query in the database after obtaining the first new query task from the task queue, and performs generation of a second new query task based on a cursor identifier generated by the database after the first query task is performed and the query condition.
Optionally, the step of adding the first new query task to the task queue by the second adding module 902 includes: adding the first new query task to a delay queue; and after the preset delay time is reached, adding a first new query task in the delay queue to the task queue.
Optionally, the step of adding the first new query task to the delay queue by the second adding module 902 includes: judging whether the query task carries a current limiting label or not; and under the condition that the current limiting label is carried in the query task, adding the first new query task into a delay queue.
The implementation process of the functions and actions of each module in the data query unit is specifically described in the implementation process of the corresponding step in the method, and is not described herein again.
For the data query unit embodiment, since it basically corresponds to the method embodiment, the relevant points can be referred to the partial description of the method embodiment. The above-described voice stream processing unit embodiments are merely illustrative, wherein the modules described as separate components may or may not be physically separate, and the components displayed as modules may or may not be physical units, that is, may be located in one place, or may be distributed on multiple network units. Some or all of the modules can be selected according to actual needs to achieve the purpose of the scheme of the application. One of ordinary skill in the art can understand and implement it without inventive effort.
The systems, devices, modules or units illustrated in the above embodiments may be implemented by a computer chip or an entity, or by a product with certain functions. A typical implementation device is a computer, which may take the form of a personal computer, laptop computer, cellular telephone, camera phone, smart phone, personal digital assistant, media player, navigation device, email messaging device, game console, tablet computer, wearable device, or a combination of any of these devices.
In a typical configuration, a computer includes one or more processors (CPUs), input/output interfaces, network interfaces, and memory.
The memory may include forms of volatile memory in a computer readable medium, Random Access Memory (RAM) and/or non-volatile memory, such as Read Only Memory (ROM) or flash memory (flash RAM). Memory is an example of a computer-readable medium.
Computer-readable media, including both non-transitory and non-transitory, removable and non-removable media, may implement information storage by any method or technology. The information may be computer readable instructions, data structures, modules of a program, or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), Static Random Access Memory (SRAM), Dynamic Random Access Memory (DRAM), other types of Random Access Memory (RAM), Read Only Memory (ROM), Electrically Erasable Programmable Read Only Memory (EEPROM), flash memory or other memory technology, compact disc read only memory (CD-ROM), Digital Versatile Discs (DVD) or other optical storage, magnetic cassettes, magnetic disk storage, quantum memory, graphene-based storage media or other magnetic storage devices, or any other non-transmission medium that can be used to store information that can be accessed by a computing device. As defined herein, a computer readable medium does not include a transitory computer readable medium such as a modulated data signal and a carrier wave.
In correspondence with the foregoing embodiments of the data query method, the present application also provides a computer-readable storage medium having stored thereon a computer program which, when executed by a processor, implements the steps of:
monitoring a task queue of a message system, and acquiring a query task from the task queue, wherein the query task comprises a query condition;
performing data query in a database based on the query task, and acquiring a cursor identifier generated by the database after the query task is executed;
after the cursor identification is obtained, a new query task corresponding to the cursor identification is generated based on the cursor identification and the query condition, and the new query task is sent to a message system so that the message system can add the new query task to the task queue.
Optionally, the step of performing data query in the database based on the query task includes: under the condition that the cursor identification corresponding to the query task is not included in the query task, performing data query in a database based on the query condition; and under the condition that the query task also comprises a cursor identifier corresponding to the query task, carrying out data query in a database based on the cursor identifier and the query condition.
Optionally, the step of querying data further includes: judging whether the query task carries a current limiting label or not; and limiting the flow of the new query task based on a preset flow limiting measure under the condition that the query task carries a flow limiting label.
Optionally, the step of limiting the flow of the new query task based on a preset flow limiting measure includes: starting timing after the new query task is generated; and after the preset delay time is reached, sending the new query task to a message system.
Optionally, the step of limiting the flow of the new query task based on a preset flow limiting measure includes: starting timing after the cursor identification is obtained; and after the preset delay time is reached, executing a new query task which is generated based on the cursor identification and the query condition and corresponds to the cursor identification, and sending the new query task to a message system.
Optionally, the step of querying data further includes: processing the data inquired from the database; and after the data processing is finished, updating the processed data to the database in batch.
In correspondence with the foregoing embodiments of the data query method, the present application also provides another computer-readable storage medium having stored thereon a computer program which, when executed by a processor, implements the steps of:
responding to a query task sent by a query initiator, adding the query task into a task queue, wherein the query task comprises a query condition, so that a database query client performs data query in a database after acquiring the query task from the task queue, acquires a cursor identifier generated by the database after the query task is executed, and generates a first new query task corresponding to the cursor identifier based on the cursor identifier and the query condition;
responding to a first new query task sent by a database query client, adding the first new query task into the task queue, so that the database query client can perform data query in the database after acquiring the first new query task from the task queue, and executing generation of a second new query task based on a cursor identifier generated by the database after the first query task is executed and the query condition.
Optionally, the step of adding the first new query task to the task queue includes: adding the first new query task to a delay queue; and after the preset delay time is reached, adding a first new query task in the delay queue to the task queue.
Optionally, the step of adding the first new query task to the delay queue includes: judging whether the query task carries a current limiting label or not; and under the condition that the current limiting label is carried in the query task, adding the first new query task into a delay queue.
The foregoing description of specific embodiments of the present application has been presented. Other embodiments are within the scope of the following claims. In some cases, the actions or steps recited in the claims may be performed in a different order than in the embodiments and still achieve desirable results. In addition, the processes depicted in the accompanying figures do not necessarily require the particular order shown, or sequential order, to achieve desirable results. In some embodiments, multitasking and parallel processing may also be possible or may be advantageous.
The above description is only exemplary of the present application and should not be taken as limiting the present application, as any modification, equivalent replacement, or improvement made within the spirit and principle of the present application should be included in the scope of protection of the present application.
Claims (13)
1. A data query method is applied to a database query client, and comprises the following steps:
monitoring a task queue of a message system, and acquiring a query task from the task queue, wherein the query task comprises a query condition;
performing data query in a database based on the query task, and acquiring a cursor identifier generated by the database after the query task is executed;
after the cursor identification is obtained, a new query task corresponding to the cursor identification is generated based on the cursor identification and the query condition, and the new query task is sent to a message system so that the message system can add the new query task to the task queue.
2. The method of claim 1, the querying data in a database based on the query task, comprising:
under the condition that the cursor identification corresponding to the query task is not included in the query task, performing data query in a database based on the query condition;
and under the condition that the query task also comprises a cursor identifier corresponding to the query task, carrying out data query in a database based on the cursor identifier and the query condition.
3. The method of claim 1, further comprising:
judging whether the query task carries a current limiting label or not;
and limiting the flow of the new query task based on a preset flow limiting measure under the condition that the query task carries a flow limiting label.
4. The method of claim 3, the limiting the flow of the new query task based on a preset flow limiting measure, comprising:
starting timing after the new query task is generated;
and after the preset delay time is reached, sending the new query task to a message system.
5. The method of claim 3, the limiting the flow of the new query task based on a preset flow limiting measure, comprising:
starting timing after the cursor identification is obtained;
and after the preset delay time is reached, executing a new query task which is generated based on the cursor identification and the query condition and corresponds to the cursor identification, and sending the new query task to a message system.
6. The method of claim 1, further comprising:
processing the data inquired from the database;
and after the data processing is finished, updating the processed data to the database in batch.
7. A data query method is applied to a message system and comprises the following steps:
responding to a query task sent by a query initiator, adding the query task into a task queue, wherein the query task comprises a query condition, so that a database query client performs data query in a database after acquiring the query task from the task queue, acquires a cursor identifier generated by the database after the query task is executed, and generates a first new query task corresponding to the cursor identifier based on the cursor identifier and the query condition;
responding to a first new query task sent by a database query client, adding the first new query task into the task queue, so that the database query client can perform data query in the database after acquiring the first new query task from the task queue, and executing generation of a second new query task based on a cursor identifier generated by the database after the first query task is executed and the query condition.
8. The method of claim 7, the adding the first new query task to the task queue, comprising:
adding the first new query task to a delay queue;
and after the preset delay time is reached, adding a first new query task in the delay queue to the task queue.
9. The method of claim 8, the adding the first new query task to a delay queue, comprising:
judging whether the query task carries a current limiting label or not;
and under the condition that the current limiting label is carried in the query task, adding the first new query task into a delay queue.
10. A data query unit is applied to a database query client, and comprises:
the task monitoring module monitors a task queue of the message system and acquires a query task from the task queue, wherein the query task comprises a query condition;
the data query module is used for performing data query in a database based on the query task and acquiring a cursor identifier generated by the database after the query task is executed;
and the task generation module is used for generating a new query task corresponding to the cursor identification based on the cursor identification and the query condition after the cursor identification is obtained, and sending the new query task to a message system so that the message system can add the new query task to the task queue.
11. A data query unit applied to a message system comprises:
the first adding module is used for responding to a query task sent by a query initiator, adding the query task into a task queue, wherein the query task comprises a query condition, so that a database query client side can perform data query in a database after acquiring the query task from the task queue, acquire a cursor identifier generated by the database after the query task is executed, and generate a first new query task corresponding to the cursor identifier based on the cursor identifier and the query condition;
and the second adding module is used for responding to a first new query task sent by the database query client, adding the first new query task into the task queue so that the database query client can perform data query in the database after acquiring the first new query task from the task queue, and executing generation of a second new query task based on the cursor identifier generated by the database after the first query task is executed and the query condition.
12. An electronic device, comprising:
a processor;
a memory for storing processor-executable instructions;
wherein the processor implements the method of any one of claims 1-9 by executing the executable instructions.
13. A computer-readable storage medium having stored thereon computer instructions, which, when executed by a processor, carry out the steps of the method according to any one of claims 1-9.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202210579957.0A CN114969256A (en) | 2022-05-25 | 2022-05-25 | Data query method and unit |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202210579957.0A CN114969256A (en) | 2022-05-25 | 2022-05-25 | Data query method and unit |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN114969256A true CN114969256A (en) | 2022-08-30 |
Family
ID=82956201
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202210579957.0A Pending CN114969256A (en) | 2022-05-25 | 2022-05-25 | Data query method and unit |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN114969256A (en) |
Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20150242474A1 (en) * | 2014-02-27 | 2015-08-27 | Microsoft Corporation | Inline and context aware query box |
| CN110008262A (en) * | 2019-02-02 | 2019-07-12 | 阿里巴巴集团控股有限公司 | A kind of data export method and device |
| CN111814029A (en) * | 2020-05-25 | 2020-10-23 | 车智互联（北京）科技有限公司 | Data query method, system and computing device |
| CN112948431A (en) * | 2019-11-26 | 2021-06-11 | 中国电信股份有限公司 | Cursor-based database query method and gateway equipment |
| CN114356999A (en) * | 2022-01-11 | 2022-04-15 | 上海途虎信息技术有限公司 | Data processing method, system and computer readable storage medium |
-
2022
- 2022-05-25 CN CN202210579957.0A patent/CN114969256A/en active Pending
Patent Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20150242474A1 (en) * | 2014-02-27 | 2015-08-27 | Microsoft Corporation | Inline and context aware query box |
| CN110008262A (en) * | 2019-02-02 | 2019-07-12 | 阿里巴巴集团控股有限公司 | A kind of data export method and device |
| CN112948431A (en) * | 2019-11-26 | 2021-06-11 | 中国电信股份有限公司 | Cursor-based database query method and gateway equipment |
| CN111814029A (en) * | 2020-05-25 | 2020-10-23 | 车智互联（北京）科技有限公司 | Data query method, system and computing device |
| CN114356999A (en) * | 2022-01-11 | 2022-04-15 | 上海途虎信息技术有限公司 | Data processing method, system and computer readable storage medium |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| US10439937B2 (en) | Service addressing in distributed environment | |
| WO2019205717A1 (en) | Method and device for searching for information in application program | |
| CN110704418A (en) | Block chain information query method, device and equipment | |
| WO2019100853A1 (en) | Page query method, device, and electronic apparatus | |
| EP3058481A1 (en) | Acceleration based on cached flows | |
| CN114780537A (en) | Flow table storage and message forwarding method, device, computing equipment and medium | |
| CN114416224B (en) | Method and device for calling micro service under multi-micro service environment | |
| CN111694639A (en) | Method and device for updating address of process container and electronic equipment | |
| CN113515717B (en) | Navigation page updating method and device, electronic equipment and storage medium | |
| CN110659406B (en) | Searching method and device | |
| CN108228248A (en) | A kind of determining method and apparatus of dependence | |
| CN107193833A (en) | The monitoring method and device of database access | |
| WO2019237949A1 (en) | Search method and device | |
| CN110266610B (en) | Traffic identification method and device and electronic equipment | |
| CN114969256A (en) | Data query method and unit | |
| WO2024021409A1 (en) | Data retrieval method and device | |
| CN112907198B (en) | Service state circulation maintenance method and device and electronic equipment | |
| CN115827280A (en) | Message processing method and device, electronic equipment and storage medium | |
| CN115495171A (en) | Command management method, device, equipment and machine-readable storage medium | |
| CN111047415B (en) | Clothing accessory order processing method, system, electronic equipment and storage medium | |
| CN109214884B (en) | Demand matching method and device and electronic equipment | |
| CN114327941A (en) | Service providing method and device | |
| CN113282617A (en) | Data query method and business system page turning method | |
| CN111414162B (en) | Data processing method, device and equipment thereof | |
| CN117176448B (en) | Service access method, client, electronic equipment and storage medium for instant messaging application |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |