CN113760638B - A log service method and device based on kubernetes cluster - Google Patents
A log service method and device based on kubernetes cluster Download PDFInfo
- Publication number
- CN113760638B CN113760638B CN202011103286.8A CN202011103286A CN113760638B CN 113760638 B CN113760638 B CN 113760638B CN 202011103286 A CN202011103286 A CN 202011103286A CN 113760638 B CN113760638 B CN 113760638B
- Authority
- CN
- China
- Prior art keywords
- application
- log
- running
- local
- cloud
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
- 238000000034 method Methods 0.000 title claims abstract description 72
- 230000008569 process Effects 0.000 claims abstract description 33
- 230000000295 complement effect Effects 0.000 claims description 9
- 238000004590 computer program Methods 0.000 claims description 9
- 230000004044 response Effects 0.000 claims description 6
- 238000010586 diagram Methods 0.000 description 13
- 238000004891 communication Methods 0.000 description 7
- 230000000694 effects Effects 0.000 description 6
- 230000006870 function Effects 0.000 description 6
- 238000012545 processing Methods 0.000 description 6
- 230000009286 beneficial effect Effects 0.000 description 4
- 238000007726 management method Methods 0.000 description 4
- 230000003287 optical effect Effects 0.000 description 4
- 230000036541 health Effects 0.000 description 3
- 238000011835 investigation Methods 0.000 description 3
- 238000012986 modification Methods 0.000 description 3
- 230000004048 modification Effects 0.000 description 3
- 238000012544 monitoring process Methods 0.000 description 3
- 230000008901 benefit Effects 0.000 description 2
- 238000004364 calculation method Methods 0.000 description 2
- 239000013307 optical fiber Substances 0.000 description 2
- 230000000644 propagated effect Effects 0.000 description 2
- 239000004065 semiconductor Substances 0.000 description 2
- 238000004458 analytical method Methods 0.000 description 1
- 238000004140 cleaning Methods 0.000 description 1
- 238000010276 construction Methods 0.000 description 1
- 238000007418 data mining Methods 0.000 description 1
- 238000013461 design Methods 0.000 description 1
- 238000005516 engineering process Methods 0.000 description 1
- 239000000835 fiber Substances 0.000 description 1
- 239000004973 liquid crystal related substance Substances 0.000 description 1
- 238000010801 machine learning Methods 0.000 description 1
- 238000012423 maintenance Methods 0.000 description 1
- 230000007246 mechanism Effects 0.000 description 1
- 238000006467 substitution reaction Methods 0.000 description 1
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F11/00—Error detection; Error correction; Monitoring
- G06F11/30—Monitoring
- G06F11/3003—Monitoring arrangements specially adapted to the computing system or computing system component being monitored
- G06F11/3006—Monitoring arrangements specially adapted to the computing system or computing system component being monitored where the computing system is distributed, e.g. networked systems, clusters, multiprocessor systems
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F11/00—Error detection; Error correction; Monitoring
- G06F11/30—Monitoring
- G06F11/3065—Monitoring arrangements determined by the means or processing involved in reporting the monitored data
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/22—Indexing; Data structures therefor; Storage structures
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/24—Querying
- G06F16/245—Query processing
-
- Y—GENERAL TAGGING OF NEW TECHNOLOGICAL DEVELOPMENTS; GENERAL TAGGING OF CROSS-SECTIONAL TECHNOLOGIES SPANNING OVER SEVERAL SECTIONS OF THE IPC; TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
- Y02—TECHNOLOGIES OR APPLICATIONS FOR MITIGATION OR ADAPTATION AGAINST CLIMATE CHANGE
- Y02D—CLIMATE CHANGE MITIGATION TECHNOLOGIES IN INFORMATION AND COMMUNICATION TECHNOLOGIES [ICT], I.E. INFORMATION AND COMMUNICATION TECHNOLOGIES AIMING AT THE REDUCTION OF THEIR OWN ENERGY USE
- Y02D10/00—Energy efficient computing, e.g. low power processors, power management or thermal management
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Databases & Information Systems (AREA)
- Data Mining & Analysis (AREA)
- Computing Systems (AREA)
- Quality & Reliability (AREA)
- Computational Linguistics (AREA)
- Software Systems (AREA)
- Mathematical Physics (AREA)
- Debugging And Monitoring (AREA)
Abstract
The invention discloses a kubernetes cluster-based log service method and device, and relates to the technical field of computers. The method comprises the steps of obtaining an operation state of an application operated on a kubernetes cluster, wherein an operation node corresponding to the application carries out local storage on an application log generated in an operation process of the application to obtain a local application log set, determining an application which is operated completely according to the operation state, carrying out cloud storage on the locally stored application log corresponding to the application which is operated completely to obtain a cloud application log set, responding to an application log query request initiated by a user, sequentially querying the local application log set and the cloud application log set according to an application number included in the application log query request to determine a target application log, and sending the target application log to the user. According to the embodiment, storage resources are saved, the log query flow is simplified, the log query efficiency is improved, and the user experience is improved.
Description
Technical Field
The present invention relates to the field of computer technologies, and in particular, to a kubernetes cluster-based log service method and apparatus.
Background
Kubernetes, an open source, is used for managing containerized applications on multiple hosts in a cloud platform, and Kubernetes clusters are used as a resource scheduling framework for scheduling online services, and the goal is to make deploying containerized applications simple and efficient (powerful), and Kubernetes provides a mechanism for application deployment, planning, updating, and maintenance. Conventional distributed log solutions such as ElK (ELASTICSEARCH, LOGSTASH, KIBANA, which is a log collection management system built by three open source frameworks) are commonly adopted in the current Kubernetes cluster.
Spark is a popular big data computing framework, and can be better applied to algorithms requiring iteration such as data mining and machine learning, and the traditional Spark generally uses a traditional Yarn scheduler to schedule resources.
In the process of implementing the present invention, the inventor finds that at least the following problems exist in the prior art:
1. The existing log service method for running the application in kubernetes clusters mainly adopts the traditional distributed log solution, consumes storage resources and has low log query efficiency;
2. When the Spark application runs in kubernetes clusters, the logs need to be queried and displayed through specific page, search, query, display and other components, and the log query flow is complex, so that the user experience is poor.
Disclosure of Invention
In view of this, the embodiment of the invention provides a method and a device for log service based on kubernetes clusters, which can save storage resources, simplify log query flow, improve log query efficiency and improve user experience.
To achieve the above object, according to a first aspect of the embodiments of the present invention, there is provided a log service method based on kubernetes clusters, including:
the method comprises the steps of obtaining the running state of an application running on a kubernetes cluster, wherein an application log generated in the running process of the application is locally stored by a running node corresponding to the application to obtain a local application log set;
Determining an application which is completed in running according to the running state, and carrying out cloud storage on locally stored application logs corresponding to the application which is completed in running to obtain a cloud application log set;
And responding to an application log query request initiated by a user, sequentially querying a local application log set and a cloud application log set according to an application number included in the application log query request to determine a target application log, and sending the target application log to the user.
Further, the step of obtaining an operating state of an application operating on the kubernetes clusters includes:
and acquiring an application number list from the kubernetes clusters at regular time, wherein the running state corresponding to each application number in the application number list is running on the kubernetes clusters.
Further, the step of determining the running completed application according to the running state includes:
And comparing the application number list with the local list, and determining the application number list and the application in the complement of the local list as the application which is completed, wherein the local list comprises the application number corresponding to the application applied to be run on the kubernetes cluster.
Further, after the step of cloud storing the locally stored application log corresponding to the running application to obtain the local application log set, the method further includes:
and deleting the application log corresponding to the running application from the local application log set.
Further, the application is Spark application, the number of the operation nodes is multiple, and each operation node respectively performs local storage on application logs generated in the operation process of the Spark application to obtain a local application log set corresponding to each operation node.
Further, the application log query request also comprises an operation node number;
According to the application number included in the application log query request, sequentially querying the local application log set and the cloud application log set to determine a target application log, and further comprising:
And sequentially inquiring a local application log set and a cloud application log set corresponding to the running node number according to the application number and the running node number included in the log inquiry request so as to determine a target application log.
According to a second aspect of the embodiment of the invention, a kubernetes cluster-based log service device is provided, which comprises an operation state acquisition module, a log storage module and a log query module, wherein,
The running state acquisition module is used for acquiring the running state of the application running on the kubernetes cluster and sending the acquired running state to the log storage module, wherein the running node corresponding to the application locally stores the application log generated in the running process of the application to obtain a local application log set;
The log storage module is used for determining the application which is completed according to the running state, and carrying out cloud storage on the locally stored application log corresponding to the application which is completed, so as to obtain a cloud application log set;
The log query module is used for responding to an application log query request initiated by a user, sequentially querying a local application log set and a cloud application log set according to an application number included in the application log query request so as to determine a target application log, and sending the target application log to the user.
Further, the application is Spark application, the number of the operation nodes is multiple, and each operation node respectively performs local storage on application logs generated in the operation process of the Spark application to obtain a local application log set corresponding to each operation node.
According to a third aspect of an embodiment of the present invention, there is provided an electronic apparatus including:
One or more processors;
Storage means for storing one or more programs,
When the one or more programs are executed by the one or more processors, the one or more processors are caused to implement any of the kubernetes cluster-based log service methods described above.
According to a fourth aspect of embodiments of the present invention, there is provided a computer readable medium having stored thereon a computer program which, when executed by a processor, implements any of the kubernetes cluster-based log service methods described above.
One embodiment of the above described invention has the advantage or benefit of employing an acquisition of the running state of an application running on the kubernetes cluster; the method comprises the steps of obtaining a local application log set by an application corresponding operation node, carrying out local storage on an application log generated in an operation process of the application to obtain the local application log set, determining the application which is operated according to an operation state, carrying out cloud storage on the locally stored application log corresponding to the application which is operated to be operated to obtain the cloud application log set, responding to an application log query request initiated by a user, sequentially querying the local application log set and the cloud application log set according to an application number included in the application log query request to determine a target application log, and sending the target application log to a technical means of the user, so that the technical problems that the existing log service method for the application operated in an kubernetes cluster consumes storage resources and is low in log query efficiency are solved, further, the storage resources are saved, the log query process is simplified, the log query efficiency is improved, and the technical effect of user experience is improved.
Further effects of the above-described non-conventional alternatives are described below in connection with the embodiments.
Drawings
The drawings are included to provide a better understanding of the invention and are not to be construed as unduly limiting the invention. Wherein:
Fig. 1 is a schematic diagram of main flow of a kubernetes cluster-based log service method according to a first embodiment of the present invention;
FIG. 2 is a schematic diagram of a main flow of a kubernetes cluster-based log service method according to a second embodiment of the present invention;
FIG. 3 is a flow chart of an embodiment of the method of FIG. 2;
FIG. 4 is a schematic diagram of main modules of a kubernetes-cluster-based log service device according to an embodiment of the present invention;
FIG. 5 is an exemplary system architecture diagram in which embodiments of the present invention may be applied;
fig. 6 is a schematic diagram of a computer system suitable for use in implementing an embodiment of the invention.
Detailed Description
Exemplary embodiments of the present invention will now be described with reference to the accompanying drawings, in which various details of the embodiments of the present invention are included to facilitate understanding, and are to be considered merely exemplary. Accordingly, those of ordinary skill in the art will recognize that various changes and modifications of the embodiments described herein can be made without departing from the scope and spirit of the invention. Also, descriptions of well-known functions and constructions are omitted in the following description for clarity and conciseness.
Fig. 1 is a schematic diagram of main flow of a log service method based on kubernetes clusters according to a first embodiment of the present invention, where the log service method according to an embodiment of the present invention mainly relates to collection (storage), query and presentation of application logs. As shown in fig. 1, the method for service of logs based on kubernetes clusters according to the embodiment of the present invention mainly includes:
And step S101, acquiring the running state of the application running on the kubernetes cluster, wherein the running node corresponding to the application locally stores the application log generated in the running process of the application to obtain a local application log set.
In the prior art, for the collection of application logs of applications running on kubernetes clusters, a set of log service systems with multiple components (including log collection components, log storage management components, log analysis display components and the like) mounted are built again mainly through the traditional distributed log collection scheme such as ELK, for example, if the log service systems are used for storing logs, the application logs generated by the applications running on kubernetes clusters are mounted on network storage for storage, and the application start delay is usually caused when the applications run, and the application start failure rate is also high.
In the implementation of the invention, the local storage device is mounted aiming at the operation node corresponding to the application, so that the operation node can directly store the application log generated in the application operation process. Because the local storage speed is high, the log storage efficiency is improved, and the problem of storage faults caused by the simultaneous operation of a plurality of kubernetes clusters is solved.
Further, according to an embodiment of the present invention, the step of obtaining the running state of the application running on the kubernetes clusters includes:
and acquiring an application number list from the kubernetes clusters at regular time, wherein the running state corresponding to each application number in the application number list is running on the kubernetes clusters.
In the actual running process, the running states include "running" (running), running retry, running failure, submitting, running success "(wherein the latter four states all represent that the running is completed in a certain stage or all), and other intermediate states, not just the two states of" running "and" running completion ", by the above arrangement, the running state is obtained as the running application number list, and thus the running application can be accurately obtained, and the running application can be determined. The method is beneficial to collecting the application logs corresponding to the application which is not in the running state, and is beneficial to avoiding the loss of the application logs.
Step S102, determining the application with the running completion according to the running state, and performing cloud storage on locally stored application logs corresponding to the application with the running completion to obtain a cloud application log set.
Specifically, according to an embodiment of the present invention, the step of determining the running completed application according to the running state includes:
And comparing the application number list with the local list, and determining the application number list and the application in the complement of the local list as the application which is completed, wherein the local list comprises the application number corresponding to the application applied to be run on the kubernetes cluster.
All application numbers of the application running in kubernetes cluster submission applications are recorded in the local list, and the complement of the application numbers corresponding to the running application is an application which is not in a running state, namely the running application is completed, and the application logs corresponding to the type of application are transferred from the local storage device to the cloud storage device so as to facilitate subsequent log query by comparing the application number list corresponding to the running application with the local list.
Further, according to a specific implementation manner of the embodiment of the present invention, a first application number may be further determined from the local list, whether the first application number exists in the received application number list corresponding to the running application, if so, the application corresponding to the first application number is in the running state, and if not, the first application number may be cloud-stored in an application log corresponding to the local storage device.
Further, according to the embodiment of the invention, after the step of cloud storage of the locally stored application log corresponding to the running application to obtain the local application log set, the method further comprises the step of deleting the application log corresponding to the running application from the local application log set.
Through the arrangement, after cloud storage is carried out on the corresponding application logs in the local storage device, the corresponding application logs in the local storage device are deleted, so that the storage space of the local storage device can be released, the burden of an operation node is reduced, and the operation efficiency of the operation node is prevented from being influenced due to insufficient local storage space.
Step S103, responding to an application log query request initiated by a user, sequentially querying a local application log set and a cloud application log set according to an application number included in the application log query request to determine a target application log, and sending the target application log to the user.
Specifically, according to the embodiment of the invention, the application log query request can further include application running time (for the application log named by taking the running time as a storage file, the efficiency and accuracy of querying the target application log can be improved).
Further, according to the embodiment of the present invention, the application is a Spark application, the number of operation nodes is multiple, and each operation node respectively locally stores an application log generated in the operation process of the application, so as to obtain a local application log set corresponding to each operation node.
According to an embodiment of the present invention, the application is a distributed computing program, preferably a Spark application. The distributed computing program has large calculation amount, needs a plurality of operation nodes to cooperate, and is more suitable for operation on kubernetes clusters.
The Spark application is used as a popular big data computing framework, the application range is quite wide, and in the prior art, the Spark application runs in a Yarn (another scheduling framework) cluster, but due to the fact that the Yarn has insufficient expansibility, the flexibility is low, the resource utilization rate is low, and the offline and online mixed deployment cannot be well realized. Through the combination of Spark application and kubernetes clusters (applications for managing containerization on a plurality of hosts in a cloud platform), not only can a framework be uniformly scheduled and a computing flow be optimized, but also hybrid deployment of online service and offline service can be realized.
Preferably, according to the embodiment of the present invention, the application log query request further includes an operation node number, and the step of sequentially querying the local application log set and the cloud application log set according to the application number included in the application log query request to determine the target application log further includes:
And sequentially inquiring a local application log set and a cloud application log set corresponding to the running node number according to the application number and the running node number included in the log inquiry request so as to determine a target application log.
Specifically, after the application is started, tasks are respectively allocated to a plurality of operation nodes, the plurality of operation nodes respectively execute part of operation tasks, the operation node number is further specified in the application log query request, a target application log corresponding to the operation node can be obtained, and then the operation condition, the resource use condition and the like of the application on the operation node can be determined according to the target application log, so that the monitoring and the investigation of the operation health condition of the application on each operation node are realized.
According to the technical scheme, the running state of the application running on the kubernetes cluster is acquired, the running node corresponding to the application locally stores the application log generated in the running process of the application to obtain a local application log set, the running-completed application is determined according to the running state, the locally stored application log corresponding to the running-completed application is stored in the cloud to obtain a cloud application log set, the local application log set and the cloud application log set are sequentially queried according to the application number included in the application log query request in response to an application log query request initiated by a user, so that a target application log is determined, and the target application log is sent to a technical means of the user, and therefore the technical problem that the existing log service method for running the application in the kubernetes cluster consumes storage resources due to the fact that the traditional distributed log solution is mainly adopted, the log query efficiency is low is solved, the technical problem that storage resources are saved, the log query flow is simplified, the log query efficiency is improved, the technical effect of user experience is improved, the application is Spark application, when the Spark application in the prior art is run in kubernetes, the technical experience is improved, the problem of searching and the user experience is improved through a special page, the user experience is improved, the problem is solved, the user experience is improved, the problem is poor in the user experience is seen, and the user experience is improved is the problem is the user experience is the is improved.
Fig. 2 is a schematic diagram of main flow of a kubernetes cluster-based log service method according to a second embodiment of the present invention, where an application scenario of the embodiment of the present invention is a log service method running on Spark of kubernetes clusters, as shown in fig. 2, mainly including:
Step S201, an application number list is obtained from the kubernetes clusters at regular time, wherein the running state corresponding to each application number in the application number list is running on the kubernetes clusters.
Specifically, the Spark application runs on kubernetes clusters, the Spark application runs on a plurality of running nodes, and each running node locally stores the application log generated in the running process to obtain a local application log set. Specifically, according to an embodiment of the present invention, as shown in fig. 3, after Spark submits an operation application to kubernetes clusters through a submit program, multiple tasks are respectively allocated to multiple operation nodes (computing nodes, executor, and computing programs) through drivers, processing of the tasks (i.e., operation application) is performed by each operation node, and an application log generated in the operation process is stored in a local disk (localvolume), i.e., local storage is performed. kubernetes the cluster is used to query the running state of the application (which may be specific to each running node) through SPARKTASKSVC (running state acquisition module).
Because the Spark has a plurality of intermediate states in the running process, not only the running state and the running completion state, by the setting, the running application number list with the running state of running can be obtained accurately, and further the running application can be determined.
Furthermore, according to the embodiment of the present invention, the above application is not only suitable for Spark application, but also suitable for running on kubernetes clusters for distributed computing programs due to the large amount of computation involved in the running process, and the multiple running nodes are required to cooperate.
Spark applications are a popular big data computing framework and are widely used. In the prior art, spark application runs in a Yarn (another scheduling framework) cluster, but the Yarn has low flexibility and low resource utilization rate due to insufficient expansibility, so that offline and online hybrid deployment cannot be well realized. Through the combination of Spark application and kubernetes clusters (applications for managing containerization on a plurality of hosts in a cloud platform), not only can a framework be uniformly scheduled and a computing flow be optimized, but also hybrid deployment of online service and offline service can be realized.
Step S202, judging whether the application corresponding to the first application number is operated according to the application number list and the first application number. If yes, the running of the application corresponding to the first application number is completed, step S203 is executed, and if not, the running state corresponding to the first application number is "running", step S201 is shifted to.
Further, according to the embodiment of the invention, the local list and the application number list can be directly utilized for comparison, the application in the intersection of the application number list and the local list is determined to be the application which is not run completely, and the application in the complement of the application number list and the local list is determined to be the application which is run completely. Because the local list records all application numbers of the application running in kubernetes cluster submission application, the complement of the application numbers corresponding to the running application is the application which is not in the running state by comparing the application number list with the local list, and the application logs corresponding to the type of application are transferred to the cloud storage device from the local storage device so as to facilitate subsequent log inquiry.
Step S203, determining an application log corresponding to the first application number in the local storage device, performing cloud storage on the application log, and deleting the corresponding application log in the local storage device.
Through the arrangement, after the cloud storage is carried out on the corresponding application logs in the local storage device, the corresponding application logs in the local storage device are deleted, so that the storage space of the local storage device can be released, the burden of the operation node is reduced, and the operation efficiency of the operation node is prevented from being influenced due to insufficient local storage space. Specifically, according to an embodiment of the present invention, by setting cleaner (a cleaning module, which is equivalent to a log storage module that is hereinafter set in each node of the kubernetes cluster), cleaner (a log storage module) obtains an operation state from SPARKTASKSVC (an operation state obtaining module), and stores an application log corresponding to an application that is "completed" in a cloud (to CS, closed storage), and at the same time, deletes the corresponding application log in the local disk (localvolume).
Step S204, responding to an application log query request initiated by a user, and sequentially querying a local application log set and a cloud application log set corresponding to the running node number according to the application number and the running node number included in the log query request so as to determine a target application log.
Specifically, according to the embodiment of the invention, the application log query request can further include application running time (for the application log named by taking the running time as a storage file, the efficiency and accuracy of querying the target application log can be improved).
Through the arrangement, the target application log corresponding to the operation node can be obtained, and then the operation condition, the resource use condition and the like of the application on the operation node can be determined according to the target application log, so that the monitoring and the investigation of the operation health condition of the application on each operation node are realized. Specifically, as shown in fig. 3, according to an embodiment of the present invention, a log service (log query module, providing log query service) is set, and after receiving a log query request initiated by a user, the log query module sequentially queries a local disk and a cloud storage device according to information (including an application number and an operating node number) indicated in the log query request, so as to determine a target application log.
Step S205, the target application log is sent to the user.
Specifically, as shown in fig. 3, according to an embodiment of the present invention, a user may initiate a log query request through SparkUI or HistoryServer (history service program) of a Spark application (HistoryServer and SparkUI pages provide a set of web ui user interfaces (the interfaces provide log query links of each running node) for the user to monitor the running state of the Spark application and the resource consumption of the kubernetes cluster, and generally, use SparkUI for running application tasks and use Histo ryServer for completed application tasks). A user can initiate a log query request comprising application running time, application number and running node number to log service (log query module) through a page, receive a target application log returned by the log service, and finally display the target application log to the user on a SparkUI or HistoryServer page. Through the arrangement, the query process of the application log is hidden from the user, the habit of viewing the application log through SparkUI or HistoryServe r pages in the past under the scheduling frames of Yarn and the like is reserved, and the user experience is improved.
According to the technical scheme, the running state of the application running on the kubernetes cluster is acquired, the running node corresponding to the application locally stores the application log generated in the running process of the application to obtain a local application log set, the running-completed application is determined according to the running state, the locally stored application log corresponding to the running-completed application is stored in the cloud to obtain a cloud application log set, the local application log set and the cloud application log set are sequentially queried according to the application number included in the application log query request in response to an application log query request initiated by a user, so that a target application log is determined, and the target application log is sent to a technical means of the user, and therefore the technical problem that the existing log service method for running the application in the kubernetes cluster consumes storage resources due to the fact that the traditional distributed log solution is mainly adopted, the log query efficiency is low is solved, the technical problem that storage resources are saved, the log query flow is simplified, the log query efficiency is improved, the technical effect of user experience is improved, the application is Spark application, when the Spark application in the prior art is run in kubernetes, the technical experience is improved, the problem of searching and the user experience is improved through a special page, the user experience is improved, the problem is solved, the user experience is improved, the problem is poor in the user experience is seen, and the user experience is improved is the problem is the user experience is the is improved.
Fig. 4 is a schematic diagram of main modules of a kubernetes-cluster-based log service device according to an embodiment of the present invention, and as shown in fig. 4, a kuber netes-cluster-based log service device 400 according to an embodiment of the present invention mainly includes:
The running state obtaining module 401 is configured to obtain a running state of an application running on the kubernetes cluster, where an running node corresponding to the application locally stores an application log generated in the running process of the application, to obtain a local application log set.
In the implementation of the invention, the local storage device is mounted aiming at the operation node corresponding to the application, so that the operation node can directly store the application log generated in the application operation process. Because the local storage speed is high, the log storage efficiency is improved, and the problem of storage faults caused by the simultaneous operation of a plurality of kubernet es clusters is solved.
Further, according to the embodiment of the present invention, the operation state obtaining module 401 is further configured to obtain an application number list from the kubernetes clusters at regular time, where an operation state corresponding to each application number in the application number list is running on the kubernetes clusters.
In the actual running process, the running state is not only the running state and the successful running state, and by the setting, the running application number list with the running state being the running state is obtained, so that the running application can be accurately obtained, and further, the application which is not running can be determined. The method is beneficial to collecting the application logs corresponding to the application which is not in the running state, and is beneficial to avoiding the loss of the application logs.
The log storage module 402 is configured to determine an application that is running completely according to the running state, and store the locally stored application log corresponding to the running application in a cloud to obtain a cloud application log set.
Specifically, according to an embodiment of the present invention, the log storage module 402 is further configured to:
And comparing the application number list with the local list, and determining the application number list and the application in the complement of the local list as the application which is completed, wherein the local list comprises the application number corresponding to the application applied to be run on the kubernetes cluster.
All application numbers of the application running in kubernetes cluster submission applications are recorded in the local list, and the complement of the application numbers corresponding to the running application is an application which is not in a running state by comparing the application number list with the local list, and an application log corresponding to the type of application is transferred to a cloud storage device from the local storage device so as to facilitate subsequent log query.
Further, according to an embodiment of the present invention, after the step of cloud storing the locally stored application log corresponding to the running application to obtain the local application log set, the log storage module 402 is further configured to:
and deleting the application log corresponding to the running application from the local application log set.
Through the arrangement, after cloud storage is carried out on the corresponding application logs in the local storage device, the corresponding application logs in the local storage device are deleted, so that the storage space of the local storage device can be released, the burden of an operation node is reduced, and the operation efficiency of the operation node is prevented from being influenced due to insufficient local storage space.
The log query module 403 is configured to respond to an application log query request initiated by a user, sequentially query a local application log set and a cloud application log set according to an application number included in the application log query request, so as to determine a target application log, and send the target application log to the user.
Specifically, according to the embodiment of the invention, the application log query request can further include application running time (for the application log named by taking the running time as a storage file, the efficiency and accuracy of querying the target application log can be improved).
Further, according to the embodiment of the present invention, the application is a Spark application, the number of operation nodes is multiple, and each operation node respectively locally stores an application log generated in the operation process of the application, so as to obtain a local application log set corresponding to each operation node.
According to an embodiment of the present invention, the application is a distributed computing program, preferably a Spark application. The distributed computing program design has large calculation amount, and a plurality of operation nodes are required to cooperate, so that the distributed computing program is more suitable for operation on kubernetes clusters.
The Spark application is used as a popular big data computing frame, the application range is very wide, in the prior art, the Spark application runs in a Yarn (another scheduling frame) cluster, and due to the insufficient expansibility of the Yarn, the flexibility is low, the resource utilization rate is low, and the offline and online hybrid deployment cannot be well realized. Through the combination of Spark application and kubernetes clusters (applications for managing containerization on a plurality of hosts in a cloud platform), not only can a framework be uniformly scheduled and a computing flow be optimized, but also hybrid deployment of online service and offline service can be realized.
Preferably, according to the embodiment of the present invention, the application log query request further includes an operation node number, and the log query module 403 is further configured to:
And sequentially inquiring a local application log set and a cloud application log set corresponding to the running node number according to the application number and the running node number included in the log inquiry request so as to determine a target application log.
Specifically, after the application is started, tasks are respectively allocated to a plurality of operation nodes, the plurality of operation nodes respectively execute part of operation tasks, the operation node number is further specified in the application log query request, a target application log corresponding to the operation node can be obtained, and then the operation condition, the resource use condition and the like of the application on the operation node can be determined according to the target application log, so that the monitoring and the investigation of the operation health condition of the application on each operation node are realized.
According to the technical scheme, the running state of the application running on the kubernetes cluster is acquired, the running node corresponding to the application locally stores the application log generated in the running process of the application to obtain a local application log set, the running-completed application is determined according to the running state, the locally stored application log corresponding to the running-completed application is stored in the cloud to obtain a cloud application log set, the local application log set and the cloud application log set are sequentially queried according to the application number included in the application log query request in response to an application log query request initiated by a user, so that a target application log is determined, and the target application log is sent to a technical means of the user, and therefore the technical problem that the existing log service method for running the application in the kubernetes cluster consumes storage resources due to the fact that the traditional distributed log solution is mainly adopted, the log query efficiency is low is solved, the technical problem that storage resources are saved, the log query flow is simplified, the log query efficiency is improved, the technical effect of user experience is improved, the application is Spark application, when the Spark application in the prior art is run in kubernetes, the technical experience is improved, the problem of searching and the user experience is improved through a special page, the user experience is improved, the problem is solved, the user experience is improved, the problem is poor in the user experience is seen, and the user experience is improved is the problem is the user experience is the is improved.
Fig. 5 illustrates an exemplary system architecture 500 of a kubernetes cluster-based log service method or kubernetes cluster-based log service device to which embodiments of the invention may be applied.
As shown in fig. 5, a system architecture 500 may include terminal devices 501, 502, 503, a network 504, and a server 505 (this architecture is merely an example, and the components contained in a particular architecture may be tailored to the application specific case). The network 504 is used as a medium to provide communication links between the terminal devices 501, 502, 503 and the server 505. The network 504 may include various connection types, such as wired, wireless communication links, or fiber optic cables, among others.
A user may interact with the server 505 via the network 504 using the terminal devices 501, 502, 503 to receive or send messages or the like. Various communication client applications, such as a big data computing class application, a data processing class application, a log service class application, etc. (only examples) may be installed on the terminal devices 501, 502, 503.
The terminal devices 501, 502, 503 may be a variety of electronic devices having a display screen and supporting web browsing, including but not limited to smartphones, tablets, laptop and desktop computers, and the like.
The server 505 may be a server providing various services, such as a background management server (by way of example only) providing support for shopping-type websites browsed by users using the terminal devices 501, 502, 503. The background management server may analyze and process the received data such as the application log query request, and feed back the processing result (e.g., the target application log—only an example) to the terminal device.
It should be noted that, the kubernetes-cluster-based log service method provided by the embodiment of the present invention is generally executed by the server 505, and accordingly, the kubernetes-cluster-based log service device is generally disposed in the server 505.
It should be understood that the number of terminal devices, networks and servers in fig. 5 is merely illustrative. There may be any number of terminal devices, networks, and servers, as desired for implementation.
Referring now to FIG. 6, there is illustrated a schematic diagram of a computer system 600 suitable for use in implementing a terminal device or server in accordance with an embodiment of the present invention. The terminal device or server shown in fig. 6 is only an example, and should not impose any limitation on the functions and scope of use of the embodiments of the present invention.
As shown in fig. 6, the computer system 600 includes a Central Processing Unit (CPU) 601, which can perform various appropriate actions and processes according to a program stored in a Read Only Memory (ROM) 602 or a program loaded from a storage section 608 into a Random Access Memory (RAM) 603. In the RAM 603, various programs and data required for the operation of the system 600 are also stored. The CPU 601, ROM 602, and RAM 603 are connected to each other through a bus 604. An input/output (I/O) interface 605 is also connected to bus 604.
Connected to the I/O interface 605 are an input section 606 including a keyboard, a mouse, and the like, an output section 607 including a Cathode Ray Tube (CRT), a Liquid Crystal Display (LCD), and the like, a speaker, and the like, a storage section 608 including a hard disk, and the like, and a communication section 609 including a network interface card such as a LAN card, a modem, and the like. The communication section 609 performs communication processing via a network such as the internet. The drive 610 is also connected to the I/O interface 605 as needed. Removable media 611 such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, or the like is installed as needed on drive 610 so that a computer program read therefrom is installed as needed into storage section 608.
In particular, according to embodiments of the present disclosure, the processes described above with reference to flowcharts may be implemented as computer software programs. For example, embodiments of the present disclosure include a computer program product comprising a computer program embodied on a computer readable medium, the computer program comprising program code for performing the method shown in the flow chart. In such an embodiment, the computer program may be downloaded and installed from a network through the communication portion 609, and/or installed from the removable medium 611. The above-described functions defined in the system of the present invention are performed when the computer program is executed by a Central Processing Unit (CPU) 601.
The computer readable medium shown in the present invention may be a computer readable signal medium or a computer readable storage medium, or any combination of the two. The computer readable storage medium can be, for example, but not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or a combination of any of the foregoing. More specific examples of a computer-readable storage medium may include, but are not limited to, an electrical connection having one or more wires, a portable computer diskette, a hard disk, a Random Access Memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing. In the context of this document, a computer readable storage medium may be any tangible medium that can contain, or store a program for use by or in connection with an instruction execution system, apparatus, or device. In the present invention, however, the computer-readable signal medium may include a data signal propagated in baseband or as part of a carrier wave, with the computer-readable program code embodied therein. Such a propagated data signal may take any of a variety of forms, including, but not limited to, electro-magnetic, optical, or any suitable combination of the foregoing. A computer readable signal medium may also be any computer readable medium that is not a computer readable storage medium and that can communicate, propagate, or transport a program for use by or in connection with an instruction execution system, apparatus, or device. Program code embodied on a computer readable medium may be transmitted using any appropriate medium, including but not limited to wireless, wireline, optical fiber cable, RF, etc., or any suitable combination of the foregoing.
The flowcharts and block diagrams in the figures illustrate the architecture, functionality, and operation of possible implementations of systems, methods and computer program products according to various embodiments of the present invention. In this regard, each block in the flowchart or block diagrams may represent a module, segment, or portion of code, which comprises one or more executable instructions for implementing the specified logical function(s). It should also be noted that, in some alternative implementations, the functions noted in the block may occur out of the order noted in the figures. For example, two blocks shown in succession may, in fact, be executed substantially concurrently, or the blocks may sometimes be executed in the reverse order, depending upon the functionality involved. It will also be noted that each block of the block diagrams or flowchart illustration, and combinations of blocks in the block diagrams or flowchart illustration, can be implemented by special purpose hardware-based systems which perform the specified functions or acts, or combinations of special purpose hardware and computer instructions.
The modules involved in the embodiments of the present invention may be implemented in software or in hardware. The described modules may also be provided in a processor, for example, a processor may be described as including an operating state acquisition module, a log storage module, and a log query module. The names of the modules do not limit the module itself in some cases, for example, the running state obtaining module may also be described as "a module for obtaining a running state of an application running on the kubernetes clusters and sending the obtained running state to the log storage module, where an application log generated in the running process of the application is locally stored by an application corresponding running node, so as to obtain a local application log set".
As a further aspect, the invention also provides a computer readable medium which may be comprised in the device described in the above embodiments or may be present alone without being fitted into the device. The computer readable medium carries one or more programs, when the one or more programs are executed by the equipment, the equipment comprises an operation state of an application running on a kubernetes cluster, wherein an operation node corresponding to the application locally stores an application log generated in an application running process to obtain a local application log set, the running completed application is determined according to the operation state, the locally stored application log corresponding to the running completed application is cloud-stored to obtain a cloud application log set, the local application log set and the cloud application log set are sequentially queried according to an application number included in the application log query request in response to an application log query request initiated by a user, so that a target application log is determined, and the target application log is sent to the user.
According to the technical scheme, the running state of the application running on the kubernetes cluster is acquired, the running node corresponding to the application locally stores the application log generated in the running process of the application to obtain a local application log set, the running-completed application is determined according to the running state, the locally stored application log corresponding to the running-completed application is stored in the cloud to obtain a cloud application log set, the local application log set and the cloud application log set are sequentially queried according to the application number included in the application log query request in response to an application log query request initiated by a user, so that a target application log is determined, and the target application log is sent to a technical means of the user, and therefore the technical problem that the existing log service method for running the application in the kubernetes cluster consumes storage resources due to the fact that the traditional distributed log solution is mainly adopted, the log query efficiency is low is solved, the technical problem that storage resources are saved, the log query flow is simplified, the log query efficiency is improved, the technical effect of user experience is improved, the application is Spark application, when the Spark application in the prior art is run in kubernetes, the technical experience is improved, the problem of searching and the user experience is improved through a special page, the user experience is improved, the problem is solved, the user experience is improved, the problem is poor in the user experience is seen, and the user experience is improved is the problem is the user experience is the is improved.
The above embodiments do not limit the scope of the present invention. It will be apparent to those skilled in the art that various modifications, combinations, sub-combinations and alternatives can occur depending upon design requirements and other factors. Any modifications, equivalent substitutions and improvements made within the spirit and principles of the present invention should be included in the scope of the present invention.
Claims (8)
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202011103286.8A CN113760638B (en) | 2020-10-15 | 2020-10-15 | A log service method and device based on kubernetes cluster |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202011103286.8A CN113760638B (en) | 2020-10-15 | 2020-10-15 | A log service method and device based on kubernetes cluster |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN113760638A CN113760638A (en) | 2021-12-07 |
| CN113760638B true CN113760638B (en) | 2024-12-06 |
Family
ID=78786036
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202011103286.8A Active CN113760638B (en) | 2020-10-15 | 2020-10-15 | A log service method and device based on kubernetes cluster |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN113760638B (en) |
Families Citing this family (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN114356718A (en) * | 2022-01-10 | 2022-04-15 | 网易（杭州）网络有限公司 | Log processing method, medium, system and computing device |
| CN114500254B (en) * | 2022-02-16 | 2023-08-22 | 平安科技（深圳）有限公司 | Historical log collection method, device and storage medium |
| CN114691050B (en) * | 2022-05-26 | 2022-09-06 | 深圳前海环融联易信息科技服务有限公司 | Cloud native storage method, device, equipment and medium based on kubernets |
| CN118468046A (en) * | 2024-07-15 | 2024-08-09 | 山东高速建筑设计有限公司 | Building planning platform and application method thereof |
Citations (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN110955578A (en) * | 2019-11-26 | 2020-04-03 | 深圳市信联征信有限公司 | Log collection method and device based on host machine, computer equipment and storage medium |
Family Cites Families (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN108153532A (en) * | 2017-12-26 | 2018-06-12 | 江苏润和软件股份有限公司 | A kind of cloud application dispositions method based on Web log mining |
| US10757215B2 (en) * | 2018-09-13 | 2020-08-25 | Pivotal Software, Inc. | Allocation of computing resources for a computing system hosting multiple applications |
| CN109245931B (en) * | 2018-09-19 | 2021-04-16 | 四川虹微技术有限公司 | Log management and monitoring alarm realization method of container cloud platform based on kubernets |
| CN110311817B (en) * | 2019-06-28 | 2021-09-28 | 四川长虹电器股份有限公司 | Container log processing system for Kubernetes cluster |
| CN111104368B (en) * | 2019-12-16 | 2023-08-22 | 浪潮云信息技术股份公司 | Method and system for realizing storage capacity expansion of container |
-
2020
- 2020-10-15 CN CN202011103286.8A patent/CN113760638B/en active Active
Patent Citations (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN110955578A (en) * | 2019-11-26 | 2020-04-03 | 深圳市信联征信有限公司 | Log collection method and device based on host machine, computer equipment and storage medium |
Also Published As
| Publication number | Publication date |
|---|---|
| CN113760638A (en) | 2021-12-07 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| US12007996B2 (en) | Management of distributed computing framework components | |
| US11921672B2 (en) | Query execution at a remote heterogeneous data store of a data fabric service | |
| US20220327125A1 (en) | Query scheduling based on a query-resource allocation and resource availability | |
| US11615087B2 (en) | Search time estimate in a data intake and query system | |
| US11580107B2 (en) | Bucket data distribution for exporting data to worker nodes | |
| US11593377B2 (en) | Assigning processing tasks in a data intake and query system | |
| US11599541B2 (en) | Determining records generated by a processing task of a query | |
| US11321321B2 (en) | Record expansion and reduction based on a processing task in a data intake and query system | |
| US11586627B2 (en) | Partitioning and reducing records at ingest of a worker node | |
| US11442935B2 (en) | Determining a record generation estimate of a processing task | |
| CN113760638B (en) | A log service method and device based on kubernetes cluster | |
| US20200050607A1 (en) | Reassigning processing tasks to an external storage system | |
| CN113760488A (en) | Method, device, equipment and computer readable medium for scheduling task | |
| CN111831503B (en) | Monitoring method based on monitoring agent and monitoring agent device | |
| CN113722007B (en) | Configuration method, device and system of VPN branch equipment | |
| CN111767126A (en) | System and method for distributed batch processing | |
| WO2024045646A1 (en) | Method, apparatus and system for managing cluster access permission | |
| CN111488268A (en) | Scheduling method and scheduling device for automated testing | |
| CN113407331B (en) | A task processing method, device and storage medium | |
| US11567800B2 (en) | Early identification of problems in execution of background processes | |
| CN114756173A (en) | Method, system, device and computer readable medium for file merging | |
| CN113821519B (en) | Data processing method and domain drive design architecture | |
| CN115080143B (en) | Page resource preloading method, device, equipment and storage medium | |
| CN118819788B (en) | Cluster scheduling method, device and equipment | |
| CN111767113B (en) | A method and device for implementing container expulsion |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |