CN110879773B

CN110879773B - CGroup-based memory monitoring method and device

Info

Publication number: CN110879773B
Application number: CN201911203941.4A
Authority: CN
Inventors: 张一罡
Original assignee: Suzhou Inspur Intelligent Technology Co Ltd
Current assignee: Suzhou Inspur Intelligent Technology Co Ltd
Priority date: 2019-11-29
Filing date: 2019-11-29
Publication date: 2023-01-06
Anticipated expiration: 2039-11-29
Also published as: CN110879773A

Abstract

The invention provides a CGroup-based memory monitoring method, which comprises the following steps: periodically traversing all the child CGroup trees, checking a memory.failcnt file of each CGroup, and recording the times that the memory usage amount recorded in the files reaches a limit value; in response to that the number of times that the memory usage of one of the CGroups reaches the limit value is increased compared with the number of times of last recording, checking a CGroup. Selecting directory files of a plurality of processes with the highest OOM scores according to the order of the OOM scores from large to small, and matching the selected process IDs with the process IDs in the printed CGroup to determine the process with the larger OOM score in the CGroup; and taking the matched process ID in the CGroup as alarm information to be notified. The invention can record the use condition of the Linux memory in detail, and is easy to position and solve the memory related errors in the running process of the target program.

Description

CGroup-based memory monitoring method and device

Technical Field

The present invention relates to the field of computers, and in particular, to a CGroup-based memory monitoring method and apparatus.

Background

For the Linux operating system, memory management is always very important. Although the memory capacity of a computer is increased along with the development of hardware, it is still impossible to put all processes and data into the memory, so that the allocation and monitoring of the memory are still very important. If the memory is not used properly or distributed unreasonably, the problems of memory overflow, memory coverage and the like are easily caused.

When a memory error occurs in a program, for example, a memory overflow occurs, the kernel prints out some memory error information to the kernel log to help locate and solve the problem. However, the printing information of the kernel is generally simpler, and the memory error problem is influenced by a plurality of conditions and is often difficult to locate under a more complicated condition. The Linux kernel 2.6.24 introduces CGroup (Control Group), which is a mechanism for managing processes in groups under Linux, and each CGroup has its own independently allocated memory capacity and managed process. If system memory is sufficient, CGroup memory shortage will also cause OOM. The introduction of CGroup makes memory management more complex and memory problems more difficult to find.

Disclosure of Invention

In view of this, an object of the embodiments of the present invention is to provide a CGroup-based memory monitoring method and apparatus, so as to check the current operating condition of a memory through a Linux system memory query interface, and print and output the current operating condition to a log file.

Based on the above object, an aspect of the embodiments of the present invention provides a CGroup-based memory monitoring method, including the following steps:

periodically traversing all the child CGroup trees, checking a memory.failcnt file of each CGroup, and recording the times that the memory usage amount recorded in the files reaches a limit value;

in response to that the number of times that the memory usage of one of the CGroups reaches the limit value is increased compared with the number of times of last recording, checking a CGroup.

Selecting directory files of a plurality of processes with the highest OOM scores according to the order of the OOM scores from large to small, and matching the selected process IDs with the process IDs in the printed CGroup;

and notifying the matched process ID in the CGroup as alarm information.

In some embodiments, the periodically traversing all the child CGroup trees, checking a memory.failcnt file of each CGroup, and recording the number of times that the memory usage amount recorded in the file reaches the limit value includes:

and viewing the memory.use _ in _ bytes file and the memory.limit _ in _ bytes file to respectively view and record the used memory of the CGroup and the memory limit limited by the CGroup.

In some embodiments, if the number of times that the memory usage amount of the record in response to one of the cgroups reaches the limit value is increased from the number of times of last record, checking a CGroup.

And circularly printing the affiliated user, the process ID, the parent process ID and the process execution instruction of each process in the CGroup by using a ps command.

In some embodiments, the selecting the directory files of the processes with the highest OOM scores in order of ascending OOM scores includes:

and matching the directory files of the 10 processes with the maximum OOM score value by using a regular expression.

In some embodiments, the notifying, as the alarm information, the process ID in the CGroup that is matched includes:

and notifying the matched process ID in the CGroup and other related printed information as alarm information through a simple network management protocol.

In some embodiments, the notifying the matched process ID in the CGroup as the alarm information further includes:

and informing the matched CGroup limited memory limit and the used memory information of the CGroup as alarm information.

Another aspect of the embodiments of the present invention provides a CGroup-based memory monitoring apparatus, including:

at least one processor; and

a memory storing program code executable by the processor, the program code implementing the following steps when executed by the processor:

periodically traversing all the sub CGroup trees, checking a memory.failcnt file of each CGroup, and recording the times that the memory usage amount recorded in the files reaches a limit value;

Selecting directory files of a plurality of processes with the highest OOM scores according to the order of the OOM scores from large to small, and matching the selected process IDs with the process IDs in the printed CGroup to determine the process with the larger OOM score in the CGroup;

and taking the matched process ID in the CGroup as alarm information to be notified.

In some embodiments, the periodically traversing all the child CGroup trees, checking a memory.failcnt file of each CGroup, and recording the number of times that the memory usage amount recorded in the file reaches the limit value further includes:

and matching directory files of 10 processes with the maximum OOM score value by using a regular expression.

The invention has the following beneficial technical effects: the CGroup-based memory monitoring method and the CGroup-based memory monitoring device provided by the embodiment of the invention can record the use condition of the Linux memory in detail, are easy to locate and solve memory-related errors in the running process of a target program, such as CGroup memory overflow, are beneficial to debugging the program and monitoring the state of a system, and help to locate and solve memory-related problems in the program development process.

Drawings

In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings used in the description of the embodiments or the prior art will be briefly described below, and it is obvious that the drawings in the following description are only some embodiments of the present invention, and it is obvious for those skilled in the art that other embodiments can be obtained by using the drawings without creative efforts.

FIG. 1 is a flow chart of a CGroup-based memory monitoring method according to the present invention;

FIG. 2 is a schematic diagram of an alarm notification sent over an SNMP service in accordance with the present invention;

fig. 3 is a schematic diagram of a hardware structure of a CGroup-based memory monitoring apparatus according to the present invention.

Detailed Description

Embodiments of the present invention are described below. However, it is to be understood that the disclosed embodiments are merely examples and that other embodiments may take various and alternative forms. The figures are not necessarily to scale; certain features may be exaggerated or minimized to show details of particular components. Therefore, specific structural and functional details disclosed herein are not to be interpreted as limiting, but merely as a representative basis for teaching one skilled in the art to variously employ the present invention. As one of ordinary skill in the art will appreciate, various features illustrated and described with reference to any one of the figures may be combined with features illustrated in one or more other figures to produce embodiments that are not explicitly illustrated or described. The combination of features shown provides a representative embodiment for a typical application. However, various combinations and modifications of the features consistent with the teachings of the present invention may be desired for certain specific applications or implementations.

In order to make the objects, technical solutions and advantages of the present invention more apparent, the following embodiments of the present invention are described in further detail with reference to the accompanying drawings.

Based on the above object, an embodiment of the present invention provides a CGroup-based memory monitoring method, as shown in fig. 1, including the following steps:

step S101: periodically traversing all the child CGroup trees, checking a memory.failcnt file of each CGroup (the file displays the times that the memory usage reaches the limit value), and recording the times that the memory usage recorded in the file reaches the limit value;

step S102: in response to that the number of times that the memory usage of one of the CGroup records reaches the limit value is increased compared with the number of times of last recording, checking a CGroup _ proc file of the CGroup (the file records the ID information of all processes in the current CGroup) to print all running processes in the CGroup in a circulating manner;

step S103: selecting directory files Of a plurality Of processes with the highest OOM scores according to the order Of OOM (Out Of Memory) scores from large to small, and matching the selected process IDs with the process IDs in the printed CGroup to determine the process with the larger OOM score in the CGroup;

step S104: and notifying the matched process ID in the CGroup as alarm information.

In some embodiments, the date and memory may be printed first, for example, using the date command to format the output system time, and using the free command to display the total usage of the system memory.

In some embodiments, the periodically traversing all the child CGroup trees, checking a memory.failcnt file of each CGroup, and recording the number of times that the memory usage amount recorded in the file reaches the limit value includes: and viewing the memory.use _ in _ bytes file and the memory.limit _ in _ bytes file to respectively view and record the used memory of the CGroup and the memory limit limited by the CGroup.

In some embodiments, in a Linux system,/sys/fs/CGroup/memory/directory is a CGroup tree, with all subdirectories under the directory forming nodes on the CGroup tree. All child CGroup trees, i.e., all subdirectories, are traversed by a recursive function. Under each CGroup, the contents of the following files are viewed: use _ in _ bytes, which shows the memory used by the current CGroup; limit _ in _ bytes, which displays the memory limit limited by the current CGroup; failcnt, which shows the number of times the memory usage reaches a limit.

Comparing the used memory and the memory limit of the current CGroup, the closer the used memory is to the memory limit, the greater the risk of OOM of the CGroup. When a process in the CGroup applies for memory, if the memory in the CGroup is exhausted, then the value of memory. Therefore, when the value of memory, failcnt, is increased, it indicates that the CGroup memory is under-allocated, and it is necessary to consider increasing the CGroup memory. The monitoring script executes a traversal operation at regular intervals (set by the user, default 5 seconds), which is called a cycle. When the value of memory, failcnt is checked each time, the value of the last period is compared, if the value is increased, it is indicated that the CGroup is under-allocated, and the CGroup has the risk of memory overflow, and the information of the CGroup is recorded.

In some embodiments, the number of times that the memory usage amount of the record in response to one of the cgroups reaches the limit value is increased compared with the number of times of last record, then viewing a CGroup. And circularly printing the affiliated user, the process ID, the parent process ID and the process execution instruction of each process in the CGroup by using a ps command.

And c, recording pids of all processes in the current CGroup by a c group. Using the ps command, the user (belonging user), pid (process id), ppid (parent process id), cmd (instruction for process execution) of each process is printed cyclically. The printing information is helpful for developers to know the relationship between the process and the Memory, and when the CGroup Memory overflows and triggers the OOM (Out Of Memory is insufficient), the process with the overlarge Memory occupation can be quickly found.

When CGroup runs out because of insufficient memory, the OOM killer (kernel handler after running out) will score each process and then choose the process with the highest score to kill. Oom _ score is the primary reference value for scoring, and the higher the score of oom _ score, the more preferentially killed by the kernel.

In some embodiments, the selecting the directory files of the plurality of processes with the highest OOM scores in order of large to small of the OOM scores (OOM _ score) includes: and matching the directory files of the 10 processes with the maximum OOM score value by using a regular expression.

In some embodiments, under the/proc directory, a regular expression is used to match a directory file in the form of/proc/[ 0-9] + where this directory file name is the process ID. Each directory file records details of a process. In each directory, the contents of a file named oom _ score, which outputs a score for process oom _ score, are printed. Selecting 10 processes from large to small according to oom _ score, and if a process belongs to the 10 processes in the CGroup with the memory.

In some embodiments, the notifying, as the alarm information, the process ID in the CGroup that is matched includes: and informing the matched process ID in the CGroup and other related printed information as alarm information through a simple network management protocol.

In some embodiments, the notifying the matched process ID in the CGroup as the alarm information further includes: and taking the matched memory limit limited by the CGroup and the used memory information of the CGroup as alarm information for notification.

The Simple Network Management Protocol (SNMP) is an application layer protocol of the TCP/IP protocol suite, and is mainly used for management of network devices. In the above steps, as shown in fig. 2, if there is an overflow risk in the memory, the memory monitoring program may print details of the CGroup and details of the process (if there is no overflow risk in the memory, no alarm information is generated, and this step will not be executed), that is, the process ID with a large OOM score in the determined CGroup and other related information. The detailed information related to the memory and the process is input into a local log file for storage on one hand, and is used as a data part of the snmp message on the other hand. The memory monitoring program forms the snmp trap message together with the data part of the snmp message according to the snmp message sending address and the snmp configuration items such as the group word and the like set by a developer in advance. The monitor then sends a snmp trap message to the destination address, typically the developer's own network device. Therefore, once the system memory has overflow risk, the developer receives the alarm information.

Where technically feasible, the technical features listed above for different embodiments may be combined with each other or changed, added, omitted, etc. to form further embodiments within the scope of the invention.

It can be seen from the foregoing embodiments that the CGroup-based memory monitoring method provided in the embodiments of the present invention can record the use condition of the Linux memory in detail, is easy to locate and solve memory-related errors occurring during the running process of the target program, such as CGroup memory overflow, and is beneficial to debugging the program, monitoring the system state, and helping to locate and solve memory-related problems occurring during the program development process.

In view of the foregoing, in another aspect of the embodiments of the present invention, a CGroup-based memory monitoring apparatus is provided, which includes:

at least one processor; and

a memory storing processor executable program code which when executed by the processor performs the steps of:

and taking the matched process ID in the CGroup as alarm information for notification.

In some embodiments, the periodically traversing all the child CGroup trees, checking a memory.failcnt file of each CGroup, and recording the number of times that the memory usage amount recorded in the file reaches the limit value further includes: and viewing the memory.use _ in _ bytes file and the memory.limit _ in _ bytes file to respectively view and record the used memory of the CGroup and the memory limit limited by the CGroup.

In some embodiments, the selecting the directory files of the processes with the highest OOM scores in order of ascending OOM scores includes: and matching directory files of 10 processes with the maximum OOM score value by using a regular expression.

Fig. 3 is a schematic diagram of a hardware structure of an embodiment of the CGroup-based memory monitoring apparatus according to the present invention.

Taking the computer device shown in fig. 3 as an example, the computer device includes a processor 301 and a memory 302, and may further include: an input device 303 and an output device 304.

The processor 301, the memory 302, the input device 303 and the output device 304 may be connected by a bus or other means, and fig. 3 illustrates the connection by a bus as an example.

The memory 302 is used as a non-volatile computer-readable storage medium, and may be used to store non-volatile software programs, non-volatile computer-executable programs, and modules, such as program instructions/modules corresponding to the CGroup-based memory monitoring method in this embodiment of the present application. The processor 301 executes various functional applications and data processing of the server by running the nonvolatile software program, instructions and modules stored in the memory 302, that is, implements the CGroup-based memory monitoring method of the above-described method embodiment.

The memory 302 may include a storage program area and a storage data area, wherein the storage program area may store an operating system, an application program required for at least one function; the storage data area may store data created according to the CGroup-based memory monitoring method, and the like. Further, the memory 302 may include high speed random access memory, and may also include non-volatile memory, such as at least one magnetic disk storage device, flash memory device, or other non-volatile solid state storage device. In some embodiments, memory 302 optionally includes memory located remotely from processor 301, which may be connected to a local module via a network. Examples of such networks include, but are not limited to, the internet, intranets, local area networks, mobile communication networks, and combinations thereof.

The input device 303 may receive input numeric or character information and generate key signal inputs related to user settings and function control of the computer apparatus based on the CGroup memory monitoring method. The output means 304 may comprise a display device such as a display screen.

Program instructions/modules corresponding to the one or more CGroup-based memory monitoring methods are stored in the memory 302, and when executed by the processor 301, perform the CGroup-based memory monitoring method in any of the above-described method embodiments.

Any embodiment of the computer device executing the CGroup-based memory monitoring method may achieve the same or similar effects as any corresponding embodiment of the method described above.

Finally, it should be noted that, as will be understood by those skilled in the art, all or part of the processes in the methods of the above embodiments may be implemented by a computer program, which may be stored in a computer-readable storage medium, and when executed, may include the processes of the embodiments of the methods described above. The storage medium may be a magnetic disk, an optical disk, a read-only memory (ROM), a Random Access Memory (RAM), or the like.

In addition, the apparatuses, devices and the like disclosed in the embodiments of the present invention may be various electronic terminal devices, such as a mobile phone, a Personal Digital Assistant (PDA), a tablet computer (PAD), a smart television and the like, or may be a large terminal device, such as a server and the like, and therefore the scope of protection disclosed in the embodiments of the present invention should not be limited to a specific type of apparatus, device. The client disclosed by the embodiment of the invention can be applied to any one of the electronic terminal devices in the form of electronic hardware, computer software or a combination of the electronic hardware and the computer software.

Furthermore, the method disclosed according to an embodiment of the present invention may also be implemented as a computer program executed by a CPU, and the computer program may be stored in a computer-readable storage medium. The computer program, when executed by the CPU, performs the above-described functions defined in the method disclosed in the embodiments of the present invention.

Further, the above method steps and system elements may also be implemented using a controller and a computer readable storage medium for storing a computer program for causing the controller to implement the functions of the above steps or elements.

Further, it should be appreciated that the computer-readable storage media (e.g., memory) described herein can be either volatile memory or nonvolatile memory, or can include both volatile and nonvolatile memory. By way of example, and not limitation, nonvolatile memory can include Read Only Memory (ROM), programmable ROM (PROM), electrically Programmable ROM (EPROM), electrically Erasable Programmable ROM (EEPROM), or flash memory. Volatile memory can include Random Access Memory (RAM), which can act as external cache memory. By way of example and not limitation, RAM may be available in a variety of forms such as synchronous RAM (DRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDR SDRAM), enhanced SDRAM (ESDRAM), synchlink DRAM (SLDRAM), and Direct Rambus RAM (DRRAM). The storage devices of the disclosed aspects are intended to comprise, without being limited to, these and other suitable types of memory.

Those of skill would further appreciate that the various illustrative logical blocks, modules, circuits, and algorithm steps described in connection with the disclosure herein may be implemented as electronic hardware, computer software, or combinations of both. To clearly illustrate this interchangeability of hardware and software, various illustrative components, blocks, modules, circuits, and steps have been described above generally in terms of their functionality. Whether such functionality is implemented as software or hardware depends upon the particular application and design constraints imposed on the overall system. Skilled artisans may implement the described functionality in varying ways for each particular application, but such implementation decisions should not be interpreted as causing a departure from the scope of the disclosed embodiments of the present invention.

The various illustrative logical blocks, modules, and circuits described in connection with the disclosure herein may be implemented or performed with the following components designed to perform the functions described herein: a general purpose processor, a Digital Signal Processor (DSP), an Application Specific Integrated Circuit (ASIC), a Field Programmable Gate Array (FPGA) or other programmable logic device, discrete gate or transistor logic, discrete hardware components, or any combination of these components. A general purpose processor may be a microprocessor, but in the alternative, the processor may be any conventional processor, controller, microcontroller, or state machine. A processor may also be implemented as a combination of computing devices, e.g., a combination of a DSP and a microprocessor, a plurality of microprocessors, one or more microprocessors in conjunction with a DSP, and/or any other such configuration.

The steps of a method or algorithm described in connection with the disclosure herein may be embodied directly in hardware, in a software module executed by a processor, or in a combination of the two. A software module may reside in RAM memory, flash memory, ROM memory, EPROM memory, EEPROM memory, registers, hard disk, a removable disk, a CD-ROM, or any other form of storage medium known in the art. An exemplary storage medium is coupled to the processor such the processor can read information from, and write information to, the storage medium. In the alternative, the storage medium may be integral to the processor. The processor and the storage medium may reside in an ASIC. The ASIC may reside in a user terminal. In the alternative, the processor and the storage medium may reside as discrete components in a user terminal.

In one or more exemplary designs, the functions may be implemented in hardware, software, firmware, or any combination thereof. If implemented in software, the functions may be stored on or transmitted over as one or more instructions or code on a computer-readable medium. Computer-readable media includes both computer storage media and communication media including any medium that facilitates transfer of a computer program from one place to another. A storage media may be any available media that can be accessed by a general purpose or special purpose computer. By way of example, and not limitation, such computer-readable media can comprise RAM, ROM, EEPROM, CD-ROM or other optical disk storage, magnetic disk storage or other magnetic storage devices, or any other medium that can be used to carry or store desired program code in the form of instructions or data structures and that can be accessed by a general-purpose or special-purpose computer, or a general-purpose or special-purpose processor. Also, any connection is properly termed a computer-readable medium. For example, if the software is transmitted from a website, server, or other remote source using a coaxial cable, fiber optic cable, twisted pair, digital Subscriber Line (DSL), or wireless technologies such as infrared, radio, and microwave, then the coaxial cable, fiber optic cable, twisted pair, DSL, or wireless technologies such as infrared, radio, and microwave are included in the definition of medium. Disk and disc, as used herein, includes Compact Disc (CD), laser disc, optical disc, digital Versatile Disc (DVD), floppy disk, blu-ray disc where disks usually reproduce data magnetically, while discs reproduce data optically with lasers. Combinations of the above should also be included within the scope of computer-readable media.

It should be understood that, as used herein, the singular forms "a", "an" and "the" are intended to include the plural forms as well, unless the context clearly supports the exception. It should also be understood that "and/or" as used herein is meant to include any and all possible combinations of one or more of the associated listed items.

The numbers of the embodiments disclosed in the embodiments of the present invention are merely for description, and do not represent the merits of the embodiments.

It will be understood by those skilled in the art that all or part of the steps for implementing the above embodiments may be implemented by hardware, or may be implemented by a program instructing relevant hardware, where the program may be stored in a computer-readable storage medium, and the above-mentioned storage medium may be a read-only memory, a magnetic disk, an optical disk, or the like.

The above-described embodiments are possible examples of implementations and are presented merely for a clear understanding of the principles of the invention. Those of ordinary skill in the art will understand that: the discussion of any embodiment above is meant to be exemplary only, and is not intended to intimate that the scope of the disclosure, including the claims, of embodiments of the invention is limited to these examples; within the idea of an embodiment of the invention, also technical features in the above embodiment or in different embodiments may be combined and there are many other variations of the different aspects of an embodiment of the invention as described above, which are not provided in detail for the sake of brevity. Therefore, any omissions, modifications, substitutions, improvements, and the like that may be made without departing from the spirit and principles of the embodiments of the present invention are intended to be included within the scope of the embodiments of the present invention.

Claims

1. A CGroup-based memory monitoring method is characterized by comprising the following steps:

2. The method of claim 1, wherein the periodically traversing all child CGroup trees, checking a memory.failcnt file of each CGroup, and recording the number of times that the memory usage amount recorded in the file reaches a limit value comprises:

3. The method of claim 1, wherein the step of viewing a group.proc file of the CGroup to print all running processes in the CGroup in a loop in response to the number of times the memory usage of the record of one CGroup reaches a limit value being increased compared to the number of times the record was last recorded comprises:

and circularly printing the belonged user, the process ID, the parent process ID and the instruction executed by the process of each process in the CGroup by using the ps command.

4. The method of claim 1, wherein the selecting the directory files of the plurality of processes with the highest OOM scores in order of their OOM scores from large to small comprises:

and matching the directory files of the 10 processes with the maximum OOM scores by using regular expressions.

5. The method of claim 3, wherein the notifying the matched process ID in the CGroup as alarm information comprises:

and notifying the matched process ID in the CGroup and other printed related information as alarm information through a simple network management protocol.

6. The method of claim 2, wherein the notifying the matched process ID in the CGroup as alarm information further comprises:

and notifying the matched CGroup limited memory limit and the used memory of the CGroup as alarm information.

7. A CGroup-based memory monitoring device is characterized by comprising:

at least one processor; and

8. The apparatus of claim 7, wherein the periodically traversing all child CGroup trees, checking a memory.failcnt file of each CGroup, and recording the number of times that the memory usage amount recorded in the file reaches a limit value further comprises:

9. The apparatus of claim 7, wherein the number of times the memory usage of the record in response to one of the cgroups reaches a limit value is increased from the number of times of last recording, then viewing a CGroup.

10. The apparatus of claim 7, wherein the selecting the directory files of the plurality of processes with the highest OOM scores in order of the OOM scores from large to small comprises: