Understanding Persistent-memory-related Issues in the Linux Kernel

A wide range of non-volatile memory (NVM) devices have been proposed and are being developed with different degrees of maturity. To facilitate standardization, the JEDEC specification [11] classifies NVM devices into the following three types based on the method used for persistence:

In addition, to facilitate NVM programming, the Storage Networking Industry Association [22] differentiates the concepts of NVM and PM [18]: NVM refers to any type of memory-based, persistent media (including flash memory packaged as solid state disks), while PM refers to a subset of NVM technology with performance characteristics suitable for a load and store programming (e.g., PCM, STT-RAM, and Optane DCPMM). In this article, we follow the definition and study the Linux kernel issues related to such PM devices.

PM devices reside on the memory bus along with DRAM devices. Unlike traditional block storage devices that provide a block IO interface to access data, PM devices use memory load and store instructions. These devices typically offer lower access latency and higher endurance compared to flash-based storage devices. For example, STT-RAM and PCM devices may offer latencies in the range of 5–220 ns with an endurance rate around \(10^{12}\) writes per bit [103].

Among existing PM devices, Intel Optane DCPMM has been the most prominent one. Great efforts have been made to integrate the device to the existing ecosystem. For example, new programming model with special CPU instructions (e.g., clwb, clflush, and clflushopt) were introduced to provide durability guarantees. In addition, new persistence domains were defined to better understand the persistence guarantees offered by these devices. Figure 1(a) shows the two persistence domain supported on Intel platforms. Asynchronous DRAM Refresh (ADR) domain specifies that all updates that reach the write pending queue in the integrated memory controller are guaranteed to be persisted in an event of system failure, whereas Enhanced ADR (eADR) domain specifies that all updates in the caches and memory controller are persisted in an event of system failure. Therefore, there is no need for developers to explicitly flush cachelines using special CPU instructions. Note that eADR is a relatively recent feature that may not be available in all platforms. Also, while eADR may provide stronger persistence guarantee at the hardware level, PM software still needs to be carefully designed to achieve desired high level properties and correctness guarantees (e.g., atomicity of a transaction) [34]. These efforts have inspired the development of many new PM-optimized software [19, 61].

Most recently, vendor-neutral interfaces such as CXL have been pushing the evolution of PM devices further. For example, CXL 2.0 specification has included the support for PM devices [4, 91]. These CXL-based PM devices are expected to be compatible with the existing NVDIMM specifications, and they will still show up as special memory devices to the OS kernel. Therefore, although Intel is winding down the Optane DCPMM business with limited support in the next 3 to 5 years [29], the community is expecting to see CXL-based PM devices in the near future that are compatible with the NVDIMM ecosystem [4, 29, 34, 91].

To support PM devices in the storage system, multiple subsystems in the Linux kernel were modified. New Unified Extensible Firmware Interface specifications were introduced, and a new NVDIMM Firmware Interface Table driver was added to the OS kernel [16]. Moreover, to manage and access PM devices efficiently, the Linux kernel exposes the PM storage space over multiple abstractions. Specifically, the PM devices attached to NVDIMM can be configured into either interleaved or non-interleaved mode. In the interleaved mode (Figure 1(b)), an NVDIMM region is created over multiple NVDIMM devices; in the non-interleaved mode, each device has one NVDIMM region. Additionally, multiple namespaces may be created within one NVDIMM region, which is conceptually similar to creating multiple partitions on one block device.

The NVDIMM devices are registered as special memory devices in the Linux kernel. Each NVDIMM namespace maintains a dedicated memory area to store Page Frame Number (PFN) metadata, which supports the necessary address translation when accessing the storage media. Moreover, to bypass the page cache and enable direct access to PM, major file systems such as Ext4 and XFS have introduced the direct-access (DAX) feature based on the traditional Direct IO feature. As will be shown in our study, however, it is challenging to implement such PM-oriented optimizations correctly due to the system complexity.

All changes to the Linux kernel occur in the form of patches [23], including but not limited to bug fixes. We collect PM-related patches from the Linux source tree for study via four steps as follows:

First, we collected all patches committed to the Linux kernel between January 2011 and December 2021, which generated a dataset containing about 772,000 patches.

Second, to effectively identify PM-related patches, we refine the dataset using a wide set of PM-related keywords, such as “persistent memory,” “pmem,” “dax,” “ndctl,” “nvdimm,” “clflushopt,” and so on. The resulting dataset contains 3,050 patches. Note that this step is similar to the keyword search in previous studies [55, 80].

Third, to prune the potential noise, we refine the dataset further by manual examination. Each patch is analyzed at least twice by different researchers, and those irrelevant to PM are excluded based on our domain knowledge. The final dataset contains 1,553 PM-related patches in total.

Based on the 1,553 PM-related patches, we conduct a comprehensive study to answer four sets of questions as follows:

To answer these questions, we manually analyzed each patch in depth to understand its purpose and logic. The patches typically follow a standard format containing a description and code changes [23], which enables us to characterize them along multiple dimensions. For patches that contain limited information, we further looked into relevant source code and design documents. Moreover, we conduct experiments to validate the reproducibility of PM bugs and the capability of state-of-the-art bug detectors. We present our findings for the four sets of questions above in Section 4, Section 5, Section 6, and Section 7, respectively.

The results of our study should be interpreted with the method in mind. The dataset was refined via PM-related keywords and manual examination, which might be incomplete. Also, we only studied PM bugs that have been triggered and fixed in the mainline Linux kernel, which is biased: There might be other latent (potentially trickier) bugs not yet discovered. Nevertheless, we believe our study is one important step toward addressing the challenge. We release our results publicly to facilitate follow-up research [3].

Figure 2 shows the distribution of PM bug patches in the Linux kernel source tree. For clarity, we only show the major top-level directories in Linux, which represent major subsystems (e.g., “fs” for file systems and “mm” for memory management). In case a patch modifies multiple files across different directories (which is not uncommon as will be discussed in Section 5.4.1), we count it toward all directories involved. Therefore, the total count is larger than the number of PM bug patches.

We can see that “driver” is involved in most patches, which is consistent with previous studies [43]. In the PM context, this is largely due to the complexity of adding the nvdimm driver and the support for the CXL interface. Also, “fs” accounts for the second most patches, largely due to the complexity of adding DAX support for file systems [6]. The fact that PM bug patches involve many major kernel subsystems implies that we cannot only focus on one (e.g., “fs”) to address the challenge.

We also count the occurrences of individual files involved in the bug patches. Table 2 shows the top 10 most “buggy” files based on the occurrences and the average lines of code (LoC) changed per 100 LoC, which verifies that adding DAX and NVDIMM support are the two major sources of introducing PM bugs in the kernel. In addition, the table also shows a fine-grained view (i.e., individual files) on where the bugs exist within a subsystem. Therefore, applying bug detection techniques (e.g., static code analysis) on these specific files and/or code paths may help identify most issues.

To build reliable and secure PM systems, it is important to understand the types of bugs that occurred. We analyze the PM bug patches in depth and classify the bugs into five main types (Table 3) as follows: Hardware Dependent, Semantic, Concurrency, Memory, and Error Code. Each type includes multiple subtypes. The last column of Table 3 shows the major subsystems affected by each type of bugs. For clarity, the column only lists up to three subsystems for each type. We can see that the same type of bugs may affect multiple subsystems (e.g., “drivers,” “fs,” “mm”), and each subsystem may suffer from multiple types of bugs.

Figure 3 shows the relative percentages of the five main types. We can see that the Semantic type is dominant (49.7%), followed by the Hardware Dependent type (19.0%). Similarly, Figure 4 further shows the relative percentages of the subtypes within each main type. We elaborate on multiple representative types below based on these classifications, with an emphasis on the ones that are different from previous studies [78, 80].

To understand how severe the PM bugs are, we classify them based on the symptoms reported in the patches. We find that there are eight types of consequence, including Missing Device, Inaccessible Device, Security, Corruption, Crash, Hang, Wrong Return Value, and Resource Leak. We elaborate on the first four types as they are relatively more unique to our dataset, while the others are similar to previous studies [78].

First, Missing Device implies the kernel is unable to detect PM devices, which is often the consequence of hardware dependent bugs. For example, the e820_pmem driver is responsible for registering resources that surface as pmem ranges. However, the buggy “e820_pmem_probe” method may fail to register the pmem ranges into the System-Physical Address (SPA) space, which makes the PM device not recognizable by the kernel.

Second, Inaccessible Device means the PM device is detectable by the kernel but not accessible. For example, the “start_pad” variable was introduced in “struct nd_pfn_sb” of the nvdimm driver to record the padding size for aligning namespaces with the Linux memory hotplug section. But the buggy “nd_pfn_validate” method of the driver does not check for the variable, which leads to an alignment issue and makes the namespace not recognizable by the kernel.

Third, in terms of Security, we observe two interesting issues. In one case, write operations may be allowed on read-only DAX mappings, which exposes wrong access permissions to the end user. In another case, an NVDIMM’s security attribute remains in “unlocked” state even when the admin issues an “overwrite” operation, which could potentially allow malicious accesses to the PM device.

Fourth, Corruption means the data or metadata stored on PM devices are corrupted, which could lead to permanent data loss if there is no additional backup available on other systems. One special type of corruption is the Crash-Consistency issues, which means tricky corruptions (i.e., data or metadata inconsistencies) triggered by a crash event (e.g., power outage, kernel panic). Such issues have been investigated intensively in the literature due to its importance [85, 111, 113], and we have observed multiple crash-consistency issues in our dataset as well. For example, the routine dax_mapping_entry_mkclean may fail to clean and write protect DAX PMD entries; consequently, memory-mapped writes to DAX PMD pages may not be flushed to PM even when a sync operation is invoked. As a result, a crash may leave the system in an inconsistent state.

Multiple PM-specific bug detection tools have been proposed recently, including PMTest [77], XFDetector [76], and AGAMOTTO [88]. These tools mostly focus on user-level PM programs. We have performed bug detection experiments using these tools, and we are able to verify their effectiveness by reproducing most of the bug detection results reported in the literature. Unfortunately, we find that they are fundamentally limited for capturing the PM bugs in our dataset. For example, XFDetector [76] relies on Intel Pin [21], which can only instrument user-level programs. PMTest [77] can be applied to kernel modules, but it requires manual annotations, which is impractical for major kernel subsystems. AGAMOTTO [88] relies on KLEE [35] to symbolically explore user-level PM programs. While it is possible to integrate KLEE with virtual machines to enable full-stack symbolic execution (as in S2E [42]), novel PM-specific path reduction algorithms are likely needed to avoid the state explosion problem [67]. One recent work Jaaru [53] leverages commit stores, a common coding pattern in user-level PM programs, to reduce the number of execution paths that need to be explored for model checking. Nevertheless, such elegant pattern has not been observed in our dataset due to the complexity of kernel-level PM optimizations (Section 5). Therefore, additional innovations on path reduction are likely needed to apply model checking to detect diverse PM-related bugs in the kernel effectively.

Great efforts have been made to detect non-PM bugs in the kernel [13, 25, 27, 63, 66, 85, 99, 101]. For example, CrashMonkey [85] logs the bio requests and emulates crashed disk states to test the crash consistency of traditional file systems. As discussed in Section 5.3, such tricky crash consistency issues exist in PM subsystems, too. Nevertheless, extending CrashMonkey to detect PM bugs may require substantial modifications including tracking PM accesses and PM-critical instructions (e.g., mfence), designing PM-specific workloads, among others.

Similarly, fuzzing-based tools have proven to be effective for kernel bug detection [24, 63, 66, 99, 101]. For example, Syzkaller [24] is a kernel fuzzer that executes kernel code paths by randomizing inputs for various system calls and has been the foundation for building other fuzzers; Razzer [63] combines fuzzing with static analysis and detects data races in multiple kernel subsystems (e.g., “driver,” “fs,” and “mm”), which could potentially be extended to cover a large portion of concurrency PM bugs in our dataset. Since Syzkaller, Razzer, and similar fuzzers heavily rely on virtualized (e.g., QEMU [33]) or simplified (e.g., LKL [15]) environments to achieve high efficiency for kenel fuzzing, one common challenge and opportunity for extending them is to emulate PM devices and interfaces precisely to ensure the fidelity.

Also, Linux kernel developers have incorporated tools such as Kernel Address Sanitizer [25], Undefined Behavior Sanitizer [27], and memory leak detectors (Kmemcheck) [13] within the kernel code to detect various memory bugs (e.g., null pointers, use-after-free, resource leak). These sanitizers instrument the kernel code during compilation and examine bug patterns at runtime. Similarly to other dynamic tools, these tools can only detect issues on the executed code paths. In other words, their effectiveness heavily depends on the quality of the inputs. As discussed in Section 6, many PM issues in our dataset require specific configuration parameters from utilities (e.g., mkfs, ndctl) and workloads (e.g., mmap(MAP_SHARED), open(O_TRUNC)) to trigger, so we believe it is important to consider such PM-critical configurations when leveraging existing kernel sanitizers for detecting the issues exposed in our study.

As discussed above, while various bug detectors have been proposed and used in practice, addressing PM-related issues in our dataset will likely require PM-oriented innovations, including precise PM emulation, PM-specific configuration and workload support, and so on, which we leave as future work.

However, to better understand the feasibility of extending existing tools for PM bug detection, we present our efforts and results on extending one existing bug detector called Dr.Checker [82] in this section. We select Dr.Checker for three main reasons: First, it has proven effective for analyzing kernel-level drivers [82], and, as shown in Figure 2 (Section 5.1), drivers account for the majority of bugs in our dataset. second, Dr.Checker is based on static analysis without dynamic execution, which makes it less sensitive to the limitations discussed in the previous sections (e.g., device emulation, input generation). Third, Dr. Checker employs multiple bug detection algorithms that can detect multiple bug patterns identified in our study (Section 5.2). We name our extension as Dr.Checker+ and release it on Git to facilitate follow-up research on PM bug detection [3].

About Dr.Checker and Its Limitations. Dr.Checker [82] mainly uses two static analysis techniques (i.e., points-to and taint analysis) to detect memory-related bugs (e.g., buffer overflow) in generic Linux drivers. It performs flow-sensitive and context-sensitive code traversal to achieve high precision for driver code analysis. One special requirement for applying Dr.Checker is to identify correct entry points (i.e., functions invoking the driver code) as well as the argument types of entry points. For example, Figure 10 shows a VFS interface (“struct file_operations”) with function pointers that allow userspace programs to invoke operations on libnvdimm driver module. The functions included in the structure (e.g., “nvdimm_ioctl”) are entry points that enable us to manipulate the underlying NVDIMM devices through the driver code. The entry points and their argument types collectively determine the entry types, which in turn determines the taint sources for initiating relevant analysis. For example, the vanilla Dr. Checker describes an IOCTL entry type where the first argument of the entry point function is marked as PointerArgs (i.e., the argument points to a kernel location, which contains the tainted data) and the second argument is marked as TaintedArgs (i.e., the argument contains tainted data and is referenced directly). This entry type is applicable to entry point functions that have similar signature. In the case of the libnvdimm kernel module, the IOCTL entry type is applicable to two entry point functions (i.e., nd_ioctl, nvdimm_ioctl). In addition, Dr.Checker includes a number of detectors to check specific bug patterns based on the taint analysis (e.g., IntegerOverflowDetector and GlobalVariableRaceDetector).

While Dr.Checker has been applied to analyze a number of Linux device drivers [82], it does not support major PM drivers directly. Table 7 summarizes the seven PM driver modules involved in our study, including nfit.ko, nd_pmem.ko, nd_blk.ko, nd_btt.ko, libnvdimm.ko, device_dax.ko, and dax.ko. These driver modules provide various supports to make PM devices usable on Linux-based systems. For example, dax.ko provides generic support for direct access to PM, which is critical for implementing the DAX feature for Linux file systems including Ext4 and XFS. As discussed in Section 5 and Section 6, these PM drivers tend to contain the most PM bugs in our dataset, and many PM bugs require PM driver features to trigger (e.g., -o dax). Based on our study of the bug patterns and relevant source code, we find that these PM drivers have much more diverse entry types compared to the embedded drivers analyzed by the vanilla Dr.Checker [82]. As shown in the second to the last column of Table 7, the vanilla Dr.Checker [82] only supports one entry type (i.e., ioctl(file)) used in the libnvdimm module, leaving the majority of PM driver code unattended.

Extending Dr.Checker to Dr.Checker+. To make Dr.Checker works for the major PM drivers, we manually examine the source code of PM drivers and identify critical entry points. As summarized in Table 8, we have added five new entry types (i.e., DEV_OP, DAX_OP, BLK_OP, GETGEO, MOUNT) besides the original IOCTL entry type. The new entry types include the major entry point functions used in the PM driver modules. Moreover, we identify the critical input arguments and map them to the appropriate taint types defined by Dr.Checker (i.e., TaintedArgs for arguments directly passed by the userspace, and PointerArgs and TaintedArgsData for arguments pointing to a kernel memory location that contains tainted data). In addition, to make Dr.Checker work for newer versions of Linux kernel, we have ported the implementation to the latest version of the LLVM/Clang compiler infrastructure [26]. Overall, as summarized in the last column of Table 7, the enhanced Dr. Checker+ is able to support bug detection in the seven major PM driver modules based on our comprehensive study of the PM driver bugs and the relevant source code.

Experimental Results of Dr.Checker+. We have applied the extended Dr.Checker+ to analyze seven major PM kernel modules in Linux kernel v5.2. In this set of experiments, we applied four detectors: (1) IntegerOverflowDetector checks for tainted data used in operations (e.g., add, sub, or mul) that may cause an integer overflow or underflow, (2) TaintedPointerDereferenceChecker detects pointers that are tainted and directly dereferenced, (3) GlobalVariableRaceDetector checks for global variables that are accessed without a mutex, and (4) TaintedSizeDetector checks for tainted data that is used as a size argument in any of the copy_to_ or copy_from_ functions that may result in information leak or buffer overflows. Table 9 summarizes the experimental results. Overall, Dr. Checker+ can process all the target kernel modules successfully, and we have observed warnings (i.e., potential issues) reported by its four detectors in five of the seven kernel modules (i.e., nfit.ko, nd_pmem.ko, nd_btt.ko, libnvdimm.ko, and dax.ko). For example, the TaintedPointerDereferenceChecker was able to identify a potential null pointer dereference in the nd_btt driver module, where the pointer variable bdev in the btt_getgeo entry point of nd_btt was accessed without a check for its validity. If the routines invoking the entry point function do not check for the null value before using it, then this code may lead to a kernel crash.

Note that the warnings reported by Dr.Checker+ do not necessarily imply PM bugs due to the conservative static analysis used in Dr.Checker. For example, the GlobalVariableRaceDetector in Dr. Checker would falsely report a warning for any access to a global variable outside of a critical section. By looking into the warnings reported in our experiments, we observed a similar false alarm: Dr. Checker+ may report a warning when the driver code invokes the macro “WARN_ON” to print to the kernel error log, which is benign.

Also, since Dr.Checker’s detectors are stateless, they may report a warning for every occurrence of the same issue. For example, the page offset in pmem_dax_direct_access is used to calculate the physical address of a page, which involves bit manipulation operations and the resulting value may overflow the range of possible integer values. The IntegerOverflowDetector may report a warning whenever the method is invoked.

Overall, our experience is that extending Dr.Checker for analyzing PM-related issues in the Linux kernel is non-trivial and time-consuming due to the complexity of the PM subsystem. However, the actual code modification is minor: We only need to modify about 100LoC in Dr.Checker to cover the major PM driver modules. While the effectiveness is still fundamentally limited by the capability of the core techniques used in the existing tools (e.g., Dr.Checker’s static analysis may report false alarms), we believe that extending existing tools to make them work for the PM subsystem in the kernel can be an important first step toward addressing the PM-related challenges exposed in our study. We leave the investigation of improving Dr.Checker further (e.g., improving the static analysis algorithms, adding diagnosis support for understanding the root causes of warnings, or fixing the warnings detected) and building new detection tools as future work.

As briefly mentioned in previous sections, this work was inspired by many existing research efforts on studying the characteristics of bugs in real-world software systems, including bug patterns in multi-threaded applications [80], Linux and/or OpenBSD kernels [39, 43], (non-PM) file systems [78] and utilities [84], cryptographic software [68], cloud systems [54], and so on. Due to the complexity of real-world systems, it is practically impossible to build effective or efficient tools to address the issues induced by various bugs without thorough understanding of the bug characteristics. Therefore, such empirical studies of real-world bug patterns, which typically requires substantial manual efforts and domain knowledge, often serve as the foundation for understanding the issues and deriving practical solutions later. For example, the seminal work of S. Lu et al. [80] studied the concurrency bug patterns from four applications; while the study did not build any tools to address the bugs directly, it has inspired a series of follow-up research on concurrency bug detection and improving multi-threaded software in general [47, 50, 81, 94]. Similarly, the study of Linux file system patches by L. Lu et al. [78] has exposed general bug patterns in the context of file systems and has inspired various follow-up innovations on improving file systems reliability [51, 86]. One common lesson learned from these classic works is that “history repeats itself” and “learning from mistakes” is important for a better future [78, 80].

Our study follows a similar methodology (Section 3) but focuses on a dataset that has never been covered by existing works (to the best of our knowledge). As discussed in Section 5, we have identified a wide range of PM-related bug patterns, some of which are consistent to previous findings but others are not. For example, our study shows that cache-related issues only contribute to a small subset (1.7%) of the bugs in our dataset, and crash-consistency issues may be caused by semantic and concurrency bugs besides misusing low-level instructions. This observation is in contrast to recent research focus on PM bug detection in the community that mostly only consider crash-consistency issues caused by the misuse of cacheline flush instructions [53, 76, 77, 88]. In other words, additional research efforts are likely needed to address the PM-related issues in general.

As exemplified by previous studies, we envision multiple directions for follow-up research based on our empirical study of PM bug patches. First, the detailed characterisation of PM bug patterns (Section 5) may help derive concrete rules for building new PM bug detection tools, similar to our extension to Dr. Checker (Section 7.3). Second, the critical conditions identified in our reproducibility experiments (Section 6) may guide the design of future tools that need to consider how to trigger the bugs for bug detection or failure diagnosis. Third, the curated dataset of various types of PM bugs may serve as the test cases for evaluating the effectiveness of newly built tools, similarly to the concept of BugBench [79, 108]. We hope that by publishing our study results and releasing the dataset, the work can facilitate building the next generation of more robust PM-based storage systems.

As mentioned in Section 1 and Section 2, PM technologies are not limited to Intel Optane DCPMM [10], and the PM ecosystem in Linux is not solely designed for Intel. Various other PM technologies (e.g., PCM [95], STT-RAM [62], and CXL-based PM [34, 91]) will likely require similar support from the Linux kernel (e.g., the DAX feature). Therefore, our study of PM-related issues in the Linux kernel is not limited to Intel Optane technology either.

While Intel is winding down its Optane DCPMM business and will only provide limited support in the next 3 to 5 years [29], we expect to see other PM devices in the near future that will be largely compatible with the Linux PM ecosystem. Particularly, CXL [45] is a new open standard cache-coherent interconnect for processors, memory expansion, and accelerators. CXL 2.0 specification describes that PM devices can be realized using CXL.io and CXL.mem protocols [91]. Hence, it is expected that future PM devices would reside on CXL slots instead of memory DIMM slots. Although this change may impact how the host system recognizes PM devices, it is expected to have little impact on the PM software stack. For example, CXL-PM driver code in the Linux kernel (drivers/cxl/pmem.c) relies on existing NVDIMM driver code for functions such as register, un-register, and probe devices [74]. CXL-based PM devices would still appear as special memory devices (similarly to existing PM devices), and existing software interfaces (e.g., DAX) are expected to work without any modifications. Therefore, our study on the current Linux PM subsystem, which is the foundation for supporting the next generation of CXL-based PM devices, is still relevant. However, we expect that CXL-based devices will likely introduce new CXL-specific drivers and specifications, which will add to the complexity of Linux PM subsystem and may introduce new bug patterns. Therefore, additional studies will likely be needed in the future.