PiDRAM: A Holistic End-to-end FPGA-based Framework for Processing-in-DRAM

DRAM-based main memory is organized hierarchically. Figure 1 (left) depicts this organization. A processor is connected to one or more memory channels (DDRx in the figure) ➊. Each channel has its own command, address, and data buses. Multiple memory modules can be plugged into a single channel. Each module contains several DRAM chips ➋. Each chip contains multiple DRAM banks that can be accessed independently ➌. Data transfers between DRAM memory modules and processors occur at cache block granularity. The cache block size is typically 64 bytes in current systems.

Inside a DRAM bank, DRAM cells are laid out as a two-dimensional array of wordlines (i.e., DRAM rows) and bitlines (i.e., DRAM columns) ➍. Wordlines are depicted in blue and bitlines are depicted in red in Figure 1. Wordline drivers drive the wordlines and sense amplifers read the values on the bitlines. A DRAM cell is connected to a bitline via an access transistor ➎. When enabled, an access transistor allows charge to flow between a DRAM cell and the cell’s bitline.

DRAM Operation. When all DRAM rows in a bank are closed, DRAM bitlines are precharged to a reference voltage level of \(\frac{V_{DD}}{2}\) . The memory controller sends an activate (ACT) command to the DRAM module to drive a DRAM wordline (i.e., enable a DRAM row). Enabling a DRAM row starts the charge sharing process. Each DRAM cell connected to the DRAM row starts sharing its charge with its bitline. This causes the bitline voltage to deviate from \(\frac{V_{DD}}{2}\) (i.e., the charge in the cell perturbs the bitline voltage). The sense amplifier senses the deviation in the bitline and amplifies the voltage of the bitline either to \(V_{DD}\) or to 0. As such, an ACT command copies one DRAM row to the sense amplifiers (i.e., row buffer). The memory controller can send READ/WRITE commands to transfer data from/to the sense amplifier array. Once the memory controller needs to access another DRAM row, the memory controller can close the enabled DRAM row by sending a precharge (PRE) command on the command bus. The PRE command first disconnects DRAM cells from their bitlines by disabling the enabled wordline and then precharges the bitlines to \(\frac{V_{DD}}{2}\) .

DRAM Timing Parameters. DRAM datasheets specify a set of timing parameters that define the minimum time window between valid combinations of DRAM commands [26, 27, 81, 97]. The memory controller must wait for tRCD, tRAS, and tRP nanoseconds between successive ACT \(\rightarrow\) RD, ACT \(\rightarrow\) ACT, and PRE \(\rightarrow\) ACT commands, respectively (Figure 1, right). Prior works show that these timing parameters can be violated (e.g., successive ACT \(\rightarrow\) RD commands may be issued with a shorter time window than tRCD) to improve DRAM access latency [26, 27, 81, 96, 97], implement physical unclonable functions [13, 82, 126], generate true random numbers [83, 123, 124], copy data [44, 145], and perform bitwise AND/OR operations [44, 142, 146, 147, 148] in commodity DRAM devices.

DRAM Internal Address Mapping. DRAM manufacturers use DRAM-internal address mapping schemes [30, 89, 130] to translate from logical (e.g., row, bank, column) DRAM addresses that are used by the memory controller to physical DRAM addresses that are internal to the DRAM chip (e.g., the physical position of a DRAM row within the chip). These schemes allow (i) post-manufacturing row repair techniques to map erroneous DRAM rows to redundant DRAM rows and (ii) DRAM manufacturers to organize DRAM internals in a cost-efficient and reliable way [76, 158]. DRAM-internal address mapping schemes can be substantially different across different DRAM chips [15, 30, 63, 70, 75, 76, 77, 88, 96, 110, 127, 129, 130, 141]. Thus, consecutive logical DRAM row addresses might not point to physical DRAM rows in the same subarray.

Prior work proposes a variety of in-DRAM computation mechanisms (i.e., PuM techniques) that (i) have great potential to improve system performance and energy efficiency [6, 28, 40, 54, 144, 145, 146, 147, 148, 149, 150] or (ii) can provide low-cost security primitives [13, 14, 82, 83, 124, 126]. A subset of these in-DRAM computation mechanisms are demonstrated on real DRAM chips [13, 44, 82, 83, 124, 126]. We describe the major relevant prior works briefly:

RowClone [145] is a low-cost DRAM architecture that can perform bulk data movement operations (e.g., copy, initialization) inside DRAM chips at high performance and low energy.

Ambit [144, 146, 147, 149, 150] is a new DRAM substrate that can perform (i) bitwise majority (and thus bitwise AND/OR) operations across three DRAM rows by simultaneously activating three DRAM rows and (ii) bitwise NOT operations on a DRAM row using 2-transistor 1-capacitor DRAM cells [72, 112].

ComputeDRAM [44] demonstrates in-DRAM copy (previously proposed by RowClone [145]) and bitwise AND/OR operations (previously proposed by Ambit [147]) on real DDR3 chips. ComputeDRAM performs in-DRAM operations by issuing carefully engineered, valid sequences of DRAM commands with violated tRAS and tRP timing parameters (i.e., by not obeying manufacturer-recommended timing parameters defined in DRAM chip specifications [116]). By issuing command sequences with violated timing parameters, ComputeDRAM activates two or three DRAM rows in a DRAM bank in quick succession (i.e., performs two or three row activations). ComputeDRAM leverages (i) two row activations to transfer data between two DRAM rows and (ii) three row activations to perform the majority function in real unmodified DRAM chips.

D-RaNGe [83] is a state-of-the-art high-throughput DRAM-based true random number generation technique. D-RaNGe leverages the randomness in DRAM activation (tRCD) failures as its entropy source. D-RaNGe extracts random bits from DRAM cells that fail with \(50\%\) probability when accessed with a reduced (i.e., violated) tRCD. D-RaNGe demonstrates high-quality true random number generation on a vast number of real DRAM chips across multiple generations.

QUAC-TRNG [124] demonstrates that four DRAM rows can be activated in a quick succession using an ACT-PRE-ACT command sequence (called QUAC) with violated tRAS and tRP timing parameters in real DDR4 DRAM chips. QUAC-TRNG uses QUAC to generate true random numbers at high throughput and low latency.

PiDRAM comprises two key hardware components. Both of these components are designed with the goal to provide a flexible and easy to use framework for evaluating PuM techniques.

❶ PuM Operations Controller. POC decodes and executes PiDRAM instructions that are used by the programmer to perform PuM operations. POC communicates with the rest of the system over two well-defined interfaces. First, it communicates with the CPU over a memory-mapped interface, where the CPU can send data to or receive data from POC using memory store and load instructions. The CPU accesses the memory-mapped registers (instruction, data, and flag registers) in POC to execute in-DRAM operations. This improves the portability of the framework and facilitates porting the framework to systems that employ different instruction set architectures. Second, POC communicates with the memory controller to perform PuM operations in the DRAM chip over a simple hardware interface. To do so, POC (i) requests the memory controller to perform a PuM operation, (ii) waits until the memory controller performs the operation, and (iii) receives the result of the PuM operation from the memory controller. The CPU can read the result of the operation by executing load instructions that target the data register in POC.

❷ Custom Memory Controller. PiDRAM’s memory controller provides an easy-to-extend basis for commodity DRAM-based PuM techniques that require issuing DRAM commands with violated timing parameters [13, 44, 82, 83, 124]. The memory controller is designed modularly and requires easy-to-make modifications to its scheduler to implement new PuM techniques. For instance, RowClone operations (Section 5) is enabled in just 60 lines of Verilog code on top of the baseline custom memory controller’s scheduler that implements conventional DRAM operations (e.g., read, write).

The custom memory controller employs three key sub-modules to facilitate the implementation of new PuM techniques. (i) The Periodic Operations Module periodically issues DDR3 refresh [117] and interface maintenance commands [52]. (ii) A simple DDR3 Command Scheduler supports conventional DRAM operations (e.g., activate, precharge, read, and write). This scheduler applies an open-bank policy (i.e., DRAM banks are left open following a DRAM row activation) to exploit temporal locality in memory accesses to the DRAM module. LOAD/STORE memory requests are simply handled by the command scheduler in a latency-optimized way. Thus, new modules that are implemented to provide new PuM functionality (e.g., a state machine that controls the execution of a new PuM operation) in the custom memory controller do not compromise the performance of LOAD/STORE memory requests.

(iii) The Configuration Register File (CRF) comprises 16 user-programmable registers that store the violated timing parameters used for DDRx sequences that trigger PuM operations (e.g., activation latency used in generating true random numbers using D-RaNGe [83], see Section 6) and miscellaneous parameters for PuM implementations (e.g., true random number generation period for D-RaNGe, see Section 6). In our implementation, CRF stores only the timing parameters used for performing PuM operations (e.g., RowClone and D-RaNGe). We do not store every standard DDRx timing parameter (i.e., non-violated, which are used exactly as defined as in DRAM chip specifications) in the CRF. Instead these timings are embedded in the command scheduler.

PiDRAM comprises two key software components that complement and control PiDRAM’s hardware components to provide a flexible and easy to use end-to-end PuM framework.

❸ PuM Operations Library (pumolib). This extensible library allows system designers to implement software support for PuM techniques. Pumolib contains customizable functions that interface with POC to perform PuM operations in real unmodified DRAM chips. The customizable functions hide the hardware implementation details of PuM techniques from software developers (that use pimolib). For example, although we expose PuM techniques to software via memory LOAD/STORE operations (POC is exposed as a memory-mapped module, Section 4.1), PuM techniques can also be exposed via specialized instructions provided by ISA extensions. Pumolib hides such implementation details from the user of the library and contributes to the modular design of the framework.

We implement a general protocol that defines how programmers express the information required to execute PuM operations to the POC. A typical function in pumolib performs a PuM operation in four steps: It (i) writes a PiDRAM instruction to the POC’s instruction register, (ii) sets the Start flag in POC’s flag register, (iii) waits for the POC to set the Ack flag in POC’s flag register, and (iv) reads the result of the PuM operation from POC’s data register (e.g., the true random number after performing an in-DRAM true random number generation operation, Section 6). We list the currently implemented pumolib functions in Table 1.

❹ Custom Supervisor Software. PiDRAM provides a custom supervisor software that implements the necessary OS primitives (i.e., virtual memory management, memory allocation, and alignment) for end-to-end implementation of PuM techniques. This facilitates developing end-to-end integration of PuM techniques in the system as these techniques require modifications across the software stack. For example, integrating RowClone end-to-end in the full system requires a new memory allocation mechanism (Section 5.1) that can satisfy the memory allocation constraints of RowClone [145]. Thus, we implement the necessary functions and data structures in the custom supervisor software to implement an allocation mechanism that satisfies RowClone’s constraints. This allows PiDRAM to be extended easily to implement support for new PuM techniques that share similar memory allocation constraints (e.g., Ambit [147], SIMDRAM [54], and QUAC-TRNG [124], as shown in Table 2).

We describe the general workflow for a PiDRAM operation (e.g., RowClone-Copy [145]) in Figure 3 over an example copy_row() function that is called by the user to perform a RowClone-Copy operation in DRAM.

The user makes a system call to the custom supervisor software ① that in turn calls the copy_row(source, destination) function in the pumolib ②. The function executes two store instructions in the RISC-V core ③. The first store instruction updates the instruction register with the copy_row instruction (that performs a RowClone-Copy operation in DRAM) ④ and the second store instruction sets the Start flag in the flag register to logic-1 ⑤ in POC. When the Start flag is set, POC instructs the PiDRAM memory controller to perform a RowClone-Copy operation using violated timing parameters ⑥. The POC waits until the memory controller starts executing the operation, after which it sets the Start flag to logic-0 and the Ack flag to logic-1 ⑦, indicating that it started the execution of the PuM operation. The PiDRAM memory controller performs the RowClone-Copy operation by issuing a set of DRAM commands with violated timing parameters ⑧. When the last DRAM command is issued, the memory controller sets the Finish flag (denoted as Fin. in Figure 3) in the flag register to logic-1 ⑨, indicating the end of execution for the last PuM operation that the memory controller acknowledged. The copy function periodically checks either the Ack or the Finish flag in the flag register (depending on a user-supplied argument) by executing load instructions that target the flag register ⑩. When the periodically checked flag is set, the copy function returns. This way, the copy function optionally blocks until the start (i.e., the Ack flag is set) or the end (i.e., the Finish flag is set) of the execution of the PuM operation (in this example, RowClone-Copy).²

Beyond commodity DRAM-based PuM techniques [13, 44, 82, 83, 123], which PiDRAM can be used to study, many prior works propose minor modifications to DRAM arrays to enable various arithmetic [6, 32, 40, 54] and bitwise operations [6, 144, 147, 148, 150] and security primitives [126].

These PuM techniques share common memory allocation and coherence requirements (Section 5.1). PiDRAM facilitates developing new mechanisms that satisfy these requirements. Table 2 describes some of the PuM case studies PiDRAM can enable.³

Other than providing an easy-to-use basis for end-to-end implementations of commodity DRAM-based PuM techniques, PiDRAM can be easily extended with a programmable microprocessor placed near the memory controller to study system integration challenges of PnM techniques (e.g., efficient pointer chasing [57, 58, 66], general-purpose compute [157], machine learning [74, 84, 94, 101, 122], databases [21, 22, 99], and graph processing [16]).

We develop a prototype of the PiDRAM framework on an FPGA-based platform. We use the Xilinx ZC706 FPGA board [166] to interface with real DDR3 modules. Xilinx provides a DDR3 PHY IP [163] that exposes a low-level “DFI” interface [33] to the DDR3 module on the board. We use this interface to issue DRAM commmands to the DDR3 module. We use the existing RISC-V-based SoC generator, Rocket Chip [11], to generate the RISC-V hardware system. Our custom supervisor software extends the RISC-V Proxy Kernel [135] to support the necessary OS primitives on PiDRAM’s prototype.

Figure 4 shows our prototype. Simulation Infrastructure. To aid the users in testing the correctness of any modifications they make on top of PiDRAM, we provide the developers with a Verilog simulation environment that injects regular READ/WRITE commands and custom commands (e.g., update the CRF, perform RowClone-Copy, generate random numbers) to the memory controller. When used in conjunction with the Micron DDR3 Verilog model provided by Xilinx [163], the simulation environment can help the developers to easily understand if something unexpected is happening in their implementation (e.g., if timing parameters are violated).

Open Source Repository. We make PiDRAM freely available to the research community as open source software at https://github.com/CMU-SAFARI/PiDRAM. Our repository includes the full PiDRAM prototype that has RowClone (Section 5) and D-RaNGe (Section 6) implemented end-to-end on the RISC-V system.

Data Mapping. RowClone has four data mapping and alignment requirements that cannot be satisfied by current memory allocation mechanisms (e.g., malloc [106]). First, the source and destination operands (i.e., page (4-KiB)-sized arrays) of the copy operation must reside in the same DRAM subarray. We refer to this as the mapping requirement. Second, the source and destination operands must be aligned to DRAM rows. We refer to this as the alignment requirement. Third, the size of the copied data must be a multiple of the DRAM row size. The size constraint defines the granularity at which we can perform bulk-copy operations using RowClone. We refer to this as the granularity requirement. Fourth, RowClone must operate on up-to-date data that resides in main memory. Modern systems employ caches to exploit locality in memory accesses and reduce memory latency. Thus, cache blocks (typically 64 B) of either the source or the destination operands of the RowClone operation may have cache block copies present in the cache hierarchy. Before performing RowClone, the cached copies of pieces of both source and destination operands must be invalidated and written back to main memory. We refer to this as the memory coherence requirement.

We explain the data mapping and alignment requirements of RowClone using Figure 5(a). The operand Source 1 cannot be copied to the operand Target 1 as the operands do not satisfy the granularity requirement (➊). Performing such a copy operation would overwrite the remaining (i.e., non-Target 1) data in Target 1’s DRAM row with the remaining (i.e., non-Source 1) data in Source 1’s DRAM row. Source 2 cannot be copied to Target 2 as Target 2 is not aligned to its DRAM row (➋). Source 3 cannot be copied to Target 3, as these operands are not mapped to the same DRAM subarray (➌). In contrast, Source 4 can be copied to Target 4 using in-DRAM copy, because these operands are (i) mapped to the same DRAM subarray, (ii) aligned to their DRAM rows and (iii) occupy their rows completely (i.e., the operands have sizes equal to DRAM row size) (➍).

Computing systems employ various layers of address mappings that obfuscate the DRAM row-bank-column address mapping from the programmer [30, 61], which makes allocating source and target operands as depicted in Figure 5(a) (➍) difficult. Only the virtual addresses are exposed to the programmer. Without control over the virtual address to DRAM address mapping, the programmer cannot easily place data in a way that satisfies the mapping and alignment requirements of an in-DRAM copy operation.

We implement a new memory allocation mechanism that can perform memory allocation for RowClone (in-DRAM copy/initialization) operations. This mechanism enables page-granularity RowClone operations (i.e., a virtual page can be copied to another virtual page using RowClone) without introducing any changes to the programming model. Figure 5(b) depicts an overview of our memory allocation mechanism.

At a high level, our memory allocation mechanism (i) splits the source and destination operands into page-sized virtually addressed memory blocks, (ii) allocates two physical pages in different DRAM rows in the same DRAM subarray, (iii) assigns these physical pages to virtual pages that correspond to the source and destination memory blocks at the same index such that the source block can be copied to the destination block using RowClone. We repeat this process until we exhaust the page-sized memory blocks. As the mechanism processes subsequent page-sized memory blocks of the two operands, it allocates physical pages from a different DRAM bank to maximize bank-level parallelism in streaming accesses to these operands.

To overcome the mapping, alignment, and granularity problems, we implement our memory management mechanism in the custom supervisor software of PiDRAM. We expose the allocation mechanism using the alloc_align(N, ID) system call. The system call returns a pointer to a contiguous array of N bytes in the virtual address space (i.e., one operand). Multiple calls with the same ID to alloc_align(N, ID) place the allocated arrays in the same subarray in DRAM, such that they can be copied from one to another using RowClone. If N is too large such that it exceeds the size of available physical memory, then alloc_align fails and causes an exception. Our implementation of RowClone requires application developers to directly use alloc_align to allocate data instead of malloc and similar function calls.

The custom supervisor software maintains three key structures to make alloc_align() work: (i) Subarray Mapping Table (SAMT), (ii) Allocation ID Table (AIT), and (iii) Initializer Rows Table (IRT).

(1) Subarray Mapping Table. We use the SAMT to maintain a list of physical page addresses that point to DRAM rows that are in the same DRAM subarray. alloc_align() queries SAMT to find physical addresses that map to rows in one subarray.

SAMT contains the physical pages that point to DRAM rows in each subarray. SAMT is indexed using subarray identifiers in the range [0, number of subarrays). SAMT contains an entry for every subarray. An entry consists of two elements: (i) the number of free physical address tuples and (ii) a list of physical address tuples. Each tuple in the list contains two physical addresses that respectively point to the first and second halves of the same DRAM row. The list of tuples contains all the physical addresses that point to DRAM rows in the DRAM subarray indexed by the SAMT entry. We allocate free physical pages listed in an entry and assign them to the virtual pages (i.e., memory blocks) that make up the row-copy operands (i.e., arrays) allocated by alloc_align(). We slightly modify our high-level memory allocation mechanism to allow for two memory blocks (4 KiB virtually addressed pages) of an array to be placed in the same DRAM row, as the page size in our system is 4 KiB and the size of a DRAM row is 8 KiB. We call two memory blocks in the same operand that are placed in the same DRAM row sibling memory blocks (also called sibling pages). The parameter N of the alloc_align() call defines this relationship: We designate memory blocks that are precisely N/2 bytes apart as sibling memory blocks.

Finding the DRAM Rows in a Subarray. Finding the DRAM row addresses that belong to the same subarray is not straightforward due to DRAM-internal mapping schemes employed by DRAM manufacturers (Section 2.1). It is extremely difficult to learn which DRAM address (i.e., bank-row-column) is actually mapped to a physical location (e.g., a subarray) in the DRAM device, as these mappings are not exposed through publicly accessible datasheets or standard definitions [71, 116, 130]. We make the key observation that the entire mapping scheme need not be available to successfully perform RowClone operations.

We observe that for a set of {source, destination} DRAM row address pairs, RowClone operations repeatedly succeed with a 100% probability. We hypothesize that these pairs of DRAM row addresses are mapped to the same DRAM subarray. We identify these row address pairs by conducting a RowClone success rate experiment where we repeatedly perform RowClone operations between every source, destination row address pair in a DRAM bank. Our experiment works in three steps: We (i) initialize both the source and the destination row with random data, (ii) perform a RowClone operation from the source to the destination row, and (iii) compare the data in the destination row with the source row. RowClone success rate is calculated as the number of bits that differ between the source and destination rows’ data divided by the number of bits stored in a row (8 KiB in our prototype). If there is no difference between the source and the destination rows’ data (i.e., the RowClone success rate for the source and the destination row is 100%), then we infer that the RowClone operation was successful. We repeat the experiment for 1000 iterations for each row address pair and if every iteration is successful, we store the address pair in the SAMT, indicating that the row address pair is mapped to different rows in the same DRAM subarray.⁴

(2) Allocation ID Table. To keep track of different operands that are allocated by alloc_align using the same ID (used to place different arrays in the same subarray), we use the AIT. AIT entries are indexed by allocation IDs (the parameter ID of the alloc_align call). Each AIT entry stores a pointer to an SAMT entry. The SAMT entry pointed by the AIT entry contains the set of physical addresses that were allocated using the same allocation ID. AIT entries are used by the alloc_align function to find which DRAM subarray can be used to allocate DRAM rows from, such that the newly allocated array can be copied to other arrays allocated using the same ID.

(3) Initializer Rows Table. To find which row in a DRAM subarray can be used as the source operand in RowClone-Initialize operations, we maintain the IRT. The IRT is indexed using physical page numbers. RowCopy-Initialize operations query the IRT to obtain the physical address of the DRAM row initialized with zeros and that belongs to the same subarray as the destination operand (i.e., the DRAM row to be initialized with zeros).

Figure 6describes how alloc_align() works over an end-to-end example. Using the RowClone success rate experiment (described above), the custom supervisor software (CSS for short) finds the DRAM rows that are in the same subarray (➊) and initializes the SAMT. The programmer allocates two 128-KiB arrays, A and B, via alloc_align() using the same allocation id (0), with the intent to copy from A to B (➋). CSS allocates contiguous ranges of virtual addresses to A and B and then splits the virtual address ranges into page-sized memory blocks (➌). CSS assigns consecutive memory blocks to consecutive DRAM banks and accesses the AIT with the allocation id (➍) for each memory block. By accessing the AIT, CSS retrieves the subarray id that points to a SAMT entry. The SAMT entry corresponds to the subarray that contains the arrays that are allocated using the allocation id (➎). CSS accesses the SAMT entry to retrieve two physical addresses that point to the same DRAM row. CSS maps a memory block and its sibling memory block (i.e., the memory block that is N/2 bytes away from this memory block, where N is the size argument of the alloc_align() call) to these two physical addresses, such that they are mapped to the first and the second halves of the same DRAM row (➏). Once allocated, these physical addresses are pinned to main memory and cannot be swapped out to storage. Finally, CSS updates the page table with the physical addresses to map the memory blocks to the same DRAM row (➐).

Since memory instructions update the cached copies of data (Section 5.1), a naive implementation of RowClone can potentially operate on stale data, because cached copies of RowClone operands can be modified by CPU store instructions. Thus, we need to ensure memory coherence to prevent RowClone from operating on stale data.

We implement a new custom RISC-V instruction, called CLFLUSH, to flush dirty cache blocks to DRAM (RISC-V does not implement any cache management operations [160]) so as to ensure RowClone operates on up-to-date data. A CLFLUSH instruction flushes (invalidates) a physically addressed dirty (clean) cache block. CLFLUSH or other cache management operations with similar semantics are supported in X86 [68] and ARM architectures [10]. Thus, the CLFLUSH instruction (that we implement) provides a minimally invasive solution (i.e., it requires no changes to the specification of commercial ISAs) to the memory coherence problem. Before executing a RowClone Copy or Initialization operation (see Section 5.4), the custom supervisor software flushes (invalidates) the cache blocks of the source (destination) row of the RowClone operation using CLFLUSH.

We support the RowClone-Copy and RowClone-Initialize operations in our custom supervisor software via two functions: (i) RowClone-Copy, rcc(void *dest, void *src, int size) and (ii) RowClone-Initialize, rci(void* dest, int size). rcc copies size number of bytes in the virtual address space starting from the src memory address to the dest memory address. rci initializes size number of bytes in the virtual address space starting from the dest memory address. We expose rcc and rci to user-level programs using system calls defined in the custom supervisor software.

rcc (i) splits the source and destination operands into page-aligned, page-sized blocks, (ii) traverses the page table (Figure 6 ➑) to find the physical address of each block (i.e., the address of a DRAM row), (iii) flushes all cache blocks corresponding to the source operand and invalidates all cache blocks corresponding to the destination operand, and (iv) performs a RowClone operation from the source row to the destination row using pumolib’s copy_row() function.

rci (i) splits the destination operand into page-aligned, page-sized blocks, (ii) traverses the page table to find the physical address of the destination operand, (iii) queries the IRT (see Section 5.2) to obtain the physical address of the initializer row (i.e., source operand), (iv) invalidates the cache blocks corresponding to the destination operand, and (v) performs a RowClone operation from the initializer row to the destination row using using pumolib’s copy_row() function.

We evaluate our solutions for the challenges in implementing RowClone end-to-end on a real system using PiDRAM. We modify the custom memory controller to implement DRAM command sequences ( \(ACT\rightarrow {}PRE\rightarrow {}ACT\) ) to trigger RowClone operations. We set the \(tRAS\) and \(tRP\) parameters to 10 ns (below the manufacturer-recommended 37.5ns for tRAS and 13.5ns for tRP [117]).

We implement a simple version of D-RaNGe in PiDRAM. PiDRAM’s D-RaNGe controller collects true random numbers from four DRAM cells in the same DRAM cache block inside one DRAM bank. We implement the D-RaNGe controller within the Periodic Operations Module (Section 4.1). The D-RaNGe controller (i) periodically accesses a DRAM cache block with reduced tRCD, (ii) reads four of the TRNG DRAM cells in the cache block, (iii) stores the four bits read from the TRNG cells in a 1 KiB random number buffer. We reserve multiple configuration registers in the CRF to configure (i) the TRNG period (in nanoseconds) used by the D-RaNGe controller to periodically generate random numbers by accessing DRAM with reduced activation latency while the buffer is not full (the D-RaNGe controller accesses DRAM every TRNG period), (ii) the timing parameter ( \(tRCD\) ) used to induce activation latency failures, and (iii) the physical location (DRAM bank, row, column addresses, and bit offset within the DRAM column) of the TRNG cells to read. We implement two pumolib functions: (i) buf_size(), which returns the number of random words (4 bytes) available in the buffer, and (ii) rand_dram(), which returns one random word that is read from the buffer. The two functions first execute PiDRAM instructions in the POC that update the data register either with (i) the number of random words available (when buf_size() is called) or (ii) a random word read from the random number buffer (when rand_dram() is called). The two functions then access the data register using LOAD instructions to retrieve either the size of the random number buffer or a random number. The application developer reads true random numbers using these two functions in pumolib.

Random Cell Characterization. D-RaNGe requires the system designer to characterize the DRAM module for activation latency failures to find DRAM cells that fail with a 50% probability (i.e., randomly) when accessed with reduced \(tRCD\) . Following the methodology presented in Reference [83], the system designer can characterize a DRAM device or use an automated procedure to find cells that fail with a 50% probability. In PiDRAM, we implement reduced latency access to DRAM by (i) extending the scheduler of the custom memory controller and (ii) adding a pumolib function activation_failure(address) that induces a \(tRCD\) failure on the DRAM cache block at address.

Experimental Methodology. We run a microbenchmark to understand the effect of the TRNG period on true random number generation throughput observed by a program running on the Rocket core. The microbenchmark consists of a loop that (i) checks the availability of random numbers using buf_size() and (ii) reads a random number from the buffer using rand_dram(). We execute the microbenchmark until we read one million bytes of random numbers. Results. The D-RaNGe controller can perform reduced-latency accesses frequently, every 220 ns. Figure 9 depicts the TRNG throughput observed by the microbenchmark for TRNG periods in the range [220 ns, 1,000 ns] with increments of 10 ns. We observe that the TRNG throughput decreases from 8.30 Mb/s at 220 ns TRNG period to 1.90 Mb/s at 1,000 ns TRNG period. D-RaNGe [83] reports 25.2 Mb/s TRNG throughput using a single DRAM bank when there are four random cells in a cache block. PiDRAM’s D-RaNGe controller can be optimized to generate random numbers more frequently to match D-RaNGe’s observed maximum throughput.⁹ We leave such optimizations to PiDRAM’s D-RaNGe controller for future work.

Including the modifications to the custom memory controller and pumolib, implementing D-RaNGe and reduced-latency DRAM access requires an additional 190 lines of Verilog and 74 lines of C code over PiDRAM’s existing codebase. We conclude that our D-RaNGe implementation (i) provides a basis for PiDRAM developers to study end-to-end implementations of DRAM-based true random number generators and (ii) shows that PiDRAM’s hardware and software components facilitate the implementation of new commodity DRAM-based PuM techniques, specifically those that are related to security. Our reduced-latency DRAM access implementation provides a basis for other PuM techniques for security purposes, such as the DRAM-latency physical unclonable functions (DL-PUF [82]) and QUAC-TRNG [124] (Section 4.4). We leave further exploration on end-to-end implementations of D-RaNGe, DL-PUF, and QUAC-TRNG, as well as end-to-end analyses of the security benefits they provide using PiDRAM for future work.