This project is under development.

See the develop branch for this project for details.

https://github.com/ikwzm/uiomem/tree/develop

Currently 1.0.0-alpha.4 is tentatively released.

https://github.com/ikwzm/uiomem/tree/v1.0.0-alpha.4.

uiomem is a Linux device driver for accessing a memory area outside the Linux Kernel management from user space.

uiomem has following features.

• uiomem can enable CPU cache, so it can access memory at high speed.
• uiomem can manually invalidiate and flush the CPU cache.
• uiomem can be freely attached and detached from Linux Kernel.

It is possible to access memory from the user space by opneing the device file(e.g. /dev/uiomem0) and mapping to the user memory space, or using the read()/write() functions.

The start address and size of the allocated memory area can be specified when the device driver is loaded (e.g. when loaded via the insmod command). Some platforms allow to specify them in the device tree.

• OS : Linux Kernel Version 4.19, 5.4, 6.1 (the author tested on 5.4 and 6.1).
• CPU: ARMv7 Cortex-A9 (Xilinx ZYNQ / Altera CycloneV SoC)
• CPU: ARM64 Cortex-A53 (Xilinx ZYNQ UltraScale+ MPSoC)

The following Makefile is included in the repository.

# SPDX-License-Identifier: GPL-2.0 OR MIT
# Copyright (C) 2015-2023 Ichiro Kawazome

#
# For in kernel tree variables
# 
obj-$(CONFIG_UIOMEM) += uiomem.o

#
# For out of kernel tree variables
#
CONFIG_MODULES ?= CONFIG_UIOMEM=m

HOST_ARCH ?= $(shell uname -m | sed -e s/arm.*/arm/ -e s/aarch64.*/arm64/)
ARCH      ?= $(shell uname -m | sed -e s/arm.*/arm/ -e s/aarch64.*/arm64/)

ifeq ($(ARCH), arm)
 ifneq ($(HOST_ARCH), arm)
   CROSS_COMPILE ?= arm-linux-gnueabihf-
 endif
endif
ifeq ($(ARCH), arm64)
 ifneq ($(HOST_ARCH), arm64)
   CROSS_COMPILE ?= aarch64-linux-gnu-
 endif
endif

ifdef KERNEL_SRC
  KERNEL_SRC_DIR := $(KERNEL_SRC)
else
  KERNEL_SRC_DIR ?= /lib/modules/$(shell uname -r)/build
endif

#
# For out of kernel tree rules
#
all:
	$(MAKE) -C $(KERNEL_SRC_DIR) ARCH=$(ARCH) CROSS_COMPILE=$(CROSS_COMPILE) M=$(PWD) $(CONFIG_MODULES) modules

modules_install:
	$(MAKE) -C $(KERNEL_SRC_DIR) ARCH=$(ARCH) CROSS_COMPILE=$(CROSS_COMPILE) M=$(PWD) $(CONFIG_MODULES) modules_install

clean:
	$(MAKE) -C $(KERNEL_SRC_DIR) ARCH=$(ARCH) CROSS_COMPILE=$(CROSS_COMPILE) M=$(PWD) clean

Load the uiomem kernel driver using insmod. The start address and size of the allocated memory area should be provided as an argument as follows. The device driver is created, and allocates memory area with the specified address and size. The memory area that can be specified must be aligned with the page size. The maximum number of memory area that can be allocated using insmod is 8 (uiomem0/1/2/3/4/5/6/7).

shell$ sudo insmod uiomem.ko uiomem0_addr=0x0400000000 uiomem0_size=0x00040000
[  562.657246] uiomem uiomem0: driver version = 1.0.0-alpha.4
[  562.657264] uiomem uiomem0: major number   = 238
[  562.657270] uiomem uiomem0: minor number   = 0
[  562.657275] uiomem uiomem0: range address  = 0x0000000400000000
[  562.657282] uiomem uiomem0: range size     = 262144
[  562.657287] uiomem uiomem.0: driver installed.
shell$ ls -la /dev/uiomem0
crw------- 1 root root 238, 0 Oct 21 17:36 /dev/uiomem0

The module can be uninstalled by the rmmod command.

shell$ sudo rmmod uiomem
[  657.672544] uiomem uiomem.0: driver removed.

In addition to the allocation via the insmod command and its arguments, memory area can be allocated by specifying the reg property in the device tree file. When a device tree file contains an entry like the following, uiomem will allocate memory area and create device drivers when loaded by insmod.

		#address-cells = <2>;
		#size-cells    = <2>;
		uiomem_plmem {
			compatible = "ikwzm,uiomem";
			device-name = "uiomem0";
			minor-number = <0>;
			reg = <0x04 0x00000000 0x0 0x00040000>;
		};

shell$ sudo insmod uiomem.ko
[  773.889476] uiomem uiomem0: driver version = 1.0.0-alpha.4
[  773.889496] uiomem uiomem0: major number   = 237
[  773.889501] uiomem uiomem0: minor number   = 0
[  773.889506] uiomem uiomem0: range address  = 0x0000000400000000
[  773.889512] uiomem uiomem0: range size     = 262144
[  773.889518] uiomem 400000000.uiomem_plbram: driver installed.
shell$ ls -la /dev/uiomem0
crw------- 1 root root 237, 0 Oct 21 17:40 /dev/uiomem0

The following properties can be set in the device tree.

• compatible
• reg
• memory-region
• shareable
• minor-number
• device-name
• sync-offset
• sync-size
• sync-direction

The compatible property is used to set the corresponding device driver when loading uiomem. The compatible property is mandatory. Be sure to specify compatible property as "ikwzm,uiomem".

The reg property specifies the physical address and size. The reg property is used when uiomem allocates a buffer outside the management of Linux Kernel. The memory area that can be specified must be aligned with the page size. Either the reg property or the memory-region property is required.

		#address-cells = <2>;
		#size-cells = <2>;
		uiomem@0xFFFC0000 {
			compatible = "ikwzm,uiomem";
			reg = <0x0 0xFFFC0000 0x0 0x00040000>;
		};

The memory-region property specifies the memory region allocated for reserved memory. The memory region specified by the memory-region property must always have the no-map property specified. Either the reg property or the memory-region property is required.

		#address-cells = <2>;
		#size-cells = <2>;
		reserved_memory {
			ranges;
			image_buf0: image_buf@0 {
				no-map;
				reg = <0x0 0x70000000 0x0 0x10000000>;
				label = "image_buf0";
			};
		};
		uiomem@image_buf0 {
			compatible = "ikwzm,uiomem";
			memory-region = <&image_buf0>;
		};

The shareable property is specified when multiple uiomem shares the memory space specified by reg property.

		#address-cells = <2>;
		#size-cells = <2>;
		uiomem0 {
			compatible = "ikwzm,uiomem";
			reg = <0x0 0xFFFC0000 0x0 0x00040000>;
			shareable;
		};
		uiomem1 {
			compatible = "ikwzm,uiomem";
			reg = <0x0 0xFFFC0000 0x0 0x00040000>;
			shareable;
		};

The minor-number property is used to set the minor number. The valid minor number range is 0 to 255. A minor number provided as insmod argument will has higher precedence, and when definition in the device tree has colliding number, creation of the device defined in the device tree will fail.

The minor-number property is optional. When the minor-number property is not specified, uiomem automatically assigns an appropriate one.

		uiomem0 {
			compatible = "ikwzm,uiomem";
			minor-number = <0>;
			reg = <0x0 0xFFFC0000 0x0 0x00040000>;
		};

The device-name property is used to set the name of device.

The device-name property is optional. The device name is determined as follow:

1. If device-name property is specified, the value of device-name property is used.
2. If device-name property is not present, and if minor-number property is specified, sprintf("uiomem%d", minor-number) is used.

The sync-offset property is used to set the start of the buffer range when manually controlling the cache of uiomem.

The sync-offset property is optional. When the sync-offset property is not specified, sync-offset is set to <0>.

The sync-size property is used to set the size of the buffer range when manually controlling the cache of uiomem.

The sync-size property is optional. When the sync-size property is not specified, sync-size is set to ths size specified by the reg property.

The sync-direction property is used to set the direction of DMA when manually controlling the cache of uiomem

• sync-direction=<0>: Read and Write
• sync-direction=<1>: Write Only
• sync-direction=<2>: Read Only

The sync-direction property is optional. When the sync-direction property is not specified, sync-direction is set to <0>.

		uiomem0 {
			compatible = "ikwzm,uiomem";
			reg = <0x0 0xFFFC0000 0x0 0x00040000>;
			sync-offset = <0x00010000>;
			sync-size = <0x000F0000>;
			sync-direction = <2>;
		};

When uiomem is loaded into the kernel, the following device files are created. <device-name> is a placeholder for the device name described in the previous section.

• /dev/<device-name>
• /sys/class/uiomem/<device-name>/phys_addr
• /sys/class/uiomem/<device-name>/size
• /sys/class/uiomem/<device-name>/sync_offset
• /sys/class/uiomem/<device-name>/sync_size
• /sys/class/uiomem/<device-name>/sync_direction
• /sys/class/uiomem/<device-name>/sync_owner
• /sys/class/uiomem/<device-name>/sync_for_cpu
• /sys/class/uiomem/<device-name>/sync_for_device

/dev/<device-name> is used when mmap()-ed to the user space or accessed via read()/write().

    if ((fd  = open("/dev/uiomem0", O_RDWR)) != -1) {
        buf = mmap(NULL, buf_size, PROT_READ|PROT_WRITE, MAP_SHARED, fd, 0);
        /* Do some read/write access to buf */
        close(fd);
    }

The device file can be directly read/written by specifying the device as the target of dd in the shell.

shell$  dd if=/dev/urandom of=/dev/uiomem0 bs=4096 count=64
64+0 records in
64+0 records out
262144 bytes (262 kB, 256 KiB) copied, 0.00341051 s, 76.9 MB/s

shell$ dd if=/dev/uiomem0 of=random.bin bs=4096
64+0 records in
64+0 records out
262144 bytes (262 kB, 256 KiB) copied, 0.00192588 s, 136 MB/s

The physical address of a memory area can be retrieved by reading /sys/class/uiomem/<device-name>/phys_addr.

    unsigned char  attr[1024];
    unsigned long  phys_addr;
    if ((fd  = open("/sys/class/uiomem/uiomem0/phys_addr", O_RDONLY)) != -1) {
        read(fd, attr, 1024);
        sscanf(attr, "%x", &phys_addr);
        close(fd);
    }

The size of a memory area can be retrieved by reading /sys/class/uiomem/<device-name>/size.

    unsigned char  attr[1024];
    unsigned int   buf_size;
    if ((fd  = open("/sys/class/uiomem/uiomem0/size", O_RDONLY)) != -1) {
        read(fd, attr, 1024);
        sscanf(attr, "%d", &buf_size);
        close(fd);
    }

The device file /sys/class/uiomem/<device-name>/sync_offset is used to specify the start address of a memory block of which cache is manually managed.

    unsigned char  attr[1024];
    unsigned long  sync_offset = 0x00000000;
    if ((fd  = open("/sys/class/uiomem/uiomem0/sync_offset", O_WRONLY)) != -1) {
        sprintf(attr, "%d", sync_offset); /* or sprintf(attr, "0x%x", sync_offset); */
        write(fd, attr, strlen(attr));
        close(fd);
    }

The device file /sys/class/uiomem/<device-name>/sync_size is used to specify the size of a memory block of which cache is manually managed.

    unsigned char  attr[1024];
    unsigned long  sync_size = 1024;
    if ((fd  = open("/sys/class/uiomem/uiomem0/sync_size", O_WRONLY)) != -1) {
        sprintf(attr, "%d", sync_size); /* or sprintf(attr, "0x%x", sync_size); */
        write(fd, attr, strlen(attr));
        close(fd);
    }

The device file /sys/class/uiomem/<device-name>/sync_direction is used to set the direction(Read/Write) of memory area of which cache is manually managed.

• 0: sets Read and Write
• 1: sets Write Only
• 2: sets Read Only

    unsigned char  attr[1024];
    unsigned long  sync_direction = 1;
    if ((fd  = open("/sys/class/uiomem/uiomem0/sync_direction", O_WRONLY)) != -1) {
        sprintf(attr, "%d", sync_direction);
        write(fd, attr, strlen(attr));
        close(fd);
    }

The device file /sys/class/uiomem/<device-name>/sync_owner reports the owner of the memory block in the manual cache management mode. If this value is 1, the buffer is owned by the device. If this value is 0, the buffer is owned by the cpu.

    unsigned char  attr[1024];
    int sync_owner;
    if ((fd  = open("/sys/class/uiomem/uiomem0/sync_owner", O_RDONLY)) != -1) {
        read(fd, attr, 1024);
        sscanf(attr, "%x", &sync_owner);
        close(fd);
    }

In the manual cache management mode, CPU can be the owner of the buffer by writing non-zero to the device file /sys/class/uiomem/<device-name>/sync_for_cpu. This device file is write only.

If '1' is written to device file, if sync_direction is 2(=Read Only) or 0(=Read and Write), the write to the device file invalidates a cache specified by sync_offset and sync_size.

    unsigned char  attr[1024];
    unsigned long  sync_for_cpu = 1;
    if ((fd  = open("/sys/class/uiomem/uiomem0/sync_for_cpu", O_WRONLY)) != -1) {
        sprintf(attr, "%d", sync_for_cpu);
        write(fd, attr, strlen(attr));
        close(fd);
    }

The value written to this device file can include sync_offset, sync_size, and sync_direction.

    unsigned char  attr[1024];
    unsigned long  sync_offset    = 0;
    unsigned long  sync_size      = 0x10000;
    unsigned int   sync_direction = 1;
    unsigned long  sync_for_cpu   = 1;
    if ((fd  = open("/sys/class/uiomem/uiomem0/sync_for_cpu", O_WRONLY)) != -1) {
        sprintf(attr, "0x%08X%08X", (sync_offset & 0xFFFFFFFF), (sync_size & 0xFFFFFFF0) | (sync_direction << 2) | sync_for_cpu);
        write(fd, attr, strlen(attr));
        close(fd);
    }

The sync_offset/sync_size/sync_direction specified by sync_for_cpu is temporary and does not affect the sync_offset or sync_size or sync_direction device files.

In the manual cache management mode, DEVICE can be the owner of the buffer by writing non-zero to the device file /sys/class/uiomem/<device-name>/sync_for_device. This device file is write only.

If '1' is written to device file, if sync_direction is 1(=Write Only) or 0(=Read and Write), the write to the device file flushes a cache specified by sync_offset and sync_size (i.e. the cached data, if any, will be updated with data on DDR memory).

    unsigned char  attr[1024];
    unsigned long  sync_for_device = 1;
    if ((fd  = open("/sys/class/uiomem/uiomem0/sync_for_device", O_WRONLY)) != -1) {
        sprintf(attr, "%d", sync_for_device);
        write(fd, attr, strlen(attr));
        close(fd);
    }

The value written to this device file can include sync_offset, sync_size, and sync_direction.

    unsigned char  attr[1024];
    unsigned long  sync_offset     = 0;
    unsigned long  sync_size       = 0x10000;
    unsigned int   sync_direction  = 1;
    unsigned long  sync_for_device = 1;
    if ((fd  = open("/sys/class/uiomem/uiomem0/sync_for_device", O_WRONLY)) != -1) {
        sprintf(attr, "0x%08X%08X", (sync_offset & 0xFFFFFFFF), (sync_size & 0xFFFFFFF0) | (sync_direction << 2) | sync_for_device);
        write(fd, attr, strlen(attr));
        close(fd);
    }

The sync_offset/sync_size/sync_direction specified by sync_for_device is temporary and does not affect the sync_offset or sync_size or sync_direction device files.

• https://github.com/ikwzm/PLBRAM-Ultra96
• https://github.com/ikwzm/PLBRAM-ZYBO-Z7