- The document discusses porting HIGHMEM support to 32-bit RISC-V Linux to allow the kernel to access physical memory above 896MB. It involves deciding the memory layout, creating a PKMAP region for temporary mappings, allocating FIXMAP slots for kmap_atomic(), and setting up a page table for the PKMAP region. However, maintaining HIGHMEM comes with performance costs and some upstream developers prefer to avoid it on new architectures if possible.

1. Experience on porting
HIGHMEM to 32bit RISC-V Linux
Eric Lin
2020.8.2

2. About me
• 2016 ~ 2018 NCKU
– MS
• 2018.12 ~ 2020.7 Andes technology
– Software engineer in Linux kernel team
• Experience
– RISC-V Linux kernel
– Device driver
– U-BOOT

3. Outline
• Why need highmem?
• Porting highmem
• An end to high memory?

4. Why need high memory ?
• Earliest days, kernel maintained ′direct map ′ to map all physical memory in
kernel space.
– It easy for the kernel to manipulate any page in the system
• 32bit platform only have 4GB virtual address space
– To reduce TLB flush cost between kernel and user space.
– Split 4G address space to 1:3 ( 1GB => kernel, 3GB => user )
• With direct map, kernel can only map 1GB physical memory.
1G
3G
1G
0x00000000
0xC0000000
0xFFFFFFFF
Physical memory
User
kernel
(direct map)
va_pa_offset

5. Why need high memory? (cont.)
• Reserved ~896MB for linear mapping (direct map) => low memory
• > 896MB virtual address space :
– Temporary mapping
• PKMAP => kmap() 、VMALLOC => vmalloc()、vmap()
– Permanent mapping
• FIXMAP => DTB、early_ioremap()
(ZONE_NORMAL)
low mem
3G
1G
Physical memory > 1G
user
kernel
(ZONE_HIGHMEM)
high mem
linear mapping
( direct map )
896MB
0xC0000000
VMALLOC
PKMAP
FIXMAP
896MB
i386
va_pa_offset

6. Porting highmem
• Decide RV32 linux memory layout
– Refer other architecture (arm, x86, nds32)
– (option) Move VMALLOC、FIXMAP..etc after PAGE_OFFSET
• Leave user-process more address space.
• Add a PKMAP region in virtual address space
– for kmap()
• Temporary mapping for a single page
• alloc_page(__GFP_HIGHMEM) from ZONE_HIGHMEM
• Create a page table for pkmap
• Add memory slots in FIXMAP for kmap_atomic()
• Add architecture kmap() and kmap_atomic()

7. 7
• 64bit platform needn’t highmem => kernel have 128GB address space.
• After porting highmem, we would like RV32 linux memory layout as below:
RISC-V 5.6 Linux Memory layout
linear mapping
(direct map)
PKMAP
kernel
user 3G
VMALLOC
FIXMAP
Reserved
VMEMMAP
PCI_IO
linear mapping
(direct map)
kernel
user
VMALLOC
FIXMAP
VMEMMAP
PCI_IO
0xffffffe0_00000000
0xffffffff_ffffffff
0xffffffff
0xc0000000
0x00000000 0x00000000_00000000
128 GB
16777215 TB
PAGE_OFFSET
PAGE_OFFSET
RV64
RV32

8. Move memory layout
(arch/riscv/include/asm/pgtable.h)
+#define VMALLOC_SIZE (SZ_128M)
+/* Reserve 4MB from top of RAM to align with PGDIR_SIZE */
+#define VMALLOC_END (0xffc00000UL)
+#define VMALLOC_START (VMALLOC_END - VMALLOC_SIZE)
…..
51 #define VMEMMAP_END (VMALLOC_START - 1)
52 #define VMEMMAP_START (VMALLOC_START - VMEMMAP_SIZE)
+#ifdef CONFIG_HIGHMEM
+/* Set LOWMEM_END alignment with PGDIR_SIZE */
+#define LOWMEM_END (ALIGN_DOWN(PKMAP_BASE, SZ_4M))
+#define LOWMEM_SIZE (LOWMEM_END - PAGE_OFFSET)
+#endif /* CONFIG_HIGHMEM */
#define TASK_SIZE PAGE_OFFSET
---------------------------
(arch/riscv/include/asm/highmem.h)
+#define PKMAP_BASE (FIXADDR_START - SZ_2M)
linear mapping
(direct map)
PKMAP 2M
user 3G
VMALLOC 128M
FIXMAP 4M
Reserved 4M
VMEMMAP 16M
PCI_IO 16M
0xffffffff
0xc0000000
0x00000000
LOWMEM_END
• Locate VMALLOC_END and LOWMEM_END
• Must Reserved 4MB from top of RAM to
align with PGDIR_SIZE
• Add a new region for PKMAP
VMALLOC_END
TASK_SIZE

9. Porting highmem
setup_bootm()
– max_low_pfn => the end of low_memory
– max_pfn => the end of physical memory
(arch/riscv/mm/init.c)
150 void __init setup_bootmem(void)
151 {
…
188 #ifdef CONFIG_HIGHMEM
189 max_low_pfn = (PFN_DOWN(__pa(LOWMEM_END)));
190 max_pfn = PFN_DOWN(memblock_end_of_DRAM());
191 memblock_set_current_limit(__pa(LOWMEM_END));
192 #else
low mem
Physical memory
high mem
max_low_pfn
max_pfn
27 static void __init zone_sizes_init(void)
28 {
…….
34 #endif
35 max_zone_pfns[ZONE_NORMAL] = max_low_pfn;
36 #ifdef CONFIG_HIGHMEM
37 max_zone_pfns[ZONE_HIGHMEM] = max_pfn;
38 #endif
39 free_area_init_nodes(max_zone_pfns);
40 }

10. Porting highmem (cont.)
• Prepare a page table and set it to swapper_pg_dir
– swapper_pg_dir is a page directory pointer for kernel.
• 32 bit use 2 level page table (RISC-V)
pte entry
pgd
pmd (pkmap_p )
PAGE
swapper_pg_dir
Physical memory

11. Porting highmem (cont.)
+static void __init pkmap_init(void)
+{
.
+ /*
+ * Permanent kmaps:
+ */
+ vaddr = PKMAP_BASE;
+
+ pgd = swapper_pg_dir + pgd_index(vaddr);
+ p4d = p4d_offset(pgd, vaddr);
+ pud = pud_offset(p4d, vaddr);
+ pmd = pmd_offset(pud, vaddr);
+ pkmap_p = (pte_t *)__va(memblock_phys_alloc(PAGE_SIZE, PAGE_SIZE));
…….
+ memset(pkmap_p, 0, PAGE_SIZE);
+ pfn = PFN_DOWN(__pa(pkmap_p));
+ set_pmd(pmd, __pmd((pfn << _PAGE_PFN_SHIFT) |
+ pgprot_val(__pgprot(_PAGE_TABLE))));
+
+ /* Adjust pkmap page table base */
+ pkmap_page_table = pkmap_p + pte_index(vaddr);
start_kernel()
-> setup_arch
-> paging_init
->pkmap_init
• Add new function pkmap_init() for creating pkmap page table

12. Porting highmem (cont.)
//arch/riscv/include/asm/fixmap.h
enum fixed_addresses {
FIX_PTE,
FIX_PMD,
FIX_EARLYCON_MEM_BASE,
+#ifdef CONFIG_HIGHMEM
+ FIX_KMAP_RESERVED,
+ FIX_KMAP_BEGIN,
+ FIX_KMAP_END = FIX_KMAP_BEGIN + (KM_TYPE_NR * NR_CPUS),
+#endif
+ __end_of_fixed_addresses,
};
23 #define FIXADDR_TOP (PKMAP_BASE)
24 #define FIXADDR_SIZE ((__end_of_fixed_addresses) << PAGE_SHIFT)
25 #define FIXADDR_START (FIXADDR_TOP - FIXADDR_SIZE)
FIXADDR_TOP
FIXADDR_START
(__end_of_fixed_address)
FIX_KMAP_BEGIN
4K
4K
4K
4K
FIX_KMAP_END
kmap_atomic
• Add memory slots
in FIXMAP

13. After porting
• If success, you will see …

14. After porting
• If fail, you will see …
????

15. An end to high memory?
• Upstream my first kernel patch, but …
• Arnd Bergmann (Linaro) reply …
– I would much prefer to not see highmem added to new architectures
at all if possible, see https://lwn.net/Articles/813201/
• Weiner like to improve memory-reclaim performance
– Inode-cache shrinking vs. highmem
• Inodes, being kernel data structures => low memory
• page-cache pages => can be placed in high memory
• With a large number of one-byte files on a 7G machine, it invoke inode
shrinker to reclaim inode with populated page cache. It can drop gigabytes of
hot and active page cache.
• Linus Torvalds say …

16. Reference
• https://www.kernel.org/doc/Documentation/vm/highmem.txt
• https://lwn.net/Articles/813201/
• https://lkml.org/lkml/2020/4/2/253

17. END