The respective talk was held by Oleksandr Shevchenko (Senior Engineering Consultant, GlobalLogic) at GlobalLogic Lviv Embedded TechTalk #2 on May 23, 2018. Oleksandr presentation is about features of software architecture, which provides parallel work of Linux and operating system real-time on different cores of a single processor. The talk is also about the Linux mechanism, which allows to connect the processor cores after the boot process has finished, the so-called "CPU Hotplug".

1. CPU hotplug implementation for the
multicore system running RTOS and Linux
simultaneously on separate cores
Oleksandr Shevchenko
Senior Consultant, Engineering

2. Agenda
1. Running RTOS and Linux simultaneously on separate
cores
2. CPU hotplug implementation
3. Questions?

3. 3
Part 1
Running RTOS and Linux
simultaneously on separate cores

4. Kernel booting
Boot Sequence
NOR Flash
RTOS
Linux Kernel
(zImage)
RootFS
(rootfs.cpio.gz)
RAM
ATAGS

5. Kernel booting
Boot Sequence
NOR Flash
RTOS
Linux Kernel
(zImage)
RootFS
(rootfs.cpio.gz)
RAM
RTOS (core 0)
By 1-st stage bootloader
ATAGS

6. Kernel booting
Boot SequenceBoot Sequence
NOR Flash
Linux Kernel
(zImage)
RootFS
(rootfs.cpio.gz)
RAM
By RTOS
Linux Kernel
(zImage)
RootFS
(rootfs.cpio.gz)
ATAGS
ATAGS
RTOS
RTOS (core 0)

7. Kernel booting
Boot SequenceBoot SequenceBoot Sequence
NOR Flash
Linux Kernel
(zImage)
RootFS
(rootfs.cpio.gz)
RAM
Linux
Kernel
RootFS
(rootfs.cpio.gz)
RootFS
Uncompress and mount by Linux Kernel
ATAGS
Linux Kernel
(zImage)
Uncompressed by itself
ATAGS
RTOS
RTOS (core 0)
(core 1,2,3)

8. Kernel booting
Boot SequenceBoot SequenceBoot Sequence
NOR Flash
Linux Kernel
(zImage)
RootFS
(rootfs.cpio.gz)
RAM
Linux
Kernel
RootFS
ATAGS
User Space
Services and applications
RTOS
RTOS (core 0)
(core 1,2,3)

9. CPU Communication
The mutex registers can only be written when it is unused
(recognizable by reading a ‘0’). Mutexes can only be reset by writing a ‘0’.
Each ARM core should write an identifier to lock the mutex.
• ARM core 0: write ‘1’
• ARM core 1: write ‘2’
• ARM core 2: write ‘3’
• ARM core 3: write ‘4’
RTOS (core 0)
Linux Kernel
(core 2)
Linux Kernel
(core 3)
Linux Kernel
(core 1)
0 (Reset)Mutex register:

10. CPU Communication
The mutex registers can only be written when it is unused
(recognizable by reading a ‘0’). Mutexes can only be reset by writing a ‘0’.
Each ARM core should write an identifier to lock the mutex.
• ARM core 0: write ‘1’
• ARM core 1: write ‘2’
• ARM core 2: write ‘3’
• ARM core 3: write ‘4’
RTOS (core 0)
Linux Kernel
(core 2)
Linux Kernel
(core 3)
Linux Kernel
(core 1)
1 (Core 0)Mutex register:
Core 0 writes 1 to Mutex
Register and enters critical
section

11. CPU Communication
The mutex registers can only be written when it is unused
(recognizable by reading a ‘0’). Mutexes can only be reset by writing a ‘0’.
Each ARM core should write an identifier to lock the mutex.
• ARM core 0: write ‘1’
• ARM core 1: write ‘2’
• ARM core 2: write ‘3’
• ARM core 3: write ‘4’
RTOS (core 0)
Linux Kernel
(core 2)
Linux Kernel
(core 3)
Linux Kernel
(core 1)
0 (Reset)Mutex register:
Core 0 exits critical section by
writing 0 to Mutex Register

12. CPU Communication
The mutex registers can only be written when it is unused
(recognizable by reading a ‘0’). Mutexes can only be reset by writing a ‘0’.
Each ARM core should write an identifier to lock the mutex.
• ARM core 0: write ‘1’
• ARM core 1: write ‘2’
• ARM core 2: write ‘3’
• ARM core 3: write ‘4’
RTOS (core 0)
Linux Kernel
(core 2)
Linux Kernel
(core 3)
Linux Kernel
(core 1)
3 (Core 2)Mutex register:
Core 2 writes 3 to Mutex
Register and enters critical
section

13. CPU Communication
The mutex registers can only be written when it is unused
(recognizable by reading a ‘0’). Mutexes can only be reset by writing a ‘0’.
Each ARM core should write an identifier to lock the mutex.
• ARM core 0: write ‘1’
• ARM core 1: write ‘2’
• ARM core 2: write ‘3’
• ARM core 3: write ‘4’
RTOS (core 0)
Linux Kernel
(core 2)
Linux Kernel
(core 3)
Linux Kernel
(core 1)
0 (Reset)Mutex register:
Core 2 exits critical section by
writing 0 to Mutex Register

14. RTOS/Linux Device Driver Architecture
Physical Devices (Hardware)
Device Drivers
Virtual File System
Kernel Space
User Space
Application
Open/
Close
Read/
Write
IOCTL
Linux

15. RTOS/Linux Device Driver Architecture
Physical Devices (Hardware)
Device Drivers
Virtual File System
Kernel
Space
User
Space
Application
Open/
Close
Read/
Write
IOCTL
Linux
XAPI
RTOS
Application

16. RTOS/Linux Device Driver Architecture
Linux Driver file opsApplication
fd=open(“/dev/module_xyz”, …); modXyzOpen()
Private
data
handle
last_error
alloc
XAPI Core API
XAPI_XYZ_Open()
XAPI_XYZ_SetX()
XAPI_XYZ_GetY()
close(fd) modXyzClose()
free
XAPI_XYZ_Close()
XAPI_XYZ_Read()
XAPI_XYZ_Write()
modXyzIoctl()ioctl(fd, cmd, …)
User Space Kernel Space
modXyzRead()
modXyzWrite()
read(fd, …)
write(fd, …)

17. 17
Part 1 Summary
1. Introduce correct boot sequence.

18. 18
Part 1 Summary
1. Introduce correct boot sequence.
2. Provide inter-CPU synchronization mechanism.

19. 19
Part 1 Summary
1. Introduce correct boot sequence.
2. Provide inter-CPU synchronization mechanism.
3. Implement common Hardware Abstraction Level
(HAL) for RTOS and Linux.

20. 20
PART 2
CPU hotplug implementation

21. Decrease Response Time on Power On
RTOS (core 0)
Linux Kernel
(core 2)
Linux Kernel
(core 3)
RTOS (core 1)
1. Start 2 samples of RTOS on two cores.

22. RTOS (core 0)
Linux Kernel
(core 2)
Linux Kernel
(core 3)
1. Start 2 samples of RTOS on two cores.
2. Once time critical task is done – withdraw from one core.
RTOS (core 1)
Decrease Response Time on Power On

23. RTOS (core 0)
Linux Kernel
(core 2)
Linux Kernel
(core 3)
Idle (core 1)
1. Start 2 samples of RTOS on two cores.
2. Once time critical task is done – withdraw from one core.
Decrease Response Time on Power On

24. RTOS (core 0)
Linux Kernel
(core 2)
Linux Kernel
(core 3)
Idle (core 1)
1. Start 2 samples of RTOS on two cores.
2. Once time critical task is done – withdraw from one core.
3. Linux Hot-plugs the idle core
Decrease Response Time on Power On

25. RTOS (core 0)
Linux Kernel
(core 2)
Linux Kernel
(core 3)
1. Start 2 samples of RTOS on two cores.
2. Once time critical task is done – withdraw from one core.
3. Linux Hot-plugs the idle core
Linux Kernel
(core 1)
Decrease Response Time on Power On

26. Patching Linux Kernel SMP Init Code
main.c
kernel_init()
wakeup_secondary() is a place where the CPU state is changed
from IDLE to EXECUTE.
smp.c
smp_prepare_cpus()
arch/arm/mach-fujitsu/
smp-hd62x.c
platform_smp_prepare_cpus()
wakeup_secondary()

27. Patching Linux Kernel SMP Init Code

28. Patching Linux Kernel SMP Init Code

29. Patching Linux Kernel SMP Init Code
main.c
kernel_init()
__cpu_up() is the place where a new process spawn for the core #cpu
with fork_idle(cpu)
smp.c
smp_init()
cpu.c
cpu_up()
_cpu_up()
__cpu_up()

30. Patching Linux Kernel SMP Init Code
In __cpu_up() added functionality to skip CPU1 initialization during kernel
start-up.
+ static unsigned flag=0;
+
+ if ((cpu == 1) && (flag == 0))
+ {
+ printk(KERN_ERR "CPU%u: Skip initialization...n", cpu);
+ flag = 1;
+ return -EBUSY;
+ }
We should quit at this point during initialization phase and do not try to
spawn a new process for the core #1 with fork_idle(cpu).
We will do it later while hot-plugging.

31. Patching Linux Kernel SMP Init Code
Linux kernel already has built-in CPU core hot-plug functionality. To
activate CPU1 core hot-plug the following sysfs write command has
to be issued:
echo 1 > /sys/devices/system/cpu/cpu1/online
But this will only work correctly if the CPU core had been previously
initialized during the startup and then disabled by command
echo 0 > /sys/devices/system/cpu/cpu1/online

32. Patching Linux Kernel SMP Init Code
While writing to the cpu subsystem related sysfs file, the store_online() function from
src/drivers/base/cpu.c is called by kernel.
We added new wakeup_secondary() function to this file
+static void wakeup_secondary(void)
…
+ jump_address = virt_to_phys(fujitsu_secondary_startup);
+ writel(jump_address, BOOT_CPU_CONTROL_CPU1_JUMP);
+ control_value = __raw_readl(BOOT_CPU_CONTROL_CPU1_STATUS);
+ control_value &= ~BOOT_CPU_CONTROL_REQUEST_MASK;
+ control_value &= ~BOOT_CPU_CONTROL_STATUS_MASK;
+ control_value |= BOOT_CPU_CONTROL_REQUEST_EXECUTE;
+ writel(control_value, BOOT_CPU_CONTROL_CPU1_STATUS);
We call this function before cpu_up() function that brings the CPU core to Linux
world.
case '1':
+ wakeup_secondary();
ret = cpu_up(cpu->sysdev.id);

33. Patching Linux Kernel SMP Init Code
And of course before hot-plugging from Linux the RTOS should
withdraw from the core 1 and send it to POLLING state by writing
correct value to the CPU control register
BOOT_CPU_CONTROL_CPU1_STATUS

34. Check if it works
1. Run Linux console command:
# cat /sys/devices/system/cpu/online
0-2
2. Run CPU3 hot-plug command from Linux console:
# echo 1 > /sys/devices/system/cpu/cpu3/online
CPU3: Booted secondary processor
CPU3: thread -1, cpu 3, socket 0, mpidr 80000003
HD62: Set local_timer Event Freq: 198000kHz Mult: 850403525 Shift: 32
CPU3: Unknown IPI message 0x1Sadf
3. Make sure that Linux is now running on 4 cores:
# cat /sys/devices/system/cpu/online
0-3

35. Check if it works
Check the output of
#htop
before and after hotplugging.

36. 36
Part 2 Summary
1. Modify smp_prepare_cpus() to not put the target
CPU core to EXECUTE state.

37. 37
Part 2 Summary
1. Modify smp_prepare_cpus() to not put the target
CPU core to EXECUTE state.
2. Modify __cpu_up to not spawn a new process for
the target core with fork_idle(cpu).

38. 38
Part 2 Summary
1. Modify smp_prepare_cpus() to not put the target
CPU core to EXECUTE state.
2. Modify __cpu_up to not spawn a new process for
the target core with fork_idle(cpu).
3. Modify the RTOS code to put the target CPU core
to POLLING state on exit.

39. Questions?

40. Thank you!