Weitere ähnliche Inhalte
Ähnlich wie Real Time Systems (20)
Mehr von Anil Kumar Pugalia (20)
Kürzlich hochgeladen (20)
Real Time Systems
- 2. What to Expect?
W's of Real Time & RTOS?
Design Methodologies of RTOS
Linux → Real Time Linux
Various Latencies & Schedulers
The “Real Time Patch” Path
Real Time Applications
Real Time Kernel Debugging
Peek into a Co-Kernel
© 2010 Anil Kumar Pugalia <email@sarika-pugs.com> 2
All Rights Reserved.
- 3. Real Time Definition
Correctness depends on
Functional Accuracy, and
Timings of the Result produced
Examples
MPEG Decoder in your DVD player
Call Response in your Mobile Phone
© 2010 Anil Kumar Pugalia <email@sarika-pugs.com> 3
All Rights Reserved.
- 4. Real Time Types
Hard Real Time
Guaranteed to meet the response reqs
Examples: Defense Systems, Vehicle Control Systems,
Satellite Systems
Soft Real Time
Once in a while may not meet the response reqs
Examples: Multimedia Devices, VoIP, CE devices
An intermediate one: Firm Real Time
Check: What is our Desktop?
© 2010 Anil Kumar Pugalia <email@sarika-pugs.com> 4
All Rights Reserved.
- 5. Is Real Time needed?
Deadline of Applications
Reduces Quality or Unacceptable
External Device Interactions with Apps
Handling of Messages – Time-bound or not
Application Task Priority
Could it be needed higher than OS services
Expressing delays & timeouts
Is it needed to be fine-grained, say in usecs
© 2010 Anil Kumar Pugalia <email@sarika-pugs.com> 5
All Rights Reserved.
- 6. Real Time OS
RTOS Phylum
Non Real-time
Soft Real-time
Hard Real-time
Conflicting Requirements
Real Time vs Rich Features of GPOS
© 2010 Anil Kumar Pugalia <email@sarika-pugs.com> 6
All Rights Reserved.
- 7. Expected Features of an RTOS
RTOS Requirements
Strictly Enforced Task Priorities
Fine-grained Preemption
External Event Handling in Bounded time frame
Which entails
Predictable Low Interrupt Latency
Predictable Low Scheduler Latency
Priority Inheritance
But wanted with Rich Features of
Multi Tasking / Processing
Interprocess Communication & Synchronization
© 2010 Anil Kumar Pugalia <email@sarika-pugs.com> 7
All Rights Reserved.
- 8. RTOS Design Methodologies
Expand an RTOS
Make a GPOS Real Time capable
Linux → Real Time Linux (Separate Session)
The Co-Kernel Approach
RT Apps in User Space
Open Source Xenomai (Separate Session)
Open Source RTAI
RT Apps in Kernel Space
RTLinux from Windriver
© 2010 Anil Kumar Pugalia <email@sarika-pugs.com> 8
All Rights Reserved.
- 9. Expand an RTOS
Difficult to ensure Determinism
As all core capabilities must be fully preemptive
Drivers for Hardware becomes very complex
Existing Software cannot be used without changes
High Maintenance Costs for Developers and Customers
So, let's understand the other one w.r.t. Linux
© 2010 Anil Kumar Pugalia <email@sarika-pugs.com> 9
All Rights Reserved.
- 10. Linux → Real Time Linux
© 2010 Anil Kumar Pugalia <email@sarika-pugs.com> 10
All Rights Reserved.
- 11. Why Linux is non Real Time?
High Interrupt Latency
High Scheduler Latency
OS services with indeterministic timing behaviour
IPC, Memory Allocation, ...
Linux is in general undeterministic in its response
Virtual Memory, System Calls, …
Later two are more of a better Coding Principles
However, the first two are major Design Changes
Let's take a closer look
© 2010 Anil Kumar Pugalia <email@sarika-pugs.com> 11
All Rights Reserved.
- 12. Real Time related Concepts
Latency
Interrupt
Scheduling
Both are related to
Preemption
And Critical Sections
© 2010 Anil Kumar Pugalia <email@sarika-pugs.com> 12
All Rights Reserved.
- 13. Latency Visualization
Interrupt ISR Runs ISR Signals RT Process
Event RT Process Runs
Time
t0 t1 t2 t3
Interrupt Interrupt Scheduling
Latency Scheduling Latency
Interrupt to Process Latency
© 2010 Anil Kumar Pugalia <email@sarika-pugs.com> 13
All Rights Reserved.
- 14. Latency Details
Why Interrupt Latency?
Attributes of ISR
Real Time Process
Typically, high priority
Otherwise, anyway it is a problem
All 3 could potentially pose problems
For Hard Real Time
© 2010 Anil Kumar Pugalia <email@sarika-pugs.com> 14
All Rights Reserved.
- 15. Possible Solutions
Key Requirement
Schedule RT Process, within a given time
Possible Helpers
Remove / Reduce interrupt-disabled code
(critical sections)
Write ISRs really really minimal in execution
time
Real Time Apps with the highest priority
May be higher than interrupts, softirqs, etc
© 2010 Anil Kumar Pugalia <email@sarika-pugs.com> 15
All Rights Reserved.
- 16. Preemption Visualization
Time
User Space
System Call
Interface
Kernel Space
Process A
Process B
© 2010 Anil Kumar Pugalia <email@sarika-pugs.com> 16
All Rights Reserved.
- 17. Preemption Details
Higher Preemption makes it more Real
Time feasible
Higher Priority (RT Tasks) could be
scheduled in a more predictable time
Sources of Preemption Latency
Critical Section Processing
Interrupt Context Processing
© 2010 Anil Kumar Pugalia <email@sarika-pugs.com> 17
All Rights Reserved.
- 18. Preemption & Critical Sections
Higher Preemption at the cost of reducing Critical Sections
Isolate & Minimize the critical sections
Protect them by disabling preemption
spin_lock / spin_unlock; preempt_disable / preempt_enable
Preemption Models for reducing Critical Sections
Preempt only safe locations
Entry & Exit of system calls, Return from interrupt processing, …
Put explicit “allow preempt” code at these places
Overall not an excellent approach
Preempt everywhere except only the Critical Sections
lock-breaking mechanism
Getting achieved by instrumenting kernel for latency measurements
And by fixing the longest latency code paths
© 2010 Anil Kumar Pugalia <email@sarika-pugs.com> 18
All Rights Reserved.
- 19. SMP World & Critical Sections
Uniprocessor case needs Protection from
Interrupt Processing
Exception Processing
Multiprocessor with additional
Multiple Kernel Threads
Posed a bigger need than Real Time for
reduced Critical Sections
Real Time has gained benefits from SMP
development, as well
© 2010 Anil Kumar Pugalia <email@sarika-pugs.com> 19
All Rights Reserved.
- 20. Real Time Kernel Patch
Critical Section Control is a major step
We have achieved good Soft Real Time Performance
Since 2.6.12, in single-digit milliseconds
On reasonably fast x86 processor
But without solving the Interrupt Latency
Hard Real Time is not possible
Recent Developments have majorly focused on this aspect
A major Real Time Kernel Patch evolved
Maintained by Ingo Molnar
Available @ http://people.redhat.com/~mingo/realtime-preempt
Could be downloaded using say, ketchup -G 2.6.22.1-rt9
And takes care of many more issues
© 2010 Anil Kumar Pugalia <email@sarika-pugs.com> 20
All Rights Reserved.
- 21. W's & How's of RT Kernel Patch?
Lot's of Questions
What all does it contain?
How to configure it?
What should be used & when?
And many more
Before answering these, let's understand
Schedulers in main-stream Kernel
Preemption Support in main-stream Kernel
© 2010 Anil Kumar Pugalia <email@sarika-pugs.com> 21
All Rights Reserved.
- 22. Linux Schedulers
Provides multi-tasking capabilities by
Time Slicing
Preemption
Based on various task priorities
Specified by its scheduling policies
Understands the following execution instances
Kernel Thread
User Process
User Thread
Linux Basic Scheduling
Normal (SCHED_OTHER) – Fairness Scheduling
Other Advanced Scheduling supported
Round Robin (SCHED_RR)
FIFO (SCHED_FIFO)
All Schedulers are O(1)
© 2010 Anil Kumar Pugalia <email@sarika-pugs.com> 22
All Rights Reserved.
- 23. Linux Kernel Preemption Levels
None (PREEMPT_NONE)
No forced preemption
Overall throughput, on average, is good
Voluntary (PREEMPT_VOLUNTARY)
First stage of latency reduction
Explicit preemption points are placed at strategic locations
Standard (PREEMPT_DESKTOP)
Preemption enabled everywhere except within critical sections
Good for soft real-time applications, like audio, multimedia, …
Kernel Parameter: preempt
© 2010 Anil Kumar Pugalia <email@sarika-pugs.com> 23
All Rights Reserved.
- 24. Preemption Level from RT Patch
Complete (PREEMPT_RT)
Spin locks replaced with preembtable
mutexes
Preemption enabled everywhere except
where protected by preempt_disable()
Smoothes out variation in latency
Allows a low & predictable latency for time-
critical real-time applications
© 2010 Anil Kumar Pugalia <email@sarika-pugs.com> 24
All Rights Reserved.
- 25. Additional Real Time Features
ISRs as Kernel Tasks
CONFIG_PREEMPT_HARDIRQ
Schedulable & Preemptible
Priority Assignment as Required
Control Window: /proc/sys/kernel/hardirq_preemption
Kernel Parameter: hardirq-preempt
Preemptable Softirqs
CONFIG_PREEMPT_SOFTIRQ
All ran in the context of ksoftirqd → Now as individual threads
A proper Linux task configured for real time
Control Window: /proc/sys/kernel/softirq_preemption
Kernel Parameter: softirq-preempt
Certain Read-Copy-Update (RCU) sections preemptible
CONFIG_PREEMPT_RCU
RCU is synchronization primitive for frequently read data
© 2010 Anil Kumar Pugalia <email@sarika-pugs.com> 25
All Rights Reserved.
- 26. Creating a Real Time Process
#include <sched.h>
#define MY_RT_PRIO 1
int main(...) {
int old_policy;
struct sched_param params = { .sched_priority = MY_RT_PRIO };
...
old_policy = sched_getscheduler(0);
if (sched_setscheduler(0, SCHED_RR, ¶ms) == -1)
handle_error();
…
}
© 2010 Anil Kumar Pugalia <email@sarika-pugs.com> 26
All Rights Reserved.
- 27. RT Application: Do's & Don'ts
Have very few Real Time Processes
Keep the RT Applications Short & Efficient
Do not do heavy duty operations
Like Memory Allocation, etc
If essential, move to Initialization Sections
Check for the working of the needed OS
services
Especially, if any RT Application has priority higher
than that OS Service
Test for the Desired Results
© 2010 Anil Kumar Pugalia <email@sarika-pugs.com> 27
All Rights Reserved.
- 28. Debugging Real Time Kernel
Soft Lockup
CONFIG_DETECT_SOFTLOCKUP
Unsafe Preemption Use Logging
CONFIG_DEBUG_PREEMPT
Deadlock Conditions Detection & Reporting
CONFIG_DEBUG_DEADLOCK
Deadlocks due to semaphores & spinlocks
Runtime Control of Locking Mode
CONFIG_DEBUG_RT_LOCKING_MODE
Mutex back to Spinlock
© 2010 Anil Kumar Pugalia <email@sarika-pugs.com> 28
All Rights Reserved.
- 29. Tracing Real Time Kernel
Latency Tracing
CONFIG_LATENCY_TRACE
For data associated with last maximum preemption
latency
Kernel Window: /proc/latency_trace
Function Call Tracing
Inserts hooks to every function via gcc
Gives function level details through
Kernel Window: /proc/latency_trace
Soon getting outdated by Ftrace
© 2010 Anil Kumar Pugalia <email@sarika-pugs.com> 29
All Rights Reserved.
- 30. Further Tracing RT Kernel
High Priority Process Wakeup
CONFIG_WAKEUP_TIMING
CONFIG_WAKEUP_LATENCY_HIST
Or, Interrupt Off Timing
CONFIG_CRITICAL_IRQSOFF_TIMING
CONFIG_INTERRUPT_OFF_HIST
Kernel Windows
/proc/sys/kernel/preempt_max_latency
/proc/latency_hist/interrupt_off_latency/CPU0 (0-10000us)
Time spent with Preemption disabled (in critical sections)
CONFIG_CRITICAL_PREEMPT_TIMING
CONFIG_PREEMPT_OFF_HIST
© 2010 Anil Kumar Pugalia <email@sarika-pugs.com> 30
All Rights Reserved.
- 31. Real Time Linux
Myths & Realities
With all these understood
Is Linux really Hard Real Time?
Can it be used in all Real Time Embedded activities?
Can it be used in a Nuclear Reactor?
Answers
Can be used in most Real Time Embedded Applications, such as Robotics
But cannot be “one-size-fits-it-all” approach to Real Time
Is still a patch and hence not that extensibly tested
Not all Device Drivers have yet been verified as Real Time safe
More & more users (such as multimedia users), using the patch would verify more
drivers, as they are the users noticing longer latency delays caused by these devices
But even if they are, it is after all a GPOS
Harder to verify a lack of bugs
Could have potential bugs (from real time perspective)
© 2010 Anil Kumar Pugalia <email@sarika-pugs.com> 31
All Rights Reserved.
- 33. W's of Co-Kernel Approach
Do not convert the GPOS into an RTOS
Place a small Real Time Kernel besides it
On the same Hardware
This would be the Co-Kernel
Taking care of all Real Time needs
Non-RT tasks are done by the GPOS
So,
© 2010 Anil Kumar Pugalia <email@sarika-pugs.com> 33
All Rights Reserved.
- 34. What is a Co-Kernel?
Co-Kernel is a Subsystem
Integrated with the main Kernel
Handling the Real Time part
© 2010 Anil Kumar Pugalia <email@sarika-pugs.com> 34
All Rights Reserved.
- 35. Types of Co-Kernel Approaches
Two Prevalent Approaches
Support running of RT Apps in User Space
RT Apps to be embodied into Kernel Space
Our Focus of Discussion is the First One
With Xenomai in consideration
© 2010 Anil Kumar Pugalia <email@sarika-pugs.com> 35
All Rights Reserved.
- 36. Benefits of Xenomai
Co-Kernel or Core running under Kernel, licensed under GPL 2
Kernel Support
Both 2.4 & 2.6 for with & without MMU systems
Provides generic building blocks to implement RT APIs (skins)
RT API Support for
VxWorks, pSOS+, VRTX, uITRON, POSIX 1003.1b
User Space i/f libraries: LGPL 2.1
Architecture Support
PowerPC*, Blackfin, ARM, x86*
Excellent Resources
Documentation, Technical Articles, …
Website: http://www.xenomai.org
© 2010 Anil Kumar Pugalia <email@sarika-pugs.com> 36
All Rights Reserved.
- 37. Xenomai Architecture
User Space User Space User Space User Space
Application Application Application Application
Linux syscall interface
VxWorks pSOS VRTX POSIX
... Kernel-based
Skin Skin Skin Skin Applications
Abstract RTOS Core
SAL / HAL
Portability
Layers
I-Pipe
© 2010 Anil Kumar Pugalia <email@sarika-pugs.com> 37
All Rights Reserved.
- 39. How Xenomai Works?
Real Time Shadow
New Sets of System Calls
Sharing Kernel Features
By Domain Migration
Real Time Driver Model Mediation
© 2010 Anil Kumar Pugalia <email@sarika-pugs.com> 39
All Rights Reserved.
- 40. What all have we learnt?
W's of Real Time & RTOS?
Design Methodologies of RTOS
Linux → Real Time Linux
Various Latencies & Schedulers
The “Real Time Patch” Path
Real Time Applications
Real Time Kernel Debugging
Peek into a Co-Kernel
© 2010 Anil Kumar Pugalia <email@sarika-pugs.com> 40
All Rights Reserved.