22 лютого відбувся Embedded Webinar #17 “Low-level Network Testing in Embedded Devices Development” від спікера Сергія Корнієнка. Під час вебінару ми говорили на такі теми: - Підхід до низькорівневого тестування мережевих протоколів; - Інструменти, які можна використати в реальних проєктах; - Знайдені баги та способи знаходження корневих причин на прикладі реального R&D проєкту. Відео та деталі заходу: https://bit.ly/embedded_webinar_17 Приєднатись до спільноти: https://www.facebook.com/groups/EmbeddedCommunity Відкриті Embedded-позиції у GlobalLogic: https://bit.ly/Embedded_Positions

1. Network Security in
Embedded devices
Serhii Korniienko - Embedded QA engineer

2. Examples of properties of typical
embedded computers when compared
with general-purpose counterparts are:
• low power consumption
• small size
• rugged operating ranges
• low per-unit cost
• often ‘non-standard‘ solution

3. Why low-level network testing is a
must in embedded:
• Often custom protocols used to reduce
CPU/RAM/ROM/bandwidth usage
• Low cost
• Timing properties of the program depend heavily
on its environment
• Environment =
+Process or & Memory Hierarchy
+ Operating System, other processes/threads, …
+ Network
+ I/O Devices
+…

4. Our experience:
1. Requirements
2. Approach
3. Tools
4. Test setup
5. Logs gathering and
analysis
6. Issues
7. Test cases
8. Process

5. 1. Our requirements:
• Low cost – cheapest HW components approved by
customer (BoM < 20$)
• Http and custom UDP support
• Stack: freeRTOS + LwIP
• Rigid requirements on synchronization jitter (PTP)
• IC network switch to build daisy-chained network
structure
• Stable work in heavy loaded network (1Gb
wireshark dump file per minute)

6. 2. Low –level testing approach:
• List of all low-level network stuff:
– Ports inventory:
– 80/tcp – http
– 554/tcp - rtsp
– 67/udp - dhcps
– 68/udp - dhcpc
– 319/udp - ptp-event
– 320/udp - ptp-general
– 2467/udp - custom protocol
– 17100/udp - custom protocol

7. L4 protocols inventory:
•ICMP
•UDP
•TCP
•IGMP

8. 3. Tools:
1. NMAP – enumeration and packet sending
- sS
- sU
- sO
- sX
- sN
- sF
- sX
- sA
- sY
- --scanflags

9. 3. Tools:
2. Ostinato –packet editing and traffic generation
- Set a value for any field of any protocol
- Open and edit PCAP files, replay and save back
- Support for the most common standard protocols
- Free, Libre, Open-Source Software

10. 3. Tools:
3. LOIC – open-source network stress testing
and denial-of-service attack application
- IC Switch throughput test

11. 4. Test setup

12. 5. Logs gathering and analysis:
The key thing is to gather and analyze all available info from all
system components (switch and microcontroller) to recognize
and eliminate bottlenecks and issues:
- Buffer overflows and underflows
- Queue overflows and underflows
- Protocol/link state changes
- Tx/Rx error counters
- Switch counters
- Error messages
- Power dropdown
- Unauthorized recording
- Voltage levels
- …

13. 6. Issues found:
Issue Web server goes
down during polite
TCP scan
Discovered/
Investigated
NMAP scan
Root cause LwIP configuration issue

14. Issues found:
Issue Discovered/
investigated
Root cause
Hard-fault occurs if send
fragmented UDP-packet
NMAP/
Wireshark/
Ostinato
LwIP incorrectly reassemble
fragmented UDP packets

15. Issues found:
Issue Discovered/
investigated
Root
cause
Device can be shutdown by specially crafted
control message
Script/fuzzing Input
validation
issue
Device can turn off it’s power supply and network chip power supply ☺

16. Issues found:
Issue Device hard-fault if send
abnormally large or small
message on control port -
Discovered/
investigated
LOIC/
Script/fuzzing
Root cause Input validation issue

17. Issues found:
Issue Discovered/
investigated
Root cause
Sending short ptp packet cause ptp
recalibration and packet loss
NMAP scan
Ostinato
LwIP issue

18. Issues found:
Issue Discovered/
investigated
Root cause
Network link goes down and Up
during nmap scan
NMAP/
Wireshark/
Ostinato
PSU issue for new IC
switch
Network link goes down during
nmap scan
NMAP/
Wireshark/
Ostinato
unauthorized command
on switch management
interface

19. Issues found:
Issue Discovered/
investigated
Root
cause
Device become unreachable if send short UDP
packet on port 319/320
NMAP scan
Ostinato
IC switch
issue
All devices in chain become unavailable if perform
UDP flood on port 319/320 of last device in chain
LOIC/Ostinato IC switch
issue

20. 7. Test cases:
Fuzzing!

21. Test cases:

22. Test cases:

23. 8. Process:
• Perform low-level testing to
every hardware changes
• Preform low-level testing for
every configuration changes
• Preform low-level testing for
every low-level structures
changes
• Preform low-level testing
periodically

24. Conclusion:
If you still hesitate about
the necessity to perform
low-level network testing
– go to the issues found
chapter!