Translated presentation slides of my talk about the FormulaPi Autonomous Robot Programming Challenge (Raspberry Pi based, summer and winter series 2017)

1. PROGRAMMING-
CHALLENGE
AUTONOMOUS CARS
Racing Series Formula Pi
https://www.formulapi.com
Thomas Effing, Bern Switzerland April 2018

2. MONSTER BORG
Image from www.FormulaPi.com
Buying Monsterborg: https://www.piborg.org/robots/monsterborg

3. LAMBDA.P.RACING
• λ.p.racing is my team name
• the name refers to the Lambda Calculus von Church
• I won both racing series Summer and Winter 2017!!
• they were the first two racing series with
MonsterBorg cars

5. RACING-RULES
• It is a pure software programming challenge
• all participants use the same cars.The cars have a Raspberry Pi 3 Computer with a
PiCam camera
• The races are taking place in England
• Before the race every participant uploads his racing code
• The racing code is written on the SD Card by the FormulaPi team. The SD cards goes
into the Raspberry Pi
• The car drives autonomously only input are the images of the camera
• MonsterBorg cars have 4 motors, 2 on both site parallel connected

6. RACING SERIES
• Summer 2017
• April bis July 2017
• 11deadlines ( 6 races inclusive final race, others test sessions)
• cars were not available, but there is a Java simulation
• Winter 2017
• November bis March 2018
• 9 deadlines (2 test runs , 3 normal races, 3 obstacle races und the final)
• obstacle race is a duel with 5 not moving obstacle cars
• in the final are the best 5 teams

7. DEMO SIMULATION

8. HOW DO I GETTOTHIS
CHALLENGE?
• I have started programming at the age of 12. My nephew is now
11 and has similar skills
• To start with programming I think to control something is quite a
good task
• Therefore I’ve searched for programmable cars and found the
Kickstarter campaign of the Monsterborg cars (the 3 before
closing the campaign)
• After that I saw the programming challenge

9. MY REQUIREMENTS FOR
PARTICIPATION
• My first look was what programming language has to be used
• There is base code in Python, but you can create a custom SD card
and then use whatever programming language you like! Great!!
• This was very important to me. I want to use the challenge to learn
a new programming language
• Idea was to use RUST or SWIFT
• The winning source code will be provided to the participants

10. JAVA
• I am the only one not using Python in the summer 2017 serie and I think it was the same in the
winter 2017 series
• For the first races in each series I did the programming in Python. In summer 2017 because the Java
implementation was not ready, in winter 2017 because I was not sure if I would continue with Java
• I won races with Python Code too
• Why Java?
• Stetting up RUST or SWIFT took too much time and is not that easy
• I’m familiar with the Java tooling
• I want a strong and statically typed language. It makes understanding the existing code and the
refactoring easier

11. WHY DID I WONTHIS?
• My interpretation why I won
• spend much time in log file analysis and video analysis
• Focusing on the main weak points of my current implementation
• Creative ideas, the main fun part of the whole challenge
• I understand the base code quite well, especially because I ported it to Java
• no fear of making bigger changes (which quite often didn't work out)
• Took every race as a testing field (this was quite risky, but just enough to get in the
finals)
• The use of Java was not a direct advantage

12. INITIAL CONCERNS
• With the need to process 30 images per second and the MonsterBorg moving
almost 1.5 meters per second I was concerned about the Java Garbage Collection
time
• How time-consuming is it to control the motors? I have no experience in doing
something like that
• result: GC is really not an issue.The GC times are irrelevant. I can process the images
at a quite constant rate of 9 ms
• result motor controller: it was really simple.They are conrolled via the I2C interface,
, I’ve used a library pi4j. I bought a PicoBorgRev car, because the controller is similar
to have something to test. MonsterBorgs are at that point not available

13. FIRST IDEAS
• Use a higher resolution for the images
• drive more the best racing line instead of following
the lines
• make a lot of images on the track at slow speed
I gave up all this ideas!

14. FIRST IDEAS
• Using higher resolution needs later to average over
the pixel values.And the processing takes longer
• The fastest lap is identical with the shortest. Just
take the inner line
• At slow speed there is less blurring so the images
are not realistic

15. THE FIRST TWO RACES WITH
JAVA CODE
• went very bad
• just starting with the 4th race I have competitive
code
• It has taken a lot of time to find the reason for the
problems

16.
DEMO
2ND RACE SUMMER 2017
https://youtu.be/buKcEEGg8z4?t=26m9s
Hint: 2nd from right. 5 seconds are enough. Straight to the wall. No lap
in 10 minutes

17. ROOT CAUSE?
• Observation: very high IOWait values in the operation system
• since then I always start SAR in every race (always monitoring IOWait, CPU and memory utilisation)
• at high IOWait values my Java Code isn’t processed at all!!
• For the 3rd race I assigned Java less memory.This mitigates the problem, but still no lap accomplished
• Updating everything, switch to Jessie Lite and OpenCV 3.2 (used for image processing)
• in total 3 SD cards sent to England
• Jessie Lite shows resource usages in a different way.This leads to finding the root cause of the problem!
• Memory Leak on the C side of OpenCV.!! A problem with the Java-C Bridge to OpenCV. The bridge is doing the memory
release of the C stuff in the Java finalize() method (not a good solution!)
• Java has enough memory resulting that the C stuff is not removed
• not very well documented
• Only solution: manual memory management of the OpenCV C stuff from Java. Not very nice

18. ORACLE JVM IS GOOD BUT …
• the JVM (Oracle Linux ARM 64 Hard Float ABI) on the Raspberry Pi is really good. I will use it in
the future for other stuff on the Pi
• There were disadvantages using Java:
• the bad java bridge to OpenCV.
• Frame rate can not be set.There is somewhere an implementation missing. (in v4l2?). I am
stuck at 30 images per second
• Java Array processing is quite slow. Python is faster with its NumPy. I need to do some tricks
and move a lot of stuff to OpenCV (it is implemented in C)
• porting to Python from Java was expensive (and I made a few errors)
• OpenCV needs manual memory management on the Java side. Python not

19. All images in the race have only 160
*120 pixels!
The PiCam can make images with 8-
megapixel but for me the
disadvantages were bigger than the
advantages.

20. Image with full driving speed are
quite blurry. But they are surprisingly
usable.This is a average quality image

21. ALGORITHM1
• an important step in the image processing is to
process the colours of the trace to pure colours
• the color changes are the lines the car is following
• the 5 lines are from left to right: red/blue, blue/red,
red/green, green/blue, blue/green

22. ALGORITHM 2
• at start the cars are in the WAIT_FOR_LIGHTS
mode.They are waiting for the 2nd green light
• After the start is is quite common to drive about
2 seconds in FIRST_STRAIGHT mode, full power
on all motors

23. ALGORITHM 3
• driving in the wrong direction (WRONG_WAY), is detectable
by a green/red line which normally (correct way) does not exist
• the cars can drive upside down (FLIPPED). Die motors are then
powered in the opposite direction
• the normal driving mode is to follow the lines
(FOLLOW_TRACK) done with a PID (proportional–integral–
derivative) controller (see z.B. https://www.youtube.com/watch?
v=JEpWlTl95Tw )

24. ALGORITHM 4
• at the start line is a dark red bar crossing the track.
This is for recognising a finished lap (NEW_LAP)
• a special situation is the mode CRASHED.We believe
then that there was a collision.
• Hint: still unsolved is to distinguish between a collision
with the wall from one with another car. This would
be very helpful!

25. CHALLENGES
• Essential for the success is to recognise the modes in a reliable way
• You have to work all the time with threshold values which must be well
adjusted. But there are still wrong detections
• Cause there a no further test runs after the initial ones I’ve introduced
new features quite carefully:
• not used all the time just randomly
• or just wrote values to the log to verify the feature first and activate
it in the next race

26. MADE MISTAKES
• The cam images were saved as JPG in the log directory.
But I did not understand why I later can’t reproduce the
calculated values. Of cause JPG is a lossy compresssion….
• image data are in bytes (8 bit). In Java there is no unsigned
data type.Therefore you must be carefully, for example
making comparisons.This mistake I did porting to Java:
instead of bytes[0] > 0 it should be
(bytes[0] & 0xff) > 0

27. MY JAVA SIGNED-BYTE MISTAKE
• top: original image of the race
(only lower half of the image)
• middle: faulty procession
• bottom: after removing the
unsigned hype problem
• hint:
• currently I do not make any
erode on the three colours.

28. R: 99
G: 88
B: 35
high red value resulting to wrong interpretations

29. FINAL SUMMER 2017
https://youtu.be/WwlhxiE3LpA?t=25m55s (the whole race)
https://youtu.be/WwlhxiE3LpA?t=27m5s (11 seconds)
hint for second link: after 11 seconds the lights go off!

30. CHALLENGE 3 WINTER 2017
https://youtu.be/4rOitCd_D3E?t=4m31s
(wrong way)
hint: right car. watch till 5:06.After that a lot of behaviour is repeating

31. FINAL WINTER 2018
https://youtu.be/8P2mM5XHjpM?t=16m34s
(avoidance!)
2nd from left. big crash. First to leave the crash and afterwards
managing the situation. First minute of video is enough

32. INNOVATIONS1
• FLIPPED:The upper and lower part of the image are compared. Normally
the variance is compared (track has less variance). I search for bright and
white image areas (indicating the top)
• image processing: current racing line is the line for which we find the most
points (colour changes).The image processing makes still too many
mistakes. (light condition a quite different on one lap). I have better
algorithms but they are still too slow.
• CRASHED: detection normally only if two consecutive images are not
different enough (stuck). Added crash if there is too much black at the
image button or the sides

33. INNOVATIONEN 2
• After power interruption drive immediately and don’t wait for the traffic light, it will
never come!This innovation was implemented first for the final Summer 2017.
Without I would have won this important race
• Dynamic laps: is a lap bad the next lap I use a different line. NEW_LAP recognition is
important here.There is a mirror effect resulting at erroneously thinking the black
wall is the red start line bar.Tried other implementations but they are still not better
• WRONG_WAY: one of the biggest problems of all participants. I think I have solved
the problem.The only innovation I came up with for the final winter 2017.Time
problems prevent an earlier implementation. Many ideas haven’t been efficiently
enough. green/red line was the only indicator. now also black/green and red/black

34. INNOVATIONEN 3
• OVERTAKE: Spent much time on this.This was necessary for the
obstacle races.There were sadly bad interactions with the CRASHED
mode.Therefore almost completely deactivated for the winter final.
Just breaking very shortly. Detection if there are many black pixels in
the lower middle rectangle

35. MACHINE LEARNING
• Still not considered
• possible problems (just speculating)
• not enough training data (images)
• not enough computing power with the
Raspberry Pi 3 in the race

36. PLANES FOR SUMMER 2018
(VERYTENTATIVE)
• complete rewrite / new program structure shifting
from being competitive to nice & clean code
• will not be in Java or Python
• porting this to Python as base for my nephew (11)
for a second team (lambda.q.racing)

37. NEW CARS 2019
• for 2019 the formula pi team plans new cars
• PiBorg Raptor (Pozyx GPS,
Beschleunigungssensoren, wheel speed encoder)
• siehe https://www.youtube.com/watch?
v=R8nce9c6Fi8