The document discusses accelerating software development for MIPI CSI-2 cameras by leveraging Linux resources. It describes using the V4L2 API and modular design approach with media controller and device tree to model hardware in a way that reduces effort and allows software to adapt to changing hardware configurations. Key aspects covered include defining relationships between devices and sub-devices to represent the hardware pipeline, and using the device tree to describe hardware details that drivers can access to configure themselves. This approach allows high-level interaction with different hardware through a standardized interface and modular design.

1. Accelerating Software
Development for MIPI
CSI-2℠ Cameras
Licinio Sousa
Synopsys, Inc.

2. The Challenge
Developing SW for Multiple HW Configs
• Nowadays HW platforms are always evolving
• It is hard for SW to keep up with the changes
• Using a monolithic approach might be simple to begin,
but harder to mantain
• To support a new device, the designer then needs to either split the
code or add support for another device in the initial code
• After a while the code will become “spaghetti code”
• Each device has specific configuration requirements and
characteristics, which complicates the tasks of providing support
and integration, due to limited code reusability
• Steep learning curve
2

3. The Challenge
Developing SW for Multiple HW Configs
3
Complete MIPI Camera and Display Solu6on

4. The Solution
Leverage Linux Resources to Reduce Design Efforts
• Systems with embedded Linux can leverage existing
libraries, APIs and driver frameworks to reduce effort
and accelerate SW development
• V4L2 API and modular design approach
• Media controller
• Device tree

5. Agenda
• Introduce a few Linux concepts and resources
• Share our experience
• Conclusion

6. Linux
# facts
- Linux is a free and open source software
~18 000 000 lines of code
- 5.64 patches accepted in mainline kernel / h
- More than 1000 engineers work in each release
# who is working in Linux
- Only 17.9 % make kernel hacking for free
- The other 82.1% is made by top tech players

7. Linux
Wearables Smartphones
Servers
Mainframes
Embedded systems

8. When is Linux’s Birthday?
8
From:torvalds@klaava.Helsinki.FI (Linus Benedict Torvalds)
Newsgroup: comp.os.minix
Subject: GCC-1.40 and a posix ques6on
Message-ID: 1991Jul13, 100050.9886@klaava.Helsinki.FI
Date: 3 Jul 91 10:00:50 GMT
Hello netlanders, Due a project I'm working on (in minix), I'm interested in the posix
standard defini6on. Could somebody please point me to a (preferably) machine-
readable format of the latest posix rules? Ftp-sites would be
nice.
Linus Torvalds torvalds@kruuna.helsinki.fi
From:torvalds@klaava.Helsinki.FI (Linus Benedict Torvalds)
Newsgroup: comp.os.minix
Subject: What would you like to see most in minix?
Summary: small poll for my new operaIng system
Message-ID: 1991Aug25, 20578.9541@klaava.Helsinki.FI
Date: 25 Aug 91 20:57:08 GMT
OrganizaIon: University of Helsinki.
Hello everybody out there using minix- I'm doing a (free) operaIng system (just a
hobby, won't be big and professional like gnu) for 386(486) AT clones.
(…)
Any suggesIons are welcome, but I won't promise I'll implement them :-)
Linus Torvalds torvalds@kruuna.helsinki.fi
From: torvalds@klaava.Helsinki.FI (Linus Benedict Torvalds)
Newsgroups: comp.os.minix
Subject: Free minix-like kernel sources for 386-AT
Message-ID: <1991Oct5.054106.4647@klaava.Helsinki.FI>
Date: 5 Oct 91 05:41:06 GMT
Organiza6on: University of Helsinki
Do you pine for the nice days of minix-1.1, when men were men and wrote their own
device drivers? Are you without a nice project and just dying
to cut your teeth on a OS you can try to modify for your needs? Are you finding it
frustra6ng when everything works on minix? No more all-
nighters to get a ni`y program working? Then this post might be just for you :-)
As I menIoned a month(?) ago, I'm working on a free version of a minix-lookalike
for AT-386 computers. It has finally reached the stage where it's even usable
(though may not be depending on what you want),
and I am willing to put out the sources for wider distribuIon. It is just version 0.02
(+1 (very small) patch already), but I've successfully run bash/gcc/gnu-make/gnu-
sed/compress etc under it. There are actually three
"birthdays" one can
idenIfy for Linux!!

9. Inside Linux – Architecture

10. User Space vs. Kernel Space
• User Space
• If SW is running in userspace it’s usually called an Application
• Memory area where application software will run
• Kernel Space
• Strictly reserved for running privileged OS kernel
• Runs kernel extensions and device drivers
• A driver has direct access to the HW

11. User-Kernel Interface
• User-kernel
interaction is made
via Linux APIs
• Linux provides APIs in
order to allow
programmers to write
applications
consistent with the
operating environment
• Some APIs can be
divided in 2 parts,
which run in two
different areas
• Kernel space
• User space

12. Video4Linux2 API
API for Video capture
• One of the Linux APIs is V4L (Video4Linux) a.k.a. V4L2
• V4L2 is an API for supporting real-time video capture in
Linux
• V4L2 is used to build drivers for many USB webcams,
TV tuners, smartphone cameras, and related devices
• With V4L2, applications can easily add video support in
a standard way

13. V4L2 API
Kernel Space
ApplicaIon Space
V4L2 Linux Drivers
V4L2 IOCTLs
mplayer ffmpeg vlc
V4L2 IOCTLs
Hardware
Hardware

14. V4L2 Devices and Subdevices
Modular Approach to Model the HW
• Hardware is often
made of several
integrated circuits that
need to interact with
each other
• This can lead to complex
drivers.
• The drivers must
reflect the hardware
model in software
• Modelling the different
hardware components as
software blocks called
sub-devices.
V4L2 device
V4L2 sub-device
V4L2 sub-device
V4L2 sub-device V4L2 sub-device

15. Media Controller
API to Define Relation Between Device/Sub-Devices
• Allows to discover a platform configuration and
configure it
• Devices are described as building blocks
• Connected through pads
• Pads can be either sinks or sources
• No limit for pads and sinks on each entity

16. Device/Sub-device and Media Entity
Source & Sink Pads to Represent Data Flow

17. Media Device - HW Pipeline
V4L2 Device/Sub-Devices Structured Using
MediaController API
Media
Device
Camera
CSI-2
Host
Colorspace
converter
Video
Device
/dev/video

18. Device Driver Interaction
Only High-Level Interaction is Needed
• The main driver will call the sub-device drivers to
• power on,
• start streaming,
• change video mode,
• and many more.
• All this without needing to know anything about the HW
it is talking to
Video
Device
/dev/video mplayer
Kernel Space
ApplicaIon Space

19. Device Tree
Describes the HW Configuration
• The format is expressive and
able to describe most board
design aspects including
• the number and type of CPUs
• base addresses and size of RAM
• busses and bridges
• peripheral device connections
• interrupt controllers and IRQ line
connections
• and others

20. Device Tree
Hardware Details Listed
• The HW specific details will be available in the Device
Tree
• HW addresses, IRQs and more are always available in
the Device Tree
• In the case of V4L2 cameras, information like number
of lanes, frequency and others is also available

21. V4L2, Media Controller and Device Tree
Putting it All Together
21

22. V4L2, Media Controller and Device Tree
Putting it All Together
• Device driver is designed using V4L2 API – API for
video capture
• It is modular and captures the HW elements – Sub-
devices
• Media controller API is used to define the relationship
between the device and sub-devices, forming a HW
pipeline that represents the actual HW platform
• Device tree describes the HW platform and individual
block specific details (address, interrupts, version,
number of lanes)
22

23. V4L2, Media Controller and Device Tree
Putting it All Together
• When kernel starts, it will look at the device tree and
configure it self
• The sub-device drivers will be loaded if present in the device tree
and will get the necessary details from it
• The device driver HW pipeline will be formed
• Application interacts with video device using high level
commands
• The video device is accessible under /dev
• The application will open /dev/videoX and interact using the high
level commands (such as change resolution, shutdown, power on,
start capture, etc.)
• Commands will be translated into low-level commands, specific to
each individual sub-device (CSI-2 controller, camera, etc.)
23

24. V4L2, Media Controller and Device Tree
Putting it All Together
• If a change in the platform is required (replace camera
or controller, etc.)
• The new individual sub-devices need to be added to the kernel
• The device tree needs to be updated and recompiled to reflect the
HW changes
• By re-starting the Kernel, the Device driver HW pipeline will be
updated automatically
24

25. OUR EXPERIENCE

26. CSI-2 SW Development
The Media Device HW Pipeline
26

27. CSI-2 SW Development
The Device Tree
Camera
PHY
Video DMA
CSI
Controller

28. CSI-2 SW Development
Simple to Change CSI-2 Camera
Camera

29. MIPI CSI-2 Host IP Prototyping Kit
• Synopsys offers an integrated HW/SW solution that
takes advantage of all these concepts
Visit Synopsys booth
to see demo!

30. Leverage Kernel Tools to Accelerate CSI-2
SW Development
• Application interacts with driver only using high level
Commands
• No need to use specific commands for each HW element
• The same kernel can serve for several HW platforms
by simple changing the Device Tree
• No effort in recompiling the kernel
• Modular code is easily split among team members
• Each driver just needs to provide standard interaction commands
• Easy to maintain and add support for new HW
All this accelerates driver development by reducing
the code complexity and turnaround time!
30

31. Thank You!
Synopsys’ broad porkolio of DesignWare MIPI IP soluIons
synopsys.com/mipi