The ‘complex camera’ user-space library is a fairly new and hot topic in the Linux kernel video community. The issue is debated and targets a real technical debt of Linux systems. The Linux kernel video input subsystem and its APIs changed greatly in the last years to keep up with the increasing complexity and processing power embedded in modern SoCs. Namely, the biggest game-changing feature introduced already 5 years ago is the media controller subsystem and its pad oriented APIs, that allows composing pipelines of processing blocks to model the acquisition and re-processing of video and images. While most of the media subsystem drivers in mainline Linux have been ported to fully support media-controller, the real missing component is now user space support to automate setting up and controlling the image processing pipelines. The Video4Linux community is now tackling the issue by implementing support for “complex camera” systems by providing a user space library that aims to support the most recent use cases represented by mobile consumer devices and high end laptops. This talks provides an overview of the in-kernel APIs of modern media-controller capable video device drivers, their userspace APIs and the challenges the currently in-development library has to face. It also aims to provide to both driver and application developers an overview of the most modern implementation of the Linux video input stack architecture, that will likely be found in most system in next years.

1. Complex Cameras are complex
Jacopo Mondi
Linux Lab 2018
Jacopo Mondi - Linux Lab 2018 Complex Cameras are complex (1/ 65)

2. Hello, I’m Jacopo
Embedded Linux software engineer
I mostly work on integrating cameras and
multimedia devices
I (mostly) work with Renesas Electronics mainline
kernel team
jacopo@jmondi.org
irc: jmondi freenode.net
Jacopo Mondi - Linux Lab 2018 Complex Cameras are complex (1/ 65)

3. Motivations
Rapidly increasing complexity of image capture
process is moving from the embedded and mobile
space to the more canonical personal computing
space (say, laptops).
Are Linux systems ready for this?
Jacopo Mondi - Linux Lab 2018 Complex Cameras are complex (2/ 65)

4. Motivations
Rapidly increasing complexity of image capture
process is moving from the embedded and mobile
space to the more canonical personal computing
space (say... laptops).
Are Linux systems ready for this?
No they’re not
(I wouldn’t be giving this presentation otherwise :p )
Jacopo Mondi - Linux Lab 2018 Complex Cameras are complex (3/ 65)

5. Complex Cameras are complex
Image acquisition is complex
Image capture
Smart sensors
Smart(er) ISPs
The media controller
The media graph
Handling pipelines
Intel’s IPU3
Libcamera
Jacopo Mondi - Linux Lab 2018 Complex Cameras are complex (4/ 65)

6. Image Capture
The image acquisition process is a pipeline of digital
signals processing blocks, that spans from the
application processor to external components
connected through a data and a control bus
Jacopo Mondi - Linux Lab 2018 Complex Cameras are complex (5/ 65)

7. Image Capture: Sensors
Sensors
grid of photo-detectors (pixels)
measure intensity of photo-electric properties
filters: each pixel transports a single color
information
Red Green Blue
Jacopo Mondi - Linux Lab 2018 Complex Cameras are complex (6/ 65)

8. Image Capture: Raw Bayer
Bayer Patterns
disposition of color filters
BGGR - RGBG - GRBG - RGGB
50% Green - 25% Red - 25% Blue
Jacopo Mondi - Linux Lab 2018 Complex Cameras are complex (7/ 65)

9. Image Capture: Raw Bayer
Bayer Patterns
disposition of color filters
BGGR - RGBG - GRBG - RGGB
50% Green - 25% Red - 25% Blues
The full pixel color has to be reconstructed
demosaicing or de-bayering
Jacopo Mondi - Linux Lab 2018 Complex Cameras are complex (8/ 65)

10. Image Capture: Raw Bayer
It’s a tough job!
interpolation of colors from neighbour pixels
computational intensive problems
Figure: From: http://www.cambridgeincolour.com
Jacopo Mondi - Linux Lab 2018 Complex Cameras are complex (9/ 65)

11. Image Capture: Color Encoding
The RAW digital values have to be interpreted and
encoded
RAW usually converted to commonly used
colorspaces as sRGB, Adobe RGB, CYMK
from a known colorspace, pixels get color encoded
to ease transportation
media bus formats: RGB565, RGB666, YUYV, VYUY
easier to transport, store and manipulate
More computations!
Jacopo Mondi - Linux Lab 2018 Complex Cameras are complex (10/ 65)

12. Image Capture: Feedback loop
What about 3A, black level calibration, enhancements
etc ?
feedback loop on captured images
even more computationally expensive
often based on user application inputs
Application
SoC
Sensor
Jacopo Mondi - Linux Lab 2018 Complex Cameras are complex (11/ 65)

13. Image Capture: Feedback loop
What about 3A, black level calibration, enhancements
etc ?
feedback loop on captured images
even more computationally expensive
often based on user application inputs
Application
SoC
Sensor
Jacopo Mondi - Linux Lab 2018 Complex Cameras are complex (12/ 65)

14. Image Capture: Feedback loop
What about 3A, black level calibration, enhancements
etc ?
feedback loop on captured images
even more computationally expensive
often based on user application inputs
Application
SoC
Sensor
Jacopo Mondi - Linux Lab 2018 Complex Cameras are complex (13/ 65)

15. Image Capture: Feedback loop
What about 3A, black level calibration, enhancements
etc ?
feedback loop on captured images
even more computationally expensive
often based on user application inputs
Application
SoC
Sensor
Jacopo Mondi - Linux Lab 2018 Complex Cameras are complex (14/ 65)

16. Image Capture: Feedback loop
What about 3A, black level calibration, enhancements
etc ?
feedback loop on captured images
even more computationally expensive
often based on user application inputs
Application
SoC
Sensor
Jacopo Mondi - Linux Lab 2018 Complex Cameras are complex (15/ 65)

17. Image Capture: Summary
Digital imaging is a computational expensive process!
Jacopo Mondi - Linux Lab 2018 Complex Cameras are complex (16/ 65)

18. Image Capture: Summary
Digital imaging is a computational expensive process!
complexity in the image acquisition
de-bayering, enhancing, pixel encoding
Jacopo Mondi - Linux Lab 2018 Complex Cameras are complex (17/ 65)

19. Image Capture: Summary
Digital imaging is a computational expensive process!
complexity in the image acquisition
de-bayering, enhancing, pixel encoding
complexity in image quality tuning
3A parameters computation, tuning
Jacopo Mondi - Linux Lab 2018 Complex Cameras are complex (18/ 65)

20. Image Capture: Summary
Digital imaging is a computational expensive process!
complexity in the image acquisition
de-bayering, enhancing, pixel encoding
complexity in image quality tuning
3A parameters computation, tuning
strict correlation with application requests
feedback loop
Jacopo Mondi - Linux Lab 2018 Complex Cameras are complex (19/ 65)

21. Image Capture: Summary
Digital imaging is a computational expensive process!
complexity in the image acquisition
de-bayering, enanching, pixel encoding
complexity in image quality tuning
3A parameters computation, tuning
strict correlation with application requests
feedback loop
don’t forget we’re operating on -big- buffers
1080p in YUYV format is 4MB of data
Jacopo Mondi - Linux Lab 2018 Complex Cameras are complex (20/ 65)

22. Image Capture: Architecture
SoC
(fat) CPUs
ISP
Sensor
RAM
DMA
I2c control
image data
CSI-2
Jacopo Mondi - Linux Lab 2018 Complex Cameras are complex (21/ 65)

23. Smart sensors
In the pre-mobile computing era, embedded systems
where:
much less powerful
Jacopo Mondi - Linux Lab 2018 Complex Cameras are complex (22/ 65)

24. Smart sensors
In the pre-mobile computing era, embedded systems
where:
much less powerful
much more proprietary
Jacopo Mondi - Linux Lab 2018 Complex Cameras are complex (23/ 65)

25. Smart sensors
In the pre-mobile computing era, embedded systems
where:
much less powerful
much more proprietary
much less standardized
Jacopo Mondi - Linux Lab 2018 Complex Cameras are complex (24/ 65)

26. Smart sensors
SoC
(fat) CPUs
ISP
Sensor
RAM
DMA
I2c control
image data
CSI-2
Jacopo Mondi - Linux Lab 2018 Complex Cameras are complex (25/ 65)

27. Smart sensors
Let the sensor do the computations!
complex sensor architectures -> complex drivers
Jacopo Mondi - Linux Lab 2018 Complex Cameras are complex (26/ 65)

28. Smart sensors
Let the sensor do the computations!
complex sensor architectures -> complex drivers
complex sensor architectures -> expensive sensors
Jacopo Mondi - Linux Lab 2018 Complex Cameras are complex (27/ 65)

29. Smart sensors
Let the sensor do the computations!
complex sensor architectures -> complex drivers
complex sensor architectures -> expensive sensors
complex sensor architectures -> power hungry
sensors
Jacopo Mondi - Linux Lab 2018 Complex Cameras are complex (28/ 65)

30. Smart sensors
Let the sensor do the computations!
complex sensor architectures -> complex drivers
complex sensor architectures -> expensive sensors
complex sensor architectures -> power hungry
sensors
more computing blocks on the sensor =
less space for picture quality
Jacopo Mondi - Linux Lab 2018 Complex Cameras are complex (29/ 65)

31. Smart(er) ISPs
Image Signal Processor
system peripheral that operates on the digital
images
usually features configurable processing blocks
each block performs a specific operation, including:
Jacopo Mondi - Linux Lab 2018 Complex Cameras are complex (30/ 65)

32. Smart(er) ISPs
Image Signal Processor
system peripheral that operates on the digital
images
usually features configurable processing blocks
each block perform a specific operation, including:
memory transfers
Jacopo Mondi - Linux Lab 2018 Complex Cameras are complex (31/ 65)

33. Smart(er) ISPs
Image Signal Processor
system peripheral that operates on the digital
images
usually features configurable processing blocks
each block perform a specific operation, including:
memory transfers
image manipulation
Jacopo Mondi - Linux Lab 2018 Complex Cameras are complex (32/ 65)

34. Smart(er) ISPs
Image Signal Processor
system peripheral that operates on the digital
images
usually features configurable processing blocks
each block perform a specific operation, including:
memory transfers
image manipulation
format conversions
Jacopo Mondi - Linux Lab 2018 Complex Cameras are complex (33/ 65)

35. Smart(er) ISPs
Image Signal Processor
system peripheral that operates on the digital
images
usually features configurable processing blocks
each block perform a specific operations, including:
memory transfers
image manipulation
format conversions
the design is very platform specific
Jacopo Mondi - Linux Lab 2018 Complex Cameras are complex (34/ 65)

36. Smart(er) ISPs
Image Signal Processor
fast interconnections with very fast CPUs
interfaces with (a lot of) system memory
(possibly) interfaced with other system peripherals
as GPUs, DRM, codecs etc.
Jacopo Mondi - Linux Lab 2018 Complex Cameras are complex (35/ 65)

37. Smart(er) ISPs
Image Signal Processor
it comes with your SoC
silicon vendors are competing on this
Jacopo Mondi - Linux Lab 2018 Complex Cameras are complex (36/ 65)

38. Smart(er) ISPs
Image Signal Processor
it comes with your SoC
silicon vendors are competing on this
So, yes, we should really use ISPs as much as we
could
Jacopo Mondi - Linux Lab 2018 Complex Cameras are complex (37/ 65)

39. Smart(er) ISPs, and somewhat in the middle :/
Image Signal Processor, not on the SoC
some silicon vendors de-couple SoC and ISP..
off load image manipulation to tiny co-processors
and build stacks of crazy complex software
components to deal with that
Is this still a thing?
Jacopo Mondi - Linux Lab 2018 Complex Cameras are complex (38/ 65)

40. Media Controller
The V4L2 APIs are designed for ’simple’ device
basically, DMA engines that transfers ’untouched’
images
i2creceiver
sensor HARDWARE
pixel data i2c messages
KERNEL
receiver driver sensor driver
V4L2
/dev/video0
system memory
Jacopo Mondi - Linux Lab 2018 Complex Cameras are complex (39/ 65)

41. Media Controller
Support for more complex devices, as ISPs
the media controller exposes ISP processing blocks
to userspace
i2c
receiver
sensor
HARDWAREpixel data i2c messages
resizer
converter
formatter
/dev/v4l-subdev-1 /dev/video0
/dev/v4l-subdev-2
/dev/v4l-subdev-4
/dev/v4l-subdev-5
system memory
receiver driver
isp driver
pixel formatter
KERNEL
Jacopo Mondi - Linux Lab 2018 Complex Cameras are complex (40/ 65)

42. Media Controller
Support for more complex devices, as ISPs
the media graph is controlled through /dev/mediaX
i2c
receiver
sensor
HARDWAREpixel data i2c messages
resizer
converter
formatter
/dev/v4l-subdev-1 /dev/video0
/dev/v4l-subdev-2
/dev/v4l-subdev-4
/dev/v4l-subdev-5
/dev/media0
system memory
receiver driver
isp driver
pixel formatter
KERNEL
Jacopo Mondi - Linux Lab 2018 Complex Cameras are complex (41/ 65)

43. Media Controller
Nice, but.... subdevices have to be linked to form a
pipeline
MEDIA_IOC_SETUP_LINK ioctl
media−ctl −d /dev/media0
−l ” ’ rcar_csi2 feaa0000 . csi2 ’ : 1
−> ’VIN2 output ’ : 0 [ 1 ] ”
Jacopo Mondi - Linux Lab 2018 Complex Cameras are complex (42/ 65)

44. Media Controller
Nice, but.... subdevices have to be configured properly
VIDIOC_SUBDEV_S_FMT ioctl
media−ctl −d /dev/media0
−V ” ’ rcar_csi2 feaa0000 . csi2 ’ : 1
[fmt : RGB888_1X24/640x480 field : none] ”
Jacopo Mondi - Linux Lab 2018 Complex Cameras are complex (43/ 65)

45. Media Controller
Nice, but....
the system boots in unusable state
applications are requested to know a lot of details
on the underlying hardware
triggering capture operations requires precise
setup of the whole pipeline
Jacopo Mondi - Linux Lab 2018 Complex Cameras are complex (44/ 65)

46. Media Controller
Nice, but....
the system boots in unusable state
applications are requested to know a lot of details
on the underlying hardware
triggering capture operations requires setup of the
whole pipeline
This is (sort of) acceptable for embedded systems
Jacopo Mondi - Linux Lab 2018 Complex Cameras are complex (45/ 65)

47. Media Controller
Nice, but....
the system boots in unusable state
applications are requested to know a lot of details
on the underlying hardware
triggering capture operations requires setup of the
whole pipeline
This is (sort of) acceptable for embedded systems
Is it for consumer laptops?
Jacopo Mondi - Linux Lab 2018 Complex Cameras are complex (46/ 65)

48. Media Controller - now on laptops!
Until today, cameras in laptop have usually been USB
cameras
super-smart cameras: they talk USB!
fit well in the single device node abstraction
platform agnostic solution
controlled through the UVC control protocol
Jacopo Mondi - Linux Lab 2018 Complex Cameras are complex (47/ 65)

49. Media Controller - now on laptops!
Intel IPU3
Intel is now shipping processors with powerful ISPs
(some) KabyLake and (some) SkyLake platforms
have IPU3 ISP
patches on linux-media and on their way to
mainline: yey!
Jacopo Mondi - Linux Lab 2018 Complex Cameras are complex (48/ 65)

50. Media Controller - now on laptops!
(simplified) IPU3 media graph
CSI-2 CSI-2
capture
media0 media1
input params
viewfinder output stat
imgu
Jacopo Mondi - Linux Lab 2018 Complex Cameras are complex (49/ 65)

51. Media Controller - now on laptops!
(simplified) IPU3 media graph
CSI-2 CSI-2
capture
media0 media1
input params
viewfinder output stat
imgu
RAW
Jacopo Mondi - Linux Lab 2018 Complex Cameras are complex (50/ 65)

52. Media Controller - now on laptops!
(simplified) IPU3 media graph
CSI-2 CSI-2
capture
media0 media1
input params
viewfinder output stat
imgu
RAW
RAW
Jacopo Mondi - Linux Lab 2018 Complex Cameras are complex (51/ 65)

53. Media Controller - now on laptops!
(simplified) IPU3 media graph
CSI-2 CSI-2
capture
media0 media1
input params
viewfinder output stat
imgu
RAW
RAW
Jacopo Mondi - Linux Lab 2018 Complex Cameras are complex (52/ 65)

54. Media Controller - now on laptops!
(simplified) IPU3 media graph
CSI-2 CSI-2
capture
media0 media1
input params
viewfinder output stat
imgu
RAW
RAW
Jacopo Mondi - Linux Lab 2018 Complex Cameras are complex (53/ 65)

55. Media Controller - now on laptops!
(simplified) IPU3 media graph
CSI-2 CSI-2
capture
media0 media1
input params
viewfinder output stat
imgu
RAW
RAW
YUV
YUV
meta
Jacopo Mondi - Linux Lab 2018 Complex Cameras are complex (54/ 65)

56. Media Controller - now on laptops!
(simplified) IPU3 media graph
CSI-2 CSI-2
capture
media0 media1
input params
viewfinder output stat
imgu
RAW
3A - tuning meta
Jacopo Mondi - Linux Lab 2018 Complex Cameras are complex (55/ 65)

57. Media Controller - now on laptops!
”Re: Webcams not recognized on a Dell Latitude 5285
laptop”
https://www.spinics.net/lists/linux-media/msg131388.html
This laptop embeds one of these new ”complex” cameras
from Intel. They requires IPU3 driver. Though, unlike
traditional webcam, you need special userspace to use it
(there is no embedded firmware to manage focus,
whitebalance, etc, userspace code need to read the stats
and manage that). As of now, there is no good plan on how
to support this in userspace.
Jacopo Mondi - Linux Lab 2018 Complex Cameras are complex (56/ 65)

58. Libcamera!
V4L2 community joins forces with the industry to
provide a full camera stack for Linux-based systems
libcamera.org
user-space camera stack designed to work with
complex cameras
aims to be compatible with V4L2 applications,
Android and ChromeOs
free software - open development - safe space for
vendor IP
Jacopo Mondi - Linux Lab 2018 Complex Cameras are complex (57/ 65)

59. Media Controller - now on laptops!
We already have devices on the market using IPU3 for
cameras
Linux distributions won’t support cameras on those
platforms :(
vendors of Linux based systems (say, Android and
ChromeOS) will keep using their in-house solutions
Jacopo Mondi - Linux Lab 2018 Complex Cameras are complex (58/ 65)

60. Media Controller - now on laptops!
We already have devices on the market using IPU3 for
cameras
Linux distributions won’t support cameras on that
platforms :(
vendors of Linux based systems (say, Android and
ChromeOS) will keep using their in-house solutions
Don’t despair though! We might have a solution!
Jacopo Mondi - Linux Lab 2018 Complex Cameras are complex (59/ 65)

61. Libcamera!
Jacopo Mondi - Linux Lab 2018 Complex Cameras are complex (60/ 65)

62. Libcamera!
Jacopo Mondi - Linux Lab 2018 Complex Cameras are complex (61/ 65)

63. Libcamera!
Please join!
read the doc: http://libcamera.org/docs.htm
read patches: git://linuxtv.org/libcamera.git
have a chat: #libcamera on freenode
we welcome inputs from anyone working with
cameras! please help!
Jacopo Mondi - Linux Lab 2018 Complex Cameras are complex (62/ 65)

64. Complex Cameras are debated
ELC-E:
Why Embedded Cameras are Difficult, and How to
Make Them Easy - Laurent Pinchart, Ideas on Board
ELC-E:
Complex Cameras on Linux - Mauro Carvalho Chehab,
Samsung
Meeting notes on the Tokyo complex cameras
workshop:
https://linuxtv.org/news.php?entry=2018-07-13.mchehab
Jacopo Mondi - Linux Lab 2018 Complex Cameras are complex (63/ 65)

65. Complex Cameras
Thanks!
Jacopo Mondi - Linux Lab 2018 Complex Cameras are complex (64/ 65)

66. Complex Cameras
Thanks!
Questions?
Jacopo Mondi - Linux Lab 2018 Complex Cameras are complex (65/ 65)