Nowadays the Web is one of the primary ways for multimedia content consumption and real-time communication (through WebRTC). During this talk Philippe will present the WPEWebKit web-engine that has been deployed on a wide range of embedded platforms and how you can add it to your own Linux-based embedded device. WPEWebKit is the official WebKit upstream port for embedded platforms. For multimedia playback and real-time communication it heavily relies on the GStreamer multimedia framework. Philippe will give an overview of the W3C specifications supported by WPEWebKit. WPEWebKit products have been deployed in various embedded environments and hardware platforms. Philippe will focus on i.MX platforms, outlining the steps required to enable WPEWebKit in Yocto-based BSPs. WPEWebKit can also be used in server-side innovative ways, such as dynamic HTML/JS/CSS powered video overlaying. Philippe will present this use-case, detailing how live video streams can be augmented with overlays. GstWPE is a GStreamer plugin embedding a WPEWebKit WebView, allowing to inject a live audio/video representation of any Web page into a GStreamer pipeline. Both GPU-based hardware-accelerated and software rasterisers runtimes are supported. (c) Embedded Linux Conference - North America (ELC-NA 2021) September 27-30, 2021 Hyatt Regency Seattle | Seattle, Washington + Virtual https://events.linuxfoundation.org/embedded-linux-conference-north-america/

1. Reaching the Multimedia Web
Reaching the Multimedia Web
from Embedded Platforms with
from Embedded Platforms with
WPEWebKit
WPEWebKit
Philippe Normand
Philippe Normand
Embedded Linux Conference 2021
Embedded Linux Conference 2021
1

2. Talk outline
Talk outline
Intro
WPE architecture overview
Supported W3C specifications
Cog
WPE on Embedded platforms with Yocto
Using WPE in GStreamer Multimedia applications
2

3. Who am I
Who am I
WebKit committer and reviewer
GStreamer committer
Partner at Igalia:
Worker-owned coop, currently around 110 happy Igalians around the
world
Provides consulting services for various Free Software projects
3

4. WPE architecture
WPE architecture
overview
overview
4

5. What is WebKit
What is WebKit
WebEngine aimed for embedding HTML/CSS/JS in native applications
Forked from KHTML by Apple in 2004
Powers Safari, but also dozens of applications on various platforms
APIs provided by WebKit ports
5

6. Multi-process!
Multi-process!
6

7. WPE, a decoupled WebKit port
WPE, a decoupled WebKit port
Upstream in webkit.org
6 months release cycle, security updates
Not tied to any widget toolkit
Rendering and input events handling via loadable backends
7

8. Rendering backends
Rendering backends
Various implementations, most notably:
FDO backend
RDK backend (various devices supported under the RDK/Comcast
umbrella)
8

9. WPEBackend-FDO
WPEBackend-FDO
Depends on EGL, usually provided by Mesa or binary drivers
High level API provided for applications (browsers, but also other apps!)
Recommended by upstream WPE community
9

10. GStreamer
GStreamer
10

11. W3C Specs
W3C Specs
11

12. Specs, specs everywhere
Specs, specs everywhere
https://webkit.org/status/
12

13. MediaSource Extensions
MediaSource Extensions
Adaptive streaming for the Web
Widely used by most major video streaming platforms (Youtube, etc)
Can ingest DASH (with DASH.js)
WPE: enabled by default at build and runtime
Spec: https://w3c.github.io/media-source/
13

14. MSE in WPE
MSE in WPE
Chunks queued from JavaScript world to a SourceBuffer
One GStreamer WebKit Append pipeline per SourceBuffer
Demuxing and parsing of samples
Samples stored at WebCore’s MSE layer
Playback pipeline using a dedicated MediaPlayerPrivate implementation
Playbin3-based
Custom source element
14

15. Encrypted Media Extensions
Encrypted Media Extensions
Protected media content playback
Used by most video streaming platforms (Netflix, etc)
WPE: disabled by default due to runtime requirements (Content-
Decryption-Module)
CDMs integrated on per-product basis in WPE
Spec: https://www.w3.org/TR/encrypted-media/
15

16. EME in WPE
EME in WPE
Protected content signaled from demuxers to WPE through GStreamer
Protection events
WPE probes for supported platform CDMs through the OpenCDM API
Two CDM runtimes supported:
RDK Thunder
Sparkle-CDM
Content decryption in WPE through custom GStreamer decryptors
16

17. MediaCapabilities
MediaCapabilities
Media engine decoding/encoding capabilities probing
WPE: enabled by default at build time, disabled at runtime
Spec: https://w3c.github.io/media-capabilities/
17

18. MediaCapabilities in WPE
MediaCapabilities in WPE
Probing implemented through a GStreamer registry scanner
Runtime identification of encoders, decoders, (de)muxers
WebKit mimetype <-> GStreamer caps
18

19. WebAudio
WebAudio
Low-latency audio pipelines
Examples of use: games, music creations apps, any app requiring audio
feedback for user interations
WPE: enabled by default at build and runtime
Spec: https://www.w3.org/TR/webaudio/
19

20. WebAudio in WPE
WebAudio in WPE
Decoding pipeline relying on GStreamer decodebin
Playback pipeline sourcing WebKit audio samples from custom GStreamer
source
element
20

21. Media Capture & Streams
Media Capture & Streams
Access to Webcam/microphone/screencasting
Used by WebRTC apps mostly (Jitsi, etc)
WPE: enabled only in developer builds, plans for enabling by default in WPE
2.36
Spec: https://www.w3.org/TR/mediacapture-streams/
21

22. Media Capture & Streams in WPE
Media Capture & Streams in WPE
Capture device management with GstDeviceMonitor
One capture pipeline per device, feeding one-to-many appsinks for
consumption by WebKit
Rendering by MediaPlayer through custom mediastream GStreamer source
element
Screencasting through PipeWire
22

23. WebRTC
WebRTC
Real-Time P2P communication for browsers
Used for audio/video chatting, low-latency one-to-many broadcasting, ...
WPE:
LibWebRTC enabled only in developer builds
Unusable in releases (LibWebRTC is not bundled in tarballs + licensing
issues due to BoringSSL)
Plans for a GStreamer-WebRTC backend in WPE 2.36
Spec: https://www.w3.org/TR/webrtc/
23

24. MediaStream recording
MediaStream recording
Use-case: mostly WebRTC calls recording
Not enabled in WPE yet. WIP implementation:
Based on the new GstTranscoder GStreamer library (1.20)
Makes use of existing internal WPE WebRTC video encoder GStreamer
element
Spec: https://w3c.github.io/mediacapture-record/
24

25. Cog, the official WPE-based
Cog, the official WPE-based
browser
browser
Minimalistic design leveraging platform renderer modules:
Wayland, X11 (!), GTK4
DRM, Headless
Auto-probing of platform for renderer selection
Single web view, for now
Can be controlled through DBus
25

26. Running Cog without Wayland
Running Cog without Wayland
compositor
compositor
App use-cases: Kiosks, Set-Top-Box UIs, ... any fullscreen display
Wayland buffers (or DMABufs) imported as GBM Buffer objects
Rendering through DRM backend
Input events handled with libinput (keyboard, pointer, touch)
26

27. Headless Cog
Headless Cog
Use-case example: Apple Music audio player
No rendering at all, no GPU needed.
Need for custom DBus bridge for user-interaction:
-> example: Interaction
with JS Apple Music SDK for playback control
27

28. Deploying WPE/Cog on
Deploying WPE/Cog on
i.MX with Yocto
i.MX with Yocto
28

29. Yocto layers
Yocto layers
WPEBackends
Cog browser
Poky reference distro (including GStreamer 1.18.x)
(meta-freescale)
https://github.com/Igalia/meta-webkit/
29

30. Open-source etnaviv driver
Open-source etnaviv driver
Nowadays well supported in the kernel and Mesa, depending on specifics of
the
SoC though
Usable WPEBackends, only working in Weston:
WPEBackend-fdo (recommended)
WPEBackend-RDK/wayland
Upstream v4l2 plugin from gst-plugins-good for hardware decoding support
30

31. i.MX6 video decoding
i.MX6 video decoding
CODA960 kernel driver
Mature integration with gst-plugins-good v4l2 plugin
On QuadPlus: H.264 1080P@30 handled but sometimes drops frames:
720P recommended:
$ export
WEBKIT_GST_MAX_AVC1_RESOLUTION=720P
31

32. i.MX8M video decoding
i.MX8M video decoding
Hantro G1 kernel driver and integration with gst-plugins-good
Development by Collabora and Pengutronix
Released in kernel 5.12
Status:
1080P@30 H.264 smooth playback
Should allow for up to 4K@30 with VP8 and H.264 and 4K@60 with VP9
and HEVC
HEVC and VP9 support: Work in progress
32

33. YUV video rendering in WPE
YUV video rendering in WPE
Internal video sink used to support RGBA only
Since then, additional formats are now supported:
YUV textures through external-oes
I420, Y444, YV12, Y41B, Y42B, NV12, NV21, VUYA, A420
33

34. Alternative fallback: Proprietary
Alternative fallback: Proprietary
NXP drivers
NXP drivers
Where etnaviv / v4l2 decoders are not usable, fallback to NXP drivers
WPE supports the gstreamer-imx decoders as well
GL sink makes use of imxvideoconvert_g2d for zero-copy rendering
34

35. Using WPE in GStreamer
Using WPE in GStreamer
applications
applications
35

36. Use-case #1: Stinger transitions
Use-case #1: Stinger transitions
https://obsproject.com/wiki/Getting-Started-With-Track-Matte-Stinger-Transitions
36

37. Use-case #2: Lower thirds
Use-case #2: Lower thirds
37

38. Use-case #3: Score boards
Use-case #3: Score boards
38

39. Why use WPE/GStreamer for
Why use WPE/GStreamer for
this?
this?
No vendor-lockin in overlay editors
Vast pool of Web designers
Flexibility and broad platform support provided by GStreamer
39

40. GstWPE
GstWPE
Use-cases: HTML overlays, streaming/cloud browsers
GStreamer source element producing audio/video streams from a WPE
WebView
Two runtime modes:
Zero-copy from WPEBackend-FDO EGLImages to GL GStreamer sink
Software rasterizer with LLVMpipe to non-GL GStreamer sink
40

41. Simple examples
Simple examples
$ gst-play-1.0 --videosink gtkglsink
wpe://https://linuxfoundation.org/
# Only with GStreamer git, should work in 1.20:
$ gst-play-1.0 --videosink gtksink
wpe://https://linuxfoundation.org/
41

42. Sample pseudo-pipeline for
Sample pseudo-pipeline for
audio/video mixing
audio/video mixing
Don't forget to set wpesrc::draw-background to 0
Configure zorder value on compositor sink pads
wpesrc
asrc0
vsrc
src
filesrc
autovideosink
sink
autoaudiosink
sink
decodebin
asrc0
vsrc
sink
src
compositor
sink0
sink1
src
audiomixer
sink0
sink1
42

43. Web-augmented one-to-many
Web-augmented one-to-many
broadcasting
broadcasting
Dynamic overlays controlled through a NodeJS app
Mixed video encoded and broadcasted to a remote Janus video room
Video consumption through WebRTC
https://www.youtube.com/watch?v=QNZJYOuVGiE
43

44. Wrap-up
Wrap-up
WPEWebKit, WPEBackend-FDO, Cog:
GstWPE:
Yocto overlay:
Questions?
https://wpewebkit.org
https://gstreamer.freedesktop.org/documentation/wpe/
https://github.com/Igalia/meta-webkit
44