These are the slides of my talk WebKit and GStreamer of the GStreamer Conference on 2013, cohosted with LinuxCon. The HTML5 version with its effects can be found at http://people.igalia.com/xrcalvar/talks/20131022-GstConf-WebKit

2. WebKit and GStreamer
Bringing lolcatz videos to your desktop
Xabier Rodríguez Calvar
Igalia

3. Who is this dude?
· Born in the Basque Country
· Grown and living in Galicia
- One of the seven Celtic Nations, like Scotland
· Studied Computer Science at Universidade da Coruña
· Partner and developer at Igalia
- Working in Multimedia and WebKit

4. From html5demos.com

5. Index
· Quick review of WebKit
· WebKit Multimedia
· Power features
- WebAudio
- Hardware acceleration
- Accelerated video compositing
5/36

6. Quick Review

7. WebKit
· Web content engine
· Processes and renders web content
· Started as a fork of KTHML and KJS
· Open source since 2005
· Goals: open source, compatibility, compliance, stability,
performance, security, portability, usability, hackability
· Non-goals: being a full web browser, being a science project,
having reusable components, unlimited scope
· Forked by Google as Blink
· Split into ports: GTK+, Qt, EFL, Mac, Chromium, Windows
7/36

8. Architecture
8/36

9. WebKitGTK+
· Each WebKit port is composed of
- Platform interfaces
- API layer
· WebKitGTK+ platform layer:
- libsoup for networking
- cairo for rasterization
- OpenGL for making the scene graph and WebGL
- GStreamer for media
- Various GTK+ APIs for talking with the system
· API layer is a GtkWidget and a set of GObject APIs
· WebKitGTK+ is used by Epiphany, Midori, yelp, devhelp
9/36

10. WebKit2
· Give the web rendering parts of WebKit their own process
· Page crashes don't crash the browser
· Can put vulnerable data into a separate address space
· Sandbox web rendering
· Prevent pages from accessing the disk and operating system
interface
10/36

11. WebKit2 Architecture
11/36

12. The Near Future
12/36

13. WebKit Multimedia

14. WebKit Multimedia
· Mac port: QuickTime, QtKit, AVFoundations
· Apple Win port: QuickTime
· GTK+ and EFL ports: GStreamer 1.0 backend
· Blackberry port: Bridges over to its own Media playback framework
· Chromium: bridges over to WebKit/Chromium backend which
relays to FFMPeg implementations
· Qt port: GStreamer on Linux, QtKit on Mac, QuickTime on
Windows
14/36

15. 15/36
remaertSG
mroftalP
- - - - - - - - - - - reyalPaideM
eroCbeW
tiKbeW
WebKit Multimedia Architecture

16. GStreamer MediaPlayer - Overview
· Mainly developed and tested on Linux platform
· Successfully ported to MacOS/Win platfoms (not fully upstream
yet though)
· Playbin-based: the best GStreamer element for playback usecases
- Playbin2 with GStreamer 0.10
· MediaPlayer registers and uses two WebKit-specific GStreamer
elements: webkitwebsrc and a video-sink
16/36

17. GStreamer MediaPlayer - Main Features
· On-disk-buffering
· Frame-accurate seeking
· Basic metrics reporting
· Trick-modes
· WebAudio
· Hardware decoding support with VA-API (gst 1.2.x)
· Video accelerated compositing
· Codec installer support
17/36

18. WebKit and GStreamer Architecture
MediaPlayerClient
-eiPaeCin
mdalyrlet
+lyakhne(
pabcCagd)
+ouehne(
vlmCagd)
HTMLMediaElement
-eiPae
mdalyr
+ly)
pa(
+lyakhne(
pabcCagd)
+eVlm(nvlm:la)
stouei ouefot
+ouehne(
vlmCagd)
-_rvt
mpiae
MediaPlayer
+ly)
pa(
+eVlm(nvlm:la)
stouei ouefot
MediaPlayerPrivateInterface
+ly)
pa(
+eVlm(nvlm:la)
stouei ouefot
MediaPlayerPrivateGStreamerBase
+eVlm(nvlm:la)
stouei ouefot
HTMLVideoElement
HTMLAudioElement
GstPlaybin
-lyi
pabn
MediaPlayerPrivateGStreamer
+ly)
pa(
18/36

19. GStreamer MediaPlayer - HTTP source element
· a GStreamer bin implementing the URI handler interface for
http(s):// and blob:// uris.
· A single source pad provides raw data access to downstream
elements.
· appsrc is used internally to interface with the WebCore
SubResourceLoader object
· Element is registered with high rank in the GStreamer registry so
playbin is more prone to pick it up to read http:// URIs.
19/36

20. GStreamer MediaPlayer - Video rendering
· Element inheriting from GstVideoSink
- In practical sense, it is a GstAppSink with video capabilities
negotiation
· Takes video GstBuffers as input
· Pass those to the MediaPlayerPrivateGStreamer via a signal
20/36

21. GStreamer MediaPlayer - Audio rendering
· Use autoaudiosink
· Check for pulseaudio for playback request messages
· Pitch preservation
· Preserve audio passthrough when possible
· Volume managed by the sink
21/36

22. GStreamer MediaPlayer - Next steps
· Improvements and bug fixes in the player:
- Store on-disk buffered media in WebCore's cache directory
- Make the webkitwebsrc element private to WebKit
· MediaStream (getUserMedia) and WebRTC spec implementation
· MediaSource adaptive streaming API spec implementation
· Integration of WebAudio with media elements and mediastream
stack
· Finish porting to 1.2
· Remove GStreamer 0.10 code path
22/36

23. WebKit Multimedia Power Features

24. WebAudio
· A JavaScript API allowing to create a new generation of Web
games and applications
· Effects, convolution engine for room simulation, etc
· Realtime analysis, playback and visualization
· W3C spec driven and implemented mainly by Chromium and
Firefox
· In WebKit, currently implemented for:
- Mac
- Chromium
- GStreamer based Ports: GTK+, EFL, Qt
24/36

25. WebAudio GStreamer - Reading Audio
· Spec allows two ways to read audio data:
- from memory: we can use giostreamsrc GStreamer element
- from a file path: we can use the filesrc GStremer element
· Pipeline: src ! decodebin2 ! audioconvert ! audioresample !
capsfilter ! deinterleave ! queue ! appsink
· deinterleave separates multi-channel audio to n audio mono
channels (Pipeline above can contain multiple queue ! appsink
chains)
· appsink used to collect mono audio buffers, store them in a list and
convert data to floating point values
· Data is stored in a WebCore::AudioBus, containing one or multiple
channel Float arrays
25/36

26. WebAudio GStreamer - Playing Audio
· The WebAudio stack processed or generated some audio data to be
sent to the sound card
· Similarly to the "Reading Data" step, a WebCore::AudioBus is used
to store the data
· Custom GStreamer element to convert an AudioBus to an
interleaved audio data flow: webkitwebaudiosrc
· webkitwebaudiosrc: a bin with n internal pads pushing to an
interleave element. Source pad outputs a single, interleave, wavencoded data flow
· Pipeline: webkitwebaudiosrc ! wavparse ! audioconvert !
autoaudiosink
26/36

27. WebAudio GStreamer - Work in Progress
· Some layout tests don't pass yet
· Not yet enabled by default in release builds
· Support for stereo+ streams in the reader side. Playback side
already handles those.
· Integration with <video> and <audio> elements: WebAudio
MediaElementSourceNode
27/36

28. Hardware Accelerated Video
· We use GStreamer VA-API. (1.2.x)
· That's it! Hardcore comes when compositing that.
28/36

29. Accelerated Video Compositing - Acceleration in
WebKit
From Joone Hur
29/36

30. Accelerated Video Compositing
· Element inheriting from GstVideoSink
- In practical sense, it is a GstAppSink with video capabilities
negotiation
· Takes video GstBuffers as input.
· Video frame is transferred to the MediaPlayerPrivateGStreamer via
a GLib signal.
· MediaPlayerPrivateGStreamer uploads it to a GL texture and
renders it asynchronously in the WebKit context.
30/36

31. Accelerated Video Compositing - Architecture
TextureMapperPlatformLayer
MediaPlayerPrivateInterface
+an(
pit)
MediaPlayerPrivateGStreamerBase
+an(
pit)
GstPlaybin
-lyi
pabn
MediaPlayerPrivateGStreamer
*
GstElement
...
WebKitVideoSink
<cets>
<rae>
31/36

32. Accelerated Video Compositing - Rendering paths
· Software rendering path
- It uses Cairo
· Accelerated compositing paths
- TextureMapperLayer:
- Light-weight implementation of a scene-graph
- The TextureMapperLayer tree is equivalent to the
GraphicsLayer tree.
- TextureMapperGL
- Provides a GPU accelerated implementation of the
drawing primitives, using shaders compatible with GL/ES
2.0
32/36

33. Accelerated Video Compositing - Next steps
· More video decoders with GstVideoGLTextureUploadMeta support
· Support EGLImage in WebKit as video frames handlers
- And video decoders that deliver this kind of capabilities.
· Handle the vertical blank interval in WebKit
- GTK+ 3.8 seems to fix this issue.
33/36

34. From html5demos.com

35. From html5demos.com
35/36

36. Thank You!
(q&a)
twitter @calvaris
www blogs.igalia.com/xrcalvar