Chromium's process architecture has graphics access restricted to a separate GPU-process. There are several reasons why this could make sense, three common ones are: Security, Robustness and Dependency Separation. GPU access restricted to a single process requires an efficient framework for communication over IPC from the other processes, and most likely a framework for composition of surfaces. This talk describes both the possible motivations for this kind of architecture and Chromium's solution for the IPC framework. We will demonstrate how a multi-process program can compose into a single window on Linux.

1. T
S
@pati_gallardo

2. Isolating GPU Access
in its own process
Patricia Aas, T S
NDC TechTown 2018
T
S
@pati_gallardo

3. Patricia Aas - Consultant
C++ Programmer, Application Security
Currently : T S
Previously : Vivaldi, Cisco Systems, Knowit, Opera Software
Master in Computer Science - main language Java
Pronouns: she/her T
S
@pati_gallardo

4. - What is Chromium?
- Communication Architecture
- Passing A Video Frame
- Why have a GPU process?
- Can I Use?
@pati_gallardo

5. Some Browser Trivia @pati_gallardo

6. Konqueror Safari Chrome
Brave
Vivaldi
Opera
KHTML Webkit Blink
KDE Apple Google

7. Composition
The Browser Window is
composed of many views
produced by many
cooperating processes
@pati_gallardo

8. Demo of Composition
...I might have made a browser...
...I might have decided to have a demo last night...
@pati_gallardo
T
S

9. Renderer Process
Webkit
Browser Process
Software
Composition
Original Chromium Software Composition Architecture
GUI

10. Renderer Process
Webkit
Browser Process
GUI
Gpu Process
Hardware
Composition
Moving Composition to
the GPU Process

11. - What is Chromium?
- Communication Architecture
- Passing A Video Frame
- Why have a GPU process?
- Can I Use?
@pati_gallardo

12. Communication Architecture
@pati_gallardo

13. Process Architecture
Browser
Gpu BrokerZygote
Renderer
GpuZygote Init
Renderer
Renderer
Process
Relationships
Tabs
IPC &
Commands

14. Components of Communication
Renderer
Gpu Process
Browser Renderer
IPC Channels
Shared Memory
Gpu Memory
Buffers
Command Buffers
(Ring buffer)
Gpu Memory
Buffers
Gpu Memory
BuffersCommand
Command
Command

15. Faking OpenGL ES 2 (for fun and profit?)
Command CommandCommand Command
Render/Browser Process Gpu Process
Client Encoder/Proxy ServerDecoder/Validator
Shared
Memory
OpenGL ES 2
Interface

16. - Write Commands to
Command Buffer in Shared
Memory
- Update ‘put’ pointer
- Signal GPU process
@pati_gallardo
Client
Renderer / Browser

17. - Read Commands from
Command Buffer in Shared
Memory
- Validate Command and
arguments
- Make actual call@pati_gallardo
Server
GPU Process

18. Server
(Gpu
Process)
IPC Channel
Command
Client
(Renderer /
Browser
process)
CommandCommandCommand Command
Command Stream Command BufferCommand
Conceptual Model

19. Synchronization Architecture
@pati_gallardo

20. - Inserts a synchronization
fence into the command
stream
- Can be attached to a
resource (texture) that
cannot be used before all
previous commands have
been processed
@pati_gallardo
Sync Token

21. SyncToken
@pati_gallardo
SyncToken
CommandBufferNamespace
release_count_
CommandBufferId

22. Gpu
Process
Command Buffer
IPC Channel
Command Stream
Ordering
Barrier
Unverified Sync Token
Wait Sync Token
Wait Sync TokenCommandCommand
Command
Command
Command Command
CommandVerified Sync TokenBrowser
Renderer
Renderer
Command
Command
CommandCommand
CommandCommand

23. - What is Chromium?
- Communication Architecture
- Passing A Video Frame
- Why have a GPU process?
- Can I Use?
@pati_gallardo

24. Getting a Video Frame into the Page
@pati_gallardo

25. Software Decoded Video Frame
- Decoded Frame in Memory in RENDERER PROCESS
- GPU Composition is done in the GPU PROCESS
- The Frame needs to be uploaded to the GPU as a
Texture BEFORE it can be composed
@pati_gallardo

26. Decode Frame into
Renderer Memory
Copy Frame to GPU
Memory Buffer
Issue Draw
Commands to GPU
Wait
SyncToken
Using the SyncToken to
Reorder

27. Insert Some Hand Waving
The full architecture is
massive
We will follow one path
A software decoded video
frame
@pati_gallardo

28. “At a high enough level of abstraction,
everything looks the same.”
Law of PowerPoint Architecture
Patricia Aas, 2018
@pati_gallardo

29. Decode
@pati_gallardo

30. Decoding Video
Browser Process
Network
stack
Renderer
Decoder*
VideoFrame
Memory Buffer
Y Plane
U Plane
V Plane
Internet
* Sometimes decoding is done in the GPU process

31. @pati_gallardo
“Texturize”

32. 1. Mailbox - unique name
2. SyncToken - fence
3. Texture Target Type (if
texture backed)
@pati_gallardo
Mailbox Holder

33. VideoFrame VideoFrame
Memory Buffer
V Plane
Y Plane
U Plane
Shared Memory Gpu ProcessRenderer
Transform the Video Frame into a GPU Resource
Y Plane Texture
UV Plane Texture
Plane Resources
Y Plane
GpuMemoryBuffer
UV Planes
GpuMemoryBuffer
MailboxHolder
SyncToken
MailboxMailbox
MailboxHolder

34. Prepare
@pati_gallardo

35. VideoFrame
Shared Memory
Gpu ProcessRenderer
Y Plane Texture
UV Plane Texture
Plane Resources
Y Plane
GpuMemoryBufferUV Planes
GpuMemoryBuffer
MailboxHolder
SyncToken
Mailbox
Mailbox
MailboxHolder
Transferrable
Resource
Texture filter
GL_LINEAR
Texture target
GL_TEXTURE_2D
Transferrable
Resource
Texture filter
GL_LINEAR
Texture target
GL_TEXTURE_2D
Id: 0
Id: 1
Move into a Transferrable Resource

36. Add to Render
@pati_gallardo

37. YUVVideoDrawQuad
Gpu ProcessRenderer
Y Plane Texture
UV Plane Texture
MailboxHolder
SyncToken
Mailbox
MailboxHolder
Mailbox
RenderPass
Resources
Id: 0 Id: 1
Transferrable
Resource
Texture filter
GL_LINEAR
Texture target
GL_TEXTURE_2D
Transferrable
Resource
Texture filter
GL_LINEAR
Texture target
GL_TEXTURE_2D
Id: 0
Id: 1
LayerTreeResourceProvider

38. Render!
@pati_gallardo

39. YUVVideoDrawQuad
Resources
Id: 0 Id: 1
Render The Frame!
GLRenderer::DrawYUVVideoQuad
clip_region

40. Gpu
Process
Wait Sync Token
Command
Command
Verified Sync Token
Browser
Renderer
CommandCommand
Command

41. GLES2 Extensions
@pati_gallardo

42. Examples : Chromium GLES2 Extensions
● CHROMIUM_image
● CHROMIUM_texture_mailbox
● CHROMIUM_sync_point
@pati_gallardo

43. VideoFrame
FrameResources
gfx::Size
PlaneResource
Mailbox
Unique Name
SyncToken
MailboxHolder
PlaneResource
PlaneResource
2. CreateImageCHROMIUM
GpuMemoryBuffer
GpuMemoryBufferVideoFramePool
Resource lifetime
ownership
MailboxHolder
MailboxHolder
3. BindTexImage2DCHROMIUM
image_id
1. BindTexture
texture_target
texture_id
1 to 3
1 to 3

44. - What is Chromium?
- Communication Architecture
- Passing A Video Frame
- Why have a GPU process?
- Can I Use?
@pati_gallardo

45. @pati_gallardo
Why Not Do GPU Composition in The Browser Process?

46. Well, Actually… On Android It Does… But I Digress…
@pati_gallardo

47. 1. Security
2. Robustness
3. Dependency Separation
4. Performance ?
@pati_gallardo

48. Security
@pati_gallardo

49. Gives Fine Grained Control
Texture memory being leaked across processes
- From Other Programs on the Users Machine
- From Other Tabs
- From the Browser @pati_gallardo

50. User : Lxgr
security.stackexchange.com

52. Robustness
@pati_gallardo

53. Graphics Drivers Crashing the Browser
- Prevent bugs in GPU drivers from crashing the browser
- Make sure graphics code in WebGL can’t crash the browser
- Compensate for Graphics Driver Bugs/Inconsistencies
@pati_gallardo

54. Dependency Separation
@pati_gallardo

55. Keep GPU Process Dependencies Out
of the Renderer process
- Minimize the renderer sandbox
- Can Have Different Dependencies
@pati_gallardo

56. Performance? ¯_(ツ)_/¯
@pati_gallardo

57. “We can solve any problem by introducing an extra
level of indirection.
…except for the problem of too many levels of
indirection”
Fundamental theorem of software engineering
Andrew Koenig/Butler Lampson/David J. Wheeler
@pati_gallardo

58. - What is Chromium?
- Communication Architecture
- Passing A Video Frame
- Why have a GPU process?
- Can I Use?
@pati_gallardo

59. - Ok, but… Can I Use?
- Hm, don’t know…
Maybe? ¯_(ツ)_/¯
@pati_gallardo

60. Not Exactly Cut And Paste
@pati_gallardo

61. Check : <chrome>://gpu
@pati_gallardo

62. Three APIs are in use in the renderer
1. Opengl ES2
2. Chromium GL ES2 Extensions
3. Chromium APIs
@pati_gallardo

63. “All non-trivial abstractions, to
some degree, are leaky.”
Law of Leaky Abstractions
Joel Spolsky, 2002
@pati_gallardo

64. - Ok, ok, but… Can I Use?
@pati_gallardo

65. ...I’d probably advice against it
@pati_gallardo

66. But knowing that it can be done
has value. It makes giving it a go
less crazy.
@pati_gallardo

67. So… maybe? ¯_(ツ)_/¯
@pati_gallardo

68. Patricia Aas, Consultant
T S
C++ and Application Security
T
S
@pati_gallardo

69. T
SD P

70. @pati_gallardo
T
S

71. Appendix / Some Notes
@pati_gallardo

72. High Level Design
Client - Server Architecture
Emulates OpenGl ES2.0
Actual Graphics Implementation is Platform Specific
Composition in GPU Process
Page Composition Controlled From Renderer
@pati_gallardo

73. Copy Video Frame To GPU Memory Buffer
Interesting Code
- CopyVideoFrameToGpuMemoryBuffers
- OutputFormat::NV12_SINGLE_GMB
- CopyRowsToNV12Buffer
- libyuv::I420ToNV12
- GpuMemoryBufferImplSharedMemory
@pati_gallardo

74. VideoFrame
FrameResources
gfx::Size
PlaneResource
Mailbox
Unique Name
SyncToken
MailboxHolder
PlaneResource
PlaneResource
2. CreateImageCHROMIUM
GpuMemoryBuffer
GpuMemoryBufferVideoFramePool
Resource lifetime
ownership
MailboxHolder
MailboxHolder
3. BindTexImage2DCHROMIUM
image_id
1. BindTexture
texture_target
texture_id
1 to 3
1 to 3

75. texture_target
Mac
GL_TEXTURE_RECTANGLE_ARB
Android/Linux
GL_TEXTURE_EXTERNAL_OES
Fallback
GL_TEXTURE_2D
@pati_gallardo

76. OES_EGL_image_external
Extension that creates EGLImage texture targets from EGLImages
“Each TEXTURE_EXTERNAL_OES texture object may require up to 3
texture image units for each texture unit to which it is bound.”
@pati_gallardo

77. CHROMIUM_image
CreateImageCHROMIUM
ReleaseTexImage2DCHROMIUM
BindTexImage2DCHROMIUM
DestroyImageCHROMIUM
@pati_gallardo

78. Share Group
- Command Buffers in the same share group
must be in the same Command Stream
- gl::GLFence
- eglFenceSyncKHR (EGL_KHR_fence_sync)
- eglWaitSyncKHR (EGL_KHR_wait_sync)
@pati_gallardo

79. VideoFrameProvider
Client
VideoFrameController
Client
InputHandler
Client
LayerTreeHostImpl VideoFrameCompositor
VideoRendererSink
OnBeginFrame
DidDrawFrame
UpdateCurrentFrame
GetCurrentFrame
PutCurrentFrame
VideoRendererImpl
Render
OnFrameDropped
VideoFrameProviderClientImplVideoFrameProviderClientImplVideoFrameProviderClientImpl
Video Frame Painting
VideoFrame
current_frame_
VideoLayerImpl
active_video_layer_
DecodersVideoResourceUpdater

80. Useful files to read
gpu_memory_buffer_video_frame_pool.cc
video_resource_updater.cc
gl_renderer.cc (GLRenderer::DrawYUVVideoQuad)
program_binding.cc (ProgramKey::YUVVideo)
@pati_gallardo

81. P f .
Patricia Aas, T S
@pati_gallardo
T
S

82. @pati_gallardo
T
S