This is the story of the glory and struggle of bringing a high quality YouTube experience to the mobile web. Once upon a time there was a web developer who wanted to play videos on the web. So he filmed a cat and wrote a Flash and a HTML5 player. He spent many hours making it work on his favorite desktop browsers and even the one his grandfather still used. People could watch his cat video, he smiled, and the world was good. Then one day someone put a browser in a phone and soon there were many phones with many different browsers. This new set of environments were even harder to develop for and had a slew of new terrifying bugs. The web developer was miserable knowing people couldn’t watch his cat video. With much time and effort he figured out many of the secrets needed to combat the evils of the different mobile platforms. Once again people could watch his cat video, he smiled, and the world was good. The end.

1. Mobile Meow
Bring YouTube to the Mobile Web
By YouTube's Greg Schechter and Matt Ward

2. Greg Schechter Matt Ward
The Web Warrior Seek Bar Engineer
schechter@google.com mattward@google.com

3. Video

4. Video

7. A Brief History

8. Video

9. Flash

10. <video>

11. <video> <video> <video>
IE Chrome Firefox
<video> <video>
Opera Safari

12. Video

13. Playback Stats

14. Flash Flash

15. Flash Native <video>

16. Flash
Flash
== Flash

17. Native

18. Native
○ For fallbacks, use rtsp:// protocol (serving .3pg) if the
device won't support HTML5
○ Use custom protocol / URL scheme to launch your own
Android, iOS native app (no Windows Phone yet)
■ youtube://video_id

19. <video>

20. The Different Browsers
Safari Android

21. The Different Browsers
Opera
Safari Android Chrome Silk
Mini
Firefox Opera
IE

22. The Different Browsers
Opera
Safari Android Chrome Silk
Mini
Firefox Opera
IE
AMP LE
S

23. Fragmentation

24. By brownpau at http://www.flickr.com/photos/brownpau/533267369/

25. How do we start
<video>

26. The Source
<video src="funny_cat_video">

27. The Source
<video src="funny_cat_video.webm/mp4/ogg">

28. Mobile Formats Support
Chrome Safari Firefox Opera IE Android
H.264
WebM Android 4+ Android 4+ Android 4+
HLS Android 3+
Platform Versions and Distribution

29. The Source
<video>
<source src="funny_cat_video.mp4">
<source src="funny_cat_video.webm">
</video>

30. HLS = HTTP Live Streaming
<video src="playlist.m3u8">

31. HLS
video-0-1.ts
format-0.m3u8 video-0-2.ts
...
video-1-1.ts
index.m3u8 format-0.m3u8 video-1-2.ts
...
video-2-1.ts
format-0.m3u8 video-2-2.ts
...

32. HLS
Big Features
○ Live and Adaptive Streaming
○ Built in CDN failover
○ Metadata + Cue Points
Apple App Requirements
○ > 10 minutes in duration or 5 mb in 5 minutes
○ at least one 64 kbps or lower stream required
Tools and more information on Apple's website:
Apple's Using HTTP Live Streaming

33. Power

34. Power Consumption: H264 vs WebM
**fullscreen flash
**windowed flash
Research done by Yossi Oren For more information visit http://iss.oy.ne.ro/

35. Stick it in a page

36. Basic video tag
Safari
○ Renders a thumbnail poster and play button (flaky)
○ Background is set to black and cannot be overwritten
○ If a parent element has display:none set at any point the video fails to
playback properly
Opera
○ Gives video dimentions black background if the information is
available
○ Has issues with stretching WebM
IE and Android
○ Play button/film icon for all video tags regardless if browser can play
the format
○ Background is set to black/grey and cannot be overwritten

37. Give it a poster and click it

38. Poster Attribute
Use the poster attribute to get a thumbnail
Safari, Chrome, Android, and Opera
○ Preserve aspect ratio of poster image
IE and Firefox
○ Stretch image to size of video tag
IE
○ Keeps poster as paused state
All Others
○ Replace video with the paused frame

39. Basic video tag
Chrome
○ Default click to toggle play/pause
○ Remove with an empty onclick handler
Safari
○ Will launch any supported video with or without an onclick handler
Android and IE
○ Shows click interactions but does not play with empty onclick

40. Basic video tag
Pro tips:
○ Create your own cued state to convey you can play the video
○ Set the background color to black for continuity across platforms

41. Controls
<video controls>

42. Lets get some controls

43. Controls
Chrome Safari Firefox Opera IE Android
Play/pause
Button
Volume Only in Mute toggle Don't work in
Controls Fullscreen only Android 4
Seek bar
Fullscreen Button Gesture Button
button
Playback Both Fullscreen Inline only Determined Fullscreen Both
Type only by hardware only

44. Fullscreen

45. requestFullScreen

46. requestFullScreen
● Desktop Only
○ Firefox, Safari, and Chrome
○ Vendor Prefixed

47. Fullscreen
var elem = document.getElementById("my-element");
document.onwebkitfullscreenchange = function() {
// The fullscreen element:
// document.webkitFullScreenElement;
console.log ("We went fullscreen!");
};
elem.webkitRequestFullScreen();

48. What about Mobile?

49. Isn't it already fullscreen?

50. Yes (sort of)

51. Fullscreen
● Desktop
○ requestFullScreen
● Mobile

52. Fullscreen
● Desktop
○ requestFullScreen
● Mobile
○ Open New Tab

53. Fullscreen
● Desktop
○ requestFullScreen
● Mobile
○ Open New Tab
○ webkitEnterFullscreen
● Webkit only
● Video element only
● metadata must be loaded

54. Fullscreen
● Desktop
○ requestFullScreen
● Mobile
○ Open New Tab
○ webkitEnterFullscreen
● Webkit only
● Video element only
● metadata must be loaded
● Pro Tip: Remember size context changes so
use viewport to scale icons and controls

55. @viewport { width: device-width; }

56. Custom Controls
Lets get some custom controls

57. Custom Controls
● Preserve your brand
● Unified experience across platforms and
browsers
● Allows us to expand the set of controls and add our own
○ annotations
○ playlist
○ captions
○ more

58. Custom Controls Pro Tips
● User expect to be able to drag the progress
bar
○ Need to remender to prevent scroll on touchstart
● Volume can't be set everywhere and users
are accustomed to using device controls
○ So don't build controls for it

59. Custom Controls Pro Tips
● Fullscreen
○ The browsing context is always fullscreen so fake it
● Don't overload clicking for both play/pause
and show/hiding controls
● Don't trigger content with hover

60. Custom Controls Pro Tips
● Fingers are fat
○ Average finger is 11mm so make targets at least
40px with 10px padding
○ Use SVG's so icons can be scaled and shared with
desktop application
○ Do what you can in css

61. autoplay
<video autoplay>

63. autoplay
In Safari on iOS (for all devices, including iPad),
where the user may be on a cellular network and
be charged per data unit, preload and autoplay are
disabled. No data is loaded until the user initiates it.

64. autoplay
In Safari on iOS (for all devices, including iPad),
where the user may be on a cellular network and
be charged per data unit, preload and autoplay are
disabled. No data is loaded until the user initiates it.

65. What about everyone else?

66. Autoplay
Chrome Safari Firefox Opera IE Android
Attribute
Scripted Buggy

67. autoplay

68. autoplay
<video onclick="this.play()">

71. autoplay
function someClickEvent(evt) {
// In a user initiated thread.
myVideoElement.load();
getVideoData(); // Triggers an ajax call.
}
function onGetVideoDataReturned(data) {
// Not in a user initiated thread.
setVideoElementSrc(data);
myVideoElement.load();
myVideoElement.play();
}

72. autoplay
function someClickEvent(evt) {
// In a user initiated thread.
myVideoElement.load();
getVideoData(); // Triggers an ajax call.
}
function onGetVideoDataReturned(data) {
// Not in a user initiated thread.
setVideoElementSrc(data);
myVideoElement.load();
// For Android
window.setTimeout(function() {
myVideoElement.play();
}, 0);
}

73. autoplay
setMediaPlaybackRequiresUserAction

74. Embeds

75. Embeds
<script>

76. Embeds
<script>
○ We need our content to be sandboxed
○ More than just a video tag

77. Embeds
<script>
○ We need our content to be sandboxed
○ More than just a video tag
<object>

78. Embeds
<script>
○ We need our content to be sandboxed
○ More than just a video tag
<object>
○ Can load content with the data
attribute
○ But no way to interact with it via
JavaScript

79. Embeds
<iframe>

80. Embeds
<iframe>
○ Allows our content to be sandboxed
○ JavaScript API communication

81. Embeds
<iframe type="text/html"
width="640"
height="385"
frameborder="0"
src="http://www.youtube.com/embed/VIDEO_ID"
allowfullscreen>
</iframe>

82. Daily impressions of Embed
Total
object
iframe
April 2011 April 2012

83. Embeds
Pro tip: Plan for the future (if you can)
<iframe type="text/html"
width="640"
height="385"
frameborder="0"
src="http://www.youtube.com/embed/VIDEO_ID"
allowfullscreen>
</iframe>

84. Embeds Pro Tips:
html {
/** Hack to fix iPhone resizing. */
overflow: hidden;
}

85. Embeds Pro Tips:
body {
/** Dymanic Resizing **/
background-color: #000;
height: 100%;
width: 100%;
/** Remove highlight when use clicks **/
-webkit-tap-highlight-color: rgba(0, 0, 0, 0);
}

86. API

87. API Communication

88. API Communication
● Poll the URL fragment?
○ http://youtube.com/embed/video_id#fragment

89. API Communication
● Poll the URL fragment?
○ http://youtube.com/embed/video_id#fragment
● Messages are one dimensional.
● Polling eats up CPU and is not instant.
● Both directions of communication use the same
fragment.

90. API Communication
● Better idea: PostMessage API
○ someWindow.postMessage(message, targetOrigin);

91. API Communication
● Better idea: PostMessage API.
○ someWindow.postMessage(message, targetOrigin);
● Uses JSON for native encoding and decoding of data.
● No polling.
● Native event listeners.
● Communications are sandboxed per-window.
● Calls are asynchronous.
● Great for Mobile, but desktop support is missing in IE7

92. Remember all communication is
asynchronous

93. Testing

94. Testing: The Old Fashioned Way
● Device lab
○ accurate
○ costly
○ space inefficient
○ boring

95. Testing: With Software
● Hardware Emulators / Simulators
○ available for major systems
○ approximation varies
○ still boring
● Android Emulator
○ bulky, slow, no video codecs
● iOs Simulator
○ both tablet and phone

96. Testing: With Software
● Browser Simulators
○ Poorer approximation
● Opera Mobile Simulator
○ missing video tag support
● Fennec (FF) Simulator
○ poor touch control mapping
○ sends desktop user agent

97. Testing: Automation
● Selenium!
○ use with simulated or real devices
○ not boring

98. Testing: Automation
● Android Webdriver
○ Still no video support
● iOS Webdriver
○ Need to register as ios dev
○ Intermittent issues with playback
● IE
○ No webdriver APIs for mobile

99. Testing: Automation
● Chrome, Opera
○ Driver APIs built-in
○ Remote debugging
● FFMobile
○ No webdriver APIs

100. Testing: Automation Strategy
● Test API methods in mobile context first
○ cheap
○ can approximate video playback by video.
currentTime

101. Testing: Automation Strategy
● More sophisticated tests to follow
○ screenreader
● Screen cap processing
○ check for distortion
○ playback accuracy

102. Testing: Automation Strategy
● Screen cap processing

103. Testing: Hardware Assistance

104. The Future
Cats in Space by WF&TD

106. Or

107. PointerEvents
http://msdn.microsoft.com/en-us/library/ie/hh673557(v=vs.85).aspx

108. http://imgs.xkcd.com/comics/in_ur_reality.png

109. Track
<video>
<track src="cats_meow" kind="subtitles">
</video>

110. getUserMedia

111. More to learn
● HTML5 Video Spec
○ http://dev.w3.org/html5/spec-author-view/video.html
● Browser Blogs
○ http://dev.opera.com/
○ http://hacks.mozilla.org/
○ http://blogs.msdn.com/b/ie/
○ http://peter.sh/
● Documentation
○ https://developer.mozilla.org/en-US/
● Other
○ http://developer.apple.
com/library/safari/#documentation/AudioVideo/Conceptual/Using_HT
ML5_Audio_Video/Device-SpecificConsiderations/Device-
SpecificConsiderations.html

112. Questions?
can haz question?
By cloudzilla http://www.flickr.com/photos/cloudzilla/378829651/
schechter@google.com
mattward@google.com