watchOS 2.0 brings media functionality to Apple Watch, offering audio and video playback and audio capture. But lest you plan on writing Logic or Final Cut for the watch: what's available on the wrist has its limits, and you hit them quickly. In this session, we'll see what the WKInterfaceController offers us for miniature mobile media, and how we can get the benefits of AV Foundation and Core Audio by moving our movies, songs, and podcasts back and forth between the watch and the iPhone.

1. Video Killed The Rolex
Star
Chris Adamson • @invalidname
CocoaConf Columbus • July, 2015

2. Disclaimer/Apology

5. Media Support in
watchOS 2.0

6. watchOS 2.0
• App Extension runs on watch, not on iPhone
• New watchOS APIs for media playback and
recording
• Prepare yourself, they’re limited!

7. From watchOS 2.0 Transition Guide
You must implement your extension using
the frameworks in the watchOS SDK instead
of the iOS SDK. For any features not
available in the provided frameworks, you
must rely on your iPhone app to perform the
corresponding task.

8. From watchOS 2.0 Transition Guide
Your extension now stores files and data on
Apple Watch. Any data that is not part of your
Watch app or WatchKit extension bundle must
be fetched the network or from the companion
iOS app running on the user’s iPhone. You
cannot rely on a shared group container to
exchange files with your iOS app. Fetching files
involves transferring them wirelessly to Apple
Watch.

9. Media files. The Watch app handles audio
and video playback in your app. If your
WatchKit extension downloads media files
from the network or the companion iOS app,
you must place those files in a shared group
container that is accessible to both your Watch
app and WatchKit extension. For more
information about managing media-related
files, see Managing Your Media
From watchOS 2.0 Transition Guide

10. Media functionality
• Video playback
• Audio playback
• Audio recording

11. Video Playback
• WKInterfaceMovie — Canned UI component for
movie playback
• WKInterfaceController — A/V features provided
by primary UI controller class

12. WKInterfaceMovie
A WKInterfaceMovie object lets you play
back video and audio content directly from
your interface. A movie object displays a
poster image with a play button on top of it.
When the user taps the play button, WatchKit
plays the movie in a modal interface.

14. // WKInterfaceMovie.h
// WatchKit
WK_AVAILABLE_WATCHOS_ONLY(2.0)
@interface WKInterfaceMovie : WKInterfaceObject
- (void)setMovieURL:(NSURL *)URL;
- (void)setVideoGravity:(WKVideoGravity)videoGravity; // default is
WKVideoGravityResizeAspect
- (void)setLoops:(BOOL)loops;
- (void)setPosterImage:(nullable WKImage *)posterImage;
@end

15. Video Gravity
• Conceptually identical to video gravity constants in AV
Foundation
• Resize — stretch pixels to fill container
• Aspect (Fit) — honoring aspect ratio, scale to reach
one set of bounds (top/bottom or right/left), then
letter-/pillar-box
• Aspect Fill — honoring aspect ratio, scale to reach
both sets of bounds, allowing contents to be
clipped if needed

16. Original 16:9 Frame

17. WKVideoGravity.
ResizeAspect
This is the default for WKInterfaceMovie.videoGravity

18. WKVideoGravity.
ResizeAspectFill

19. WKVideoGravity.Resize
Please never do this

20. WKInterfaceController
• Media playback and recording methods
provided by the base controller class
• Can use these to play video whenever the app
decides it’s time to do so

21. - (void)presentMediaPlayerControllerWithURL:(NSURL *)URL
options:(nullable NSDictionary *)options
completion:(void(^)(BOOL didPlayToEnd,
NSTimeInterval endTime,
NSError * __nullable error))completion
WK_AVAILABLE_WATCHOS_ONLY(2.0);
- (void)dismissMediaPlayerController WK_AVAILABLE_WATCHOS_ONLY(2.0);
options dictionary of presentMediaPlayerController takes
WKVideoGravity key, uses the WKVideoGravity constants

22. Video Considerations

23. 1280x720 320x180
921,600 pixels 57,600 pixels
1/16 the size!

24. Encoding Guidelines
• Video Codec: H.264 High profile
• Bitrate: 160 kbps, 30 frames/sec
• Size: 320x180 (landscape), 208x260 (portrait)
• Audio: 32 kbps

25. 160 kbps video

26. 93% smaller file size!

27. WKInterfaceController Audio
• Playback works just like video
• Same recommendation for audio bitrate: 32
kbps
• Audio playback always uses Bluetooth
headphones/speakers if paired, otherwise
internal speaker

28. Audio Recording
-(void)presentAudioRecordingControllerWithOutputURL:(NSURL *)URL
preset:(WKAudioRecordingPreset)preset
maximumDuration:(NSTimeInterval)maximumDuration
actionTitle:(nullable NSString *)actionTitle
completion:(void (^)(BOOL didSave,
NSError * __nullable error))completion
WK_AVAILABLE_WATCHOS_ONLY(2.0);
- (void)dismissAudioRecordingController
WK_AVAILABLE_WATCHOS_ONLY(2.0);

29. presentAudioRecording
ControllerWithOutputURL:
• actionTitle — A string to use in an “end recording”
button once audio capture is underway
• preset — WKAudioRecording quality preset
• NarrowBandSpeech (8 kHz sampling, 24 kbps
AAC)
• WideBandSpeech (16 kHz sampling, 32 kbps AAC)
• HighQualityAudio (44.1 kHz sampling, 96 kbps
AAC)

30. Audio Player
• “Headless” API for playing audio
programmatically
• Assumes app provides own UI, or doesn’t
have one

31. WKAudioFileAsset
+ (instancetype)assetWithURL:(NSURL *)URL;
+ (instancetype)assetWithURL:(NSURL *)URL
title:(nullable NSString *)title
albumTitle:(nullable NSString *)albumTitle
artist:(nullable NSString *)artist;

32. WKAudioFilePlayerItem
• status — .Unknown, .ReadyToPlay, .Failed
• Notifications — Time jumped, Played to End,
Failed to Play to End
+ (WKAudioFilePlayerItem *)playerItemWithAsset:(WKAudioFileAsset
*)asset;
@property (nonatomic, readonly) WKAudioFileAsset *asset;
@property (nonatomic, readonly) WKAudioFilePlayerItemStatus status;
@property (nonatomic, readonly, nullable) NSError *error;
@property (nonatomic, readonly) NSTimeInterval currentTime;

33. WKAudioFilePlayer
+ (instancetype)playerWithPlayerItem:
(WKAudioFilePlayerItem *)item;
- (void)play;
- (void)pause;
- (void)replaceCurrentItemWithPlayerItem:
(nullable WKAudioFilePlayerItem *)item;
@property(nonatomic, readonly, nullable) WKAudioFilePlayerItem
*currentItem;
@property (nonatomic, readonly) WKAudioFilePlayerStatus
status;
@property (nonatomic, readonly, nullable) NSError *error;
@property (nonatomic) float rate;
@property (nonatomic, readonly) NSTimeInterval currentTime;

34. WKAudioFileQueuePlayer
@interface WKAudioFileQueuePlayer : WKAudioFilePlayer
+ (instancetype)queuePlayerWithItems:
(NSArray<WKAudioFilePlayerItem *> *)items;
- (void)advanceToNextItem;
- (void)appendItem:(WKAudioFilePlayerItem *)item;
- (void)removeItem:(WKAudioFilePlayerItem *)item;
- (void)removeAllItems;
@property(nonatomic, readonly) NSArray<WKAudioFilePlayerItem
*> *items;
@end

35. And that’s it!

36. Video Killed The Rolex
Star
Chris Adamson • @invalidname
CocoaConf Columbus • July, 2015
Slides available at slideshare.net/invalidname
Code (eventually) at github.com/invalidname

37. Now wait a darn
minute!

39. AVFoundation, Core Image, and Core Audio are
huge and complex, but required for lots of app
types. Will any audio, video, or image APIs be
available? Will they only be possible through
limited, high-level interfaces?
http://www.marco.org/2015/05/28/watch-sdk-questions

40. Available System Technologies
Extensions built specifically for watchOS 2 have access to the
following system frameworks:
ClockKit
Contacts
Core Data
Core Foundation
Core Graphics
Core Location
Core Motion
EventKit
Foundation
HealthKit
HomeKit
ImageIO
MapKit
Mobile Core Services
PassKit
Security
Watch Connectivity
WatchKit
Notice the absence of AV Foundation, Core Audio, Core
Media, and Core Video

41. From watchOS 2.0 Transition Guide
You must implement your extension using
the frameworks in the watchOS SDK instead
of the iOS SDK. For any features not
available in the provided frameworks, you
must rely on your iPhone app to perform the
corresponding task.

42. overcast.fm

44. AUGraph
AUFilePlayer AURemoteIO
AVAudioEngine is conceptually similar, but can’t do a
step we need later, so this is the Core Audio approach

45. AUGraph
AUFilePlayer AURemoteIO
AUNew
TimePitch

46. Offline AUGraphs
AUFilePlayer
AUGeneric
Output
AUNew
TimePitch
AudioUnit
Render()
+
ExtAudioFile
Write()

47. OSStatus timeShift(NSURL *inSourceURL, NSURL *inDestinationURL, float inSpeed) {
OSStatus err = noErr;
// crate graph
AUGraph auGraph;
err = NewAUGraph(&auGraph);
if (err != noErr) {goto fail;} // goto fail, go directly to fail...
AudioComponentDescription compDesc = {0};
// file player
NSLog (@"Making AUFilePlayer");
AUNode filePlayerNode;
AudioUnit filePlayerUnit;
compDesc.componentType = kAudioUnitType_Generator;
compDesc.componentSubType = kAudioUnitSubType_AudioFilePlayer;
compDesc.componentManufacturer = kAudioUnitManufacturer_Apple;
err = AUGraphAddNode(auGraph, &compDesc, &filePlayerNode);
if (err != noErr) {goto fail;} // goto fail, go directly to fail...
err = AUGraphNodeInfo(auGraph, filePlayerNode, NULL, &filePlayerUnit);
if (err != noErr) {goto fail;} // goto fail, go directly to fail...
// AUNewTimePitch
NSLog (@"Making AUNewTimePitch");
AUNode timePitchNode;
AudioUnit timePitchUnit;
memset(&compDesc, 0, sizeof(compDesc));
compDesc.componentType = kAudioUnitType_FormatConverter;
compDesc.componentSubType = kAudioUnitSubType_NewTimePitch;
compDesc.componentManufacturer = kAudioUnitManufacturer_Apple;
// and another 100 lines or so of this!

48. ?

49. Place media files that you download from
the network (or transfer from your iOS
app) in a shared group container. Both your
Watch app and WatchKit extension must
have access to the shared group container. In
your extension code, create URLs for any
media files inside the container and use them
to configure the media interfaces.
From watchOS 2.0 Transition Guide

50. NSFileManager.containerURLForSecurityApplicationGroupIdentifier

58. Phone: Write to Container
if let container =
NSFileManager.defaultManager().containerURLForSecurityApplication
GroupIdentifier (
"group.com.cocoaconf.extensionclass.CocoaConfExtensions") {
let fileURL = container.URLByAppendingPathComponent(
"podcast-file-with-effects.m4a")
try NSFileManager.defaultManager().copyItemAtURL(original,
toURL: fileURL)
// TODO: handle possible write error
}

59. Watch: Play from container
if let container =
NSFileManager.defaultManager().containerURLForSecurityApplication
GroupIdentifier (
"group.com.cocoaconf.extensionclass.CocoaConfExtensions") {
let fileURL = container.URLByAppendingPathComponent(
“podcast-file-with-effects.m4a“)
if NSFileManager.defaultManager().fileExistsAtPath(
fileURL.path!) {
// got it!
let audioAsset = WKAudioFileAsset(URL: fileURL)
let playerItem = WKAudioFilePlayerItem(asset: audioAsset)
self.player = WKAudioFilePlayer(playerItem: playerItem)
self.player.play()
}
}

60. What about video?

63. AVAssetExportSession
AVAssetExportSession

64. Export Preset Names for Apple Devices
You use these export options to produce files that can be played on the specific
Apple devices.
Declaration
SWIFT
let AVAssetExportPresetAppleM4VCellular: String
let AVAssetExportPresetAppleM4ViPod: String
let AVAssetExportPresetAppleM4V480pSD: String
let AVAssetExportPresetAppleM4VAppleTV: String
let AVAssetExportPresetAppleM4VWiFi: String
let AVAssetExportPresetAppleM4V720pHD: String
let AVAssetExportPresetAppleM4V1080pHD: String
let AVAssetExportPresetAppleProRes422LPCM: String

65. AVAssetWriter
• Low-level access for writing media files
• Allows you to specify output size, encoding
settings, bitrate, etc.
• Requires you to write each CMSampleBuffer
individually

66. AVAssetWriterInput
SWIFT
let AVVideoCodecKey: String
let AVVideoCodecH264: String
let AVVideoCodecJPEG: String
let AVVideoCodecAppleProRes4444: String
let AVVideoCodecAppleProRes422: String
let AVVideoWidthKey: String
let AVVideoHeightKey: String
let AVVideoCompressionPropertiesKey: String
let AVVideoAverageBitRateKey: String
let AVVideoQualityKey: String
let AVVideoMaxKeyFrameIntervalKey: String
let AVVideoProfileLevelKey: String
let AVVideoProfileLevelH264Baseline30: String
let AVVideoProfileLevelH264Baseline31: String
let AVVideoProfileLevelH264Baseline41: String
let AVVideoProfileLevelH264Main30: String
let AVVideoProfileLevelH264Main31: String
let AVVideoProfileLevelH264Main32: String
let AVVideoProfileLevelH264Main41: String
let AVVideoProfileLevelH264High40: String
let AVVideoProfileLevelH264High41: String
let AVVideoPixelAspectRatioKey: String
let AVVideoPixelAspectRatioHorizontalSpacingKey:
String
let AVVideoPixelAspectRatioVerticalSpacingKey:
String
let AVVideoCleanApertureKey: String
let AVVideoCleanApertureWidthKey: String
let AVVideoCleanApertureHeightKey: String
let AVVideoCleanApertureHorizontalOffsetKey:
String
let AVVideoCleanApertureVerticalOffsetKey:
String
Video Settings
These constants define dictionary keys for configuring video
compression and compression settings for video assets.
- initWithMediaType:outputSettings:sourceFormatHint:

67. AVAssetReader
Output
AVAssetReader
Output
AVAssetWriter
Input
AVAssetWriter
Input
Audio path (output settings to change bitrate)
Video path (output settings to change size, encoding, bitrate)

68. Takeaways
• Basic support for file-based audio/video playback
and audio recording
• Playback files are either in your bundle or
downloaded by your iOS app + extension
• Any downloading or media processing needs to be
performed on your iPhone, then sent to watch
extension/app
• NSFileManager.containerURLForSecurity
ApplicationGroupIdentifier()

69. Video Killed The Rolex
Star
Chris Adamson • @invalidname
CocoaConf Columbus • July, 2015
Slides available at slideshare.net/invalidname
Code (eventually) at github.com/invalidname