1. iOS Media APIs
Chris Adamson • @invalidname
MobiDevDay • Detroit, MI • May 4, 2013
Slides will be posted to slideshare.net/invalidname
Thursday, May 2, 13

2. Where do I start?
Thursday, May 2, 13

3. iOS Media APIs
• AV Foundation
• Core Media
• Core Animation
• Media Player
• Core Audio
• Audio Toolbox
• Audio Units
• Core MIDI
• Open AL
Thursday, May 2, 13

4. Also!
• UIKit
• UIImagePickerController, remote control
events
• AirPlay
• HTTP Live Streaming
Thursday, May 2, 13

5. How Do I?
• Play music on my title
screen?
• Play a video in my app?
• Capture, edit, and
export video?
• Play streaming video
from my web site?
• Play user’s iTunes music?
• Play a video on an
AppleTV?
• Play web radio?
• Mix and perform effects
on audio?
• Use MIDI devices?
• Create in-game sounds
for a 3D game?
Thursday, May 2, 13

6. How Do I?
• Play music on my title
screen?
• Play a video in my app?
• Capture, edit, and
export video?
• Play streaming video
from my web site?
• Play user’s iTunes music?
• Play a video on an
AppleTV?
• Play web radio?
• Mix and perform effects
on audio?
• Use MIDI devices?
• Create in-game sounds
for a 3D game?
Thursday, May 2, 13

7. AV Foundation
Thursday, May 2, 13

8. AV Foundation
• High-level Obj-C audio/video framework
• Debuted in iPhone OS 2.3, not fully baked
until iOS 4
• Consists of three distinct areas of
functionality with little overlap
Thursday, May 2, 13

9. AVAudioPlayer
AVAudioRecorder
AVAudioSession
AVAudioSessionChannelDescription
AVAudioSessionPortDescription
AVAudioSessionRouteDescription
AVCaptureAudioDataOutput
AVCaptureConnection
AVCaptureDevice
AVCaptureDeviceInput
AVCaptureFileOutput
AVCaptureInput
AVCaptureMetadataOutput
AVCaptureMovieFileOutput
AVCaptureOutput
AVCaptureSession
AVCaptureStillImageOutput
AVCaptureVideoDataOutput
AVCaptureVideoPreviewLayer
AVAsset
AVAssetExportSession
AVAssetImageGenerator
AVAssetReader
AVAssetReaderAudioMixOutput
AVAssetReaderOutput
AVAssetReaderTrackOutput
AVAssetReaderVideoCompositionOutput
AVAssetResourceLoader
AVAssetResourceLoadingRequest
AVAssetTrack
AVAssetTrackSegment
AVAssetWriter
AVAssetWriterInput
AVAssetWriterInputPixelBufferAdaptor
AVAudioMix
AVAudioMixInputParameters
AVComposition
AVCompositionTrack
AVCompositionTrackSegment
AVMediaSelectionGroup
AVMediaSelectionOption
AVMetadataFaceObject
AVMetadataItem
AVMetadataObject
AVMutableAudioMix
AVMutableAudioMixInputParameters
AVMutableComposition
AVMutableCompositionTrack
AVMutableMetadataItem
AVMutableTimedMetadataGroup
AVMutableVideoComposition
AVMutableVideoCompositionInstruction
AVMutableVideoCompositionLayerInstruction
AVPlayer
AVPlayerItem
AVPlayerItemAccessLog
AVPlayerItemAccessLogEvent
AVPlayerItemErrorLog
AVPlayerItemErrorLogEvent
AVPlayerItemOutput
AVPlayerItemTrack
AVPlayerItemVideoOutput
AVPlayerLayer
AVQueuePlayer
AVSynchronizedLayer
AVTextStyleRule
AVTimedMetadataGroup
AVURLAsset
AVVideoComposition
AVVideoCompositionCoreAnimationTool
AVVideoCompositionInstruction
AVVideoCompositionLayerInstruction
Audio
Capture
Editing / Export / Playback
Thursday, May 2, 13

10. AVF audio classes
• AVAudioPlayer – plays flat files or audio
from an NSData
• AVAudioRecorder – records from most-
recently connected input device (built-in
mic, headset, etc.)
• AVAudioSession – negotiates with system
for access to audio hardware
Thursday, May 2, 13

11. AVAudioPlayer
• initWithContentsOfURL:error: or
initWithData:error:
• URL must be local
• Methods: play, playAtTime:, pause, stop
• Properties: volume, pan, numberOfLoops,
etc.
Thursday, May 2, 13

12. Playing a song
NSURL *songURL = [[NSBundle mainBundle]
URLForResource:@"Bossa Lounger Long"
! ! ! ! ! ! ! withExtension:@"caf"];
// set up av audio player
NSError *err = nil;
self.player = [[AVAudioPlayer alloc]
initWithContentsOfURL:songURL
error:&err];
if (err) {
! NSLog (@"Error creating player: %@", err);
} else {
! [self.player play];
}
Thursday, May 2, 13

13. How Do I?
• Play music on my title
screen?
• Play a video in my app?
• Capture, edit, and
export video?
• Play streaming video
from my web site?
• Play user’s iTunes music?
• Play a video on an
AppleTV?
• Play web radio?
• Mix and perform effects
on audio?
• Use MIDI devices?
• Create in-game sounds
for a 3D game?
Thursday, May 2, 13

14. Video Playback
• Two options:AVPlayer and
MPMoviePlayerController
• Since we’re already talking about AV
Foundation, let’s do AVPlayer and friends
for now
Thursday, May 2, 13

15. AVF essentials
• AVAsset – A time-based, playable item
(local or remote)
• AVPlayer – Handles playback of one or
more AVPlayerItems
• Each AVPlayerItem is associated with an
AVAsset
• Player does play/pause, seekToTime:, etc.
Thursday, May 2, 13

16. AVPlayer video
• AVPlayerLayer – A CALayer to present
video from an AVLayer
• CALayer is not a UIResponder, doesn’t
handle touches.You need to provide your
own playback UI.
• gravity property determines “stretching”
of video to fit layer’s bounds
Thursday, May 2, 13

17. Playing a video
AVURLAsset *asset = [AVURLAsset URLAssetWithURL:url
! ! ! ! ! ! ! options:nil];
AVPlayerItem *playerItem = [AVPlayerItem playerItemWithAsset:asset];
self.player = [AVPlayer playerWithPlayerItem:playerItem];
NSArray *visualTracks = [asset tracksWithMediaCharacteristic:
AVMediaCharacteristicVisual];
if ([visualTracks count] > 0) {
! AVPlayerLayer *playerLayer = [AVPlayerLayer playerLayerWithPlayer:player];
! [playerView.layer addSublayer:playerLayer];
! playerLayer.frame = playerView.layer.bounds;
! playerLayer.videoGravity = AVLayerVideoGravityResizeAspect;
}
Thursday, May 2, 13

18. How Do I?
• Play music on my title
screen?
• Play a video in my app?
• Capture, edit, and
export video?
• Play streaming video
from my web site?
• Play user’s iTunes music?
• Play a video on an
AppleTV?
• Play web radio?
• Mix and perform effects
on audio?
• Use MIDI devices?
• Create in-game sounds
for a 3D game?
Thursday, May 2, 13

19. AVF Capture
• AVCaptureSession – Coordinates capture
activities
• Discover AVCaptureDevices (mics,
camera), create AVCaptureInputs from
them, connect to session
• Create AVCaptureOutputs (file, data
callbacks), connect to session
Thursday, May 2, 13

20. AVF Capture// create capture session, attach default video input
self.captureSession = [[AVCaptureSession alloc] init];
NSError *setUpError = nil;
AVCaptureDevice *videoDevice = [AVCaptureDevice
defaultDeviceWithMediaType: AVMediaTypeVideo];
if (videoDevice) {
! AVCaptureDeviceInput *videoInput = [AVCaptureDeviceInput
deviceInputWithDevice:videoDevice
! error:&setUpError];
! if (videoInput) {
! ! [captureSession addInput: videoInput];
! }
}
// create a preview layer from the session and add it to UI
AVCaptureVideoPreviewLayer *previewLayer = [AVCaptureVideoPreviewLayer
layerWithSession:captureSession];
previewLayer.frame = captureView.layer.bounds;
previewLayer.videoGravity = AVLayerVideoGravityResizeAspect;
[captureView.layer addSublayer:previewLayer];
// start capture session and write to file
captureMovieOutput = [[AVCaptureMovieFileOutput alloc] init];
[captureSession addOutput:captureMovieOutput];
[captureSession startRunning];
[captureMovieOutput startRecordingToOutputFileURL:captureMovieURL
! ! ! ! ! ! recordingDelegate:self];
Thursday, May 2, 13

21. AVF Editing
• AVMutableComposition – An AVAsset for a
multi-track movie you build from
references to other AVAssets
• AVMutableCompositionTrack – Built up
with
insertTimeRange:ofTrack:atTime:error:
Thursday, May 2, 13

22. AVFVideo Effects
• Create AVMutableMutableVideoInstruction
made up of
AVMutableVideoCompositionLayerInstructi
ons
• Each instruction works with a video track
• Instruction ramps opacity or affine
transform over a time range
Thursday, May 2, 13

23. AVF Text/Image effects
• Create an AVSynchronizedLayer to show
the composition
• Create CALayers, CATextLayers and set
CAAnimations on them, then add as sub-
layers to the AVSynchronizedLayer
Thursday, May 2, 13

24. AVF Export
• Create AVAssetExportSession with one of the
canned presets (audio only, or QuickTime .mov
at preset size or quality)
• If you used layer-based animations, add an
AVVideoComposition, with an
AVVideoCompositionCoreAnimationTool
(yes, it’s as hard as it sounds)
• Call
exportAsynchronouslyWithCompletionHandler:
Thursday, May 2, 13

25. AVF Editing
-(IBAction) handlePerformTapped: (id) sender {
! NSLog (@"handlePerformTapped");
! NSError *performError = nil;
! // create composition
! self.composition = [[AVMutableComposition alloc] init];
! // create video tracks a and b
! // note: mediatypes are defined in AVMediaFormat.h
! [trackA! release];
! trackA = [self.composition addMutableTrackWithMediaType:AVMediaTypeVideo
! ! ! ! ! ! ! ! ! !
preferredTrackID:kCMPersistentTrackID_Invalid];
! [trackB release];
! trackB = [self.composition addMutableTrackWithMediaType:AVMediaTypeVideo
! ! ! ! ! ! ! ! ! !
preferredTrackID:kCMPersistentTrackID_Invalid];
// create video instructions
NSMutableArray *videoInstructions = [[[NSMutableArray alloc] init] autorelease];
! // create music track
trackMusic = [self.composition addMutableTrackWithMediaType:AVMediaTypeAudio
preferredTrackID:kCMPersistentTrackID_Invalid];
! CMTimeRange musicTrackTimeRange = CMTimeRangeMake(kCMTimeZero,
musicTrackAudioAsset.duration);
! NSError *trackMusicError = nil;
! [trackMusic insertTimeRange:musicTrackTimeRange
! ! ! ! ! ! ofTrack:[musicTrackAudioAsset.tracks objectAtIndex:0]
! ! ! ! ! ! atTime:kCMTimeZero
! ! ! ! ! ! error:&trackMusicError];
! if (trackMusicError) {
! ! NSLog(@"couldn't create trackMusic: %@", trackMusicError);
! } else {
! ! NSLog (@"created trackMusic");
! }
!
! // setup the player
! [compositionPlayer release];
! compositionPlayer = [[AVPlayer playerWithPlayerItem: [AVPlayerItem playerItemWithAsset: composition]] retain];
! [compositionPlayerLayer removeFromSuperlayer];
! compositionPlayerLayer = [[AVPlayerLayer playerLayerWithPlayer:compositionPlayer] retain];
! [compositionView.layer addSublayer:compositionPlayerLayer];
! compositionPlayerLayer.frame = compositionView.layer.bounds;
! compositionPlayerLayer.videoGravity = AVLayerVideoGravityResizeAspect;
!
! [updateScrubberTimer invalidate];
! [updateScrubberTimer release];
! updateScrubberTimer = [[NSTimer scheduledTimerWithTimeInterval:0.1
! ! ! ! ! ! ! ! ! ! ! !
! ! target:self
! ! ! ! ! ! ! ! ! ! ! !
! ! selector:@selector(updateScrubber:)
! ! ! ! ! ! ! ! ! ! ! !
! ! userInfo:nil
! ! ! ! ! ! ! ! ! ! ! !
! ! repeats:YES]
! ! ! ! ! ! retain];
// the video tracks
AVAssetTrack *sourceVideoTrack = [[sourceVideoAsset tracksWithMediaType: AVMediaTypeVideo] objectAtIndex: 0];
! // pad out the opening with five seconds of blank
CMTime videoTracksTime = CMTimeMake(0, VIDEO_TIME_SCALE);
CMTime postEditTime = CMTimeAdd (videoTracksTime, CMTimeMakeWithSeconds(FIRST_CUT_TRACK_A_IN_TIME, VIDEO_TIME_SCALE));
[trackA insertEmptyTimeRange:CMTimeRangeMake(kCMTimeZero, postEditTime)];
videoTracksTime = postEditTime;
// first shot
postEditTime = CMTimeAdd (videoTracksTime, CMTimeMakeWithSeconds(FIRST_CUT_DURATION,VIDEO_TIME_SCALE));
CMTimeRange firstShotRange = CMTimeRangeMake(kCMTimeZero, CMTimeMakeWithSeconds(FIRST_CUT_DURATION, VIDEO_TIME_SCALE));
! [trackA insertTimeRange:firstShotRange
! ! ! ! ! ofTrack:sourceVideoTrack
! ! ! ! ! atTime:videoTracksTime
! ! ! ! ! error:&performError];
videoTracksTime = postEditTime;
// track b needs to insert empty segment up to its first use
[trackB insertEmptyTimeRange:CMTimeRangeMake(kCMTimeZero, videoTracksTime)];
postEditTime = CMTimeAdd (videoTracksTime, CMTimeMakeWithSeconds(SECOND_CUT_TRACK_B_IN_TIME,VIDEO_TIME_SCALE));
CMTimeRange secondShotRange = CMTimeRangeMake(CMTimeMakeWithSeconds(SECOND_CUT_SOURCE_TIME, VIDEO_TIME_SCALE),
CMTimeMakeWithSeconds(SECOND_CUT_DURATION, VIDEO_TIME_SCALE));
[trackB insertTimeRange:secondShotRange
ofTrack:sourceVideoTrack
atTime:videoTracksTime
error:&performError];
videoTracksTime = postEditTime;
// TODO: later segments
// desperation cheese - works
AVMutableVideoCompositionInstruction *transitionInstruction = [AVMutableVideoCompositionInstruction videoCompositionInstruction];
transitionInstruction.timeRange = CMTimeRangeMake(kCMTimeZero, composition.duration);
AVMutableVideoCompositionLayerInstruction *aInstruction = [AVMutableVideoCompositionLayerInstruction
videoCompositionLayerInstructionWithAssetTrack: trackA];
[aInstruction setOpacityRampFromStartOpacity:0.0 toEndOpacity:1.0
timeRange:CMTimeRangeMake(CMTimeMakeWithSeconds(2.9, VIDEO_TIME_SCALE),
CMTimeMakeWithSeconds(6.0, VIDEO_TIME_SCALE))];
AVMutableVideoCompositionLayerInstruction *bInstruction = [AVMutableVideoCompositionLayerInstruction
videoCompositionLayerInstructionWithAssetTrack: trackB];
[bInstruction setOpacity:0 atTime:kCMTimeZero];
transitionInstruction.layerInstructions = [NSArray arrayWithObjects:aInstruction, bInstruction, nil];
[videoInstructions addObject: transitionInstruction];
// end of desperation cheese
// synchronized layer to own all the title layers
AVSynchronizedLayer *synchronizedLayer = [AVSynchronizedLayer synchronizedLayerWithPlayerItem:compositionPlayer.currentItem];
synchronizedLayer.frame = [compositionView frame];
[self.view.layer addSublayer:synchronizedLayer];
// main titles
CATextLayer *mainTitleLayer = [CATextLayer layer];
mainTitleLayer.string = NSLocalizedString(@"Running Start", nil);
mainTitleLayer.font = @"Verdana-Bold";
mainTitleLayer.fontSize = videoSize.height / 8;
mainTitleLayer.foregroundColor = [[UIColor yellowColor] CGColor];
mainTitleLayer.alignmentMode = kCAAlignmentCenter;
mainTitleLayer.frame = CGRectMake(0.0, 0.0, videoSize.width, videoSize.height);
mainTitleLayer.opacity = 0.0; // initially invisible
[synchronizedLayer addSublayer:mainTitleLayer];
// main title opacity animation
[CATransaction begin];
[CATransaction setDisableActions:YES];
CABasicAnimation *mainTitleInAnimation = [CABasicAnimation animationWithKeyPath:@"opacity"];
mainTitleInAnimation.fromValue = [NSNumber numberWithFloat: 0.0];
mainTitleInAnimation.toValue = [NSNumber numberWithFloat: 1.0];
mainTitleInAnimation.removedOnCompletion = NO;
mainTitleInAnimation.beginTime = AVCoreAnimationBeginTimeAtZero;
mainTitleInAnimation.duration = 5.0;
[mainTitleLayer addAnimation:mainTitleInAnimation forKey:@"in-animation"];
CABasicAnimation *mainTitleOutAnimation = [CABasicAnimation animationWithKeyPath:@"opacity"];
mainTitleOutAnimation.fromValue = [NSNumber numberWithFloat: 1.0];
mainTitleOutAnimation.toValue = [NSNumber numberWithFloat: 0.0];
mainTitleOutAnimation.removedOnCompletion = NO;
mainTitleOutAnimation.beginTime = 5.0;
mainTitleOutAnimation.duration = 2.0;
[mainTitleLayer addAnimation:mainTitleOutAnimation forKey:@"out-animation"];
[CATransaction commit];
// TODO: end credits
// tell the player about our effects
AVMutableVideoComposition *videoComposition = [AVMutableVideoComposition videoComposition];
videoComposition.instructions = videoInstructions;
videoComposition.renderSize = videoSize;
videoComposition.frameDuration = CMTimeMake(1, 30); // 30 fps
compositionPlayer.currentItem.videoComposition = videoComposition;
// set up the duration label at the right side of the scrubber
int durationSeconds = (int) CMTimeGetSeconds (composition.duration);
self.durationLabel.text = [NSString stringWithFormat: @"%02d:%02d",
durationSeconds / 60, durationSeconds % 60];
// reset rate field and play/pause button state
rateField.text = @"0.0";
playPauseButton.selected = NO;
! NSLog (@"bottom of handlePerformTapped");
!
}
No, that’s not supposed to be legible.
Thursday, May 2, 13

26. How Do I?
• Play music on my title
screen?
• Play a video in my app?
• Capture, edit, and
export video?
• Play streaming video
from my web site?
• Play user’s iTunes music?
• Play a video on an
AppleTV?
• Play web radio?
• Mix and perform effects
on audio?
• Use MIDI devices?
• Create in-game sounds
for a 3D game?
Thursday, May 2, 13

27. HTTP Live Streaming
Thursday, May 2, 13

28. HTTP Live Streaming
• Apple-led semi-standard streaming format
• Only stream format allowed on App Store
• Streams video as series of small (10 sec.)
files, via ordinary web server on port 80
• Create streams with command-line tools,
Pro apps (Compressor, Final Cut), server-
side transcoders (Wowza)
Thursday, May 2, 13

29. Thursday, May 2, 13

30. Thursday, May 2, 13

31. HLS Advantages
• Mobile-friendly: works over spotty cellular
connections, stream can provide multiple
bitrates (client switches on the fly)
• No special server software required: can
stream from Dropbox
• Wide adoption: Roku, Xbox,Android,
GoogleTV, etc.
Thursday, May 2, 13

32. Client-side HLS
• Create an AVPlayer or
MPMoviePlayerController with the
stream’s .m3u8 URL just like any other URL
• There is no step 2
Thursday, May 2, 13

33. How Do I?
• Play music on my title
screen?
• Play a video in my app?
• Capture, edit, and
export video?
• Play streaming video
from my web site?
• Play user’s iTunes music?
• Play a video on an
AppleTV?
• Play web radio?
• Mix and perform effects
on audio?
• Use MIDI devices?
• Create in-game sounds
for a 3D game?
Thursday, May 2, 13

34. Media Player
Thursday, May 2, 13

35. Media Player
• Allows access to the device’s iTunes library
• Audio-only: songs, podcasts, audiobooks
• Discover contents with MPMediaQuery, or
show a MPMediaPickerController
• MPMediaItems have metadata (title, artist,
album, cover art, etc.)
• Play with MPMusicPlayerController
Thursday, May 2, 13

36. MPMediaQuery
NSString *searchText = searchController.searchBar.text;
MPMediaQuery *query = [MPMediaQuery songsQuery];
MPMediaPropertyPredicate *titlePredicate =
[MPMediaPropertyPredicate predicateWithValue:searchText
! ! ! forProperty:MPMediaItemPropertyTitle
! ! ! comparisonType:
MPMediaPredicateComparisonContains];
[query addFilterPredicate:titlePredicate];
MPMusicPlayerController *iPodController =
! [MPMusicPlayerController iPodMusicPlayer];
[iPodController stop];
[iPodController setQueueWithQuery: query];
[iPodController play];
Thursday, May 2, 13

37. MPMoviePlayerController
• Simple video player, alternative to AVPlayer
• Provides its own view and controls
• Lighter-weight than AVPlayer, which helps
on really old devices (iPhone 3GS)
Thursday, May 2, 13

38. MP novelties
• MPNowPlayingInfoCenter – Access to
metadata shown on lock screen and
external media displays (e.g., in-car
entertainment systems)
• MPVolumeView – System-wide volume
slider, with AirPlay button if available
Thursday, May 2, 13

39. Remote Controls
• Receive play/pause, forward/back from
headset, dock keyboard, in-car systems,
other external devices
• -[UIApplication
beginReceivingRemoteControlEvents]
• Must be able to become first responder
Thursday, May 2, 13

40. How Do I?
• Play music on my title
screen?
• Play a video in my app?
• Capture, edit, and
export video?
• Play streaming video
from my web site?
• Play user’s iTunes music?
• Play a video on an
AppleTV?
• Play web radio?
• Mix and perform effects
on audio?
• Use MIDI devices?
• Create in-game sounds
for a 3D game?
Thursday, May 2, 13

41. AirPlay
Thursday, May 2, 13

42. AirPlay
• Wireless audio/video streaming from iOS
device to Apple TV,AirPort Express, some
speakers
• Unofficial third-party clients for Mac
(Reflector,Air Sharing), PC,Android, etc.
Thursday, May 2, 13

43. AirPlay API
• There basically isn’t one
• User either mirrors device, or uses
AirPlay menu on MPVolumeView or
MPMoviePlayerController
• You can deny (but please don’t) with
allowsAirPlay (MPMovieController),
mediaPlaybackAllowsAirPlay (UIWebView),
allowsExternalPlayback (AVPlayer)
Thursday, May 2, 13

44. Second screens
• If user connects toVGA/DVI via Dock/
Lightning adapter, or connects to AirPlay
without mirroring, your app will see a
second screen
• Discover with -[UIScreen screens], create
new UIWindow for it
• Chat on the device, video on the second
screen?
Thursday, May 2, 13

45. How Do I?
• Play music on my title
screen?
• Play a video in my app?
• Capture, edit, and
export video?
• Play streaming video
from my web site?
• Play user’s iTunes music?
• Play a video on an
AppleTV?
• Play web radio?
• Mix and perform effects
on audio?
• Use MIDI devices?
• Create in-game sounds
for a 3D game?
Thursday, May 2, 13

46. Core Audio
Thursday, May 2, 13

47. Core Audio
• C frameworks for real-time audio
processing
• Legendary performance, legendary
difficulty
• Basis of OS X pro audio apps like Logic
and GarageBand
Thursday, May 2, 13

48. Core Audio
• Audio Queue
• Audio Units
• Open AL
• Audio File Services
• Audio Converter Svcs.
• Extended Audio File
Svcs.
• Audio File Stream Svcs.
• Audio Session Svcs.
Thursday, May 2, 13

49. Core Audio
• Small number of structures and functions
• Most behavior is specified by getting and
setting properties
• All functions return OSStatus, must check
for noErr before continuing
• “Create”-style functions take pointer as a
parameter and populate it
Thursday, May 2, 13

50. Audio Queue
• Convenience API for play-out or capture
• Wrapped by AV Foundation’s
AVAudioRecorder,AVAudioPlayer for file-
only scenarios
• For play-out, app provides buffers to play
• For captuer, queue gives app with buffers of
capture data
Thursday, May 2, 13

51. Audio Queue
Thursday, May 2, 13

52. Audio Queue
Thursday, May 2, 13

53. Audio Queue
Thursday, May 2, 13

54. Audio Queue
Thursday, May 2, 13

55. Parsing Web Radio
Thursday, May 2, 13

56. Parsing Web Radio
NSData NSData
Packets Packets Packets Packets Packets
NSURLConnection delivers
NSData buffers, containing audio
and framing info.We pass it to
Audio File Services.
Thursday, May 2, 13

57. Parsing Web Radio
NSData NSData
Packets Packets Packets Packets Packets
Packets Packets
Packets Packets Packets
NSURLConnection delivers
NSData buffers, containing audio
and framing info.We pass it to
Audio File Services.
Audio File Services calls us back
with parsed packets of audio data.
Thursday, May 2, 13

58. Parsing Web Radio
NSData NSData
Packets Packets Packets Packets Packets
Packets Packets
Packets Packets Packets
012
Packets
Packets
Packets
Packets
Packets
Packets
NSURLConnection delivers
NSData buffers, containing audio
and framing info.We pass it to
Audio File Services.
Audio File Services calls us back
with parsed packets of audio data.
We create an AudioQueueBuffer
with those packets and enqueue it
for play-out.
Thursday, May 2, 13

59. How Do I?
• Play music on my title
screen?
• Play a video in my app?
• Capture, edit, and
export video?
• Play streaming video
from my web site?
• Play user’s iTunes music?
• Play a video on an
AppleTV?
• Play web radio?
• Mix and perform effects
on audio?
• Use MIDI devices?
• Create in-game sounds
for a 3D game?
Thursday, May 2, 13

60. Audio Units
Thursday, May 2, 13

61. Audio Units
• Lowest level of audio processing available
to third parties
• I/O, effects, mixing, file player, synthesis
• Extremely low latency I/O (< 10 ms)
Thursday, May 2, 13

62. Pull Model
AUSomething
Thursday, May 2, 13

63. Pull Model
AUSomething
AudioUnitRender()
Thursday, May 2, 13

64. Pull Model
AUSomethingAUSomethingElse
Thursday, May 2, 13

65. Buses (aka, Elements)
AUSomething
AUSomethingElse
AUSomethingElse
Thursday, May 2, 13

66. AUGraph
AUSomething
AUSomethingElse
AUSomethingElse
Thursday, May 2, 13

67. AURemoteIO Buses
AURemoteIO
Thursday, May 2, 13

68. AURemoteIO Buses
AURemoteIO
bus 0
to output H/W
Thursday, May 2, 13

69. AURemoteIO Buses
AURemoteIO
bus 0
to output H/W
bus 0
from app
Thursday, May 2, 13

70. AURemoteIO Buses
AURemoteIO
bus 0
to output H/W
bus 1
from input H/W
bus 0
from app
Thursday, May 2, 13

71. AURemoteIO Buses
AURemoteIO
bus 0
to output H/W
bus 1
from input H/W
bus 1
to app
bus 0
from app
Thursday, May 2, 13

72. Pass Through
AURemoteIO
bus 1
from input H/W
bus 0
to output H/W
Thursday, May 2, 13

73. Pass-Through with Effect
bus 0
to output H/W
AURemoteIO
AUEffect
bus 1
from input H/W
Thursday, May 2, 13

74. Core Audio code
-(void) setUpAUGraph {
! if (self.auGraph) {
! ! CheckError(AUGraphClose(self.auGraph),
! ! ! ! "Couldn't close old AUGraph");
! ! CheckError (DisposeAUGraph(self.auGraph),
! ! ! ! ! "Couldn't dispose old AUGraph");
! }
!
! CheckError(NewAUGraph(&_auGraph),
! ! ! "Couldn't create new AUGraph");
!
! CheckError(AUGraphOpen(self.auGraph),
! ! ! "Couldn't open AUGraph");
!
! AudioComponentDescription outputcd = {0};
! outputcd.componentType = kAudioUnitType_Output;
! outputcd.componentSubType = kAudioUnitSubType_RemoteIO;
! outputcd.componentManufacturer = kAudioUnitManufacturer_Apple;
!
! AUNode ioNode;
! CheckError(AUGraphAddNode(self.auGraph,
! ! ! ! &outputcd,
! ! ! ! &ioNode),
! ! ! "couldn't add remote io node");
!
! // get the remote io unit from the node
! CheckError(AUGraphNodeInfo(self.auGraph,
! ! ! ! ioNode,
! ! ! ! NULL,
! ! ! ! &_ioUnit),
! ! ! "couldn't get remote io unit");
!
!
! // effect unit here
! AudioComponentDescription effectcd = {0};
! effectcd.componentType = kAudioUnitType_FormatConverter;
! effectcd.componentSubType = kAudioUnitSubType_NewTimePitch;
! effectcd.componentManufacturer = kAudioUnitManufacturer_Apple;
!
! AUNode effectNode;
! CheckError(AUGraphAddNode(self.auGraph,
! ! ! ! &effectcd,
! ! ! ! &effectNode),
! ! ! "couldn't get effect node [time/pitch]");
!
! // get effect unit from the node
! CheckError(AUGraphNodeInfo(self.auGraph,
! ! ! ! effectNode,
! ! ! ! NULL,
! ! ! ! &_effectUnit),
! ! ! "couldn't get effect unit from node");
!
! // enable input on the remote io unit
UInt32 oneFlag = 1;
UInt32 busZero = 0;
! CheckError(AudioUnitSetProperty(self.ioUnit,
! ! ! ! kAudioOutputUnitProperty_EnableIO,
! ! ! ! kAudioUnitScope_Output,
! ! ! ! busZero,
! ! ! ! &oneFlag,
! ! ! ! sizeof(oneFlag)),
! ! ! "Couldn't enable output on bus 0");
! UInt32 busOne = 1;
! CheckError(AudioUnitSetProperty(self.ioUnit,
! ! ! ! kAudioOutputUnitProperty_EnableIO,
! ! ! ! kAudioUnitScope_Input,
! ! ! ! busOne,
! ! ! ! &oneFlag,
! ! ! ! sizeof(oneFlag)),
! ! ! "Couldn't enable input on bus 1");
!
! // set stream format that the effect wants
! AudioStreamBasicDescription streamFormat;
! UInt32 propertySize = sizeof (streamFormat);
! CheckError(AudioUnitGetProperty(self.effectUnit,
kAudioUnitProperty_StreamFormat,
! ! ! ! ! kAudioUnitScope_Input,
! ! ! ! ! 0,
! ! ! ! ! &streamFormat,
! ! ! ! ! &propertySize),
! ! ! "Couldn't get effect unit stream format");
!
! CheckError(AudioUnitSetProperty(self.ioUnit,
kAudioUnitProperty_StreamFormat,
! ! ! ! ! kAudioUnitScope_Output,
! ! ! ! ! busOne,
! ! ! ! ! &streamFormat,
! ! ! ! ! sizeof(streamFormat)),
! ! ! "couldn't set stream format on iounit bus 1 output");
! CheckError(AudioUnitSetProperty(self.ioUnit,
! kAudioUnitProperty_StreamFormat,
! ! ! ! kAudioUnitScope_Input,
! ! ! ! ! busZero,
! ! ! ! ! &streamFormat,
! ! ! ! ! sizeof(streamFormat)),
! ! ! "couldn't set stream format on iounit bus 0 input");
! !
!
! CheckError(AUGraphConnectNodeInput(self.auGraph,
! ! ! ! ! ioNode,
! ! ! 1,
! ! ! ! ! effectNode,
! ! ! ! ! 0),
! ! ! "couldn't connect remoteio bus 1 output to effect bus 0 input");
!
! CheckError(AUGraphConnectNodeInput(self.auGraph,
! ! ! ! ! effectNode,
! ! ! ! ! 0,
! ! ! ! ! ioNode,
! ! ! ! ! 0),
! ! ! "couldn't connect effect bus 0 output to remoteio bus 0 input");
!
!
! CheckError(AUGraphInitialize(self.auGraph),
! ! ! "Couldn't initialize AUGraph");
!
! CheckError(AUGraphStart(self.auGraph),
! ! ! "Couldn't start AUGraph");
!
!
!
! NSLog (@"bottom of setUpAUGraph");
}
No, this isn’t supposed to be readable either
Thursday, May 2, 13

75. How Do I?
• Play music on my title
screen?
• Play a video in my app?
• Capture, edit, and
export video?
• Play streaming video
from my web site?
• Play user’s iTunes music?
• Play a video on an
AppleTV?
• Play web radio?
• Mix and perform effects
on audio?
• Use MIDI devices?
• Create in-game sounds
for a 3D game?
Thursday, May 2, 13

76. Core MIDI
Thursday, May 2, 13

77. Core MIDI
• Handles MIDI events from external devices
• Connect via Dock/Lightning connectors
or Camera Connection Kit (USB)
• Only provides callbacks on events (key up,
key down, pitch bend, etc.) — up to you do
something with it
Thursday, May 2, 13

78. MIDI Packets
• High nybble of status is command, low is
channel number
• Data 1 & 2 depend on command
STATUS DATA 1 DATA 2
Thursday, May 2, 13

79. Handling MIDI packetsstatic void! MyMIDIReadProc(const MIDIPacketList *pktlist, void *refCon, void
*connRefCon) {
! SNFMasterViewController *myVC = (__bridge SNFMasterViewController*) refCon;
! MIDIPacket *packet = (MIDIPacket *)pktlist->packet;!
! for (int i=0; i < pktlist->numPackets; i++) {
! ! Byte midiStatus = packet->data[0];
! ! Byte midiCommand = midiStatus >> 4;
! ! // is it a note-on or note-off
! ! if ((midiCommand == 0x09) ||
! ! ! (midiCommand == 0x08)) {
! ! ! Byte note = packet->data[1] & 0x7F;
! ! ! Byte velocity = packet->data[2] & 0x7F;
! ! ! printf("midiCommand=%d. Note=%d, Velocity=%dn",
midiCommand, note, velocity);
! ! !
! ! ! // send to augraph
! ! ! CheckError(MusicDeviceMIDIEvent (myVC.auSampler,
! ! ! ! ! ! ! ! ! ! ! midiStatus,
! ! ! ! ! ! ! ! ! ! ! note,
! ! ! ! ! ! ! ! ! ! ! velocity,
! ! ! ! ! ! ! ! ! ! ! 0),
! ! ! ! ! "Couldn't send MIDI event");
! ! !
! ! }
! ! packet = MIDIPacketNext(packet);
! }
}
Thursday, May 2, 13

80. How Do I?
• Play music on my title
screen?
• Play a video in my app?
• Capture, edit, and
export video?
• Play streaming video
from my web site?
• Play user’s iTunes music?
• Play a video on an
AppleTV?
• Play web radio?
• Mix and perform effects
on audio?
• Use MIDI devices?
• Create in-game sounds
for a 3D game?
Thursday, May 2, 13

81. OpenAL
Thursday, May 2, 13

82. OpenAL
• Cross-platform C API to create positional
sound
• Designed to resemble OpenGL
• Low latency, ideal for games
• Calls set an error flag, must check with
alGetError() after every AL call
Thursday, May 2, 13

83. AL example
// set up OpenAL source
alGenSources(1, player.sources);
alSourcei(player.sources[0], AL_LOOPING, AL_TRUE);
alSourcef(player.sources[0], AL_GAIN, AL_MAX_GAIN);
updateSourceLocation(player);
// connect buffer to source
alSourcei(player.sources[0], AL_BUFFER, buffers[0]);
// set up listener
alListener3f (AL_POSITION, 0.0, 0.0, 0.0);
// start playing
alSourcePlay(player.sources[0]);
Note: error checking removed for clarity
Thursday, May 2, 13

84. OpenAL Concepts
• ALListener – The listener, who has an x,y,z
position, orientation, etc.
• ALSource – A sound producing object in
the space. Has position, orientation,
motion, sound cone, much more
• ALBuffer – Buffers of sound attached to a
source. Source can loop one buffer or
receive a stream of buffers
Thursday, May 2, 13

85. How Did I?
• Play music on my title
screen?
• Play a video in my app?
• Capture, edit, and
export video?
• Play streaming video
from my web site?
• Play user’s iTunes music?
• Play a video on an
AppleTV?
• Play web radio?
• Mix and perform effects
on audio?
• Use MIDI devices?
• Create in-game sounds
for a 3D game?
Thursday, May 2, 13

86. iOS Media APIs
• There are a LOT of them!
• Each fills a specific role, not a lot of overlap
• Simple stuff is easy, complex stuff is possible
Thursday, May 2, 13

87. Questions?
• “Audio andVideo Starting Point” in Apple
developer documentation
• http://devforums.apple.com
• coreaudio-api@lists.apple.com
Thursday, May 2, 13

88. iOS Media APIs
Chris Adamson • @invalidname
MobiDevDay • Detroit, MI • May 4, 2013
Slides will be posted to slideshare.net/invalidname
Thursday, May 2, 13