11. Core Audio
Why?
"Easy" and "CoreAudio" can't be used in the
same sentence. CoreAudio is very powerful,
very complex, and under-documented. Be
prepared for a steep learning curve, APIs with
millions of tiny little pieces, and puzzling things
out from sample code rather than reading
high-level documentation.
–Jens Alfke, coreaudio-api list
Feb 9, 2009
12. Core Audio
Why?
• Problem domain is hard
• Performance is hard
• Low latency is hard
• Reusability is hard
13. Core Audio
Why?
• Doing without would suck
• Slowness would suck
• Latency would suck
• Non-reusability would suck
22. Core Audio Overview
The Canonical Format
• AudioSampleType
• Used for I/O
• 32-bit float (Mac)
• 16-bit integer (iPhone)
• AudioUnitSampleType
• Used for DSP
• 32-bit float (Mac)
• 8.24 fixed (iPhone)
23. Core Audio Overview
AudioStreamBasicDescription - “ASBD”
• Contains the minimal information needed to describe
audio data
• Some formats may not use all of the fields
• Unused fields need to be set to zero
struct AudioStreamBasicDescription {
Float64 mSampleRate;
UInt32 mFormatID;
UInt32 mFormatFlags;
UInt32 mBytesPerPacket;
UInt32 mFramesPerPacket;
UInt32 mBytesPerFrame;
UInt32 mChannelsPerFrame;
UInt32 mBitsPerChannel;
UInt32 mReserved;
};
24. Core Audio Overview
Properties
• Key/Value pair used to describe/manipulate API attributes.
• Scope and element selectors are used by some API to further
qualify a property
• The value of a property can be of whatever type the API needs.
• APIs share several common functions
• GetProperty, SetProperty, and GetPropertyInfo
• AddPropertyListener and RemovePropertyListener
25. Core Audio Overview
Scopes and Elements
• An element is the same as a bus
input scope
element 0
global scope
output scope output scope
element 0 element 1
26. Core Audio Overview
AudioFormat
• Provides information about installed codecs
• Fills out ASBDs based on Format ID
• Provides more information about a formatʼs parameters
27. Core Audio Overview
AudioFormat
• ASBDs can be complicated, let the system do the work for
you!
asbd.mSampleRate = 44100.0;
asbd.mFormatID = kAudioFormatMPEG4AAC;
asbd.mChannelsPerFrame = 2;
UInt32 propSize = sizeof(AudioStreamBasicDescription);
AudioFormatGetProperty(kAudioFormatProperty_FormatInfo,
0, NULL,
!
! ! ! ! &propSize, &asbd);
28. Core Audio Overview
AudioConverter
• Converts
• bit depths
• sample rate
• interleaving & deinterleaving
• channel ordering
• PCM <-> compressed/encoded
• Can use all installed codecs
29. Core Audio Overview
AudioFile
• Parses a file and provides access to the raw data
• Uses properties to query information about the file
• ExtendedAudioFile
• High-level access to an audio file
• Combines AudioFile + AudioConverter
35. Audio Session
Fundamental Concepts
• Describes an applicationʼs interaction with the audio system
• Represents the current state of audio on the device
• Current Settings
• Preferences
• State Transitions
• Interruptions
• Route Changes
37. Audio Session
Categories
• Identify a set of audio features for your application
• Mixable with others
• Have input or output
• Silence on Screen Lock or Ringer Switch
41. • The pathway for audio signals
• Where is audio output to?
• Where is audio input from?
• “Last in Wins” rule
• Notification when route changes
• Reason why the route changed
AudioSessionAddPropertyListener(kAudioSessionProperty_AudioRouteChange,
audioRouteChangeListenerCallback, self);
44. Audio Session
Enabling Hardware Acceleration
• Must set “Mix With Others” to false
• Overrides not persistent across category changes
// Override our current categories ‘mix with others’ attribute
UInt32 value = 1;
AudioSessionSetProperty(kAudioSessionProperty_OverrideCategoryMixWithOthers,
sizeof(value), &value);
46. OpenAL
• Powerful API for 3D audio mixing
• Designed for games
• Cross-platform, used everywhere!
• Models audio in 3D space, as heard by a single listener
• Designed as a compliment to OpenGL
• Mimics OpenGL conventions
• Uses the same coordinate system
• Implemented using Core Audioʼs 3D Mixer AU
47. OpenAL
Fundamental Concepts
• Context
• The spatial environment
• Contains the listener
• Source
• 3D point emitting audio
• Many attributes to control rendering
• Buffer
• Container for audio data
• alBufferData() - copies data to internal buffers
• alBufferDataStatic() - application owns memory
48. OpenAL
Architecture
Buffer Buffer Buffer Buffer n
Source Source Source n
Listener
OpenAL
Context
OpenAL Device
iPhone Hardware
49. OpenAL
Listener
• Only 1 per context
• Positionable just like Sources
• Represents the userʼs experience in the 3D environment
• Orientation described by two Vectors:
• AT = Direction the Listener is facing
• UP = Direction pointing up from the top of the Listenerʼs head
// Orient the Listener facing +Z
ALfloat listenerOrientation[6] = {!0.0, 0.0, 1.0, // AT
!! 0.0, 1.0, 0.0} // UP
alListenerfv(AL_ORIENTATION, listenerOrientation);
53. OpenAL
Basic Setup
// open an OpenAL Device
1 oalDevice = alcOpenDevice(NULL);
// Create a new OpenAL Context (and listener)
2 oalContext = alcCreateContext(oalDevice, NULL);
// Set our new context to be the current OpenAL Context
3 alcMakeContextCurrent(oalContext);
54. OpenAL
Creating Buffers and Sources
// Create an OpenAL buffer to hold our audio data
4 alGenBuffers(1, &oalBuffer);
// Fill the OpenAL buffer with data
alBufferDataStatic(oalBuffer, AL_FORMAT_MONO16,
audioData, audioDataSize, 44100);
// Create an OpenAL Source object
5 alGenSources(1, &oalSource);
// Attach the OpenAL Buffer to the OpenAL Source
alSourcei(oalSource, AL_BUFFER, oalBuffer);
56. OpenAL
Attenuation by Distance
• Describes the reduction in volume based on the distance
to the listener.
• Set distance model on the context with alDistanceModel()
AL_INVERSE_DISTANCE
AL_INVERSE_DISTANCE_CLAMPED
AL_NONE
• Configure Source attributes
AL_REFERENCE_DISTANCE
AL_MAX_DISTANCE
AL_ROLLOFF_FACTOR
AL_SOURCE_RELATIVE
57. OpenAL
Attenuation by Distance
// Set the distance model to be used
alDistanceModel(AL_INVERSE_DISTANCE_CLAMPED);
+20
dB 0
-20
0 Distance from Listener 20
58. OpenAL
Attenuation by Distance
// Set the Source’s Reference Distance
alSourcef(mySource, AL_REFERENCE_DISTANCE, 2.0);
reference
distance
+20
dB 0
-20
0 Distance from Listener 20
59. OpenAL
Attenuation by Distance
// Set the Maximum Distance
alSourcef(mySource, AL_MAX_DISTANCE, 30.0);
reference maximum
distance distance
+20
dB 0
-20
0 Distance from Listener 50
60. OpenAL
Attenuation by Distance
// Set the Rolloff Factor
alSourcef(mySource, AL_ROLLOFF_FACTOR, 2.0);
reference maximum
distance distance
+20
dB 0
-20
0 Distance from Listener 50
71. Sample Rates
• High-Quality laser *pew pew!* and *beeps* unnecessary
• Example: ʻafconvert -f caff -d LEI16@8000ʼ
• The more SRCs the less performance you get