2. -MN Developer and Instructor
-Owner of TapHarmonic, LLC.
-Author of Learning AV Foundation
About...
Bob McCune
http://LearningAVFoundation.com
3. -What is AVAudioEngine?
- Goals and Capabilities
-Understanding AVAudioEngine
- Understanding Audio Nodes
- Playing and Recording Audio
- Audio Mixing and Effect Processing
- Working with MIDI and Samplers
Agenda
What will I learn?
4. AV Foundation Evolution
Humble Beginnings
AV Foundation
MediaPlayer AVKit
UIKit AppKit
Core Video Core MediaCore Animation
Audio-Only
Core Audio
MavericksiOS 7
5. AV Foundation Evolution
Humble Beginnings
AVKit AVKit
UIKit AppKit
Core Video Core MediaCore Animation Core Audio
AVF Video AVF Audio
YosemiteiOS 8
7. -Objective-C API simplifying low-latency, real-time audio
-Features and Capabilities:
- Read and write audio files in all Core Audio supported formats
- Play and record audio using files and buffers
- Dynamically configure audio processing blocks
- Perform audio tap processing
- Perform Stereo and 3D mixing off audio signals
- MIDI playback and control over sampler instruments
AVAudioEngine
Core Audio for Mortals
9. -Manages graphs of audio nodes
-Connects audio node into active chains
-Dynamically attach and reconfigure graph
-Start and stop the engine
The Engine
AVAudioEngine
11. -AVAudioEngine provides 3 implicit nodes:
- AVAudioInputNode: system input, cannot be created
- AVAudioOutputNode: system output, cannot be created
- AVAudioMixerNode: mixes multiple inputs to a single output
-Nodes are connected via their input and output busses
- Most nodes have one input and one output
- AVAudioMixerNode has multiple inputs and one output
- Busses have an associated audio format
Nodes
AVAudioNode
12. -Nodes are connected to form an active chain
- Source Node Destination Node = Active Chain
- Establishes and active render thread
Node Connections
Establishing Active Chains
Destination
Node
(Output)
Source
Node
(Player)
13. Node Connections
Establishing Active Chains
Destination
Node
(Output)
Source
Node
(Player)
Processing
Node
(Mixer)
-Nodes are connected to form an active chain
- Source Node Destination Node = Active Chain
- Establishes and active render thread
14. Node Connections
Establishing Active Chains
Destination
Node
(Output)
Source
Node
(Player)
Processing
Node
(Mixer)
X
-Nodes are connected to form an active chain
- Source Node Destination Node = Active Chain
- Establishes and active render thread
15. 1. Create the engine
2. Create the nodes
3. Attach the nodes to the engine
4. Connect the nodes together
5. Start the engine
Basic Recipe
Configuring the Graph
16. Engine Setup
Basic Recipe
// 1. Create engine (example only, needs to be strong reference)
AVAudioEngine *engine = [[AVAudioEngine alloc] init];
// 2. Create a player node
AVAudioPlayerNode *player = [[AVAudioPlayerNode alloc] init];
// 3. Attach node to the engine
[engine attachNode:player];
// 4. Connect player node to engine's main mixer
AVAudioMixerNode *mixer = engine.mainMixerNode;
[engine connect:player to:mixer format:[mixer outputFormatForBus:0]];
// 5. Start engine
NSError *error;
if (![engine startAndReturnError:&error]) {
// handle error
}
19. -Reads and writes files in all Core Audio supported formats
-Automatically decodes when reading, encodes when writing
- Does not support sample rate conversion
-File has both a file format and a processing format
- fileFormat: on-disk format
- processingFormat: uncompressed, in-memory format
- Both are instances of AVAudioFormat
Audio Files
AVAudioFile
20. -Provides a format descriptor for the digital audio samples
- Provides access to sample rate, channel count, interleaving, etc.
- Wrapper over Core Audio AudioStreamBasicDescription
-Core Audio uses a “Standard” format for both platforms
- Noninterleaved linear PCM, 32-bit floating point samples
- Canonical formats are deprecated!
-Additionally supports “Common” formats
- AVAudioCommonFormat: 16/32-bit integer, 32/64-but floating point
Audio Formats
AVAudioFormat
21. -Memory buffer for audio data in any Linear PCM format
- Format and buffer capacity defined upon creation
-Provides a wrapper over a Core Audio AudioBufferList
- audioBufferList and mutableAudioBufferList properties
Audio Buffers
AVAudioPCMBuffer
@property (nonatomic, readonly) float * const *floatChannelData;
@property (nonatomic, readonly) int16_t * const *int16ChannelData;
@property (nonatomic, readonly) int32_t * const *int32ChannelData;
-Sample data accessed using:
22. -Pushes audio data onto the active render thread
-Schedule audio data from files and buffers
- Scheduled to play immediately or at future time
- Future times specified with AVAudioTime
- Files
- Schedule file or file segment with completion callback
- Buffers
- Schedule multiple buffers with individual completion callbacks
- Schedule looping buffer
Player Nodes
AVAudioPlayerNode
28. -Node that mixes multiple inputs into a single output
- Efficiently performs sample rate conversions
- Can upmix/downmix channel counts where needed
Audio Mixing
AVAudioMixerNode
AVAudioMixerNode
Input 1
Input 2
29. -Group and process similar inputs
- Simplify and improve efficiency of similar audio processing
Audio Mixing
Using Submixes
AVAudioMixerNode
Input 1
Input 2
AVAudioMixerNode
Input 3
Input 4
AVAudioMixerNode
(main mixer)
30. -AVAudioMixing Protocol:
- Defines properties to be applied to an input bus of a mixer node
- Source and mixer nodes conform to this protocol
Audio Mixing
AVAudioMixing
AVAudioOutputNodeAVAudioMixerNode
AVAudioPlayerNode
AVAudioPlayerNode
volume pan
volume pan
Input Bus 0
Input Bus 1
32. -Node tap pulls data off the render thread
-Captures the output of a particular node
- Record data from microphone
- Record data from a pre-recorded or live audio mix
- Perform data visualization or analysis
-Can install one tap per output bus
-Dynamically install and remove taps
-Audio data returned in a block callback
Node Taps
Pulling Data
35. Effect Nodes
Digital Signal Processing
-There are two main categories of effects available:
- AVAudioUnitEffect: performs real-time audio processing
- AVAudioUnitTimeEffect: performs non real-time audio processing
AVAudioUnitDelay
AVAudioUnitEQ
AVAudioUnitDistortion
AVAudioUnitReverb
AVAudioUnitEffect AVAudioUnitTimeEffect
AVAudioUnitTimePitch
AVAudioUnitVarispeed
36. -Delays original signal by delay time and mixes with original
-Configuration parameters:
- delayTime: The delay time of the input signal (up to 2 seconds)
- feedback: Amount of output fed back into delay line
- lowPassCutoff: Frequency past which high frequencies rolled off
- wetDryMix: The blend of wet/dry signals (0% to 100%)
Delay
AVAudioUnitDelay
37. -Multi-band equalizer and filter unit
-Configuration parameters:
- bands: Array of AVAudioUnitEQFilterParameters objects
- globalGain: Overall gain adjustment applied to input signal (-96 to 24db)
Equalization
AVAudioUnitEQ
38. -Used to define the EQ parameters to be applied
- Retrieved from the AVAudioUnitEQ object’s bands property
-Configuration parameters:
- filterType: Parametric, Low/High Pass, Band Pass, Low/High Shelf
- frequency: The center frequency or cutoff frequency
- bandwidth: The width around the main frequency in octaves
- gain: The gain adjustment (boost or cut) to the frequency
- bypass: The bypass state
Equalization (Continued)
AVAudioUnitEQFilterParameters
39. -Multi-stage distortion effect of original signal
-Configuration presets:
- loadFactoryPreset:(AVAudioUnitDistortionPreset)preset
- DrumsBitBrush, MultiBrokenSpeaker, SpeechWaves, etc
-Configuration parameters:
- preGain: Gain applied to signal before distorted (-80dB to 20dB)
- wetDryMix: The blend of wet/dry signals (0% to 100%)
Distortion
AVAudioUnitDistortion
40. -Simulates the reflective qualities of a particular environment
-Configuration presets:
- loadFactoryPreset:(AVAudioUnitReverbPreset)preset
- Small Room, Large Hall, Cathedral, etc.
-Configuration parameters:
- wetDryMix: The blend of wet/dry signals (0% to 100%)
Reverb
AVAudioUnitReverb
43. -Musical Instrument Digital Interface
-Specification defining:
- Communication protocol for controlling electronic instruments
- Hardware cables and connectors
- Standard MIDI file format
-Extensions:
- General MIDI (GM)
- Downloadable Sounds (DLS)
MIDI
What is MIDI?
http://www.midi.org
44. -High-quality sampler instrument to play sampled sounds
-Loads samples in EXS, SF2, or DLS formats
- Can additionally load and arbitrary array of sample data
-Responds to all standard MIDI messages
- Note on/off, controller messages, pitch bend, etc.
-Great solution for a live performance instrument
- What about playing sequences?
Sampling Instrument
AVAudioUnitSampler
45. -AVAudioEngine has a musicSequence property
- AudioToolbox MusicSequence type
-Use the MusicPlayer and MusicSequence APIs
- Attach the MusicSequence to the engine to play through Sampler
MusicSequence
Core MIDI Type
47. -Powerful new addition to AV Foundation
- Simplifies low-latency, realtime audio
- Great solution for music, audio, and gaming applications
- Core Audio for Mortals
-Enables you to build advanced audio applications
- Read and write audio files
- Play and record audio using files and buffers
- Add DSP effects: Reverb, Delays, etc.
- Perform Stereo and 3D mixing off audio signals
- MIDI playback and control over sampler instruments
Summary
AVAudioEngine