ST Developers Conference 2016 - Solutions for Playing high-quality Music and Voice over BLE

1. October 4, 2016
Santa Clara Convention Center
Mission City Ballroom
Solutions for playing high-quality
music and voice over BLE
Roberto Sannino

2. Audio Over Bluetooth Smart: Scenarios 2
Voice controlled TV remoteAudio Entertainment & Gaming
Smart Home Wearables

3. Voice Terminal
• Audio capture & render
• Signal processing
• Low power
• Constained geometry
Voice & data
Gateway
Seamless connectivity
BlueVoice: IoT Audio Augmented 3
Cloud
• Natural Language Processing
• Dialogue Management
• Services
Play Music
Control Lighting, heating, …
News, sport, traffic, weather, …
Answer questions, create to-do lists, shopping lists, …
Place orders online, use other online services: taxi, pizza, …

4. A BlueVoice to Cloud Architecture
Integrated Terminal
Audio Front End
Signal Processing
Communication
& 3rd Party SW libs
Cloud-based
ServicesGateway
4

5. BlueVoice at a Glance 5
MEMS Microphone Microcontroller BlueNRG RF connectivity
The ST HW and SW solution for
Ultra low-power Voice streaming over Bluetooth Low-Energy
STM32 Signal Processing and Application Firmware & Software
BLUEVOICELINK BLE and microphone reference application based on STM32Cube
OSXBLUEVOICE Voice-over-BLE vendor-specific profile library for STM32 and BlueNRG
BLUEMICROSYSTEM Voice and Sensor data over a BLE link to an Android - iOS Smartphone App
BlueMS, BlueST-SDK Bluetooth Smart and Sensors Technology Application for Android and iOS

6. BlueVoice Mapping over Standard
Bluetooth 4.0 protocol stack
6
BlueVoice Vendor Specific Profile
Audio processing
Generic Access
Profile (GAP)
Generic Attribute
Profile (GATT)
Attribute Protocol Security Manager
Logical Link Control and Adaptation Protocol
Host-Controller Interface
Link Layer Direct Test Mode
Physical Layer
Bluetooth Low Energy Stack
GAP configuration
GATT configuration
Application: BlueVoice Profile
(vendor specific)
Central Unit
(Master)
Peripheral
Unit
(Slave)
Client Server
ClientServer
Service
Characteristic
Characteristic
Descriptor
Audio is
exported as
a Service
Service
Characteristic
Characteristic
Descriptor

7. BlueVoice Server Attributes
Characterstic Value
Compressed audio data
Audio Characteristic
Service Declaration
• BVS UUID
• Type: primary
Characteristic Declaration
• AudioData UUID
• Char properties:
• Notification
• Maximum value length: 20 B
Characteristic Declaraton
Characteristic Value
Characterstic Value
Synchronization data
Sync Characteristic
Characteristic Declaration
• SyncData UUID
• Char properties:
• Notification
• Maximum value length: 6 B
Characteristic Declaraton
Characteristic Value
Reference chart
BlueVoice Service
Service
Characteristic
Characteristic
Descriptor
Server
7

8. BlueVoice Packet Exchange Sequence
Advertise
Connect Request
Notification
Notification
Notification
Advertise
Advertisement
ModeDiscovery
Mode
Central Unit
(Master)
Peripheral
Unit
(Slave)
Connection created
Notification
Notification
Notification
Peripheral-to-
Central
communication
Central-to-Peripheral
communication
......
No response is required for
Notifications:
• OK for Audio streaming
• Optimal Bandwidth
exploitation
8

9. BlueVoice Architecture:
Audio processing and transmission
PDM to PCM
conversion
PCMPDM
Audio
Compression
Raw Data
Server - TX
Audio
Acquisition
symmetrical architecture for bi-directional communication
Audio
Decompression
Serial Audio Out
USB, I2S, …
Client - RX
9

10. BlueVoice Development Kit
BlueVoice on open.Audio 10
Voice Gateway
Cloud
BlueVoice Terminal
• Microphone Acquisition
• PDM to PCM conversion: 8 and 16 KHz
• ADPCM compression: 4 bit per sample
• BTLE streaming: 32 and 64 kbps
SensorTile
BlueCoin
BlueMS app

11. Software IP and ST Eco-system
Open Software Design Environment
11
Algorithms and system demonstrators for the Internet of Things.
Unleashing the power of embedded software
Bring your ideas to now!
BlueMicroSystem
STM32 ODE
STM32 Nucleo
development
boards
STM32 Nucleo
expansion
boards
STM32 Cube
software
STM32 Cube
expansion
software
Software libraries
BlueVoiceLink
SmartAcoustics
BlueMS
Android & iOS
Example Projects

12. Open.AUDIO OSXBlueVoice
Voice over BLE Vendor Specific Profile library
12
PDM to PCM
conversion
PCMPDM ADPCM Audio
Compression
ADPCM Data
Server - TX
Microphone
Acquisition
ADPCM Audio
Decompression
Serial Audio Out
USB, I2S, …
Client - RX
16k or 8k sample/s
4 bit/sample
16bit/sample
@16kHz or
@ 8kHz
OSXBlueVoice Speech Tx library
Open Software
Expansion
Open.AUDIO
OSXBlueVoice
The BlueVoice Library
OSXBlueVoice Speech Rx library
ADPCM Data
16k or 8k sample/s
4 bit/sample

13. Audio BLE Streaming 13
Connection
Event
10ms t
Audio Data Packet, 20 bytes
160ms
Connection interval
Connection
Event
…
ADPCM Synch interval
Sync Data Packet, 6 bytes
… ………
Data streaming is implemented as periodical data exchange
• Connection Event: data packet exchange sequence
• Connection Interval: the time between two consecutive Connection Events
Radio
Idle
Connection
Event
Connection
Event
Connection
Event
Low-power state between connection events
Voice Streaming
• Audio Format: PCM 16 bit @ 16 kHz
• ADPCM Compression @ 64 Kbps: Low
latency and low complexity
• Side Information at low data rate
ehnances error resilience

14. OSXBLUEVOICE v2.0.0 Library 14
OSXBLUEVOICE 2.0.0

15. Working mode:
• TRANSMITTER
• RECEIVER
• HALF-DUPLEX
Configurable:
• Audio Sampling Frequency: 8 or 16 kHz
• Audio input channel select
Audio input buffer:
1, 2, 5 or 10 ms of PCM
audio data.
BLE service and
characteristic
modularity
Compatible with Android and iOS app: STBlueMS
osxBlueVoice 2.0.0 Library 15
osxBlueVoice 2.0.0
Features
Compiled for different
ARM Cortex core:
• M0+
• M4
STM32 Example code for:
• STM32F4
• STM32L4
• STM32L0
ST development Platforms:
• STM32 Nucleo
• SensorTile
• BlueCoin
Hardware CompatibilitySTM32 application project

16. osxBlueVoice 2.0.0 Library Architecture 16
Input
ch1 ch2 ch1 ch2 ch1 ch2 ch1 ch2……
PCM buffer:
- 1, 2, 5 or 10 ms audio
- One or more audio channel
- Audio sampling frequency: 8 or 16 kHz.
- Total audio channels acquired.
- Audio Channel chosen: one of the
acquired channels (1, 2 or … n).
- Service and characteristics configuration
Configuration
osxBlueVoice Library 2.0.0
Service and characteristics creation/configuration
BLE packetization and
sending
BLE packet parsing and
ADPCM decompression
Audio parameters configuration
BLE packets:
- Audio data
20 Bytes
- Sync data
6 Bytes
Output
Audio send application
TX Control
Audio receive application
RX Control
SendData ready
GATT
Notification
BLE Event
ADPCM
compression
Internal double
buffering
Data Control
Input
Output

17. BlueVoiceLink reference application
17
www.st.com/bluevoicelink
BLUEVOICELINK1
osxBlueVoice 2.0.0

18. BLUEVOICELINK1
Half-Duplex communication based on STM32 Open development Environment
Audio configuration
8 kHz,16 kHz
4:1 ADPCM
USB USB
BlueVoiceLink1 2.0.0 - Peripheral
osxBlueVoice 2.0.0
BlueVoiceLink1 2.0.0 - Central
osxBlueVoice 2.0.0
Source code reference application based on
osxBlueVoice 2.0.0
NUCLEO-F401RE
NUCLEO-L476RG
Microphone
Bluetooth Low Energy
X-NUCLEO-CCA02M1
X-NUCLEO-IDB04A1
STM32 Nucleo
18

19. ST BlueMS + BlueVoiceLink1
Simplex voice over BLE communication from STM32 ODE to Android/iOS
19
Cloud-based
ASR Service
In configuration
8 kHz
ADPCM
Available on Google Play
and App Store
NUCLEO-F401RE
NUCLEO-L476RG
NUCLEO-L053R8
Microphone
BLE
X-NUCLEO-CCA02M1
X-NUCLEO-IDB04A1
STM32 Nucleo
BlueVoiceLink1 2.0.0 - Peripheral
osxBlueVoice 2.0.0

20. ST BlueMS + Bluemicrosystem2
BlueVoice Service Integration in BlueMicrosystem environment
Cloud-based
ASR Service
ST BlueMS available on Google Play and App Store
8 kHz
ADPCM
20
BlueVoiceLink1 2.0.0 - Peripheral
osxBlueVoice 2.0.0

21. BlueVoice Applications
ADPCM
@ 8 kHz
ADPCM
@8 or 16 kHz
ST BlueMS
3.0.0
BlueMicrosystem2.1.0 BlueVoiceLink1 2.0.0
Peripheral
BlueVoiceLink1 2.0.0
Central
data rates ≤ 32kbps
to maximize
compatibility
From version 4.4 KITKAT
API 19
R&D
From version 8
NUCLEO F401
NUCLEO L476
SensorTile
NUCLEO F401
NUCLEO L476
NUCLEO L053
BlueCoin
osxBlueVoice 2.0.0 osxBlueVoice 2.0.0
osxBlueVoice 2.0.0
NUCLEO F401
NUCLEO L476
Cloud-based
ASR Service
21

22. BlueVoice Advanced Application

23. SensorTile
Integrated Development Platform
Core System
Cradle eXpansion
Cradle
23

24. SensorTile
Software platform
24
Starter Projects BlueMicrosystem BlueVoice Function Pack
Core System

25. SensorTile Native Support for BlueVoice 25
SensorTile
Terminal
Platform Signals
Comm
Interface
Cloud-based
Services
Mobile
Device
«Natural Language»

26. BlueVoice over BlueNRG-1
BLE Application Processor
26
Audio Sampling frequency: 8 kHz
Compression algorithm: ADPCM
Bitrate 32kbps
Receiver can be any STM32
BlueVoice client or an Android/iOS
device running ST BlueMS app
STEVAL-IDB007V1
BlueNRG-1 evaluation platform
Demo application:

27. BlueVoice R&D 27
R&D
R&D
BlueVoice
FullDuplex Speech
BlueVoice
FullBand Music

28. BlueVoice R&D – FullBand Music 28
Compression algorithm: Opus
• Highly-flexible speech and audio codec
• Bit-rates from 6 kb/s to 510 kb/s
• Narrowband (8 kHz) to fullband (48 kHz)
• Frame sizes from 2.5 ms to 60 ms
• Speech and Music support
• Mono and Stereo
• Flexible rate control
• Flexible complexity
• All changeable dynamically
• Completely free
• Royalty-free licensing
• Open-source implementation
• IETF RFC 6716
Stereo
Full Band Music @ up to 48 kHz

29. Full Band Audio
Streaming
USB
I2S
Stereo
Notifications
BlueVoice Control Data
Central/Client - RX
USB /I2S
16bit@48 kHz
Compressed
AudioPCM
BLE
48kbps
PCM
Compressed
Audio
BLE
Peripheral/Server - TX
16bit@48kHz
USB
.wav 48kbps
Full band
music @ 48kbps Notifications Write without resp
Full Band Audio Streaming
• BlueNRG-MS
• STM32F446, STM32F7
• Audio Format: PCM @ 48 kHz stereo
• Opus Compression @ 48kbps
BlueVoice FullBand Demo Overview 29

30. BlueVoice R&D – FullDuplex Speech 30
Bidirectional (Full-Duplex)
audio streaming
Input: PCM @ 16 kHz
Opus Compression @ 10 kbps
or higher (Variable Bit Rate)

31. BlueVoice R&D – FullDuplex Speech
Central Data Packet Peripheral Data Packet
Radio
Idle
t
…
Connection
Event
Connection
Event
t
……
Radio
Idle
Connection
Event
Connection
Event
Connection
Event
Connection
Event
Central GATT Interval
Peripheral GATT Interval
Different Central-Peripheral interleaving schemas may be implemented
31

32. BlueVoice Solutions
Half-duplex High quality musicFull-duplex Full band music
OpusADPCM
32/64 kbps
8 - 16 kbps
SBC
<64 kbps
<64 kbps >64 kbps
8 - 16 kbps
32/64 kbps
32 - 64 kbps
8 - 10 kbps
Compression schema
R&D
Demo
R&D
Demo
Simplex
R&D
32/64 kbps Demo
<64 kbps
R&D
32/64 kbps
32/64 kbps
R&D
Demo
R&D
32

33. BlueVoice Solutions
Advanced
features
Ultra-Low Power
BlueVoice-L
• STM32L4 or BlueNRG-1
• Features
• ADPCM @8kHz
• Hw PCM filtering
• Low power mic
BlueVoice+
• Advanced audio
processing
• Beamforming
• ASR
BlueVoiceLink
• STM32F4/STM32L4/STM32L0
• Features
• Half Duplex Speech
• ADPCM @16/8kHz
• ST BlueMS app Android/iOS
Basic
functonalities
High Performance
BlueVoice Full Band
• Opus Compression
• High Quality Stereo
Music
• 48+ KHz sampling rate
BlueVoice-Full Duplex
• Full Duplex Speech
• Opus @ 16 kHz
• Low Latency
BlueVoice-Android
• Half Duplex Speech
• ADPCM @ 8/16kHz
• SBC/Opus @ 16 kHz
33

34. Thank You