'Hear' & Now: Software Integration for the Qualcomm Snapdragon Audio Feature

‘Hear’ & Now
Qualcomm® Snapdragon™ Audio
Ravi Satyanarayanan, Director Product Management, Qualcomm Technologies, Inc.
Tony Rodriguez, CTO, Digimarc
Richard Kjerstadius, Director, DSP Software, Opalum
Qualcomm Snapdragon is a product of Qualcomm Technologies, Inc.

‘Hear’ & Now: Snapdragon Audio
User
Experience
1
Ecosystem
Enablement
2
Digimarc
Watermark Detection
3
Opalum
Speaker Protection
4

Audio user experience
Enhancing and expanding the phone experience
Camcorder
Entertainment
Natural user
interaction
Voice call
Music playback

Voice call. Hear. And be heard.
Voice enhancement technology for clearer conversations
©2013-2014 Qualcomm Technologies, Inc. and/or its affiliated companies. All Rights Reserved.
• Fluence & Fluence Pro noise cancellation
technology echo cancellation and noise
suppression
• Freedom to hold the phone how you want
− Landscape or portrait speakerphone
− Handset
• Hear your conversation & not the background sound
− Active Noise Cancellation to suppress noise around you
− Automatically amplify speech relative to background sound
• HD Voice in all calls
• Reduce noise on the incoming calls

Music playback
Hear music as the musicians intended
6
• Hi-Fi audio at the same quality as studio masters
- High quality WCD codec
- End-to-end 24bit / 192kHz
• Listen all day with low power playback on the DSP
• Active Noise Cancellation to hear only what you want to hear

Home theatre on-the-go
Enjoy the same surround sound available in home
theaters wherever you are
7
• Virtual surround sound over headphones to bring the immersive
experience with you
• Big booming sound
- Bigger bass, clearer vocals, louder sounds
• Collaborations with industry leaders: Dolby, DTS

Camcorder
Capture what you want to hear
• Capture what you see with Sound Focus
- Suppress unwanted noise from outside of the
field of view
• Audio zoom when the camera zooms in
• Surround Sound Record for the Full HD / 4K
camcorder experience
• High quality audio, with high dynamic range

Voice natural user interface
• Always on, low power voice activation
- No need to press a button to interact with your device
- Launch applications and give commands quickly
• Activate & Unlock in one step
- No need to enter a PIN
• Hands free operation with Fluence
- Drive mode

User Experience
1
Ecosystem
Enablement
2
Digimarc
Watermark Detection
3
Opalum
Speaker Protection
4

Audio ecosystem
• Added on Snapdragon 810, 808
• Great for low latency, timing
sensitive algorithms
• Speaker Protection by Opalum
Audio front-endVoice
Audio
3rd-party relationships for OEM
differentiation
• End-to-end high-fidelity audio
– Native 24bit/192kHz
framework
• Low power decode on Qualcomm®
Hexagon™ DSP
• Snapdragon Voice Activation
• ISVs can enable similar
solutions using the framework
Always on, low power
• DSP-accelerated voice codecs
and ECNS for VoIP
• Hexagon Access Program support
to add voice codecs and
processing modules
Qualcomm Hexagon is a product of Qualcomm Technologies, Inc.

How others add value,
and find success

User Experience
1
Ecosystem
Enablement
2
Digimarc
Watermark Detection
3
Opalum
Speaker Protection
4

Qualcomm® Snapdragon SenseTM Audio +
Digimarc Discover
Enabling Low-Power Always-On Contextual Awareness
Tony Rodriguez
Qualcomm Snapdragon Sense is a product of Qualcomm Technologies, Inc.

— Slide 15 — ©2014 Digimarc Corporation
Digimarc
Vision and Mission
• Enable mobile devices to see, hear, and understand by providing
digital identities for all media objects (Internet of Everything)
Approach
• Lessons learned from the last decade of deployment (started
w/DoJa & BrewTM):
• ‘Seeing & Hearing’ is a function of how fast, accurate and
reliable the underlying object/media recognition algorithms are.
• Deterministic object recognition is significantly quicker, more
accurate and reliable.
• Developers are best served when fast, reliable and accurate
object recognition is available as a system service that minimizes
bandwidth consumption and is always-on & low-power.
• 2013: Delivered Intuitive Computing Platform (ICP) to our
retail, publishing and brand partners who have deployed apps
to enable engagement with print, packaging, radio, etc.
BREW is a product of Qualcomm Technologies, Inc.

Digimarc Discover built on ICP: Enabling Shoppers Journey
• Digimarc® Discover app & SDK’s
built on Intuitive Computing
Platform.
• Enables object and media
recognition through all phases of
Shopper’s Journey
• Discovery & transaction via mobile
devices across all touch points of
the Journey
• Digimarc® Barcode: Imperceptible
barcodes for CPG that can double
throughput at front-of-store.

Examples: Publications, TV, Retail
Print to Digital Contextual Second-Screen Shopping

Deterministic Identification Powered by Digimarc ID’s
We can’t see or hear Digimarc ID’s, but mobile and wearable devices can
Digimarc ID
Original Digital
Media
Digimarc ID
Imperceptibly
Embedded
Enabled Content
Print
Music, Movies or Images
Television and Radio

Comparison of App Experience in Noisy Environment
• Digimarc Audio ID’s
effective in hostile
environments.
• Accuracy maintained,
no missed matches
(deterministic)
• Developers can
assign multiple ID’s,
creating richer,
contextually aware
experience.

Digimarc Audio ID: Details
• Signal Construction
• Weak signal in 120 Hz to 8 kHz
range, communicated via host audio
(channel)
• Extensible & key based payload
protocols
• Embedding
• Psychoacoustic principles and
empirical analysis used to hide
watermark
• Ambient Detection
• Spectral Processing
• Payload Recovery
Technique Frequency Domain WM
Description Frequency Domain Spread
Spectrum
Characteristics Watermark signal added to audio
spectrum (frequency magnitude)
Payload Extensible 24 bit
Audio required for
detection
1-3 Seconds

Porting to the Qualcomm® HexagonTM
Goal: People don’t have “ear-lids” – sound is the most pervasive and rich source of context in the
human experience, our sub-conscious is continually (even when asleep) categorizing audio and
applying policies on when and how to engage our consciousness. Mobile & wearable devices
should do the same!
• Phase 1: Re-factoring (pipeline architecture), Fixed Point, Memory Reduction
• Phase 2: Hexagon Port of Detector
• Phase 3: Low-Power Activator Implementation
• Result: Developers able to create always-on contextually aware applications that are fast, reliable and
accurate.
Qualcomm Hexagon is a product of Qualcomm Technologies, Inc.

Phase 1: Re-factoring Detector (Pipeline Architecture)
• Mobile
implementation
optimized for
OS &
application
processor
• Each block re-
coded, reducing
memory
consumption
and functional
granularity to
aide in porting
• 8MB -> 256K
Runtime
FFT
Intermediate
Accumulation
Magnitudes
Spectral
Filter
Final
Accumulation
Magnitudes
Normalize
Magnitudes
For every 512 samples @ 16 Khz For each 1.024 second
CRCSNR Gating Decode
Select Decode
Candidates
Version
Detection
For each 1.024 second For each 1.024 second + Scaling
Audio
Digimarc
ID
Spectral Processing
Payload Recovery

Phase 2: Hexagon Port
DetectorApplication
Hexagon & Audio Codec
Audio
• Application reliant on High
Level Operating System
(HLOS) to retrieve audio
samples and deliver to
Detector
• Audio sampled at 16Khz but
processed only once a
second.
• Difficult for Application
Processor cores to enter low-
power state.
Codec
Application Processor
Digimarc Qualcomm Technologies Hexagon SDK

Phase 2: Hexagon Port
Detector StubApplication
Skel
Fast
RPC
Stub
Detector
• Hexagon SDK to Port
detector C++ source.
• Significantly reduced load
on Application Processor
cores
• Utilized library of optimized
functions for Hexagon (FFT,
etc.)
• QAIC handled complexity of
FastRPC marshaling
Audio
Codec

Phase 3: Low-Power Activator
Audio
Skel
Fast
RPC
Stub
Codec
“Activator”Service&PolicyEngine
• Worked with Qualcomm Technologies to
build Activator and route audio from Codec
• Eliminated need for HLOS to manage audio.
Detector

Audio
Skel
Fast
RPC
Stub
Detector
Codec
• Policy Engine able to wake Application
Processor and invoke appropriate application
when Digimarc Audio ID detected
• Network bandwidth consumed after object
identification.
Policy Engine

Audio
Skel
Fast
RPC
Stub
Detector
Codec
• Policy Engine able to wake Application
Processor and invoke appropriate application
when Digimarc Audio ID detected
• Network bandwidth consumed after object
identification.
• Result: Application Processor & RF allowed
to enter low-power state.
• Developers able to create always-on
contextually aware applications that
leverage the ubiquity of audio in our
everyday lives Hexagon & Audio Codec
Policy Engine

Conclusion : Snapdragon Sense Audio + Digimarc Audio ID’s
• People don’t have “ear-lids” – sound is the most pervasive and rich source of context in the human
experience, our sub-conscious is continually (even when asleep) categorizing audio and applying
policies on when and how to engage our consciousness.
• Mobile and wearable devices are now able to do the same, remain always aware and wake the
Application Processor as needed.
• Able to maintain an always-on posture with no network dependencies – ramps power and
bandwidth consumption after identification has occurred (unique to Digimarc ID’s).
• Enabled by Snapdragon Sense Audio & Digimarc Audio ID’s that provide accurate, reliable context.
• Result: Accurate and application specific context can be incorporated in all sources of audio.
• Applications enabled for developers: Retail, Second-Screen, Indoor Location, Radio, Concerts, etc.

‘Hear’ & Now: Snapdragon Sense Audio
User Experience
1
Ecosystem
Enablement
2
Digimarc
Watermark Detection
3
Opalum
Speaker Protection
4

© 2014Opalum AB
Agenda
• Trends in MobileAudio
• OpalumOverview andTech
• Collaboration with QualcommTechnologies
• What’s next?

© 2014Opalum AB
Trends in Mobile Audio
• Maximizing loudness and bass
– Power doesn’t kill a speaker
– Pushing the limits on thinner form-factors
• Multi-channel audio goes mobile
– Virtualization techniques for movies and games.
– Parametric audio
• Reducing power consumption
– “Always on” features demand DSP core offloading
– Increasing level of integration into main chipset
– Other ways of reducing power consumption?

© 2014Opalum AB
Collaborating with QualcommTechnologies
• Seamless integration between LabSuite and Snapdragon
• Audio tuning lead-time drastically reduced
• Working with the Hexagon tools
• Dynamic loading enables really cool android app possibilities

© 2014Opalum AB
For more information on Qualcomm, visit us at:
www.qualcomm.com & www.qualcomm.com/blog
©2013-2014 Qualcomm Technologies, Inc. and/or its affiliated companies. All Rights
Reserved.
Qualcomm, Snapdragon, Hexagon, Brew and Fluence are trademarks of Qualcomm
Incorporated, registered in the United States and other countries. Uplinq and Snapdragon
Sense iaretrademarks of Qualcomm Incorporated. All Qualcomm Incorporated trademarks
are used with permission. HY-FI is a trademark of Qualcomm Atheros, Inc., registered in the
United States and other countries, used with permission. Other products and brand names
may be trademarks or registered trademarks of their respective owners.of their respective
owners.
References in this presentation to “Qualcomm” may mean Qualcomm Incorporated,
Qualcomm Technologies, Inc., and/or other subsidiaries or business units within the
Qualcomm corporate structure, as applicable.
Qualcomm Incorporated includes Qualcomm’s licensing business, QTL, and the vast majority
of its patent portfolio. Qualcomm Technologies, Inc., a wholly-owned subsidiary of
Qualcomm Incorporated, operates, along with its subsidiaries, substantially all of
Qualcomm’s engineering, research and development functions, and substantially all of its
product and services businesses, including its semiconductor business, QCT.
Thank you FOLLOW US ON:

'Hear' & Now: Software Integration for the Qualcomm Snapdragon Audio Feature

Empfohlen

Empfohlen

Weitere ähnliche Inhalte

Was ist angesagt?

Was ist angesagt? (20)

Ähnlich wie 'Hear' & Now: Software Integration for the Qualcomm Snapdragon Audio Feature

Ähnlich wie 'Hear' & Now: Software Integration for the Qualcomm Snapdragon Audio Feature (20)

Mehr von Qualcomm Developer Network

Mehr von Qualcomm Developer Network (20)

Kürzlich hochgeladen

Kürzlich hochgeladen (20)

'Hear' & Now: Software Integration for the Qualcomm Snapdragon Audio Feature