BPM JS

jsbpm
onset detection in the Browser
Janessa Det - WEB eng @TWITTERNYC

hello.
bpmjs
janessa det (@jandet)
twitter nyc
Software engineer
Web Team
columbia university
M.S. Computer science
Graphics & Vision
duke university
B.S.E. biomedical engineering

IRON MAN
onset detection
bpm. what?
bpmjs
onset detection
fundamental to:
sonar, speech, EKG's,
movement detection
this
thing
IRON MAN

AUTOMATIC
TRANSCRIPTION
OF MUSIC
onset detection
bpm. what?
bpmjs
AUTOMATIC
TRANSCRIPTION
OF MUSIC
this
thing
1. instrument identification
2. instrument isolation
3. pitch detection
4. dynamics

accessible
applicable
sharable
cross-platform
fast
digital signals processing
in the browser. why?
bpmjs
accessible
applicable
sharable
cross-platform
fast
digital signals processing

javascript
in the browser. why?
bpmjs
because
javascript
and it would be cool
is fun
to play with

diverse
diversity is powerful.
bpmjs
the browseris
diverse
the community

bpmjs
yeah,
that’s me
reality
the dream
"finding ways to breed
engineering research, music, and art
with the trusty steed of javascript
to make the magical unicorns
of my imagination a reality"

bpmjs
yeah,
that’s me
reality
the dream
"finding ways to breed
engineering research, music, and art
with the trusty steed of javascript
to make the magical unicorns
of my imagination a reality"
ok, let’s
breed some
unicorns

onset detection?
how do we do
onset detection?

GENRES
tempos
beats
onsets
what is bpm?
bpmjs
1. pitch
2. notes
3. percussive beats
DRUM AND BASS 160-180 BPM
House 118-135 BPM
hip-hop 115 BPM
Concert marches 120 BPM
Screamers 130-150 BPM
beats
GENRES
what is a beat?
1. pitched non-percussive
2. pitched percussive
3. non-pitched percussive
4. complex mixed
onsets
tempos
Largo 40-60 BPM
Larghetto 60-66 BPM
Adagio 66-76 BPM
Andante 76-108 BPM
Moderato 108-120 BPM
Allegro 120-168 BPM
Presto 168-200 BPM
Prestissimo 200+ BPM

bpmjs
detection method.
DETECTION
FUNCTION
stft
DETECTION
FUNCTION
STFT PEAK PICKINGPEAK PICKING BPMBPM
Prep the signal
for Detection
Functions
Selectively maximize
for onset properties
Pick out peak times from
Detection Function
Calculate
BPM
ಠ_ಠ

bpmjs
detection method.
DETECTION
FUNCTION
stft
DETECTION
FUNCTION
STFT PEAK PICKINGPEAK PICKING BPMBPM
Prep the signal
for Detection
Functions
Selectively maximize
for onset properties
Pick out peak times from
Detection Function
Calculate
BPM

stft. short-time fourier transform.
bpmjs
4. zero padding
Zero pad to nearest power of 2
2. hamming window
Apply windowing function -- Hamming
3. normalization
Normalize magnitude
5. fast fourier transform
Convert to frequency domain
1. window slicing
Slice signal into overlapping windows

detection functions.
bpmjs
frequency based methods
3. phase deviation
1. High frequency content
2. spectral difference
4. euclidian distance

peak picking
build threshold
peak picking.
bpmjs
build threshold
1. median filter
Smooth out the signal for thresholding
2. adaptive thresholding
Threshold by median-filtered signal
3. initial peak finding
Identify local maxima
4. thresholding peaks
Apply adaptive threshold to peak findings
5. refractory period
Apply minimum time between peaks as threshold
peak picking

120 BPM
bpm calculation.
bpmjs
0.5 1.0 1.5 2.0 2.5 3.0 3.5 4.0
peak times (in seconds)
0.5
1 beat / 0.5 sec * 60 sec / 1 min
= 120 BPM

in javascript?
how do we do it
in javascript?

color profile of
music characterization.
bpmjs
color profile of
so insane by discovery
average ~ 115 bpm
0:30 1:000:15 0:45 1:15 1:45 2:151:30 2:000:00 2:30 3:002:45 3:15

dsp.js
web audio
complex arithmetic
a lot of loops & math
browserify
tools used.
bpmjs
dsp.js
web audio
complex arithmetic
a lot of loops & math
browserify
i

modules.
bpmjs
peak picking
detection
bpm
web audio
AudioBuffer node
hfc sd pd e. dist.
dsp.js
fft windowing
complex
stft

how to not break chrome
js challenges.
bpmjs
float32array
compatibility
complex arithmetic
matrices

js challenges.
bpmjs
float32array
compatibility
complex arithmetic
matrices
NaNNaNNaNNaNNaNNaNNaNNaNNaNNaNNaNNaN
NaNNaNNaNNaNNaNNaNNaNNaNNaNNaNNaNNaN...
NaNNaNNaNNaNNaNNaNNaNNaN

0. signal winndow
stft. short-time fourier transform.
bpmjs
3. ZERO PADDING
1. HAMMING WINDOW
2. NORMALIZATION
4. FFT -> Complex objects

bpmjs
detection functions.
hfc
sd
pd
e. dist.
not bad...

bpmjs
peak picking.
detection
function (ed)
median filtered
thresholded
etc

THE DREAM:
matlab!!
benchmarks
optimizations
& performance
bpmjs
1. web workers for stft
2. SLICE WELL-SELECTED WINDOWS
benchmarks
vs. matlab!!
THE DREAM: “REAL-TIME”
3. asm.js, enscripten, LLjs

weaknesses.
future work.
weaknesses.
bpmjs
1. sensitive tuning parameters
2. a priori knowledge helpful to select detection functions
3. median filtering or aggressive thresholding can cripple
the detection function
future work.
1. only scratching the surface of research
2. many detection functions
3. onset detection for different applications

is the unicorn beautiful?
ok so how did i do?
is the unicorn beautiful?

bpmjs
ok, it feels a
little forced
but we’re getting
there!!

the web is changing.
bpmjs
web audio api
dsp.js
God Bless
npm

rethinking
web engineering.
bpmjs
what does it mean to be a
web engineer?
how about more engineering?
this is not the
final frontier

bpmjs
embrace diversity
open community of learning
knowledge builds upon itself
combining fields is very powerful
think outside the box
breed those unicorns
rethinking
web engineering.

bpmjs
thanks.
go forth and breed unicorns

main references
Beat Detection for Automated Music Transcription:
An exploration of Onset Detection Algorithms.
http://bingweb.binghamton.edu/~ahess2/Onset_Detection_Nov302011.pdf
The Scientist and Engineer's Guide to
Digital Signal Processing
By Steven W. Smith, Ph.D.
http://www.dspguide.com/pdfbook.htm
additional resources:
Gamedev: beat detection algorithms
http://archive.gamedev.net/archive/
reference/programming/features/beatdetection/index.html
BPM Database
http://www.bpmdatabase.com/

BPM JS

Empfohlen

Empfohlen

Weitere ähnliche Inhalte

Ähnlich wie BPM JS

Ähnlich wie BPM JS (20)

Kürzlich hochgeladen

Kürzlich hochgeladen (20)

BPM JS