Presentation given by Janani Mukundan and Richard Daskas at Watson DevCon 2016. As AI advances, scientists are pushing the boundaries of computational creativity. IBMers from Research are teaching Watson to compose original music based on human emotion. Using machine learning algorithms, the Researchers are teaching artificial neural networks to understand music theory, structure (pitch, time signature, key signature), and emotional intent to co-create music with a human partner. "Watson Beat" generates new and unique musical song combinations based on the beat played by the individual and the emotional tone the human chooses. Here’s how it works.

1. IBM confidential ©IBM Corporation
Watson Beat:
New Cognitive Era

2. 2©IBM Corporation
FutureCurrent

3. IBM Confidential | Do Not Distribute © 2015 IBM Corporation
▪ Why Music?
▪ Why unsupervised learning?
▪ Why Watson Beat?
▪ Representing music based on parameters is not very intuitive for most people
▪ For example: I like songs with 120 bpm, following B# major etc.
▪ Watson Beat provides a new way to query and compose new music based on reference tracks
▪ Ease of use: no requirement of prior knowledge, music theory etc.
▪ Provide input track(s) to our model, it gives you a new output track(s)
Motivation:
3

4. IBM Confidential | Do Not Distribute © 2015 IBM Corporation
▪ Demo of potential application
▪ I am an amateur video game developer, and I need music for my game, and I like how “Game of Thrones” sounds
▪ Feed it through Watson Beat and get 100s of variations on same music
▪ Ability to steer compositions based on intent — slow, sad, fast, happy, vibrant
Learning Music: Basic Idea
Learned
“Game Of Thrones”
Input
Track(s)
Perturb model
using creativity
genes
(known influences)
Extract
musical
characteristics
(pitch, rhythm,
dynamics etc.)
Reconstruct track by
iteratively learning
input musical
characteristics with
added perturbation
Output
Track(s)
4
Original
“Game Of Thrones”

5. IBM Confidential | Do Not Distribute © 2015 IBM Corporation
▪ RBM: Stochastic Neural Net with
▪ one layer of visible units
▪ one layer of hidden units
▪ Learning RBMs: Contrastive Divergence
▪ DBN: Stack multiple Restricted Boltzmann
Machines (RBM)
Deep Belief Networks (DBN):
5
x: Visible layer
h: Hidden Layer
W: weights
Input Vector
Perturbed Input Vector
unsupervised learning of NNs
(RBMs, Auto encoder etc.)
Output Vector
RBM

6. IBM Confidential | Do Not Distribute © 2015 IBM Corporation
▪ Watson Beat Pandora station
▪ Suggest recreated version of songs that you like, you’ve been listening to etc.
▪ Producers, composers, music engineers create music based on intent (slow, fast, happy vibrant)
▪ Ability for retail stores, small businesses to create their own music based on original tracks
▪ Loop Pedal Mixing: pedal -> DJ Watson mixer -> amp
▪ https://www.youtube.com/watch?v=qX2eJsj9MiQ
Applications: Cloud based cognitive music service
6

7. IBM Confidential | Do Not Distribute © 2015 IBM Corporation
Backup Slides
7

8. IBM Confidential | Do Not Distribute © 2015 IBM Corporation
Training RBMs
8

9. IBM Confidential | Do Not Distribute © 2015 IBM Corporation
Recreate original Music using RBMs
C# E B
Time1/16 1/16
h: Hidden Layer
(Holds extracted features of visible layer)
x: Visible layer
(Holds pitch information)
9
C E# B
Time1/16 1/16
x~: Learned visible layer
(Holds learned pitch information)
p(h|x) p(x|h)
Demo: Recreated
“Mary had a little lamb”

10. Example 1: Create new music by adding perturbation
10
h: Hidden Layer
(Holds extracted features of visible
Demo: Learned
“Mary” (less perturbation)
Demo: Learned
“Mary” (more perturbation)
x: Visible layer
(Holds initial pitch information)
h1: Hidden Layer
(Holds extracted features of visible
C# rand
B
Time
x: Visible layer
(Holds perturbed pitch information)
x~: Learned visible layer
(Holds learned pitch information)
p(h|x) p(x|h)
E rand E# rand`
B
Time
A rand`
p1(h|x)
C# E B
Time
Demo: Original “Mary
had a little lamb”

11. 11
h: Hidden Layer
(Holds extracted features of visible
Demo: Spooky
String Quartet
x: Visible layer
(Holds initial pitch information)
h1: Hidden Layer
(Holds extracted features of visible
C#
Oct
B
Time
x: Visible layer
(Holds perturbed pitch information)
x~: Learned visible layer
(Holds learned pitch information)
p(h|x) p(x|h)
E Oct
E# Oct`
B
Time
A Oct`
p1(h|x)
C# E B
Time
Demo: Original “String
Quartet”
Example 2: Create new music by steering learning based on emotional intent

12. Example 3: Create new music by learning from two songs and adding perturbation
C# minor
B
Time
Visible layer
(Holds pitch + bias information)
Hidden Layer
(Holds extracted features of visible layer)
minor
E
Song A
D#
Song B
Weights
12
Demo: Learned
“Willie Nelson”
and
”Miley Cyrus”

13. 13
Original Adele
Learned Adele
“Vibrant version”
Learned Adele
“Mellow version”
Example 4: Create new music by steering learning based on emotional intent

14. 14
x1: Visible layer for RBM1
x2: Visible layer for RBM2
h1: Hidden Layer for RBM 1
W1: weights for RBM1
x3: Visible Layer for RBM3
h2: hidden layer for RBM2
h3: hidden layer for RBM3
W2: weights for RBM2
W3: weights for RBM3