Video from PhD thesis defence: Marius Miron - Source Separation Methods for Orchestral Music: Timbre-Informed and Score-Informed Strategies More info here: http://mariusmiron.com/research/thesis/ http://mtg.upf.edu/node/3863 Video here: https://youtu.be/FDrFTTVOtr0

1. Source separation methods for
orchestral music: timbre-informed
and score-informed strategies
Universitat Pompeu Fabra, Barcelona, Music Technology Group
Thesis supervisors:
Dr. Jordi Janer
Dr. Emilia Gómez
Marius Miron
Thesis defence committee:
Dr. Emmanuel Vincent
Dr. Xavier Serra
Dr. Maximo Cobos
Doctoral thesis presentation

2. Outline
2
ApplicationsDatasetsIntroduction Deep learningMatrix decomposition Conclusions

3. Outline
3
Applications
PART V
Datasets
PART II
Introduction
PART I
Deep learning
PART IV
Matrix decomposition Conclusions
PART III

4. Introduction

5. Music source separation
5
Audio sources
+
Music source
separation
Estimated audio sources

6. Orchestral music
6

7. Orchestral music source separation
7

8. Orchestral music source separation
8

9. Informed source separation
9
Audio sources
+
Music source
separation
Estimated audio sources
Musical score
Sources’ timbre

10. Timbre-informed source separation
10
Frequency (Hz)
STFTmagnitude

11. Score-informed source separation
11
Frequency(Hz)
STFT magnitude spectrogram
Time (s)
Mahler’s Symphony no. 1
Mixture
French horns

12. Score-informed source separation
12
Frequency(Hz)
STFT magnitude spectrogram
Time (s)
Mahler’s Symphony no. 1
Mixture
French horns

13. Score-informed source separation
13
Frequency(Hz)
STFT magnitude spectrogram
Time (s)
Mahler’s Symphony no. 1
Mixture
French horns

14. Informed source separation
14
Audio sources
+
Music source
separation
Estimated audio sources
Musical score
Isolated samples
Score
follower
Timbre
learning

15. Context
15
Performances as Highly Enriched aNd
Interactive Concert eXperiences
Score following
Concert notes
Conductor gestures
Visualization: chords

16. Motivation
16
Music source
separation
Active
music
listening
Music
Analysis

17. Motivation
17
Active music listening

18. Motivation
18
Music analysis

19. Challenges
19
DSD100 dataset Bach10 dataset

20. Challenges
20
Multi-microphone recordings: leakage between sources

21. Challenges
21
Large number of sources of similar timbre

22. Challenges
22
Less disjoint representations
Time (s)
Mahler’s Symphony no. 1
Frequency(Hz)

23. Challenges
23

24. Opportunities
24
Multi-microphone source separation

25. Opportunities
25

26. Opportunities
26

27. Opportunities
27
Tempo
Dynamics
Timbre
Local timing
Widmer, G., & Goebl, W. (2004). Computational models of expressive music performance:
The state of the art. JNMR, 33(3), 203-216.

28. Audio-to-score alignment
28
Audio-to-score
Alignment
Global
Alignment
Tempo
Dynamics
Timbre
Local timing

29. Audio-to-score alignment
29
Score
follower

30. Audio-to-score alignment
30
Alignment
Errors
Tempo
Dynamics
Timbre
Local timing

31. Note refinement
31
Score
follower
Niedermayer, B. (2012). Audio-to-score alignemnt: data acquisition in the context of computational musicology.
PhD Thesis, Johannes Kepler University, Linz

32. Opportunities
32
Part III
Tempo
Dynamics
Timbre
Local timing

33. Opportunities
33
Widmer, G., & Goebl, W. (2004). Computational models of expressive music performance:
The state of the art. JNMR, 33(3), 203-216.
Part IV
Tempo
Dynamics
Timbre
Local timing

34. Opportunities
34

35. Contributions
• Note refinement using image processing to fix local misalignments
• Deep learning framework for classical music source separation
• Deep learning score-informed source separation
35

36. Contributions
• Reproducible research (datasets, frameworks, code, results)
• Orchestral dataset with score annotations
• Cloud-based source separation framework on Repovizz.
• Deep learning repository for audio applications.
36

37. Datasets

38. PHENICX-Anechoic
38
Piece Dura
tion
Period No.
sources
Total no.
instruments
Max. instruments/
source
Mozart 3min
47s
classical 8 10 2
Beethoven 3min
11s
classical 10 20 4
Mahler 2min
12s
romantic 10 30 4
Bruckner 1min
27s
romantic 10 39 12

39. PHENICX-Anechoic
39

40. PHENICX-Anechoic
40
Roomsim generation

41. Other dataset
Bach10 Dataset
10 Bach chorales, 20-40 seconds each
Perfectly and automatically aligned scores
41
Duan, Z. and Pardo B.,(2011), Bach10 dataset

42. PHENICX recordings
42

43. Source separation using
matrix decomposition

44. Score-informed source separation
44
Logfrequency
(1/4semitoneres)
Time (s)
Trained timbre bases Time gains initialised with score
Magnitude

45. Note refinement
45
Score
follower
Niedermayer, B. (2012). Audio-to-score alignemnt: data acquisition in the context of computational musicology.
PhD Thesis, Johannes Kepler University, Linz

46. Note refinement for source separation
46
Logfrequency
(1/4semitoneres)
Time (s)
Trained timbre bases Time gains initialised with score
Magnitude
Miron et al (2015). Improving score-informed source separation for classical music through note refinement ISMIR 2015

47. Note refinement using NMF gains
47
Miron et al (2015). Improving score-informed source separation for classical music through note refinement ISMIR 2015
Time gains after NMF
Time (s)
Blob detection for a single note

48. Note refinement using NMF gains
48
Miron et al (2015). Improving score-informed source separation for classical music through note refinement ISMIR 2015
Time gains after NMF
Time (s)
Blob detection for a single note

49. Note refinement using pitch salience
49
Miron et al (2014). Audio-to-score alignment at the note level for orchestral recordings, ISMIR 2014
frequencyrelativetonote'sf0(centbins)
time (seconds)
➩
A B
time (seconds)

50. Note refinement using NMF gains
50Miron et al (2015). Improving score-informed source separation for classical music through note refinement ISMIR 2015
Evaluation with Bach10 dataset: alignment
NMF gains
score follower
NMF gains
pitch salience
pitch salience
score follower
Error threshold (seconds)
Alignmentrate

51. Note refinement using NMF gains
51
Miron et al (2015). Improving score-informed source separation for classical music through note refinement ISMIR 2015
Source separation evaluation metrics - BSS EVAL :
• Signal to Distortion Ratio (SDR)
• Signal to Interference Ratio (SIR)
• Signal to Artefacts Ratio (SAR)
• Image to Spatial Distortion Ratio (ISR)

52. Note refinement using NMF gains
52
Miron et al (2015). Improving score-informed source separation for classical music through note refinement ISMIR 2015
Time gains after NMF
Time (s)
Blob detection for a single note

53. Note refinement using NMF gains
53
Miron et al (2015). Improving score-informed source separation for classical music through note refinement ISMIR 2015
A
B
C
D
E
F
ground truth
misaligned
refined pitch salience
refined NMF gains
implicit
refined NMF gains
submatrix
time & frequency
Evaluation with Bach10 dataset: source separation

54. Multi-channel source separation
54
Miron et al (2016). Score-informed source separation for multi-channel orchestral recordings,
Journal of Electrical Engineering and computer science 2016

55. Multi-channel source separation
55
Miron et al (2016). Score-informed source separation for multi-channel orchestral recordings,
Journal of Electrical Engineering and computer science 2016
Ref1

56. Multi-channel source separation
56
Miron et al (2016). Score-informed source separation for multi-channel orchestral recordings,
Journal of Electrical Engineering and computer science 2016
Ref2

57. PARAFAC gains estimation
57
[ ]…
Ref2
Miron et al (2016). Score-informed source separation for multi-channel orchestral recordings,
Journal of Electrical Engineering and computer science 2016

58. Multi-channel source separation
58
GT
Ali
Ext
Ref1
Ref2
ground truth
refined gains
extended
refined gains multi
submatrix
aligned
Miron et al (2016). Score-informed source separation for multi-channel orchestral recordings,
Journal of Electrical Engineering and computer science 2016

59. Multi-channel source separation
59
SDR(dB)
Mozart Beethoven Mahler Bruckner
Miron et al (2016). Score-informed source separation for multi-channel orchestral recordings,
Journal of Electrical Engineering and computer science 2016

60. Multi-channel source separation
60
SDR(dB)
Mozart Beethoven Mahler Bruckner
Miron et al (2016). Score-informed source separation for multi-channel orchestral recordings,
Journal of Electrical Engineering and computer science 2016

61. Multi-channel source separation
61
SIR(dB)
Mozart Beethoven Mahler Bruckner
Miron et al (2016). Score-informed source separation for multi-channel orchestral recordings,
Journal of Electrical Engineering and computer science 2016

62. Multi-channel source separation
62
SAR(dB)
Mozart Beethoven Mahler Bruckner
Miron et al (2016). Score-informed source separation for multi-channel orchestral recordings,
Journal of Electrical Engineering and computer science 2016

63. Source separation using
deep learning

64. Deep learning source separation
64
(J,T,F)
(T,F)
Neural
network

65. Deep learning source separation
65
(J,T,F)
(T,F)
Neural
network
Advantages:
- faster processing
- non-linear, multi-layered
Disadvantages:
- Slower training
- Dependent on training data

66. Deep learning source separation
66
conv1
fshape(1,30)
stride(1,4)
conv2
fshape(20,1)
stride(1,1)
dense1
256
inverse
conv2
inverse
conv1
(1,T,F) (J,T,F)
(30,T,F)
11
(30,T,F)
2 2
dense2
Jx30xTxF
1 1
=
with
(J,T,F) (J,T,F)
(T,F)
Jx(30,T,F)
11
Jx(30,T,F)
2 2
Chandna et al,(2017). Monaural source separation using convolutional neural networks
LVA/ICA 2017

67. Data driven methods
67
DSD100 dataset Bach10 dataset
We do not have training data for orchestral music

68. Data generation method
68
Tempo
Dynamics
Timbre
Local timing
Widmer, G., & Goebl, W. (2004). Computational models of expressive music performance:
The state of the art. JNMR, 33(3), 203-216.

69. Data generation method
69
Tempo
Dynamics
Timbre
Local timing
Audio
Isolated
Instruments

70. Data generation method
70
Miron et al,(2017). Generating data to train neural networks for classical music source separation
SMC 2017
Score
Augmented
Synthesis
Audio
Augmentation
Multi-track
Renditions

71. Data generation method
71
Miron et al,(2017). Generating data to train neural networks for classical music source separation
SMC 2017
STFT
Data
Processing
Multi-track
Renditions
CNN
Training
Trained model

72. Data generation method
72
Miron et al,(2017). Generating data to train neural networks for classical music source separation
SMC 2017
STFT
Data
Processing
Multi-track
Renditions
CNN
Training
Trained model
STFT
Data
Processing
Target
piece
CNN
Separation
Separated
Sources

73. Score-informed separation with CNN
73
Score-based binary matrices
Miron et al,(2017). Monoaural score-informed source separation for classical music using deep convolutional neural networks.
ISMIR 2017

74. 74
Score-based
soft masks
Score-informed separation with CNN

75. 75
STFT magnitude spectrogram
Score-informed separation with CNN

76. 76
Score-filtered
spectrum
Score-informed separation with CNN

77. Score-informed separation with CNN
77
Miron et al,(2017). Monoaural score-informed source separation for classical music using deep convolutional neural networks.
ISMIR 2017
conv1
fshape(1,30)
stride(1,4)
conv2
fshape(20,1)
stride(1,1)
dense1
256
inverse
conv2
inverse
conv1
(J,T,F)
(J,T,F)
(30,T,F)
11
(30,T,F)
2 2
=
with
(J,T,F) (J,T,F)
(T,F)
(30,T,F)
11
(30,T,F)
2 2
dense2

78. Experiments
CNN autoencoder NMF
78
Multi-source filter model
Score informed
Trained on RWC
vs
Score-informed
Trained on renditions
synthesised with RWC
Duan, Z. and Pardo B.,(2011), Bach10 dataset

79. Experiments
PA
79
Automatically aligned score
Tolerance window around
-Onsets
-Offsets
andPerfectly aligned score
Duan, Z. and Pardo B.,(2011), Bach10 dataset

80. Results
80
Mean(dB)
SDR SIR SAR
Miron et al,(2017). Monoaural score-informed source separation for classical music using deep convolutional neural networks.
ISMIR 2017

81. Results
81
Mean(dB)
SDR SIR SAR
Miron et al,(2017). Monoaural score-informed source separation for classical music using deep convolutional neural networks.
ISMIR 2017

82. Results
82
Mean(dB)
SDR SIR SAR
Miron et al,(2017). Monoaural score-informed source separation for classical music using deep convolutional neural networks.
ISMIR 2017

83. Results
How much data we need?
83
No. of instances
SDR(dB)
Sampling:
Bootstrapping
with replacement
Fixed samples
Miron et al,(2017). Monoaural score-informed source separation for classical music using deep convolutional neural networks.
ISMIR 2017

84. Applications

85. Applications
85
Repovizz server Source separation server
Cloud-based source separation

86. Applications
86
https://youtu.be/4xthvs7O9q0

87. Applications
87
https://youtu.be/4xthvs7O9q0

88. Applications
88

89. Conclusions

90. Contributions
90

91. Contributions
91

92. Contributions
92
blob note before blob note after
t=0.2
t=1
t=1.4
time(analysis windows)
centbins

93. Contributions
93
[ ]…

94. Contributions
94
Tempo
Dynamics
Timbre
Local timing
Audio
Isolated
Instruments

95. Contributions
95

96. Contributions
96
Code:
https://github.com/MTG/DeepConvSep Data: .wav and .mat
DOIDOI 10.5281/zenodo.100913610.5281/zenodo.1009136
Reproducible research

97. Contributions
97

98. Media coverage
98

99. Media coverage
99

100. Future work
• Instrument recognition using deep learning
• Deepconvsep is already a starting point and a baseline
• Better databases : NSynth to generate training data
• Explore other music traditions, genres, and a more flexible context
100

101. Publications by the author
101
Peer-reviewed journals
• Miron, M., Carabias-Orti, J.J., Bosch, J.J., Gomez, & Janer, J. (2016). Score-Informed Source Separation
for Multichannel Orchestral Recordings. Journal of Electrical and Computer Engineering
Full articles in peer-reviewed conferences
• Miron, M., Gomez, & Janer, J. (2017). Monaural score-informed source separation for classical music using
convolutional neural networks. In Proceedings of the 18th International Society for Music Information
Retrieval Conference (ISMIR)
• Miron, M., Gomez,E., & Janer, J. (2017). Generating data to train convolutional neural networks for
classical music source separation. In Sound and Music Computing Conference (SMC)
• Martel, H., Miron, M. (2017). Data augmentation for deep learning source separation of HipHop songs. In
10th International Workshop on Machine Learning and Music (MML)
• Chandna, P.,Miron, M., Janer, J., & Gomez, E. (2017). Monoaural Audio Source Separation Using Deep
Convolutional Neural Networks. In 13th International Conference on Latent Variable Analysis and Signal
Separation (LVA/ICA)
• Miron, M., Carabias-Orti, J.J., & Janer, J. (2015). Improving score-informed source separation for classical
music through note refinement. In 16th International Society for Music Information Retrieval Conference
(ISMIR)
• Miron, M., Carabias-Orti, J.J., & Janer, J. (2014). Audio-to-score alignment at the note level for orchestral
recordings. In 15th International Society for Music Information Retrieval Conference (ISMIR)

102. Source separation methods for
orchestral music: timbre-informed
and score-informed strategies
Universitat Pompeu Fabra, Barcelona, Music Technology Group
Thesis supervisors:
Dr. Jordi Janer
Dr. Emilia Gómez
Marius Miron
Thesis defence committee:
Dr. Emmanuel Vincent
Dr. Xavier Serra
Dr. Maximo Cobos
Doctoral thesis presentation