LyriSys: An Interactive Support System for Writing Lyrics Based on Topic Transition

LyriSys:
An Interactive Support System
for Writing Lyrics Based on Topic Transition
Kento Watanabe1
Yuichiroh Matsubayashi1, Kentaro Inui1,
Tomoyasu Nakano2, Satoru Fukayama2, Masataka Goto2
1Graduate School of Information Sciences, Tohoku University, Japan
2National Institute of Advanced Industrial Science and Technology (AIST), Japan

Difficulty of Writing Lyrics
2017/3/16 The 22nd annual meeting of the intelligent user interfaces community (IUI2017) 2
It is difficult to consider many techniques at the same time.
Select words whose syllables correspond to the melody notes.
[Austin+2010, Ueda 2010]

D . . . . . . D . . . . . . . . E .
"
. . . D
. . . . . . . .# .
"
.# .
!
.# .
!
-
D E .
"
. . . . D E
. . . . . . . . D E .
"
. . .
. .
. . . . . . . . . .
1
&

17
&
D . . . . . . D . . . . . . . . E .
"
.
24
&
. . . . . . . .# .
"
.# .
!
.# .
!
-
D E .
"
. . . .
30
&
. . . . . . . . D E .
"
. . .
. .
. . . . . . .
I re-mem-ber our love And I re-mem-ber every word un- spo- ken
1 3 1 1 1 1 3 2 1 3Syllable Counts
Lyrics

Difficulty of Writing Lyrics
It is difficult to consider many techniques at the same time.
[Austin+2010, Ueda 2010]
Block1 Block2 Block3
I drop into the ocean
My friend the stormy sea
As long as the blindness
Won't leave me tonight
I felt the wrath of the chosen one
I wanted forgiveness but was given none
Let all the lights scream in my eyes
That 's how I want it to fee
I fell in love
I fell in love
Topic:
Dark
Topic:
Scene
Topic:
Sweet Love
• Verse-Bridge-Chorus segment (called Block).
• Text in each block has an topic.
• The topic transition constitutes a story.

Development of a Lyrics Writing Support System
System enables rich interaction by the topic transition model
Topic Transition Model
Language Model
Sweet love
Scene
Life
Exciting
Ardent Love
…… night
day
light
…
love
know
baby
…
probability
wordword
…
probability
SYSTEM INTELIGENCE
Over 100000 lyrics
…
TRAINING DATA
USER INPUT
hoge
Glassy FM Lead126
1
&

17
&
D . . . . . . D . . . . . . . . E .
"
. . . D E .
"
. . . .
24
&
. . . . . . . .# .
"
.# .
!
.# .
!
-
D E .
"
. . . . D E .
"
. . . .
30
&
. . . . . . . . D E .
"
. . .
. .
. . . . . . . . . .
D . .
35
&
. . . . . . . . . . E .
"
. . . . . . . . . . . . . . . . . .
40
&
. . . . . . . . . . . . . . . . . . . . . .
. .# . . -
44
&
. . . . . . . . . . D E .
"
. . . . . . . . . . .
49
&
E .
"
. . . . . . . . . . . . . . . . . . . . E .
"
. . . . . . . . D
54
&
D E .
"
. . . .
. . . . . . . . . . D . . . . . . . . .
hoge
Glassy FM Lead=126.
1
&

17
&
D . . . . . . D . . . . . . . . E .
"
. . . D E .
"
. . . .
24
&
. . . . . . . .# .
"
.# .
!
.# .
!
-
D E .
"
. . . . D E .
"
. . . .
30
&
. . . . . . . . D E .
"
. . .
. .
. . . . . . . . . .
D . .
35
&
. . . . . . . . . . E .
"
. . . . . . . . . . . . . . . . . .
40
&
. . . . . . . . . . . . . . . . . . . . . .
. .# . . -
44
&
. . . . . . . . . . D E .
"
. . . . . . . . . . .
49
&
E .
"
. . . . . . . . . . . . . . . . . . . . E .
"
. . . . . . . . D
54
&
D E .
"
. . . .
. . . . . . . . . . D . . . . . . . . .
Syllable counts
e.g., “Time”→”Life”→”Love”
Number of blocks
Number of lines
Structure of an entire song
…
Sequence of topics
e.g.,1-3-1-1
SYSTEM OUTPUT
Recommended lyrics
If tomorrow never comes
I remember the day
You realize time flies
…
h
Glas=126.
1
&

17
&
D . . . . . . D .
24
&
. . . . . . . .# .
"
.# .
!
.#
30
&
. . . . . . . . D E .
"
. . .
35
&
. . . . . . . .
. . E .
"
. .
40
&
. . . . . . . . . . . .
44
&
. . . . . . . . . .
=126.
1
&

17
&
D . . . . . .
24
&
. . . . . . . .# .
"
.
30
&
. . . . . . . . D E .
"
35
&
. . . . . . . .
. . E
40
&
. . . . . . . . .
44
&
. . . . . . . . .
…
Unsupervised
Learning
Trial-and-Error

System Introduction

Example of Created Lyrics
. . . . E .
"
. . . . . . . . D
. . D . . . . . . . . .
. . . 1
54
&
D E .
"
. . . .
. . . . . . . . . . D . .
58
&
. . . . . . . . . . - D . . . . .
1
54
&
D E .
"
. . . .
. . . . . . . . . . D . .
58
&
. . . . . . . . . . - D . . . . .
If you really un- der- stand I promise you’-ll be An ex- traor- di- nar- y man
1
44
&
. . . . . . . . . . D E .
"
. . . . . . . . . . .
49
&
E .
"
. . .
. . . . . . . . . . . . . . . . . E .
"
. . . . . . . . D
54
&
D E .
"
. . . .
. . . . . . . . . . D . . . . . . . . .
58
&
. . . . . . . . . . - D . . . . .
If you really understand I promise you'll be An extraordinary man
I don't really understand I promise you'll be Oh supernatural love
Example of lyrics when the user uses our system
Fully automatically recommended lyrics
The love phrases “I promise…” and ”supernatural love” were
recommended when the topic was ⟨sweet love⟩.
1 1 2 3 1 2 2 1 1 5 1Syllable
Lyrics
Inputted Topic: Sweet Love

Writing Support System: LyriSys

Implementation of LyriSys

Recommendation Algorithm
LyriSys searches the word strings so that the word
probability is large according to the beam search.
I
You
So
…
1 syllable words
need
don’t
when
…
1 syllable words
always
really
better
…
2 syllable words
remember
everything
understand
…
3 syllable words
: Word probability 𝑃(𝑤𝑜𝑟𝑑)|𝑤𝑜𝑟𝑑)+,, 𝑡𝑜𝑝𝑖𝑐)
Example of beam search when syllable count is “1-1-2-3” and topic is “sweet love”.

How does the system handle topic
Block1 Block2 Block3
I felt the wrath of the chosen one
I wanted forgiveness but was given none
Let all the lights scream in my eyes
That 's how I want it to fee
I fell in love
I fell in love
Topic:
Dark
Topic:
Scene
Topic:
Sweet Love
𝑃(𝑡𝑜𝑝𝑖𝑐3|𝑡𝑜𝑝𝑖𝑐,) 𝑃(𝑡𝑜𝑝𝑖𝑐4|𝑡𝑜𝑝𝑖𝑐3)
∏𝑃(𝑤𝑜𝑟𝑑|𝑡𝑜𝑝𝑖𝑐,) ∏𝑃(𝑤𝑜𝑟𝑑|𝑡𝑜𝑝𝑖𝑐3) ∏𝑃(𝑤𝑜𝑟𝑑|𝑡𝑜𝑝𝑖𝑐4)
The story is represented by two kinds of probabilities
Transition probability
Word probability
Enhanced Hidden Markov Model [watanabe+ 2014]

How to Get Probabilities ?
𝑃(𝑤𝑜𝑟𝑑|𝑡𝑜𝑝𝑖𝑐)𝑃(𝑡𝑜𝑝𝑖𝑐)|𝑡𝑜𝑝𝑖𝑐)+,)
Enhanced HMM [Watanabe+2014]
(1) Transition prob: (2) Word prob:
……
night
day
light
…
love
know
baby
…
probability
wordword
…
Scene
Exciting
Slang
Time
Sweet love
Life
Foreign
Dark
Religious
Ardent love
Scene
Exciting
Slang
Time
Sweetlove
Life
Foreign
Dark
Religious
Ardentlove
Transition Matrix
𝑡𝑜𝑝𝑖𝑐)
𝑡𝑜𝑝𝑖𝑐)+,Previous
73 5 4 1 7 4 5 1 1 1
6 62 3 1 15 1 5 1 5 1
9 5 42 6 8 9 13 1 5 1
1 1 3 85 1 5 3 1 2 1
6 9 2 1 72 1 3 1 7 1
8 1 6 8 1 71 4 1 1 1
6 5 5 2 7 2 67 1 4 1
1 1 0 1 1 1 1 93 1 1
2 6 2 2 13 0 4 0 69 1
2 4 2 3 4 1 3 0 4 78 (%)
System automatically learns
(1) typical topic transitions and
(2) semantically appropriate wordings
from a large collection of human lyrics.
probability
Over 10000 lyrics
Unsupervised
learning
…

User Feedback

User Feedback: Setup
nWriting Japanese Lyrics
p5 Japanese (One user was a school teacher of music).
pWe randomly selected songs from RWC music database.
n4 Tasks
1. Base line : Without interface
2. Method 1: Fully Automatic lyrics generation
Users can only select topics and input syllables,
cannot edit/select the generated lyrics.
3. Method 2: Interaction without topic
System calculates the simple N-gram probability,
cannot handle topic.
4. Proposed : LyriSys

User Feedback: Results
Task Positive Comments Negative Comments
(Base line)
Without
system
In comparison to other tasks, it was
comfortable in not specifying the
number of syllable.
It was difficult to come up with the
words that satisfy the melody of song,
because of lacking in vocabulary.
(Method 1)
Automatically
generation
It was easy to write the lyrics
because I didn’t need to determine
which words to use.
I sometimes felt boring because
users couldn’t edit the generated
lyrics.
(Method 2)
Interaction
without topic
It was useful to select the candidate
of lyrics when the generated result
was partially good.
It was difficult to write the lyrics
that represent the story, because
only a limited variety of words are
generated.
(Proposed)
LyriSys
1) In comparison to the previous
method 2, selecting topics made it
easy to write the lyrics that specifies
my intention. 2) The generated
lyrics are more expressive than the
result of other interface because of
the consideration of topic.
1) The list of the 10 topics was too
restricted and coarse-grained. 2)
Although the system generates an
abstract story, I thought that it
would be interesting if the system
could generate a concrete story.

Conclusions and Future Work
Our interaction design
System is designed as a recommendation system.
User can input the structure of entire an song and the story.
Our system intelligence
System automatically learns typical topic transitions and
semantically appropriate wordings from a large collection of
human lyrics.
Novel lyric-writing system: LyriSys
It might be too much of a burden for the user to specify the
number of syllable counts.
We plan to introduce extended functions on the Web.
Next Step

Appendix

Related works
Many researchers have studied rhyme of lyrics
[Nichols+09 Genzel+10, Abe+10 Ramakrishnan+10, Berbieri+12].
Prior studies report systems that can generate only a
single line of lyrics independently of the rest [Abe+2012]
Our system has a strong advantage in capturing
topic transition (i.e., story) of an entire lyrics.

Story Estimation

What topics were learned ?
z Label Representative words in each topic: top words from 𝑃(𝑤𝑜𝑟𝑑|𝑧)
1 Sweet love love,know,baby,want,need,make,wan,feel,one,tell,give,more,heart,good,only,hold,'cause,please,
kiss,mine,stay,true,cry,crazy,touch,someone,miss,somebody,mean,woman,nobody,enough,care
2 scene night,day,light,eye,fall,dream,sun,sky,rain,shine,fly,into,home,star,walk,blue,wind,burn,fire,mo
on,christmas,cold,morning,watch,sleep,tear,dark,water,close,summer,open,remember,river,
3 exciting yeah,down,come,hey,gon,ooh,back,rock,keep,right,dance,tonight,stop,here,run,roll,alright,whoa,
everybody,round,turn,ready,move,music,ride,party,bring,doo,c'mon,slow,beat,high,gim,ohh
4 slang nigga,shit,fuck,bitch,cause,money,niggaz,'em,ass,hit,real,hoe,game,wit,big,street,fuckin,bout,get
tin,rap,hood,gun,block,motherfucker,dick,check,thug,catch,smoke,tryin,young,throw,straight
5 time have,all,time,say,never,see,way,take,think,life,find,try,thing,leave,too,look,nothing,lose,believe,l
ive,always,everything,mind,change,long,something,wait,much,break,end,wrong,word,inside
6 Ardent love get,like,girl,off,body,put,boy,shake,hot,show,work,club,floor,lady,drop,cuz,sexy,got,sex,jump,lo
w,lookin,freak,top,shawty,damn,pop,aint,boom,bounce,chick,dont,hair,thang,lil,shorty,push
7 life out,well,little,call,old,play,friend,talk,new,town,bad,use,car,kid,mama,drive,sit,lot,door,pay,drink
,meet,pretty,house,buy,first,daddy,fun,guy,wear,next,write,phone,bed,school,hang,trouble,blues
8 foreign que,por,con,amor,como,una,quiero,para,sin,esta,pero,todo,solo,las,cuando,hay,soy,corazon,voy,v
ida,del,porque,los,tengo,bien,ella,estoy,ser,vez,hoy,aqui,les,este,puedo,siempre,dale,tan,quien
9 dark die,hand,head,lie,blood,face,dead,kill,fight,death,fear,hell,black,while,hate,wall,line,cut,scream,
skin,bleed,pull,speak,become,bone,devil,full,thought,sick,blind,human,chain,build,stone,breath,
10 religious dem,god,man,lord,yuh,sing,nah,world,jesus,king,child,nuh,hear,gal,name,fus,song,praise,people
,heaven,bear,inna,soul,pray,holy,war,free,pon,peace,hallelujah,glory,joy,thank,bless,mary,mek

Enhanced HMM
Generation process
For each topic z = 1, … , 𝐽:
Draw 𝜃>→~𝐷𝑖𝑟 𝛼
Draw 𝜙>~𝐷𝑖𝑟(𝛽)
For each lyrics 𝑚 = 1,2, … , 𝑀:
For each Block 𝑏 = 1,2, … , 𝐵J:
Draw 𝑧K~𝑀𝑢𝑙𝑡𝑖 𝜃>NOP→
For each word 𝑤 in block 𝑏:
Draw 𝑤~𝑀𝑢𝑙𝑡𝑖(𝜙>N
)
𝑤
𝑁,
𝑀
𝑧,
𝑁3 𝑁4
𝑤
𝑧3
𝑤
𝑧4
𝜙>
𝐽
𝜃>→
𝐽
𝛽
𝛼
Plate notation of enhanced HMM
𝑃 𝐿𝑦𝑟𝑖𝑐𝑠J = U V 𝜃>NOP→ V 𝜙>NWNX
YN
Z[,
]
K[,>_``
Generation probability of lyrics
topic
Transition distribution
Word distribution

Dataset
nPurchasable at online shop.
＄260

Our interaction design
System is designed as a recommendation system.
USER INPUT SYSTEM OUTPUT
hoge
Glassy FM Lead6
1
&

17
&
D . . . . . . D . . . . . . . . E .
"
. . . D E .
"
. . . .
24
&
. . . . . . . .# .
"
.# .
!
.# .
!
-
D E .
"
. . . . D E .
"
. . . .
30
&
. . . . . . . . D E .
"
. . .
. .
. . . . . . . . . . D . .
35
&
. . . . . . . .
. . E .
"
. . . . . . . . . . . . . . . . . .
hoge
Glassy FM Lead=126.
1
&

17
&
D . . . . . . D . . . . . . . . E .
"
. . . D E .
"
. . . .
24
&
. . . . . . . .# .
"
.# .
!
.# .
!
-
D E .
"
. . . . D E .
"
. . . .
30
&
. . . . . . . . D E .
"
. . .
. .
. . . . . . . . . . D . .
35
&
. . . . . . . .
. . E .
"
. . . . . . . . . . . . . . . . . .
Syllable counts
e.g., “Time”→”Life”→”Love”
Number of blocks
Number of lines
Structure of an entire song
…
Sequence of topics
e.g., 1-3-1-1
Trial-and-Error
Recommended lyrics
If tomorrow never comes
I remember the day
You realize time flies
…
h
Glas=126.
1
&

17
&
D . . . . . . D .
24
&
. . . . . . . .# .
"
.# .
!
.#
30
&
. . . . . . . . D E .
"
. . .
35
&
. . . . . . . .
. . E . . .
=126.
1
&

17
&
D . . . . . .
24
&
. . . . . . . .# .
"
.#
30
&
. . . . . . . . D E .
"
35
&
. . . . . . . .
. . E
…

Our system intelligence
System automatically learns typical topic transitions and semantically
appropriate wordings from a large collection of human lyrics.
Topic Transition Model
Language Model
Sweet love
Scene
Life
Exciting
Ardent Love
……
night
day
light
…
love
know
baby
…
probability
wordword
…probability
Over 100000 lyrics
…
Unsupervised
Learning
SYSTEM INTELIGENCE
TRAINING DATA

Result: Automatic lyrics generation
# Block1: time
I know everything is
And I remember every word unspoken
Remember waiting there inside my head
Conversation circles
# Block2: time
I know everything is
And I remember every word every thought
Tomorrow is another story about love
The situation became
# Block3: life
And I remember everything
California face reality
And you realize you're looking
You'll always remember
And you'll always be
And everybody knows
# Block4: ardent love
I know you're watching every dream
I really don't want nobody sitting on top
And I remember feeling like letting go
I really don't matter
# Block5: sweet love
I remember December
I promise you'll remember the first degree
I don't really understand
I promise you'll be
Oh supernatural love

Result: User interaction with system
# Block1: time
I remember our love
And I remember every word unspoken
Remember waiting there inside my head
Remember I believe
# Block2: time
I know everything ends
But I remember every word every thought
Tomorrow is another story about love
So appreciate it all
# Block3: life
You can remember yesterday
Reminiscing on everyone
And you realize you're looking
You'll always remember
And every single day
You probably say
# Block4: ardent love
It's all because you're wasting time
But you're the one you're looking out for
And I remember feeling like letting go
'Cause I don't need anybody
# Block5: sweet love
I remember December
And every little tenderness, and I believe
If you really understand
I promise you'll be
An extraordinary man

This result shows that the created lyrics correspond to
the input parameters (i.e., syllables and topics)
We can see the sentimental phrases “in my mind” and ”I just
wanna be with you” were created when the topic was ⟨Sweet
Love⟩.
Example 2 (made by user)
世界中でこんなにすれ違い.涙を見せず独り振る舞うだけで.あなたのそばに.
(We could not understand each other in this world. I didn’t show my tears, I was lonely. I just wanna be with you.)
世界中でこんなにたくさんの.心に決めて思い出した思い出に.あなたのそばに.
(So many things in the world. Memories that I remembered in my mind. I just wanna be with you.)
Block 2, Topic: Sweet Love
se- ka- i- ju- u de ko-n- na- ni su-re-chi-ga- i na-mi- da wo mi-se-zu hi-to-ri hu-ru-ma-u da-ke de a- na- ta no so- ba ni
Fully automatically generated lyrics.
Example outcome of an user’s interactions with the system.

This result shows that the created lyrics correspond to
the input parameters (i.e., syllables and topics)
We can see the scenic phrases “the way of memories” and
”wet in the rain” were created when the topic was ⟨Scene⟩.
Example (made by user)
思い出の坂道は久しぶりの昼下がりへ.恋人の足跡はアスファルトの雨に濡れる.
o-mo-i- de no sa-ka-mi-chi wa hi- sa-si-bu-ri no hi- ru sa-ga-ri e ko- i- bi-to no a- shi-a- to ha a- su-fa-ru-to no a- me ni nu-re-ru
(Afternoon came to the way of memories after a long time. Lovers' footprint get wet in the rain on asphalt.)
思い出の坂道を.雨上がりの交差点で. 思い出の坂道を.心にない雨に濡れて.
(The way of memories. At the intersection of the rain. The way of memories. I got wet in the rain without heart.)
Block 1, Topic: Scene
Fully automatically generated lyrics.
Example outcome of an user’s interactions with the system.

LyriSys: An Interactive Support System for Writing Lyrics Based on Topic Transition

Empfohlen

Empfohlen

Weitere ähnliche Inhalte

Ähnlich wie LyriSys: An Interactive Support System for Writing Lyrics Based on Topic Transition

Ähnlich wie LyriSys: An Interactive Support System for Writing Lyrics Based on Topic Transition (20)

Kürzlich hochgeladen

Kürzlich hochgeladen (20)

LyriSys: An Interactive Support System for Writing Lyrics Based on Topic Transition