The document summarizes an experiment using human participants to simulate the functions of a neural network. The participants worked together to encode an image by describing its elements, then generated a poem by collectively proposing and voting on words and phrases. This process aimed to demonstrate how machines encode and generate text from images. The experiment showed the participants taking turns proposing words or phrases to build a poem corresponding to several different encoded images. In the conclusion, the author acknowledges contributions from researchers in the field of neural networks and natural language processing.
11. Generating Text (decoding) (1)
Toe upon ___, a snowing flesh,
A gold of lemon, root and rind,
She sifts in sunlight down the ____
With nothing on. Nor on her mind.
We spy beneath the banister
A constant thresh of thigh on thigh--
Her lips imprint the swinging ___
That parts to let her parts go __.
One-woman waterfall, she wears
Her slow descent like a long cape
And pausing, on the final stair
Collects her motions into shape.
Nude Descending a Staircase,
Duchamp, 1912 X. J. Kennedy (1961)
12. Generating Text (decoding) (1)
Toe upon toe, a snowing flesh,
A gold of lemon, root and rind,
She sifts in sunlight down the ____
With nothing on. Nor on her mind.
We spy beneath the banister
A constant thresh of thigh on thigh--
Her lips imprint the swinging ___
That parts to let her parts go __.
One-woman waterfall, she wears
Her slow descent like a long cape
And pausing, on the final stair
Collects her motions into shape.
Nude Descending a Staircase,
Duchamp, 1912 X. J. Kennedy (1961)
13. Generating Text (decoding) (1)
Toe upon toe, a snowing flesh,
A gold of lemon, root and rind,
She sifts in sunlight down the stairs
With nothing on. Nor on her mind.
We spy beneath the banister
A constant thresh of thigh on thigh--
Her lips imprint the swinging ___
That parts to let her parts go __.
One-woman waterfall, she wears
Her slow descent like a long cape
And pausing, on the final stair
Collects her motions into shape.
Nude Descending a Staircase,
Duchamp, 1912 X. J. Kennedy (1961)
14. Generating Text (decoding) (1)
Toe upon toe, a snowing flesh,
A gold of lemon, root and rind,
She sifts in sunlight down the stairs
With nothing on. Nor on her mind.
We spy beneath the banister
A constant thresh of thigh on thigh--
Her lips imprint the swinging air
That parts to let her parts go __.
One-woman waterfall, she wears
Her slow descent like a long cape
And pausing, on the final stair
Collects her motions into shape.
Nude Descending a Staircase,
Duchamp, 1912 X. J. Kennedy (1961)
15. Generating Text (decoding) (1)
Toe upon toe, a snowing flesh,
A gold of lemon, root and rind,
She sifts in sunlight down the stairs
With nothing on. Nor on her mind.
We spy beneath the banister
A constant thresh of thigh on thigh--
Her lips imprint the swinging air
That parts to let her parts go by.
One-woman waterfall, she wears
Her slow descent like a long cape
And pausing, on the final stair
Collects her motions into shape.
Nude Descending a Staircase,
Duchamp, 1912 X. J. Kennedy (1961)
16. • Recurrent Neural Networks (RNNs)
Input
Output
Generating Text (decoding) (2)
<START> Toe upon
Toe upon toe
17. • Recurrent Neural Networks (RNNs)
Input
Output
Generating Text (decoding) (2)
<START> Toe upon
Toe upon toe
18. • Recurrent Neural Networks (RNNs)
Input
Output
Generating Text (decoding) (2)
<START> Toe upon
Toe upon toe
Aardvark 0.1%
.
.
.
Toe 20%
.
.
.
Zebra 0.2%
20. Exercise Goals
• Humans become a machine
• Each group is a neural machine
• Each individual is a neuron
• Adaptation
• Communicate in words (not numbers)
• More flexibility
21. Encoding an image
• Describe what you see
• Pieces of an image
• Objects in a scene
• A story for the scene
22. Encoding an Image – Pieces
• For each piece, choose 3 words that describe its content
Blue
Uniform
Empty
Sea
Blue
Calm
Sky
Cloud
Sunny
Cloud
Sky
Stick
Sky
Cloud
Cotton
Tightrope
Man
Balancing
Clouds
Sky
Lines
Grey
Cloud
Tripod
City
Landscape
River
Skyscrapers
Landscape
Horizon
Corner
Buildings
Window
Puddle
Mirror
Pavement
Riverside
Town
Park
Buildings
Roads
Below
City
Brownish
Metal
River
Brown
City
23.
24. Encoding an Image – Objects (individual)
• Group together pieces to identify up to 5 objects
• Describe each with 1 word
• A story should start forming
Blue
Uniform
Empty
Sea
Blue
Calm
Sky
Cloud
Sunny
Cloud
Sky
Stick
Sky
Cloud
Cotton
Tightrope
Man
Balancing
Clouds
Sky
Lines
Grey
Cloud
Tripod
City
Landscape
River
Skyscrapers
Landscape
Horizon
Corner
Buildings
Window
Puddle
Mirror
Pavement
Riverside
Town
Park
Buildings
Roads
Below
City
Brownish
Metal
River
Brown
City
Sky
Man
Roof
City
25. Encoding an Image – Objects (group)
• Reach an agreement as a group (up to 5 objects, 1 word each)
Sky
Man Tightrope
Roof
Town
Sky
Someone
Reflection
Town
Sky
Man Tightrope
Roof
City
26.
27. Encoding an Image – Scene/Story
• Decide on story of up to 5 words
Man
walks
tightrope
above
town
Sky
Man Tightrope
Roof
Town
28. Man walks tightrope above town
Blue
Uniform
Empty
Sea
Blue
Calm
Sky
Cloud
Sunny
Cloud
Sky
Stick
Sky
Cloud
Cotton
Tightrope
Man
Balancing
Clouds
Sky
Lines
Grey
Cloud
Tripod
City
Landscape
River
Skyscrapers
Landscape
Horizon
Corner
Buildings
Window
Puddle
Mirror
Pavement
Riverside
Town
Park
Buildings
Roads
Below
City
Brownish
Metal
River
Brown
City
Sky
Man Tightrope
Roof
Town
The Encoded Image
29. Generating the Poem (Decoding)
• React to what we saw
• Write a poem word-by-word
• Individually propose alternative continuations
• Collaboratively select one
30. Decoding – Choose First Word
I
He
The
High
I 4
He 1
The 0
High 1
• Each person
proposes 1 word
• Each person votes
(up to 2 votes) for
‘best’ word
• Cannot vote
yourself!
• Count votes
• Write highest
scoring word to
poem
• If tied, repeat
voting only
between tied
words (or flip coin)
POEM
I
31. Decoding – Choose Next Word
really
only
walk
stand
really 3
only 0
walk 0
stand 1
• Each person
proposes 1 word
• Each person votes
(up to 2 votes) for
‘best’ word
• Cannot vote
yourself!
• Count votes
• Write highest
scoring word to
poem
• If tied, repeat
voting only
between tied
words (or flip coin)
POEM
I really
32. Decoding – Choose Next Word
knowing
walking
trotting
standing
knowing 0
walking 3
trotting 0
standing 1
• Each person
proposes 1 word
• Each person votes
(up to 2 votes) for
‘best’ word
• Cannot vote
yourself!
• Count votes
• Write highest
scoring word to
poem
• If tied, repeat
voting only
between tied
words (or flip coin)
POEM
I really had an easy way of walking
33. Decoding – Speed it up
• One word at a time is too slow for humans…
• Propose whole phrases (as many words as you like)
34. Decoding – Choose Next Phrase
However tall
I never thought
Looking at a small
Despite
However tall 0
I never thought 1
Looking at a small 3
Despite 1
• Each person
proposes 1 phrase
• Each person votes
(up to 2 votes) for
‘best’ phrase
• Cannot vote
yourself!
• Count votes
• Write highest
scoring phrase to
poem
• If tied, repeat
voting only
between tied
words (or flip coin)
POEM
I really had an easy way of walking
Looking at a small
37. Image: Man leads caravan through desert
The camel holds the hand of the poor man
He’ll watch us tread, he’ll watch us fall
The long shadow of a donkey across the cracked sand
Shabbily clad but standing tall
We each of us must go, all
Surrounded by another expedition,
Across an ocean on the sea of sand,
Mountains gaze upon a vast Egyptian,
A field of sand beneath the silver strand.
38. Image: Man free falls to ground
We gotta hide behind the Beaver lake!
I wanna know a better place or where,
Another day a little kiss and take,
An angel on the other side of there.
The forest and cliffs standing against the sky
A human being in free fall
Sailing through the air like a fly
Come to me, the grass, a call.
39. Image: Crowd watches fishes at aquarium
In a box of Nothing
far from the deep
where waves are thrusting
where light goes to sleep
Expecting something from an empty zoo,
Surrounded by an ocean full of fish,
On the other side of me and you,
Beneath the carpet like a jellyfish.
40. Image: Two men round a campfire
A living fire becomes a doubles title.
To stay protected by the sons of men,
We stuck together like a semi final,
The one and two and three or four of ten.
Marshmallows at dawn
Freshly cut wood burns in the fire
Time slips out a wide yawn
They all sing out in choir
41. Acknowledgements
• Zena Edwards, CV:iD
• Daniela Paolucci, Apples and Snakes
• Sebastian Riedel, UCL
• Piotr Mirowski, HumanMachine/Deepmind
• Mandana Seyfeddinipur, SOAS
• Marjan Ghazvininejad, USC
• Generating Topical Poetry, EMNLP 2016
Toe upon toe, a snowing flesh, A gold of lemon, root and rind, She sifts in sunlight down the stairs With nothing on. Nor on her mind.
We spy beneath the banister A constant thresh of thigh on thigh-- Her lips imprint the swinging air That parts to let her parts go by.
One-woman waterfall, she wears Her slow descent like a long cape And pausing, on the final stair Collects her motions into shape.