This document discusses the building blocks of neural networks. It identifies 5 key building blocks: 1) dense connections, 2) embeddings for processing text, 3) tied weights for reducing parameters, 4) recurrence for handling sequences, and 5) convolutions for processing images. It provides examples of how each building block can be used, such as using embeddings to represent words, recurrence to handle sequential data like text, and convolutions to process image data. The document argues that these building blocks can be combined in different ways to solve a variety of problems, such as predicting the next word in a sequence, detecting objects in images, or generating image captions.

1. MAT KELCEY
Machine Learning Principal
@mat_kelcey #EvolutionTW #ThoughtWorks

2. THE BUILDING BLOCKS OF
NEURAL NETWORKS
Mat Kelcey @mat_kelcey

3. BUILDING BLOCK #1

4. PROBLEM: PREDICTING HOUSE PRICES
size (m2) price
190 ?

5. PROBLEM: PREDICTING HOUSE PRICES
size (m2) price
160 $650,000
140 $560,000
200 $820,000
170 $660,000
180 $630,000
210 $800,000
190 ?

6. PROBLEM: PREDICTING HOUSE PRICES

7. PROBLEM: PREDICTING HOUSE PRICES

8. PROBLEM: PREDICTING HOUSE PRICES

9. A SIMPLE MODEL
price = m x size

10. A SIMPLE MODEL
price = m x size
something we
know

11. A SIMPLE MODEL
price = m x size
something we
want to know

12. A SIMPLE MODEL
price = m x size
something we need to
estimate

13. A GRAPHICAL REPRESENTATION
size price
m
price = m x size

14. A GRAPHICAL REPRESENTATION
price
m1
m2
size
distance
to city
price = m1 x size + m2 x distance

15. A GRAPHICAL REPRESENTATION
house size
distance
block size
age
...
price

16. A GRAPHICAL REPRESENTATION
house size
distance
block size
age
...
price

17. A GRAPHICAL REPRESENTATION

18. A GRAPHICAL REPRESENTATION

19. A GRAPHICAL REPRESENTATION

20. BUILDING BLOCKS
1. dense connections

21. WHAT ABOUT DIFFERENT TYPES OF DATA
block size 62 m2
distance to city 11.0 km
block size 250 m2

22. WHAT ABOUT DIFFERENT TYPES OF DATA
block size 62 m2
distance to city 11.0 km
block size 250 m2
"To be, or not to be."

23. WHAT ABOUT DIFFERENT TYPES OF DATA
block size 62 m2
distance to city 11.0 km
block size 250 m2
"To be, or not to be."

24. WHAT ABOUT TEXT?
"To be, or not to be."

25. PROBLEM: PREDICTING THE NEXT WORD
input: output:
To be or ?

26. REPRESENTING TEXT
{be, not, or, to}
"To be, or not to be."

27. REPRESENTING TEXT
{be, not, or, to}
"To be, or not to be."
{b, e, n, o, r, t}

28. REPRESENTING A WORD
be not or to

29. REPRESENTING A WORD
1 0 0 0
be not or to

30. REPRESENTING A WORD
0 0 1 0
be not or to

31. REPRESENTING A WORD
1 0 0 0
be not or to
-0.3 0.1 0.4

32. REPRESENTING A WORD
0 0 1 0
be not or to
0.1 0.5 -0.9
1 0 0 0
be not or to
-0.3 0.1 0.4

33. BUILDING BLOCKS
1. dense connections
2. embeddings for processing text

34. PREDICTING THE NEXT WORD
0 0 0 1
be not or to
be not or to
0 1 0 0
1 0 0 0
be not or to
0 0 1 0
be not or to

35. TIED WEIGHTS
keep all these
weights the same!!

36. PARTS OF A NETWORK
input words
represented as 0s
& 1s

37. PARTS OF A NETWORK
input words
represented as 3
values

38. PARTS OF A NETWORK
a combination of the
three words that
represents the
phrase

39. PARTS OF A NETWORK
the expected the
next word

40. BUILDING BLOCKS
1. dense connections
2. embeddings for processing text
3. tied weights for reducing parameters

41. PREDICTING THE NEXT WORD: v0

42. PREDICTING THE NEXT WORD: v0

43. PREDICTING THE NEXT WORD: v1

44. PREDICTING THE NEXT WORD: v1

45. PREDICTING THE NEXT WORD: v1

46. PREDICTING THE NEXT WORD: v1

47. PREDICTING THE NEXT WORD: v1

48. PREDICTING THE NEXT WORD: v1

49. PREDICTING THE NEXT WORD: v1

50. PREDICTING THE NEXT WORD: v1
tied weights!

51. BUILDING BLOCKS
1. dense connections
2. embeddings for processing text
3. tied weights for reducing parameters
4. recurrence for handling sequences

52. BUT WHAT ABOUT IMAGES?

53. PROBLEM: "IS OUTDOOR" DETECTOR
input: output:
True

54. "IS OUTDOOR" DETECTOR

55. "IS OUTDOOR" DETECTOR
768 x 512 x 3 = 1,179,648 pixels
512
768
3

56. "IS OUTDOOR" DETECTOR

57. "IS OUTDOOR" DETECTOR
number of parameters
= 768 x 512 x 3
= 1,179,648

58. "IS OUTDOOR" DETECTOR
number of parameters
= 384 x 256 x 3
= 294,912

59. "IS OUTDOOR" DETECTOR
number of parameters
= 384 x 256 x 3
= 294,912

60. BUILDING BLOCKS
1. dense connections
2. embeddings for processing text
3. tied weights for reducing parameters
4. recurrence for handling sequences
5. convolutions for processing images

61. BUILDING WITH BLOCKS

62. PROBLEM: CAPTIONING IMAGES
Show and Tell: A Neural Image Caption Generator: Vinyals et al
A group of young
people playing a
game of frisbee.
output:input:

63. PROBLEM: CAPTIONING IMAGES

64. PROBLEM: CAPTIONING IMAGES
conv net!

65. PROBLEM: CAPTIONING IMAGES
embeddings!
embeddings!

66. PROBLEM: CAPTIONING IMAGES

67. PROBLEM: CAPTIONING IMAGES
recurrence!!

68. PROBLEM: CAPTIONING IMAGES

69. THANK YOU
For questions or suggestions:
Mat Kelcey
matkelcey@ThoughtWorks.com