Slides of Cambridge Geek Night talk about using convolutional neural networks to do optical character recognition, specifically in the Longan library.

1. Neural Networks for OCR

2. OCR: picture of text -> text And Bruno, conqueror of Carthage, strode up to me and said: "Devil take you, Edith!" "Finally, you scoundrel - are you going to confess your love for me?" I retorted. The German warrior stood stoically. He surveyed the landscape before him; grinned; spoke: "You are much better with an axe than Jane - I grant you that." (Killing, I admit, was my favourite pastime. Long before I enlisted in the Order of the Knights of Malta, I liked playing with knives. No-one objected.) "Overall, how would you rank/rate my performance in axing?" "Performance evaluations are meaningless!" (Quite true.) Radically changing the topic, I asked: "So, what are your thoughts on Empress Teresa?" "Unshareable; irrelevant; bitter." "Secret? Very unsurprising. We warrior/troubadours are quite reserved - nay - silent." (Xenophobia played a role too. You knew that. So did my friend, Zoe.)

3. OCR Step 1: Find letters

6. Rotated / squashed / skewed

7. In different fonts

9. Texture from paper

11. Find major lines, use heuristics, eg “vertical line on left, vertical line on right, horizontal line in the middle -> H”

12. Clustering algorithms

13. Etc...

14. How do humans do it? -> neural networks

16. Can be trained to categorize inputs

17. Real Neurons

18. Neuronal Connections

19. Firing neurons excite others Firing threshold

20. Firing neurons excite others Firing threshold

21. Firing neurons excite others Firing threshold

22. ...which in turn excite others Firing threshold

23. Inputs can be weighted Firing threshold 0.7 0.4

24. Neurons can suppress others Firing threshold 0.7 0.4 -0.5

25. And they can have a starting bias Firing threshold Bias 0.3

26. (So they're basically logic gates) 0.5 0.5 1 1 -1 Bias: 1 AND OR NOT

27. Smooth threshold Smaller effect Larger effect

29. Excitation of neurons in first layer set according to the input

30. “ Hidden” layer(s) in between

31. Final layer is output In Hid Out

33. One output neuron per possible letter

34. Train network to excite the output neuron that corresponds to the letter input A B C D E F G H I J K L M ...

36. Present network with a sample input and a desired output

37. Then adjust the input weightings of its neurons so that the output becomes more like the desired one

38. The algorithm for this is called back-propagation

39. Do this 100000 times: What's this letter? Is it an A? No, it's a Q. What's this letter? Is it a B? No, it's a P. What's this letter? Is it a W? Yes, it's a W. etc.

41. At least one layer between the input and the output, of similar size -> 1 million connections -> lots of CPU time

42. All neurons in the hidden layer(s) are given the same inputs and targets, so they gravitate towards behaving the same

44. Gives pretty much the same response to all inputs: “Meh, it could be any letter really”

46. Different parts of hidden layer notice different bits about image

47. Far fewer connections

49. Causes network to notice local features (eg horizontal lines) consistently -0.133 -0.133 -0.133 -0.133

51. Train network to output different patterns for each letter instead of just exciting one neuron per letter. NNs learn better if each output neuron is trained to be active roughly half of the time.

52. -> “Convolutional neural network” (LeCun '95)

55. Easy to install

56. Easy to use

60. Simple neural network implementation in Python: http://arctrix.com/nas/python/bpnn.py

61. Back propagation paper: http://citeseer.ist.psu.edu/viewdoc/download?doi=10.1.1.70.878&rep=rep1&type=pdf

62. Convolutional neural networks: http://yann.lecun.com/exdb/lenet/index.html

63. Exhaustive NN FAQ: ftp://ftp.sas.com/pub/neural/FAQ.html