1. Cognitive Learning and the Multimodal Memory Game: Toward Human-Level Machine Learning 2008 IEEE World Congress on Computational Intelligence (WCCI 2008) Cognitive Architectures: Towards Human-Level Intelligence Session June 5, 2008, Hong Kong Byoung-Tak Zhang Biointelligence Laboratory School of Computer Science and Engineering Cognitive Science, Brain Science, and Bioinformatics Programs Seoul National University Seoul 151-744, Korea [email_address] http://bi.snu.ac.kr/

3. Cognitive Learning

11. Research Platform for Cognitive Learning

12. © 2007, SNU Biointelligence Lab, http://bi.snu.ac.kr/ Image Sound Text But, I'm getting married tomorrow Well, maybe I am... I keep thinking about you. And I'm wondering if we made a mistake giving up so fast. Are you thinking about me? But if you are, call me tonight. Image-to-Text Generator Text-to-Image Generator Text Hint But, I'm getting married tomorrow Well, maybe I am... I keep thinking about you. And I'm wondering if we made a mistake giving up so fast. Are you thinking about me? But if you are, call me tonight. But, I'm getting married tomorrow Well, maybe I am... I keep thinking about you. And I'm wondering if we made a mistake giving up so fast. Are you thinking about me? But if you are, call me tonight. But, I'm getting married tomorrow Well, maybe I am... I keep thinking about you. And I'm wondering if we made a mistake giving up so fast. Are you thinking about me? But if you are, call me tonight. But, I'm getting married tomorrow Well, maybe I am... I keep thinking about you. And I'm wondering if we made a mistake giving up so fast. Are you thinking about me? But if you are, call me tonight. Hint Image Machine Learner Toward Human-Level Machine Learning: Multimodal Memory Game (MMG)

13. Text Generation Game (from Image) Image Sound © 2008, SNU Biointelligence Lab, http://bi.snu.ac.kr/ Text I2T Learning by Viewing T2I Game Manager Text Hint T

14. Image Generation Game (from Text) Text Image Sound © 2008, SNU Biointelligence Lab, http://bi.snu.ac.kr/ I2T Learning by Viewing T2I Game Manager Hint Image I

15. Some Experimental Results

20. Answer Query I don't know what happened There's a kitty in my guitar case Maybe there's something I can do to make sure I get pregnant Maybe there's something there's something I … I get pregnant There's a a kitty in … in my guitar case I don't know don't know what know what happened Matching & Completion Image-to-Text Translation Results © 2008, SNU Biointelligence Lab, http://bi.snu.ac.kr/

21. Query Matching & Completion I don't know what happened Take a look at this There's a kitty in my guitar case Maybe there's something I can do to make sure I get pregnant Answer Text-to-Image Translation Results © 2008, SNU Biointelligence Lab, http://bi.snu.ac.kr/

22. Hypernetwork Architecture for Cognitive Learning

23. © 2008, SNU Biointelligence Lab, http://bi.snu.ac.kr/ 4 Data Items 1 2 3 1 2 3 4 4 Round 1 Round 2 Round 3 Learning x8 x9 x12 x1 x2 x3 x4 x5 x6 x7 x10 x11 x13 x14 x15 x1 =1 x2 =0 x3 =0 x4 =1 x5 =0 x6 =0 x7 =0 x8 =0 x9 =0 x10 =1 x11 =0 x12 =1 x13 =0 x14 =0 x15 =0 y = 1 x1 =0 x2 =1 x3 =1 x4 =0 x5 =0 x6 =0 x7 =0 x8 =0 x9 =1 x10 =0 x11 =0 x12 =0 x13 =0 x14 =1 x15 =0 y = 0 x1 =0 x2 =0 x3 =1 x4 =0 x5 =0 x6 =1 x7 =0 x8 =1 x9 =0 x10 =0 x11 =0 x12 =0 x13 =1 x14 =0 x15 =0 y =1 x 4 x 10 y=1 x 1 x 4 x 12 y=1 x 1 x 10 x 12 y=1 x 4 x 3 x 9 y=0 x 2 x 3 x 14 y=0 x 2 x 9 x 14 y=0 x 3 x 6 x 8 y=1 x 3 x 6 x 13 y=1 x 3 x 8 x 13 y=1 x 6 x1 =0 x2 =0 x3 =0 x4 =0 x5 =0 x6 =0 x7 =0 x8 =1 x9 =0 x10 =0 x11 =1 x12 =0 x13 =0 x14 =0 x15 =1 y =1 x 11 x 15 y=0 x 8

24. Hypernetwork of DNA Molecules [Zhang, DNA-2006]

25. Hypernetwork as Chemical Associative Memory © 2008, SNU Biointelligence Lab, http://bi.snu.ac.kr/ [Zhang, DNA-2006] For more details: Zhang, B.-T., IEEE Computational Intelligence Magazine , August 2008 (in press)

26. Image to Text (Recall Rate) © 2008, SNU Biointelligence Lab, http://bi.snu.ac.kr/

27. Text to Image (Recall Rate) © 2008, SNU Biointelligence Lab, http://bi.snu.ac.kr/

28. Toward Human-Level Intelligence

29. From Mind to Molecules and Back © 2008, SNU Biointelligence Lab, http://bi.snu.ac.kr/ Mind Brain Cell Molecule ∞ memory 10 11 cells >10 3 molecules

30. Paradigms for Computational Intelligence © 2008, SNU Biointelligence Lab, http://bi.snu.ac.kr/ [Zhang, IEEE Comp. Intel. Mag. , August 2008] Symbolism Connectionism Dynamicism Hyperinter- actionism Metaphor symbol system neural system dynamical system biomolecular system Mechanism logical electrical mechanical chemical Description syntactic functional behavioral relational Representation localist distributed continuous collective Organization structural connectionist differential combinatorial Adaptation substitution tuning rate change self-assembly Processing sequential parallel dynamical massively parallel Structure procedure network equation hypergraph Mathematics logic, formal language linear algebra, statistics geometry, calculus graph theory, probabilistic logic Space/time formal spatial temporal spatiotemporal

33. Derivation of the Learning Rule

37. Multimodal Memory Game as a Platform for Cognitive Machine Learning © 2008, SNU Biointelligence Lab, http://bi.snu.ac.kr/ Image Sound Text I2T Learning by Viewing T2I

38. Illustrative Results © 2008, SNU Biointelligence Lab, http://bi.snu.ac.kr/ Query Completion Classification who are you ? are you who ? you who are ? what are you who are you who are you Friends Friends Friends you need to wear it ? need to wear it you ? to wear it you need ? wear it you need to ? it you need to wear ? I need to wear it you want to wear it you need to wear it you need to do it you need to wear a 24 24 24 House 24