Diese Präsentation wurde erfolgreich gemeldet.
Wir verwenden Ihre LinkedIn Profilangaben und Informationen zu Ihren Aktivitäten, um Anzeigen zu personalisieren und Ihnen relevantere Inhalte anzuzeigen. Sie können Ihre Anzeigeneinstellungen jederzeit ändern.
2014/11/22 EMNLP2014ㄞ䜏఍@PFI 
A Fast and Accurate Dependency 
Parser using Neural Networks 
Danqi Chen and Christopher Mann...
⮬ᕫ⤂௓䠖ᚨỌᣅஅ 
● twitter: @tkng 
● Preferred Infrastructure໅ົ䠄䛣䛾఍ሙ䠅 
● 䜸䞁䝷䜲䞁ᶵᲔᏛ⩦䛸䛔䛖ᮏ䛜ฟ䜎䛩 
○ ᾏ㔝, ᒸ㔝ཎ, ᚓᒃ, ᚨỌ䛾4ே䛷ᇳ➹୰
䜒䛧CRF䛜䜟䛛䜙䛺䛔ே䛿… 
CRF, viterbi, forward-backward䛺䛹㍕䛳䛶䜎䛩 
㟁Ꮚ᭩⡠∧䛜ฟ䜎䛩
A Fast and Accurate Dependency 
Parser using Neural Networks 
● ಀ䜚ཷ䛡ゎᯒ䛾ヰ 
● Transition based parser䛾ศ㢮ჾ䜢䝙䝳䞊䝷䝹 
䝛䝑䝖䛻䛧䛯䜙ᛶ⬟䛜3...
Transition based vs Graph based 
● Transition based: ㏿䛔䛜ᛶ⬟䛜䛱䜗䛳䛸ᝏ䛔 
● Graph based: 㐜䛔䛜ᛶ⬟䛜䛱䜗䛳䛸䛔䛔
Transition based parsing䛾䜔䜚᪉ 
● 䝇䝍䝑䜽䛻༢ㄒ䜢✚䜏䚸ศ㢮ჾ䛷ḟ䛻䛹䛖䛩䜛䛛 
䜢Ỵ䜑䜛 
○ Shift: 䝞䝑䝣䜯䛛䜙䝇䝍䝑䜽䛻༢ㄒ䜢䜂䛸䛴✚䜐 
○ Arc: 䝇䝍䝑䜽䛛䜙༢ㄒ䜢ྲྀ䜚ฟ䛧䛶Arc䜢ᙇ䜛 
■...
᪤Ꮡ䛾Transition based parser䛾ㄢ㢟 
● ⣲ᛶ䛜䝇䝟䞊䝇䛷䛒䜛 
○ ఝ䛯༢ㄒ䚸ఝ䛯ရモ䛜䜎䛳䛯䛟㐪䛖⾲⌧䛻䛺䜛 
● ᡭసᴗ䛷స䛳䛯⣲ᛶ䛿୙᏶඲䛷䛒䜛 
○ 䜶䜻䝇䝟䞊䝖䛜స䛳䛯⣲ᛶ䛻䜒ぢⴠ䛸䛧䛜䛒䜛 
● ィ⟬䝁䝇䝖...
ᮏㄽᩥ䛾ᡭἲ 
● ศ㢮ჾ䜢NN䛷⨨䛝᥮䛘䜛 
○ 䛣䜜䜎䛷䛿SVM䛸䛛䜢౑䛖䛾䛜䝯䝆䝱䞊 
● ๓䝨䞊䝆䛾ၥ㢟䜢ゎỴ䛩䜛䛯䜑䛻䛔䛟䛴䛛ᕤኵ 
䛜䛒䜛
ศ㢮⏝NN䛾ᵓᡂ 
● 3ᒙ䛾䝙䝳䞊䝷䝹䝛䝑䝖 
○ 1ᒙ䠖༢ㄒ䛾ศᩓ⾲⌧ 
○ 2ᒙ䠖hidden layer 
○ 3ᒙ䠖softmax 
● Ꮫ⩦䛿back propagation, Ꮫ⩦⋡䛿AdaGrad 
䛷ㄪᩚ 
○ ➨1ᒙ䛿䛒䜙䛛...
䝫䜲䞁䝖 
● ධຊ䛿ᩥ䛷䛿䛺䛟ᩥ⬦䛺䛾䛷ྍኚ㛗䛾ධຊ䜢 
ᢅ䛖ᚲせ䛿䛺䛔 
● ༢ㄒ䛰䛡䛷䛿䛺䛟䚸ရモ䚸౫Ꮡ䝷䝧䝹䜒dḟඖ䛾 
పḟඖᐦ䝧䜽䝖䝹䛻ኚ᥮䛩䜛 
○ ఝ䛶䜛ရモ䛸䛛䛾᝟ሗ䜢౑䛘䜛
䝫䜲䞁䝖2: cube activation function 
● 㞃䜜ᒙ䛾activation function䛻cube activation 
function䜢౑䛖 
● cube: f(x) = x^3 
● ㄝ᫂䜢ㄞ䜣䛷䜒Ⓨ᝿䛜⌮...
䝫䜲䞁䝖3: 䜻䝱䝑䝅䝳 
● 䛒䜛༢ㄒ䛜䝁䞁䝔䜻䝇䝖䛻ฟ⌧䛧䛯ሙྜ䛻䚸➨2 
ᒙ䠄㞃䜜ᒙ䠅䛻ᑐ䛩䜛ධຊ䛜䛹䛖䛺䜛䛛䛿஦๓ 
䛻ィ⟬䛷䛝䜛 
○ ༢ㄒ䛾ᩘ䛿ୖ㝈䛜䛒䜛䛾䛷䜻䝱䝑䝅䝳ྍ⬟ 
○ 8ࠥ10ಸ䛠䜙䛔䛾㧗㏿໬ 
● ACL2014䛾䝧䝇...
ᐇ㦂⤖ᯝ䠖௚ᡭἲ䛸䛾ẚ㍑
ᐇ㦂⤖ᯝ䠖cube activation 
0.8ࠥ1.2%䛠䜙䛔ᛶ⬟䛜ྥୖ䛧䛶䛔䜛
ᐇ㦂⤖ᯝ䠖word2vec vs random 
➨1ᒙ䜢word2vec䛷ึᮇ໬䛩䜛䛸0.7ࠥ1.7%䛾ᛶ 
⬟ྥୖ
ᐇ㦂⤖ᯝ䠖POS embedding 
CTB䛰䛸10%䛠䜙䛔ᛶ⬟ྥୖ
POS embedding䛾どぬ໬
䜎䛸䜑 
● Shift-reduceᆺ䛾ಀ䜚ཷ䛡ゎᯒჾ䛷ศ㢮ჾ䜢NN䛻䛩䜛䛸ᛶ 
⬟䛜ୖ䛜䜛 
● MST䝟䞊䝃䞊䛸ྠ䛨䛠䜙䛔⢭ᗘ䛜䜘䛟䛶䚸㏿ᗘ䛿100ಸ䛠䜙 
䛔㏿䛔 
● cube activation䛿⡆༢䛰䛧䚸1%䛸䛛ᛶ⬟ୖ䛜䜛䜏䛯䛔...
䝛䝑䝖ୖ䛛䜙䛾ឤ᝿ 
● 䜻䝱䝑䝅䝳䛺䛧䛰䛸5ಸ䛠䜙䛔㐜䛟䛺䛳䛶䛔䜛䛾 
䛷䚸㏿䛟䛺䛳䛯䛸䛔䛖䛾䛿䜰䞁䝣䜵䜰 
○ http://nlpers.blogspot.jp/2014/11/emnlp-2015-paper-list-with-min...
ឤ᝿ 
● ➨1ᒙ䜢word2vec䛷ึᮇ໬䛩䜛䛾䛿䚸ゝ䜟䜜 
䛶䜏䜜䜀ᙜ䛯䜚๓䛰䛜䚸௚䛾ㄽᩥ䛷ぢ䛯䛣䛸䛜 
䛺䛛䛳䛯䛾䛷ឤᚰ䛧䛯 
○ ༢䛻pretrain䛰䜘䛽䛸䛔䛖䛸䜎䛑䛭䛖䛺䜣䛰䛡䛹 
● ⤖ᒁ䚸Deep Learning䛷䛿䛺䛔 
●...
Nächste SlideShare
Wird geladen in …5
×

EMNLP2014読み会 徳永

EMNLP 2014読み会@PFIでの発表資料

Ähnliche Bücher

Kostenlos mit einer 30-tägigen Testversion von Scribd

Alle anzeigen

Ähnliche Hörbücher

Kostenlos mit einer 30-tägigen Testversion von Scribd

Alle anzeigen
  • Loggen Sie sich ein, um Kommentare anzuzeigen.

EMNLP2014読み会 徳永

  1. 1. 2014/11/22 EMNLP2014ㄞ䜏఍@PFI A Fast and Accurate Dependency Parser using Neural Networks Danqi Chen and Christopher Manning ㄞ䜐ே䠖ᚨỌᣅஅ
  2. 2. ⮬ᕫ⤂௓䠖ᚨỌᣅஅ ● twitter: @tkng ● Preferred Infrastructure໅ົ䠄䛣䛾఍ሙ䠅 ● 䜸䞁䝷䜲䞁ᶵᲔᏛ⩦䛸䛔䛖ᮏ䛜ฟ䜎䛩 ○ ᾏ㔝, ᒸ㔝ཎ, ᚓᒃ, ᚨỌ䛾4ே䛷ᇳ➹୰
  3. 3. 䜒䛧CRF䛜䜟䛛䜙䛺䛔ே䛿… CRF, viterbi, forward-backward䛺䛹㍕䛳䛶䜎䛩 㟁Ꮚ᭩⡠∧䛜ฟ䜎䛩
  4. 4. A Fast and Accurate Dependency Parser using Neural Networks ● ಀ䜚ཷ䛡ゎᯒ䛾ヰ ● Transition based parser䛾ศ㢮ჾ䜢䝙䝳䞊䝷䝹 䝛䝑䝖䛻䛧䛯䜙ᛶ⬟䛜3%䛠䜙䛔ୖ䛜䛳䛯
  5. 5. Transition based vs Graph based ● Transition based: ㏿䛔䛜ᛶ⬟䛜䛱䜗䛳䛸ᝏ䛔 ● Graph based: 㐜䛔䛜ᛶ⬟䛜䛱䜗䛳䛸䛔䛔
  6. 6. Transition based parsing䛾䜔䜚᪉ ● 䝇䝍䝑䜽䛻༢ㄒ䜢✚䜏䚸ศ㢮ჾ䛷ḟ䛻䛹䛖䛩䜛䛛 䜢Ỵ䜑䜛 ○ Shift: 䝞䝑䝣䜯䛛䜙䝇䝍䝑䜽䛻༢ㄒ䜢䜂䛸䛴✚䜐 ○ Arc: 䝇䝍䝑䜽䛛䜙༢ㄒ䜢ྲྀ䜚ฟ䛧䛶Arc䜢ᙇ䜛 ■ Left-Arc, Right-Arc䛾2✀㢮䛜䛒䜛 ● 䝇䝍䝑䜽䛾䛹䛣䛛䜙༢ㄒ䜢ྲྀ䜚ฟ䛩䛛䛜ኚ䜟䜛 ● ヲ䛧䛟䛿ⴭ⪅䛾䝇䝷䜲䝗䛷
  7. 7. ᪤Ꮡ䛾Transition based parser䛾ㄢ㢟 ● ⣲ᛶ䛜䝇䝟䞊䝇䛷䛒䜛 ○ ఝ䛯༢ㄒ䚸ఝ䛯ရモ䛜䜎䛳䛯䛟㐪䛖⾲⌧䛻䛺䜛 ● ᡭసᴗ䛷స䛳䛯⣲ᛶ䛿୙᏶඲䛷䛒䜛 ○ 䜶䜻䝇䝟䞊䝖䛜స䛳䛯⣲ᛶ䛻䜒ぢⴠ䛸䛧䛜䛒䜛 ● ィ⟬䝁䝇䝖䛜㧗䛔 ○ feature string䜢స䛳䛶䝝䝑䝅䝳䝔䞊䝤䝹䜢lookup䛧䛺䛔 䛸䛔䛡䛺䛔 ○ ṇ┤䚸䛣䛣䛿ᐇ⿦䛾ၥ㢟䛨䜓䛺䛔䛛䛸ᛮ䛖
  8. 8. ᮏㄽᩥ䛾ᡭἲ ● ศ㢮ჾ䜢NN䛷⨨䛝᥮䛘䜛 ○ 䛣䜜䜎䛷䛿SVM䛸䛛䜢౑䛖䛾䛜䝯䝆䝱䞊 ● ๓䝨䞊䝆䛾ၥ㢟䜢ゎỴ䛩䜛䛯䜑䛻䛔䛟䛴䛛ᕤኵ 䛜䛒䜛
  9. 9. ศ㢮⏝NN䛾ᵓᡂ ● 3ᒙ䛾䝙䝳䞊䝷䝹䝛䝑䝖 ○ 1ᒙ䠖༢ㄒ䛾ศᩓ⾲⌧ ○ 2ᒙ䠖hidden layer ○ 3ᒙ䠖softmax ● Ꮫ⩦䛿back propagation, Ꮫ⩦⋡䛿AdaGrad 䛷ㄪᩚ ○ ➨1ᒙ䛿䛒䜙䛛䛨䜑word2vec䛷Ꮫ⩦䛥䛫䛶䛚䛟
  10. 10. 䝫䜲䞁䝖 ● ධຊ䛿ᩥ䛷䛿䛺䛟ᩥ⬦䛺䛾䛷ྍኚ㛗䛾ධຊ䜢 ᢅ䛖ᚲせ䛿䛺䛔 ● ༢ㄒ䛰䛡䛷䛿䛺䛟䚸ရモ䚸౫Ꮡ䝷䝧䝹䜒dḟඖ䛾 పḟඖᐦ䝧䜽䝖䝹䛻ኚ᥮䛩䜛 ○ ఝ䛶䜛ရモ䛸䛛䛾᝟ሗ䜢౑䛘䜛
  11. 11. 䝫䜲䞁䝖2: cube activation function ● 㞃䜜ᒙ䛾activation function䛻cube activation function䜢౑䛖 ● cube: f(x) = x^3 ● ㄝ᫂䜢ㄞ䜣䛷䜒Ⓨ᝿䛜⌮ゎ䛷䛝䛺䛔… ○ 䛹䛳䛛䜙⪃䛘䛴䛔䛯䛾䛛ㅦ
  12. 12. 䝫䜲䞁䝖3: 䜻䝱䝑䝅䝳 ● 䛒䜛༢ㄒ䛜䝁䞁䝔䜻䝇䝖䛻ฟ⌧䛧䛯ሙྜ䛻䚸➨2 ᒙ䠄㞃䜜ᒙ䠅䛻ᑐ䛩䜛ධຊ䛜䛹䛖䛺䜛䛛䛿஦๓ 䛻ィ⟬䛷䛝䜛 ○ ༢ㄒ䛾ᩘ䛿ୖ㝈䛜䛒䜛䛾䛷䜻䝱䝑䝅䝳ྍ⬟ ○ 8ࠥ10ಸ䛠䜙䛔䛾㧗㏿໬ ● ACL2014䛾䝧䝇䝖䝨䞊䝟䞊䜒ྠᵝ䛾䝔䜽䝙䝑䜽 䜢⏝䛔䛶䛔䛯
  13. 13. ᐇ㦂⤖ᯝ䠖௚ᡭἲ䛸䛾ẚ㍑
  14. 14. ᐇ㦂⤖ᯝ䠖cube activation 0.8ࠥ1.2%䛠䜙䛔ᛶ⬟䛜ྥୖ䛧䛶䛔䜛
  15. 15. ᐇ㦂⤖ᯝ䠖word2vec vs random ➨1ᒙ䜢word2vec䛷ึᮇ໬䛩䜛䛸0.7ࠥ1.7%䛾ᛶ ⬟ྥୖ
  16. 16. ᐇ㦂⤖ᯝ䠖POS embedding CTB䛰䛸10%䛠䜙䛔ᛶ⬟ྥୖ
  17. 17. POS embedding䛾どぬ໬
  18. 18. 䜎䛸䜑 ● Shift-reduceᆺ䛾ಀ䜚ཷ䛡ゎᯒჾ䛷ศ㢮ჾ䜢NN䛻䛩䜛䛸ᛶ ⬟䛜ୖ䛜䜛 ● MST䝟䞊䝃䞊䛸ྠ䛨䛠䜙䛔⢭ᗘ䛜䜘䛟䛶䚸㏿ᗘ䛿100ಸ䛠䜙 䛔㏿䛔 ● cube activation䛿⡆༢䛰䛧䚸1%䛸䛛ᛶ⬟ୖ䛜䜛䜏䛯䛔䛺䛾 䛷䛚䛔䛧䛔䛛䜒 ● POS embedding䛿౑䛘䜛᫬䛜㝈䜙䜜䛭䛖䛰䛜䜘䛟ຠ䛟
  19. 19. 䝛䝑䝖ୖ䛛䜙䛾ឤ᝿ ● 䜻䝱䝑䝅䝳䛺䛧䛰䛸5ಸ䛠䜙䛔㐜䛟䛺䛳䛶䛔䜛䛾 䛷䚸㏿䛟䛺䛳䛯䛸䛔䛖䛾䛿䜰䞁䝣䜵䜰 ○ http://nlpers.blogspot.jp/2014/11/emnlp-2015-paper-list-with-mini-reviews. html ● ᐦ䝧䜽䝖䝹䛻䛩䜛䛣䛸䛷䚸௒ᚋ䛿GPU䛷䛾㧗㏿ ໬䛜ᮇᚅ䛷䛝䜛 ○ https://twitter.com/yoavgo/status/528625970752667648
  20. 20. ឤ᝿ ● ➨1ᒙ䜢word2vec䛷ึᮇ໬䛩䜛䛾䛿䚸ゝ䜟䜜 䛶䜏䜜䜀ᙜ䛯䜚๓䛰䛜䚸௚䛾ㄽᩥ䛷ぢ䛯䛣䛸䛜 䛺䛛䛳䛯䛾䛷ឤᚰ䛧䛯 ○ ༢䛻pretrain䛰䜘䛽䛸䛔䛖䛸䜎䛑䛭䛖䛺䜣䛰䛡䛹 ● ⤖ᒁ䚸Deep Learning䛷䛿䛺䛔 ● ᑠ㞴䛧䛔䛣䛸䜢䛧䛶䛺䛔䛾䛜ዲ༳㇟

×