Deep Multi-Task Learning with Shared Memory

•

2 gefällt mir•1,623 views

marujirou

EMNLP reading group @TMU

Technologie

Deep Multi-Task Learning with
Shared Memory
Pengfei Liu, Xipeng Qiu, Xuanjing Huang
EMNLP2016 reading group
presenter: ryosuke miyazaki

Abstract
Due to the large number of parameters neural models need a large-scale
corpus.
→ unsupervised pre-training is effective
Multi-task learning also improve the final performance.

This paper propose LSTM with external memory for multi-task learning.

Model: ME-LSTM
Key vector, Erase vector, Add vector

Model: ME-LSTM
Reading operation
K segment, M dimensions per one segment
,

Training
Task-specific output layer
Linear combination of cost function
λm is the weights for each task m

Analysis: Visualize deep fusion gate
Sentiment score
Dimensions of
deep fusion gate gt
Activate → black

Conclusion
・ This paper propose two deep architectures for multi-task
learning.
・ They design an external memory to store the knowledge
by related tasks.
・ Deep fusion strategy enabling the model to give shared
information.

Empfohlen

Exploring Strategies for Training Deep Neural Networks paper reviewVimukthi Wickramasinghe

Using Multi-layered Feed-forward Neural Network (MLFNN) Architecture as Bidir...IOSR Journals

Long Zhou - 2017 - Neural System Combination for Machine TransaltionAssociation for Computational Linguistics

Neural netorksmatchingMasa Kato

18IMPULSE_TECHNOLOGY

Switch Transformers: Scaling to Trillion Parameter Models with Simple and Eff...taeseon ryu

Multi-task Learning for Dense Prediction Tasks in Computer VisionArun Talkad

[딥논읽] Meta-Transfer Learning for Zero-Shot Super-Resolution paper reviewtaeseon ryu

Empfohlen

Exploring Strategies for Training Deep Neural Networks paper reviewVimukthi Wickramasinghe

Using Multi-layered Feed-forward Neural Network (MLFNN) Architecture as Bidir...IOSR Journals

Long Zhou - 2017 - Neural System Combination for Machine TransaltionAssociation for Computational Linguistics

Neural netorksmatchingMasa Kato

18IMPULSE_TECHNOLOGY

Switch Transformers: Scaling to Trillion Parameter Models with Simple and Eff...taeseon ryu

Multi-task Learning for Dense Prediction Tasks in Computer VisionArun Talkad

[딥논읽] Meta-Transfer Learning for Zero-Shot Super-Resolution paper reviewtaeseon ryu

On the cross domain reusability of neural modules for general video game playingAlexander Braylan

Attention Is All You NeedSEMINARGROOT

Introduction to CNNShuai Zhang

Mlp mixer image_process_210613 deeplearning paper review!taeseon ryu

Deep Belief NetworksHasan H Topcu

Restricting the Flow: Information Bottlenecks for Attributiontaeseon ryu

Data Applied:ForecastDataminingTools Inc

N ns 1Thy Selaroth

Reading group nfm - 20170312Shuai Zhang

Digit recognition using mnist databasebtandale

Introduction to Machine learning & Neural NetworksAnkur Nair

doc1.docxRamyadevi62

Convolutional neural networkItachi SK

AttentionSEMINARGROOT

Learning weighted lower linear envelope potentials in binary markov random fi...jpstudcorner

Classification by backpropacationSiva Priya

SummaRuNNer: A Recurrent Neural Network based Sequence Model for Extractive ...Shubhangi Tandon

Learning to compare: relation network for few shot learningSimon John

Icacci2017 poster templatevinaykumar R

1D Convolutional Neural Networks for Time Series Modeling - Nathan Janos, Jef...PyData

怖くない誤差逆伝播法 Chainerを添えてmarujirou

ニューラル日英翻訳における出力文の態制御Hayahide Yamagishi

Weitere ähnliche Inhalte

Was ist angesagt?

On the cross domain reusability of neural modules for general video game playingAlexander Braylan

Attention Is All You NeedSEMINARGROOT

Introduction to CNNShuai Zhang

Mlp mixer image_process_210613 deeplearning paper review!taeseon ryu

Deep Belief NetworksHasan H Topcu

Restricting the Flow: Information Bottlenecks for Attributiontaeseon ryu

Data Applied:ForecastDataminingTools Inc

N ns 1Thy Selaroth

Reading group nfm - 20170312Shuai Zhang

Digit recognition using mnist databasebtandale

Introduction to Machine learning & Neural NetworksAnkur Nair

doc1.docxRamyadevi62

Convolutional neural networkItachi SK

AttentionSEMINARGROOT

Learning weighted lower linear envelope potentials in binary markov random fi...jpstudcorner

Classification by backpropacationSiva Priya

SummaRuNNer: A Recurrent Neural Network based Sequence Model for Extractive ...Shubhangi Tandon

Learning to compare: relation network for few shot learningSimon John

Icacci2017 poster templatevinaykumar R

1D Convolutional Neural Networks for Time Series Modeling - Nathan Janos, Jef...PyData

Was ist angesagt? (20)

On the cross domain reusability of neural modules for general video game playing

Attention Is All You Need

Introduction to CNN

Mlp mixer image_process_210613 deeplearning paper review!

Deep Belief Networks

Restricting the Flow: Information Bottlenecks for Attribution

Data Applied:Forecast

N ns 1

Reading group nfm - 20170312

Digit recognition using mnist database

Introduction to Machine learning & Neural Networks

doc1.docx

Convolutional neural network

Attention

Learning weighted lower linear envelope potentials in binary markov random fi...

Classification by backpropacation

SummaRuNNer: A Recurrent Neural Network based Sequence Model for Extractive ...

Learning to compare: relation network for few shot learning

Icacci2017 poster template

1D Convolutional Neural Networks for Time Series Modeling - Nathan Janos, Jef...

Andere mochten auch

怖くない誤差逆伝播法 Chainerを添えてmarujirou

ニューラル日英翻訳における出力文の態制御Hayahide Yamagishi

GoogLeNet InsightsAuro Tripathy

ディープボルツマンマシン入門〜後半〜sakaizawa

tmu_science_cafe02Tomoyuki Kajiwara

ディープボルツマンマシン入門Saya Katafuchi

ECS for Amazon Deep Learning and Amazon Machine LearningAmanda Mackay (she/her)

[ACL2016] Achieving Open Vocabulary Neural Machine Translation with Hybrid Wo...Hayahide Yamagishi

[EMNLP2016読み会] Memory-enhanced Decoder for Neural Machine TranslationHayahide Yamagishi

Chainerチュートリアル -v1.5向け- ViEW2015Ryosuke Okuta

Andere mochten auch (10)

怖くない誤差逆伝播法 Chainerを添えて

ニューラル日英翻訳における出力文の態制御

GoogLeNet Insights

ディープボルツマンマシン入門〜後半〜

tmu_science_cafe02

ディープボルツマンマシン入門

ECS for Amazon Deep Learning and Amazon Machine Learning

[ACL2016] Achieving Open Vocabulary Neural Machine Translation with Hybrid Wo...

[EMNLP2016読み会] Memory-enhanced Decoder for Neural Machine Translation

Chainerチュートリアル -v1.5向け- ViEW2015

Ähnlich wie Deep Multi-Task Learning with Shared Memory

Multi tasking learningShreyusPuthiyapurail

Local Applications of Large Language Models based on RAG.pptxlwz614595250

Transfer Learning in NLP: A SurveyNUPUR YADAV

DLD meetup 2017, Efficient Deep LearningBrodmann17

Machine learning in science and industry — day 4arogozhnikov

Knowledge distillation deeplabFrozen Paradise

Feature Extraction and Analysis of Natural Language Processing for Deep Learn...Sharmila Sathish

Nn kbKushal Arora

ODSC East: Effective Transfer Learning for NLPindico data

Performance Comparison between Pytorch and Mindsporeijdms

Neural Network in Knowledge BasesKushal Arora

SENTIMENT ANALYSIS IN MYANMAR LANGUAGE USING CONVOLUTIONAL LSTM NEURAL NETWORKijnlc

Sentiment Analysis In Myanmar Language Using Convolutional Lstm Neural Networkkevig

Nips 2017 in a nutshellLULU CHENG

Multi Task Learning and Meta LearningSrilalitha Veerubhotla

Life-long / Incremental Learning (DLAI D6L1 2017 UPC Deep Learning for Artifi...Universitat Politècnica de Catalunya

Lifelong / Incremental Deep Learning - Ramon Morros - UPC Barcelona 2018Universitat Politècnica de Catalunya

Lifelong Learning for Dynamically Expandable NetworksNAVER Engineering

Implementing a neural network potential for exascale molecular dynamicsPFHub PFHub

DL for sentence classification project Write-upHoàng Triều Trịnh

Ähnlich wie Deep Multi-Task Learning with Shared Memory (20)

Multi tasking learning

Local Applications of Large Language Models based on RAG.pptx

Transfer Learning in NLP: A Survey

DLD meetup 2017, Efficient Deep Learning

Machine learning in science and industry — day 4

Knowledge distillation deeplab

Feature Extraction and Analysis of Natural Language Processing for Deep Learn...

Nn kb

ODSC East: Effective Transfer Learning for NLP

Performance Comparison between Pytorch and Mindspore

Neural Network in Knowledge Bases

SENTIMENT ANALYSIS IN MYANMAR LANGUAGE USING CONVOLUTIONAL LSTM NEURAL NETWORK

Sentiment Analysis In Myanmar Language Using Convolutional Lstm Neural Network

Nips 2017 in a nutshell

Multi Task Learning and Meta Learning

Life-long / Incremental Learning (DLAI D6L1 2017 UPC Deep Learning for Artifi...

Lifelong / Incremental Deep Learning - Ramon Morros - UPC Barcelona 2018

Lifelong Learning for Dynamically Expandable Networks

Implementing a neural network potential for exascale molecular dynamics

DL for sentence classification project Write-up

Mehr von marujirou

Cross-Lingual Sentiment Analysis using modified BRAEmarujirou

Learning Tag Embeddings and Tag-specific Composition Functions in Recursive N...marujirou

2015 08 surveymarujirou

Representation Learning Using Multi-Task Deep Neural Networks for Semantic Cl...marujirou

Combining Distant and Partial Supervision for Relation Extraction (Angeli et ...marujirou

Semantic Compositionality through Recursive Matrix-Vector Spaces (Socher et al.)marujirou

Relation Classification via Convolutional Deep Neural Network (Zeng et al.)marujirou

DL勉強会 01ディープボルツマンマシンmarujirou

Mehr von marujirou (8)

Cross-Lingual Sentiment Analysis using modified BRAE

Learning Tag Embeddings and Tag-specific Composition Functions in Recursive N...

2015 08 survey

Representation Learning Using Multi-Task Deep Neural Networks for Semantic Cl...

Combining Distant and Partial Supervision for Relation Extraction (Angeli et ...

Semantic Compositionality through Recursive Matrix-Vector Spaces (Socher et al.)

Relation Classification via Convolutional Deep Neural Network (Zeng et al.)

DL勉強会 01ディープボルツマンマシン

Kürzlich hochgeladen

Sample pptx for embedding into website for demoHarshalMandlekar2

Arizona Broadband Policy Past, Present, and Future Presentation 3/25/24Mark Goldstein

Potential of AI (Generative AI) in Business: Learnings and InsightsRavi Sanghani

The State of Passkeys with FIDO Alliance.pptxLoriGlavin3

How to write a Business Continuity PlanDatabarracks

Why device, WIFI, and ISP insights are crucial to supporting remote Microsoft...panagenda

What is DBT - The Ultimate Data Build Tool.pdfMounikaPolabathina

Passkey Providers and Enabling Portability: FIDO Paris Seminar.pptxLoriGlavin3

DevEX - reference for building teams, processes, and platformsSergiu Bodiu

Assure Ecommerce and Retail Operations Uptime with ThousandEyesThousandEyes

Unleashing Real-time Insights with ClickHouse_ Navigating the Landscape in 20...Alkin Tezuysal

The Future Roadmap for the Composable Data Stack - Wes McKinney - Data Counci...Wes McKinney

The Fit for Passkeys for Employee and Consumer Sign-ins: FIDO Paris Seminar.pptxLoriGlavin3

Use of FIDO in the Payments and Identity Landscape: FIDO Paris Seminar.pptxLoriGlavin3

The Role of FIDO in a Cyber Secure Netherlands: FIDO Paris Seminar.pptxLoriGlavin3

Generative Artificial Intelligence: How generative AI works.pdfIngrid Airi González

So einfach geht modernes Roaming fuer Notes und Nomad.pdfpanagenda

Testing tools and AI - ideas what to try with some tool examplesKari Kakkonen

A Journey Into the Emotions of Software DevelopersNicole Novielli

Digital Identity is Under Attack: FIDO Paris Seminar.pptxLoriGlavin3

Kürzlich hochgeladen (20)

Sample pptx for embedding into website for demo

Arizona Broadband Policy Past, Present, and Future Presentation 3/25/24

Potential of AI (Generative AI) in Business: Learnings and Insights

The State of Passkeys with FIDO Alliance.pptx

How to write a Business Continuity Plan

Why device, WIFI, and ISP insights are crucial to supporting remote Microsoft...

What is DBT - The Ultimate Data Build Tool.pdf

Passkey Providers and Enabling Portability: FIDO Paris Seminar.pptx

DevEX - reference for building teams, processes, and platforms

Assure Ecommerce and Retail Operations Uptime with ThousandEyes

Unleashing Real-time Insights with ClickHouse_ Navigating the Landscape in 20...

The Future Roadmap for the Composable Data Stack - Wes McKinney - Data Counci...

The Fit for Passkeys for Employee and Consumer Sign-ins: FIDO Paris Seminar.pptx

Use of FIDO in the Payments and Identity Landscape: FIDO Paris Seminar.pptx

The Role of FIDO in a Cyber Secure Netherlands: FIDO Paris Seminar.pptx

Generative Artificial Intelligence: How generative AI works.pdf

So einfach geht modernes Roaming fuer Notes und Nomad.pdf

Testing tools and AI - ideas what to try with some tool examples

A Journey Into the Emotions of Software Developers

Digital Identity is Under Attack: FIDO Paris Seminar.pptx

Deep Multi-Task Learning with Shared Memory

1. Deep Multi-Task Learning with Shared Memory Pengfei Liu, Xipeng Qiu, Xuanjing Huang EMNLP2016 reading group presenter: ryosuke miyazaki

2. Abstract Due to the large number of parameters neural models need a large-scale corpus. → unsupervised pre-training is effective Multi-task learning also improve the final performance. This paper propose LSTM with external memory for multi-task learning.

3. Model: ME-LSTM Key vector, Erase vector, Add vector

4. Model: ME-LSTM Reading operation K segment, M dimensions per one segment ,

5. Model: ME-LSTM Deep Fusion strategy

6. Model: ME-LSTM Writing operation

7. Two architectures ARC-1 ARC-2

8. Training Task-specific output layer Linear combination of cost function λm is the weights for each task m

9. Experiment: text classification

10. Result: Movie

11. Result: Product

12. Analysis: Visualize deep fusion gate Sentiment score Dimensions of deep fusion gate gt Activate → black

13. Analysis: Visualize deep fusion gate

14. Conclusion ・ This paper propose two deep architectures for multi-task learning. ・ They design an external memory to store the knowledge by related tasks. ・ Deep fusion strategy enabling the model to give shared information.