Are Transformers Good Learners? Exploring the Limits of Transformer Training

Speaker:
Dušan Variš (ÚFAL MFF UK)
Abstract:
In recent years, research into deep neural networks lead to significant advancements in many fields, ranging from NLP, across computer vision to playing games like Chess and Go. Even though deep neural networks were originally inspired by biological neurons, there are still many differences between deep nets and their biological counterparts. In this talk, we focus on three potential weaknesses of the neural network training: generalization, catastrophic forgetting and knowledge composition. We demonstrate how deep neural networks struggle with these phenomena, even though they are crucial to learning in their biological counterparts. We also discuss current approaches that focus on solving these issues.
Length:
00:50:42
Date:
17/05/2021
views: 1550

Images:
Attachments: (video, slides, etc.)
75.5 MB
1551 downloads
1.4
488 downloads