Multilingual Zero-Shot Transfer in Low-Resource Settings

Speaker:
Gabriel Stanovsky (Hebrew University of Jerusalem)
Abstract:
I will present two recent works centered around multilingual zero-shot transfer, which occurs when models can solve instances without direct supervision in their target language. First, I will present a model capable of filling in eroded parts in ancient cuneiform tablets written thousands of years ago in Akkadian. We find that zero-shot models do better than monolingual models given the limited training data available for this task, and show their effectiveness in automatic and human evaluations. Motivated by these findings, I will present an experiment of zero-shot performance under balanced data conditions which mitigate corpus size confounds. We show that the choice of pretraining languages vastly affects downstream cross-lingual transfer for BERT-based models, and develop a method of quadratic time complexity in the number of pretraining languages to estimate these inter-language relations. Our findings can inform pretraining configurations in future large-scale multilingual language models. This work was recently awarded an outstanding paper award at NAACL 2022.
Length:
00:41:15
Date:
10/10/2022
views: 310

Images:
Attachments: (video, slides, etc.)
79.0 MB
311 downloads