Invariant Learning and Disentanglement for NLP
Kursbeschreibung
| Studiengang | Modulkürzel | Leistungs- bewertung |
|---|---|---|
| BA-2010 | AS-CL | 8 LP |
| Master | SS-CL, SS-TAC | 8 LP |
| Dozenten/-innen | Artem Sokolov |
| Veranstaltungsart |
|
| Sprache | English |
| Erster Termin | 10.10.2022 |
| Letzter Termin | 14.10.2022 |
| Zeit und Ort | täglich, 10:00-17:00, INF 326 / SR 27 |
| Commitment-Frist | tbd. |
Teilnahmevoraussetzungen
Knowledge of the following will be helpful: foundations of probability and statistics, machine learning, natural language processing and neural networks.
Leistungsnachweis
- Regular and active participation
- Paper presentation
- Implementation project in small groups (during the semester, TBD)
Inhalt
This module is an introduction to the emerging subfield of machine learning of invariance-based learning and disentanglement, which tries to uncover hidden true causal factors that govern natural data, to introduce better inductive biases for models or to increase robustness to the out-of-distribution inputs. Assuming a close approximation to the factors has been uncovered, a fine-grained control of the neural networks outputs becomes possible, which are also more robust to the presence of spurious input features. Most of the research has been done so far in computer vision, however, we will attempt applications to natural language processing.
Seminarplan
| Datum | Sitzung | Materialien |
| 10.10.2022 | Intro | slides |
| 11.10.2022 | VAE-based Disentanglement | beta-VAE (CAV, Lime) |
| 12.10.2022 | FactorVAE, TC-VAE, InfoVAE | |
| 13.10.2022 | Challenging common assumptions.., InfoGAN | |
| 14.10.2022 | Invariant Risk Minimization | IRM, When IRM works, The risks of IRM |
Literature
Zusätliche Literatur
- VAE in NLP:
- Bowman et al. 2015 (VAE+RNNs)
- Hu et al. 2018 (controlling text)
- Li et al., 2018 (GANs for dialog + imitation learning)
- Zhang et al., 2016 (CVAE+RNN for MT)
- Pagnoni et al., 2018 (CVAE+RNN for MT)
- Schulz et al., 2018 (CVAE+RNN for MT)
- Wang et al. 2021 (Transformers with denoising AE, not variational AE, for sentence embeddings)
- Fu et al. 2020 (bag-of-words as latents)
- Fang et al. 2021 (CVAE+Transformers for generation)
- Ok et al. 2022 (VAE+Transformers for LM; latent attentio)
- Park et al. 2021 (converting Transformers to VAE)
