Kristina Toutanova (Google) - Advances and Limitations in Generalization via Self-Supervised Pretrained Representations

Advances and Limitations in Generalization via Self-Supervised Pretrained Representations

Pretrained neural representations, learned from unlabeled text, have recently led to substantial improvements across many natural language problems. Yet some components of these models are still brittle and heuristic, and sizable human labeled data is typically needed to obtain competitive performance on end tasks.  

I will first talk about recent advances from our team leading to (i) improved multi-lingual generalization and ease of use through tokenization-free pretrained representations and (ii) better few-shot generalization for underrepresented task categories via neural language model-based example extrapolation. I will then point to limitations of generic pre-trained representations when tasked with handling both language variation and out-of-distribution compositional generalization, and the relative performance of induced symbolic representations.

This talk is part of the TTIC Colloquium and will be presented on Zoom, register here for details.

Host: Toyota Technology Institute at Chicago