Hugging Face Reads, Feb. 2021 - Long-range Transformers

Back to Articles Hugging Face Reads, Feb. 2021 - Long-range Transformers Published March 9, 2021 Update on GitHub Upvote 3 Victor Sanh VictorSanh Follow Efficient Transformers taxonomy from Efficient Transformers: a Survey by Tay et al. Introduction Summaries Longformer - The Long-Document...

Back to Articles Hugging Face Reads, Feb. 2021 - Long-range Transformers Published March 9, 2021 Update on GitHub Upvote 3 Victor Sanh VictorSanh Follow Efficient Transformers taxonomy from Efficient Transformers: a Survey by Tay et al. Introduction Summaries Longformer - The Long-Document Transformer Compressive Transformers for Long-Range Sequence Modelling Linformer: Self-Attention with Linear Complexity Rethinking Attention with Performers Reading group discussion @Hugging Face 🤗: Long-range modeling Co-written by Teven Le Scao, Patrick Von Platen, Suraj Patil, Yacine Jernite and Victor Sanh. Each month, we will choose a topic to focus on, reading a set of four papers recently published on the subject. We will then write a short blog post summarizing their findings and the common trends between them, and questions we had for follow-up work after reading them. The first topic for January 2021 was Sparsity and Pruning, in February 2021 we addressed Long-Range Attention in Transformers. Introduction After the rise of large transformer models in 2018 and 2019, two trends have quickly emerged to bring their compute requirements down. First, conditional computation, quantization, distillation, and pruning have unlocked inference of large models in compute-constrained environments; we’ve already touched upon this in part in our last reading group post. The research community then moved to reduce the cost of pre-training. In particular, one issue has been at the center of the efforts: the quadratic cost in memory and time of transformer models with regard to the sequence length. In order to allow efficient training of very large models, 2020 saw an onslaught of papers to address that bottleneck and scale transformers beyond the usual 512- or 1024- sequence lengths that were the default in NLP at the start of the year. This topic has been a key part of our research discussions from the start, and our own Patrick Von Platen has already dedicated a 4-part series to Reformer. In this reading group, rather than trying to cover every approach (there are so many!), we’ll focus on four main ideas: Custom attention patterns (with Longformer) Recurrence (with Compressive Transformer) Low-rank approximations (with Linformer) Kernel approximations (with Performer) For exhaustive views of the subject, check out Efficient Transfomers: A Survey and Long Range Arena. Summaries Longformer - The Long-Document Transformer Iz Beltagy, Matthew E. Peters, Arman Cohan Longformer addresses the memory bottleneck of transformers by replacing conventional self-attention with a combination of windowed/local/sparse (cf. Sparse Transformers (2019)) attention and global attention that scales linearly with the sequence length. As opposed to previous long-range transformer models (e.g. Transformer-XL (2019), Reformer (2020), Adaptive Attention Span (2019)), Longformer’s self-attention layer is designed as a drop-in replacement for the standard self-attention, thus making it possible to leverage pre-trained checkpoints for further pre-training and/or fine-tuning on long sequence tasks. The standard self-attention matrix (Figure a) scales quadratically with the input length: Figure taken from Longformer Longformer uses different attention patterns for autoregressive language modeling, encoder pre-training & fine-tuning, and sequence-to-sequence tasks. For autoregressive language modeling, the strongest results are obtained by replacing causal self-attention (a la GPT2) with dilated windowed self-attention (Figure c). With nnn being the sequence length and www being the window length, this attention pattern reduces the memory consumption from n2n^2n2 to wnwnwn, which under the assumption that w

Source: Hugging Face — Published March 9, 2021 at 00:00 UTC — Category: Models