Samira Abnar PhD Candidate University of Amsterdam

Distilling Inductive Biases

Distilling Inductive Biases feature image

Having the right inductive biases can be crucial in many tasks or scenarios where data or computing resources are a limiting factor, or where training data is not perfectly representative of the conditions at test time. However, defining, designing and efficiently adapting inductive biases is not necessarily straightforward. In this post, I discuss our paper, "Transferring Inductive Biases Through Knowledge Distillation", where we explore the power of knowledge distillation for transferring the effect of inductive biases from one model to another.


Visualizing Model Comparison

Visualizing Model Comparison feature image

In this post, I explain the representational analysis technique and how we can use it in combination with a multi dimensional scaling algorithm to visualize the similarity between multiple models, or different instances of the same model in 2d.


On the Merits of Recurrent Inductive Bias

On the Merits of Recurrent Inductive Bias feature image

In this post, we try to understand the nature of recurrent inductive bias. I will discuss different sources of inductive biases of RNNs and provide empirical results to demonstrate the benefits of each of them by comparing LSTMs to different variants of Transformers in the context of a task that having the recurrent inductive bias is shown to help achieve better generalization.


Quantifying Attention Flow in Transformers

Quantifying Attention Flow in Transformers feature image

In this post, I explain two techniques for visualising attention that address the problem of lack of token identifiability in higher layers of Transformers when using raw attention weights to interpret models' decisions. These techniques are called Attention Rollout and Attention Flow that are introduced in our paper "Quantifying Attention Flow In Transformers".


Blackbox Meets Blackbox

Blackbox Meets Blackbox feature image

We presented our paper, “Blackbox meets Blackbox⁚ Representational Similarity and Stability Analysis of Neural Language Models and Brains”, at Blackbox NLP workshop at ACL 2019 in Forence! In this paper, we define and apply representational stability analysis (ReStA), an intuitive way of analyzing neural language models.


From Attention in Transformers to Dynamic Routing in Capsule Nets

From Attention in Transformers to Dynamic Routing in Capsule Nets feature image

In this post, we go through the main building blocks of transformers and capsule networks and try to draw a connection between different components of these two models. Our main goal here is to understand if these models are inherently different, and if not, how they relate.


Incremental Reading for Question Answering

Incremental Reading for Question Answering feature image

We presented our paper on “Incremental Reading for Questions Answering” in the Continual Learning workshop at NeurIPS 2018 in Montreal. This is about the project I worked on during my internship at Google in summer 2018.


Experiential, Distributional and Dependency-based Word Embeddings have Complementary Roles in Decoding Brain Activity

Experiential, Distributional and Dependency-based Word Embeddings have Complementary Roles in Decoding Brain Activity feature image

Our paper “Experiential, Distributional and Dependency-based Word Embeddings have Complementary Roles in Decoding Brain Activity” has been accepted for poster presentation at Cognitive Modeling and Computational Linguistics workshop (2018).