Having the right inductive biases can be crucial in many tasks or scenarios where data or computing resources are a limiting factor, or where training data is not perfectly representative of the conditions at test time. However, defining, designing and efficiently adapting inductive biases is not necessarily straightforward. In this post, I discuss our paper, "Transferring Inductive Biases Through Knowledge Distillation", where we explore the power of knowledge distillation for transferring the effect of inductive biases from one model to another.

In this post, I explain the representational analysis technique and how we can use it in combination with a multi dimensional scaling algorithm to visualize the similarity between multiple models, or different instances of the same model in 2d.

In this post, we try to understand the nature of recurrent inductive bias. I will discuss different sources of inductive biases of RNNs and provide empirical results to demonstrate the benefits of each of them by comparing LSTMs to different variants of Transformers in the context of a task that having the recurrent inductive bias is shown to help achieve better generalization.

In this post, I explain two techniques for visualising attention that address the problem of lack of token identifiability in higher layers of Transformers when using raw attention weights to interpret models' decisions. These techniques are called Attention Rollout and Attention Flow that are introduced in our paper "Quantifying Attention Flow In Transformers".

We presented our paper, “Blackbox meets Blackbox⁚ Representational Similarity and Stability Analysis of Neural Language Models and Brains”, at Blackbox NLP workshop at ACL 2019 in Forence! In this paper, we define and apply representational stability analysis (ReStA), an intuitive way of analyzing neural language models.

In this post, we go through the main building blocks of transformers and capsule networks and try to draw a connection between different components of these two models. Our main goal here is to understand if these models are inherently different, and if not, how they relate.

We presented our paper on “Incremental Reading for Questions Answering” in the Continual Learning workshop at NeurIPS 2018 in Montreal. This is about the project I worked on during my internship at Google in summer 2018.

Our paper “Experiential, Distributional and Dependency-based Word Embeddings have Complementary Roles in Decoding Brain Activity” has been accepted for poster presentation at Cognitive Modeling and Computational Linguistics workshop (2018).