In this article, I present a total of 84 papers and articles published in 2020 that I found particularly interesting. For the sake of clarity, I divide them into 12 sections. My personal summary for 2020 is as follows.

In 2020, Transformer model made a huge leap forward. In natural…


  • A published study shows a sudden improvement in generalization performance from random results: overfitting starts at about 10² steps, and a sudden improvement in generalization performance from random prediction is reported at about 10⁶ steps. Thus, weighted decay seems to be the key to generalization. …


Thoughts and Theory

Vision Transformer (ViT) has been gaining momentum in recent years. This article will explain the paper “Do Vision Transformers See Like Convolutional Neural Networks?” (Raghu et al., 2021) published by Google Research and Google Brain, and explore the difference between the conventionally used CNN and Vision Transformer.

The abstract of this paper and the content of this blog

There are six…






  • A study of unsupervised learning on a large amount of video data is presented, using Transformer to train well-designed tasks in both temporal and spatial directions. …



Week 31, 2021 (Aug 8~)

Akihiro FUJII

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store