Featured Paper/News in This Week. A 3D-Transformer that can be directly applied to molecular structures has been proposed. The attention weights can be adjusted according to the interatomic distances, and the computational complexity does not seem to be that high. — — — — — — — — — —…

About this post In this post, I introduce a recently published method for self-supervised learning in a framework similar to masked language models. The two papers introduced in this article are MAE (He et al., (2021) and SimMIM (Xie et al., 2021). Each of them can be lightly summarized as follows. MAE The authors…

Featured Paper/News in This Week. SimMIM, a model for image pre-training with a structure similar to a masked language model, has been presented. The concept is similar to MAE introduced last week, but this is a simpler implementation. It may become more common to use masks to pre-train images. The…

Featured Paper/News in This Week. A method is proposed to mask the image and pre-train the model to recover it, like BERT. 75% of the image is masked and only 25% of the unmasked image is input to the encoder, which seems to be memory friendly. An image generation model…

Featured Paper/News in This Week. There have been a study in the past that have shown that ViT classifies with a more human-like behavior than CNN, but now a new study has been published that shows that ViT correctly classifies even when perturbed on a patch-by-patch basis.CNN …

Featured Paper/News in This Week. A new dataset for self-supervised learning has been released that can be used for commercial purposes and is portrait rights friendly. …

This blog post describes the paper “Patches Are All You Need?” (Under review, 2021), which was submitted to ICLR2022 (under review as of the end of the Oct.). The ConvMixer proposed in this paper is composed of CNN+BN, and unlike previous Vision Transformer series, it can achieve results even on…

Featured Paper/News in This Week. Methods to improve the performance of zero-shot inference have been presented. Since GPT-3 zero-shot inference is used in many applications, any improvement in the performance of zero-shot inference may have a significant social impact. — — — — — — — — — — —…

Featured Paper/News in This Week. — A published study shows a sudden improvement in generalization performance from random results: overfitting starts at about 10² steps, and a sudden improvement in generalization performance from random prediction is reported at about 10⁶ steps. Thus, weighted decay seems to be the key to generalization. …