A Rationale from a Frequency Perspective for Grokking in Training Neural Networks

Le résumé fourni par la source

Grokking is a phenomenon where neural networks (NNs) initially fit the training data and later generalize to the test data during training. In this paper, we empirically provide a frequency perspective to explain the emergence of this phenomenon in NNs. We employ an insufficient and nonuniform sampling approach to construct the training dataset, which results in the presence of spurious frequency components when compared to the test dataset. Our key insight reveals that neural networks initially learn spurious frequency components, and we observe this phenomenon across synthetic data, solving PDEs, and parity function datasets, offering a novel viewpoint for elucidating the grokking phenomenon by characterizing it through the lens of frequency dynamics during the training process. Our empirical frequency-based analysis sheds new light on understanding the grokking phenomenon and its underlying mechanisms.

Ce résumé expose les affirmations des auteurs. BNTIC ne l’interprète pas comme une validation indépendante des résultats.