CISPA
Browse
1139_dynamical_isometry_for_residua.pdf (954.46 kB)

Dynamical Isometry for Residual Networks

Download (954.46 kB)
preprint
posted on 2024-04-29, 07:32 authored by Advait GadhikarAdvait Gadhikar, Rebekka BurkholzRebekka Burkholz
The training success, training speed and generalization ability of neural networks rely crucially on the choice of random parameter initialization. It has been shown for multiple architectures that initial dynamical isometry is particularly advantageous. Known initialization schemes for residual blocks, however, miss this property and suffer from degrading separability of different inputs for increasing depth and instability without Batch Normalization or lack feature diversity. We propose a random initialization scheme, RISOTTO, that achieves perfect dynamical isometry for residual networks with ReLU activation functions even for finite depth and width. It balances the contributions of the residual and skip branches unlike other schemes, which initially bias towards the skip connections. In experiments, we demonstrate that in most cases our approach outperforms initialization schemes proposed to make Batch Normalization obsolete, including Fixup and SkipInit, and facilitates stable training. Also in combination with Batch Normalization, we find that RISOTTO often achieves the overall best result.

History

Primary Research Area

  • Trustworthy Information Processing

Open Access Type

  • Green

BibTeX

@misc{Gadhikar:Burkholz:2022, title = "Dynamical Isometry for Residual Networks", author = "Gadhikar, Advait" AND "Burkholz, Rebekka", year = 2022, month = 10, doi = "10.48550/arxiv.2210.02411" }

Usage metrics

    Categories

    No categories selected

    Licence

    Exports

    RefWorks
    BibTeX
    Ref. manager
    Endnote
    DataCite
    NLM
    DC