Inicializace v neuronových sítích

Hruška, Martin

This thesis examines how weight initialization affects the training of deep neural net- works, specifically targeting the ResNet architecture, which utilizes full pre-activation residual blocks and ReLU activations. We perform a theoretical derivation of gradi- ent variance propagation, illustrating that skip connections help stabilize training and mitigate the issue of vanishing gradients. With this foundation, we adapt and use the timm PyTorch training framework to evaluate different initialization methods across 30 ResNet-50 models trained using ImageNet. These extensive experiments, carried out over 20 days on a GPU cluster, identify only slight performance variations between the initialization techniques. Our findings indicate that features of modern architectures and optimizers lessen the dependency on particular initializations, highlighting the priority of design decisions over initialization techniques in deep learning.

guest :: login Digital Repository
		Search		Submit		Help		About