Optimization methods of neural networks for solving the problem of binary classification

Krutov T. Y., Afanasyev G. I., Nesterov Yu. G.

The aim of this work is to carry out a comparative analysis of existing methods for optimizing neural networks and to determine the best optimizers for solving the problem of binary classification in pattern recognition. A comparative analysis of existing methods for optimizing neural networks is carried out and a number of optimizers are identified that show the best quality of training for solving the problem of binary classification in image recognition of the used data set. The article briefly describes mathematical expressions for calculating the updated parameters of a neural network. The gradient descent method, SGD, the Nesterov method, and the Momentum method are considered. Adaptive optimization methods such as Adagrad, RMSProp, and Adam are also described. Two neural architectures are considered: the first architecture is a convolutional neural network with four convolution layers, the second network consists of a VGG19 neural network pre-trained on an ImageNet set with an added classifier. Additional training of the model is performed by freezing all layers of the VGG19 neural network except for the layers starting with the block5_conv1 layer. The composition of the network layers is described in the text and in the figures. The "Dogs vs. Cats" dataset with balanced image classes was used as a training set. The models were trained on the CPU without using graphics accelerators. The results of training and testing models are shown in the graphs of accuracy and loss. For ease of perception, each graph contains the learning curves of all models. Additionally, a boxplot diagram is constructed showing the probability distribution and median estimates on the test data set. Recommendations for choosing the architecture of neural networks are described.

Keywords:neural network optimizers, transfer learning, neural network models, convolutional neural networks, VGG19.