Home // International Journal On Advances in Software, volume 11, numbers 3 and 4, 2018 // View article

A Novel Training Algorithm based on Limited-Memory quasi-Newton Method with Nesterov’s Accelerated Gradient in Neural Networks and its Application to Highly-Nonlinear Modeling of Microwave Circuit

Authors:
Shahrzad Mahboubi
Hiroshi Ninomiya

Keywords: Neural networks; training algorithm; Limited-memory quasi-Newton method; Nesterov's accelerated gradient method; highly-nonlinear function modeling.

Abstract:
This paper describes a novel algorithm based on Limited-memory quasi-Newton method incorporating Nesterov's accelerated gradient for faster training of neural networks. Limited-memory quasi-Newton is one of the most efficient and practical algorithms for solving large-scale optimization problems. Limited-memory quasi-Newton is also the gradient-based algorithm using the limited curvature information without the approximated Hessian such as the normal quasi-Newton. Therefore, Limited-memory quasi-Newton attracts attention as the training algorithm for large-scale and complicated neural networks. On the other hand, Nesterov's accelerated gradient method has been widely utilized as the first-order training algorithm for neural networks. This method accelerated the steepest gradient method using the inertia term for the gradient vector. In this paper, it is confirmed that the inertia term is effective for the acceleration of Limited-memory quasi-Newton based training of neural networks. The acceleration of the proposed algorithm is demonstrated through the computer simulations compared with the conventional training algorithms for a benchmark problem and a real-world problem of the microwave circuit modeling.

Pages: 323 to 334

Copyright: Copyright (c) to authors, 2018. Used with permission.

Publication date: December 30, 2018

Published in: journal

ISSN: 1942-2628