Combined quasi-Newton methods with two-dimensional search for degenerate unconstrained optimization in machine learning

Abstract

This paper presents a two-dimensional search algorithm for quasi-Newton methods applied to ill-conditioned and degenerate unconstrained optimization problems. At each iteration, the space is decomposed into an orthogonal sum of two subspaces based on the spectral decomposition of the approximate Hessian (updated via the BFGS or SR1 formula) and a regularization parameter. The search direction in one subspace is computed using a quasi-Newton scheme, while an alternative optimization method (e.g., gradient descent or conjugate gradient) is employed in the complementary subspace. The next iterate is obtained by minimizing a fourth-order local model of the objective function in two dimensions with respect to the step-size parameters along these directions. The proposed approach enables efficient handling of spectral degeneracy by combining curvature-aware and gradient-based updates within a unified framework. The efficiency of the proposed method is demonstrated through numerical experiments on standard test problems from unconstrained optimization and machine learning. The results are compared with implementations from widely used software environments, including R, Scilab, Python, and PyTorch.