Shepherd, Adrian John; (1995) Novel second-order techniques and global optimisation methods for supervised training of multi-layer perceptrons. Doctoral thesis (Ph.D), UCL (University College London).

Text Novel_second-order_techniques_.pdf Download (5MB)

Abstract

Conventional training methods for multi-layer perceptrons (MLPs), derived from the traditional backpropagation algorithm, have three serious inadequacies; convergence to a solution is frequently slow; they do not always converge to the desired global solution; and their performance is highly dependent on the setting of one or more user-defined parameters. A growing body of research indicates that second-order training methods, derived from classical optimisation theory, offer substantial improvements in training speed and a reduced sensitivity to initial parameter settings. However, experiments conducted for this research suggest that most second-order methods have worse global convergence properties than conventional methods. On the other hand, training methods that are designed to have better global convergence characteristics than conventional methods - for example, stochastic training methods - are typically as slow or slower than conventional methods. The aim of this research is to develop MLP training algorithms that are both fast and 'globally-reliable' by combining second-order methods with a novel deterministic strategy for global optimisation. Expanded Range Approximation (ERA). Unlike most stochastic methods for global optimisation, the implementation of ERA with a second-order algorithm is trivial. When tested on benchmark training tasks, hybrid second-order/ERA algorithms (with appropriate parameter settings) were considerably faster and converged to a global minimum as or more frequently than conventional algorithms. This thesis also gives practical guidelines for the efficient implementation of second-order training algorithms, with particular attention paid to factors that affect the probability of a given algorithm attaining a global minimum. In addition, a novel line-search algorithm is presented that offers an efficient compromise between the reliability of safeguarded polynomial interpolation and the speed of backtracking line searches; used as part of a second-order training algorithm, only a single function evaluation is required per training iteration in the best case.

Download activity - last month

Export as CSVJSONXML

Download activity - last 12 months

Export as JSONXMLCSV

Downloads by country - last 12 months

Export as CSVXMLJSON

Archive Staff Only

View Item