Local minima and generalization

We consider a number of popular beliefs within the neural network community on the training and generalization behavior of multi-layer perceptrons, and to some extent recurrent networks: (a) 'the solution found is often close to the global minimum in terms of the magnitude of the error', (b) 'smaller networks generalize better than larger networks', (c) 'the number of parameters in the network should be less than the number of data points in order to provide good generalization'. For the tasks and methodology we consider, we show that a) the solution found is often significantly worse than the global minimum, b) oversize networks can provide improved generalization due to their ability to find better solutions, and c) that the optimal number of parameters with respect to generalization error can be much larger than the number of data points.