The approximation capabilities of feedforward neural networks with a single hidden layer and with various activation functions has been widely studied ([19], [8], [1], [2], [13]). Mhaskar and Micchelli have shown in [22] that a network using any non-polynomial locally Riemann integrable activation can approximate any continuous function of any number of variables on a compact set to any desired degree of accuracy (i.e. it has the universal approximation property). This important result has advanced the investigation of the complexity problem: If one needs to approximate a function from a known class of functions within a prescribed accuracy, how many neurons are necessary to realize this approximation for all functions in the class? De Vore et al. ([3],) proved the following result: if one approximates continuously a class of functions of d variables with bounded partial derivatives on a compacta, in order to accomplish the order of approximation O(1/n), it is necessary to use at least O(n d) number of neurons, regardless of the activation function. In other words, when the class of functions being approximated is defined in terms of bounds on the partial derivatives, a dimension independent bound for the degree of approximation is not possible. Kurková studied the relationship between approximation rates of one-hidden-layer neural networks with different types of hidden units. She showed in [14] that no sufficiently large class of functions can be approximated by one-hidden-layer networks with another type of unit than Heaviside perceptrons with a rate of approximation related to the rate of approximation by perceptron networks.

Top