Statistical Sensitivity Analysis for Training Feedforword Neural Networks by B-Splines Weight Functions

Based on the new architecture and algorithm of training neural networks by B-splines weight functions, the statistical sensitivity analysis for neural networks using B-splines weight functions is discussed in this paper. The sensitivity formula of B-splines weight function neural networks is derived. Finally, the correctness of theoretical sensitivity formula is verified by simulation.


Introduction
Sensitivity refers to how a system output is influenced by its input perturbations.Sensitivity analysis for neural networks can be dated back to the 1960s.Many scholars have studied sensitivity or the sensitivity of neural networks [1,2].In 1962, Hoff used an n-dimensional hypersphere to model Adaline for sensitivity analysis.Winter (1989) was the first one to derive an analytical expression for the probability of error in Madaline caused by weight perturbations.Stevenson continued Winter's work, established the sensitivity of Madaline to weight error, defined the mathematical model of sensitivity as hypersphere and analyzed the sensitivity of ADALINE.Piché [3] used a statistical approach to relate the output error to the change of weights for an ensemble of Madalines, with several activation functions such as linear, sigmoid, and threshold.On the related literature, other sensitivity function has been defined, such as output sensitivity, trajectory sensitivity, function sensitivity, etc.
By early algorithms (such as backpropagation (BP) or radial basis function (RBF) algorithm), the trained weights were constant data (constant weights).For the constant weights, let us assume that the connection weight vector changes from W* to W=W*+∆W, where W* is the matrix of constant weights for the global minima of neural networks, and ∆W indicates the weight perturbations.Under the assumption of statistical weight perturbations, the statistical sensitivity can be defined as follows [2].
It is well known that artificial neural networks have been widely used in many fields [4].BP algorithm is trained by the steepest descent-like algorithms and can not find global minima in many applications.Many techniques have been introduced to improve the performance of the steepest descent-like algorithms.Those studies mainly focus on: improving the learning speed [5,6], changing the network's architecture, growing and pruning algorithm [7,8], optimizing the initialized weights or some other values [9], convergence of online training procedure [10], using more additional parameters, deterministic convergence, online gradient method [11], tuning network's parameters, neural network's algorithms using genetic or evolutionary methods [12], hybrid neural networks and so on.Unfortunately, the drawbacks can not be overcome completely arising from the steepest descent-like algorithms, such as the local minima, the slow convergence speed and the limited scale of problems.Therefore, the statistical sensitivity defined in (1) can hardly be used to find correct results.
In recent years, a new class of algorithms for training feedforward neural networks has been proposed by Zhang [13], who changes the weights from constant data to the cubic spline functions (weight functions), and the arguments of the cubic spline functions are input patterns.
The new algorithms proposed by [13] not only simplify the architecture of neural networks, but also overcome the drawbacks by using early algorithms, such as local minima, slow convergence and difficult to obtain the global optimal point.
Distinct form [13], a new algorithm for training neural networks using B-spline weight functions are introduced in this paper and the mathematic expression for statistical sensitivity is also obtained, in which the definition of statistical sensitivity is different from (1).Finally, the simulation examples show that the correctness of theoretical sensitivity formula derived from this paper.

Fundamentals of training neural networks by B-splines Weight Functions
Fig. 1 shows a model of B-splines function neural network, where ( ) Suppose there are 2 N + patterns, and ( ) w x is the theoretical weight function which represents the connection with the i-th ( ) ( ) ) where , the interpolate nodes can be expressed as Add Let where ( ) Any spline function ( ) b x can be expressed as 1 , ( ) ( ) where i c is a constant.

Advanced Engineering Forum Vols. 6-7 1215
For the interval[ ] , a b ，the spline space is N k + dimensional, and the function ( ) interpolation nodes, let the interpolation points be The k-th spline interpolation satisfies the following conditions ( ) Where i y is the output value, and ( ) b x satisfied ( 10) is called the k-th spline interpolation function.
If we take 0 1 , , , N x x x ⋅⋅⋅ as spline interpolation nodes, from ( 10) and ( 8) we have ( ) We can solve the linear system (11) for finding the coefficient i c .

Statistical Sensitivity Analysis of B-splines Weight Function Neural Networks
Different from (1), the definition of statistical sensitivity can be described as: Where * 0

X
σ is standard deviation of the disturbances on input patterns, ( ) S X represent the value of sensitivity.When noise x ∆ is embedded, the theoretic error of B-splines weight function neural networks can be expressed as Where … represents some kinds of norms.Generally speaking, the theoretical noise errors of neural networks contain model error and approximation noise error., N x x + can get the maximum values, we have where p is the index of subinterval 1 , Suppose the input without perturbation is 1 , , then the approximation error will be From ( 16) and ( 6), we can get where

Information Technology for Manufacturing Systems III
Let the dimension of input and output be m and n respectively, ( ) represent approximate output and target output respectively, the output perturbation is ( If the function ( ) ) is continuous, then ( ) ( ) we have Therefore, the sensitivity of B-splines weight function can be expressed by (12) in the following ( ) ( ) ( )

Simulations
To show the results proposed in this paper, an example is given below.The architecture of the network is 3-4, and the learning cure [13] is The output patterns are obtained by ) Advanced Engineering Forum Vols.6-7 1217 When the input perturbation is very small, theoretical noise sensitivity matches approximate noise sensitivity, see Fig. 2 Fig. 3 illustrates that the output relative error is accordingly very small (close to zero), so we can conclude that the network has generation ability.As can be seen from Fig. 2 and Fig. 3, when the input perturbations increase to some extent, the sensitivity changes quite so obvious, i.e. the sensitivity may reduce, may increase, or increase and then decrease.But the output relative error increases rapidly at the same time.The network doesn't remain stable, very sensitive to the input perturbations and the output deviates from the target values.

Conclusions
Although some results of statistical sensitivity analysis have been achieved using early algorithms, further analysis of the method proposed in this paper is necessary for improved insight into its effectiveness.The sensitivity formula of B-splines weight function neural networks is derived and the correctness of theoretical sensitivity formula is verified by simulations.It can be seen that the theoretical value of sensitivity is determined by several factors.The sensitivity depends not only on the weight functions, but also on the input patterns, which could be determined by measuring the sensitivity of the network.

Fig. 1
Fig. 1 The architecture of neural network using B-splines function The approximate weight function ( ) i i b x and the theoretic weight function ( ) i i w x have the same value on the interpolation points, but there are some errors out of the interpolation points.Let

Firstly, we will
analyze model error ( ) ( ) w x b x − .Many norms can be adapted to measure errors.Here we use Chebyshev norm.For continuous function, Chebyshev norm is the maximum of absolute value.That is are continuous functions, then ( ) w x and ( ) get B-splines weight functions by the training patterns.The statistical sensitivity of B-splines weight function neural network can be calculated by (22).For the trained neural network, suppose the output without disturbance is *Y , and the disturbed output is Y , then relative error can be got by the formula as follows.

Fig. 2
Fig. 2 Sensitivity of B-splines weight functions