by Manish Myst B.E E&C 2011 batch shri sant gadge baba college of engg & tech, bhusawal SSGBCOET

2. Self Organization: An ANN can create its own organization or representation of the information it receives during learning time.

3. Real Time Operation: ANN computations may be carried out in parallel, and special hardware devices are being designed and manufactured which take advantage of this capability.

5. Satellite image where the data loss is undesirable because of image collect cost.

6. Medical image where difference in original image and uncompressed one can compress diagnostic accuracy.

7. Tele -video conferencing.

8. Remote sensing.

9. Space and hazardous waste water application

10. Control of remotely piloted vehicle in military.

12. Compression Speed

13. Mean Square Error

15. The efficiency of the software or hardware implementation of the algorithm.

18. Self Organization: An ANN can create its own organization or representation of the information it receives during learning time.

19. Real Time Operation: ANN computations may be carried out in parallel, and special hardware devices are being designed and manufactured which take advantage of this capability.

21. Deciding the type of connections among neurons for different layers, as well as among the neurons within a layer.

22. Deciding the way a neuron receives input and produces output.

23. Determining the strength of connection within the network by allowing the network learns the appropriate values of connection weights by using a training data set.

25. The activity of each hidden unit is determined by the activities of the input units and the weights on the connections between the input and the hidden units.

27. Because back propagation is a supervised learning algorithm, the desired outputs are given as part of the training vector. The actual network outputs are subtracted from the desired outputs and an error signal is produced.

30. A limit imposed on the strength of each neuron.

32. Classification: Use input values to determine the classification e.g. is the input the letter A, is blob of the video data a plane and what kind of plane is it.

33. Data association: Like classification but is also recognizes data that contains errors. E.g. not only identify the character that were scanned but identify when the scanner is not working properly.

34. Data Conceptualization: Analyze the inputs so that grouping relationships can be inferred. E.g. extract from a database the names of those most likely to by a particular product.

35. Data Filtering: Smooth an input signal. e.g. take the noise out of a telephone signal.Artificial Neural Networks (ANN) is currently a 'hot' research area in medicine and it is believed that they will receive extensive application to biomedical systems in the next few years. At the moment, the research is mostly on modeling parts of the human body and recognizing diseases from various scans (e.g. cardiograms, CAT scans, ultrasonic scans, etc.). Neural networks are ideal in recognizing diseases using scans since there is no need to provide a specific algorithm on how to identify the disease. Neural networks learn by example so the details of how to recognize the disease are not needed. What is needed is a set of examples that are representative of all the variations of the disease. The quantity of examples is not as important as the 'quantity'. The examples need to be selected very carefully if the system is to perform reliably and efficiently.<br />5. IMAGE COMPRESSION WITH NEURAL NETWORKS:<br />Apart from the existing technology on image compression represented by series of JPEG, MPEG, and H.26x standards, new technology such as neural networks and genetic algorithms are being developed to explore the future of image coding. Successful applications of neural networks to vector quantization have now become well established, and other aspects of neural network involvement in this area are stepping up to play significant roles in assisting with those traditional compression techniques. Existing research can be summarized as follows:<br />1.Back-Propagation image Compression<br />2.Hierarchical Back-Propagation Neural Network<br />3.Adaptive Back-Propagation Neural Network<br />4.Hebbian Learning Based Image Compression<br />5.Vector Quantization Neural Networks<br />6.Predictive Coding Neural Networks.<br />5.1 BASIC BACK PROPAGATION NEURAL NETWORK<br />Minsky and Papert (1969) showed that there are many simple problems such as the exclusive or problem which linear neural networks can not solve. Note that term quot;
solvequot;
means learn the desired associative links. Argument is that if such networks can not solve such simple problems how they could solve complex problems in vision, language, and motor control. Solutions to this problem were as follows:<br />•Select appropriate quot;
recodingquot;
scheme which transforms inputs.<br />•Perceptron Learning Rule Requires that you correctly quot;
guessquot;
an acceptable input to hidden unit mapping.<br />•Back propagation learning rule Learn both sets of weights simultaneously.<br />Back propagation is a form of supervised learning for multi layer nets, also known as the generalized delta rule. Error data at the output layer is quot;
back propagatedquot;
to earlier ones, allowing incoming weights to these layers to be updated. It is most often used as training algorithm in current neural network applications. The back propagation algorithm was developed by Paul Werbos in 1974 and rediscovered independently by Rumelhart and Parker. Since its rediscovery, the back propagation algorithm has been widely used as a learning algorithm in feed forward multilayer neural networks. What makes this algorithm different than the others is the process by which the weights are calculated during the learning network. In general, the difficulty with multilayer Perceptrons is calculating the weights of the hidden layers in an efficient way that result in the least (or zero) output error; the more hidden layers there are, the more difficult it becomes. To update the weights, one must calculate an error. At the output layer this error is easily measured; this is the difference between the actual and desired (target) outputs. At the hidden layers, however, there is no direct observation of the error; hence, some other technique must be used. To calculate an error at the hidden layers that will cause minimization of the output error, as this is the ultimate goal. The back propagation algorithm is an involved mathematical tool; however, execution of the training equations is based on iterative processes, and thus is easily implementable on computer.<br />The neural network structure can be illustrated in Fig.5.1. there layers, one input layer, one output layer and one hidden layer, are designed. Both input layer and output layer are fully connected to the hidden layer. Compression is achieved by designing the value of K, the number of neurons at the hidden layer, less than that of neurons at both input and output layers. The input image is split up into blocks or vectors of 8x8, 4x4 or 16x16 pixels. When the input vector is referred to as N-dimensional which connected to each neuron at the hidden layer can be represented by {wjb j=1,2,… K and I=1,2.. N}, which can also be described by a matrix of K x N. From the hidden layer to the output layer, the connections can be represented by {wij;: 1< < N, 1< j< K} which is another weight matrix of N x K. Image compression is achieved by training the network in such a way that the coupling weights { wij }, scale the input vector of N-dimension into a narrow channel of K-dimension (K<N) at the hidden layer and produce the optimum output value which makes the quadratic error between input and output minimum. In accordance with the neural network structure, the operation can be described as follows:<br />For encoding <br />for decoding.<br />where denotes the normalized pixel values for grey scale images with grey levels [0,255]. The reason of using normalized pixel values is due to the fact that neural networks can Operate more efficiently when both their inputs and outputs are limited to a range of [0,1].<br />Figure5.1 Back Propagation Neural Network<br />The above linear networks can also be designed into non-linear if a transfer function such as sigmoid is added to the hidden layer and the output layer to scale the summation down in the above equations.<br />With this basic back-propagation neural network, compression is conducted in two phases: training and encoding. In the first phase, a set of image samples is designed to train the network via back-propagation learning rule, which uses each input vector as the desired output. This is equivalent to compressing the input into the narrow channel represented by the hidden layer and then reconstructing the input from the hidden to the output layer.<br />The second phase simply involves the entropy coding of the state vector hj at the hidden layer. In cases that adaptive training is conducted, the entropy coding of those coupling weights is also required in order to catch up with some input characteristics that are not encountered at the training stage. The entropy coding is normally designed as the simple fixed length binary coding although many advanced variable length entropy-coding algorithms are available. <br />This neural network development, in fact, is in the direction of K-L transform technology, which actually provides the optimum solution for all linear narrow channel type of image compression neural networks. Equations (1) and (2) are represented in matrix form:<br />for encoding and decoding.<br />the K-L transform maps input images into a new vector space where all the coefficients in the new space is de-correlated. This means that the covariance matrix of the new vectors is a diagonal matrix whose elements along the diagonal are eigen-values of the covariance matrix of the original input vectors. Let ej and j, i=1, 2…n, be eigen-vectors and eigen values of cx the covariance matrix for input vector x, and those corresponding eigen values are arranged in a descending order so that <br /> > +1, for I=1,2,..n-1. <br />to extract the principal components, K eigen vectors corresponding to the K largest eigen-values in cx. in addition, all eigen-vactors in [AK]are ordered in such a way that the first row of {AK} is the eigen-vector corresponding to the smallest eigen-value. Hence, the forward K-L transform or encoding can defined as:<br />and the inverse K-L transform or decoding can be defined as:<br />where [mx] is the mean value of [x] and represents the reconstructed vectors or image blocks. Thus the mean square error between x and is given by the following equation:<br />where the statistical mean value E{.} is approximated by the average value over all the input vector samples which, in image coding, are all the non-overlapping blocks of 4x4 or 8x8 pixels.<br />Therefore, by selecting the K eigen-vectors associated with largest eigen-values to run K-L transform over input pixels, the resulting errors between reconstructed image and original one can be minimized due to the fact that the values of s decrease monotonically.<br />For the comparison between the equation pair (3-4) and the equation pair (5-6), it can be concluded that the linear neural network reaches the optimum solution whenever the following condition is satisfied:<br /> <br />under this circumstance, the neuron weights from input to hidden and from hidden to output can be described respectively as follows:<br /> <br />where [U] is an arbitrary K x K matrix and [U] [U]-1 gives an identity matrix of K x K. Hence, it can be seen that the liner neural network can achieve the same compression performance as that of K-L transform without necessarily obtaining its weight matrices being equal to [AK]T and [AK].<br />5.2 HIERARCHICAL BACK-PROPAGATION NEURAL NETWORK<br />The basic back-propagation network can be further extended to construct a hierarchical neural network by adding two more hidden layers into the existing network, in which the three hidden layers are termed as combiner layer, compressor layer and decomposer layer. The structure can be shown in Figure 4.2. the idea is to exploit correlation between pixels by inner hidden layer and to exploit correlation between blocks of pixels by outer hidden layers. From input layer to combiner layer and decombiner layer to output layer, local connections are designed, which has the same effect as M fully, connected neural sub-networks.<br />Figure5.2 Hierarchical Neural Network Structure<br />Training such a neural network can be conducted in terms of : <br />Outer Loop Neural Network (9OLNN) Training<br />Inner Loop Neural Network (ILNN) Training<br />Coupling weight allocation for the Overall Neural Network.<br />5.3 ADAPTIVE BACK-PROPAGATION NEURAL NETWORK<br />Adaptive back-propagation neural network is designed to make the neural network compression adaptive to the content of input image. The general structure for a typical adaptive scheme can be illustrated in Fig. 4.3, in which a group of neural networks with increasing number of hidden neurons (hmin, hmax), is designed. The basic idea is to classify the input image blocks in age blocks into a few sub-sets with different features according to their complexity measurement. A fine tuned neural network then compresses each sub-set.<br />356235224790<br />Figure 5.3 Adaptive Neural Network Structure<br />Training of such a neural network can be designed as : <br />parallel training <br />serial training<br />activity based training;<br />The parallel training scheme applies the complete training set simultaneously to all neural networks and use S/N (signal-to-noise) ratio to roughly classify the image blocks into the same number of sub-sets as the of neural networks. After this initial coarse classification is completed, each neural network is then further trained by its corresponding refined sub-set of training blocks.<br />Serial training involves an adaptive searching process to build up the necessary number of neural networks to accommodate the different patterns embedded inside the training images. Starting with a neural network with predefined minimum number of hidden neurons, hmin, the neural network is roughly trained by all the image blocks. The S/N ratio, further training is started to the next neural network with the number of hidden neurons increased and the corresponding threshold readjusted for further classification. This process is repeated until the whole training set is classified into a maximum number of sub-sets corresponding to the same number of neural networks established.<br />In the next two training schemes, extra two parameter, activity A(Pj) and four directions are defined to classify the training set rather than using the neural networks. Hence the back propagation training of each neural network can be completed in one phase by its appropriate sub-set.<br />The so-called activity of the block is defined as:<br /> <br />and <br /> <br />where AP(Pi(I,j)) is the activity of each pixel which concerns its neighboring 8 pixels as r and s vary from –1 to +1 in equation (11).<br />Prior to training, all image blocks are classified into four classes according to their activity values, which are, identified as very low, low, high and very high activities. Hence four neural networks are designed with increasing number of hidden neurons to compress the four different sub-sets of input images after the training phase is completed.<br />On top of the high activity parameter, further feature extraction technique is applied by considering four main directions presented in image details, i.e., horizontal, vertical and the two diagonal directions. These preferential direction features can be evaluated by calculating the values of mean squared differences among neighboring pixels along the four directions.<br />For the image patterns classified as high activity, further four neural network corresponding to the above directions are added to refine their structure and tune their learning processes to the preferential orientations of the input. Hence the overall neural network system is designed to have six neural networks among which two correspond to low activity and medium activity sub-sets and other four networks correspond to the high activity and four direction classifications.<br />5.4 HEBBIAN LEARNING BASED IMAGE COMPRESSION <br />While the back-propagation based narrow-channel neural network aim at achieving compression upper bounded by K-L transform, a number of Hebbian learning rules have been developed to address the issue how the principal components can be directly extracted from input image blocks to achieve image data compression. The general neural networks structural consists of one input layer and one output layer. Hebbian learning rule comes from hebb’s postulation that if two neurons are very active at the same time which is illustratyed by the high values of both its output and one of its inputs, the strength of the connection between the two neurons will grow or increase. Hence, for the output values expressed as [h]=[w]T[x], the learning rule can be described as:<br />where, Wi(t+1) = {Wi1, Wi2,….WiN}- the ith new coupling weight vector in the next cycle (t+1); 1 < I < M and M is the number of output neurons. <br />- learning rate; hi(t)- ith output value; X(t)-input vector, corresponding to each individual image block.<br />11-Euclidean norm used to normalize the updated weights and make the learning stable. <br />5.5 VECTOR QUANTIZATION NEURAL NETWORKS<br />Since neural networks are capable of learning from input information and optimizing itself to obtain the appropriate environment for a wide range of tasks, a family of learning algorithms have been developed for vector quantization. The input vector is constructed from a K-dimensional space. M neurons are designed to compute the vector quantization code-book in which each neuron relates to one code-word vitas compling weights. The coupling weights. The coupling weight, {Wij}, associated with the I’th neuron is eventually trained to represent the code-word ci in the code-book. As the neural network is being trained, all the coupling weights will be optimized to represent the best possible partition of all the input vectors. To train the network, a group of image samples known to both encoder and decoder is often designated as the training set, and the first M input vectors of the training data set are normally used to initialize all the neurons. With this general structure, various learning algorithms have been designed and developed such as Kohone’s self-organising feature mapping, competitive learning, frequency sensitive competitive learning fuzzy competitive learning, general learning, and distortion equalized fuzzy competitive learning and PVQ (predictive VQ) neural networks.<br />Let Wi(t) be the weight vector of the I’th neuron at the I’th iteration, the basic competitive learning algorithm can be summarized as follows:<br /> <br />where d(x, Wi(t)) is the distance in L2 metric between input vector x and the coupling weight vector Wi(t)= { wi1,wi2….Wik}; K=p x p ; is the leering rate, and z i is its output.<br />A so called under utilization problem occurs in competitive learning which means some of the neurons are left out of the learning process and never win the competition. Various schemes are developed to tackle this problem. Kohonen self-organising neural network overcomes the problem by updating the winning neuron as well as those in its neighborhood.<br />Frequency sensitive competitive learning algorithm address the problem by keeping a record of how frequent each neuron is the winner to maintain that all neurons is the network are updated an approximately equal number of times. To implement this scheme, the distance is ,modified to include the total number of times that the neuron I is the winner. The modified distance measurement is defined as:<br />where ui(t) is the total number of winning times for neuron I up to the t’th training cycle. Hence, the more the I’th neuron wins the competition, the greater its distance from the next input vector. Thus, the change of winning the competition diminishes. This way of tackling the under-utilization problem does not provide interactive solutions in optimizing the code-book.<br />Around the competitive learning scheme, fuzzy membership functions are introduced to control the transition from soft to crisp decision during the code-boo design process. The essential idea is that one input vector is assigned to a cluster only to a certain extent rather than either ‘in’ or ‘out’. The fuzzy assignment is useful particularly at earlier training stages, which guarantees that all input vectors are included in the formation of new code-book represented by all the neuron coupling weights. Representative examples included direct fuzzy competitive learning, fuzzy algorithms for learning vector quantization and distortion equalized fuzzy competitive learning algorithm etc.<br />5.6 PREDICTIVE CODING NEURAL NETWORKS<br />predictive coding has been proved a powerful technique in de-correlating input data for speech compression and image compression where a high degree of correlation is embedded among neighboring data samples. Although general predictive coding is classified into various models such as AR and