19 Oct 2023

Neuron saturation

"Saturating" refers to a situation where the activation function of a neuron reaches its maximum or minimum output value, and any further changes in the input have little to no effect on the output and the neuron becomes unresponsive. This can hinder the learning process in a neural network.

Examples

• Sigmoid function: as the input moves toward positive infinity, the output approaches 1, and as the input moves toward negative infinity, the output approaches 0. When the output is close to 1 or 0, the gradient of the function (which is used in the backpropagation algorithm for training) becomes very small, making it difficult for the network to learn from errors.

• Hyperbolic tangent function: the tanh function has a similar S-shaped curve, and it saturates when its input values are very large or very small.

That's one of the reasons why Rectified Linear Units (ReLUs) have become popular in neural networks. ReLUs do not saturate for positive input values; they output the input as-is if it's positive and zero for negative input values.