Lecture 6 | Training Neural Networks I Sigmoid Problems of the sigmoid activation function Problem 1: Saturated neurons kill the gradients. Problem 2: Sigmoid outputs are not zero-centered. Suppose a given feed-forward neural network has hidden layers and all activation functions are sigmoid. Then, except the first layer, the other layers get only positive inputs. […]