SESKStuff Everyone Should Know
SESK: Stuff Everyone Should Know · sesk.info/ai-activation-adds-nonlinearity

Why place nonlinear activation functions between neural network layers?

They let stacked layers model nonlinear relationships.

Start playing free

  1. To sort every input
  2. To remove all parameters
  3. To model nonlinear relationships ✔

Why: A stack of linear operations still collapses to one linear operation. Nonlinear activations let deeper networks represent far more complicated functions.

Huh, didn't know that: Different layers in one network may use different activation functions.

The dad joke
The activation function took a curve because straight lines were too predictable.
Like this card?
Play the game

References

Answer: To model nonlinear relationships. A stack of linear operations still collapses to one linear operation. Nonlinear activations let deeper networks represent far more complicated functions.

  1. Google Machine Learning Crash Course: Activation functions developers.google.com/machine-learning/crash-course/neural-networks/activation-functions
    Stacking nonlinearities on nonlinearities lets us model very complicated relationships between the inputs and the predicted outputs.
    Checked 2026-10-10.

Think this answer is wrong? Tap "Challenge this answer" on the card in the game and tell us why.

Test yourself in the game

Artificial IntelligenceWhat does the "deep" in deep learning refer to?Artificial IntelligenceWhat does the ReLU activation return for an input x?Artificial IntelligenceInto what open interval does the logistic sigmoid map real inputs?

That's stuff everyone should know.