Deep Learning - Activations, Convolutions, and Pooling Part 1
This video presents the biological background of activation functions and the classical choices that were used for neural networks.
Further Reading:
A gentle Introduction to Deep Learning