Purpose

This series goes over the basic building blocks of Neural Networks, following the API provided by PyTorch.

The PyTorch documentation provides good explanation of the API, the calculation made, and the shapes of the inputs, outputs, and weights. However, dense text descriptions and mathematical formulas aren’t necessarily intuitive. This series aims to provide intuition at a glance to serve as a reference/memo that supplements the PyTorch documentation.

Disclaimer

This series follows the conventions in PyTorch 2.1.0. The concepts discussed should apply to other deep learning frameworks and future PyTorch releases but there may be caveats and differences in nomenclature.

Table of Content

1. Multi-Layer Perceptrons
   • Linear Layers
   • Activation Functions
   • Dropout
2. Convolution Layers
   • Convolution Layers
   • Padding
   • Pooling
3. Transformer Blocks
   • Embedding
   • Positional Encoding
   • Scaled Dot-Product Attention
   • Transformer Encoder
   • Transformer Decoder
4. Recurrent Layers
   • RNN
   • GRU
   • LSTM
5. Normalization Layers
   • Batch Norm
   • Layer Norm

Special thanks for 3Blue1Brown and community members for the fantastic manim library that powers all the animations in this series.