NN Layers Reference

Every architecture in this section is built from a small set of reusable layer types. This page is the reference catalog — what each layer does and where it's actually used — for looking up a specific layer rather than reading a full architecture page.

Core Building Blocks

• Dense / Linear / Fully-Connected layer: y = Wx + b — every input connects to every output. The most basic learnable transformation; used as the final classification head in almost every architecture, and as the feed-forward sublayer inside Transformer blocks (see Attention & Transformers).
• Embedding layer: a lookup table mapping discrete tokens (words, categories) to dense learned vectors — the first layer of essentially every NLP/LLM model, converting token IDs into something a network can actually compute over.
• Flatten: reshapes a multi-dimensional tensor (e.g. a CNN's final feature map) into a 1D vector, the standard bridge between convolutional/spatial layers and a dense classification head.

Convolutional Layers

• Conv1D: convolution along a single axis (time or sequence) — used for audio, time series, and text convolutions.
• Conv2D: the standard image convolution, sliding a 2D filter across height and width — the core layer of every CNN in CNNs.
• Conv3D: convolution across height, width, and an additional axis (depth, or time in video) — used for volumetric data (medical CT/MRI scans) and video.
• Depthwise Separable Convolution: factors a standard convolution into a depthwise (per-channel, spatial-only) step and a pointwise (1×1, channel-mixing) step — see CNNs's MobileNet entry for why this is dramatically cheaper.
• Transposed Convolution (Deconvolution): the operation that upsamples rather than downsamples — learns to expand a smaller feature map back to a larger spatial size, used in the decoder half of U-Net (see Vision Architectures) and in GAN/diffusion generators (see Generative Models).
• Pooling (Max/Average): downsamples a feature map by taking the max or average value in each local region — reduces spatial size and parameter count in the layers that follow.

Recurrent Layers

• RNN cell: the basic recurrent unit, h_t = f(h_{t-1}, x_t) — see Sequence Models for the full mechanics and why it struggles with long sequences.
• LSTM cell: adds a gated cell state that explicitly controls what's remembered/forgotten — see Sequence Models.
• GRU cell: a simplified two-gate alternative to LSTM, fewer parameters, often comparable performance.

Attention & Transformer Layers

• Self-Attention layer: computes Query/Key/Value projections and attention-weighted output over a sequence — see Attention & Transformers for the full math.
• Multi-Head Attention layer: runs several attention computations in parallel and concatenates them — see Attention & Transformers.
• Cross-Attention layer: queries from one sequence attend to keys/values from another — the mechanism connecting encoder and decoder in encoder-decoder Transformers (see Attention & Transformers) and connecting image features to text tokens in VLMs.
• Positional Encoding layer: injects order information into an otherwise order-blind attention computation — sinusoidal, learned, or RoPE (see Attention & Transformers).
• Transformer Block: the composed unit — self-attention → add & norm → feed-forward → add & norm — stacked to build every Transformer-based model.

Normalization Layers

• Batch Normalization: normalizes each feature's activations across the current mini-batch, then applies a learned scale and shift — stabilizes and speeds up training of deep CNNs, standard in the CNN lineage. Dependent on batch size/statistics, which makes it a poor fit for architectures processing variable-length sequences one token at a time.
• Layer Normalization: normalizes across a single example's features instead of across the batch — batch-size-independent, which is exactly why it's the default in Transformers rather than BatchNorm.
• RMSNorm: a simplified LayerNorm that only rescales by the root-mean-square of activations (no mean-centering, no learned bias) — cheaper to compute with comparable training stability, and the current default normalization in most modern LLMs (LLaMA and successors).
• Group Normalization: normalizes within groups of channels rather than the whole batch or the whole layer — a middle ground used when batch sizes are too small for BatchNorm to be stable (common in diffusion model U-Nets, see Generative Models).

Regularization & Utility Layers

• Dropout: randomly zeroes a fraction of activations during training, forcing the network not to rely on any single unit — see Initialization, Regularization & LR Scheduling for the full mechanics and why it's turned off at inference time.
• Softmax: converts a vector of raw scores into a probability distribution (non-negative, sums to 1) — the standard final-layer activation for multi-class classification, and the operation that turns attention scores into attention weights.
• Residual / Skip Connection: adds a layer's input back to its output, y = F(x) + x — not a layer with its own parameters, but the connectivity pattern that makes very deep networks trainable (see Training Deep Networks and CNNs's ResNet entry).

Graph Layers

• Graph Convolution layer: aggregates each node's neighbor features (normalized weighted average) — see Advanced Architectures for GCN, GraphSAGE, and GAT, the three main variants of this layer type.

Deep Learning architecture catalog complete. Next: LLMs & GenAI — where these building blocks and architectures become ChatGPT-class production systems, or back to the Deep Learning Roadmap for the full checklist.