Modern Vision & Multimodal

Where vision has been heading for the last several years: attention instead of (or alongside) convolution, images understood jointly with language, and vision extended into time (video) and space (3D) instead of a single flat frame.

Vision Transformers in Production

Vision Architectures covers ViT, DeiT, and Swin Transformer in full architectural depth. In production specifically, the practical choice between a CNN backbone and a ViT-family backbone usually comes down to: available pretraining data and compute (ViTs need more of both to beat a CNN, per the same page's discussion), inference latency constraints (Swin's windowed attention closes much of the gap CNNs otherwise win on efficiency), and whether the downstream task benefits from a global receptive field from layer one (detection/segmentation on large, cluttered scenes tend to benefit more than simple classification does).

Vision-Language Models and Image-Text Alignment

CLIP-style dual encoders: covered in Advanced Architectures — Multimodal — a vision encoder and a text encoder trained contrastively into a shared embedding space, the foundation of zero-shot image classification and text-to-image retrieval.
Fusion-based VLMs (LLaVA, Qwen-VL, GPT-4V-style models): covered in Multimodal & Generative Models — an image encoder's features get projected directly into an LLM's token space, letting the model reason jointly over image and text rather than just comparing embeddings.
Serving VLMs in production: see LLM Hosting & Serving Patterns — Multimodal/VLM Inference for the image→vision-encoder→projector→LLM serving path and which inference engines support it.

Image and Video Generation

Image generation: GANs and diffusion models, covered in full depth in Generative Models — DDPM/DDIM, Latent/Stable Diffusion, ControlNet, and why diffusion overtook GANs for this task specifically.
Video generation: extends image diffusion along a time axis — the core challenge beyond per-frame image quality is temporal consistency (an object's appearance and the scene's geometry need to stay coherent frame to frame, not just look plausible in isolation). Approaches generally either extend a 2D diffusion U-Net with temporal attention/convolution layers connecting adjacent frames, or model video generation directly in a learned spatiotemporal latent space — an active, fast-moving research area where architectural details vary significantly model to model.

Generating each frame independently: the object's size and color jump around frame to frame -- individually each frame might look fine, but played as video it flickers and warps, since nothing ties consecutive frames together.

Video Understanding

Classifying or describing what's happening in a video, as opposed to generating one — the discriminative counterpart to video generation:

Action recognition: classify what action is being performed in a video clip — architecturally, either a 3D CNN (convolving across height, width, and time simultaneously, extending Conv3D) or a two-stream approach combining per-frame appearance features with optical flow motion features, or a video Transformer applying attention across both spatial patches and time steps.

Input	A stacked clip of frames (height × width × time)
Mechanism	Convolves across height, width, AND time simultaneously -- the kernel itself has a temporal extent, so a single layer already mixes information across a few adjacent frames.

Conceptually the simplest extension of a 2D CNN -- but 3D kernels are expensive, and effective temporal receptive field grows slowly without deep stacks.

Temporal action localization: not just "what action" but "when does it start and end" within a longer, untrimmed video — structurally similar to object detection, except localizing a time interval instead of a spatial bounding box.
Video captioning / video QA: describing or answering questions about video content — extends the VLM approach above by feeding a sequence of sampled frames (or learned video features) into a language model instead of a single image.

3D Vision

Moving beyond a single 2D image to genuine 3D scene understanding:

Point clouds: a set of 3D points (x, y, z, often with additional attributes like color), the standard representation for LiDAR sensor output — processed by architectures specifically designed for unordered point sets (e.g. PointNet-style networks), since a point cloud has no natural grid structure for standard convolution to exploit.

Rotation35°

336 points, rotated live — nearer points drawn larger and more opaque (a simple depth cue, painter's algorithm).

A LiDAR scan is exactly this: thousands of (x, y, z) points with no grid structure connecting them -- point A has no inherent 'neighbor' the way a pixel does. That's precisely why standard convolution doesn't apply directly, and why point-cloud architectures (PointNet and successors) are built around operations that don't assume any particular point ordering.

NeRF (Neural Radiance Fields): represents an entire 3D scene implicitly as a neural network — given a 3D coordinate and a viewing direction, the network outputs color and density at that point, and rendering a novel viewpoint means querying the network along camera rays through the scene. Trained from a set of 2D photos of a scene from different angles, producing remarkably photorealistic novel-view synthesis without ever building an explicit 3D mesh.

Ray offset from object center0.00

final pixel

Density along the ray

This ray's accumulated color weight from the object is 1.00 (vs. 0.00 background) -- computed by marching along the ray, querying density at each sample point, and accumulating color weighted by how much light survives to reach that point (transmittance) times how much this point itself absorbs (alpha). A ray straight through the object's center accumulates almost entirely object color; a ray that misses picks up almost none -- exactly what the offset slider demonstrates.

3D reconstruction: recovering 3D scene structure from 2D images — classical Structure from Motion (SfM) triangulates 3D points from matched features (see Vision Fundamentals — SIFT/ORB) across multiple photos of the same scene from different viewpoints; modern learned approaches increasingly fold this into end-to-end differentiable pipelines, including NeRF-style implicit representations above.
Applications: autonomous driving (LiDAR point cloud processing), AR/VR (scene reconstruction), robotics (spatial mapping/SLAM), and content creation (turning photos into 3D assets).

Computer Vision section complete. Next: NLP for the equivalent classical-to-modern arc in text, or back to Deep Learning for the underlying architectures this section builds on.