Vision Tasks & Models

"Computer vision" is not one task — it's a family of structurally different problems that happen to share an image as input. This page is the full taxonomy, recapping the tasks with model architectures already covered in depth elsewhere, and covering the ones that aren't: pose estimation, OCR, tracking, depth estimation, and optical flow.

The Core Three (Recap)

Covered in full architectural depth in Vision Architectures — linked here for completeness of the taxonomy:

• Classification: assign one label to an entire image. The original CNN task (CNNs's LeNet-through-ConvNeXt lineage).
• Detection: locate and classify multiple objects as bounding boxes — R-CNN family, YOLO, SSD, DETR.
• Segmentation: classify every pixel — semantic (class per pixel) vs. instance (also distinguishes individual objects) — U-Net, Mask R-CNN.

Pose Estimation

Locate the positions of specific keypoints on a subject — most commonly human joints (wrists, elbows, knees), but the same idea applies to animal or object pose. Two architectural approaches:

• Top-down: first detect each person (a standard detection model), then run a pose-estimation model on each cropped detection independently — simpler per-person accuracy, but cost scales with the number of people in the frame.
• Bottom-up: detect all keypoints across the whole image in one pass, then group them into individual people's skeletons afterward — cost stays roughly constant regardless of how many people are in frame, at the expense of a harder grouping step.
• Applications: fitness/sports motion analysis, animation and motion capture, human-computer interaction (gesture recognition), and animal behavior research.

OCR (Optical Character Recognition)

Reading text out of an image — structurally two sub-problems chained together:

• Text detection: locate where text regions are in the image (a specialized object-detection problem, output as bounding boxes or polygons around text lines/words).
• Text recognition: given a cropped text region, decode the actual character sequence — historically CNN + RNN/CTC-loss pipelines (predict a character sequence without needing pre-segmented individual characters), increasingly Transformer-based sequence-to-sequence models that treat this as image-to-text generation directly (see Attention & Transformers).
• Modern approach: end-to-end Transformer-based OCR models (and general-purpose VLMs, see Modern Vision & Multimodal) increasingly fold detection and recognition into one model rather than a two-stage pipeline, especially for documents with complex layouts (tables, forms) where layout understanding matters as much as character recognition.

Object Tracking

Given detections in each frame of a video, maintain consistent identity for each object across frames — "this is the same car in frame 47 that was in frame 12," not just "there's a car in this frame":

• Detection-based tracking (tracking-by-detection): run a detector independently on each frame, then solve an association problem — matching detections across consecutive frames based on position, motion prediction (a Kalman filter predicting where an object should be next, based on its velocity), and appearance similarity (a learned embedding, similar in spirit to Metric Learning's similarity embeddings). SORT and DeepSORT are the classic, still widely used algorithms built on exactly this pattern.
• Why it's hard: occlusion (an object temporarily hidden behind another), objects entering/leaving frame, and visually similar objects crossing paths all make the association step genuinely ambiguous, not just a bookkeeping exercise.
• Applications: autonomous driving (tracking surrounding vehicles/pedestrians), sports analytics, video surveillance, robotics.

Depth Estimation

Predicting how far each pixel is from the camera — turning a 2D image into a 2.5D/3D understanding of the scene:

• Stereo depth: given two cameras at a known offset (like human eyes), the disparity — how much a point shifts between the two images — is inversely proportional to its distance, giving depth via geometry directly, no learning required for the core computation (though modern stereo systems use learned matching for the disparity computation itself).

• Monocular depth estimation: predicting depth from a single image, with no stereo geometry to exploit — a genuinely underdetermined problem in principle (the same 2D image is consistent with infinitely many 3D scenes), solved by deep networks learning strong priors from large datasets of images paired with real depth measurements (LiDAR, structured light) — how modern phone cameras produce depth-of-field ("portrait mode") effects without a second lens.
• Applications: autonomous driving (obstacle distance), AR/VR (scene understanding for object placement), robotics navigation, computational photography.

Optical Flow

Estimating the per-pixel motion between two consecutive video frames — a dense vector field where every pixel gets an estimated (dx, dy) displacement, not just a handful of tracked keypoints. Classical approaches (Lucas-Kanade, Horn-Schunck) assume local brightness constancy (a point's brightness doesn't change between frames, so intensity differences reveal motion) and solve for the flow that best explains observed pixel changes; modern deep learning approaches (RAFT and successors) train a network end-to-end to predict flow directly, substantially outperforming classical methods especially under large motion, occlusion, and lighting change. Optical flow underlies video compression, video frame interpolation (generating in-between frames), action recognition, and provides motion features that improve object tracking's association step above.

Next: Modern Vision & Multimodal — where vision meets language, video, and 3D.