Computer Vision Basics

Image classification is the foundational task in computer vision: given an input image, the model assigns it to one or more predefined categories. Early approaches relied on hand-crafted features like SIFT (Scale-Invariant Feature Transform) and HOG (Histogram of Oriented Gradients), which were fed into traditional classifiers such as SVMs. The breakthrough came in 2012 when AlexNet, a deep convolutional neural network (CNN), dramatically outperformed all competitors on the ImageNet challenge, reducing the top-5 error rate from ~26% to ~16%.

Modern classification architectures have evolved significantly since AlexNet. ResNet introduced skip connections that allow training of networks with hundreds of layers by addressing the vanishing gradient problem. EfficientNet uses neural architecture search to find optimal scaling strategies across depth, width, and resolution. Vision Transformers (ViT) apply the transformer architecture from NLP to image patches, achieving state-of-the-art results by treating an image as a sequence of 16×16 patches. As a PM, you should understand that deeper or larger models generally improve accuracy but increase inference latency and cost—a critical trade-off for production systems.

Transfer learning is what makes modern computer vision practical for most product teams. Instead of training from scratch on millions of images, you fine-tune a model pre-trained on ImageNet (or similar large datasets) on your specific task. This dramatically reduces the amount of labeled data needed—often from millions of images to just a few thousand. Foundation models like CLIP (Contrastive Language-Image Pre-training) go even further, learning joint representations of images and text that enable zero-shot classification, where the model can classify images into categories it has never been explicitly trained on.

For product managers, the key decisions around image classification involve understanding the trade-offs between accuracy, latency, and cost. A model running on-device (edge deployment) needs to be small and fast, which is why architectures like MobileNet and EfficientNet-Lite exist. A cloud-based model can be larger and more accurate but introduces network latency and data privacy concerns. You should also understand the importance of the training data distribution matching your production data—a classifier trained on professional photos will perform poorly on user-generated smartphone images.

Object detection goes beyond classification by not only identifying what is in an image but also where it is, drawing bounding boxes around each detected object. This is essential for applications like autonomous driving (detecting pedestrians, vehicles, signs), retail (shelf monitoring, checkout-free stores), and security (surveillance, threat detection). The two main families of detectors differ fundamentally in their approach: two-stage detectors prioritize accuracy, while single-stage detectors prioritize speed.

Two-stage detectors like the R-CNN family (R-CNN → Fast R-CNN → Faster R-CNN) work in two phases. First, a Region Proposal Network (RPN) suggests candidate bounding boxes that likely contain objects. Then, each proposal is classified and its bounding box is refined. Faster R-CNN unified this into an end-to-end trainable network, achieving strong accuracy but at relatively high computational cost. These models typically run at 5-15 frames per second, making them suitable for offline analysis but challenging for real-time applications.

Single-stage detectors like YOLO (You Only Look Once) and SSD (Single Shot MultiBox Detector) process the entire image in a single pass, dividing it into a grid and predicting bounding boxes and class probabilities simultaneously. YOLOv8 and its successors achieve impressive accuracy while running at 30-100+ FPS, making them ideal for real-time applications. The key insight is that YOLO frames detection as a regression problem rather than a classification problem, which is why it's so fast. As a PM, you should understand that the YOLO family offers configurable model sizes (nano, small, medium, large, extra-large) that let you trade accuracy for speed.

More recent architectures like DETR (Detection Transformer) apply transformers to object detection, eliminating the need for hand-designed components like anchor boxes and non-maximum suppression. DETR treats detection as a set prediction problem, using a transformer encoder-decoder architecture with learned object queries. While initially slower to train, transformer-based detectors are becoming increasingly competitive and offer simpler, more elegant architectures. For product decisions, the choice between detector families depends on whether you need real-time processing (YOLO), maximum accuracy for offline analysis (Faster R-CNN), or architectural simplicity with good accuracy (DETR).

Image segmentation assigns a label to every pixel in an image, providing much finer-grained understanding than bounding boxes. There are three main types: semantic segmentation labels every pixel with a class (e.g., road, sidewalk, building) but doesn't distinguish between instances of the same class; instance segmentation (e.g., Mask R-CNN) separates individual instances, so two overlapping cars get different masks; and panoptic segmentation combines both, providing a complete scene understanding. Segmentation is critical for autonomous vehicles (understanding drivable surfaces), medical imaging (tumor boundary detection), and augmented reality (separating foreground from background).

The dominant architecture for segmentation is the U-Net and its variants, originally developed for biomedical image segmentation. U-Net uses an encoder-decoder structure with skip connections that preserve fine spatial detail. The encoder downsamples the image to capture context, while the decoder upsamples to produce pixel-level predictions. Skip connections pass high-resolution features from the encoder directly to the decoder, which is crucial for precise boundary delineation. DeepLab models introduced atrous (dilated) convolutions and the Atrous Spatial Pyramid Pooling module to capture multi-scale context without losing resolution. More recently, the Segment Anything Model (SAM) from Meta represents a foundation model approach to segmentation, trained on over 1 billion masks and capable of segmenting any object given a point, box, or text prompt.

Optical Character Recognition (OCR) extracts text from images and is one of the most commercially mature computer vision applications. Modern OCR systems typically use a two-stage pipeline: text detection (finding where text appears in the image) followed by text recognition (reading the characters). Deep learning models like CRNN (Convolutional Recurrent Neural Network) combine CNNs for feature extraction with RNNs for sequence modeling, handling variable-length text. Cloud APIs from Google (Vision API), AWS (Textract), and Azure (Computer Vision) provide production-ready OCR that handles documents, handwriting, and scene text. For structured documents like invoices or forms, document AI models go beyond raw OCR to extract key-value pairs and understand document layout.

As a PM building products with segmentation or OCR, you should know that these tasks are generally more data-hungry and computationally expensive than classification. Segmentation requires pixel-level annotations, which can cost 10-50x more than bounding boxes to create. OCR accuracy varies dramatically with image quality, font style, language, and layout complexity—always test with your actual production data. For OCR specifically, the distinction between printed text (95%+ accuracy achievable) and handwriting recognition (much harder, highly variable) is important to set stakeholder expectations correctly.

Generative models create new images rather than analyzing existing ones, and they've become one of the most transformative areas in AI. Generative Adversarial Networks (GANs), introduced by Ian Goodfellow in 2014, consist of two networks trained in competition: a generator that creates fake images and a discriminator that tries to distinguish real from fake. Through this adversarial process, the generator learns to produce increasingly realistic images. Notable GAN variants include StyleGAN (high-resolution face generation with controllable attributes), Pix2Pix (paired image-to-image translation), and CycleGAN (unpaired style transfer, like turning horses into zebras).

GANs were dominant from 2014-2021 but are notoriously difficult to train. Common issues include mode collapse (the generator produces only a few types of outputs), training instability (the generator and discriminator can become unbalanced), and the need for careful hyperparameter tuning. Techniques like progressive growing (starting at low resolution and gradually increasing), spectral normalization, and Wasserstein distance help stabilize training, but GANs remain finicky compared to newer alternatives.

Diffusion models (DDPM, Stable Diffusion, DALL-E, Midjourney) have largely superseded GANs for image generation since 2022. They work by learning to reverse a gradual noising process: during training, noise is progressively added to images, and the model learns to denoise at each step. During generation, the model starts from pure noise and iteratively denoises to produce an image. Stable Diffusion operates in a compressed latent space rather than pixel space, making it much more computationally efficient. Text-to-image generation combines diffusion models with text encoders (like CLIP) to generate images from natural language descriptions. The quality, controllability, and diversity of diffusion models have made them the foundation of modern generative AI for images.

For product managers, generative models open up applications in content creation (marketing assets, game art), design tools (prototyping, style transfer), data augmentation (generating synthetic training data), and personalization (try-on experiences, avatar creation). Key considerations include: computational cost (diffusion models require significant GPU resources, especially for high-resolution output), content safety (preventing generation of harmful or inappropriate content), IP and copyright (training data provenance and output ownership are legally unsettled), and quality control (generative outputs require human review or automated quality filters for production use).

Computer vision has transformative applications across industries. Autonomous vehicles use a fusion of cameras, LiDAR, and radar with models performing detection, segmentation, depth estimation, and tracking in real-time. Medical imaging applies CV to X-rays, CT scans, MRIs, and pathology slides for tasks like tumor detection, disease classification, and surgical planning—FDA-approved AI systems now assist radiologists in breast cancer screening, diabetic retinopathy detection, and more. Manufacturing uses CV for quality inspection, defect detection, and robotic guidance, often achieving inspection rates and accuracy that exceed human capability. Agriculture employs CV for crop monitoring, disease detection, and yield estimation using drone and satellite imagery.

Building CV systems requires high-quality, representative datasets. Key public datasets include ImageNet (14M images, 21K categories—the benchmark that launched deep learning), COCO (330K images with detection, segmentation, and captioning annotations), Open Images (9M images with rich annotations), and domain-specific datasets like CheXpert (chest X-rays) and Cityscapes (urban driving). For production systems, you'll almost always need custom datasets. Critical considerations include: ensuring your data represents the diversity of real-world conditions (lighting, angles, quality), balancing class distributions (rare defects are hard to collect), and establishing rigorous annotation guidelines with quality checks. Data flywheel strategies—where production usage generates data that improves the model—are essential for long-term competitive advantage.

Evaluation metrics for CV vary by task. For classification: accuracy, precision, recall, F1-score, and the confusion matrix. For object detection: mAP (mean Average Precision) is the standard metric, calculated by computing precision-recall curves for each class and averaging. mAP@0.5 uses a 50% IoU threshold, while mAP@[0.5:0.95] averages across multiple thresholds for a stricter evaluation. IoU (Intersection over Union) measures how well predicted bounding boxes overlap with ground truth—an IoU ≥ 0.5 is typically considered a correct detection. For segmentation: pixel accuracy, mean IoU (mIoU), and the Dice coefficient are standard. Understanding these metrics is crucial for setting quality bars and making go/no-go decisions.

As a PM, you should also track operational metrics beyond model accuracy: inference latency (p50, p95, p99), throughput (images per second), model size (memory footprint), and cost per inference. A model with 95% accuracy that runs in 50ms may be more valuable than one with 98% accuracy that takes 2 seconds. You should establish minimum viable model thresholds—the accuracy below which the product isn't useful—and understand the relationship between dataset size, labeling cost, and expected accuracy improvements. The Pareto frontier of accuracy vs. cost often reveals that the last few percentage points of accuracy require disproportionate investment.