Machine Learning Fundamentals

Machine learning is a subfield of artificial intelligence that gives computers the ability to learn patterns from data without being explicitly programmed for every scenario. Rather than writing rules by hand, ML engineers provide algorithms with large datasets and let the system discover statistical regularities on its own. The result is a model — a mathematical function that maps inputs to outputs based on what it learned during training.

As a PM, you don't need to implement these algorithms, but you need to understand what problems ML can and cannot solve. ML excels when you have large amounts of historical data, the problem has a clear objective function, and patterns in the data are stable over time. It struggles when data is scarce, the task requires causal reasoning, or the environment shifts rapidly (known as distribution shift).

A useful mental model is the distinction between traditional software and ML-powered software. Traditional software follows deterministic rules: if X, then Y. ML software is probabilistic: given X, predict Y with some confidence. This fundamentally changes how you design products, measure quality, set user expectations, and handle failure modes. Every ML-powered feature should have a graceful degradation path for when the model is wrong.

The three major paradigms of machine learning differ in the type of feedback the algorithm receives during training.

Supervised learning is the most common paradigm. The algorithm is given labeled examples — input-output pairs — and learns to predict the output for new, unseen inputs. Classification (e.g., spam vs. not spam) and regression (e.g., predicting house prices) are the two main types. Supervised learning requires high-quality labeled data, which is often the most expensive and time-consuming part of any ML project. As a PM, you'll need to plan for data labeling pipelines, quality assurance, and label consistency.

Unsupervised learning works with unlabeled data. The algorithm finds hidden structure — clusters, patterns, or compressed representations — without explicit guidance. Common techniques include k-means clustering, principal component analysis (PCA), and autoencoders. Product applications include customer segmentation, anomaly detection, and recommendation systems. The challenge is that results can be harder to evaluate since there's no ground truth to compare against.

Reinforcement learning (RL) trains agents to take sequential actions in an environment to maximize a cumulative reward signal. The agent learns through trial and error, balancing exploration (trying new actions) and exploitation (using known good actions). RL powers game-playing AI, robotics, and recommendation systems where user interaction is sequential. It's powerful but notoriously difficult to apply in production due to reward specification challenges and sample inefficiency.

You don't need to derive the math, but you should understand the intuition behind widely used algorithms and when each is appropriate.

Linear and logistic regression are the simplest supervised models. Linear regression predicts a continuous number (e.g., revenue), while logistic regression predicts a probability for classification. They're fast, interpretable, and serve as strong baselines. When someone says "start simple," they often mean start here. If a linear model performs well enough, the added complexity of a neural network may not be justified.

Decision trees and ensemble methods split data through a series of if-then rules, making them highly interpretable. Random forests and gradient-boosted trees (XGBoost, LightGBM) combine hundreds of trees to reduce variance and improve accuracy. These ensembles are the workhorses of tabular data problems and consistently win Kaggle competitions on structured datasets. They also handle missing values and mixed feature types gracefully.

Neural networks are composed of layers of interconnected neurons with learnable weights. They excel at unstructured data — images, text, audio — where manual feature engineering is impractical. However, they require large datasets, significant compute, and careful tuning. Deep learning (neural networks with many layers) has driven most of the AI breakthroughs in the last decade, from image recognition to language generation.

Support vector machines (SVMs) and k-nearest neighbors (KNN) are also worth knowing. SVMs find optimal decision boundaries and work well in high-dimensional spaces. KNN classifies based on similarity to nearby training examples. Both are useful for smaller datasets or as components of larger systems.

The bias-variance tradeoff is one of the most important concepts in ML. Bias measures how far off a model's average predictions are from the true values — high bias means the model is too simple and misses patterns (underfitting). Variance measures how much predictions change across different training sets — high variance means the model memorizes noise in the training data (overfitting).

Overfitting occurs when a model performs extremely well on training data but poorly on new, unseen data. The model has essentially memorized the training set rather than learning generalizable patterns. Signs include a large gap between training accuracy and validation accuracy. Common remedies include regularization (L1/L2), dropout (for neural networks), early stopping, data augmentation, and simply collecting more training data.

Underfitting is the opposite: the model is too simple to capture the underlying pattern. Training and validation performance are both poor. Solutions include using a more complex model, adding more features, reducing regularization, or training longer.

As a PM, the bias-variance tradeoff informs critical product decisions. An overfitting model gives a false sense of accuracy during development but disappoints users in production. An underfitting model never reaches acceptable quality. You should always ask your ML team: "How does performance compare between training and validation sets?" and "What does the learning curve look like?" These questions reveal whether the team needs more data, a better model, or better features.

How you split your data fundamentally affects whether you can trust your model's reported performance. The standard approach divides data into three sets: training (60-80%), validation (10-20%), and test (10-20%). The model learns from the training set, hyperparameters are tuned on the validation set, and the test set is used only once for a final, unbiased performance estimate.

A critical mistake is data leakage — when information from the test or validation set inadvertently influences training. This inflates performance metrics and leads to nasty surprises in production. Common leakage sources include normalizing data before splitting, using future data to predict the past (temporal leakage), and including features that are proxies for the label. As a PM, always ask: "Is there any chance our evaluation is optimistic due to leakage?"

Cross-validation is a more robust evaluation technique, especially when data is limited. In k-fold cross-validation, the data is split into k equal parts. The model is trained k times, each time holding out a different fold as validation. The average performance across folds gives a more reliable estimate than a single train-validation split. Common values are k=5 or k=10.

For time-series data, standard random splitting is invalid because it leaks future information. Instead, use time-based splits where training data always precedes validation/test data chronologically. This mimics real-world deployment where you can only predict the future, not the past. As a PM building products with temporal data (forecasting, recommendations, fraud detection), insisting on proper temporal validation is one of the highest-impact things you can do.