AI Ethics, Bias, and Fairness

AI bias is not a single phenomenon — it manifests in multiple distinct forms throughout the machine learning pipeline, and understanding these distinctions is critical for an AI PM who needs to communicate risks to stakeholders and prioritize mitigation efforts.

Data bias is the most common source and arises from the training data itself. Historical bias reflects real-world inequities embedded in the data — for example, if past hiring data shows fewer women in engineering roles, a model trained on that data will perpetuate that pattern. Representation bias occurs when certain groups are underrepresented in the training set, leading to worse performance for those groups. Measurement bias happens when the features used as proxies don't measure the intended concept equally across groups — for example, using arrest records as a proxy for criminal behavior disproportionately affects over-policed communities.

Algorithmic bias emerges from the model architecture and training process itself. Even with perfectly balanced data, certain model choices can introduce bias. Aggregation bias occurs when a single model is used for groups with fundamentally different data-generating processes. Learning bias can emerge from the optimization objective — models trained to maximize overall accuracy will naturally perform better on majority groups at the expense of minorities.

Selection bias and deployment bias occur downstream. Selection bias means the data collected doesn't represent the actual population (e.g., surveying only smartphone users). Deployment bias occurs when a system is used in contexts different from its intended design — a model trained for one demographic or geography being applied to another. Feedback loops are particularly insidious: a biased model's predictions influence future data collection, reinforcing the original bias. Predictive policing is a textbook example — sending more police to flagged neighborhoods generates more arrests there, "confirming" the model's predictions.

One of the most important lessons in AI fairness is that there is no single definition of "fair" — and that different mathematical definitions of fairness are often mutually incompatible. This isn't just an academic curiosity; it has profound implications for how you design and evaluate AI products.

Demographic parity (also called statistical parity) requires that the positive prediction rate is equal across groups. For a hiring model, this means equal percentages of men and women are selected. Equal opportunity requires that the true positive rate (sensitivity/recall) is equal across groups — qualified candidates from all groups should have an equal chance of being correctly identified. Equalized odds extends this to require equal true positive and false positive rates across groups. Predictive parity requires that precision (the fraction of positive predictions that are correct) is equal across groups. Individual fairness requires that similar individuals receive similar predictions, regardless of group membership.

The impossibility theorem (demonstrated by Chouldechova, 2017, and Kleinberg et al., 2016) proves that except in trivial cases, you cannot simultaneously satisfy demographic parity, equal opportunity, and predictive parity when base rates differ between groups. This means fairness involves inherent tradeoffs, and the "right" definition depends on the context and values at stake.

For AI PMs, this means you cannot just "add fairness" as a feature request — you need to make deliberate choices about which fairness criteria matter most for your use case, and be transparent about the tradeoffs. A loan approval system might prioritize equal opportunity (ensuring qualified borrowers from all groups are approved), while a content recommendation system might prioritize demographic parity in representation. Document your fairness choices, the reasoning behind them, and the tradeoffs you accepted. This is both good practice and increasingly a regulatory requirement.

Bias detection and mitigation span three stages of the ML pipeline: pre-processing (fixing the data), in-processing (modifying the training), and post-processing (adjusting the outputs).

Pre-processing techniques address bias at the data level. Re-sampling adjusts the training data distribution to balance representation (oversampling underrepresented groups or undersampling overrepresented ones). Re-labeling corrects potentially biased labels in the training data. Feature transformation removes or modifies features that encode protected attributes, though this must be done carefully since many features serve as proxies for protected characteristics (e.g., zip code correlating with race). Tools like IBM's AI Fairness 360 and Google's What-If Tool provide automated bias detection across datasets.

In-processing techniques modify the model training itself. Adversarial debiasing adds a secondary objective that penalizes the model for making predictions from which a protected attribute can be inferred. Fairness constraints add mathematical constraints directly to the optimization objective (e.g., requiring equalized odds). Regularization approaches add penalty terms that discourage the model from relying on features correlated with protected attributes. These techniques often involve a fairness-accuracy tradeoff — the model may become slightly less accurate overall in exchange for more equitable performance across groups.

Post-processing techniques adjust model outputs after prediction. Threshold adjustment sets different decision thresholds for different groups to achieve a desired fairness metric (e.g., lowering the classification threshold for an underperforming group). Calibration ensures that predicted probabilities are accurate across groups. Reject option classification defers to human judgment for predictions near the decision boundary where bias is most likely. For AI PMs, the choice of technique depends on your constraints: do you have access to modify training data, the model itself, or only the deployment pipeline? Pre-processing is most accessible, post-processing is quickest to implement, and in-processing is most principled but requires ML engineering investment.

Real-world case studies of AI bias are essential knowledge for AI PMs, both as cautionary tales and as frameworks for thinking about your own products.

Hiring (Amazon, 2018): Amazon developed an AI recruiting tool trained on 10 years of historical resumes. Because the tech industry had been predominantly male, the system learned to penalize resumes containing the word "women's" (as in "women's chess club") and downgraded graduates of all-women's colleges. Amazon scrapped the tool, but the case illustrates how historical bias in training data directly translates to discriminatory outputs. The lesson: always ask what historical inequities are embedded in your training data.

Criminal justice (COMPAS): The COMPAS recidivism prediction tool, analyzed by ProPublica in 2016, was shown to have significantly different false positive rates across racial groups — Black defendants were nearly twice as likely as white defendants to be falsely flagged as high-risk. Northpointe (the vendor) countered that the tool had equal predictive parity across groups. Both claims were mathematically correct, illustrating the impossibility theorem in practice. The lesson: which fairness metric you choose fundamentally changes whether a system appears "fair."

Healthcare (Optum/UnitedHealth): A 2019 study published in Science found that a widely used healthcare algorithm systematically discriminated against Black patients. The algorithm used healthcare costs as a proxy for healthcare needs, but because Black patients historically had less access to healthcare (and therefore lower costs), the system concluded they were healthier and denied them care coordination services. At a given risk score, Black patients were significantly sicker than white patients. The lesson: proxy variables can encode systemic inequities in non-obvious ways.

Facial recognition: Multiple studies (Gender Shades by Joy Buolamwini, NIST's FRVT) have documented dramatically different error rates across demographics for facial recognition systems, with error rates for dark-skinned women being up to 34% higher than for light-skinned men. This has led to real-world harms in law enforcement (wrongful arrests) and access control (exclusion). Several cities have banned government use of facial recognition as a result.

AI auditing is the systematic evaluation of AI systems for bias, fairness, and compliance. Audits can be internal (conducted by your own team) or external (performed by independent third parties). External audits carry more credibility and are increasingly required by regulation — New York City's Local Law 144, for example, mandates annual bias audits for AI hiring tools.

A comprehensive AI audit includes: data audits (examining training data composition and representativeness), model audits (testing performance across demographic subgroups), process audits (reviewing development practices, documentation, and governance), and impact audits (assessing real-world effects on different communities). The audit should produce a written report with findings, risk ratings, and remediation recommendations. Frameworks like the Algorithmic Impact Assessment (AIA) used by the Canadian government provide structured templates for these evaluations.

Inclusive AI development means building diverse teams and engaging affected communities throughout the development process. Research consistently shows that homogeneous teams are more likely to have blind spots about potential biases. But diversity alone isn't sufficient — you need processes that surface diverse perspectives. Participatory design involves the communities who will be affected by the AI system in its design and evaluation. Value-sensitive design explicitly identifies and balances the values of different stakeholders. Community advisory boards provide ongoing input from affected populations.

For AI PMs, inclusive development is both an ethical imperative and a practical strategy for building better products. Diverse perspectives catch bias issues earlier (when they're cheaper to fix), improve product-market fit for underserved populations, and reduce regulatory and reputational risk. Concrete steps include: diversifying your data labeling workforce, conducting user research across demographic groups, including fairness criteria in your model evaluation rubrics, and establishing channels for affected communities to report bias issues post-deployment.