Technical Examination of Bias and Fairness in Machine-Learning Models: Groups, Metrics and Limitations

AI & Machine Learning

A technical fairness analysis does not begin with the statement that a model is fair or unfair. The examination question, group definitions, data state, reference outcomes, model outputs, metric and comparison criterion must first be fixed so that any measured difference has a traceable technical meaning.

1. Concepts and examination context

The term bias has several meanings

In a technical context, bias can describe systematic statistical distortion. In broader AI discussions, however, the term is also used for organisational, social and human influences. A technical report should therefore not state that “bias exists” without identifying what was actually measured or observed.

NIST Special Publication 1270 distinguishes systemic, statistical and human bias and explicitly treats AI bias as a socio-technical issue rather than reducing it to model parameters, training data or one fairness metric. Statistical bias can be relevant while still representing only one part of the broader issue.

A fairness question needs a concrete decision context

Before metrics are selected, the examination should define the decision or prediction process, target population, possible effect or harm and relevant group attributes. A metric without that context can be calculated correctly while answering the wrong question.

Group definitions must be technically reproducible. The source of group attributes, handling of missing values, category combinations and any exclusions should be documented because those choices directly affect the measured subgroup results.

2. Group comparison and technical measurement basis

Overall metrics can hide differences between groups. A model can have a strong overall accuracy while producing materially different false-positive or false-negative rates in defined subgroups. An aggregate score therefore does not automatically answer a group-specific examination question.

Group metrics should use the same technical definition

When groups are compared, the same metric definition and calculation procedure should be applied to each subgroup. Fairlearn's MetricFrame provides one technical structure for evaluating metrics overall and by group, including differences, ratios and bootstrap-based confidence intervals.

Selection rate measures the share of positive predictions

Selection rate is the proportion of cases for which the model produces the designated positive prediction. It can be useful for demographic-parity-style comparisons, but equal selection rates do not establish that all groups experience equal error patterns or equal outcomes in a broader sense.

True-positive and false-positive rates require suitable reference labels. If those labels are inaccurate, incomplete or conceptually unsuitable, the group metrics inherit that limitation. The label-generating process is therefore itself part of the evidence basis.

Differences can be expressed as absolute differences or ratios. The chosen representation should be documented and interpreted together with group counts; an apparently large percentage difference in a very small subgroup can carry substantial statistical uncertainty.

3. Fairness criteria and metric selection

The choice of fairness metric is part of the examination criterion

Demographic parity, equalized odds and equal opportunity answer different statistical questions. Hardt, Price and Srebro formulate equalized odds and equal opportunity for supervised classification. Satisfying one criterion only establishes the relationship defined by that criterion; it does not prove the absence of disadvantage, identify causes or provide a legal assessment.

The metric should therefore be derived from the concrete examination question. If the concern is unequal positive selection, selection rates may be relevant. If the concern is different error treatment conditional on reference outcomes, true-positive and false-positive rates can become more relevant. No single metric is universally “the fairness metric.”

Construct validity limits fairness metrics

Fairlearn emphasizes the importance of construct validity: a technically precise metric can still be a weak representation of the underlying real-world concept. The examination should ask whether the chosen observable variables, labels and metrics actually represent the effect or harm that the analysis is intended to address.

4. Uncertainty, subgroups and aggregation

Uncertainty should remain visible in group metrics

Group metrics are estimates based on finite samples. Fairlearn supports bootstrap confidence intervals for MetricFrame, allowing estimates to be viewed together with uncertainty. Small groups can otherwise create apparently precise differences that are poorly supported by the available data.

Multiple group attributes create more granular subgroups

Combining group attributes can reveal differences hidden by broader aggregation. At the same time, the number of cases per combination can become small or zero. A subgroup that is absent from the examined data cannot yield a supportable empirical group metric from that data.

Aggregation choices should therefore be documented. Results for broad groups and intersectional subgroups can answer different questions. With many groups and metrics, the number of statistical comparisons also increases and should be considered when interpreting isolated differences.

5. Causes, limitations and separation from legal assessment

Fairness metrics are not a causal analysis

A measured difference establishes that a defined metric differs between groups. It does not by itself establish why the difference exists. Possible contributors include data composition, reference labels, model behavior, thresholds, selection mechanisms or the wider application context.

A group difference therefore does not automatically prove a model defect. Conversely, the absence of a difference on one selected metric does not establish the absence of all harms, other group effects or other relevant fairness concerns.

Representativeness and fairness remain different areas of examination. Representativeness concerns population, sampling, coverage and distributions; fairness analysis concerns defined group differences in outputs, performance or effects. Bias is broader still and should not be reduced to any one group comparison.

Technical fairness measurement and legal discrimination analysis must remain separate

A technical examination can document groups, selection rates, true-positive and false-positive rates, other performance metrics and uncertainty. Whether a measured difference is legally impermissible or normatively unjust requires a separate standard. NIST and Fairlearn provide technical and socio-technical frameworks; they do not replace binding law or case-specific legal assessment.

6. Documentation and technical synthesis

Practical documentation structure for a fairness analysis

• Technical question and examined model state.
• Data and test-data state.
• Defined effect or harm being investigated.
• Group attributes and reproducible group definitions.
• Number of cases per group and relevant subgroups.
• Reference labels and known label limitations.
• Model outputs and thresholds.
• Selection rates and group-specific performance metrics.
• True-positive and false-positive rates where applicable.
• Selected fairness criterion and justification.
• Differences or ratios and uncertainty estimates.
• Aggregation level, subgroup structure and missing combinations.
• Data and method limitations.
• Technical interpretation and explicit limitation.

Further reading

• Assessing Machine-Learning Model Robustness
• Representativeness of Training and Test Data
• Validating a Machine-Learning Model
• Machine-Learning Model as a Subject of Examination

Data Science & AI Assessments

Technical examination of data quality, model validity, reproducibility, bias, robustness and documented AI-system behaviour.

Explore this assessment service