FairLens: Benchmarking Fairness in Vision-Language Models for High-Stakes Decision-Making

Vision-language models (VLMs) are increasingly used to make decisions from visual inputs. We introduce FairLens, a benchmark and evaluation framework for measuring both the fairness and the validity of VLM responses in three high-stakes domains: hiring, legal, and healthcare. FairLens pairs real face images spanning gender, race, and age groups with closed- and open-ended questions, giving more than 100K image-question pairs per model, and evaluates responses from four complementary views: demographic parity over adverse outcome rates, soundness, demographic association over unsupported roles and statuses, and bias in free-text generation. Soundness is the central validity criterion: a response is sound when it follows the evidence stated in the question and abstains when the image cannot support an answer.

Evaluating eight VLMs, we find that the primary failure is unwarranted inference rather than unequal treatment. Models routinely infer qualifications, threat, illness, or professional role from a face instead of abstaining, and the weakest model does so on 99% of the questions its input cannot answer. These failures are most severe in legal and healthcare, where recognizing insufficient evidence matters most, and disparity metrics alone would miss them: parity gaps are small in absolute terms, yet when baseline adverse rates are low the same gap means one demographic group receives adverse labels several times as often as another, and a small gap can equally reflect a model that treats every group unsafely. Bias in free-text responses is only loosely coupled to multiple-choice accuracy, so correct structured answers do not imply safe generation. FairLens shows that fair high-stakes VLM behavior requires similar treatment across groups and refusal to infer high-stakes attributes from appearance, and its question suite transfers to any face corpus with demographic annotations.

Unlike standard VQA, FairLens is built around questions for which a face often should not provide sufficient evidence. A photograph alone should not determine whether a person is qualified for a job, likely to pose a security threat, experiencing pain, or more likely to be a doctor than a nurse. Some questions provide task-relevant context (for example, that a candidate meets all qualifications); those test whether models follow stated evidence rather than overriding it with appearance-based assumptions.

The evaluation set has 1,505 adult UTKFace images (ages 25-65), 10% of the filtered dataset, selected with stratified sampling and a fixed random seed. Each image is paired with 69 questions (38 closed-ended, 31 open-ended) across hiring, legal, and healthcare, for 103,845 image-question pairs per model.

High-stakes VLM fairness cannot be reduced to equal outcome rates. Demographic parity gaps remain small (about 1-3 pp), yet that is not evidence of similar risk: when baseline adverse rates are low, a small gap can still be a large rate ratio, and a model can treat groups similarly while labeling everyone unsafely. Unsupported inference is the dominant failure. Many FairLens questions are underdetermined by a face: appearance alone does not justify claims about qualifications, threat, criminality, pain, urgency, or professional role. Abstention is a first-class evaluation target. Soundness and open-ended bias also come apart, so correct multiple-choice behavior does not imply safe unconstrained generation.

The core contribution is the question suite rather than the image source: the benchmark transfers to any image set with demographic annotations. Across eight VLMs, similar treatment across groups and refusal to infer high-stakes attributes from a face prove to be separate requirements, and current models satisfy the first far more often than the second.