Features Relative Advantages

The Features Relative Advantages focus on testing of models for relative advantages of features comparing against different groups.

Model feature selection decisions need to be informed by an understanding of the positive or negative impact of each feature on the legally protected groups who are impacted by the contribution of each feature to the outcome of the model as a whole.

It is important to calculate the favourability of each feature (positive or negative) for each protected class as compared against the favourability of that feature for the general population.

If a feature is more favourable for a protected class than it is for the general population, then the inclusion of that feature in the feature space provides members of that protected class with an advantage relative to the general population.

This test is called “feature relative advantage”.

Why are Feature Relative Advantage tests important?

Features relative advantage tests are critical for assessing potential biases and disparities in the model’s predictions.

By quantifying how each feature influences outcomes across various protected characteristics categories such as race and gender, it allows for the identification of advantages or disadvantages experienced by members of a specific protected class.

This analysis is essential for evaluating the model’s fairness and equity, ensuring that feature selection does not inadvertently favor one gender or race over others or perpetuate discrimination.

Addressing these disparities is crucial for building models that provide unbiased, ethical, and inclusive outcomes, aligning with societal values and regulatory guidelines, and promoting trust in products that leverage these models.

How does Fairly perform Feature Relative Advantage tests?

The Feature Relative Advantage test quantifies the differential impact of model features on various protected classes relative to the general population.

This test is critical for identifying potential biases and ensuring that model decisions uphold fairness and equity standards.

Mathematical Formulation

For a given dataset (D) with features (X_1, X_2, \ldots, X_n) and target outcome (Y), the dataset is segmented by protected classes (C = {c_1, c_2, \ldots, c_k}) such as race, gender, or age. The correlation between each feature (X_i) and the target (Y) within each class (c_j) is computed to measure how changes in (X_i) affect (Y).

The correlation coefficient ( \rho_{X_i, Y}^{c_j} ) for each feature (X_i) within each class (c_j) is defined as: [ \rho_{X_i, Y}^{c_j} = \frac{\text{Cov}(X_i, Y | C = c_j)}{\sigma_{X_i | C = c_j} \sigma_{Y | C = c_j}} ] where (\text{Cov}) denotes the covariance, and (\sigma) denotes the standard deviation of (X_i) and (Y), conditioned on the class (C = c_j).

To determine the relative advantage of each feature for each class, the correlation coefficient for the class in question is compared to the average of the coefficients for that feature across all other classes. The relative advantage (A_{X_i}^{c_j}) is computed as: [ A_{X_i}^{c_j} = \rho_{X_i, Y}^{c_j} - \frac{1}{k - 1} \sum_{c_m \in C, c_m \neq c_j} \rho_{X_i, Y}^{c_m} ]

Interpretation and Reporting

A positive value of (A_{X_i}^{c_j}) indicates that feature (X_i) is more favorable to class (c_j) compared to the average effect of (X_i) across other classes, suggesting a potential advantage for (c_j) in the model decision process. Conversely, a negative value suggests a disadvantage.

Results from this analysis are compiled into a report detailing the relative advantages and disadvantages for each feature across all protected classes.

This comprehensive evaluation aids in assessing compliance with fair lending laws and ethical AI guidelines, ensuring that no protected class is unduly favored or prejudiced by the model features.