Asenion Test Agents for Predictive AI
Asenion Test Agents for Predictive AI enable testing of predictive AI models against policies and controls.
Risk and Compliance testing is often regarded as a secondary task and typically occurs when compliance reports are necessary. This task can be both time-consuming and tedious.
Fairly’s aim is to expedite and standardize this process in order to make it more efficient.
Within the Fairly platform, a policy consists of various control bundles, each composed of a set of controls.
Controls can be either qualitative which requires users to respond to specific questions or quantitative controls which necessitate the execution of model tests to satisfy the control’s criteria.
The Fairly Asenion framework supports the integration of appropriate test agents into a uniform testing system, as a result of the policy or policies that outlines the quantitative controls necessary for the project.
These agents conduct the necessary tests, compile the results and upload to the Fairly platform for further analysis.
What tests can Asenion Test Agents perform?
BISG and BIFSG
BISG stands for Bayesian Improved Surname Geocoding. It is a statistical method used to infer the likely racial or ethnic composition of a geographic area based on the surnames of its residents. BIFSG stands for Bayesian Improved First Name Surname Geocoding.
Data Drift
Data drift is the significant change in data distribution, compared to the data used to train the model. Data drift can be caused by the evolution of business processes or industry events which create discontinuities in the underlying phenomena. It does not affect the formatting of the data, but the data at its core and what it represents.
Feature Importance Analysis
Feature importance analysis is a technique in data analysis and machine learning that assesses the significance of each input variable (feature) in influencing the output or target variable in a predictive model.
Features Relative Advantages
The Features Relative Advantages focus on the testing of models for relative advantages of features comparing against different groups.
Features Stability
Feature Stability analysis calculates the Population Stability Index (PSI). PSI is a commonly used metric to assess the stability of the population distribution across different datasets or time periods. It quantifies the degree of change in the distribution of a variable between a reference population (often the training dataset) and a comparison population (e.g., validation dataset or production data).
ISO/IEC TR 24027
ISO/IEC TR 24027 is a Technical Report for Information technology — Artificial intelligence (AI) — Bias in AI systems and AI aided decision making.
Data Integrity
Data integrity analysis is a critical process that involves meticulously examining and enhancing the quality, accuracy, consistency, and reliability of data within a dataset or database.
Model Impact on Business Processes
Model Impact Analysis on Business Processes refers to the practice of analyzing probablistic models and machine learning models. The analysis is done in a way that ensures equitable and unbiased treatment of different groups or individuals. It involves addressing and mitigating potential sources of bias in the data, algorithms, and model development process to avoid discriminatory outcomes.
Challenger Models
A model risk assessment policy to perform effective challenge using challenger models. Developing a “challenger model” is another approach to measuring potential model risk. With this technique, a new model is developed, sometimes on the basis of an alternative model theory or different data set, for comparison to the “champion model” (i.e., the primary model being used).
Privacy - PII Detection
The purpose of PII leakage detection is to determine if models are using PII in training datasets and if models (LLMs) are returning any PII data.
Fair Lending Testing
Fairly performs fair lending testing using a combination of techniques and methodologies.
Performance Testing
Fairly performs performance testing using a combination of techniques and methodologies.
New Asenion Test Agents can be created to add additional tests against custom policies and controls.
How does Asenion Test Agents perform these tests?
Asenion Test Agents can run with three options:
- In Jupyter notebook as a Python client library
- As a command line script that can be integrated into the CI/CD pipeline
- As an application with a Web UI for configuration