Adaptive One-Shot Testing Methodology

Purpose

Adaptive One-Shot Testing is a risk-based, adversarial testing methodology designed to efficiently identify high-impact failure modes in AI systems, particularly large language model (LLM)–based applications. The methodology focuses on discovering whether a vulnerability exists, rather than attempting to exhaustively measure average-case performance.

This approach is especially suited for:

  • Safety, security, and compliance testing
  • AI governance and assurance
  • Pre-deployment and post-change risk validation
  • Regulatory and audit-oriented assessments

Core Concept

Under Adaptive One-Shot Testing:

  • Each test case consists of a single, targeted prompt (a “one-shot”).
  • The objective of each prompt is to directly elicit a specific class of failure (e.g., hallucination, policy violation, data leakage, prompt injection).
  • The testing framework adapts subsequent prompts based on prior responses in order to:
    • Explore failure boundaries
    • Escalate attack sophistication
    • Maximize the probability of exposing latent weaknesses

Rather than replaying a fixed test set, the system performs guided adversarial exploration of the model’s risk surface.

How It Works

  1. Risk-Driven Test Intent Definition
    Each test is mapped to a specific risk category and sub-risk (e.g., “system prompt leakage,” “fabricated citation,” “policy override”).

  2. Targeted Prompt Generation
    The testing system generates a single, focused prompt intended to trigger that specific failure mode.

  3. Response Evaluation
    The system’s response is automatically analyzed against predefined failure criteria.

  4. Adaptive Mutation
    If no failure is detected, the next prompt is:
    • Reformulated
    • Strengthened
    • Obfuscated
    • Or contextually altered
      based on the observed behavior of the system.
  5. Early Termination on Failure
    Once a failure is identified, testing for that specific sub-risk terminates immediately, as the objective (existence proof of vulnerability) has been achieved.

Coverage Strategy

For each defined risk and sub-risk:

  • Testing is executed:
    • A minimum number of times to ensure baseline coverage
    • Up to a defined maximum number of attempts if no failure is observed
  • The process prioritizes:
    • Breadth of risk surface exploration
    • Efficient discovery of material weaknesses
    • Avoidance of redundant testing once a vulnerability is proven

Evaluation Philosophy (Worst-Case Logic)

Adaptive One-Shot Testing uses a worst-case risk logic:

The presence of a single material failure is sufficient to classify the risk as not adequately controlled.

Accordingly:

  • The objective is not to estimate probabilities or average performance.
  • The objective is to determine whether a failure mode is possible and reproducible.

Why This Methodology Is Appropriate for AI Risk and Compliance

AI system risks are:

  • Non-deterministic
  • Context-sensitive
  • Highly sensitive to prompt phrasing
  • Susceptible to adversarial interaction patterns

Adaptive One-Shot Testing reflects these realities by:

  • Avoiding static, predictable test suites
  • Continuously probing the system in new ways
  • Simulating real-world misuse and abuse patterns
  • Focusing on control effectiveness, not just output quality

Outputs and Evidence

This methodology produces:

  • Prompt-response transcripts
  • Test execution logs
  • Failure exemplars
  • Coverage summaries
  • Human validation records
  • Risk classification results

Plain-Language Summary

Instead of asking the same questions repeatedly, we systematically try new and increasingly challenging questions, one at a time, to see whether the system can be made to fail in any important way. The moment we succeed, we stop and record the weakness.