Review scope and evidence labels
This page profiles AI Red Teaming as a methodology for structured adversarial testing and abuse-case validation of AI-enabled systems.
Evidence labels used on this page:
- Community security knowledge base — based on public MITRE ATLAS or similar adversary-knowledge material.
- Community security guidance — based on public security community material such as OWASP.
- Official guidance — based on public official guidance such as NIST material.
- Author analysis — interpretation for AI security, governance, architecture and operational risk on this site.
- Implementation evidence required — local records that must exist before a mitigation can be treated as operating.
This entry is a Research methodology profile, not a certification claim. It does not imply offensive AI red-team certification; it defines the governance and architecture questions that structured AI red-team work should answer.
Methodology snapshot
| Field | Value |
|---|---|
| Category | AI adversarial testing and control validation |
| Research type | Methodology |
| Core question | Can realistic AI abuse scenarios bypass intended controls or create unacceptable harm? |
| Main scope | LLM applications, RAG, agents, tools, data access, model behavior, prompts, outputs, logs and response playbooks |
| Useful outputs | Scenario library, findings, reproduction evidence, control gaps, residual risk and retest records |
| Companion methods | MITRE ATLAS, AI Threat Modeling, Prompt Injection, OWASP LLM Top 10, Audit Evidence |
| Review status | Public guidance reviewed; local work requires explicit scope, authorization and safety boundaries |
What AI Red Teaming means
AI red teaming tests whether an AI-enabled system can be manipulated, misused or made to fail in ways that matter to the organisation. Evidence: Community security knowledge base; Official guidance; Author analysis.
It should be connected to threat modeling. Random prompts are not enough; tests should map to assets, actors, abuse cases, controls and evidence. Evidence: Author analysis.
For this site, AI Red Teaming is positioned as architecture and governance validation: the point is to improve control design and risk decisions, not to claim offensive credentials. Evidence: Author analysis.
What it is not
AI Red Teaming is not:
- a certification by itself,
- unrestricted attacks against public systems,
- only jailbreak prompt lists,
- only model benchmark testing,
- a substitute for secure design,
- proof that a system is safe forever,
- permission to test without written scope.
Practical AI red-team work needs authorization, test boundaries, safety rules, evidence handling and remediation follow-up. Evidence: Author analysis; Implementation evidence required.
Practical testing areas
| Area | Practical question | Evidence to retain |
|---|---|---|
| Prompt injection | Can untrusted instructions override policy or leak data? | Test cases, prompts, results, screenshots/logs |
| RAG abuse | Can retrieved content poison context or expose restricted data? | Source samples, query traces, permission tests |
| Tool misuse | Can an agent call tools outside intended scope? | Tool-call logs, approvals, negative tests |
| Data leakage | Can sensitive context appear in answers or logs? | Redaction tests, output review, log policy |
| Model misuse | Can the system assist prohibited or harmful workflows? | Safety tests, refusal analysis, policy records |
| Monitoring | Would defenders see the abuse path? | Telemetry, alerts, incident drill results |
| Remediation | Were fixes retested and accepted? | Fix records, retest evidence, residual-risk decision |
Evidence: Community security guidance; Author analysis.
Relationship to existing Research entries
AI Red Teaming connects to:
- MITRE ATLAS — provides adversary vocabulary and technique framing.
- AI Threat Modeling — defines the scenarios that red-team tests should validate.
- Prompt Injection — common LLM abuse path requiring specific tests.
- RAG Security — retrieval systems need abuse tests for poisoning and leakage.
- AI Security Architecture — red-team findings should improve architecture decisions.
- OWASP Top 10 for LLM Applications — risk categories can become test families.
- AI Governance — results must feed ownership, risk acceptance and remediation.
- Audit Evidence — claims require reproducible evidence and retest records.
Governance relevance
AI red-team work is governance-relevant because it turns abstract AI risk into concrete scenarios, findings and decisions. It helps leadership see whether controls operate in practice and whether residual risk is acceptable. Evidence: Author analysis; Implementation evidence required.
Useful governance outputs include approved scope, scenario catalog, finding register, evidence pack, remediation plan, retest record and accepted-risk statement. Evidence: Author analysis.
Current limitations and follow-up research
This page is a methodology profile. Real work requires explicit permission, system access, safety constraints, evidence handling, legal scope and remediation ownership.
Follow-up Research nodes should cover:
- AI Abuse Case Library,
- Agent Security Testing,
- RAG Red Teaming,
- AI Incident Response,
- AI evaluation governance.