Trust & Transparency
Weight: 1.2×Confidence calibration, source attribution, capability disclosure, decision explanation
8 criteria
← Back to Work
AI Agent Evaluation Tool
A systematic framework for evaluating AI agents across trust, usability, accessibility, and compliance dimensions — combining NIST RMF, EU AI Act, Microsoft HAX, Nielsen Heuristics, and WCAG into one actionable toolkit.
Dimensions
6
Criteria
40
Frameworks
12+
Trust & Transparency
Usability
Error Recovery
Accessibility
Safety
Autonomy
The Challenge
How do you evaluate AI when the rules keep changing?
Traditional UX evaluation methods weren't designed for systems that learn, produce probabilistic outputs, and operate autonomously. When an AI agent's behavior isn't deterministic, how do you test for clarity? When the system adapts over time, how do you audit trust?
Existing frameworks address pieces of the puzzle — Microsoft HAX covers human-AI interaction, Nielsen heuristics assess usability, WCAG ensures accessibility — but none provide a unified evaluation approach for modern AI agents.
Framework Methodology
This framework synthesizes 12+ established guidelines into 6 evaluation dimensions, each with weighted criteria and severity levels for prioritized recommendations.
Source Frameworks
AI-Specific
- NIST AI Risk Management Framework
- EU AI Act Compliance
- Microsoft HAX Guidelines (18 principles)
- Google PAIR Guidelines
- Anthropic Constitutional AI
- IBM AI Ethics
Foundational UX
- Nielsen's 10 Usability Heuristics
- Google Conversational Design
- Apple Human Interface Guidelines
- WCAG 2.1 (AA level)
- WAI-ARIA Best Practices
- Stanford HAI Guidelines
6 Evaluation Dimensions
Usability & Learnability
Weight: 1.0×System status, recognition over recall, consistency, conversational turn-taking
7 criteria
Error Recovery
Weight: 1.1×Error clarity, repair strategies, human escalation, graceful degradation
6 criteria
Accessibility
Weight: 1.0×Screen reader support, keyboard navigation, cognitive load, multi-modal I/O
7 criteria
Safety & Compliance
Weight: 1.3×Risk classification, harm prevention, data privacy, audit trails, bias mitigation
6 criteria
Autonomy & Control
Weight: 1.2×Human-in-the-loop, override mechanisms, scope boundaries, feedback loops
6 criteria
Sample Evaluation: Dust.tt
Dust
Enterprise AI agent platform for building custom agents that connect to company data.
Evaluation Tool
Use this interactive tool to evaluate any AI agent. Your progress is saved automatically.
Evaluation Report
Need help evaluating your AI products?
I help teams audit AI experiences using structured frameworks, turning complex requirements into actionable design improvements.