Pydantic AI Testing & Evaluation

AgentCI provides first-class Pydantic AI testing and evaluation capabilities for type-safe agent applications. Our automated Pydantic AI CI platform discovers all your Agent instances and validates them against comprehensive evaluation suites - leveraging your existing type safety without any code modifications.

Pydantic AI agent testing that works with your existing codebase

AgentCI automatically discovers and evaluates Pydantic AI agents, including:

Agent discovery: Agent() instances with model and system_prompt configurations
Evaluation types: Accuracy, safety, and performance testing with type validation
CI/CD integration: Automated testing on pull requests via GitHub
Zero code changes: No additional decorators or modifications to your Python code required

Supported Agent Patterns

`Agent` Class

from pydantic_ai import Agent

agent = Agent(
    model='claude-3-haiku-20240307',
    system_prompt="""You are a customer service representative.

Your responsibilities:
- Address customer concerns promptly
- Escalate complex issues when needed
- Maintain a friendly and helpful tone"""
)

Supported Tool Patterns

Plain Functions

from typing import Optional
from datetime import datetime

def get_current_time() -> str:
    """Get the current time as a string."""
    return datetime.now().isoformat()

def calculate_metrics(values: list[float], metric_type: str = "mean") -> float:
    """Calculate various metrics from a list of values."""
    if not values:
        return 0.0

    if metric_type == "mean":
        return sum(values) / len(values)
    elif metric_type == "max":
        return max(values)
    return 0.0

What Gets Auto-Discovered

AgentCI automatically finds:

Agent() instances with model and system_prompt parameters
Plain Python functions with docstrings and type annotations
Async functions used as tools

No decorators or special tool classes required - Pydantic AI's type-safe approach works seamlessly with AgentCI's auto-discovery.