LlamaIndex Testing & Evaluation

AgentCI delivers automated LlamaIndex testing and evaluation for your RAG applications and agentic workflows. Our platform provides comprehensive LlamaIndex CI integration that automatically discovers FunctionAgent, ReActAgent, and all your custom tools - no code changes required.

LlamaIndex testing and evaluation for RAG applications

AgentCI automatically discovers and evaluates LlamaIndex agents, including:

  • Agent discovery: FunctionAgent() and ReActAgent.from_tools() instances
  • Evaluation types: Accuracy, safety, performance, and consistency testing
  • CI/CD integration: Automated testing on pull requests via GitHub
  • Zero code changes: No decorators, wrappers, or modifications to your Python code required

Supported Agent Patterns

FunctionAgent

from llama_index.core.agent.workflow import FunctionAgent
from llama_index.llms.openai import OpenAI

agent = FunctionAgent(
    tools=[multiply_tool, calculator_tool],
    llm=OpenAI(model="gpt-4o"),
    system_prompt="You are a mathematical assistant that helps with calculations."
)

ReActAgent

from llama_index.core.agent import ReActAgent
from llama_index.llms.openai import OpenAI

agent = ReActAgent.from_tools(
    tools=[weather_tool, get_time],
    llm=OpenAI(model="gpt-4o"),
    verbose=True
)

Supported Tool Patterns

FunctionTool.from_defaults

from llama_index.core.tools import FunctionTool
from typing import Annotated

def calculate(
    operation: Annotated[str, "The operation: add, subtract, multiply, divide"],
    x: Annotated[float, "The first number"],
    y: Annotated[float, "The second number"]
) -> float:
    """Perform basic arithmetic operations."""
    if operation == "add":
        return x + y
    elif operation == "multiply":
        return x * y
    return 0.0

calculator_tool = FunctionTool.from_defaults(
    calculate,
    name="calculator",
    description="Perform calculations"
)

Plain Functions

def multiply(a: float, b: float) -> float:
    """Useful for multiplying two numbers."""
    return a * b

# Use directly without wrapping
agent = FunctionAgent(
    tools=[multiply],
    llm=OpenAI(model="gpt-4o"),
    system_prompt="You are a math assistant."
)

What Gets Auto-Discovered

AgentCI automatically finds:

  • FunctionAgent() instances with llm, tools, and system_prompt parameters
  • ReActAgent.from_tools() calls
  • Tools created with FunctionTool.from_defaults()
  • Plain Python functions with docstrings used as tools

No configuration or code changes required.

Next Steps