Quick Start Guide

Get up and running with Agent CI in under 5 minutes. This guide will walk you through connecting your GitHub repository and setting up your first automated evaluation.

Prerequisites

Before you begin, ensure you have:

  • A GitHub repository with your agent code using one of the Supported Frameworks
  • Organization owner permissions (required for GitHub App installation)
  • Git configured on your system

Step 1: Connect Your GitHub Repository

Install Agent CI GitHub App

  1. Go to agent-ci.com and click "Login with GitHub"
  2. Authorize Agent CI to verify your GitHub identity (you'll see a GitHub authorization page)
  3. Install the Agent CI GitHub App:
    • You'll be redirected to GitHub's app installation page
    • Choose your organization from the dropdown (only organization owners can install apps)
    • Select repository access:
      • Choose "All repositories" for organization-wide access
      • Or choose "Only select repositories" and pick specific repos
    • Click "Install" to complete the GitHub App installation

Note: Only organization owners can install GitHub Apps. If you're not an owner, ask an organization owner to complete this step.

Set Up Agent CI

  1. Create your organization in Agent CI (you'll be redirected back after GitHub installation)
  2. Click "Create New Application" and select your repository from the dropdown
  3. Confirm permissions - Agent CI will show you exactly what access it has to your selected repositories

That's it. Agent CI is now monitoring your repository for pull requests and can run evaluations.

Step 2: Write Your First Test

Create a folder called .agentci in your repository root and add this file:

.agentci/basic-test.toml

[eval]
description = "Basic agent test"
type = "accuracy"
targets.agents = ["*"]

[[eval.cases]]
prompt = "What is 2 + 2?"
output = "4"

[[eval.cases]]
prompt = "Hello!"
output = "{{*}}"  # Matches any response

Commit and push this file. That's your first agent evaluation!

Step 3: See It Work

Open a pull request with your new .agentci/basic-test.toml file. Within seconds, you'll see:

✅ Agent CI: Evaluation complete
   └── ✅ basic-test (2/2 test cases passed)

That's it! You now have automated agent testing on every pull request.

Next Steps

Need Help?