GitHub Integration

Agent CI integrates natively with GitHub to provide automated evaluation and monitoring for your agent applications. Through GitHub App installation, Agent CI monitors your Git repositories and automatically runs evals on pull requests without requiring manual configuration.

What is GitHub Integration?

GitHub integration connects your Git repositories to Agent CI's evaluation platform. When you install the Agent CI GitHub App, it gains access to monitor repository events, read your codebase, and post evaluation results directly to pull requests.

Git is the version control system that tracks changes to your code. GitHub is the cloud platform that hosts Git repositories and provides collaboration features like pull requests, code review, and webhooks.

Agent CI uses both:

  • Git for version control, tracking prompt changes, and identifying evaluation contexts
  • GitHub for repository hosting, pull request automation, and workflow integration

How GitHub Integration Works

GitHub App Installation

Agent CI uses a GitHub App rather than personal access tokens or OAuth apps. This provides:

  • Fine-grained permissions - Only access what's necessary for evaluations
  • Organization-level control - Admins control which repositories Agent CI can access
  • Webhook automation - Real-time notifications when pull requests are opened or updated
  • Status checks - Block PR merges if critical evals fail
  • PR comments - Automated evaluation results posted directly to pull requests

Installation Process

  1. Login with GitHub - Authenticate using your GitHub account
  2. Install GitHub App - Choose which repositories to grant access
  3. Select repositories - Grant access to all repos or specific ones
  4. Webhook configuration - GitHub automatically configures webhooks for Agent CI
  5. Repository monitoring - Agent CI begins monitoring for pull request events

Only GitHub organization owners can install GitHub Apps. Individual users with repository access cannot complete installation without owner permissions.

Git Version Control Integration

Agent CI leverages Git's version control capabilities to track agent evolution:

Git Commit Tracking

Every evaluation run is associated with a Git commit hash, enabling:

  • Version attribution - Know exactly which code version produced which results
  • Cross-environment comparison - Compare performance of same commit across branches
  • Performance history - Track eval performance over Git commit history
  • Automatic versioning - No manual version management required

Git Branch Environments

Your Git branch structure defines deployment environments:

  • main branch - Production environment
  • staging branch - Pre-production testing
  • Feature branches - Development and experimentation

Agent CI automatically detects which branch code runs on and labels evaluation results accordingly.

Git-Based Prompt Versioning

Agent prompts are tracked through Git commits:

  • No separate interface for prompt management
  • Standard Git diff shows exactly what changed
  • Git blame shows who changed prompts and when
  • Git history provides complete prompt evolution timeline

Repository Structure

Agent CI expects a specific repository structure to function correctly:

Required Directory Structure

your-repository/
├── .agentci/
│   └── evals/
│       ├── accuracy-test.toml
│       ├── safety-test.toml
│       └── performance-test.toml
├── your_agent/
│   ├── __init__.py
│   ├── agent.py
│   └── tools.py
└── requirements.txt

Configuration Storage in Git

All evaluation configurations are stored in Git:

  • TOML files in .agentci/evals/ define test cases
  • Version controlled alongside application code
  • Branch-specific configurations for different environments
  • Pull request reviews include eval configuration changes

This Git-native approach ensures eval configurations evolve with your agent code.

Pull Request Workflow

The GitHub integration automates evaluation on every pull request:

Automated PR Evaluation

  1. Developer opens PR - Creates pull request with agent changes
  2. GitHub webhook fires - Notifies Agent CI of new PR
  3. Agent CI runs evals - Executes all configured evaluations
  4. Results posted - Automated comment added to PR with pass/fail status
  5. Status check updated - GitHub status check reflects eval results
  6. Merge decision - Team reviews results and merges if evals pass

Agent CI posts structured comments to pull requests showing evaluation results, pass/fail status for each test case, and links to detailed dashboard views.

GitHub Status Checks

Agent CI provides GitHub status checks that can be configured as required checks:

  • Passing status - All critical evals passed, PR can merge
  • Failing status - One or more evals failed, merge blocked
  • Pending status - Evals still running
  • Error status - Evaluation infrastructure issue

Configure branch protection rules to require Agent CI status checks before merging.

Repository Permissions

The Agent CI GitHub App requests specific permissions:

Required Permissions

  • Contents: Read - Access repository code and eval configurations
  • Pull requests: Read & Write - Monitor PRs and post evaluation comments
  • Webhooks: Read - Receive notifications about repository events
  • Checks: Write - Update GitHub status checks with eval results

Repository Access Control

Organization owners control repository access:

  • All repositories - Grant access to entire organization
  • Select repositories - Choose specific repositories for Agent CI
  • Revoke access - Remove Agent CI access at any time through GitHub settings

Agent CI only accesses repositories explicitly granted during installation.

GitHub Webhooks

Agent CI uses GitHub webhooks to receive real-time notifications:

Webhook Events

  • Pull request opened - Triggers initial evaluation run
  • Pull request updated - Re-runs evals with new commits
  • Pull request merged - Updates production baseline metrics
  • Push to branch - Updates environment-specific data

GitHub manages webhook configuration automatically when you install the Agent CI GitHub App.

Common GitHub Integration Questions

What is GitHub integration? GitHub integration connects your GitHub repositories to Agent CI through a GitHub App, enabling automated evaluation on pull requests.

Do I need a GitHub account? Yes - Agent CI requires GitHub for authentication and repository access. Git repositories hosted on other platforms (GitLab, Bitbucket) are not currently supported.

Can I use Agent CI with private repositories? Yes - Agent CI works with both public and private GitHub repositories. The GitHub App requires appropriate permissions to access private repos.

What GitHub permissions does Agent CI need? Agent CI needs read access to repository contents, read/write access to pull requests, and write access to status checks. See the Repository Permissions section above for details.

How do I revoke Agent CI access? Go to your GitHub organization settings, navigate to "Installed GitHub Apps," find Agent CI, and click "Uninstall" or adjust repository access.

Getting Started with GitHub Integration

  1. Go to agent-ci.com and login with GitHub
  2. Install the Agent CI GitHub App on your organization
  3. Select repositories to grant access
  4. Create .agentci/evals/ directory in your repository
  5. Add eval configuration files defining test cases
  6. Open a pull request to see automated evaluation in action

Next Steps