CLI Overview

The orcha CLI is a powerful command-line tool for running evaluations, managing authentication, and uploading custom scorers to the Orchestrated platform.

Introduction

The Orchestrated CLI (orcha) provides a streamlined interface for working with the evaluation framework. It handles authentication, state management, and execution of your evaluation files.

Key Features

  • Single-file executable - Standalone binary built with Bun, no dependencies required
  • Interactive terminal UI - Progress bars and live updates powered by Ink
  • Automatic state management - Handles configuration and credentials seamlessly
  • Multiple output formats - Terminal UI or JSONL for CI/CD pipelines
  • Credential management - Secure authentication flow with token storage

Platform Support

The CLI is available for:

  • macOS - Intel and Apple Silicon (arm64)
  • Linux - x64 and arm64
  • Windows - x64 (orcha.exe)

Installation

Install the orcha CLI using the provided build scripts or download pre-built binaries.

From Source

Build and install from the project repository:

# Clone the repository
git clone https://github.com/your-org/eval-processor.git
cd eval-processor

# Install dependencies
pnpm install

# Build the CLI
pnpm run cli:build

# Install globally (requires sudo on macOS/Linux)
pnpm run cli:install

This installs orcha to /usr/local/bin/orcha (macOS/Linux) or equivalent on Windows.

Pre-built Binaries

Download platform-specific binaries from the releases page:

# macOS
curl -L https://github.com/your-org/eval-processor/releases/latest/download/orcha-macos -o orcha
chmod +x orcha
sudo mv orcha /usr/local/bin/

# Linux
curl -L https://github.com/your-org/eval-processor/releases/latest/download/orcha-linux -o orcha
chmod +x orcha
sudo mv orcha /usr/local/bin/

# Windows (PowerShell)
Invoke-WebRequest -Uri "https://github.com/your-org/eval-processor/releases/latest/download/orcha-windows.exe" -OutFile "orcha.exe"
# Move to a directory in your PATH

Verify Installation

Confirm the CLI is installed correctly:

orcha --version
# Output: orcha version 1.0.0

orcha --help
# Shows available commands and options

Global Options

Global options can be used with any command and control CLI behavior.

Available Options

--version          # Show version information
--help, -h         # Show help for a command
--verbose          # Enable verbose logging (also: ORCHA_VERBOSE=true)
--jsonl            # Output results in JSONL format (also: ORCHA_JSONL=true)

Environment Variables

The CLI respects several environment variables for configuration:

# CLI Behavior
ORCHA_VERBOSE=true       # Enable verbose logging
ORCHA_JSONL=true         # Use JSONL output format

# State Configuration (see Configuration guide)
ORCHESTRATED_API_URL     # API Gateway endpoint
ORCHESTRATED_TENANT_ID   # Tenant identifier
ORCHESTRATED_SERVICE_NAME # Service name
ORCHESTRATED_ENVIRONMENT # Environment (development, staging, production)
ORCHESTRATED_ACCESS_TOKEN # Cognito access token

Output Formats

Terminal UI (Default)

The default output uses Ink for rich terminal UI:

orcha eval my-eval.ts

# Output:
# Evaluation: My First Eval
# ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
# Progress: [████████████████████] 100% | 10/10 test cases
#
# Scorer Results:
# ┌──────────────────┬──────┬──────┬──────┬──────┐
# │ Scorer           │ Mean │ P50  │ Min  │ Max  │
# ├──────────────────┼──────┼──────┼──────┼──────┤
# │ Effectiveness    │ 0.95 │ 1.00 │ 0.80 │ 1.00 │
# └──────────────────┴──────┴──────┴──────┴──────┘

JSONL Output

Use --jsonl for machine-readable output (ideal for CI/CD):

orcha eval my-eval.ts --jsonl

# Output (one JSON object per line):
# {"type":"start","name":"My First Eval","timestamp":"2024-01-15T10:30:00Z"}
# {"type":"result","input":"...","output":"...","scores":{"Effectiveness":0.95}}
# {"type":"summary","mean":0.95,"median":1.0,"stddev":0.05}

Verbose Mode

Enable verbose logging to see detailed information:

# Using flag
orcha eval my-eval.ts --verbose

# Using environment variable
ORCHA_VERBOSE=true orcha eval my-eval.ts

# Output includes:
# - State initialization
# - Configuration loading
# - Data source queries
# - Scorer execution details
# - Bundle downloads (for uploaded evaluations)

Getting Help

The CLI includes built-in help for all commands.

General Help

orcha --help
orcha -h

# Output:
# orcha - Orchestrated evaluation CLI
#
# Usage: orcha <command> [options]
#
# Commands:
#   eval <file>      Run an evaluation from a file
#   upload <file>    Serialize and upload project to S3
#   login            Authenticate with API Gateway
#   logout           Clear stored credentials
#   whoami           Show current user and configuration
#   list [dir]       Find all .eval.ts files
#
# Options:
#   --version        Show version
#   --help, -h       Show help
#   --verbose        Enable verbose logging
#   --jsonl          Output in JSONL format

Command-Specific Help

Get help for individual commands:

orcha eval --help
# Shows usage, options, and examples for the eval command

orcha upload --help
# Shows usage, options, and examples for the upload command

orcha login --help
# Shows authentication instructions

Common Issues

Authentication Errors

# Error: No credentials found
orcha login

# Error: Token expired
orcha login  # Re-authenticate

Configuration Errors

# Error: Missing tenantId or serviceName
# Solution: Create Orchestrated.yaml or set environment variables
echo "tenantId: your-tenant-id" > Orchestrated.yaml
echo "serviceName: your-service-name" >> Orchestrated.yaml

File Not Found

# Error: Cannot find file
# Solution: Provide absolute or relative path
orcha eval ./evals/my-eval.ts
orcha eval /Users/you/projects/evals/my-eval.ts

Additional Resources

Next: CLI CommandsQuick Start Guide

Was this page helpful?