CLI Commands

Complete reference for all orcha CLI commands with examples, options, and usage patterns.

eval

Run an evaluation from a TypeScript file.

Usage

orcha eval <file> [options]

Arguments

file - Path to the evaluation file (.ts or .eval.ts)

Options

--jsonl         # Output results in JSONL format
--verbose       # Enable verbose logging
--resume        # Resume from a previous batch operation

Examples

Basic Evaluation

# Run a single evaluation file
orcha eval ./evals/my-eval.ts

JSONL Output

# Output in JSONL format for CI/CD pipelines
orcha eval ./evals/my-eval.ts --jsonl

# Redirect to file
orcha eval ./evals/my-eval.ts --jsonl > results.jsonl

Verbose Mode

# Show detailed execution logs
orcha eval ./evals/my-eval.ts --verbose

# Also works with environment variable
ORCHA_VERBOSE=true orcha eval ./evals/my-eval.ts

Resume Batch Processing

# Resume an interrupted batch evaluation
orcha eval ./evals/batch-eval.ts --resume

Output Format

Terminal UI (Default)

Evaluation: My Eval
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
Progress: [████████████████████] 100% | 50/50 test cases

Scorer Results:
┌──────────────────┬──────┬──────┬──────┬──────┐
│ Scorer           │ Mean │ P50  │ Min  │ Max  │
├──────────────────┼──────┼──────┼──────┼──────┤
│ Effectiveness    │ 0.87 │ 0.90 │ 0.20 │ 1.00 │
│ Factuality       │ 0.92 │ 1.00 │ 0.50 │ 1.00 │
└──────────────────┴──────┴──────┴──────┴──────┘

Total Duration: 12.4s

JSONL Format

{"type":"start","name":"My Eval","timestamp":"2024-01-15T10:30:00Z"}
{"type":"result","idx":0,"input":"...","output":"...","scores":{"Effectiveness":0.95}}
{"type":"result","idx":1,"input":"...","output":"...","scores":{"Effectiveness":0.80}}
{"type":"summary","mean":0.87,"median":0.90,"stddev":0.12,"scores":{"Effectiveness":{"mean":0.87}}}

Evaluation File Format

The evaluation file should export an Eval() call:

// evals/my-eval.ts
import { Eval, Effectiveness } from 'orchestrated'

await Eval("My Eval", {
  data: [
    { input: "Question 1", output: "Answer 1" },
    { input: "Question 2", output: "Answer 2" },
  ],
  scores: [Effectiveness],
})

Exit Codes

0 - Success
1 - Evaluation error or failure
2 - Configuration error (missing credentials, invalid config)

upload

Serialize a project file and upload the bundle to S3 for backend execution.

Usage

orcha upload <file> [options]

Arguments

file - Path to the project file (typically project.ts or evals/project.ts)

Options

--verbose       # Enable verbose logging
--dry-run       # Generate files without uploading to S3

Examples

Basic Upload

# Upload project with custom scorers
orcha upload ./evals/project.ts

Dry Run

# Generate definitions.json without uploading
orcha upload ./evals/project.ts --dry-run

Verbose Upload

# Show detailed serialization and upload logs
orcha upload ./evals/project.ts --verbose

What Gets Uploaded

The upload command performs the following:

Serialization - Analyzes your project file and extracts:
- Custom scorer definitions
- Task function definitions
- Data source configurations
- Handler functions
Bundle Creation - Creates a JavaScript bundle containing all handler functions:
- Bundles with Bun for optimal size
- Includes all dependencies
- Sets __ORCHESTRATED_HANDLERS_ONLY__ flag
S3 Upload - Uploads the bundle to S3:
- Path: s3://{bucket}/{tenantId}/{serviceName}/{fingerprint}/handlers.bundle.js
- Fingerprint based on content hash for caching
Definitions Output - Generates .orchestrated/definitions.json:
- JSON representation of all definitions
- References to uploaded bundles
- Schema information for validation

Output

Uploading project: ./evals/project.ts
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━

✓ Serialized 3 custom scorers
✓ Generated handler bundle (fingerprint: a1b2c3d4)
✓ Uploaded to s3://orchestrated-bundles/acme/prod/a1b2c3d4/handlers.bundle.js
✓ Written definitions to .orchestrated/definitions.json

Upload complete!

Project File Format

The project file should define custom scorers and evaluations:

// evals/project.ts
import { createCustomScorer, Eval } from 'orchestrated'
import { z } from 'zod'

export const MyScorer = createCustomScorer({
  name: "MyScorer",
  schema: z.object({
    input: z.string(),
    output: z.string(),
  }),
  handler: async (args) => ({
    name: "MyScorer",
    score: args.output.length > 10 ? 1 : 0,
    metadata: { length: args.output.length },
  }),
})

// Evaluations using the custom scorer
await Eval("My Eval", {
  data: [/* ... */],
  scores: [MyScorer],
})

Requirements

Configuration - Must set s3Bucket in Orchestrated.yaml or ORCHESTRATED_S3_BUCKET env var
AWS Credentials - Must set AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY
Permissions - S3 bucket must allow uploads from your credentials

Authenticate with the Orchestrated API Gateway using OAuth2 device flow.

Usage

orcha login [options]

Options

--verbose       # Show detailed authentication flow

Examples

Basic Login

orcha login

# Output:
# Visit: https://console.getorchestrated.com/activate
# Enter code: ABCD-1234
# Waiting for authentication...
# ✓ Authenticated as user@example.com
# Credentials saved to ~/.orcha/credentials.json

Verbose Login

orcha login --verbose

# Shows:
# - Device code request
# - Token polling
# - JWT parsing
# - Credential storage

Authentication Flow

Device Code Request - CLI requests a device code from the authorization server
User Prompt - CLI displays activation URL and code
Browser Authentication - User visits URL and enters code
Token Polling - CLI polls for token approval
Credential Storage - Stores access token and refresh token in ~/.orcha/credentials.json

Credential Storage

Credentials are stored securely at ~/.orcha/credentials.json:

{
  "accessToken": "eyJhbGciOiJSUzI1NiIsInR5cCI6IkpXVCJ9...",
  "refreshToken": "eyJhbGciOiJSUzI1NiIsInR5cCI6IkpXVCJ9...",
  "expiresAt": "2024-01-15T11:30:00Z",
  "userId": "abc123",
  "email": "user@example.com"
}

Token Expiration

Access tokens expire after 1 hour. The CLI automatically:

Checks expiration before each command
Prompts to re-authenticate if expired
Uses refresh token to obtain new access token (when supported)

logout

Clear stored credentials and sign out from the Orchestrated platform.

Usage

orcha logout

Examples

Basic Logout

orcha logout

# Output:
# ✓ Credentials cleared
# Run 'orcha login' to authenticate again

What Gets Cleared

Deletes ~/.orcha/credentials.json
Removes access token
Removes refresh token
Clears user information

Re-authentication

After logout, run orcha login to authenticate again:

orcha logout
orcha login

whoami

Display current user information and configuration state.

Usage

orcha whoami [options]

Options

--verbose       # Show detailed configuration

Examples

Basic Usage

orcha whoami

# Output:
# User: user@example.com (ID: abc123)
# Token: Valid (expires in 45 minutes)
#
# Configuration:
#   Tenant ID: acme
#   Service Name: production
#   Environment: production
#   API URL: https://api.getorchestrated.com

Verbose Output

orcha whoami --verbose

# Additional output:
#   S3 Bucket: orchestrated-bundles
#   S3 Region: us-east-1
#   App URL: https://console.getorchestrated.com
#   Config Source: Orchestrated.yaml

Output Fields

User Information

Email - User email from JWT token
User ID - Cognito user identifier
Token Status - Valid, expired, or missing
Expiration - Time until token expires

Configuration

Tenant ID - Current tenant identifier
Service Name - Current service name
Environment - Current environment (development, staging, production)
API URL - API Gateway endpoint

Configuration Sources

In-code parameters - Highest priority
Environment variables - ORCHESTRATED_* vars
Orchestrated.yaml - Config file in project root
Defaults - Fallback values

Exit Codes

0 - Valid credentials and configuration
1 - Missing or invalid credentials
2 - Configuration error

list

Find and list all evaluation files (.eval.ts) in a directory.

Usage

orcha list [directory] [options]

Arguments

directory - Directory to search (default: current directory)

Options

--recursive, -r  # Search subdirectories recursively
--verbose        # Show additional file information

Examples

List in Current Directory

orcha list

# Output:
# Found 3 evaluation files:
#   ./my-eval.eval.ts
#   ./simple.eval.ts
#   ./batch.eval.ts

Recursive Search

orcha list ./evals --recursive

# Output:
# Found 8 evaluation files:
#   ./evals/basic.eval.ts
#   ./evals/advanced/batch.eval.ts
#   ./evals/advanced/custom.eval.ts
#   ./evals/experiments/test1.eval.ts
#   ...

Verbose Output

orcha list --verbose

# Output:
# Found 3 evaluation files:
#   ./my-eval.eval.ts (2.4 KB, modified 2 hours ago)
#   ./simple.eval.ts (1.1 KB, modified 1 day ago)
#   ./batch.eval.ts (5.2 KB, modified 3 days ago)

File Patterns

The list command finds files matching:

*.eval.ts - Standard evaluation files
*-eval.ts - Alternative naming convention

Use Cases

Run All Evaluations

# List and run each evaluation
for file in $(orcha list --recursive); do
  orcha eval "$file"
done

CI/CD Pipeline

# Find and run all evaluations in CI
orcha list ./evals --recursive | while read file; do
  orcha eval "$file" --jsonl >> results.jsonl
done

Next: CLI Configuration CLI Overview