CLI Configuration

Configure the orcha CLI using configuration files, environment variables, and managed credentials.

Config File

The Orchestrated.yaml file provides centralized configuration for your project.

File Location

Create Orchestrated.yaml in your project root:

your-project/
├── Orchestrated.yaml
├── evals/
│   └── my-eval.ts
└── package.json

Basic Configuration

# Orchestrated.yaml
# Tenant identifier (REQUIRED)
tenantId: your-tenant-id

# Service name (REQUIRED)
serviceName: your-service-name

# Environment (optional, default: development)
environment: production

# API Gateway endpoint (optional)
apiUrl: https://api.getorchestrated.com

# S3 configuration for uploads (REQUIRED for orcha upload)
s3Bucket: your-bucket-name
s3Region: us-east-1

Complete Example

# Orchestrated.yaml
# Complete configuration with all options

# API Configuration
apiUrl: https://api.getorchestrated.com
appUrl: https://console.getorchestrated.com

# Tenant and Service
tenantId: acme
serviceName: production

# Environment
environment: production

# S3 Configuration (for orcha upload)
s3Bucket: orchestrated-bundles
s3Region: us-east-1
# s3Endpoint: https://custom-s3.example.com  # Optional, for S3-compatible services

Configuration Priority

Configuration values are resolved in this order (highest to lowest priority):

In-code parameters - Passed directly to functions
Environment variables - ORCHESTRATED_* prefixed vars
Orchestrated.yaml - Configuration file
Defaults - Framework defaults

Example:

# Orchestrated.yaml
tenantId: from-file
serviceName: from-file

# Environment variable overrides config file
export ORCHESTRATED_TENANT_ID=from-env
orcha eval my-eval.ts
# Uses: tenantId=from-env, serviceName=from-file

Required vs Optional

Required for All Operations

tenantId - Tenant identifier
serviceName - Service name

Required for Upload

s3Bucket - S3 bucket name for bundle storage
s3Region - S3 region (default: us-east-1)

Optional

apiUrl - API endpoint (default: https://api.getorchestrated.com)
appUrl - Console URL (default: https://console.getorchestrated.com)
environment - Environment name (default: development)
s3Endpoint - Custom S3 endpoint

Multiple Environments

Use different config files for different environments:

# Development
cp Orchestrated.dev.yaml Orchestrated.yaml
orcha eval my-eval.ts

# Production
cp Orchestrated.prod.yaml Orchestrated.yaml
orcha eval my-eval.ts

Or use environment-specific configs:

# Orchestrated.yaml
tenantId: acme
serviceName: ${SERVICE_NAME}  # From environment
environment: ${ENVIRONMENT}   # From environment

Environment Variables

Configure the CLI using environment variables prefixed with ORCHESTRATED_.

State Configuration

# API Configuration
export ORCHESTRATED_API_URL=https://api.getorchestrated.com
export ORCHESTRATED_APP_URL=https://console.getorchestrated.com
export ORCHESTRATED_APP_CLIENT_ID=5of2tqergm9nt3n22a5dmc67u5

# Tenant and Service (REQUIRED)
export ORCHESTRATED_TENANT_ID=your-tenant-id
export ORCHESTRATED_SERVICE_NAME=your-service-name

# Environment
export ORCHESTRATED_ENVIRONMENT=production

# S3 Configuration (for orcha upload)
export ORCHESTRATED_S3_BUCKET=your-bucket-name
export ORCHESTRATED_S3_REGION=us-east-1
export ORCHESTRATED_S3_ENDPOINT=https://custom-s3.example.com  # Optional

Authentication

# Access token (loaded from credentials if not set)
export ORCHESTRATED_ACCESS_TOKEN=eyJhbGciOiJSUzI1NiIsInR5cCI6IkpXVCJ9...

Note: Prefer using orcha login instead of manually setting ORCHESTRATED_ACCESS_TOKEN.

AWS Credentials

Required for orcha upload to work:

export AWS_ACCESS_KEY_ID=AKIAIOSFODNN7EXAMPLE
export AWS_SECRET_ACCESS_KEY=wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY

CLI Behavior

# Enable verbose logging
export ORCHA_VERBOSE=true

# Use JSONL output format
export ORCHA_JSONL=true

OpenAI API Key

Required for some built-in scorers (Factuality, AnswerRelevancy):

export OPENAI_API_KEY=sk-...

Complete Example

# .env.production
# Production environment configuration

# State
ORCHESTRATED_API_URL=https://api.getorchestrated.com
ORCHESTRATED_TENANT_ID=acme
ORCHESTRATED_SERVICE_NAME=production
ORCHESTRATED_ENVIRONMENT=production

# S3
ORCHESTRATED_S3_BUCKET=orchestrated-bundles
ORCHESTRATED_S3_REGION=us-east-1

# AWS
AWS_ACCESS_KEY_ID=AKIAIOSFODNN7EXAMPLE
AWS_SECRET_ACCESS_KEY=wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY

# OpenAI
OPENAI_API_KEY=sk-...

Load with:

# Load environment variables
source .env.production

# Run evaluation
orcha eval my-eval.ts

Credentials

The CLI manages authentication credentials automatically using the orcha login flow.

Credential Storage

Credentials are stored at ~/.orcha/credentials.json:

{
  "accessToken": "eyJhbGciOiJSUzI1NiIsInR5cCI6IkpXVCJ9...",
  "refreshToken": "eyJhbGciOiJSUzI1NiIsInR5cCI6IkpXVCJ9...",
  "expiresAt": "2024-01-15T11:30:00Z",
  "userId": "abc123",
  "email": "user@example.com"
}

Authentication Flow

Login - Run orcha login to authenticate
Device Code - CLI displays activation URL and code
Browser Authentication - User visits URL and enters code
Token Storage - CLI stores access token and refresh token
Automatic Loading - CLI loads credentials for subsequent commands

Token Lifecycle

Access Token

Lifetime: 1 hour
Usage: Authentication for API requests
Renewal: Automatically refreshed using refresh token

Refresh Token

Lifetime: 30 days
Usage: Obtain new access tokens
Renewal: Re-authenticate with orcha login after expiration

Checking Credentials

# View current user and token status
orcha whoami

# Output:
# User: user@example.com (ID: abc123)
# Token: Valid (expires in 45 minutes)

Credential Management

Login

orcha login
# Stores credentials to ~/.orcha/credentials.json

Logout

orcha logout
# Removes ~/.orcha/credentials.json

Re-authenticate

# When token expires
orcha login

# Or set token manually (not recommended)
export ORCHESTRATED_ACCESS_TOKEN=eyJhbGciOiJSUzI1NiIsInR5cCI6IkpXVCJ9...

Manual Token Management

For CI/CD pipelines, you can set the token manually:

# Set token as environment variable
export ORCHESTRATED_ACCESS_TOKEN=your-token-here
orcha eval my-eval.ts

Note: Tokens obtained from orcha login should not be committed to version control.

Token Permissions

Access tokens include the following claims:

User ID - Cognito user identifier
Email - User email address
Tenant Access - List of accessible tenants
Expiration - Token expiry timestamp

Security Best Practices

Never commit credentials - Add ~/.orcha/credentials.json to .gitignore
Use short-lived tokens - Access tokens expire after 1 hour
Rotate tokens regularly - Re-authenticate periodically
Use environment-specific credentials - Different tokens for dev/staging/prod
Secure CI/CD tokens - Use secret management for pipeline tokens

Troubleshooting

Common configuration issues and solutions.

Authentication Errors

Missing Credentials

Error: No credentials found. Run 'orcha login' to authenticate.

Solution:

orcha login

Expired Token

Error: Access token expired. Please re-authenticate.

Solution:

orcha login

Invalid Token

Error: Invalid access token.

Solution:

orcha logout
orcha login

Configuration Errors

Missing Tenant ID

Error: Missing required configuration: tenantId

Solution:

# Set in Orchestrated.yaml
echo "tenantId: your-tenant-id" > Orchestrated.yaml

# Or use environment variable
export ORCHESTRATED_TENANT_ID=your-tenant-id

Missing Service Name

Error: Missing required configuration: serviceName

Solution:

# Set in Orchestrated.yaml
echo "serviceName: your-service-name" >> Orchestrated.yaml

# Or use environment variable
export ORCHESTRATED_SERVICE_NAME=your-service-name

Missing S3 Configuration

Error: Missing S3 bucket configuration for upload

Solution:

# Set in Orchestrated.yaml
echo "s3Bucket: your-bucket-name" >> Orchestrated.yaml
echo "s3Region: us-east-1" >> Orchestrated.yaml

# Or use environment variables
export ORCHESTRATED_S3_BUCKET=your-bucket-name
export ORCHESTRATED_S3_REGION=us-east-1

AWS Errors

Missing AWS Credentials

Error: Missing AWS credentials for S3 upload

Solution:

export AWS_ACCESS_KEY_ID=your-access-key
export AWS_SECRET_ACCESS_KEY=your-secret-key

S3 Permission Denied

Error: Access Denied: s3://your-bucket/path/to/bundle.js

Solution: Verify your AWS credentials have write permissions to the S3 bucket.

File Errors

Config File Not Found

Warning: Orchestrated.yaml not found, using defaults

Solution: Create Orchestrated.yaml in your project root or set environment variables.

Evaluation File Not Found

Error: Cannot find file: ./evals/my-eval.ts

Solution:

# Use correct path
orcha eval ./evals/my-eval.ts

# Or absolute path
orcha eval /Users/you/projects/evals/my-eval.ts

Debugging

Enable Verbose Logging

orcha eval my-eval.ts --verbose
# Or
ORCHA_VERBOSE=true orcha eval my-eval.ts

Check Configuration

orcha whoami --verbose
# Shows all configuration values and sources

Verify Credentials

orcha whoami
# Shows user info and token status

Next: Eval() API CLI Commands