Uploading to Cloud

Upload your evaluations and custom scorers to the cloud platform for web-based management and API access.

Prerequisites

Before uploading, ensure you have the necessary configuration and credentials.

Authentication

orcha login

This will:

Open your browser for authentication
Save credentials to ~/.orcha/credentials.json
Validate your access token

Verify authentication:

orcha whoami

Configuration File

Create an Orchestrated.yaml file in your project root:

# Orchestrated.yaml
apiUrl: https://api.getorchestrated.com
tenantId: your-tenant-id
serviceName: your-service-name
environment: production

# S3 configuration (REQUIRED for upload)
s3Bucket: your-bucket-name
s3Region: us-east-1
# s3Endpoint: https://custom-s3.example.com  # Optional

Environment Variables

Alternatively, use environment variables:

export ORCHESTRATED_TENANT_ID=your-tenant-id
export ORCHESTRATED_SERVICE_NAME=your-service-name
export ORCHESTRATED_S3_BUCKET=your-bucket-name
export AWS_ACCESS_KEY_ID=your-access-key
export AWS_SECRET_ACCESS_KEY=your-secret-key

Required Fields

For uploading, you must configure:

tenantId - Your organization identifier
serviceName - The service you're evaluating
s3Bucket - S3 bucket for bundle storage
AWS credentials - For S3 upload permissions

Serialization

Orchestrated automatically serializes your evaluations and scorers for cloud deployment.

What Gets Serialized

Custom Scorers:

Scorer definitions (name, schema, metadata)
Handler functions (bundled separately)
Prompt templates for LLM-based scorers
Choice scores and validation rules

Data Sources:

Dataset configurations
Parameter schemas
Transform functions

Task Functions:

Task handler code
Dependencies and imports
Context requirements

Serialization Process

When you run orcha upload, the CLI:

Analyzes your code to identify scorers, tasks, and data sources
Extracts definitions into JSON format
Bundles handler functions using esbuild
Generates a fingerprint for versioning
Uploads bundle to S3
Creates definitions.json with metadata

Project File Structure

Organize your project for upload:

// project.ts
import { createCustomScorer, createTypedScorer, Eval } from 'orchestrated'
import { z } from 'zod'

// Define custom scorers
export const LengthValidator = createCustomScorer({
  name: "LengthValidator",
  schema: z.object({
    output: z.string(),
  }),
  handler: async (args) => ({
    name: "LengthValidator",
    score: args.output.length >= 10 && args.output.length <= 500 ? 1 : 0,
    metadata: { length: args.output.length },
  }),
})

export const ToneScorer = createTypedScorer({
  name: "ToneScorer",
  schema: z.object({
    input: z.string(),
    output: z.string(),
  }),
  promptTemplate: `
    Evaluate the tone of this response (1-5):
    Input: {{input}}
    Output: {{output}}
  `,
  choiceScores: {
    "1": 0.0,
    "2": 0.25,
    "3": 0.5,
    "4": 0.75,
    "5": 1.0,
  },
})

// Define evaluations
export async function MyEvaluation() {
  await Eval("My Evaluation", {
    data: [
      { input: "test 1", output: "result 1" },
      { input: "test 2", output: "result 2" },
    ],
    scores: [LengthValidator, ToneScorer],
  })
}

Inline Scorers

Inline scorers are automatically detected and bundled:

// This works too - scorer is automatically registered
await Eval("Inline Scorer Eval", {
  data: [/* ... */],
  scores: [
    createCustomScorer({
      name: "InlineScorer",
      schema: z.object({ output: z.string() }),
      handler: async (args) => ({
        name: "InlineScorer",
        score: args.output.length > 0 ? 1 : 0,
        metadata: {},
      }),
    }),
  ],
})

Upload Process

Upload your project to make it available in the cloud platform.

Basic Upload

orcha upload project.ts

Upload Output

You'll see progress and confirmation:

Analyzing project.ts...
✓ Found 2 scorers
✓ Found 1 evaluation

Serializing scorers...
✓ LengthValidator (custom_scorer)
✓ ToneScorer (typed_scorer)

Bundling handlers...
✓ Created handlers.bundle.js (12.4 KB)

Uploading to S3...
✓ Uploaded to s3://bucket/tenant/service/abc123def/handlers.bundle.js

Generating definitions...
✓ Created .orchestrated/definitions.json

Upload complete!
Fingerprint: abc123def456
Scorers: 2
Evaluations: 1

Generated Files

After upload, you'll find:

.orchestrated/
  definitions.json          # Serialized definitions
  abc123def456/
    handlers.bundle.js      # Local copy of bundle

definitions.json Structure

{
  "fingerprint": "abc123def456",
  "scorers": [
    {
      "type": "custom_scorer",
      "slug": "length-validator",
      "name": "LengthValidator",
      "schema": { /* JSON Schema */ },
      "bundle": {
        "file": "s3://bucket/tenant/service/abc123def456/handlers.bundle.js",
        "fingerprint": "abc123def456",
        "handler": "LengthValidatorHandler"
      }
    },
    {
      "type": "typed_scorer",
      "slug": "tone-scorer",
      "name": "ToneScorer",
      "schema": { /* JSON Schema */ },
      "promptTemplate": "...",
      "choiceScores": { "1": 0.0, "2": 0.25, /* ... */ }
    }
  ],
  "evaluations": [
    {
      "name": "My Evaluation",
      "scorers": [
        { "type": "custom_scorer", "slug": "length-validator" },
        { "type": "typed_scorer", "slug": "tone-scorer" }
      ]
    }
  ]
}

Updating Existing Uploads

When you upload again:

New fingerprint if code changes
Same fingerprint if no changes (cached)
Old versions remain accessible

# First upload
orcha upload project.ts
# Fingerprint: abc123

# After changes
orcha upload project.ts
# Fingerprint: def456

# Both versions available in cloud

Verification

Verify your upload was successful.

Check Upload Status

The upload command confirms:

✓ Upload complete!
  Fingerprint: abc123def456
  Scorers: 2
  Evaluations: 1
  Bundle: s3://bucket/tenant/service/abc123def456/handlers.bundle.js

Verify S3 Upload

Check S3 directly (requires AWS CLI):

aws s3 ls s3://your-bucket/tenant/service/abc123def456/
# handlers.bundle.js

Test Locally

Test uploaded definitions locally:

import { Eval } from 'orchestrated'
import definitions from './.orchestrated/definitions.json'

// Use uploaded scorer definitions
await Eval("Test Uploaded", {
  data: [{ input: "test", output: "result" }],
  scores: definitions.scorers,
})

Validate Bundle Download

The first time you use uploaded scorers, they're downloaded and cached:

Downloading bundle abc123def456...
✓ Downloaded to .orchestrated/abc123def456/handlers.bundle.js
✓ Using cached bundle

Viewing in Console

Access your uploaded evaluations and scorers in the web console.

Navigate to Console

Open the web console:

# Get console URL
orcha whoami
# Console: https://console.getorchestrated.com

# Or open directly (if configured)
open https://console.getorchestrated.com

View Uploaded Scorers

Navigate to Scorers section
Filter by service name
View scorer details:
- Name and slug
- Schema definition
- Upload timestamp
- Fingerprint
- Usage statistics

View Uploaded Evaluations

Navigate to Evaluations section
Select your service
View evaluation details:
- Name and description
- Data source configuration
- Scorer list
- Historical runs

Run Evaluations

Execute evaluations from the console:

Select an evaluation
Click Run Evaluation
Configure options:
- Data source parameters
- Context overrides
- Export destinations
Click Start
Monitor progress in real-time
View results when complete

API Access

Use uploaded definitions via API:

curl -X POST https://api.getorchestrated.com/evaluations/run \
  -H "Authorization: Bearer $TOKEN" \
  -H "Content-Type: application/json" \
  -d '{
    "evaluationSlug": "my-evaluation",
    "fingerprint": "abc123def456",
    "dataSource": {
      "type": "static",
      "data": [
        { "input": "test", "output": "result" }
      ]
    }
  }'

Share with Team

Uploaded evaluations are available to your entire team:

Team members can view definitions
Run evaluations from console
Access via API
Clone and modify for new use cases

Version Management

Manage multiple versions:

View version history
Compare fingerprints
Rollback to previous versions
Tag stable versions
Archive old versions

Back to: Creating Your First Eval CLI Reference