CLI

Launch remote rollouts and browse jobs, trials, and secrets from the harbor CLI

Please ensure you have either logged in with harbor auth login or minted an API key (see Harbor Hub API key)

Launching a Job

The command harbor run --launch will attempt to launch a job under your personal organization

The --launch flag can be combined with a singular agent/model combination (e.g. harbor run --launch -a claude-code -m anthropic/fable-5 -d harbor/hello-world). Agent/model sweeps must be launched using a config file (e.g. harbor run --launch -c config.yaml)

# config.yaml
job_name: opus-vs-codex-sweep
organization: my-org          # omit to use your personal org
credential_mode: gateway      # or "direct"

n_attempts: 3                 # 1-10
n_concurrent_trials: 32       # 1-1000

# Every agent runs against every task. Trials = n_attempts x tasks x agents.
agents:
  - name: claude-code
    model_name: anthropic/claude-opus-4-1
    secrets: [ANTHROPIC_API_KEY]

  - name: claude-code
    model_name: anthropic/claude-sonnet-4-5
    secrets: [ANTHROPIC_API_KEY]

  - name: codex
    model_name: openai/gpt-5
    secrets: [OPENAI_API_KEY, HF_TOKEN]
    n_concurrent: 8           # per-agent sub-limit under n_concurrent_trials
    env:
      RUST_LOG: info          # nonsensitive only - secrets go in `secrets`

  - name: oracle              # needs no inference credential
    secrets: []

datasets:
  - name: terminal-bench/terminal-bench-2-1
    ref: "6"                  # `ref` or `version`, never both
    n_tasks: 25
    exclude_task_names:
      - "flaky-*"

tasks:
  - name: harbor/hello-world
    ref: latest

# Supplied for this job only, read from your local environment at launch.
job_secrets:
  HF_TOKEN:
    from_env: HF_TOKEN

retry:
  max_retries: 2
  include_exceptions:
    - EnvironmentStartError

Injecting Secrets

There are two methods of injecting secrets into the hosted job: flags --stored-secret <ENV_VAR_NAME> and --one-off-secret ENV_VAR_NAME=secret_value, and config fields job_secrets and secrets.

When using flags, --stored-secret will pull <ENV_VAR_NAME> from the organization’s stored credentials on the hub. --one-off-secret will encrypt the secret value and ensure that the value is not persisted in any job config files. The secret will be decrypted only when necessary and marked revoked after the job finishes. Flags apply secrets to every agent/model combination in the job config.

In other words, if I launch with the following command: harbor run --launch -c simple_config.yaml --stored-secret TEST_VAR where simple_config.yaml is

job_name: simple-config
organization: my-org          # omit to use your personal org

agents:
  - name: claude-code
    model_name: anthropic/claude-opus-4-1
    secrets: [ANTHROPIC_API_KEY]

  - name: claude-code
    model_name: anthropic/claude-sonnet-4-5
    secrets: [ANTHROPIC_API_KEY]

tasks:
  - name: harbor/hello-world
    ref: latest

The final resolved config would be as follows:

job_name: simple-config
organization: my-org

agents:
  - name: claude-code
    model_name: anthropic/claude-opus-4-1
    secrets: [ANTHROPIC_API_KEY, TEST_VAR] # notice TEST_VAR is used in both agents

  - name: claude-code
    model_name: anthropic/claude-sonnet-4-5
    secrets: [ANTHROPIC_API_KEY, TEST_VAR] # notice TEST_VAR is used in both agents

tasks:
  - name: harbor/hello-world
    ref: latest

Flag	Value	Effect
--dry-run	—	Validate without queuing. Resolves tasks, agents, and the owning org, checks selections, and reports the trial count
--org	TEXT	Organization that should own the hosted job. Defaults to your personal org
--credential-mode	gateway | direct	gateway proxies the key through the Hub, direct hands the real key to the agent
--stored-secret	NAME	Select a secret already stored in the owning org. Repeatable
--one-off-secret	NAME[=VALUE]	Supply a secret for this job only. Bare NAME reads it from your environment. Repeatable
--env-file	PATH	Load a .env. Every name in it is selected, as with --one-off-secret
--registry-secret	HOST=NAME_OR_ID	Pin a stored pull secret for a private image host. Repeatable
--no-secrets	—	Launch with no credentials at all

--launch and --upload are mutually exclusive, so the upload-side flags do not apply to a hosted launch: --upload, --public / --private, and --share.

Regrading

Add --launch to regrade recorded Hub results with an updated verifier. This creates a new private job without rerunning the agent or changing the source.

Specify --org for the launch organization’s credits, permissions, and credentials. It must match the source organization, and you must have permission to launch there; shared read access alone is insufficient.

Regrade a job

harbor job regrade <job-uuid> --launch --org <org> -d org/dataset@version
harbor job regrade <job-uuid> --launch --org <org> -p ./updated-tasks

Use -d for datasets, -p for local task directories, or -t for registry tasks. These options are repeatable and can be combined. Verifiers match by task name. Harbor selects the latest trial attempts with recorded archives.

Regrade a trial

harbor trial regrade <trial-uuid> --launch --org <org> -t org/task@ref

Use exactly one -t or -p; for -p, provide a single task directory. Local verifier tasks are uploaded as temporary private packages. Add --yes to accept launch and upload without prompting.

Requirements

Hosted regrade currently supports single-step trials with separate verifiers. Source trials must have finished, and the replacement task must have the same name and be able to grade from the recorded artifacts. Sources must be Hub UUIDs.

Use --dry-run to validate without creating a job. With local verifier tasks, it checks inputs without uploading; full source validation happens on launch.

Options

These options apply with --launch. See the secret options for credential details.

Option	Description
--launch	Run the regrade on Hosted Harbor.
--org NAME	Launch organization. Required; must match the source organization.
-p, --task-path PATH	Local verifier task. Jobs also accept parent directories; repeatable for jobs.
-t, --task REF	Registry verifier task, such as org/task@ref. Repeatable for jobs.
-d, --dataset REF	Dataset of verifier tasks. Job regrade only; repeatable.
-n, --n-concurrent N	Concurrent regrades. Job regrade only.
--job-name NAME	Name of the new hosted job. Job regrade only.
--trial-name NAME	Currently sets the new hosted job’s name for trial regrade.
--ve, --verifier-env KEY=VALUE	Verifier environment variable. Repeatable.
--stored-secret NAME	Stored verifier credential from the launch organization. Repeatable.
--one-off-secret NAME[=VALUE]	Job-only credential; bare NAME reads your local environment. Repeatable.
--registry-secret HOST=NAME_OR_ID	Select a stored credential for pulling private images. Repeatable.
--use-static-ip	Use a static outbound IP. Otherwise, the server chooses its default.
-y, --yes	Accept launch and local task upload without prompting.
--dry-run	Validate without creating a job.

Listing Jobs

Using the command harbor hub job list you can print out a list of all jobs visible to your user on the hub.

Flag	Value	Effect
--scope	my | shared | all	Visibility scope. Defaults to my, the jobs your user owns
--search	TEXT	Only jobs whose names contain the substring
--agent	NAME	Filter by agent name. Repeatable, and multiple values match any of them
--provider	NAME	Filter by model provider. Repeatable
--model	NAME	Filter by model. Repeatable

Job Overview

Use the command harbor hub job show JOB_ID JOB_ID_2 ... to print an overview similar to the one provided on the hub.

The overview reports the number of trials, errors, and retries, the average return on each metric, the cost in USD, and token usage.

Per-Task Job Breakdown

Use the command harbor hub job tasks JOB_ID to see a per-task breakdown of a job.

Flag	Value	Effect
--search	TEXT	Only tasks whose names contain the substring
--agent, --provider, --model	NAME	As for job list

Trials Overview

Use the command harbor hub job trials JOB_ID JOB_ID_2 ... to list trials across one or more jobs. In interactive terminals, you can switch pages to audit all trials without running a second command.

Flag	Value	Effect
--search	TEXT	Only trials whose names contain the substring
--agent, --provider, --model	NAME	As for job list
--limit	N	Page size. Defaults to 100 here
--failed-only	—	Only trials that errored or failed
--include-retries	—	Include retry history, not just the latest execution
--sort-by	started_at | task_name | name | error_type	Sort column
--sort-order	asc | desc	Sort direction

Comparing 2 or More Jobs

Use command harbor hub job compare JOB_ID JOB_ID_2 ... to get the side-by-side grid for performance on tasks

Job Visibility

Use command harbor hub job shares JOB_ID to see who a job is shared with. This returns orgs and users

Renaming Jobs

Use harbor hub job rename JOB_ID "NEW NAME" to change the name displayed on the Hub. This does not change the job ID, artifact identity, or submitted configuration. Add --json to print the rename RPC response.

Transferring Job Ownership

Use harbor hub job transfer JOB_ID ORGANIZATION to transfer a completed job and all of its trials to another organization you own. Job and trial IDs do not change. The destination must already have access to every execution dependency used by the job.

The command prompts for confirmation; pass --yes / -y when scripting. By default, the source organization keeps shared read access and job-specific secrets are permanently removed. Use --drop-source-access to remove the source grant or --keep-job-credentials to retain and reauthorize secrets you originally supplied.

Deleting Jobs

Use command harbor hub job delete JOB_ID JOB_ID_2 ... to permanently delete jobs you own from the hub, including all of their trials and shares.

The command prompts for confirmation before deleting anything; pass --yes / -y to skip the prompt (required when scripting or piping).

Only the job’s owner can delete a job. Jobs linked to a leaderboard submission and hosted jobs that are still running cannot be deleted.

Hosted Job Status

Use command harbor hub job status JOB_ID to get your job status. Returns counts of pending, running, failed, and completed trials.

Trial Show

Use command harbor hub trial show TRIAL_ID to show a single trial’s metadata

Downloading a Trial

Use command harbor hub trial download TRIAL_ID to download a particular trial

Flag	Value	Effect
--output-dir, -o	PATH	Directory to materialize the trial into. Defaults to ./trials
--overwrite	—	Replace an existing trial directory
--trajectory	—	Download only trajectory.json. Errors if the trial has no stored trajectory

Retrying a Hosted Trial

For a trial in a job that was launched remotely, you can use the command harbor hub trial retry TRIAL_ID to retry those trials.

These flags are cumulative filters. harbor hub trial retry --job JOB_ID --failed-only selects every trial in the job, then narrows to the ones that failed.

Flag	Value	Selects
--job	JOB_ID	Every trial in that job
--search	TEXT	Trials whose names contain the substring
--agent	TEXT	Trials run with that agent
--provider	TEXT	Trials run with that provider
--model	TEXT	Trials run with that model
--task	TEXT	Trials run against that task
--exception	TEXT	Trials that failed with that exception
--failed-only	—	Only trials that failed
--yes	—	Skip confirmation

Canceling a Hosted Trial

To cancel a hosted trial, use command harbor hub trial cancel TRIAL_ID

Flag	Value	Effect
--job	JOB_ID	Cancel every trial in the job. Same as harbor hub job cancel JOB_ID
--all	—	Cancel every currently running trial
--reason	TEXT	Record a reason for the cancellation
--yes	—	Skip confirmation

The selection filters from trial retry apply here too.

Adding Secrets

Use command harbor hub secrets add NAME to upload a secret to the Hub.

Flag	Value	Effect
--org	ORG_ID	Organization to store the secret on. Defaults to your personal org
--job	JOB_ID	Scope the secret to a single job instead of account-wide
--from-env	—	Read the value from the local environment variable of the same name
--yes, -y	—	Supersede an existing secret without prompting

Listing Secrets

Use command harbor hub secrets list to list the names and metadata of your uploaded secrets.

Flag	Value	Effect
--org	ORG_ID	Organization to list secrets for
--job	JOB_ID	List secrets scoped to that job
--include-revoked	—	Include secrets that have been revoked

Deleting Secrets

Use command harbor hub secrets delete to revoke a secret.

Flag	Value	Effect
--org	ORG_ID	Revoke a secret in that organization
--job	JOB_ID	Revoke a secret scoped to that job
--purge	—	Delete the record outright, so it no longer appears under list --include-revoked

Adding an Image Registry Secret

Use command harbor hub secrets registry add to add an image registry secret. Also supports secrets registry list and secrets registry delete.

Flag	Value	Effect
--name	TEXT	Display name used to select the credential later
--from-file	PATH	Read the credential from a file, such as a GAR service account JSON
--yes, -y	—	Supersede an existing credential without prompting

Shared Flags

The listing commands above also accept these:

Flag	Value	Effect
--quiet, -q	—	Print only IDs, for piping into xargs
--no-trunc	—	Show full cell content, wrapping instead of truncating
--no-headers	—	Omit the header row
--page	N	Fetch one specific page, disabling interactive paging
--json	—	Return the raw API response as JSON