Verifier

Verifier documentation for Harbor.

The tests/test.sh script is the entrypoint for the verifier. The tests/ directory is uploaded to the environment at /tests/ after the agent runs, or should be included in the verifier image if running separately. tests/test.sh is executed in the task’s working directory and is responsible for verifying the task’s completion. It must produce a numerical reward at /logs/verifier/reward.txt or /logs/verifier/reward.json.

Required script

OS	File
Linux	tests/test.sh
Windows	tests/test.bat

The script should:

1. Install test dependencies (if needed)
2. Verify that the agent satisfied the instruction
3. Write a reward under /logs/verifier/

We recommend using absolute paths inside the script to avoid cwd surprises.

Reward files

File	Format
/logs/verifier/reward.json	JSON key-value object of labeled numeric metrics
/logs/verifier/reward.txt	Single number, usually 1 or 0

If both are present, Harbor prefers reward.json.

Example

tests/test.sh

#!/bin/bash
set -euo pipefail

uvx pytest /tests/test_outputs.py

if [ $? -eq 0 ]; then
  echo 1 > /logs/verifier/reward.txt
else
  echo 0 > /logs/verifier/reward.txt
fi

For multi-metric or LLM-based grading, consider using RewardKit or writing a structured reward.json file.

Separate verifier environment

Harbor tasks can elect to use a separate sandbox for verification, which improves the security boundary between the agent and the verifier.

To use a separate verifier sandbox, set [verifier.environment_mode] to "separate" in task.toml or include a [verifier.environment] section with the same schema as [environment].

If either of those are set, Harbor will treat the tests/ directory as the verifier image’s build context, similar to environment/. The verifier image is responsible for including /tests/test.sh or /tests/test.bat.

Artifacts declared in the [artifacts] section of the task.toml are copied into the verifier sandbox at the same location as they are in the agent sandbox.

See separate verifier environment for more details.

Example

task.toml

[verifier]
environment_mode = "separate"

[verifier.environment]
cpus = 2

tests/Dockerfile

FROM ubuntu:24.04

WORKDIR /app

COPY test.sh /tests/test.sh

Regrading

Harbor can rerun verifiers for existing trials if the task uses a separate verifier environment. This is useful when iterating on a verifier.

See regrading jobs and trials for details.

Passing environment variables to the verifier

Environment variables can be passed to the verifier using the [verifier.env] section in task.toml.

task.toml

[verifier.env]
ANTHROPIC_API_KEY = "${ANTHROPIC_API_KEY}"
MODEL_NAME = "claude-haiku-4-5"

The ${VAR} syntax is used to read from the host environment. Harbor will ask the user to confirm before passing the environment variables to the verifier.

LLM- or agent-as-a-judge

Task authors can implement test.sh to use whatever verification approach they like, including LLM- or agent-as-a-judge.

To avoid boilerplate, consider using RewardKit to define your judge or programmatic criteria.

rewardkit

The Harbor team maintains a package called harbor-rewardkit. It is the easiest way to define and run common verifiers, including programmatic criteria and LLM- or agent-as-a-judge.