> ## Documentation Index
> Fetch the complete documentation index at: https://docs.iearena.org/llms.txt
> Use this file to discover all available pages before exploring further.

# Verifier

> Verifier documentation for Harbor.

The `tests/test.sh` script is the entrypoint for the verifier. The `tests/` directory is uploaded to the environment at `/tests/` after the agent runs, or should be included in the verifier image if [running separately](#separate-verifier-environment). `tests/test.sh` is executed in the task's working directory and is responsible for verifying the task's completion. It must produce a numerical reward at `/logs/verifier/reward.txt` or `/logs/verifier/reward.json`.

## Required script

| OS      | File             |
| ------- | ---------------- |
| Linux   | `tests/test.sh`  |
| Windows | `tests/test.bat` |

The script should:

1. Install test dependencies (if needed)
2. Verify that the agent satisfied the [instruction](/core-concepts/tasks/instruction)
3. Write a **reward** under `/logs/verifier/`

We recommend using **absolute paths** inside the script to avoid `cwd` surprises.

## Reward files

| File                         | Format                                           |
| ---------------------------- | ------------------------------------------------ |
| `/logs/verifier/reward.json` | JSON key-value object of labeled numeric metrics |
| `/logs/verifier/reward.txt`  | Single number, usually `1` or `0`                |

If both are present, Harbor prefers `reward.json`.

### Example

```bash title="tests/test.sh" theme={"system"}
#!/bin/bash
set -euo pipefail

uvx pytest /tests/test_outputs.py

if [ $? -eq 0 ]; then
  echo 1 > /logs/verifier/reward.txt
else
  echo 0 > /logs/verifier/reward.txt
fi
```

For multi-metric or LLM-based grading, consider using [RewardKit](/core-concepts/rewardkit/quick-start) or writing a structured `reward.json` file.

## Separate verifier environment

Harbor tasks can elect to use a separate sandbox for verification, which improves the security boundary between the agent and the verifier.

To use a separate verifier sandbox, set `[verifier.environment_mode]` to `"separate"` in `task.toml` or include a `[verifier.environment]` section with the same schema as `[environment]`.

If either of those are set, Harbor will treat the `tests/` directory as the verifier image's build context, similar to `environment/`. The verifier image is responsible for including `/tests/test.sh` or `/tests/test.bat`.

Artifacts declared in the `[artifacts]` section of the `task.toml` are copied into the verifier sandbox at the same location as they are in the agent sandbox.

See [separate verifier environment](/core-concepts/tasks/separate-verifier) for more details.

### Example

```toml title="task.toml" theme={"system"}
[verifier]
environment_mode = "separate"

[verifier.environment]
cpus = 2
```

```dockerfile title="tests/Dockerfile" theme={"system"}
FROM ubuntu:24.04

WORKDIR /app

COPY test.sh /tests/test.sh
```

## Regrading

Harbor can rerun verifiers for existing trials if the task uses a separate verifier environment. This is useful when iterating on a verifier.

See [regrading jobs and trials](/core-concepts/jobs/regrade) for details.

## Passing environment variables to the verifier

Environment variables can be passed to the verifier using the `[verifier.env]` section in `task.toml`.

```toml title="task.toml" theme={"system"}
[verifier.env]
ANTHROPIC_API_KEY = "${ANTHROPIC_API_KEY}"
MODEL_NAME = "claude-haiku-4-5"
```

The `${VAR}` syntax is used to read from the host environment. Harbor will ask the user to confirm before passing the environment variables to the verifier.

## LLM- or agent-as-a-judge

Task authors can implement `test.sh` to use whatever verification approach they like, including LLM- or agent-as-a-judge.

To avoid boilerplate, consider using [RewardKit](/core-concepts/rewardkit/quick-start) to define your judge or programmatic criteria.

## `rewardkit`

The Harbor team maintains a package called [`harbor-rewardkit`](/core-concepts/rewardkit/quick-start). It is the easiest way to define and run common verifiers, including programmatic criteria and LLM- or agent-as-a-judge.