> ## Documentation Index
> Fetch the complete documentation index at: https://docs.iearena.org/llms.txt
> Use this file to discover all available pages before exploring further.

# Metrics

> Metrics documentation for Harbor.

Metrics define how to aggregate rewards across tasks in a job or dataset. By default, Harbor averages rewards across tasks and treats missing rewards as 0.

However, you may want to define custom logic or handle missing rewards differently. You can do this by creating a `metric.py` file. If you are working with a `dataset.toml`, you can run the following command to create a `metric.py` file and automatically add it to the `[[files]]` section of your `dataset.toml`:

```bash theme={"system"}
harbor init --dataset "<org>/<name>" --with-metric
```

If you are running a local dataset, it will automatically use the `metric.py` if it's present in the dataset directory.

## Default behavior

By default, Harbor averages rewards across tasks and treats missing rewards as 0.

<Tabs>
  <Tab title="Single dimension">
    ```jsonl title="Rewards" theme={"system"}
    {"reward": 1}
    {"reward": 0}
    null
    {"reward": 0}
    ```

    ```json title="Aggregate" theme={"system"}
    {"mean": 0.25}
    ```
  </Tab>

  <Tab title="Multi-dimensional">
    ```jsonl title="Rewards" theme={"system"}
    {"correctness": 1, "style": 0.5}
    {"correctness": 0, "style": 1}
    null
    {"correctness": 1, "style": 0.5}
    ```

    ```json title="Aggregate" theme={"system"}
    {"correctness": 0.5, "style": 0.5}
    ```
  </Tab>

  <Tab title="Missing dimensions">
    ```jsonl title="Rewards" theme={"system"}
    {"correctness": 1}
    {"style": 1}
    {}
    null
    ```

    ```json title="Aggregate" theme={"system"}
    {"correctness": 0.25, "style": 0.25}
    ```
  </Tab>
</Tabs>

For single-dimension rewards, Harbor names the aggregate after the metric (`mean`). For multi-dimensional rewards, it aggregates each dimension independently. Missing rewards and missing dimensions count as 0.

## Custom metrics with `metric.py`

A `metric.py` script must accept the following arguments:

* `-i, --input-path`: rewards JSONL file
* `-o, --output-path`: output JSON file with computed metrics

### Example

```python title="metric.py" theme={"system"}
# /// script
# dependencies = []
# ///
# To add a dependency: uv add --script metric.py <dependency>

import argparse
import json
from pathlib import Path


def main(input_path: Path, output_path: Path):
    rewards = []

    for line in input_path.read_text().splitlines():
        reward = json.loads(line)
        if reward is None:
            rewards.append(0)
        elif len(reward) != 1:
            raise ValueError(
                f"Expected exactly one key in reward dictionary, got {len(reward)}"
            )
        else:
            rewards.extend(reward.values())

    mean = sum(rewards) / len(rewards) if rewards else 0

    output_path.write_text(json.dumps({"mean": mean}))


if __name__ == "__main__":
    parser = argparse.ArgumentParser()
    parser.add_argument(
        "-i",
        "--input-path",
        type=Path,
        required=True,
        help="Path to a jsonl file containing rewards, one json object per line.",
    )
    parser.add_argument(
        "-o",
        "--output-path",
        type=Path,
        required=True,
        help="Path to a json file where the metric will be written as a json object.",
    )
    args = parser.parse_args()
    main(args.input_path, args.output_path)
```

### Considerations

When implementing a custom metric, be sure to account for

1. Missing or invalid reward keys
2. Invalid reward values
3. Null rows in the JSONL input
4. Default aggregate reward keys

### Publishing custom metrics

When publishing a dataset, Harbor automatically publishes the `metric.py` file in the same directory:

```bash theme={"system"}
harbor publish "<path/to/dataset>" --public
```

### Output format

`metric.py` output should be a JSON object. Multiple metric values are allowed:

```json theme={"system"}
{"mean": 0.81, "pass_rate": 0.5}
```