Skip to main content
Harbor has the following core concepts:

​
Task

A task defines one or more instructions, a sandbox environment, and a verifier. Tasks are used to evaluate agents and models and are implemented as directories in the Harbor task format. See multi-step tasks for tasks with multiple instructions.

​
Dataset

A dataset is a collection of tasks. Datasets are used to evaluate agents and models. Usually, a dataset corresponds to a benchmark (e.g. Terminal-Bench or SWE-Bench Verified). Datasets can optionally be distributed through Harbor Hub.

​
Agent

An agent is a program that completes tasks. Harbor includes pre-integrated agents and supports custom agents implemented with the BaseAgent interface.

​
Sandbox

A sandbox is an isolated environment for running tasks. Harbor includes pre-integrated sandboxes such as Daytona and Modal. Other runtimes can be integrated as custom sandboxes by implementing BaseEnvironment.

​
Verifier

A verifier evaluates the agent’s work and produces the task’s reward.

​
Trial

A trial is one agent’s attempt at completing one task.

​
Job

A job is a collection of trials used to evaluate agents and models. A job can combine datasets, agents, tasks, and models. Under the hood, a job generates trials and runs them in parallel.

​
Trajectory

A trajectory is the conversation and action history of the agent and user while completing a task. Trajectories are one of the most useful tools for understanding and debugging agent behavior. Harbor’s standard trajectory format is ATIF. Trajectories can also be loaded into a later agent session.