Custom plugins

Run custom logic around Harbor jobs and trials.

Harbor uses the plugin interface internally for Harbor Hub. Its upload plugin streams completed trials and finalizes jobs.

Custom plugins use the same interface for local or private integrations. They run in the Harbor process, react to job and trial events, and are passed at runtime instead of stored in config.json.

Lifecycle

Method	When it runs
on_job_start(job)	Before the job runs. Initialize clients or register trial callbacks here.
on_job_end(job_result)	After the job finishes. Flush, upload, or close resources here.

Available trial callbacks include on_trial_started, on_environment_started, on_agent_started, on_agent_ended, on_verification_started, on_trial_ended, and on_trial_cancelled.

Implement a plugin

Extend BaseJobPlugin and implement its two lifecycle methods. Register trial callbacks from on_job_start when the plugin needs per-trial events.

trial_logger.py

from harbor.job import Job
from harbor.models.job.plugin import BaseJobPlugin
from harbor.models.job.result import JobResult
from harbor.trial.hooks import TrialHookEvent


class TrialLoggerPlugin(BaseJobPlugin):
    def __init__(self, prefix: str = "trial") -> None:
        self.prefix = prefix

    async def on_job_start(self, job: Job) -> None:
        job.on_trial_ended(self._on_trial_ended)

    async def _on_trial_ended(self, event: TrialHookEvent) -> None:
        print(f"{self.prefix}: {event.trial_name}")

    async def on_job_end(self, job_result: JobResult) -> None:
        print(f"job finished: {len(job_result.trial_results)} trials")

Plugins run outside the sandbox. Install the plugin and its dependencies in the same Python environment as Harbor. See Environment variables for the difference between Harbor-process and sandbox variables.

Run a custom plugin

Custom plugins do not need a registered name. Pass module.path:ClassName to --plugin. For TrialLoggerPlugin in trial_logger.py, use trial_logger:TrialLoggerPlugin.

export OPENAI_API_KEY="..."
harbor run \
  -t hello-world/hello-world \
  -a codex -m openai/gpt-5.6-sol \
  --plugin trial_logger:TrialLoggerPlugin \
  --plugin-kwarg prefix=completed

--plugin-kwarg (--pk) passes constructor options to the plugin. Both --plugin and --pk are repeatable. See Use multiple plugins for the multi-plugin syntax.

Examples

See Harbor’s plugin packages for more implementations. For a third-party integration guide, see Braintrust.