Skip to content
Docs
Download Markdown

Build and check a task package

Use Harbor to build a reproducible task, then record the evidence for its exact revision.

Download Markdown

A taskA specific item a contributor claims or enters package gives an agent work to do in a terminal and gives a verifier a repeatable way to check the result. Use the Harbor task format to build the files. Use Humanbased's inventory fields to describe the exact revision you checked. Harbor task format

Separate the task files by audience

FilesWhat to put thereWho needs it
instruction.mdThe work, available inputs, output paths and success criteriaThe solver
task.toml and environment/Run limits, resource requirements, input files and a reproducible environment. Pin the image by digest.The solver and runner
tests/The verifier, its fixtures and its grading dependenciesThe verifier
solution/A reference implementation that proves the task is solvableAn authorized verifier or reviewer

In separate-verifier mode, list each output the verifier needs under artifacts in task.toml. Treat those outputs as untrusted input. Keep hidden tests, expected answers and the reference implementation out of the agent image and the Marketplace preview. Record whether a reference exists. Harbor itself makes the reference optional. Separate verifier · Artifact transfer · Reference solution

Record the evidence for one revision

Inventory fieldWhat to record
Task family and package versionThe family, exact package revision, manifest digest and pinned environment digest. Count distinct tasks in the listed snapshot.
Verifier tier and revisionRecord each method and its version separately. Tier 1 is a programmatic check, Tier 2 is comparison to a reference oracle, and Tier 3 is rubric-based judgment. A tier describes the method of checking.
Reference-solution presenceRecord whether the package has a reference, with a revision or digest available to authorized readers.
CalibrationFor a frontier-agent calibration claim, pin three agent configurations and run eight trials each on the same package and verifier revision. Record models, settings, budgets, trial counts and a batch reference. Compare only runs with compatible configurations.
Run evidenceRecord execution state (completed, budget_exhausted, or infra_error), check result, and the reward-validity decision as separate fields. Use the verifier report for the check result. Count infrastructure errors under execution state and give the numerator and denominator for every reported rate.

When checking a listing, match its reported metrics to the selected package and verifier revisions. Treat missing fields as unavailable. Distinguish seller-reported evidence from results the platform ran. Share only approved summaries and references with buyers. Raw verification logs stay internal.

Run a small example

This synthetic firewall audit uses four invented input lines. It illustrates the package layout and verifier checks without exposing a real task's answer. Use Harbor 0.25.0 and a running local Docker engine. Run the following in a new working directory.

mkdir -p firewall-port-audit/{environment,solution,tests}

cat > firewall-port-audit/instruction.md <<'EOF'
Read /app/rules.txt. Each row contains a protocol and port separated by a
space. Write /app/ports.txt with the distinct TCP port numbers, in numeric
ascending order, one per line, ending with a newline.
EOF

cat > firewall-port-audit/task.toml <<'EOF'
artifacts = ["/app/ports.txt"]

[agent]
timeout_sec = 30

[environment]
build_timeout_sec = 120

[verifier]
environment_mode = "separate"
timeout_sec = 30
EOF

cat > firewall-port-audit/environment/Dockerfile <<'EOF'
FROM python:3.12-slim@sha256:4840d514c583a4d3d8cad3940e68dee6f0f6f3635c12fcd0a3d74c21a5b93a54
WORKDIR /app
RUN printf '%s\n' 'tcp 443' 'udp 53' 'tcp 22' 'tcp 443' > /app/rules.txt
EOF

cat > firewall-port-audit/solution/solve.sh <<'EOF'
#!/bin/sh
set -eu
awk '$1 == "tcp" { print $2 }' /app/rules.txt | sort -nu > /app/ports.txt
EOF

cat > firewall-port-audit/tests/test.sh <<'EOF'
#!/bin/sh
set -eu
mkdir -p /logs/verifier
if printf '22\n443\n' | cmp -s - /app/ports.txt; then
  printf '1\n' > /logs/verifier/reward.txt
else
  printf '0\n' > /logs/verifier/reward.txt
fi
EOF

chmod +x firewall-port-audit/{solution/solve.sh,tests/test.sh}
harbor run -p firewall-port-audit -a oracle -e docker -o jobs --job-name oracle
harbor run -p firewall-port-audit -a nop -e docker -o jobs --job-name nop

The reference run should report reward 1 and the no-op run should report 0, both with zero exceptions. Check each job's result.json. In a calibration record, retain the exact package and verifier revision for every run and report infrastructure errors separately. These two smoke checks establish solvability and no-op rejection for the example. A frontier-agent calibration claim requires its own model trial batch. Harbor authoring tutorial

Choose a generation recipe

Choose the recipe that matches the evidence your item needs.

For the inventory card, record the item kind, exact version, author and verifier roles, source and permitted uses, and which evidence is available. Count distinct included tasks for a task package. Count individually identified and accessible trajectories for a trajectory set. Record who corrected an item, the reason, its review state and the before-and-after check results. If a person or model scored an explanation, keep its rubric, calibration and review separate from the program result.