A taskA specific item a contributor claims or enters package gives an agent work to do in a terminal and gives a verifier a repeatable way to check the result. Use the Harbor task format to build the files. Use Humanbased's inventory fields to describe the exact revision you checked. Harbor task format
Separate the task files by audience
| Files | What to put there | Who needs it |
|---|---|---|
instruction.md | The work, available inputs, output paths and success criteria | The solver |
task.toml and environment/ | Run limits, resource requirements, input files and a reproducible environment. Pin the image by digest. | The solver and runner |
tests/ | The verifier, its fixtures and its grading dependencies | The verifier |
solution/ | A reference implementation that proves the task is solvable | An authorized verifier or reviewer |
In separate-verifier mode, list each output the verifier needs under artifacts in task.toml. Treat those outputs as untrusted input. Keep hidden tests, expected answers and the reference implementation out of the agent image and the Marketplace preview. Record whether a reference exists. Harbor itself makes the reference optional. Separate verifier · Artifact transfer · Reference solution
Record the evidence for one revision
| Inventory field | What to record |
|---|---|
| Task family and package version | The family, exact package revision, manifest digest and pinned environment digest. Count distinct tasks in the listed snapshot. |
| Verifier tier and revision | Record each method and its version separately. Tier 1 is a programmatic check, Tier 2 is comparison to a reference oracle, and Tier 3 is rubric-based judgment. A tier describes the method of checking. |
| Reference-solution presence | Record whether the package has a reference, with a revision or digest available to authorized readers. |
| Calibration | For a frontier-agent calibration claim, pin three agent configurations and run eight trials each on the same package and verifier revision. Record models, settings, budgets, trial counts and a batch reference. Compare only runs with compatible configurations. |
| Run evidence | Record execution state (completed, budget_exhausted, or infra_error), check result, and the reward-validity decision as separate fields. Use the verifier report for the check result. Count infrastructure errors under execution state and give the numerator and denominator for every reported rate. |
When checking a listing, match its reported metrics to the selected package and verifier revisions. Treat missing fields as unavailable. Distinguish seller-reported evidence from results the platform ran. Share only approved summaries and references with buyers. Raw verification logs stay internal.
Run a small example
This synthetic firewall audit uses four invented input lines. It illustrates the package layout and verifier checks without exposing a real task's answer. Use Harbor 0.25.0 and a running local Docker engine. Run the following in a new working directory.
mkdir -p firewall-port-audit/{environment,solution,tests}
cat > firewall-port-audit/instruction.md <<'EOF'
Read /app/rules.txt. Each row contains a protocol and port separated by a
space. Write /app/ports.txt with the distinct TCP port numbers, in numeric
ascending order, one per line, ending with a newline.
EOF
cat > firewall-port-audit/task.toml <<'EOF'
artifacts = ["/app/ports.txt"]
[agent]
timeout_sec = 30
[environment]
build_timeout_sec = 120
[verifier]
environment_mode = "separate"
timeout_sec = 30
EOF
cat > firewall-port-audit/environment/Dockerfile <<'EOF'
FROM python:3.12-slim@sha256:4840d514c583a4d3d8cad3940e68dee6f0f6f3635c12fcd0a3d74c21a5b93a54
WORKDIR /app
RUN printf '%s\n' 'tcp 443' 'udp 53' 'tcp 22' 'tcp 443' > /app/rules.txt
EOF
cat > firewall-port-audit/solution/solve.sh <<'EOF'
#!/bin/sh
set -eu
awk '$1 == "tcp" { print $2 }' /app/rules.txt | sort -nu > /app/ports.txt
EOF
cat > firewall-port-audit/tests/test.sh <<'EOF'
#!/bin/sh
set -eu
mkdir -p /logs/verifier
if printf '22\n443\n' | cmp -s - /app/ports.txt; then
printf '1\n' > /logs/verifier/reward.txt
else
printf '0\n' > /logs/verifier/reward.txt
fi
EOF
chmod +x firewall-port-audit/{solution/solve.sh,tests/test.sh}
harbor run -p firewall-port-audit -a oracle -e docker -o jobs --job-name oracle
harbor run -p firewall-port-audit -a nop -e docker -o jobs --job-name nopThe reference run should report reward 1 and the no-op run should report 0, both with zero exceptions. Check each job's result.json. In a calibration record, retain the exact package and verifier revision for every run and report infrastructure errors separately. These two smoke checks establish solvability and no-op rejection for the example. A frontier-agent calibration claim requires its own model trial batch. Harbor authoring tutorial
Choose a generation recipe
Choose the recipe that matches the evidence your item needs.
- Verify a software task
- Verify an ML reproduction task
- Verify a scientific calculation task
- Verify a synthetic reconciliation task
- Verify a tool-recovery trajectory
- Verify a formal proof task
For the inventory card, record the item kind, exact version, author and verifier roles, source and permitted uses, and which evidence is available. Count distinct included tasks for a task package. Count individually identified and accessible trajectories for a trajectory set. Record who corrected an item, the reason, its review state and the before-and-after check results. If a person or model scored an explanation, keep its rubric, calibration and review separate from the program result.