Use the task-package specification to choose the package layout and record evidence for the exact revision.
Start from an analysis whose result you can reproduce on licensed or original inputs. This example uses the analysis-reproduction pattern and original synthetic observations.
Author and verify
Give the solver the training observations and the feature values and IDs it must predict. Keep the targets for the held-out split with the verifier. Specify the prediction schema, mean absolute error and threshold before running. Tier 1 checks reject a missing ID, a non-finite prediction or a score above the threshold. The metric supplies the verdict without a judge.
Calibrate and correct
Use a known-good reproduction and a constant predictor as controls. Pin both split digests, metric code and the taskA specific item a contributor claims or enters revision. Mark missing or malformed predictions as failed checks. Record runner failures separately. If a constant predictor passes, revise the split or threshold with a new verifier revision and rerun the controls.
Worked example
Use Python 3.12. The script fits a slope from training observations and compares predictions with two held-out targets. The reference should pass and both the constant and missing-ID controls should fail.
python3.12 - <<'PY'
from math import isfinite
train = [(1, 2), (2, 4), (3, 6)]
features = {"a": 4, "b": 5}
targets = {"a": 8, "b": 10}
slope = sum(x * y for x, y in train) / sum(x * x for x, _ in train)
def passes(predictions):
if predictions.keys() != targets.keys():
return False
if not all(type(value) in (int, float) and isfinite(value) for value in predictions.values()):
return False
error = sum(abs(predictions[key] - value) for key, value in targets.items())
return error / len(targets) <= 0.10
assert passes({key: slope * x for key, x in features.items()})
assert not passes({"a": 4, "b": 4})
assert not passes({"a": 8})
assert not passes({"a": 8, "b": "10"})
print("reference pass, constant fail, missing ID fail")
PYWhen packaging the task, read predictions from the declared output artifact and score them against targets stored with the verifier. Preserve the actual error and threshold in the private evidence.