Run an evaluation with a reviewed execution notebook

Why use this

1. An evaluation researcher may have a prior run to learn from but lack a coherent record of commands, environment choices and failed attempts. Repeating the apparent successful steps can miss decisions that materially changed the result.

2. This job could prepare a reviewed execution notebook that distinguishes planned actions from observed outputs and interpretations. That structure supports later reproducibility review without making a new notebook look like an already completed evaluation.

Why an agent / simpler alternative

Useful judgment

Extract dependencies from the prior record, identify missing assumptions and organize a traceable execution plan with review gates.

Simpler alternative

Use an established evaluation script when its environment and reporting already capture the necessary evidence.

Unsuitable when

The task would provision resources, execute costly runs or claim reproducibility without approved scope and inspected results.

Proposed inputs

1. Name

   Prior-run materials

   Owner

   Authorized researcher

   Minimum scope

   Relevant commands, outputs and environment notes

   Quality check

   Preserve failures and version details

2. Name

   Evaluation brief

   Owner

   Research lead

   Minimum scope

   Question, dataset scope, infrastructure and resource limits

   Quality check

   Define acceptance criteria and permitted alternatives

Access and setup — proposed or unknown

Proposed minimum

1. Review synthetic run notes and draft notebook sections; no provisioning, commands, downloads or evaluation execution.

Compatibility unknowns

1. Current notebook tooling, infrastructure access, dataset availability and runtime compatibility are unknown.

External actions requiring approval

1. Infrastructure creation, compute spending, data access, execution and external sharing requires explicit plan approval.

Expected deliverable and review checks

Proposed artifact

Execution-notebook candidate with plan, command placeholders, expected artifacts and separate future result/interpretation fields.

Critical checks

1. Keep planned versus executed status explicit; record dead ends; tie every future result to environment and input versions.

Missing output evidence

This overlay adds no new output inspection, execution or validation; consult the referenced canonical source outcome limits.

Failure signals and proposed responses

1. Signal

   A failed provisioning attempt disappears from the record

   Why it matters

   A rerunner misses a necessary constraint

   Proposed response

   Retain the failure and approved alternative rationale.

2. Signal

   Interpretation is written before results exist

   Why it matters

   The notebook implies an outcome

   Proposed response

   Leave result fields empty and label hypotheses.

Effort and cost — unknown

Setup work

1. Define notebook structure and approved execution boundaries.

Ongoing review

1. Researcher reviews plan and later checks actual execution evidence.

Unknowns

1. Environment reproducibility, evaluation validity, compute needs and costs are unknown.

First check — proposed, unexecuted

Status

proposed_unexecuted

Synthetic fixture

Invent prior-run notes with a blocked resource and changed dataset version; draft a reviewed-plan notebook outline.

Acceptance checks

1. Preserve both deviations; leave outputs unexecuted; provision nothing and claim no evaluation result.

Stop conditions

1. Stop if resource authority, dataset scope or review criteria are unresolved.

Limits

1. A notebook plan cannot establish evaluation performance, reproducibility or successful execution.