Probe documentation adequacy by building from it

Why use this

1. A documentation team may believe an integration guide is complete because experienced developers can fill in its omissions. Asking a code generator to work only from the guide can expose where unstated assumptions make the task impossible to reproduce.

2. A documentation-bounded exercise plan could connect implementation questions and later test failures to specific missing instructions. It should avoid treating one generated solution’s success or failure as conclusive proof of overall documentation quality.

Why an agent / simpler alternative

Useful judgment

Attempt the stated task using only supplied documentation and report where information is insufficient.

Simpler alternative

Use a checklist review when the guide’s required fields and examples are already known.

Unsuitable when

The agent would need hidden expertise, external sources or production credentials to complete the exercise.

Proposed inputs

1. Name

   Frozen documentation set

   Owner

   Documentation owner

   Minimum scope

   Exact guide version and allowed supporting pages

   Quality check

   Confirm the intended audience and prohibit unstated external context.

2. Name

   Integration task and test expectations

   Owner

   Engineering owner

   Minimum scope

   Bounded intended behavior, safe fixture and separately defined acceptance checks

   Quality check

   Ensure tests assess the documented task rather than hidden requirements.

Access and setup — proposed or unknown

Proposed minimum

1. Read-only documentation and an isolated proposed exercise; no code execution initially.

Compatibility unknowns

1. Unknown: exact runtime, dependency availability, test environment and integration interfaces.

External actions requiring approval

1. Installing software, accessing external services or changing shared repositories outside the approved exercise.

Expected deliverable and review checks

Proposed artifact

Candidate-code plan, documentation-question log and proposed test-to-guidance mapping.

Critical checks

1. Every implementation assumption points to allowed documentation or remains explicitly missing.

2. Later failures would be triaged between code, environment and documentation causes.

Missing output evidence

No code generation, dependency installation or test execution occurs in this overlay.

Failure signals and proposed responses

1. Signal

   The agent silently uses knowledge absent from the guide.

   Why it matters

   The exercise no longer tests documentation sufficiency.

   Proposed response

   Record the unsupported assumption and stop that implementation branch.

2. Signal

   An environment failure is labelled a documentation defect.

   Why it matters

   The resulting remediation could target the wrong problem.

   Proposed response

   Separate environmental prerequisites from missing task instructions before drawing conclusions.

Effort and cost — unknown

Setup work

1. Freeze documentation scope and define a safe acceptance fixture.

Ongoing review

1. Classify questions and any later test evidence with engineers.

Unknowns

1. Unknown: setup effort, test coverage and model-dependent behavior.

First check — proposed, unexecuted

Status

proposed_unexecuted

Synthetic fixture

An invented guide describes an endpoint but omits a required authentication configuration; a separate test expectation requires it.

Acceptance checks

1. The missing prerequisite is reported without inventing a credential path.

2. No success or failure verdict is claimed from an unrun test.

Stop conditions

1. External knowledge, credentials or live service access is required.

Limits

1. Editorial documentation probe; a single bounded exercise cannot establish general documentation adequacy.