Find recurring flaky tests from CI history

Why use this

1. A team facing failed CI runs may spend time on tests that pass on retry while overlooking consistent regressions. Similar error messages across different commits or environments can make unrelated failures appear to be one flaky test.

2. A source-linked investigation queue could separate intermittent evidence from incomplete or deterministic failures. Maintainers could prioritize review using traceable run context without automatically disabling tests or creating high-priority work items.

Why an agent / simpler alternative

Useful judgment

Cluster failure signatures while preserving commit, environment and test-path differences.

Simpler alternative

Use CI rerun reports when the platform already supplies reliable per-test history.

Unsuitable when

Pass/fail comparisons lack enough context to distinguish code changes from intermittency.

Proposed inputs

1. Name

   Bounded CI history

   Owner

   Repository owner

   Minimum scope

   Relevant run IDs, outcomes, commits, environments and test paths

   Quality check

   Include passing evidence where available, not only failure logs.

2. Name

   Prioritization conventions

   Owner

   Maintainer

   Minimum scope

   Approved impact indicators and existing issue references

   Quality check

   Distinguish observed reruns from estimated developer cost.

Access and setup — proposed or unknown

Proposed minimum

1. Read-only CI exports and an offline remediation queue.

Compatibility unknowns

1. Unknown: exact runtime, GitHub/log access, retention and test-identity consistency.

External actions requiring approval

1. Creating tasks, rerunning jobs, quarantining tests or changing repository configuration.

Expected deliverable and review checks

Proposed artifact

Ranked investigation candidates with signatures, supporting runs and alternative explanations.

Critical checks

1. Intermittency claims compare appropriate code/environment conditions.

2. Queue ranking separates measured evidence from inferred impact.

Missing output evidence

No new workflow run or output inspection was performed for this overlay.

Failure signals and proposed responses

1. Signal

   A failure before a code fix is compared with a pass afterward.

   Why it matters

   A real regression could be mislabeled flaky.

   Proposed response

   Separate revisions and require comparable-state evidence.

2. Signal

   Different tests emit the same generic timeout text.

   Why it matters

   One cluster may hide distinct causes.

   Proposed response

   Retain test paths and environment dimensions before grouping.

Effort and cost — unknown

Setup work

1. Choose history bounds and normalize run/test identifiers.

Ongoing review

1. Validate candidate clusters and decide on remediation separately.

Unknowns

1. Unknown: log completeness, investigation effort and true rerun impact.

First check — proposed, unexecuted

Status

proposed_unexecuted

Synthetic fixture

Invented runs contain a fail/pass across different commits and matching timeout messages from separate test paths.

Acceptance checks

1. Neither pattern alone proves one flaky test.

2. The queue preserves the missing evidence questions.

Stop conditions

1. A rerun, issue creation or test change is required.

Limits

1. Editorial CI triage; a manifest comparison is not a test run or verified flakiness diagnosis.