Flaky Test Detector for CI Pipelines

WHAT IT DOES

Flaky Test Detector for CI Pipelines is a developer-focused tool that analyzes test run logs from continuous integration pipelines to identify flaky tests—tests that produce inconsistent results across multiple runs. It takes test execution logs with multiple runs as input, parses each run to extract test names and pass/fail statuses, then tracks the results for each test across all runs. The tool calculates pass rates, identifies tests that have both passes and failures (flaky), tests that always fail (broken), and tests that always pass (stable). Results are displayed with a summary showing the total number of tests analyzed, stable tests, and flaky/broken tests, along with detailed reports for each test including pass rate, failure count, and recommendations. This enables developers and QA engineers to identify and address non-deterministic test behavior in their CI pipelines.

HOW TO USE

Paste test run logs into the input area, separating each run with "=== RUN N ===". Each test line should indicate pass/fail using either emoji (✅/❌) or text (PASS/FAIL). Example format: "=== RUN 1 ===\n✅ test/MyTest.js: should transfer tokens (passed)\n❌ test/MyTest.js: should mint tokens (failed)". Click "Detect flaky tests" to analyze the logs. The tool will display a summary with the number of tests analyzed, stable tests, flaky tests, and test runs. Detailed results show each flaky or broken test with its pass rate, pass/fail counts, and recommendations for addressing flakiness.

TECHNICAL NOTE: PATTERN-BASED TEST LOG PARSING

The tool uses a multi-format parser to extract test names and results from log lines. It recognizes several common test output formats: emoji indicators (✅ for pass, ❌ for fail), text indicators (PASS/FAIL, passed/failed), and structured patterns like "should [action]" for test names. The tool splits the logs into separate test runs using the "=== RUN N ===" delimiter, then tracks each test's results across all runs. For each test, it calculates the pass rate and classifies the test as stable (100% pass), flaky (both passes and failures), or broken (0% pass). The detection threshold is based on the actual results pattern, not a fixed percentage.

WHAT IT CANNOT SEE

The tool cannot automatically detect flakiness without historical test run data from multiple runs. It is limited to the accuracy of the provided test logs; logs with incomplete or inconsistent formatting may not parse correctly. It cannot differentiate between flaky tests and legitimate failures without multiple runs. It cannot detect flakiness caused by external factors (network instability, RPC timeouts, environment differences) without detailed logs containing error messages. The tool is limited to the test frameworks and formats supported by the parser. It cannot detect flaky tests on non-EVM chains.

PLEASE NOTE

This tool is designed for analyzing test logs from EVM smart contract CI pipelines but can be used with any test logs in the supported format. The tool is read-only and does not execute tests or modify any state. Detection accuracy depends on the quality and completeness of the test logs. The tool provides recommendations but does not automatically fix flaky tests. Always verify flagged tests manually and investigate the root cause. The tool provides no warranty that the detection results are complete or accurate; users should supplement with additional analysis.