Test Coverage Regression Checker in CI

WHAT IT DOES

The Test Coverage Regression Checker in CI is a developer tool that analyzes test coverage reports to detect coverage regressions in CI environments. It parses coverage report JSON files from Hardhat coverage, Foundry coverage, or other EVM testing frameworks, extracts coverage percentages per contract or file, and compares against a baseline report. The tool identifies contracts where coverage has decreased, helping teams catch test quality regressions early in the development cycle and ensure that new code changes do not reduce overall test coverage.

HOW TO USE

Paste your current coverage report JSON into the first text area and your baseline coverage report into the second text area. Click "parse current" and "parse baseline" to extract coverage data from each report. Click "check regressions" to compare coverage percentages. The tool displays a detailed report showing each contract's current coverage, baseline coverage, absolute percentage difference, and status. Contracts with coverage drops exceeding the 0.5% threshold are flagged as regressions. The report also includes visual coverage bars and indicates whether the CI check would pass or fail.

TECHNICAL MECHANISM

The tool parses coverage report JSON using format detection that supports multiple common structures. It handles Hardhat coverage's "contracts" object structure with nested contract data, flat arrays of contract objects, and simple key-value objects. For each detected contract or file, it extracts the contract name and coverage percentage. The tool also supports line-level coverage data by calculating coverage from line hit counts when percentage data is not directly available. The comparison algorithm calculates the absolute coverage difference. A contract is flagged as a regression if coverage decreased by more than 0.5% of the baseline value, preventing noise from minor fluctuations. The tool also identifies improvements (increased coverage) and new contracts without baselines. Results are displayed in a sortable table with color-coded status indicators and visual coverage bars showing the current coverage level. The tool provides a summary of total regressions detected and a CI exit code recommendation.

WHAT IT CANNOT SEE

The tool cannot run tests or generate coverage measurements—it only analyzes already-collected data. It cannot detect coverage decreases caused by changes in the test suite, test conditions, or test environment. The tool cannot automatically update baseline coverage. It cannot determine if a coverage decrease is acceptable without manual review. The tool cannot analyze which specific lines or branches lost coverage—only aggregate coverage percentages are compared. It cannot differentiate between changes in contract code and changes in test coverage. The tool cannot validate that coverage measurements are accurate or reproducible across different runs or environments. It cannot account for differences between coverage reporting tools.

PLEASE NOTE

This tool is designed for Ethereum Virtual Machine (EVM) compatible chains but works with any test coverage report in the supported JSON formats. The 0.5% regression threshold is a reasonable default for catching meaningful coverage drops while ignoring noise. The tool requires coverage reports in JSON format—ensure your testing framework is configured to output JSON reports. The tool is intended for CI/CD pipelines and local development testing. Always run coverage reports under consistent conditions (same test suite, same environment) to ensure accurate comparisons. The tool provides recommendations only and should be used as part of a comprehensive quality assurance process.