Why does Codacy show unexpected coverage changes?#

You may encounter some situations where Codacy shows unexpected drops in coverage, potentially causing your quality gates to fail. Usually, these drops in coverage happen in files that the commit or pull request didn't change.

There are multiple reasons for this, but it's important to understand that each coverage report that you upload to Codacy contains information about which lines of code in your repository are tested or not in a specific commit. In particular, each coverage report provides the following information about the lines of your source code files:

Coverable lines (lines that can be tested), by listing those lines
Covered lines (lines that were tested at least once), by marking those lines as tested or having a number of test hits
Not coverable lines (lines that can't be tested), by not listing those lines

For example, the coverage report represented below includes coverage information for two source code files:

File ClassA.java has two coverable lines and all are covered by tests
File ClassB.java has three coverable lines but only line 1 is covered by tests

File	Line number	Covered by tests?
ClassA.java	2	Yes
ClassA.java	4	Yes
ClassB.java	1	Yes
	3	No
	11	No

Based on the information obtained from the coverage reports, Codacy calculates code coverage as follows:

The coverage for a file, commit, or pull request is the percentage of covered lines in the universe of coverable lines for that file, commit, or pull request. For example, a commit with 85 covered lines out of a total of 100 coverable lines has 85% coverage.

The coverage variation of a commit or pull request is the increase or drop in the percentage of coverable lines that are covered by tests in the repository because of the changes of the commit or pull request.
The diff coverage of a pull request is the percentage of coverable lines that the pull request added or modified that are covered by tests.

If a pull request doesn't add or modify any coverable lines, the diff coverage is ∅ (not applicable). This scenario happens when the only changes in a pull request are:
- Deleted lines
- Added or modified lines that aren't coverable

Important

Note that changes external to a source code file can affect the lines that are or aren't covered in that file.

Common reasons for unexpected coverage changes#

There are several reasons that could cause Codacy to report unexpected coverage results, from changes to your source code to external factors. The following is a non-exhaustive list of the most common reasons:

Adding new tests or removing existing tests from your project.

Even small and localized changes to your tests can have an impact on the amount of covered lines across your repository.
Changing the logic of your application or tests.

Changing the control flow of your application or tests can mean that different areas of your code start or stop being covered by tests. For example, inverting the result of the Boolean expression of an if statement means that a different branch of your code could now be tested.
Failing to upload coverage reports, or uploading a different number of reports between commits.

This can be caused by a failed step in your CI/CD pipeline, for example. In the case of pull requests, you should make sure that you upload all relevant coverage reports for both the common ancestor commit and the head commit of the pull request branch.
Ignoring files on Codacy.

Updating the list of ignored files can have an impact on the amount of coverable and covered lines of the commits that Codacy compares to calculate the coverage variation metric.
External factors affecting the execution of tests.

A variety of factors that are external to your code can affect the execution of tests and, consequently, the results contained in the coverage reports. A few examples of these external factors are:
- Updates to dependencies of your project that could result in different test execution paths
- Misconfiguration of repository secrets that could prevent some test execution paths
- Tests that are dependent on time, such as running test cases only on specific dates or times of the day
- "Flaky" tests caused by any inconsistent or unreliable behavior of your code, infrastructure, or third-party services

The examples below describe in more detail how specific changes in your code impact the coverage metrics that Codacy calculates.

Example: Diff coverage is 100% but pull request coverage variation is negative#

Consider an example pull request where Codacy shows the following metrics:

100% diff coverage
A negative coverage variation

Diff coverage is 100% but pull request coverage variation is negative

There are two possible scenarios that could cause this result:

Removing covered lines or tests

Since diff coverage only applies to covered lines that the pull request added or modified, removed lines don't affect the diff coverage metric. However, removing covered lines or tests means that there are now less covered lines in the repository, causing a drop in coverage.
Application logic changes

A change in the flow of execution of your application or tests can mean that a different number of coverable lines in your repository are now covered by tests, causing a drop in coverage. However, if all lines modified in the pull request continue to be covered, the diff coverage metric is 100%.

The table below represents two example coverage reports reflecting a pull request that causes line 1 of the file ClassB.java to stop being covered:

	Common ancestor commit		Head commit
File	Line number	Covered by tests?	Line number	Covered by tests?
ClassA.java	2	Yes	2	Yes
ClassA.java	4	Yes	4	Yes
ClassB.java	1	Yes	1	No
	3	No	3	No
	11	No	11	No

The table below displays the code coverage metrics as calculated by Codacy:

Initially, ClassB.java was 33.33% covered but after the changes from the pull request the file is now 0% covered, causing the repository coverage to drop 20%
As long as the pull request also modified any covered line that continues to be covered, the diff coverage is 100%

	Common ancestor commit			Head commit			Pull request results
File	Coverable lines	Covered lines	Coverage	Coverable lines	Covered lines	Coverage	Coverage variation	Diff coverage
ClassA.java	2	2	100%	2	2	100%	0%
ClassB.java	3	1	33.33%	3	0	0%	-33.33%
Total	5	3	60%	5	2	40%	-20%	100%

Example: Pull request coverage variation is negative but no files have coverage variation#

Consider an example pull request where Codacy shows the following metrics:

Negative coverage variation
There aren't any files with coverage variation

Pull request coverage variation is negative but no files have coverage variation

Removing covered lines from a file that had 100% coverage means that the file will continue to have 100% coverage. All lines in the file continue to be covered, even though there are now less covered lines. As such, there is no coverage variation for the file.

However, since the proportion between the total number of covered and coverable lines across all files in the repository is now different, there can be a drop in the coverage variation for the pull request.

The table below represents two example coverage reports reflecting a pull request that removes lines 5 and 6 of the file ClassA.java:

	Common ancestor commit		Head commit
File	Line number	Covered by tests?	Line number	Covered by tests?
ClassA.java	2	Yes	2	Yes
	4	Yes	4	Yes
	5	Yes

	6	Yes
ClassB.java	1	Yes	1	Yes
	3	No	3	No
	11	No	11	No

The table below displays the code coverage metrics as calculated by Codacy:

Initially, ClassA.java was 100% covered and continues to be 100% covered after the pull request, causing the coverage variation for the file to be 0%
However, there were 62.5% lines covered across all files in the repository but after the pull request only 60% of the lines are now covered, causing the pull request coverage variation to drop 2.5%

	Common ancestor commit			Head commit			Pull request results
File	Coverable lines	Covered lines	Coverage	Coverable lines	Covered lines	Coverage	Coverage variation
ClassA.java	4	4	100%	2	2	100%	0%
ClassB.java	3	1	33.33%	3	1	33.33%	0%
Total	7	5	62.5%	5	3	60%	-2.5%

Why does Codacy show unexpected coverage changes?#

Common reasons for unexpected coverage changes#

Example: Diff coverage is 100% but pull request coverage variation is negative#

Example: Pull request coverage variation is negative but no files have coverage variation#

See also#

Share your feedback 📢