Expected Behavior

Gradle's incremental java compiler should only re-compile unchanged java files if there is a ABI-breaking change in one of the classes that java file depends on.

Current Behavior (optional)

Gradle's incremental java compiler re-compiles unchanged java files any time one of the classes that java file depends on has changed, regardless of whether that change is ABI-compatible.

Context

Gradle already has compilation avoidance at the project level. This means that compilation for a project can be skipped entirely as long as the project's compile classpath has no ABI-breaking changes.

Gradle also already has an incremental compiler. The incremental compiler "analyzes the dependencies between classes, and only recompiles a class when it has changed, or one of the classes it depends on has changed." (From https://blog.gradle.org/incremental-compiler-avoidance). This works whether the changed class is on the compile classpath (i.e. an external or inter-project dependency), or the changed class is within the project.

The former takes ABI-compatibility into account but the latter does not. This leads to some counter-intuitive performance characterstics. For example, Gradle can only take advantage of compile avoidance for inter-class dependencies across projects, but not inter-class dependencies within a project. And even when classes are in a separate project from their dependencies, Gradle still may not be able to take advantage of compile avoidance depending on the composition and order of changes made. I created a sample project to demonstrate this: incremental-compile-test.tgz

The project structure looks like this:

./producers/build.gradle
./producers/src/main/java/com/example/ChangingAbiProducer.java
./producers/src/main/java/com/example/StableAbiProducer.java
./consumers/build.gradle
./consumers/src/main/java/com/example/ChangingAbiConsumer.java
./consumers/src/main/java/com/example/StableAbiConsumer.java
./gradle/wrapper/gradle-wrapper.jar
./gradle/wrapper/gradle-wrapper.properties
./gradlew
./gradlew.bat
./settings.gradle
./build.gradle
./generate-consumers.sh
./remove-consumers.sh

In the sample project, ChangingAbiProducer is used to simulate ABI-breaking changes (by changing a method signature), and StableAbiProducer is used to simulate ABI-compatible changes (by changing a method implementation). ChangingAbiConsumer depends on the ChangingAbiProducer, and StableAbiConsumer depends on StableAbiProducer. To figure out what classes are being recompiled, I check the class file timestamps. I'm also enabling info logging to be able to see how many classes are recompiled in each sub-project by looking at this message:

Incremental compilation of <N> classes completed in <M> secs

One interesting side-note is that in Gradle 8.3 it seems like the class count is always off by one. When I do a full compile of a project with 2 classes, it says 3 classes were compiled. If I increase the class count to 1000, it says 1001 classes were compiled.

Scenario 1

1. ./gradlew clean compileJava (to ensure everything is up to date)
2. Change the method implementation in StableAbiProducer, and change the method signature in ChangingAbiProducer.
3. ./gradlew --no-build-cache --info compileJava

Expected: Both producers are recompiled, and only ChangingAbiConsumer is recompiled.
Actual: Both producers and both consumers are recompiled.

Scenario 2

1. ./gradlew clean compileJava (to ensure everything is up to date)
2. Change only the method implementation in StableAbiProducer
3. ./gradlew --no-build-cache --info compileJava
4. Change only the method signature in ChangingAbiProducer
5. ./gradlew --no-build-cache --info compileJava

Expected: At step 3, only StableAbiProducer is recompiled. At step 5, only ChangingAbiProducer and ChangingAbiConsumer are recompiled.
Actual: At step 3, only StableAbiProducer is recompiled. At step 5, ChangingAbiProducer and both consumers are recompiled.

Scenario 3 (Note this is identical to scenario 2, with steps 2 and 4 swapped)

1. ./gradlew clean compileJava (to ensure everything is up to date)
2. Change only the method signature in ChangingAbiProducer
3. ./gradlew --no-build-cache --info compileJava
4. Change only the method implementation in StableAbiProducer
5. ./gradlew --no-build-cache --info compileJava

Expected: At step 3, only ChangingAbiProducer and ChangingAbiConsumer are recompiled. At step 5, only StableAbiProducer is recompiled.
Actual: Works as expected

Note that scenarios 1, 2, and 3 make the same changes to the same two files. None of the scenarios should require recompiling StableAbiConsumer (because it only consumes stable ABIs). However, in scenarios 1 and 2, Gradle needlessly recompiled StableAbiConsumer. Only in scenario 3 was Gradle able to skip recompiling StableAbiConsumer.

Here is my best guess as to what's going on internally.

In scenario 1: Project-level compile avoidance does not apply to consumers because producers has a mix of ABI-breaking and ABI-compatible changes. Control is passed to the incremental compiler. The incremental compiler sees that both Producer classes changed since consumers was last compiled. Because the incremental compiler does not consider ABI-compatibility it doesn't take into account that the change to StableAbiProducer was an ABI-compatible change and simply compiles both Consumers.

In scenario 2: On the first round of compilation, project-level compile avoidance does apply to consumers because producers only had ABI-compatible changes. The compileJava task is skipped entirely. Because compilation is skipped entirely, no new compilation data is recorded. The second round of compilation plays out exactly the same as scenario 1. That is, the incremental compiler acts as if both Producers changed. This is because both Producers have changed since the last time compilation data was recorded, even though only one changed since the last time compilation was successful.

In scenario 3: On the first round of compilation, project-level compile avoidance does not apply to consumers because producers had an ABI-breaking change. Control is passed to the incremental compiler. The incremental compiler sees that only ChangingAbiProducer has changed. Since ChangingAbiConsumer is the only class that depends on ChangingAbiProducer, only ChangingAbiConsumer is recompiled. This also generates up-to-date compilation data. On the second round of compilation, project-level compile avoidance does apply, and the compileJava task is skipped entirely.

The end result is that some compilation cycles take longer than necessary, and the only way to avoid this is to compile ABI-breaking/ABI-compatible changes separately, and in the correct order. Since this will sometimes happen naturally, and other times not happen naturally, I would guess most people just notice their compile cycles "randomly" taking longer than expected. It's hardly noticeable on my small sample project. However I work on some very large applications where this is very noticeable when it happens. The sample project also includes a bash script to generate additional StableAbiConsumers to demonstrate this effect.

I think a good way to address this, and improve compile performance in general, is to have the incremental compiler take ABI-compatibility into account when it is detecting what classes have changed, similar to how the project-level compile avoidance does. Ideally, this should work for inter-class dependencies within a single project, as well as inter-class dependencies between projects.