Skip to content

Conversation

ijuma
Copy link
Member

@ijuma ijuma commented May 29, 2020

Gradle 6.5 includes a fix for gradle/gradle#12866, which
affects the performance of Scala compilation.

I profiled the scalac build with async profiler and 54% of the time was on GC
even after the Gradle upgrade (it was more than 60% before), so I switched to
the throughput GC (GC latency is less important for batch builds) and it
was reduced to 38%.

I also centralized the jvm configuration in build.gradle and simplified it a bit
by removing the minHeapSize configuration from the test tasks.

On my desktop, the time to execute clean builds with no cached Gradle daemon
was reduced from 127 seconds to 97 seconds. With a cached daemon, it was
reduced from 120 seconds to 88 seconds. The performance regression when
we upgraded to Gradle 6.x was 27 seconds with a cached daemon
(#7677 (comment)), so it
should be fixed now.

Gradle 6.4 with no cached daemon:

BUILD SUCCESSFUL in 2m 7s
115 actionable tasks: 112 executed, 3 up-to-date
./gradlew clean compileScala compileJava compileTestScala compileTestJava  1.15s user 0.12s system 0% cpu 2:08.06 total

Gradle 6.4 with cached daemon:

BUILD SUCCESSFUL in 2m 0s
115 actionable tasks: 111 executed, 4 up-to-date
./gradlew clean compileScala compileJava compileTestScala compileTestJava  0.95s user 0.10s system 0% cpu 2:01.42 total

Gradle 6.5 with no cached daemon:

BUILD SUCCESSFUL in 1m 46s
115 actionable tasks: 111 executed, 4 up-to-date
./gradlew clean compileScala compileJava compileTestScala compileTestJava  1.27s user 0.12s system 1% cpu 1:47.71 total

Gradle 6.5 with cached daemon:

BUILD SUCCESSFUL in 1m 37s
115 actionable tasks: 111 executed, 4 up-to-date
./gradlew clean compileScala compileJava compileTestScala compileTestJava  1.02s user 0.10s system 1% cpu 1:38.31 total

This PR with no cached Gradle daemon:

BUILD SUCCESSFUL in 1m 37s
115 actionable tasks: 81 executed, 34 up-to-date
./gradlew clean compileScala compileJava compileTestScala compileTestJava  1.27s user 0.10s system 1% cpu 1:38.70 total

This PR with cached Gradle daemon:

BUILD SUCCESSFUL in 1m 28s
115 actionable tasks: 111 executed, 4 up-to-date
./gradlew clean compileScala compileJava compileTestScala compileTestJava  1.02s user 0.10s system 1% cpu 1:29.35 total

Committer Checklist (excluded from commit message)

  • Verify design and implementation
  • Verify test coverage and CI build status
  • Verify documentation (including upgrade notes)

@ijuma ijuma marked this pull request as ready for review June 3, 2020 06:27
@ijuma ijuma requested a review from omkreddy June 3, 2020 06:27
@ijuma
Copy link
Member Author

ijuma commented Jun 3, 2020

@omkreddy I'll rerun the numbers with 6.5 final tomorrow morning, but this should be ready for review otherwise.

@ijuma ijuma changed the title MINOR: Update to Gradle 6.5 MINOR: Update to Gradle 6.5 and tweak build jvm config Jun 3, 2020
@ijuma
Copy link
Member Author

ijuma commented Jun 3, 2020

@omkreddy Updated the PR description and made a few more changes.

@ijuma
Copy link
Member Author

ijuma commented Jun 3, 2020

JDK 8/Scala 2.12 had an out of memory issue.

@ijuma
Copy link
Member Author

ijuma commented Jun 3, 2020

retest this please

Copy link
Contributor

@omkreddy omkreddy left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@ijuma Thanks for the PR. LGTM.

@ijuma
Copy link
Member Author

ijuma commented Jun 3, 2020

Build passed.

@ijuma ijuma merged commit 3af3ab9 into apache:trunk Jun 3, 2020
@ijuma ijuma deleted the gradle-6.5 branch June 3, 2020 20:18
@ijuma
Copy link
Member Author

ijuma commented Jun 3, 2020

Cherry picked this to 2.6 as well since it's low risk and fixes a compilation speed regression.

ijuma added a commit that referenced this pull request Jun 3, 2020
Gradle 6.5 includes a fix for gradle/gradle#12866, which
affects the performance of Scala compilation.

I profiled the scalac build with async profiler and 54% of the time was on GC
even after the Gradle upgrade (it was more than 60% before), so I switched to
the throughput GC (GC latency is less important for batch builds) and it
was reduced to 38%.

I also centralized the jvm configuration in `build.gradle` and simplified it a bit
by removing the minHeapSize configuration from the test tasks.

On my desktop, the time to execute clean builds with no cached Gradle daemon
was reduced from 127 seconds to 97 seconds. With a cached daemon, it was
reduced from 120 seconds to 88 seconds. The performance regression when
we upgraded to Gradle 6.x was 27 seconds with a cached daemon 
(#7677 (comment)), so it
should be fixed now.

Gradle 6.4 with no cached daemon:

```
BUILD SUCCESSFUL in 2m 7s
115 actionable tasks: 112 executed, 3 up-to-date
./gradlew clean compileScala compileJava compileTestScala compileTestJava  1.15s user 0.12s system 0% cpu 2:08.06 total
```

Gradle 6.4 with cached daemon:

```
BUILD SUCCESSFUL in 2m 0s
115 actionable tasks: 111 executed, 4 up-to-date
./gradlew clean compileScala compileJava compileTestScala compileTestJava  0.95s user 0.10s system 0% cpu 2:01.42 total
```

Gradle 6.5 with no cached daemon:

```
BUILD SUCCESSFUL in 1m 46s
115 actionable tasks: 111 executed, 4 up-to-date
./gradlew clean compileScala compileJava compileTestScala compileTestJava  1.27s user 0.12s system 1% cpu 1:47.71 total
```

Gradle 6.5 with cached daemon:

```
BUILD SUCCESSFUL in 1m 37s
115 actionable tasks: 111 executed, 4 up-to-date
./gradlew clean compileScala compileJava compileTestScala compileTestJava  1.02s user 0.10s system 1% cpu 1:38.31 total
```

This PR with no cached Gradle daemon:

```
BUILD SUCCESSFUL in 1m 37s
115 actionable tasks: 81 executed, 34 up-to-date
./gradlew clean compileScala compileJava compileTestScala compileTestJava  1.27s user 0.10s system 1% cpu 1:38.70 total
```

This PR with cached Gradle daemon:

```
BUILD SUCCESSFUL in 1m 28s
115 actionable tasks: 111 executed, 4 up-to-date
./gradlew clean compileScala compileJava compileTestScala compileTestJava  1.02s user 0.10s system 1% cpu 1:29.35 total
```

Reviewers: Manikumar Reddy <[email protected]>, Chia-Ping Tsai <[email protected]>
ijuma added a commit that referenced this pull request Jun 4, 2020
…use_all_dns_ips-as-default

* apache-github/trunk:
  KAFKA-9788; Use distinct names for transaction and group load time sensors (#8784)
  KAFKA-9514; The protocol generator generated useless condition when a field is made nullable and flexible version is used (#8793)
  MINOR: Update to Gradle 6.5 and tweak build jvm config (#8751)
  MINOR: Upgrade spotbugs and spotbugsPlugin (#8790)
  KAFKA-10089 The stale ssl engine factory is not closed after reconfigure (#8792)
  KAFKA-10080; Fix race condition on txn completion which can cause duplicate appends (#8782)
  KAFKA-10084: Fix EosTestDriver end offset (#8785)
  KAFKA-10083: fix failed testReassignmentWithRandomSubscriptionsAndChanges tests (#8786)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants