Apply snappy compression to Configuration Cache entries #29643
Closed
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
|
@bot-gradle test APT |
|
I've triggered the following builds for you. Click here to see all build failures. |
abstratt
reviewed
Jun 21, 2024
| * Determines the maximum memory working set: [bufferCount] * [bufferCapacity]. | ||
| * The default maximum working set is `32MB`. | ||
| */ | ||
| val bufferCount = System.getProperty("org.gradle.configuration-cache.internal.buffer-count", null)?.toInt() |
There was a problem hiding this comment.
Is this right? 1 million buffers that are 32 bit long each? Normally, I'd expect the other way around.
There was a problem hiding this comment.
Is this right? 1 million buffers that are 32 bit long each? Normally, I'd expect the other way around.
Yes, perhaps the term buffer is not the best choice here. A better analogy might be of a packet (like in UDP). These are used to quickly send the encoded information from the producer to the writer thread for compression >> encryption >> disk IO.
There was a problem hiding this comment.
@abstratt Please take another look, I changed the code to the packet analogy.
mlopatkin
reviewed
Jun 21, 2024
...n/configuration-cache/src/main/kotlin/org/gradle/internal/cc/impl/io/ParallelOutputStream.kt
Outdated
Show resolved
Hide resolved
...n/configuration-cache/src/main/kotlin/org/gradle/internal/cc/impl/io/ParallelOutputStream.kt
Show resolved
Hide resolved
...n/configuration-cache/src/main/kotlin/org/gradle/internal/cc/impl/io/ParallelOutputStream.kt
Outdated
Show resolved
Hide resolved
| } finally { | ||
| // in case of failure, this releases some memory until the | ||
| // producer realizes there was a failure | ||
| ready.clear() |
Apply snappy compression to Configuration Cache entries
Compression and encryption are offloaded to a separate thread, utilizing a pool of byte buffers capped at a maximum memory working set of 16MB.
The resulting files are 1/5th of their original size. Storing might be marginally slower in some cases, but loading is as fast or faster in all tested scenarios.
Reviewing cheatsheet
Before merging the PR, comments starting with