Skip to content

Conversation

BenWhitehead
Copy link
Collaborator

They still retain their public factory methods, just the constructors are now private -- they should have been to begin with.

@BenWhitehead BenWhitehead requested review from a team as code owners August 5, 2025 19:25
@product-auto-label product-auto-label bot added size: s Pull request size is small. api: storage Issues related to the googleapis/java-storage API. labels Aug 5, 2025
They still retain their public factory methods, just the constructors are now private -- they should have been to begin with.
@BenWhitehead BenWhitehead force-pushed the nonblocking-appendable/01/fix-flush-config-scope branch from 6e74313 to 71edb38 Compare August 5, 2025 19:50
BenWhitehead added a commit that referenced this pull request Aug 20, 2025
## Description
feat: *breaking behavior* rewrite Storage.blobAppendableUpload to be non-blocking and have improved throughput (#3231)

Rewrite internals of BlobAppendableUpload to provide non-blocking write calls, and it take advantage of grpc async message handling.

When `AppendableUploadWriteableByteChannel#write(ByteBuffer)` is called, an attempt will be made to enqueue the bytes in the outbound queue to GCS.
If there is only enough room to partially consume the bytes provided in the `ByteBuffer` the write call will return early specifying the number of bytes actually consumed.

As acknowledgements come in from gcs, enqueued messages will be evicted freeing space in the outbound queue. Thereby allowing more bytes to be consumed and enqueued.

Given appendable objects are still in private preview I can't quote any metrics here, however preliminary benchmarking of several million objects across a range of sizes show across the board throughput improvments.

Because the channel's write call is now non-blocking, if you want to block your application until the full buffer is consumed some new helper methods have been added in StorageChannelUtils to provide blocking behavior.

A new method `MinFlushSizeFlushPolicy#withMaxPendingBytes(long)` has been added to allow limiting the number of pending outbound bytes. The default values is 16MiB, but can be configured lower if necessary.

## Release Notes

BEGIN_COMMIT_OVERRIDE

BEGIN_NESTED_COMMIT
feat: *breaking behavior* rewrite Storage.blobAppendableUpload to be non-blocking and have improved throughput (#3231)
END_NESTED_COMMIT

BEGIN_NESTED_COMMIT
feat: add StorageChannelUtils to provide helper methods to perform blocking read/write to/from non-blocking channels (#3231)
END_NESTED_COMMIT

BEGIN_NESTED_COMMIT
feat: add MinFlushSizeFlushPolicy#withMaxPendingBytes(long) (#3231)
END_NESTED_COMMIT

BEGIN_NESTED_COMMIT
fix: update BlobAppendableUploadConfig and FlushPolicy.MinFlushSizeFlushPolicy to default to 4MiB minFlushSize and 16MiB maxPendingBytes (#3249)
END_NESTED_COMMIT

BEGIN_NESTED_COMMIT
fix: make FlushPolicy${Min,Max}FlushSizeFlushPolicy constructors private (#3217)
END_NESTED_COMMIT

END_COMMIMT_OVERRIDE

## Sub PRs
This PR is made of up the following PRs, in sequence
1. #3217
2. #3218 
3. #3219
4. #3220
5. #3221
6. #3222
7. #3223
8. #3224 
9. #3225 
10. #3226 
11. #3227 
12. #3228 
13. #3229 
14. #3230 
15. #3235 
16. #3236 
17. #3241
18. #3242
19. #3246
20. #3248
21. #3249
22. #3252
@BenWhitehead
Copy link
Collaborator Author

Merged in #3231

@BenWhitehead BenWhitehead deleted the nonblocking-appendable/01/fix-flush-config-scope branch August 20, 2025 21:13
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
api: storage Issues related to the googleapis/java-storage API. size: s Pull request size is small.
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants