-
Notifications
You must be signed in to change notification settings - Fork 85
chore: integration PR to validate merge train against main
and run full integration suite for e2e-crc32c-media
#3197
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Merged
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
* Add `Crc32cValue#zero()` * `Crc32cValue.zero().concat(v) == v` * Update WriteCtx to use `Crc32cValue.zero()` as it's base values rather than `null` * Update Hasher.nullSafeConcat to account for the use of `zero` * if Hasher.noop() always return null * if Hasher.enabled() only return null if left side is null * Add nullability annotation to Crc32cValue.nullSafeConcat
… as it's base values rather than `null`'
…initial crc32c value The desired default will be `Crc32cValue.zero()`, but not all tests or uploads are ready to handle this. Stick with the existing null as default, and provide the overload to selectively opt into things incrementally.
By default, it will be the same as Hasher.enabled(). If running on java < 9, a log message will be logged making it apparent there isn't a fast implementation of crc32c. A system property `com.google.cloud.storage.Hasher.default` can be set to "disabled" to instead make the default Hasher.noop().
…o the set of Opts defined for an operation. Add `UnifiedOpts.HasherSelector` (also add `UnifiedOptions.BucketObjectHmacKeyAllOpt` which extends all ``{Bucket,Object,HmacKey}{Target,List,Source}Opt` and provides an anchor for those opts which apply to all 9 permutations [UserProject & Headers]) Add Opts#getHasher() that will return Hasher.defaultInstance() or a hasher defined based on the last occurring HasherProvider present in the instance of Opts. Both Crc32cMatch and Md5Match are configured to return Hasher.noop(). If an explicit checksum value is provided by the customer, use it and disable our automatic crc32c calculation.
…via grpc transport * Storage#blobWriteSession for BlobWriteSessionConfigs.getDefault() * Storage#blobWriteSession for BlobWriteSessionConfigs.bufferToTempDirThenUpload() * Storage#blobWriteSession for BlobWriteSessionConfigs.bufferToDiskThenUpload(*) * Storage#create(BlobInfo, InputStream) * Storage#createFrom(BlobInfo, InputStream) * Storage#createFrom(BlobInfo, Path) * Storage#writer
…ionConfig Create TestUtils.rmDashRm to recursively delete a directory, and update ITSyncAndUploadUnbufferedWritableByteChannelPropertyTest to use it. Add TmpDir auto closable to allow managed lifecycle of a temporary directory in a test. Remove md5Base64 from ChecksummedTestContent#toString(). Base64 values can contain path values that are not valid for filesystems. Add ITObjectChecksumSupportTest for journaling uploads. Update ITSyncAndUploadUnbufferedWritableByteChannelPropertyTest property tests to enable and expect crc32c values in all messages.
…eate methods that accept byte[] * `Storage#create(BlobInfo, BlobTargetOption...)` * `Storage#create(BlobInfo, byte[], BlobTargetOption...)` * `Storage#create(BlobInfo, byte[], int, int, BlobTargetOption...)` If no user provided checksum is specified, crc32c will still be calculated. Applies to both `StorageOptions.http()` and `StorageOptions.grpc()` instances.
…henUpload BlobWriteSession config
ddelgrosso1
approved these changes
Jul 18, 2025
This was referenced Jul 23, 2025
feat: add default end-to-end crc32c checksumming for several upload methods via grpc transport
#3176
Closed
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Labels
api: storage
Issues related to the googleapis/java-storage API.
size: xl
Pull request size is extra large.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Merge train for
BEGIN_COMMIT_OVERRIDE
BEGIN_NESTED_COMMIT
feat: add default end-to-end crc32c checksumming for several upload methods via grpc transport #3176
#3176
END_NESTED_COMMIT
BEGIN_NESTED_COMMIT
feat: add default end-to-end checksumming for JournalingBlobWriteSessionConfig #3180
#3180
END_NESTED_COMMIT
BEGIN_NESTED_COMMIT
fix: give user provided checksum precondition priority for Storage#create methods that accept byte[] #3182
#3182
END_NESTED_COMMIT
BEGIN_NESTED_COMMIT
fix: move crc32c computation before writing to disk for BufferToDiskThenUpload BlobWriteSession config #3187
#3187
END_NESTED_COMMIT
END_COMMIT_OVERRIDE