Skip to content

Conversation

BenWhitehead
Copy link
Collaborator

@BenWhitehead BenWhitehead commented Jul 18, 2025

Merge train for

  1. chore: cleanup Crc32cValue and associated Hasher operations #3167
  2. chore: add Hasher#defaultHasher() #3173
  3. chore: add UnifiedOpts.Opts#getHasher() to return a hasher relative to the set of Opts defined for an operation. #3174
  4. feat: add default end-to-end crc32c checksumming for several upload methods via grpc transport #3176
  5. feat: add default end-to-end checksumming for JournalingBlobWriteSessionConfig #3180
  6. fix: give user provided checksum precondition priority for Storage#create methods that accept byte[] #3182
  7. fix: move crc32c computation before writing to disk for BufferToDiskThenUpload BlobWriteSession config #3187

BEGIN_COMMIT_OVERRIDE

BEGIN_NESTED_COMMIT
feat: add default end-to-end crc32c checksumming for several upload methods via grpc transport #3176

#3176
END_NESTED_COMMIT

BEGIN_NESTED_COMMIT
feat: add default end-to-end checksumming for JournalingBlobWriteSessionConfig #3180

#3180
END_NESTED_COMMIT

BEGIN_NESTED_COMMIT
fix: give user provided checksum precondition priority for Storage#create methods that accept byte[] #3182

#3182
END_NESTED_COMMIT

BEGIN_NESTED_COMMIT
fix: move crc32c computation before writing to disk for BufferToDiskThenUpload BlobWriteSession config #3187

#3187
END_NESTED_COMMIT

END_COMMIT_OVERRIDE

* Add `Crc32cValue#zero()`
  * `Crc32cValue.zero().concat(v) == v`
* Update WriteCtx to use `Crc32cValue.zero()` as it's base values rather than `null`
* Update Hasher.nullSafeConcat to account for the use of `zero`
  * if Hasher.noop() always return null
  * if Hasher.enabled() only return null if left side is null
* Add nullability annotation to Crc32cValue.nullSafeConcat
…initial crc32c value

The desired default will be `Crc32cValue.zero()`, but not all tests or uploads are ready to handle this. Stick with the existing null as default, and provide the overload to selectively opt into things incrementally.
By default, it will be the same as Hasher.enabled(). If running on java < 9, a log message will be logged making it apparent there isn't a fast implementation of crc32c.

A system property `com.google.cloud.storage.Hasher.default` can be set to "disabled" to instead make the default Hasher.noop().
…o the set of Opts defined for an operation.

Add `UnifiedOpts.HasherSelector` (also add `UnifiedOptions.BucketObjectHmacKeyAllOpt` which extends all ``{Bucket,Object,HmacKey}{Target,List,Source}Opt` and provides an anchor for those opts which apply to all 9 permutations [UserProject & Headers])

Add Opts#getHasher() that will return Hasher.defaultInstance() or a hasher defined based on the last occurring HasherProvider present in the instance of Opts.

Both Crc32cMatch and Md5Match are configured to return Hasher.noop(). If an explicit checksum value is provided by the customer, use it and disable our automatic crc32c calculation.
…via grpc transport

* Storage#blobWriteSession for BlobWriteSessionConfigs.getDefault()
* Storage#blobWriteSession for BlobWriteSessionConfigs.bufferToTempDirThenUpload()
* Storage#blobWriteSession for BlobWriteSessionConfigs.bufferToDiskThenUpload(*)
* Storage#create(BlobInfo, InputStream)
* Storage#createFrom(BlobInfo, InputStream)
* Storage#createFrom(BlobInfo, Path)
* Storage#writer
…ionConfig

Create TestUtils.rmDashRm to recursively delete a directory, and update ITSyncAndUploadUnbufferedWritableByteChannelPropertyTest to use it.

Add TmpDir auto closable to allow managed lifecycle of a temporary directory in a test.

Remove md5Base64 from ChecksummedTestContent#toString(). Base64 values can contain path values that are not valid for filesystems.

Add ITObjectChecksumSupportTest for journaling uploads.

Update ITSyncAndUploadUnbufferedWritableByteChannelPropertyTest property tests to enable and expect crc32c values in all messages.
…eate methods that accept byte[]

* `Storage#create(BlobInfo, BlobTargetOption...)`
* `Storage#create(BlobInfo, byte[], BlobTargetOption...)`
* `Storage#create(BlobInfo, byte[], int, int, BlobTargetOption...)`

If no user provided checksum is specified, crc32c will still be calculated.

Applies to both `StorageOptions.http()` and `StorageOptions.grpc()` instances.
@product-auto-label product-auto-label bot added size: xl Pull request size is extra large. api: storage Issues related to the googleapis/java-storage API. labels Jul 18, 2025
@BenWhitehead BenWhitehead marked this pull request as ready for review July 18, 2025 17:07
@BenWhitehead BenWhitehead requested a review from a team as a code owner July 18, 2025 17:07
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
api: storage Issues related to the googleapis/java-storage API. size: xl Pull request size is extra large.
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants