-
Notifications
You must be signed in to change notification settings - Fork 389
feat: implement parallel operations #2067
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Changes from 1 commit
8717527
22097df
9a59f1f
d2b241f
30b45d3
16a5f05
5a94aa7
1cc8bae
face55f
524a310
bac3ed8
b4bc333
46687c6
721aab6
f5e8121
886dc03
94b1c02
9793cc6
89e8204
c153ab6
4feb1c2
0780e50
c47130b
4c2dda4
aad8f2b
File filter
Filter by extension
Conversations
Jump to
Diff view
Diff view
- Loading branch information
There are no files selected for viewing
Original file line number | Diff line number | Diff line change |
---|---|---|
|
@@ -60,6 +60,14 @@ export interface DownloadMultiCallback { | |
(err: Error | null, contents?: Buffer[]): void; | ||
} | ||
|
||
/** | ||
* Create a TransferManager object to perform parallel transfer operations on a Cloud Storage bucket. | ||
* | ||
* @class | ||
* @hideconstructor | ||
* | ||
* @param {Bucket} bucket A {@link Bucket} instance | ||
*/ | ||
export class TransferManager { | ||
bucket: Bucket; | ||
constructor(bucket: Bucket) { | ||
|
@@ -79,6 +87,60 @@ export class TransferManager { | |
options: UploadMultiOptions, | ||
callback: UploadMultiCallback | ||
): Promise<void>; | ||
/** | ||
* @typedef {object} UploadMultiOptions | ||
* @property {number} [concurrencyLimit] The number of concurrently executing promises | ||
shaffeeullah marked this conversation as resolved.
Show resolved
Hide resolved
|
||
* to use when uploading the files. | ||
* @property {boolean} [skipIfExists] Do not upload the file if it already exists in | ||
* the bucket. This will set the precondition ifGenerationMatch = 0. | ||
* @property {string} [prefix] A prefix to append to all of the uploaded files. | ||
* @property {object} [passthroughOptions] {@link UploadOptions} Options to be passed through | ||
* to each individual upload operation. | ||
*/ | ||
/** | ||
* @typedef {array} UploadResponse | ||
* @property {object} The uploaded {@link File} | ||
* @property {object} The uploaded {@link Metadata} | ||
*/ | ||
/** | ||
* @callback UploadMultiCallback | ||
* @param {?Error} err Rewuest error if any | ||
* @param {array} files Array of uploaded {@link File}. | ||
* @param {array} metadata Array of uploaded {@link Metadata} | ||
*/ | ||
/** | ||
* Upload multiple files in parallel to the bucket. This is a convenience method | ||
* that utilizes {@link Bucket#upload} to perform the upload. | ||
* | ||
* @param {array} [filePaths] An array of fully qualified paths to the files. | ||
* you wish to upload to the bucket | ||
* @param {UploadMultiOptions} [options] Configuration options. | ||
* @param {UploadMultiCallback} [callback] Callback function. | ||
* @returns {Promise<UploadResponse[] | void>} | ||
* | ||
* @example | ||
* ``` | ||
* const {Storage} = require('@google-cloud/storage'); | ||
* const storage = new Storage(); | ||
* const bucket = storage.bucket('my-bucket'); | ||
* const transferManager = new TransferManager(bucket); | ||
* | ||
* //- | ||
* // Upload multiple files. | ||
* //- | ||
* transferManager.uploadMulti(['/local/path/file1.txt, 'local/path/file2.txt'], function(err, files, metadata) { | ||
* // Your bucket now contains: | ||
* // - "file1.txt" (with the contents of '/local/path/file1.txt') | ||
ddelgrosso1 marked this conversation as resolved.
Outdated
Show resolved
Hide resolved
|
||
* // - "file2.txt" (with the contents of '/local/path/file2.txt') | ||
* // `files` is an array of instances of File objects that refers to the new files. | ||
* }); | ||
* | ||
* //- | ||
* // If the callback if omitted, we will return a Promise. | ||
* //- | ||
* const response = transferManager.uploadMulti(['/local/path/file1.txt, 'local/path/file2.txt']); | ||
* ``` | ||
*/ | ||
async uploadMulti( | ||
filePaths: string[], | ||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. [Optional] We have an opportunity to get fancy here to really emphasize throughput; what if we allowed iterators and async iterators? Use cases:
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. We can probably wait until customers ask for this functionality though. There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. It is a good idea, maybe after the initial release we can circle up. I would definitely be interested in your thoughts on optimizations here. |
||
optionsOrCallback?: UploadMultiOptions | UploadMultiCallback, | ||
|
Uh oh!
There was an error while loading. Please reload this page.