Skip to content
Merged
Changes from 1 commit
Commits
Show all changes
25 commits
Select commit Hold shift + click to select a range
8717527
feat: implement parallel operations
ddelgrosso1 Aug 15, 2022
22097df
add more parallel operations
ddelgrosso1 Sep 15, 2022
9a59f1f
add header to test file
ddelgrosso1 Sep 15, 2022
d2b241f
update import of fs/promises
ddelgrosso1 Sep 15, 2022
30b45d3
fix pathing on windows, fix mocking of fs promises
ddelgrosso1 Sep 16, 2022
16a5f05
add jsdoc headers to class and uploadMulti
ddelgrosso1 Sep 16, 2022
5a94aa7
add jsdoc comments to remaining functions
ddelgrosso1 Sep 30, 2022
1cc8bae
update comment wording
ddelgrosso1 Oct 24, 2022
face55f
add experimental jsdoc tags
ddelgrosso1 Oct 27, 2022
524a310
feat: add directory generator to performance test framework
ddelgrosso1 Nov 3, 2022
bac3ed8
clarify variable names and comments
ddelgrosso1 Nov 3, 2022
b4bc333
capitalization
ddelgrosso1 Nov 3, 2022
46687c6
wip: transfer manager performance tests
ddelgrosso1 Nov 10, 2022
721aab6
feat: merged in application performance tests (#2100)
shaffeeullah Nov 10, 2022
f5e8121
fix: fixed many bugs (#2102)
shaffeeullah Nov 15, 2022
886dc03
fix: more work on transfer manager perf metrics (#2103)
ddelgrosso1 Nov 15, 2022
94b1c02
fix: performance test refactoring, comments (#2104)
ddelgrosso1 Nov 16, 2022
9793cc6
refactor: refactor constants (#2105)
shaffeeullah Nov 16, 2022
89e8204
linter fixes, download to disk for performance test
ddelgrosso1 Nov 29, 2022
c153ab6
rename transfer manager functions
ddelgrosso1 Nov 29, 2022
4feb1c2
remove callbacks from transfer manager
ddelgrosso1 Nov 29, 2022
0780e50
add more experimental tags, update comments
ddelgrosso1 Nov 29, 2022
c47130b
change signature of downloadManyFiles to accept array of strings or a…
ddelgrosso1 Nov 30, 2022
4c2dda4
linter fix
ddelgrosso1 Nov 30, 2022
aad8f2b
add transfer manager samples and samples tests
ddelgrosso1 Dec 1, 2022
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
add jsdoc headers to class and uploadMulti
  • Loading branch information
ddelgrosso1 committed Oct 24, 2022
commit 16a5f05a8161e5a8264e7db364073889a45eca33
62 changes: 62 additions & 0 deletions src/transfer-manager.ts
Original file line number Diff line number Diff line change
Expand Up @@ -60,6 +60,14 @@ export interface DownloadMultiCallback {
(err: Error | null, contents?: Buffer[]): void;
}

/**
* Create a TransferManager object to perform parallel transfer operations on a Cloud Storage bucket.
*
* @class
* @hideconstructor
*
* @param {Bucket} bucket A {@link Bucket} instance
*/
export class TransferManager {
bucket: Bucket;
constructor(bucket: Bucket) {
Expand All @@ -79,6 +87,60 @@ export class TransferManager {
options: UploadMultiOptions,
callback: UploadMultiCallback
): Promise<void>;
/**
* @typedef {object} UploadMultiOptions
* @property {number} [concurrencyLimit] The number of concurrently executing promises
* to use when uploading the files.
* @property {boolean} [skipIfExists] Do not upload the file if it already exists in
* the bucket. This will set the precondition ifGenerationMatch = 0.
* @property {string} [prefix] A prefix to append to all of the uploaded files.
* @property {object} [passthroughOptions] {@link UploadOptions} Options to be passed through
* to each individual upload operation.
*/
/**
* @typedef {array} UploadResponse
* @property {object} The uploaded {@link File}
* @property {object} The uploaded {@link Metadata}
*/
/**
* @callback UploadMultiCallback
* @param {?Error} err Rewuest error if any
* @param {array} files Array of uploaded {@link File}.
* @param {array} metadata Array of uploaded {@link Metadata}
*/
/**
* Upload multiple files in parallel to the bucket. This is a convenience method
* that utilizes {@link Bucket#upload} to perform the upload.
*
* @param {array} [filePaths] An array of fully qualified paths to the files.
* you wish to upload to the bucket
* @param {UploadMultiOptions} [options] Configuration options.
* @param {UploadMultiCallback} [callback] Callback function.
* @returns {Promise<UploadResponse[] | void>}
*
* @example
* ```
* const {Storage} = require('@google-cloud/storage');
* const storage = new Storage();
* const bucket = storage.bucket('my-bucket');
* const transferManager = new TransferManager(bucket);
*
* //-
* // Upload multiple files.
* //-
* transferManager.uploadMulti(['/local/path/file1.txt, 'local/path/file2.txt'], function(err, files, metadata) {
* // Your bucket now contains:
* // - "file1.txt" (with the contents of '/local/path/file1.txt')
* // - "file2.txt" (with the contents of '/local/path/file2.txt')
* // `files` is an array of instances of File objects that refers to the new files.
* });
*
* //-
* // If the callback if omitted, we will return a Promise.
* //-
* const response = transferManager.uploadMulti(['/local/path/file1.txt, 'local/path/file2.txt']);
* ```
*/
async uploadMulti(
filePaths: string[],
Copy link
Contributor

@danielbankhead danielbankhead Dec 2, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[Optional] We have an opportunity to get fancy here to really emphasize throughput; what if we allowed iterators and async iterators?

Use cases:

  • walking large directory you want to walk asynchronously (reduces memory usage)
  • situations where one may parse a list of files in chunks and want to chain operations together (such as an HTTP server receiving incoming data)

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We can probably wait until customers ask for this functionality though.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It is a good idea, maybe after the initial release we can circle up. I would definitely be interested in your thoughts on optimizations here.

optionsOrCallback?: UploadMultiOptions | UploadMultiCallback,
Expand Down