Skip to content

feat: implement parallel operations #2067

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 25 commits into from
Dec 5, 2022
Merged

feat: implement parallel operations #2067

Show file tree
Hide file tree
Changes from 1 commit
Commits
Show all changes
25 commits
Select commit Hold shift + click to select a range
8717527
feat: implement parallel operations
ddelgrosso1 Aug 15, 2022
22097df
add more parallel operations
ddelgrosso1 Sep 15, 2022
9a59f1f
add header to test file
ddelgrosso1 Sep 15, 2022
d2b241f
update import of fs/promises
ddelgrosso1 Sep 15, 2022
30b45d3
fix pathing on windows, fix mocking of fs promises
ddelgrosso1 Sep 16, 2022
16a5f05
add jsdoc headers to class and uploadMulti
ddelgrosso1 Sep 16, 2022
5a94aa7
add jsdoc comments to remaining functions
ddelgrosso1 Sep 30, 2022
1cc8bae
update comment wording
ddelgrosso1 Oct 24, 2022
face55f
add experimental jsdoc tags
ddelgrosso1 Oct 27, 2022
524a310
feat: add directory generator to performance test framework
ddelgrosso1 Nov 3, 2022
bac3ed8
clarify variable names and comments
ddelgrosso1 Nov 3, 2022
b4bc333
capitalization
ddelgrosso1 Nov 3, 2022
46687c6
wip: transfer manager performance tests
ddelgrosso1 Nov 10, 2022
721aab6
feat: merged in application performance tests (#2100)
shaffeeullah Nov 10, 2022
f5e8121
fix: fixed many bugs (#2102)
shaffeeullah Nov 15, 2022
886dc03
fix: more work on transfer manager perf metrics (#2103)
ddelgrosso1 Nov 15, 2022
94b1c02
fix: performance test refactoring, comments (#2104)
ddelgrosso1 Nov 16, 2022
9793cc6
refactor: refactor constants (#2105)
shaffeeullah Nov 16, 2022
89e8204
linter fixes, download to disk for performance test
ddelgrosso1 Nov 29, 2022
c153ab6
rename transfer manager functions
ddelgrosso1 Nov 29, 2022
4feb1c2
remove callbacks from transfer manager
ddelgrosso1 Nov 29, 2022
0780e50
add more experimental tags, update comments
ddelgrosso1 Nov 29, 2022
c47130b
change signature of downloadManyFiles to accept array of strings or a…
ddelgrosso1 Nov 30, 2022
4c2dda4
linter fix
ddelgrosso1 Nov 30, 2022
aad8f2b
add transfer manager samples and samples tests
ddelgrosso1 Dec 1, 2022
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
linter fixes, download to disk for performance test
  • Loading branch information
@ddelgrosso1
ddelgrosso1 committed Nov 29, 2022
commit 89e8204a35394533030aba4efab807143b166641
9 changes: 1 addition & 8 deletions internal-tooling/performApplicationPerformanceTest.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -16,13 +16,7 @@

import yargs from 'yargs';
import {promises as fsp, rmSync} from 'fs';
import {
Bucket,
DownloadOptions,
DownloadResponse,
Storage,
UploadOptions,
} from '../src';
import {Bucket, DownloadOptions, DownloadResponse, UploadOptions} from '../src';
import {performance} from 'perf_hooks';
// eslint-disable-next-line node/no-unsupported-features/node-builtins
import {parentPort} from 'worker_threads';
Expand All @@ -43,7 +37,6 @@ import {TRANSFER_MANAGER_TEST_TYPES} from './performanceTest';
const TEST_NAME_STRING = 'nodejs-perf-metrics-application';
const DEFAULT_BUCKET_NAME = 'nodejs-perf-metrics-shaffeeullah';

let stg: Storage;
let bucket: Bucket;

const checkType = getValidationType();
Expand Down
2 changes: 2 additions & 0 deletions internal-tooling/performTransferManagerTest.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -36,6 +36,7 @@ import {
} from './performanceUtils';
import {performance} from 'perf_hooks';
import {rmSync} from 'fs';
import * as path from 'path';

const TEST_NAME_STRING = 'tm-perf-metrics';
const DEFAULT_BUCKET_NAME = 'nodejs-transfer-manager-perf-metrics';
Expand Down Expand Up @@ -178,6 +179,7 @@ async function performDownloadMultipleObjectsTest(): Promise<TestResult> {
const getFilesResult = await bucket.getFiles();
const start = performance.now();
await transferManager.downloadMulti(getFilesResult[0], {
prefix: path.join(__dirname, '..', '..'),
concurrencyLimit: argv.numpromises,
passthroughOptions: {
validation: checkType,
Expand Down