Speed up conformance runs with parallelism #720
Merged
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
…RPC parallelism to 4*num CPUs
|
@smaye81, ping |
|
Nice! Tested this on my machine too and it was much faster. |
smaye81
approved these changes
Dec 5, 2023
jhump
added a commit
that referenced
this pull request
Dec 6, 2023
This is a follow-up to #720. Since that PR is merged, accessing the `outWriter` and `errWriter` from multiple goroutines is racy. Even if the underlying writer were thread-safe, this means that writing a message or line could end up being "interrupted" by a concurrent write, interleaving lines and leaving the output unreadable. This commit fixes that by introducing a `Printer` interface with a thread-safe implementation to do logging. To make sure there aren't any actual data races in this code, this also adds a new test that exercises a more significant portion of the conformance test runner, including the concurrent aspects of the runner. So running `go test -race` can now verify that these aspects are race-free.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This introduces two new flags to the
connectconformancerunner:max-servers: the number of concurrent server processes to start. Test cases for all of them will be written to the client's stdin in parallel (thread-safe, but interleaving test cases for various servers). Defaults to 4.parallel: the number of RPCs to issue in parallel when--modeisserver(so this only controls the reference client). In--modeofclient, it's up to the client-under-test to implement parallelism. Defaults to 4 * GOMAXPROC (and GOMAXPROC defaults to the number of CPU cores detected).This speeds up conformance runs on my MBP from about 3 minutes down to less than 15 seconds,.