Allow ignoring individual vars #40
Merged
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
```
clj -X:dev:test :only metabase.util.queue-test :ignored '{:vars ["metabase.util.queue-test/bounded-transfer-queue-test" "metabase.util.queue-test/take-batch-test"]}'
ignoring var: metabase.util.queue-test/bounded-transfer-queue-test
ignoring var: metabase.util.queue-test/take-batch-test
Running tests with options {:mode :cli/local, :namespace-pattern #"^(?:(?:metabase.*)|(?:hooks\..*))", :exclude-directories [".clj-kondo/src" "classes" "dev" "enterprise/backend/src" "local" "resources" "resources-ee" "src" "target" "test_config" "test_resources"], :test-warn-time 3000, :exclude-tags [:metabot-v3/e2e], :only metabase.util.queue-test, :ignored {:vars #{"metabase.util.queue-test/bounded-transfer-queue-test" "metabase.util.queue-test/take-batch-test"}}}
Running tests in metabase.util.queue-test
Finding tests took 7.4 s.
Running 4 tests
Running before-run hooks...
```
vs without this:
```
❯ clj -X:dev:test :only metabase.util.queue-test
Running tests with options {:mode :cli/local, :namespace-pattern #"^(?:(?:metabase.*)|(?:hooks\..*))", :exclude-directories [".clj-kondo/src" "classes" "dev" "enterprise/backend/src" "local" "resources" "resources-ee" "src" "target" "test_config" "test_resources"], :test-warn-time 3000, :exclude-tags [:metabot-v3/e2e], :only metabase.util.queue-test}
Running tests in metabase.util.queue-test
Finding tests took 6.8 s.
Running 6 tests
Running before-run hooks...
LONG TEST in metabase.util.queue-test/take-batch-test
Test took 4.721 seconds seconds to run
LONG TEST in metabase.util.queue-test/bounded-transfer-queue-test
Test took 7.015 seconds seconds to run
6/6 100% [==================================================] ETA: 00:00
Ran 6 tests in 11.882 seconds
53 assertions, 0 failures, 0 errors.
{:test 6, :pass 53, :fail 0, :error 0, :type :summary, :duration 11881.563167, :single-threaded 6}
Ran 0 tests in parallel, 6 single-threaded.
Finding and running tests took 18.7 s.
All tests passed.
```
We can annotate existing tests, but it is helpful to be able to ignore
from the CLI. This will give us the ability to quickly suppress known,
flaky tests in all of CI without having to change source, land PRs, etc.
escherize
approved these changes
Aug 20, 2025
| @@ -28,6 +33,22 @@ | |||
| (is (seq tests)) | |||
| (is (every? var? tests)) | |||
| (is (contains? (set tests) (resolve 'mb.hawk.core-test/find-tests-test))))) | |||
| (testing "ignore var options" | |||
| ;; note this is a set here. but that gets normalized and the cli can pass a sequence | |||
There was a problem hiding this comment.
What do you think of passing a seq of ignored vars in the test, since it mirrors the usual usage more closely?
There was a problem hiding this comment.
i want it to be the same, but the entrypoint here means it would convert this to a set many times rather than just once. I normalize the options at the beginning of the find-tests bit, but this multi-method (defmethod find-tests clojure.lang.Symbol gets called once for each symbol; so if a vector comes in i have to turn it into a set each time and that is just very wasteful.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
ignoring some tests
with
ignored '{:vars ["metabase.util.queue-test/bounded-transfer-queue-test" "metabase.util.queue-test/take-batch-test"]}'we run 4 testsvs without this:
We can annotate existing tests, but it is helpful to be able to ignore from the CLI. This will give us the ability to quickly suppress known, flaky tests in all of CI without having to change source, land PRs, etc.
API in CI
alternatively, perhaps easier to pass a filename of either json or edn and let the runner convert from json into clojure data structures. doing this all on the CLI isn't crazy but it's not super elegant