Skip to content

Fix not-reported flaky tests #22639

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
stianst merged 1 commit into from
Aug 23, 2023
Merged

Fix not-reported flaky tests #22639

stianst merged 1 commit into from
Aug 23, 2023

Conversation

@mhajas
Copy link
Contributor

@mhajas mhajas commented Aug 23, 2023 •
edited
Loading

In the current state, not all flaky tests are reported. The reason for that is that sometimes the failed attempt is reported as a failure and sometimes as an error. The current upload-flaky-test is looking only for <flakyFailure string in surefire-reports, however, we need to search also for <flakyError.

More info here: https://maven.apache.org/surefire/maven-surefire-plugin/examples/rerun-failing-tests.html#output-flaky-re-run-information-in-test-report-xml

@mhajas
Fix not-reported flaky tests
1465f07 
Signed-off-by: Michal Hajas <[email protected]>
@mhajas mhajas requested a review from a team as a code owner August 23, 2023 07:51
@mhajas
Copy link
Contributor Author

mhajas commented Aug 23, 2023

I checked also keycloak-github-bot and it does not seem to be affected by this as it is checking only for testcase and stackTrace elements which are the same in both cases (failure and error).

@cypress
Copy link

cypress bot commented Aug 23, 2023

3 flaky tests on run #8589 ↗︎

0 527 48 0 Flakiness 3

Details:

Merge 1465f07 into afaa8c2...
Project: Keycloak Admin UI Commit: 6e03e3e261 ℹ️
Status: Passed Duration: 15:59 💡
Started: Aug 23, 2023 7:59 AM Ended: Aug 23, 2023 8:15 AM
Flakiness  authentication_test.spec.ts • 1 flaky test • chrome

View Output Video

Test Artifacts
Authentication test > should add a condition Output Screenshots
Flakiness  realm_settings_events_test.spec.ts • 1 flaky test • chrome

View Output Video

Test Artifacts
Realm settings events tab tests > Should remove event from event listener Output Screenshots
Flakiness  realm_settings_general_tab_test.spec.ts • 1 flaky test • chrome

View Output Video

Test Artifacts
Realm settings general tab tests > Test all general tab switches Output Screenshots

This comment has been generated by cypress-bot as a result of this project's GitHub integration settings.

@stianst stianst enabled auto-merge (squash) August 23, 2023 08:42
@ghost ghost added the flaky-test label Aug 23, 2023
@ghost
Copy link

ghost commented Aug 23, 2023

Unreported flaky test detected

If the below flaky tests below are affected by the changes, please review and update the changes accordingly. Otherwise, a maintainer should report the flaky tests prior to merging the PR.

org.keycloak.testsuite.forms.BrowserFlowTest#testAlternativeNonInteractiveExecutorInSubflow

Keycloak CI - Forms IT (firefox)

org.openqa.selenium.TimeoutException: 
Navigation timed out after 10000 ms
Build info: version: '3.14.0', revision: 'aacccce0', time: '2018-08-02T20:19:58.91Z'
System info: host: 'fv-az173-551', ip: '10.1.0.25', os.name: 'Linux', os.arch: 'amd64', os.version: '5.15.0-1041-azure', java.version: '17.0.8'
Driver info: org.openqa.selenium.firefox.FirefoxDriver
...

Report flaky test

org.keycloak.testsuite.model.ConcurrentHashMapStorageTest#testStorageSeparation

Keycloak CI - Store Model Tests

org.keycloak.models.locking.LockAcquiringTimeoutException: Lock [org.keycloak.models.RealmModel] already acquired by keycloak instance [null] at the time [null]
	at org.keycloak.models.map.lock.MapGlobalLockProvider.withLock(MapGlobalLockProvider.java:94)
	at org.keycloak.models.locking.GlobalLockProvider.withLock(GlobalLockProvider.java:45)
	at org.keycloak.models.map.storage.jpa.JpaMapStorageProviderFactory.update(JpaMapStorageProviderFactory.java:549)
	at org.keycloak.models.map.storage.jpa.JpaMapStorageProviderFactory.validateAndUpdateSchema(JpaMapStorageProviderFactory.java:471)
...

Report flaky test

ghost
ghost reviewed Aug 23, 2023
View reviewed changes
Copy link

@ghost ghost left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Unreported flaky test detected, please review

@stianst stianst merged commit fa1f1eb into keycloak:main Aug 23, 2023
This was referenced Aug 23, 2023
Closed
Closed
@mhajas mhajas deleted the flakyTestIssueHunting branch August 23, 2023 10:06
stianst pushed a commit to stianst/keycloak that referenced this pull request Sep 4, 2023
@mhajas @stianst
Fix not-reported flaky tests (keycloak#22639)
e3bcb2e 
Signed-off-by: Michal Hajas <[email protected]>
stianst pushed a commit that referenced this pull request Sep 4, 2023
@mhajas @stianst
Fix not-reported flaky tests (#22639)
6a7cd22 
Signed-off-by: Michal Hajas <[email protected]>
This was referenced Sep 6, 2023
Closed
Closed
@stianst stianst mentioned this pull request Nov 14, 2023
Closed
@stianst stianst mentioned this pull request May 6, 2024
Merged
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@stianst stianst stianst approved these changes

+1 more reviewer
Reviewers whose approvals may not affect merge requirements

Assignees

No one assigned

Labels

flaky-test

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

2 participants

@mhajas @stianst