Keycloak health checks failing & server repeatedly restarting after second deployment after upgrade to 26.4.0 on AWS Fargate #43194
Replies: 5 comments 25 replies
-
DB connection errors will under some circumstances cause the "Keycloak database connections async health check" to fail. From what you are showing the problem you are having is with the "Keycloak cluster health check" - which is new for the 26.4 release. cc @pruivo @ahus1 How many keycloak instances are you trying to cluster? Can you try setting the log level to debug and look for logs from org.keycloak.infinispan.health.impl.JdbcPingClusterHealthImpl - that should give a indication of why that check is failing. |
Beta Was this translation helpful? Give feedback.
4 replies
-
|
The error you saw in 26.3.x is actual indicating a critical problem that the Keycloak nodes do not form a correct cluster. The following is assuming you are using the default As outlined in https://www.keycloak.org/server/caching#network-ports the different Keycloak Pods needs need to connect to each other on port 7800 to form a cluster. I am not an expert on Fargate, still I see that your Dockerfile doesn't expose port 7800 which might be part of the problem. The health check that you see failing is checking that only one coordinator is listed in The additional sections "Network bind address" and "Running instances on different networks" might be required on Fargate and Docker to tune the address each instance binds to and the address it publishes to other instances. About checking the health: You are currently calling the overall health endpoint. There are two sub-URLs for the readiness and the liveness probe which offer a more differentiated check: When the liveness probe fails, it is time to restart the container. If the readiness probe fails, no traffic should be sent to the container. |
Beta Was this translation helpful? Give feedback.
-
|
Thank you for your help @shawkins and @ahus1
I am only trying to use 1 instance, behind a load balancer. Unfortunately I mostly don't know what I'm doing with Keycloak, and this is a configuration which was working for me and now isn't.
Unfortunately that would be very difficult to do, as I don't have direct access to view the DB. Is there a specific part of this table that I would view or modify? Changes
What HappenedIt's... working?
This happened again after going from org.keycloak.infinispan.health.impl.JdbcPingClusterHealthImpl Logs are repetitive, and look like this: And after that time, they were all "healthy". Here is some context around the unhealthy logs, except I dropped []: The INFO [io.smallrye.health] logs did NOT change between "unhealthy" and "healthy", they stayed the same. Between the last "unhealthy" and the first "healthy" this is what I see: QuestionWhy did |
Beta Was this translation helpful? Give feedback.
-
|
I can take a look at it this week. |
Beta Was this translation helpful? Give feedback.
-
|
Another thought on this: Maybe Fargate is starting the new instance of Keycloak while the old instance is still running. If this the case, and the network connectivity between the two is not working, it would explain the problems you see above. One way to explore is to enforce in Fargate that it shouldn't do that, so that it shuts down the old instance first. The other way is to fix the networking. There seems to be a user who solved a similar setup some time ago: #23400 Since then, Keycloak evolved and JDBC_PING is the default and built-in. Still the following bits would need to be translated: This now needs to be translated to the the new Keycloak configuration options |
Beta Was this translation helpful? Give feedback.
-
|
@pruivo @ahus1 thank you for the information. In #43194 (reply in thread) I neglected to adjust a security group mapping which I realized because it was mentioned in #23400. Recap
Recap Log SummaryAfter deploying again, I now see two patterns of Those messages stop after 1 minute and 20 seconds. @pruivo This makes me think the DB is being updated & the way the container is stopped isn't the cause of the problem. Then I start to see a similar but different message: After that, another task starts up. Right after that new task starts up, I start to see both the above "Connect timed out" IP and a new one too: My Current Understanding
Attempts to fix the problemI'm trying some things - both https://www.keycloak.org/server/caching#_running_instances_on_different_networks and changing how health checks are read and looking at adjusting the entire setup - and will post again tomorrow. |
Beta Was this translation helpful? Give feedback.
16 replies
-
|
Logs - A only: A and B and C, where C is Keycloak/Keycloak/bde07cda99674106865f27d747432c2e which escaped the loop: |
Beta Was this translation helpful? Give feedback.
-
I do not know.
0 (default). It is configurable. |
Beta Was this translation helpful? Give feedback.
-
|
I found 6 different IPs from the remove statements. The container with ID Can all these containers talk to each other? I can see 2 containers inserting their information at the same time in the database; if they are not reachable, then it explains why you have partitions.
I'll delay it at least to 20 seconds (it is the default timeout if the database has invalid entries). Last, but not least, how difficult would it be for me to replicate your environment in AWS Fargate? |
Beta Was this translation helpful? Give feedback.
-
|
I am working on reproducing with a minimal CloudFormation template. When I get that working I'll post an update. |
Beta Was this translation helpful? Give feedback.
-
|
Thank you. |
Beta Was this translation helpful? Give feedback.
-
|
Thank you everyone for your help. I seem to have found a configuration which both works and doesn't sometimes restart. For future reference - this is a working gist using Fargate + RDS / Aurora, in CloudFormation with Dockerfile & deployment instructions: https://gist.github.com/Gregory-Ledray/b1dd6a8958c6ed224aed4838450922fb Main changes from my post were:
These changes resulted in:
As for why it was that not having these changes applied resulted in the exact behavior I saw, that I'm not sure. I hadn't set container-level health checks in Fargate, so container health wasn't being evaluated using the HEALTHCHECK in the Dockerfile. Tasks cycled because restarts were initiated by load balancer health checks. And the errors were due to misconfiguration. But why did those errors sometimes go away, resulting in a stable deployment? That I do not know and don't have time to investigate. Thank you again to everyone who helped! |
Beta Was this translation helpful? Give feedback.
5 replies
-
|
You need both 7800 and 57800 ports in your security group to allow traffic between instances. See https://www.keycloak.org/server/caching#network-ports |
Beta Was this translation helpful? Give feedback.
-
|
I updated the gist with that. |
Beta Was this translation helpful? Give feedback.
-
|
@pruivo I updated again, making the 57800 port open, and now it's not able to deploy. Same issue as the original one in this Discussion :( Maybe the issue is that Keycloak is what #43194 (reply in thread) said. "By default, it picks a site local (non-routable) IP address, for example, from the 192.168.0.0/16 or 10.0.0.0/8 address range." which wouldn't work, as the tasks don't share those local IP addresses. And there was a link to someone who fixed that. I am currently looking at this to try to fix that: https://www.keycloak.org/server/caching#_running_instances_on_different_networks Alternatively, maybe I should set |
Beta Was this translation helpful? Give feedback.
-
What do you mean? How was your cluster working before?
There is no communication between Keycloak instances. Sessions created in one instance won't be visible in the other instance, and vice versa. If you're using just a single instance (no high availability), that's fine. |
Beta Was this translation helpful? Give feedback.
-
|
Once you set So with that, there are some benefits to keep the current behavior - obviously this needs fixing for the networking problems. When adding the parameters, note that all CLI options are available as environment variables. This might make your scripting a bit simpler. So as listed on https://www.keycloak.org/server/all-config, the option CLI option |
Beta Was this translation helpful? Give feedback.
-
|
I never got this to work on This was probably caused by whatever caused #43561 which caused shutdown to never run. I checked the logs for After updating to |
Beta Was this translation helpful? Give feedback.
0 replies
-
|
Thank you for sharing your approach in the gist. Some notes on adding RPM packages. The docs state an approach that might be slightly simpler as it avoids copying over single files or an extra dependency. See the snippet just after "It is possible to install new RPMs if absolutely required..." There you could copy those extra binaries and their dependencies onto the official Keycloak image, which would make your script a lot smaller. It would also then be less fragile towards changes in our build file like, for example, changing the JDK version. Our latest images use JDK21, while your script is using JDK17. The docs also state how to run your custom endpoint scripts. Looking at your example, |
Beta Was this translation helpful? Give feedback.
0 replies
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Uh oh!
There was an error while loading. Please reload this page.
Uh oh!
There was an error while loading. Please reload this page.
-
Hi. After upgrading to 26.4.0 I've been unable to keep Keycloak stable, as the service has repeatedly reported failing health checks & when that happens the service/task restarts.
During functional testing, the service tends to work as expected, ie I can log in to both the admin portal and as a user.
Timeline
2025-09-25T11:15:48.330-05:00deployed upgrade to 26.3.5Saw this error shortly after startup. This error was NOT new with 26.3.5:
2025-09-25T11:16:25.006-05:002025-09-25 16:16:25,006 ERROR [org.jgroups.protocols.TCP] (pd-bundler-8,ip-10-30-181-178-15876) ip-10-30-181-178-15876: failed sending message to 10.30.184.253:7800: java.net.SocketTimeoutException: Connect timed outThis error appeared once and then stopped. In a subsequent deployment at
2025-09-26T16:37:47.720-05:00, still on 26.3.5, this error would appear twice during startup and then stop.2025-09-30T18:01:28.699-05:00deployed upgrade to 26.4.0Starting with the
26.4.0release, I began to see the following log message accompany the above ERROR in the logs, intially preceding the above error but also appearing after & at other intervals:In the first deployment, these failing health checks occurred for 1 minute and 30 seconds and then stopped. Everything seemed to be OK.
2025-10-02T11:21:14.048-05:00deployed with 26.4.0 for the second timeStarting in this deployment, the above health down status was repeatedly logged over 8 minutes and 30 seconds. The "failed sending message to..." error was interspersed with these problems. From an admin perspective and an end user perspective the service seems functional. I'm still able to log in to the admin page & to the application.
Because so many health checks failed, the service was terminated and a new one was started up. The new one had the same problem as the last one - it started up, reported unhealthy, and shut down. The service performed this loop several hundred times. I downgraded to 26.3.5 to "fix" the problem.
Does anyone have any advice on how to fix this?
Infrastructure / Setup
The service runs on AWS Fargate as a Task with Docker. The service restarts if health checks fail. Health check path is
/healthon port 9000. Docker configured health checks as follows:Dockerfile
Not shown in Dockerfile are default username/password of admin & KC_HOSTNAME.
What I Have Tried
LOG: could not receive data from client: Connection reset by peerbut these seem to happen during service restarts, as they only happened every 7 to 8 minutes.Questions
26.4.0to cause these DB connection errors to result in the service as being marked unhealthy?26.4.0cause this problem, but not the first? I presume that the first startup included migrating to the new version, so maybe there is something in there which delayed startup which caused this issue to not manifest?Beta Was this translation helpful? Give feedback.
All reactions