- {project_name} deployments are now able to handle user requests simultaneously in both sites.

- Active monitoring of the connectivity between the sites is now required to update the replication between the sites in case of a failure.

- {project_name} deployments are now able to handle user requests simultaneously in both sites. Previous load balancer configurations handling requests only in one site at a time will continue to work.

- Active monitoring of the connectivity between the sites is now required to the replication between the sites in case of a failure. The blueprints describe a setup with Alertmanager and AWS Lambda.

. While previous load balancer configurations will continue to work with {project_name}, consider upgrading

an existing Route53 configuration to avoid prolonged failover times due to client side DNS caching.

title="Deploy an AWS Lambda to disable a non-responding site"

This {section} explains how to resolve a split-brain scenarios between two sites in a multi-site deployment.

It also disables replication if one site fails, so the other site can continue to serve requests.

In the event of a network communication failure between sites in a multi-site deployment, it is no longer possible for the two sites to continue to replicate data between them.

The {jdgserver_name} is configured with a `FAIL` failure policy, which ensures consistency over availability. Consequently, all user requests are served with an error message until the failure is resolved, either by restoring the network connection or by disabling cross-site replication.

In such scenarios, a quorum is commonly used to determine which sites are marked as online or offline.

However, as multi-site deployments only consist of two sites, this is not possible.

Instead, we leverage "`fencing`" to ensure that when one of the sites is unable to connect to the other site, only one site remains in the loadbalancer configuration, and hence only this site is able to serve subsequent users requests.

In addition to the loadbalancer configuration, the fencing procedure disables replication between the two {jdgserver_name} clusters to allow serving user requests from the site that remains in the loadbalancer configuration.

As a result, the sites will be out-of-sync once the replication has been disabled.

To recover from the out-of-sync state, a manual re-sync is necessary as described in <@links.ha id="operate-synchronize" />.

This is why a site which is removed via fencing will not be re-added automatically when the network communication failure is resolved. The remove site should only be re-added once the two sites have been synchronized using the outlined procedure <@links.ha id="operate-site-online" />.

and AWS Lambda functions.

A Prometheus Alert is triggered when split-brain is detected by the {jdgserver_name} server metrics, which results in the Prometheus AlertManager calling the AWS Lambda based webhook.

The triggered Lambda function inspects the current Global Accelerator configuration and removes the site reported to be offline.

In a true split-brain scenario, where both sites are still up but network communication is down, it is possible that both sites will trigger the webhook simultaneously.

We guard against this by ensuring that only a single Lambda instance can be executed at a given time.

The logic in the AWS Lambda ensures that always one site entry remains in the loadbalancer configuration.