---
title: Circuit Breaking
weight: 55
---
:icons: font
:sectlinks:
:linkattrs:

Distributed systems will benefit from failing quickly and applying back pressure, as opposed to propagating delays and errors through the system.

Circuit breaking is an important technique used to limit the impact of failures, latency spikes, and other types of network problems.

This step will show how to apply a Circuit Breaker into the *travels* service in order to limit the number of concurrent requests and connections.

NOTE: *Step 1:* Deploy a new *loadtester* portal in the *travel-portal* namespace

In this example we are going to deploy a new workload that will simulate an important increase in the load of the system.

[source,bash]
----
kubectl apply -f <(curl -L https://raw.githubusercontent.com/kiali/demos/master/travels/travel_loadtester.yaml) -n travel-portal
----

The *loadtester* workload will try to create 50 concurrent connections to the *travels* service, adding considerable pressure to the *travels-agency* namespace.

++++
<a class="image-popup-fit-height" href="/images/tutorial/05-06-loadtester-graph.png" title="Loadtester Graph">
    <img src="/images/tutorial/05-06-loadtester-graph.png" style="display:block;margin: 0 auto;" />
</a>
++++

{nbsp} +
The Travels Demo application is capable of handling this load and in a first look it doesn't show unhealthy status.

++++
<a class="image-popup-fit-height" href="/images/tutorial/05-06-loadtester-details.png" title="Loadtester Details">
    <img src="/images/tutorial/05-06-loadtester-details.png" style="display:block;margin: 0 auto;" />
</a>
++++

{nbsp} +
But in a real scenario an unexpected increase in the load of a service like this may have a significant impact in the overall system status.

NOTE: *Step 2:* Use the Traffic Shifting Wizard on *travels* service to generate a traffic rule

Use the "Traffic Shifting" Wizard to distribute traffic to the *travels* workloads and use the "Advanced Options" to add a "Circuit Breaker" to the scenario.

++++
<a class="image-popup-fit-height" href="/images/tutorial/05-06-traffic-shifting-circuit-breaker.png" title="Traffic Shifting with Circuit Breaker">
    <img src="/images/tutorial/05-06-traffic-shifting-circuit-breaker.png" style="display:block;margin: 0 auto;" />
</a>
++++

{nbsp} +
The "Connection Pool" settings will indicate that the proxy sidecar will reject requests when the number of concurrent connections and requests exceeds more than one.

{nbsp} +
The "Outlier Detection" will eject a host from the connection pool if there is more than one consecutive error.

NOTE: *Step 3:* Study the behavior of the Circuit Breaker in the *travels* service

In the *loadbalancer* workload Graph we can see that the *travels* service's Circuit Breaker accepts some, but fails most, connections.

Remember, that these connections are stopped by the proxy on the *loadbalancer* side. That "fail sooner" pattern prevents overloading the network.

Using the Graph we can select the failed edge, check the Flags tab, and see that those requests are closed by the Circuit breaker.

++++
<a class="image-popup-fit-height" href="/images/tutorial/05-06-loadtester-flags-graph.png" title="Loadtester Flags Graph">
    <img src="/images/tutorial/05-06-loadtester-flags-graph.png" style="display:block;margin: 0 auto;" />
</a>
++++

{nbsp} +
If we examine the "Request volume" metric from the "Outbound Metrics" tab we can see the evolution of the requests, and how the introduction of the Circuit Breaker made the proxy reduce the request volume.

++++
<a class="image-popup-fit-height" href="/images/tutorial/05-06-loadtester-flags-details.png" title="Loadtester Outbound Metrics">
    <img src="/images/tutorial/05-06-loadtester-flags-details.png" style="display:block;margin: 0 auto;" />
</a>
++++

NOTE: *Step 4:* Update or delete Istio Configuration

As part of this step you can update the scenarios defined around the *travels* service to experiment with more Circuit Breaker settings, or you can delete the generated Istio config in the service.

{nbsp} +
Understanding what happened:

https://istio.io/latest/docs/tasks/traffic-management/circuit-breaking/[(i) Circuit Breaking, window="_blank"]

https://istio.io/latest/docs/reference/config/networking/destination-rule/#OutlierDetection[(ii) Outlier Detection, window="_blank"]

https://istio.io/latest/docs/reference/config/networking/destination-rule/#ConnectionPoolSettings[(iii) Connection Pool Settings, window="_blank"]

https://www.envoyproxy.io/docs/envoy/latest/intro/arch_overview/upstream/circuit_breaking[(iv) Envoy's Circuit breaking Architecture, window="_blank"]