---
title: How do I inspect the underlying metrics used to generate the Kiali Graph?
weight: 110
---
:icons: font
:imagesdir: /images/faq/graph/

It is not uncommon for the Kiali graph to show traffic that surprises the user.  Often the thought is that
Kiali may have a bug. But in general Kiali is just visualizing the metrics generated by Istio. The next
thought is that the Istio telemetry generation may have a bug. But in general Istio is generating the
expected metrics given the defined configuration for the application.

To determine whether there is an actual bug it can be useful to look directly at the metrics collected
by and stored in the Prometheus database.  Prometheus provides a basic console that can be opened
using the https://istio.io/latest/docs/reference/commands/istioctl/#istioctl-dashboard[istioctl dashboard command]:

```
> istioctl dashboard prometheus
```

The above command, assuming Istio and Prometheus are in the default istio-system namespace, should open
the Prometheus console in your browser.

Kiali uses a variety of metrics but the primary request traffic metrics for graph generation are these:

* istio_requests_total
* istio_tcp_sent_bytes_total

The https://prometheus.io/docs/prometheus/latest/querying/basics/[Prometheus query language] is very rich
but a few basic queries is often enough to gather time-series of interest.

First, check to ensure Istio is in fact generating telemetry.

This query will return results if Istio has generated any HTTP or GRPC telemetry at any point:

`istio_requests_total{}`

This query will return results if Istio has generated any HTTP or GRPC telemetry in the last hour:

`sum(rate(istio_requests_total{}[1h])) by (reporter) > 0`

This query will return results if Istio has generated any TCP telemetry at any point:

`istio_tcp_sent_bytes_total{}`

This query will return results if Istio has generated any TCP telemetry in the last hour:

`sum(rate(istio_tcp_sent_bytes_total{}[1h])) by (reporter) > 0`

You can refine the searches as needed to look for more specific telemetry.  For example, if you want to check
for HTTP traffic to the service __myservice__ in namespace __mynamespace__:

`istio_requests_total{destination_service_namespace="mynamespace", destination_service_name="myservice"}`

A similar example using the Bookinfo sample app, and looking for traffic to the __reviews__ service:

`istio_requests_total{reporter="source", destination_service_name="reviews"}`

And here is an example of the results:

image::prom-ui.png[Prometheus Console - all attributes]

The query above is good for dumping all of the attributes but it can be useful to aggregate results by desired attributes.
The next query will get the request counts for the __reviews__ service broken down by source and destination workloads:

`sum(istio_requests_total{reporter="source", destination_service_name="reviews"}) by (source_workload, destination_workload)`

image::prom-ui-2.png[Prometheus Console - aggregation]

The first step to explaining your Kiali graph is to inspect the metrics used to generate the graph. Kiali devs may ask for
this info when working with you to solve a problem, so it is useful to know how to get to the Prometheus console.


