…3046)

#### What this PR does

I noticed the ingest storage config is still listed as experimental in
the doc. It shouldn't anymore.

#### Which issue(s) this PR fixes or relates to

N/A

#### Checklist

- [ ] Tests updated.
- [ ] Documentation added.
- [ ] `CHANGELOG.md` updated - the order of entries should be
`[CHANGE]`, `[FEATURE]`, `[ENHANCEMENT]`, `[BUGFIX]`. If changelog entry
is not needed, please add the `changelog-not-needed` label to the PR.
- [ ]
[`about-versioning.md`](https://github.com/grafana/mimir/blob/main/docs/sources/mimir/configure/about-versioning.md)
updated with experimental features.

---------

Signed-off-by: Marco Pracucci <[email protected]>

)

there was a problem where the store-gateway wouldn't accept the
/prepare-shutdown call because it couldn't persist the marker on disk.
In this particular case it could persist the marker because there
weren't any blocks synced for any tenant yet.

This is the place which failed
https://github.com/grafana/mimir/blob/77c8cab8dffbe71557b6591f73d077e8c1da5c1c/pkg/storegateway/gateway_prepare_shutdown_http.go#L33-L73



The fix is to create the sync dir during startup

…component basis (#13043)

#### What this PR does

`_config.multi_zone_distributor_schedule_toleration` and
`_config.multi_zone_etcd_schedule_toleration` are un necessarily
complication. In this PR I'm removing them. I'm also introducing a new
utility function `newMimirMultiZoneToleration()` to simplify the
definition of the toleration.

#### Which issue(s) this PR fixes or relates to

N/A

#### Checklist

- [ ] Tests updated.
- [ ] Documentation added.
- [x] `CHANGELOG.md` updated - the order of entries should be
`[CHANGE]`, `[FEATURE]`, `[ENHANCEMENT]`, `[BUGFIX]`. If changelog entry
is not needed, please add the `changelog-not-needed` label to the PR.
- [ ]
[`about-versioning.md`](https://github.com/grafana/mimir/blob/main/docs/sources/mimir/configure/about-versioning.md)
updated with experimental features.

---------

Signed-off-by: Marco Pracucci <[email protected]>

…distributor.libsonnet (#13044)

#### What this PR does

`multi_zone_availability_zones` is used by all Mimir components and it's
not distributor specific. In this I'm moving it out of
`multi-zone-distributor.libsonnet` and adding an assertion to ensure no
more than 3 AZs are specified.

_This PR is expected to be a no-op._

#### Which issue(s) this PR fixes or relates to

N/A

#### Checklist

- [ ] Tests updated.
- [ ] Documentation added.
- [ ] `CHANGELOG.md` updated - the order of entries should be
`[CHANGE]`, `[FEATURE]`, `[ENHANCEMENT]`, `[BUGFIX]`. If changelog entry
is not needed, please add the `changelog-not-needed` label to the PR.
- [ ]
[`about-versioning.md`](https://github.com/grafana/mimir/blob/main/docs/sources/mimir/configure/about-versioning.md)
updated with experimental features.

Signed-off-by: Marco Pracucci <[email protected]>

#### What this PR does

We currently have `-querier.prefer-availability-zone` that allows to
specify the preferred zone when querying ingesters. In this PR I'm
extending it to store-gateways too. The CLI flag was experimental and
hidden. In this PR I'm exposing it in the doc, but it's still
experimental.

In this PR I've also added 2 new metrics, tracking the data transfer
between queriers and ingesters / store-gateways respectively:

- `cortex_ingester_client_transferred_bytes_total{ingester_zone="..."}`
-
`cortex_storegateway_client_transferred_bytes_total{store_gateway_zone="..."}`

These two metrics should make it easily to monitor the cross-AZ data
transfer that happens during fallbacks.

#### Which issue(s) this PR fixes or relates to

N/A

#### Checklist

- [x] Tests updated.
- [ ] Documentation added.
- [x] `CHANGELOG.md` updated - the order of entries should be
`[CHANGE]`, `[FEATURE]`, `[ENHANCEMENT]`, `[BUGFIX]`. If changelog entry
is not needed, please add the `changelog-not-needed` label to the PR.
- [x]
[`about-versioning.md`](https://github.com/grafana/mimir/blob/main/docs/sources/mimir/configure/about-versioning.md)
updated with experimental features.

---------

Signed-off-by: Marco Pracucci <[email protected]>