that either splits the incoming batches into multiple batches, one per trace

Does this exist? might be better just to mention groupbytrace if it's the only existing option.

It will, once #1235 is addressed :-) I think groupbytrace isn't suitable for all cases, although I would certainly recommend it for most cases that I can think of.

should the example show groupbytrace since it's basically required?

I thought about it, but I think I would prefer to have the future "splitbatch" there instead.

this example is cracking me up. I love how flexible the collector is.

perhaps use a jumphash? I'm pretty sure it will work in this use case and it's significantly less code.

https://github.com/dgryski/go-jump

We present jump consistent hash, a fast, minimal memory, consistent hash algorithm that can be expressed in about 5 lines of code. In comparison to the algorithm of Karger et al., jump consistent hash requires no storage, is faster, and does a better job of evenly dividing the key space among the buckets and of evenly dividing the workload when the number of buckets changes. Its main limitation is that the buckets must be numbered sequentially, which makes it more suitable for data storage applications than for distributed web caching.

I'm no expert, but I think this will work if you just order the IPs you're load balancing over in increasing order.

I saw it the first time in @annanay25's branch and found it highly interesting, especially because it's implemented 5 lines written by a PerlMonks. I decided to read the paper, and this statement in the abstract intrigued me:

Its main limitation is that the buckets must be numbered sequentially, which makes it more suitable for data storage applications than for distributed web caching

Reading further, I realized that jumphash might not be a good choice for our use case at all:

This ability to add or remove buckets in any order can be valuable for cache servers where the servers are purely a performance improvement. But for data storage applications, where each bucket represents a different shard of the data, it is not acceptable for shards to simply disappear, because that shard is only place where the corresponding data is stored. Typically this is handled by either making the shards redundant (with several replicas), or being able to quickly recover a replacement, or accepting lower availability for some data. Server death thus does not cause reallocation of data; only capacity changes do. This means that shards can be assigned numerical ids in increasing order as they are added, so that the active bucket ids always fill the range [0, num_buckets)

Basically, we can scale up, but can't scale down.

Ah, thank you. I knew we used jumphash with memcached in Cortex so I thought it might be useful here, but you are correct. We use a statefulset to force identity/order so that new shards are always appended to the end.

Replace double negative (never false) to positive (always true)