Skip to content

allocator: research support for MemberSpec load factor #237

@jgraettinger

Description

@jgraettinger

As a Gazette operator, I'd like a dynamic control feedback loop which allows brokers having very busy journals to discharge those journals more equitably around the cluster.

Today the allocator (falsely) assumes all journals are utilized equally. As an operator, it's on me to partitionin journals to best approximate this assumption, but it's impractical to achieve perfectly. Journals may also exhibit shorter-lived write skew, eg due to time of day patterns.

I'd like to be able to determine a load factor for each broker (through a process not defined here, but likely kubernetes top or something similar), and then periodically update that factor on each broker's advertised MemberSpec. The allocator should then use that factor to determine the "fair-share" scaling for each broker, such that busy brokers discharge journals and idle brokers take on more.

I should have the means to automate this control system (eg, running every 5 minutes) to balance our broker load over the day.

Metadata

Metadata

Assignees

No one assigned

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions