Steps to recreate:

Launch mmlu on an instance with multiple gpus. Run:

ilab model evaluate --model models/instructlab/granite-7b-lab --benchmark mmlu

Only 1 gpu is consumed. Adjusting batch-size doesn't seem to have any effect.