Seems like FIRST scores drop while RZ scores remain.

python src/rank_llm/scripts/run_rank_llm.py --model_path=castorini/first_mistral --top_k_candidates=100 --dataset=dl20 --retrieval_method=SPLADE++_EnsembleDistil_ONNX --prompt_template_path=src/rank_llm/rerank/prompt_templates/rank_zephyr_template.yaml --context_size=4096 --variable_passages --use_logits --use_alpha --num_gpus 1

Firstly, this should be changed in README to be rank_zephyr_alpha_template.yaml.

But both seem to result in lower scores than the base SPLADE model

~ ndcg_cut_10 all 0.7192

5 points lower than the scores reported in the paper