Official PyTorch implementation of DLP: Dynamic Layerwise Pruning in Large Language Models

Perplexity results on WikiText. We produce the Uniform, OWL and DLP(Ours) with 70% unstructured sparsity on LLaMA1, LLaMA2 models. The best performance result is indicated in bold.

Installation instructions can be found in INSTALL.md.

We provide a quick overview of the arguments:

• --model: The identifier for the LLaMA model on the Hugging Face model hub.
• --cache_dir: Directory for loading or storing LLM weights. The default is llm_weights.
• --prune_method: We have implemented three pruning methods, namely [magnitude_dlp, wanda_dlp, sparsegpt_dlp.
• --sparsity_ratio: Denotes the percentage of weights to be pruned.
• --sparsity_type: Specifies the type of sparsity [unstructured, 2:4, 4:8].
• --eval_zero_shot: Whether to compute accuracy on a zero-shot dataset.
• --save: Specifies the directory where the result will be stored.

Below is an example command for pruning LLaMA-7B with DLP, to achieve unstructured 70% sparsity.

python   run.py \
    --model "Enoch/llama-7b-hf" \
    --alpha 0.15 \
    --prune_method "wanda_dlp" \
    --sparsity_ratio 0.7 \
    --sparsity_type "unstructured"

zero-shot evaluation

python   run.py \
    --model "Enoch/llama-7b-hf" \
    --alpha 0.15 \
    --prune_method "wanda_dlp" \
    --sparsity_ratio 0.7 \
    --sparsity_type "unstructured" \
    --save_model "pruned/llama-7b-hf_sparsity_0.7_wanda_dlp" \
    --eval_zero_shot 

This repository is build upon the OWL repositories.

@misc{chen2025dlpdynamiclayerwisepruning,
      title={DLP: Dynamic Layerwise Pruning in Large Language Models}, 
      author={Yuli Chen and Bo Cheng and Jiale Han and Yingying Zhang and Yingting Li and Shuhao Zhang},
      year={2025},
      eprint={2505.23807},
      archivePrefix={arXiv},
      primaryClass={cs.CL},
      url={https://arxiv.org/abs/2505.23807}, 
}