Overview | Training Dynamics | Benchmarking | Getting Started | Behind the Name

Tora is a project built on torchtune that provides LoRA-based RL methods for post-training.

Building on the torchtune library, Tora extends its functionality for RL post-training. It integrates PEFT methods like (Q)LoRA and (Q)DoRA into RL, providing an efficient and memory-friendly framework. Tora enables researchers to reduce the computational resources required for fine-tuning large language models via RL.

📚 Key References: LoRA for RL

• October 2025: LoRA Without Regret by Thinking Machines. link
• May 2025: Tina: Tiny Reasoning Models via LoRA. link

The following table summarizes the key features of RL methods and LoRA-based techniques supported in Tora.

DoRA w/ Cache: The standard DoRA layer in torchtune (link) recalculates the weight norm and magnitude scale on every forward pass. This is inefficient for GRPO's completion generation step, as these values remain static between weight updates. DoRA w/ Cache optimizes this by caching these expensive computations. It computes the values once and reuses them on subsequent forward passes, avoiding redundant calculations and significantly improving performance. However, the current caching implementation is not compatible with torch.compile.

Unless specified otherwise, our experimental settings are as follows:

• We used Qwen2.5 base models in five sizes: 1.5B, 3B, 7B, 14B, and 32B parameters.
• All experiments were conducted on two NVIDIA RTX A40 GPUs using the GSM8K training dataset.
• We used a per-GPU batch size of 2 and a generation sequence length of 512.
• For all LoRA-based methods, LoRA was applied to all layers with a rank of 1, an alpha of 2, and zero dropout.
• In QLoRA and QDoRA, the base model was quantized to 4-bits.
• We enabled activation checkpointing and used Fully Sharded Data Parallelism (FSDP) across all experiments.
• The learning rate for LoRA-based methods was set to 20x that of full-parameter GRPO training.

We show the reward dynamics during GRPO training of Qwen2.5-3B with different methods on GSM8K. From the results, we can see that LoRA-based methods (with rank 1), even with base model quantization, achieve comparable performance with full-parameter GRPO training.

In the tables below, we benchmark the peak memory usage per GPU, the number of generated tokens per second during GRPO completion generation, and the seconds per gradient step for different GRPO methods.

DoRA w/ Cache significantly speeds up the generation process by caching intermediate calculations, and it has comparable performance with torch.compile optimizations.

Clone the repository and install the required packages.

git clone https://github.com/shangshang-wang/Tora.git && cd Tora
pip install torch torchvision torchao
pip install -e .
pip install wandb math_verify

Download a model from the Hugging Face Hub.

MODEL_SIZE=1.5B  # 1.5B, 3B, 7B, 14B, or 32B
tune download "Qwen/Qwen2.5-${MODEL_SIZE}" \
--output-dir "/tmp/Qwen2.5-${MODEL_SIZE}" \
--hf-token <HF_TOKEN>

Below are example commands for running distributed GRPO training on 2 GPUs. You can easily switch between LoRA methods by modifying the lora_type parameter in the config file or overriding it on the command line.

Full-Parameter RL:

tune run --nproc_per_node 2 full_grpo_distributed --config qwen2_5/1.5B_full_grpo

LoRA-Based RL:

# In the config file, set lora_type to "lora", "dora", or "dora_cache"
tune run --nproc_per_node 2 lora_grpo_distributed --config qwen2_5/1.5B_lora_grpo model.lora_type="lora"

The name Tora (虎) means Tiger in Japanese. It's also a blend of TorchTune and LoRA. The name is inspired by the film Crouching Tiger, Hidden Dragon, which refers to masters with hidden strengths. This symbolism captures the role of LoRA in RL post-training: by updating only a tiny fraction of a model's parameters, LoRA unleashes significant performance gains—a "crouching tiger" of potential within the base model.