This is the github repository for paper Generative Representational Learning of Foundation Models for Recommendation.

1. Install dependencies

pip install -r requirements.txt

2. Install GradCache

cd training/GradCache
pip install -e .
cd ../..

3. Environment Setup

This project requires modifications to specific packages for training and inference environments.

For Training Environment:

• Replace transformers/models/llama/modeling_llama.py with env/Training Environment/transformers/modeling_llama.py
• Replace peft/tuners/lora/model.py with env/Training Environment/peft/model.py
• Apply similar replacements for other files in the Training Environment folder

For Inference Environment:

• Replace files similarly using the env/Inference Environment/ versions

Here is a simplified command to run the program. For more detailed parameter settings, please refer to the corresponding configuration file.

python scripts/run_script.py \
    --base_model_path /path/to/base/model \
    --train_data_path /path/to/train/data \
    --output_dir /path/to/output

For Generative Tasks:

python scripts/run_inference_eval_gen.py \
    --base_model_path /path/to/base/model \
    --peft_path /path/to/trained/adapter \
    --test_data_path /path/to/test/data

For Embedding Tasks:

python scripts/run_inference_eval_emb.py \
    --base_model_path /path/to/base/model \
    --peft_path /path/to/trained/adapter \
    --test_data_path /path/to/test/data

python merge/merge_moe.py

The Dataset is available in https://huggingface.co/datasets/Anonqwq/RecFound.

The TMoLE model checkpoint is available in https://huggingface.co/Anonqwq/RecFound-7B. You can load the module on Mistral-7B-v0.3-Instruct after getting ready for the environment.

Built upon:

• GritLM
• PEFT
• Transformers

MIT License