pip install -r requirements.txt

1. Input Data: All data come from the ADNI1GO database.

   • Images must be registered and preprocessed to a size of 32x256x256.

2. Preprocessing Scripts:

   • The data preprocessing code will be released soon.

To pre-train the vision encoder, follow these steps:

1. Open the file script/train_clip_MIND_LLM.sh and modify the following parameters:

   • training_json_path: Path to the training data JSON file.
   • output_dir: Directory where the model checkpoints will be saved.

2. Run the training script:

   bash script/train_clip_MIND_LLM.sh

To train the multi-modality projector, follow these steps:

1. Edit the file script/pretrain_phi3_MIND_LLM_LISA_addClinicInfo.sh:

   • Modify pretrain_vision_model to point to the pre-trained vision encoder model.
   • Update output_dir to specify where the projector checkpoints will be saved.

2. Update the DataArguments in the file src/train/train_MIND_LLM_LISA_addClinicInfo.py:

   • Adjust the parameters to match the data configuration.

3. Run the training script:

   bash script/pretrain_phi3_MIND_LLM_LISA_addClinicInfo.sh

1. Open and modify the following parameters in the fine-tuning scripts script/finetune_lora_phi3_MIND_LLM_diagnosis_LISA_addCli*.sh:

   • pretrain_vision_model: Path to the pre-trained vision encoder model.
   • pretrain_mm_mlp_adapter: Path to the multi-modality projector model.
   • output_dir: Directory where the fine-tuned model checkpoints will be saved.

2. Choose and execute the appropriate script based on your setup:

   • Without Gradient Checkpointing:

     bash script/finetune_lora_phi3_MIND_LLM_diagnosis_LISA_addCli_nockpt_grad.sh

   • With Gradient Checkpointing:

     bash script/finetune_lora_phi3_MIND_LLM_diagnosis_LISA_addCli_addckpt_grad.sh

• Enabling gradient checkpointing allows the batch size to increase from 2 to 4 with 2x NVIDIA A6000 GPUs.