[Paper (NeurIPS 2025)] [Paper (arXiv)]

by Dongjun Hwang, Yejin Kim, Minyoung Lee, Seong Joon Oh, Junsuk Choe

This repo contains the code for our paper "OVS Meets Continual Learning: Towards Sustainable Open-Vocabulary Segmentation".

This code is based on fc-clip and mask2former. Many thanks to all for sharing the excellent code!

Abstract: Open-Vocabulary Segmentation (OVS) aims to segment classes that are not present in the training dataset. However, most existing studies assume that the training data is fixed in advance, overlooking more practical scenarios where new datasets are continuously collected over time. To address this, we first analyze how existing OVS models perform under such conditions. In this context, we explore several approaches such as retraining, fine-tuning, and continual learning but find that each of them has clear limitations. To address these issues, we propose ConOVS, a novel continual learning method based on a Mixture-of-Experts framework. ConOVS dynamically combines expert decoders based on the probability that an input sample belongs to the distribution of each incremental dataset. Through extensive experiments, we show that ConOVS consistently outperforms existing methods across pre-training, incremental, and zero-shot test datasets, effectively expanding the recognition capabilities of OVS models when data is collected sequentially.

• Upload the paper to arXiv
• Update code

See Installation Instructions.

See Preparing Datasets.

Please run the below command and put that on the root directory of this repo.

gdown 1csp3trVKhc90aUZO4S__V23iwsQTe0od

And move this file to ./checkpoints/fcclip_cocopan.pth.

You can evaluate all the below method without training the model as we already provide the checkpoint [download link].

Downloaded files should be in the project directory like below:

ConOVS/
├── ...
├── checkpoints/
│   ├── fcclip_cocopan.pth
│   ├── finetuning/
│   │   ├── cityscapes.pth
│   ...
├── mvn_dist/
│   ├── cityscapes_gmm.pkl
│   ...

Notes:

• Fine-tuning: Name of the file indicates the sequence of the incremental datasets.
• Retraining: Name of the file indicates all incremental datasets.
• ConOVS: You should download the checkpoint from ./finetuning in the link above, and MVN distribution from here.

This document provides a brief intro of the usage of our method and other comparisons.

This document is derived from Getting Started with Detectron2.

Our method consists of two main steps:

• (1) fine-tuning the fc-clip model on the first dataset, and
• (2) making the MVN distribution (prototypes) to combine the fine-tuned models.

We already provided the fine-tuned models and the prototypes in this link.

Therefore, if you want to evaluate our method, you can skip the steps (1) and (2) and directly go to the Evaluation section.

We provide a script train_net.py, that is made to train all the configs.

To train a model with "train_net.py", first setup the corresponding datasets following datasets/README.md, then beginning with fine-tuning fc-clip:

# scripts/(scenario number)/(method name).sh
bash scripts/s1/finetuning.sh

Other methods can be trained in a similar manner. For example, to train the model with EWC, use

bash scripts/s1/ewc.sh

To generate the multivariate normal distribution,

bash scripts/s1/conovs_gmm.sh

To evaluate ConOVS's performance, use

bash scripts/s1/conovs_eval.sh

Other methods can be evaluated in a similar manner. For example, to evaluate the fine-tuned model, use

bash scripts/s1/finetuning_eval.sh

For more options, see python train_net.py -h.

TBD