📄 Paper | 🎬 Video Figure

This repository accompanies our research paper (CSCW 2025) titled "ThingMoji: User-Captured Cut-Outs For In-Stream Visual Communication" ThingMoji are user-captured cut-outs you can embed directly into chat to reference precise objects and moments in a live stream. StreamThing is the reference web app implementing creation, reuse, and time-based navigation of ThingMojis in one-to-many streaming.

• Brush-to-capture cut-outs from the live video (auto-pauses during selection for precision).

• Inline messaging: drop ThingMojis into chat like emojis; hover to enlarge; click to reveal the original frame for context.

• Gallery & reuse: personal + shared collections to quickly re-insert prior ThingMojis.

• Timeline overview: color-grouped segments show when ThingMojis appeared; hover to preview, click to jump to related chat.

• Scalable streaming: WebRTC via ION-SFU; chat via socket.io.

• Node.js 16+ (recommend 18+)
• npm (for the client and server)
• Python 3.9 (Conda recommended)
• GPU drivers/CUDA if you plan to use PyTorch with CUDA
• Go 1.20+ (only if you want to run Ion SFU locally)

ThingMoji-System/
  client/             # Web client (React + Webpack)
  server/             # Node signaling/WebSocket server (port 5000)
  ion-sfu/            # Ion SFU source (optional local media SFU)
  embedding.py        # Python embedding service (port 8000)
  weights/            # MobileSAM weights
  ...

• We do not include MobileSAM model weights in this repository (see weights/ in .gitignore).
• Please download mobile_sam.pt from the official MobileSAM repository and place it at weights/mobile_sam.pt.
  • MobileSAM: https://github.com/ChaoningZhang/MobileSAM
• The Python service looks for the weight file at ./weights/mobile_sam.pt.

Setup steps:

1. Create the folder if it doesn't exist:

mkdir weights

2. Download mobile_sam.pt from the MobileSAM project and put it in weights/.
3. Verify the file exists before starting services.

1. One-time install (or after dependency changes):

install.bat

2. Start all services (server, client, Python):

start.bat

• start.bat auto-detects Conda and runs conda run -n MobileSAM python embedding.py for Python. If Conda is missing, it starts without Python; run the Python service manually (see below).
• If you use a local Ion SFU, start it first in a separate terminal; start.bat does not launch Ion SFU.

Open three terminals for fine-grained control or troubleshooting.

Terminal A — Node server (port 5000):

cd server && npm install
npm start

Terminal B — Web client:

cd client && npm install
npm start

• Dev server port is configured in client/configs/webpack/dev.js (default 443). Change if needed.

Terminal C — Python embedding service (port 8000): install environment for MobileSAM.

conda create -n MobileSAM python=3.9
conda activate MobileSAM
pip install menpo opencv-python matplotlib flask flask_cors onnxruntime timm onnx scikit-learn
pip install torch torchvision torchaudio pytorch-cuda=11.8 -c pytorch -c nvidia

If you haven't already, clone and install MobileSAM:

git clone [email protected]:ChaoningZhang/MobileSAM.git
conda activate MobileSAM
cd MobileSAM
pip install -e .
python embedding.py

By default the client streams via a hosted Ion SFU. To use a local SFU instead:

1. Install Go

• Download and install from Go downloads. Restart your terminal after install.

2. Build and run the Ion SFU in ion-sfu/

- git clone https://github.com/ionorg/ion-sfu.git
- cd ion-sfu
- go build ./cmd/signal/json-rpc/main.go && ./main -c config.toml

websocket client side setup:

• see src/routes/socket.js

ion-sfu streamer side:

• src/routes/Streamer.js line 161-167

• src/routes/Viewer.js line 113-131

• The JSON-RPC signal server listens on port 7000 by default. Ensure your firewall allows local access.

Note: start.bat does not start Ion SFU. If you choose local SFU, run it in a separate terminal before starting the client.

• If WebSocket cannot connect, ensure the Node server is running on port 5000 and that your firewall allows local connections.
• If embedding requests fail, ensure the Python server is running on port 8000.

• OBS Virtual Camera issue in Google Chrome: see https://support.google.com/chrome/thread/2232679?hl=en.

If you use this repository in academic work, please cite the associated paper:

@article{Hu2025ThingMoji,
  author    = {Hu, Erzhen and Wan, Qian and Zhou, Changkong and Azim, Md Aashikur Rahman and
               Wang, PiaoHong and Hu, Xingyi and Zeng, Yuhan and Lu, Zhicong and Heo, Seongkook},
  title     = {ThingMoji: User-Captured Cut-Outs For In-Stream Visual Communication},
  journal   = {Proceedings of the ACM on Human-Computer Interaction},
  volume    = {9},
  number    = {7},
  articleno = {CSCW495},
  year      = {2025},
  month     = {November},
  doi       = {10.1145/3757676}
}

• MobileSAM: Based on the work by Chaoning Zhang et al. See the MobileSAM repository for details and licensing: ChaoningZhang/MobileSAM.
• Ion SFU: Streaming uses the Ion SFU signaling/server from the ION/Pion project. See: ionorg/ion-sfu.
• Thanks to our streamers and viewers for participating in the in-the-wild deployments, and to prior systems inspiring our design space (e.g., SnapStream, StreamSketch, timeline/backchannel work). Full related-work discussion is in the paper.