Google Colab notebook environment with embedded Gemini assistant. Students could interact with Gemini in two main ways: code cell generations (generateCode) and a conversational side chatbot (converse). There are also additional buttons that support error explanation, code explanation, and visualization generation.

Content Directory

├── README.md
├── hw_materials/
│   ├── colab_save_history.js
│   ├── DSPM-HW0.ipynb
│   ├── DSPM-HW0-refsol-rubrics.ipynb
│   └── DSPM-HW1-4-refsol-rubrics.ipynb
└── data_process_analysis/
    ├── data_analysis_utils.py
    ├── Data_Process.ipynb
    ├── Log_Analysis.ipynb
    ├── prompts/
    │   ├── seg_prompt.md
    │   └── step_prompt.md
    └── Data_Visualization.ipynb

Course materials and assignments used in the study:

• colab_save_history.js: TamperMonkey userscript that automatically saves code execution history and chat interactions with Gemini in Google Colab notebooks for data collection purposes
• DSPM-HW0.ipynb: Template & instructions shared to student. In-lecture activity focused on Honda Accord car price analysis - used as a warm-up for homework setup & diagnostic tasks for students
• DSPM-HW0-refsol-rubrics.ipynb: Reference solution and grading rubrics for HW0
• DSPM-HW1-4-refsol-rubrics.ipynb: Reference solutions and grading rubrics for homework assignments 1-4

Data processing and analysis pipeline for the study:

• data_analysis_utils.py: Python utility functions and mappings to process and prepare (survey) data for behavioral coding and statistical analysis
• Data_Process.ipynb: Main notebook for cleaning and processing raw surveys & interaction logs, and preparing data for analysis
• Log_Analysis.ipynb: LLM-automated analysis of behavioral logs, including episode segmentation, step and AI usage behaviors annotations
  • prompts/seg_prompt.md: Prompt for segmenting continuous interaction logs into episodes based on student intent
  • prompts/step_prompt.md: Prompt for breaking down episodes into fine-grained steps (Intent → Input → Understand → Assess) and annotating AI usage patterns
• Data_Visualization.ipynb: Visualizations for paper figures and exploratory data analysis

Qianou Ma, Kenneth Koedinger, and Tongshuang Wu. 2025. Not Everyone Wins with LLMs: Behavioral Patterns and Pedagogical Implications in AI-assisted Data Analysis.

@misc{ma2025promptsreflectiondesigningreflective,
      title={Not Everyone Wins with LLMs: Behavioral Patterns and Pedagogical Implications in AI-assisted Data Analysis}, 
      author={Qianou Ma and Kenneth Koedinger and Tongshuang Wu},
      year={2025},
      eprint={2509.21890},
      archivePrefix={arXiv},
      primaryClass={cs.HC},
      url={http://arxiv.org/abs/2509.21890}, 
}