Disclaimer:
This project is a learning exercise only, created to explore the interaction between git source code, language models (OpenAI & Ollama), and automated test generation.
The output should not be relied upon, use at your own discretion.
This project extracts C source code from a git commit, summarises it using an AI model (OpenAI or Ollama), and generates Python test cases using pytest and lib389.
- Summarizes C code from git commits
- Supports OpenAI GPT-4 (cloud) and Ollama (local)
- Generates pytest test stubs based on code analysis
- Saves summaries and test cases to file prepended with the commit hash
- CLI based evaluation pipeline
Below is the setup for the environment I used during playtime. Your setup may differ slightly.
curl -fsSL https://ollama.com/install.sh | shollama pull codellama:7b-instructollama run codellama:7b-instructmkdir -p ~/projects && cd ~/projectsgit clone https://github.com/389ds/389-ds-base.gitgit clone https://github.com/jchapma/ai.git
cd ai/test-generationpython3 -m venv .venv
source .venv/bin/activate
pip install -r requirements.txtecho 'export OPENAI_API_KEY=[YOUR OPENAI API KEY]' >> ~/.bashrc
source ~/.bashrcusage: test_generation.py [-h] -r REPO -c COMMIT -m {ollama,openai} [-o OUTPUTDIR] (--summary | --train)
Summarise C source code and generate pytest stubs from a specific git commit using either OpenAI or Ollama language models. Intended as a learning exercise.
options:
-h, --help show this help message and exit
-r, --repo REPO path to the Git repository
-c, --commit COMMIT commit hash to analyse
-m, --model {ollama,openai}
AI model to use
-o, --outputdir OUTPUTDIR
output directory (default: summary)
--summary Run in summary mode (extracts and analyses code)
--train Run in training mode (Not implemented)To help demonstrate the output of the tool, some example results have been included.
I picked a commit with only C code and an associated pytest, the pytest could have been used as a comparison.
python test-generation/test_generation.py --summary -r ../389-ds-base/ -c 2108b4f63abbce6e58efc25b83993b0c1eccd5a3 -m ollama -o summary_ollama
summary_ollama/
├── 2108b4f6_code.c # Extracted source code from commit hash
├── 2108b4f6_summary.log # Summary and pytest, generated by ollama model
└── 2108b4f6_test.py # Extracted pytest from commit hash
python test-generation/test_generation.py --summary -r ../389-ds-base/ -c 2108b4f63abbce6e58efc25b83993b0c1eccd5a3 -m openai -o summary_openai
summary_openai/
├── 2108b4f6_code.c # Extracted source code from commit hash
├── 2108b4f6_summary.log # Summary and pytest, generated by openai model
└── 2108b4f6_test.py # Extracted pytest from commit hash