TUTT (The Ultimate Time Tracker) is a prototype time tracker that sends screenshots to the OpenAI Vision API to categorize what you're doing on your computer.

If you create a good prompt that describes your normal activities, the result can be quite accurate.

Here's how the UI might look like if your day job is at Acme.ai, you're working on a food app idea called "Keto App" as a side project, and sometimes you get sucked into Twitter or Instagram that is completely unrelated to anything.

List of time tracking events	Details of time tracking event

I need to track billable hours for freelance work, and I want a tool that requires the least amount of effort. The best app I could find was Timely App, but I feel their auto-categorization feature isn't accurate enough for my needs and I still need to put in manual effort to categorize the hours.

• This is just a proof of concept and is very unpolished. But if you're interested in a polished version, give the repo a ⭐!
• The OpenAI API costs are approximately $1/day with the default settings. Open an issue if you are interested in support for running a Llava open source model.
• Since your screenshots are being sent to OpenAI, there are a lot of security and privacy risks with this app. You should only use it if you understand those risks. Note that the data is not sent anywhere else besides OpenAI.

1. OSX - this only works on OSX, due to the quickgrab dependency.
2. GPT4 access - At the time of writing, according to the OpenAI website:

GPT-4 with vision is currently available to all developers who have access to GPT-4 via the gpt-4-vision-preview model and the Chat Completions API which has been updated to support image inputs. If you're a Pay-As-You-Go customer and you've made a successful payment of $1 or more, you'll be able to access the GPT-4 API (8k).

git clone https://github.com/akrabat/QuickGrab.git
cp QuickGrab/quickgrab /usr/local/bin

This should already be installed, run python --version or python3 --version to double check. If not, you can install it with brew install python.

brew install node

It has been tested with:

• nodejs version v18.16.0
• npm version 9.5.1

The example prompt and example categories won't give any meaningful results, so you will need to create a custom prompt and corresponding categories. It should only take a few minutes.

Copy the example prompt:

cp prompt_example.txt prompt.txt

and customize prompt.txt to fit your use case.

The categories listed in the prompt must be formatted with category numbers like Category #<num>, as is done in the example prompt:

Category #1 - <title of category 1>

... 

Category #2 - <title of category 2>

You can add as many categories as you want, but these category ids need to match the category_id fields in the categories.json file that you will create in the next step.

It's recommended to spend some time designing your prompt! You will get much more accurate results if your prompt is good.

cp categories_example.json categories.json

Add the categories you mentioned in your custom prompt.

The category ids need to match the category_id fields in the categories.json file, and the category names should be concise.

Tutt consists of two components:

1. A python screen grabber
2. A Vue.js web UI to view the captured screenshots with their categories

This section walks you through getting the python screen grabber running.

You can run it in the background with tmux or similar.

To isolate your python dependencies and avoid any potential dependency conflicts, you can create and activate a virtual environment:

python3 -m venv tutt
source tutt/bin/activate

or with conda:

conda create --name tutt python=3.11
conda activate tutt

pip install -r requirements.txt

You must have an OpenAI account that allows access to the Vision API.

export OPENAI_API_KEY=<your OPENAI API key>

cd src
python tutt.py

You can run it in the background with tmux or similar. When your machine restarts, you will have to manually restart the script.

cd ui/tutt
npm install

./start_server.sh

It will display the localhost address that you can view in a browser.

None yet, but if you are interested in this app, give the repo a star and I will try to put some more time into building this into a usable app! Feel free to file issues for any improvements you'd like to see.

• rem - an open source approach to locally record and enable searching everything you view on your Apple Silicon.
• ActivityWatch - open source time tracker.