Skip to content

Review gpt-5 batch processing feature request #1820

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Draft
wants to merge 1 commit into
base: main
Choose a base branch
from
+61 −4
Draft

Review gpt-5 batch processing feature request #1820

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
50 changes: 48 additions & 2 deletions docs/concepts/batch.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -5,7 +5,7 @@ Batch processing allows you to send multiple requests in a single operation, whi
## Supported Providers

### OpenAI
- **Models**: gpt-4o, gpt-4.1-mini, gpt-4-turbo, etc.
- **Models**: gpt-4o, gpt-4.1-mini, gpt-4-turbo, gpt-5-turbo, gpt-5, gpt-5-preview, o1-preview, o1-mini, etc.
- **Cost Savings**: 50% discount on batch requests
- **Format**: Uses OpenAI's batch API with JSON schema for structured outputs

Expand All @@ -21,6 +21,49 @@ Batch processing allows you to send multiple requests in a single operation, whi
- **Format**: Uses Google Cloud Vertex AI batch prediction API
- **Documentation**: [Google Cloud Batch Prediction](https://cloud.google.com/vertex-ai/generative-ai/docs/multimodal/batch-prediction-gemini)

## GPT-5 Support

Instructor fully supports GPT-5 models for batch processing, including:

- `gpt-5-turbo` - Latest GPT-5 model optimized for speed and cost
- `gpt-5` - Standard GPT-5 model
- `gpt-5-preview` - Preview version of GPT-5
- `o1-preview` - Advanced reasoning model
- `o1-mini` - Smaller version of the O1 model

All GPT-5 models work with the same batch API and provide the same 50% cost savings.

### GPT-5 Example

```python
from instructor.batch import BatchProcessor
from pydantic import BaseModel

class Analysis(BaseModel):
reasoning: str
conclusion: str
confidence: float

# Use GPT-5 for complex reasoning tasks
processor = BatchProcessor("openai/o1-preview", Analysis)

messages_list = [
[
{"role": "system", "content": "Analyze the provided data and give a detailed reasoning."},
{"role": "user", "content": "Sales data shows a 15% increase in Q1, but customer complaints rose by 23%. Analyze the situation."}
]
]

# Create and submit batch
batch_file = processor.create_batch_from_messages(
messages_list=messages_list,
max_tokens=500,
temperature=0.1
)

batch_id = processor.submit_batch(batch_file)
```

## Basic Usage

The `BatchProcessor` provides a complete interface for batch processing including job submission, status monitoring, and result retrieval.
Expand All @@ -36,8 +79,11 @@ class User(BaseModel):
name: str
age: int

# Create processor with model specification
# Create processor with model specification (supports GPT-5 models)
processor = BatchProcessor("openai/gpt-4.1-mini", User)
# GPT-5 examples:
# processor = BatchProcessor("openai/gpt-5-turbo", User)
# processor = BatchProcessor("openai/o1-preview", User)

# Prepare your message conversations
messages_list = [
Expand Down
12 changes: 11 additions & 1 deletion examples/batch_api/run_batch_test.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -5,7 +5,7 @@
Creates a batch job to extract User(name: str, age: int) data from text examples.

Supports:
- OpenAI: openai/gpt-4o-mini, openai/gpt-4o, etc.
- OpenAI: openai/gpt-4o-mini, openai/gpt-4o, openai/gpt-5-turbo, openai/o1-preview, openai/o1-mini, etc.
- Anthropic: anthropic/claude-3-5-sonnet-20241022, anthropic/claude-3-opus-20240229, etc.
- Google: google/gemini-2.5-flash, google/gemini-pro, etc.

Expand All @@ -17,6 +17,11 @@
# OpenAI
export OPENAI_API_KEY="your-key"
python run_batch_test.py --model "openai/gpt-4o-mini"

# OpenAI GPT-5
export OPENAI_API_KEY="your-key"
python run_batch_test.py --model "openai/gpt-5-turbo"
python run_batch_test.py --model "openai/o1-preview"

# Anthropic
export ANTHROPIC_API_KEY="your-key"
Expand Down Expand Up @@ -830,6 +835,11 @@ def list_models():
typer.echo(" • openai/gpt-4o-mini")
typer.echo(" • openai/gpt-4o")
typer.echo(" • openai/gpt-4-turbo")
typer.echo(" • openai/gpt-5-turbo")
typer.echo(" • openai/gpt-5")
typer.echo(" • openai/gpt-5-preview")
typer.echo(" • openai/o1-preview")
typer.echo(" • openai/o1-mini")
typer.echo()

typer.echo("Anthropic:")
Expand Down
3 changes: 2 additions & 1 deletion instructor/batch/__init__.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -6,7 +6,7 @@
tracking for type-safe handling of batch results.

Supported Providers:
- OpenAI: 50% cost savings on batch requests
- OpenAI: 50% cost savings on batch requests (supports GPT-4, GPT-5, O1 models)
- Anthropic: 50% cost savings on batch requests (Message Batches API)

Features:
Expand All @@ -24,6 +24,7 @@ class User(BaseModel):
age: int

processor = BatchProcessor("openai/gpt-4o-mini", User)
# GPT-5 example: processor = BatchProcessor("openai/gpt-5-turbo", User)
batch_id = processor.submit_batch("requests.jsonl")

# Results are BatchSuccess[T] | BatchError union types
Expand Down
Loading