Skip to content

Build reproducible, configurable workflows in the terminal for research, development, and analysis with intelligent document processing and sophisticated context management.

License

Notifications You must be signed in to change notification settings

jdmonaco/wireflow

Folders and files

NameName
Last commit message
Last commit date

Latest commit

Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 

Repository files navigation

WireFlow: Reproducible AI Workflows for Research and Development

Version: 0.3.0 (pre-release)

A terminal-based tool for building reproducible AI workflows for research, development, and analysis, featuring flexible configuration cascades, comprehensive document processing (PDFs, Office files, images), workflow dependencies and chaining, and intelligent context management.

Key Features

  • 🎯 Git-like Project Discovery: Automatic .workflow/ directory detection walking up from any subdirectory, enabling project-aware execution from anywhere in your tree (stops at $HOME for safety).
  • πŸ“„ Native Document Processing: Unified handling of PDFs (32MB, joint text+visual analysis), Microsoft Office files (.docx/.pptx auto-converted via LibreOffice), images (Vision API with automatic resizing), and text files with intelligent format detection and caching.
  • πŸ”§ Configuration Cascade with Pass-Through: Multi-tier inheritance (global β†’ ancestors β†’ project β†’ workflow β†’ CLI) where empty values automatically inherit from parent tiers while explicit values override and decouple, enabling change-once affect-many configuration management.
  • πŸ—οΈ Nested Project Support: Automatic discovery and inheritance from all ancestor projects in the directory hierarchy, with transparent source tracking showing exactly where each configuration value originates.
  • πŸ”— Workflow Dependencies & Chaining: Create multi-stage processing pipelines with DEPENDS_ON declarations, automatically passing outputs as context to dependent workflows via hardlinks for efficient DAG-based orchestration.
  • πŸ“¦ Semantic Content Aggregation: Distinguish INPUT documents (primary analysis targets) from CONTEXT materials (supporting information) using three methods: glob patterns, explicit file lists, and workflow dependencies, with optimized ordering for cost-effective caching.
  • πŸ’° Prompt Caching Architecture: Strategic cache breakpoint placement (max 4) at semantic boundaries enables 90% cost reduction on stable content, with intelligent ordering (system prompts β†’ project descriptions β†’ PDFs β†’ text β†’ images β†’ task) and date-only timestamps to prevent minute-by-minute invalidation.
  • πŸ“š Citations Support: Optional Anthropic citations API integration with document mapping for source attribution, generating sidecar citation files and enabling proper references for AI-generated content.
  • ⚑ Dual Execution Modes: Persistent workflows with configuration, context, dependencies, and outputs for iterative development, or lightweight task mode for one-off queries without workflow directories, both sharing optimized execution logic.
  • πŸ’Ύ Safe Output Management: Automatic timestamped backups before overwriting, hardlinked copies for convenient access (.workflow/output/), atomic file operations, and format-specific post-processing (mdformat, jq).
  • πŸ“Š Dual Token Estimation: Fast heuristic character-based estimates plus exact counts via Anthropic's Token Counting API, with detailed breakdowns showing contribution from system prompts, task, input documents, context, and images.
  • 🌊 Streaming & Batch Modes: Real-time streaming output with SSE parsing for immediate feedback, or single-request batch mode with pager display, both supporting identical configuration and context aggregation.

Quick Start

Install

First, clone the repo and then link the script into your PATH. For example:

# Clone repository
git clone https://github.com/jdmonaco/wireflow.git
cd wireflow

# Add to PATH (example using ~/.local/bin)
ln -s "$(pwd)/wireflow.sh" ~/.local/bin/wfw

Setup

export ANTHROPIC_API_KEY="sk-ant-..."
export PATH="$HOME/.local/bin:$PATH"

Create Your First Workflow

# Initialize project
cd my-project
wfw init .

# Create workflow
wfw new analyze-data

# Edit workflow config
wfw edit analyze-data

# Run with context
wfw run analyze-data --context-file data.csv --stream

Your project files and folders are treated as read-only. All WireFlow files are maintained in a .workflow/ subfolder.

Documentation

πŸ“š Complete documentation: https://docs.joemona.co/wireflow/

Quick Links

Core Concepts

Workflows

Persistent, named tasks with configuration and outputs:

wfw new 01-analysis
wfw run 01-analysis --stream

Tasks

Lightweight, one-off execution without persistence:

wfw task -i "Summarize these notes" --context-file notes.md

Dependencies

Chain workflows to build pipelines:

wfw run 02-report --depends-on 01-analysis --stream

Configuration

Multi-tier cascade with pass-through:

Global (~/.config/wireflow/config)
    ↓
Ancestor Projects (grandparent β†’ parent)
    ↓
Project (.workflow/config)
    ↓
Workflow (.workflow/<workflow_name>/config)
    ↓
CLI Flags (--model, --temperature, etc.)

Usage Examples

Simple Analysis

wfw init my-analysis
wfw new analyze-data
wfw run analyze-data --context-file data.csv --stream

Workflow Chain

wfw run 00-context --stream
wfw run 01-outline --depends-on 00-context --stream
wfw run 02-draft --depends-on 00-context,01-outline --stream

Quick Query

wfw task -i "Extract action items" --context-file meeting-notes.md

Requirements

Configuration

Global Configuration

Auto-created on first use at ~/.config/wireflow/:

  • config - Global defaults for all projects
  • prompts/base.txt - Default system prompt
  • tasks/ - Named task templates (optional)

Project Configuration

Created by wfw init:

  • .workflow/config - Project-level settings
  • .workflow/project.txt - Project description (optional)
  • .workflow/<workflow-name>/ - Individual workflows

Help

wfw help              # Show all subcommands
wfw help <subcommand> # Detailed subcommand help
wfw <subcommand> -h   # Quick help

Contributing

Contributions welcome! See CONTRIBUTING.md for guidelines.

License

MIT License - see LICENSE for details.

Resources


Made with Claude Code

About

Build reproducible, configurable workflows in the terminal for research, development, and analysis with intelligent document processing and sophisticated context management.

Topics

Resources

License

Stars

Watchers

Forks

Contributors 2

  •  
  •  

Languages