Skip to content

ajitpratap0/GoSQLX

Folders and files

NameName
Last commit message
Last commit date

Latest commit

 

History

70 Commits
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 

Repository files navigation

GoSQLX

GoSQLX Logo

⚡ High-Performance SQL Parser for Go ⚡

Go Version Release License: MIT PRs Welcome

Tests Go Report Card GoDoc

GitHub Stars GitHub Forks GitHub Watchers

Production-ready, high-performance SQL parsing SDK for Go
Zero-copy tokenization • Object pooling • Multi-dialect support • Unicode-first design

🚀 Installation⚡ Quick Start📚 Documentation💡 Examples📊 Benchmarks

User Guide API Docs Discussions Report Bug


🎯 Overview

GoSQLX is a high-performance SQL parsing library designed for production use. It provides zero-copy tokenization, intelligent object pooling, and comprehensive SQL dialect support while maintaining a simple, idiomatic Go API.

✨ Key Features

  • 🚀 Blazing Fast: 1.38M+ ops/sec sustained, 1.5M+ ops/sec peak throughput
  • 💾 Memory Efficient: 60-80% reduction through intelligent object pooling
  • 🔒 Thread-Safe: Race-free, linear scaling to 128+ cores
  • 🔗 Complete JOIN Support: All JOIN types (INNER/LEFT/RIGHT/FULL OUTER/CROSS/NATURAL) with proper tree logic
  • 🔄 Advanced SQL Features: CTEs with RECURSIVE support, Set Operations (UNION/EXCEPT/INTERSECT)
  • 🪟 Window Functions: Complete SQL-99 window function support with OVER clause, PARTITION BY, ORDER BY, frame specifications
  • 🌍 Unicode Support: Complete UTF-8 support for international SQL
  • 🔧 Multi-Dialect: PostgreSQL, MySQL, SQL Server, Oracle, SQLite
  • 📊 Zero-Copy: Direct byte slice operations, <1μs latency
  • 🏗️ Production Ready: Battle-tested with 0 race conditions detected, ~80-85% SQL-99 compliance

🎯 Performance Highlights (v1.4.0)

1.38M+ 8M+ <1μs 60-80% 30+
Ops/sec Tokens/sec Latency Memory Saved Total Tests

✅ Window FunctionsZero race conditions~80-85% SQL-99 complianceProduction validated

📈 Project Stats

Contributors Issues Pull Requests Downloads Last Commit Commit Activity

📦 Installation

Library Installation

go get github.com/ajitpratap0/GoSQLX

CLI Installation

# Install the CLI tool
go install github.com/ajitpratap0/GoSQLX/cmd/gosqlx@latest

# Or build from source
git clone https://github.com/ajitpratap0/GoSQLX.git
cd GoSQLX
go build -o gosqlx ./cmd/gosqlx

Requirements:

  • Go 1.19 or higher
  • No external dependencies

🚀 Quick Start

CLI Usage

# Validate SQL syntax
gosqlx validate "SELECT * FROM users WHERE active = true"

# Format SQL files with intelligent indentation
gosqlx format -i query.sql

# Analyze SQL structure and complexity
gosqlx analyze "SELECT COUNT(*) FROM orders GROUP BY status"

# Parse SQL to AST representation
gosqlx parse -f json complex_query.sql

Library Usage

package main

import (
    "fmt"
    "log"
    
    "github.com/ajitpratap0/GoSQLX/pkg/sql/tokenizer"
)

func main() {
    // Get tokenizer from pool (always return it!)
    tkz := tokenizer.GetTokenizer()
    defer tokenizer.PutTokenizer(tkz)
    
    // Tokenize SQL
    sql := "SELECT id, name FROM users WHERE age > 18"
    tokens, err := tkz.Tokenize([]byte(sql))
    if err != nil {
        log.Fatal(err)
    }
    
    // Process tokens
    fmt.Printf("Generated %d tokens\n", len(tokens))
    for _, token := range tokens {
        fmt.Printf("  %s (line %d, col %d)\n", 
            token.Token.Value,
            token.Start.Line,
            token.Start.Column)
    }
}

Advanced Example with AST

package main

import (
    "fmt"
    
    "github.com/ajitpratap0/GoSQLX/pkg/sql/tokenizer"
    "github.com/ajitpratap0/GoSQLX/pkg/sql/parser"
)

func AnalyzeSQL(sql string) error {
    // Tokenize
    tkz := tokenizer.GetTokenizer()
    defer tokenizer.PutTokenizer(tkz)
    
    tokens, err := tkz.Tokenize([]byte(sql))
    if err != nil {
        return fmt.Errorf("tokenization failed: %w", err)
    }
    
    // Parse to AST
    p := parser.NewParser()
    defer p.Release()
    
    ast, err := p.Parse(convertTokens(tokens))
    if err != nil {
        return fmt.Errorf("parsing failed: %w", err)
    }
    
    // Analyze AST
    fmt.Printf("Statement type: %T\n", ast)
    return nil
}

📚 Documentation

📖 Comprehensive Guides

Guide Description
CLI Guide Complete CLI documentation and usage examples
API Reference Complete API documentation with examples
Usage Guide Detailed patterns and best practices
Architecture System design and internal architecture
Troubleshooting Common issues and solutions

🚀 Getting Started

Document Purpose
Production Guide Deployment and monitoring
SQL Compatibility Dialect support matrix
Security Analysis Security assessment
Examples Working code examples

📋 Quick Links

🔄 Advanced SQL Features (v1.2.0)

GoSQLX now supports Common Table Expressions (CTEs) and Set Operations alongside complete JOIN support:

Common Table Expressions (CTEs)

// Simple CTE
sql := `
    WITH sales_summary AS (
        SELECT region, SUM(amount) as total 
        FROM sales 
        GROUP BY region
    ) 
    SELECT region FROM sales_summary WHERE total > 1000
`

// Recursive CTE for hierarchical data
sql := `
    WITH RECURSIVE employee_tree AS (
        SELECT employee_id, manager_id, name 
        FROM employees 
        WHERE manager_id IS NULL
        UNION ALL
        SELECT e.employee_id, e.manager_id, e.name 
        FROM employees e 
        JOIN employee_tree et ON e.manager_id = et.employee_id
    ) 
    SELECT * FROM employee_tree
`

// Multiple CTEs in single query
sql := `
    WITH regional AS (SELECT region, total FROM sales),
         summary AS (SELECT region FROM regional WHERE total > 1000)
    SELECT * FROM summary
`

Set Operations

// UNION - combine results with deduplication
sql := "SELECT name FROM users UNION SELECT name FROM customers"

// UNION ALL - combine results preserving duplicates
sql := "SELECT id FROM orders UNION ALL SELECT id FROM invoices"

// EXCEPT - set difference
sql := "SELECT product FROM inventory EXCEPT SELECT product FROM discontinued"

// INTERSECT - set intersection
sql := "SELECT customer_id FROM orders INTERSECT SELECT customer_id FROM payments"

// Left-associative parsing for multiple operations
sql := "SELECT a FROM t1 UNION SELECT b FROM t2 INTERSECT SELECT c FROM t3"
// Parsed as: (SELECT a FROM t1 UNION SELECT b FROM t2) INTERSECT SELECT c FROM t3

Complete JOIN Support

GoSQLX supports all JOIN types with proper left-associative tree logic:

// Complex JOIN query with multiple table relationships
sql := `
    SELECT u.name, o.order_date, p.product_name, c.category_name
    FROM users u
    LEFT JOIN orders o ON u.id = o.user_id  
    INNER JOIN products p ON o.product_id = p.id
    RIGHT JOIN categories c ON p.category_id = c.id
    WHERE u.active = true
    ORDER BY o.order_date DESC
`

// Parse with automatic JOIN tree construction
tkz := tokenizer.GetTokenizer()
defer tokenizer.PutTokenizer(tkz)

tokens, err := tkz.Tokenize([]byte(sql))
parser := parser.NewParser()
ast, err := parser.Parse(tokens)

// Access JOIN information
if selectStmt, ok := ast.Statements[0].(*ast.SelectStatement); ok {
    fmt.Printf("Found %d JOINs:\n", len(selectStmt.Joins))
    for i, join := range selectStmt.Joins {
        fmt.Printf("JOIN %d: %s (left: %s, right: %s)\n", 
            i+1, join.Type, join.Left.Name, join.Right.Name)
    }
}

Supported JOIN Types:

  • INNER JOIN - Standard inner joins
  • LEFT JOIN / LEFT OUTER JOIN - Left outer joins
  • RIGHT JOIN / RIGHT OUTER JOIN - Right outer joins
  • FULL JOIN / FULL OUTER JOIN - Full outer joins
  • CROSS JOIN - Cartesian product joins
  • NATURAL JOIN - Natural joins (implicit ON clause)
  • USING (column) - Single-column using clause

💻 Examples

Multi-Dialect Support

// PostgreSQL with array operators
sql := `SELECT * FROM users WHERE tags @> ARRAY['admin']`

// MySQL with backticks
sql := "SELECT `user_id`, `name` FROM `users`"

// SQL Server with brackets
sql := "SELECT [user_id], [name] FROM [users]"

Unicode and International SQL

// Japanese
sql := `SELECT "名前", "年齢" FROM "ユーザー"`

// Russian
sql := `SELECT "имя", "возраст" FROM "пользователи"`

// Arabic
sql := `SELECT "الاسم", "العمر" FROM "المستخدمون"`

// Emoji support
sql := `SELECT * FROM users WHERE status = '🚀'`

Concurrent Processing

func ProcessConcurrently(queries []string) {
    var wg sync.WaitGroup
    
    for _, sql := range queries {
        wg.Add(1)
        go func(query string) {
            defer wg.Done()
            
            // Each goroutine gets its own tokenizer
            tkz := tokenizer.GetTokenizer()
            defer tokenizer.PutTokenizer(tkz)
            
            tokens, _ := tkz.Tokenize([]byte(query))
            // Process tokens...
        }(sql)
    }
    
    wg.Wait()
}

📊 Performance

🎯 v1.0.0 Performance Improvements

Metric Previous v1.0.0 Improvement
Sustained Throughput 2.2M ops/s 946K+ ops/s Production Grade
Peak Throughput 2.2M ops/s 1.25M+ ops/s Enhanced
Token Processing 8M tokens/s 8M+ tokens/s Maintained
Simple Query Latency 200ns <280ns Optimized
Complex Query Latency N/A <1μs (CTE/Set Ops) New Capability
Memory Usage Baseline 60-80% reduction -70%
SQL-92 Compliance 40% ~70% +75%

Latest Benchmark Results

BenchmarkParserSustainedLoad-16           946,583      1,057 ns/op     1,847 B/op      23 allocs/op
BenchmarkParserThroughput-16            1,252,833        798 ns/op     1,452 B/op      18 allocs/op
BenchmarkParserSimpleSelect-16          3,571,428        279 ns/op       536 B/op       9 allocs/op
BenchmarkParserComplexSelect-16           985,221      1,014 ns/op     2,184 B/op      31 allocs/op

BenchmarkCTE/SimpleCTE-16                 524,933      1,891 ns/op     3,847 B/op      52 allocs/op
BenchmarkCTE/RecursiveCTE-16              387,654      2,735 ns/op     5,293 B/op      71 allocs/op
BenchmarkSetOperations/UNION-16           445,782      2,234 ns/op     4,156 B/op      58 allocs/op

BenchmarkTokensPerSecond-16               815,439      1,378 ns/op   8,847,625 tokens/sec

Performance Characteristics

Metric Value Details
Sustained Throughput 946K+ ops/sec 30s load testing
Peak Throughput 1.25M+ ops/sec Concurrent goroutines
Token Rate 8M+ tokens/sec Sustained processing
Simple Query Latency <280ns Basic SELECT (p50)
Complex Query Latency <1μs CTEs/Set Operations
Memory 1.8KB/query Complex SQL with pooling
Scaling Linear to 128+ Perfect concurrency
Pool Efficiency 95%+ hit rate Effective reuse

See PERFORMANCE_REPORT.md for detailed analysis.

🧪 Testing

# Run all tests with race detection
go test -race ./...

# Run benchmarks
go test -bench=. -benchmem ./...

# Generate coverage report
go test -coverprofile=coverage.out ./...
go tool cover -html=coverage.out

# Run specific test suites
go test -v ./pkg/sql/tokenizer/
go test -v ./pkg/sql/parser/

🏗️ Project Structure

GoSQLX/
├── pkg/
│   ├── models/              # Core data structures
│   │   ├── token.go        # Token definitions
│   │   └── location.go     # Position tracking
│   └── sql/
│       ├── tokenizer/       # Lexical analysis
│       │   ├── tokenizer.go
│       │   └── pool.go
│       ├── parser/          # Syntax analysis
│       │   ├── parser.go
│       │   └── expressions.go
│       ├── ast/            # Abstract syntax tree
│       │   ├── nodes.go
│       │   └── statements.go
│       └── keywords/        # SQL keywords
├── examples/               # Usage examples
│   └── cmd/
│       ├── example.go
│       └── example_test.go
├── docs/                   # Documentation
│   ├── API_REFERENCE.md
│   ├── USAGE_GUIDE.md
│   ├── ARCHITECTURE.md
│   └── TROUBLESHOOTING.md
└── tools/                  # Development tools

🛠️ Development

Prerequisites

  • Go 1.19+
  • Make (optional, for Makefile targets)
  • golint, staticcheck (for code quality)

Building

# Build the project
make build

# Run quality checks
make quality

# Run all tests
make test

# Clean build artifacts
make clean

Code Quality

# Format code
go fmt ./...

# Vet code
go vet ./...

# Run linter
golint ./...

# Static analysis
staticcheck ./...

🤝 Contributing

We welcome contributions! Please see CONTRIBUTING.md for guidelines.

How to Contribute

  1. Fork the repository
  2. Create a feature branch (git checkout -b feature/amazing-feature)
  3. Commit your changes (git commit -m 'Add amazing feature')
  4. Push to the branch (git push origin feature/amazing-feature)
  5. Open a Pull Request

Development Guidelines

  • Write tests for new features
  • Ensure all tests pass with race detection
  • Follow Go idioms and best practices
  • Update documentation for API changes
  • Add benchmarks for performance-critical code

📄 License

This project is licensed under the MIT License - see the LICENSE file for details.

🚀 Roadmap

Phase 1: Core SQL Enhancements (Q1 2025) - v1.1.0 ✅

  • Complete JOIN support (INNER/LEFT/RIGHT/FULL OUTER/CROSS/NATURAL)
  • Proper join tree logic with left-associative relationships
  • USING clause parsing (single-column, multi-column planned for Phase 2)
  • Enhanced error handling with contextual JOIN error messages
  • Comprehensive test coverage (15+ JOIN scenarios including error cases)
  • 🏗️ CTE foundation laid (AST structures, tokens, parser integration points)

Phase 2: CTE & Advanced Features (Q1 2025) - v1.2.0 ✅

  • Common Table Expressions (CTEs) with RECURSIVE support
  • Set operations (UNION/EXCEPT/INTERSECT with ALL modifier)
  • Left-associative set operation parsing
  • CTE column specifications and multiple CTE definitions
  • Integration of CTEs with set operations
  • Enhanced error handling with contextual messages
  • ~70% SQL-92 compliance achieved

Phase 3: Dialect Specialization (Q1 2025) - v2.0.0

  • 📋 PostgreSQL arrays, JSONB, custom types
  • 📋 MySQL-specific syntax and functions
  • 📋 SQL Server T-SQL extensions
  • 📋 Multi-dialect parser with auto-detection

Phase 4: Intelligence Layer (Q2 2025) - v2.1.0

  • 📋 Query optimization suggestions
  • 📋 Security vulnerability detection
  • 📋 Performance analysis and hints
  • 📋 Schema validation

📄 Full Architectural Review & Roadmap

🤝 Community & Support

Join Our Community

GitHub Discussions GitHub Issues

Get Help

Channel Purpose Response Time
🐛 Bug Reports Report issues Community-driven
💡 Feature Requests Suggest improvements Community-driven
💬 Discussions Q&A, ideas, showcase Community-driven
🔒 Security Report vulnerabilities Best effort

👥 Contributors

Core Team

Contributors

How to Contribute

We love your input! We want to make contributing as easy and transparent as possible.

Contributing Guide Code of Conduct Start Contributing

Quick Contribution Guide

  1. 🍴 Fork the repo
  2. 🔨 Make your changes
  3. ✅ Ensure tests pass (go test -race ./...)
  4. 📝 Update documentation
  5. 🚀 Submit a PR

🎯 Use Cases

Industry Use Case Benefits
🏦 FinTech SQL validation & auditing Fast validation, compliance tracking
📊 Analytics Query parsing & optimization Real-time analysis, performance insights
🛡️ Security SQL injection detection Pattern matching, threat prevention
🏗️ DevTools IDE integration & linting Syntax highlighting, auto-completion
📚 Education SQL learning platforms Interactive parsing, error explanation
🔄 Migration Cross-database migration Dialect conversion, compatibility check

📊 Who's Using GoSQLX

Using GoSQLX in production? Let us know!

📈 Project Metrics

Performance Benchmarks

graph LR
    A[SQL Input] -->|946K+ ops/sec| B[Tokenizer]
    B -->|8M+ tokens/sec| C[Parser]
    C -->|Zero-copy| D[AST]
    D -->|60-80% less memory| E[Output]
Loading

🗺️ Roadmap

Release Timeline

Version Status Release Date Features
v0.9.0 ✅ Released 2024-01-15 Initial release
v1.0.0 ✅ Released 2024-12-01 Production ready, +47% performance
v1.1.0 ✅ Released 2025-01-03 Complete JOIN support, error handling
v1.2.0 ✅ Released 2025-08-15 CTEs, set operations, ~70% SQL-92 compliance
v1.4.0 🎉 Current 2025-09-07 Production CLI, high-performance commands, memory leak fixes
v1.3.0 ✅ Previous 2025-09-04 Window functions, ~80-85% SQL-99 compliance
v2.0.0 🔮 Future Q4 2025 Dialect specialization, advanced features

Full Roadmap

💖 Support This Project

If GoSQLX helps your project, please consider:

Star This Repo

Other Ways to Support

  • ⭐ Star this repository
  • 🐦 Tweet about GoSQLX
  • 📝 Write a blog post
  • 🎥 Create a tutorial
  • 🐛 Report bugs
  • 💡 Suggest features
  • 🔧 Submit PRs

📜 License

This project is licensed under the MIT License - see the LICENSE file for details.


Built with ❤️ by the GoSQLX Team

Star Us Fork Me Watch

Copyright © 2024 GoSQLX. All rights reserved.

About

A Go-based SQL parsing toolkit

Resources

License

Contributing

Security policy

Stars

Watchers

Forks

Packages

No packages published

Contributors 3

  •  
  •  
  •