Production-ready, high-performance SQL parsing SDK for Go
Zero-copy tokenization โข Object pooling โข Multi-dialect support โข Unicode-first design
๐ Installation โข โก Quick Start โข ๐ Documentation โข ๐ก Examples โข ๐ Benchmarks
GoSQLX is a high-performance SQL parsing library designed for production use. It provides zero-copy tokenization, intelligent object pooling, and comprehensive SQL dialect support while maintaining a simple, idiomatic Go API.
- ๐ Blazing Fast: 1.38M+ ops/sec sustained, 1.5M+ ops/sec peak throughput
- ๐พ Memory Efficient: 60-80% reduction through intelligent object pooling
- ๐ Thread-Safe: Race-free, linear scaling to 128+ cores
- ๐ Complete JOIN Support: All JOIN types (INNER/LEFT/RIGHT/FULL OUTER/CROSS/NATURAL) with proper tree logic
- ๐ Advanced SQL Features: CTEs with RECURSIVE support, Set Operations (UNION/EXCEPT/INTERSECT)
- ๐ช Window Functions: Complete SQL-99 window function support with OVER clause, PARTITION BY, ORDER BY, frame specifications
- ๐ Unicode Support: Complete UTF-8 support for international SQL
- ๐ง Multi-Dialect: PostgreSQL, MySQL, SQL Server, Oracle, SQLite
- ๐ Zero-Copy: Direct byte slice operations, <1ฮผs latency
- ๐๏ธ Production Ready: Battle-tested with 0 race conditions detected, ~80-85% SQL-99 compliance
| 1.38M+ | 8M+ | <1ฮผs | 60-80% | 30+ |
|---|---|---|---|---|
| Ops/sec | Tokens/sec | Latency | Memory Saved | Total Tests |
โ Window Functions โข Zero race conditions โข ~80-85% SQL-99 compliance โข Production validated
go get github.com/ajitpratap0/GoSQLX# Install the CLI tool
go install github.com/ajitpratap0/GoSQLX/cmd/gosqlx@latest
# Or build from source
git clone https://github.com/ajitpratap0/GoSQLX.git
cd GoSQLX
go build -o gosqlx ./cmd/gosqlxRequirements:
- Go 1.19 or higher
- No external dependencies
# Validate SQL syntax
gosqlx validate "SELECT * FROM users WHERE active = true"
# Format SQL files with intelligent indentation
gosqlx format -i query.sql
# Analyze SQL structure and complexity
gosqlx analyze "SELECT COUNT(*) FROM orders GROUP BY status"
# Parse SQL to AST representation
gosqlx parse -f json complex_query.sqlpackage main
import (
"fmt"
"log"
"github.com/ajitpratap0/GoSQLX/pkg/sql/tokenizer"
)
func main() {
// Get tokenizer from pool (always return it!)
tkz := tokenizer.GetTokenizer()
defer tokenizer.PutTokenizer(tkz)
// Tokenize SQL
sql := "SELECT id, name FROM users WHERE age > 18"
tokens, err := tkz.Tokenize([]byte(sql))
if err != nil {
log.Fatal(err)
}
// Process tokens
fmt.Printf("Generated %d tokens\n", len(tokens))
for _, token := range tokens {
fmt.Printf(" %s (line %d, col %d)\n",
token.Token.Value,
token.Start.Line,
token.Start.Column)
}
}package main
import (
"fmt"
"github.com/ajitpratap0/GoSQLX/pkg/sql/tokenizer"
"github.com/ajitpratap0/GoSQLX/pkg/sql/parser"
)
func AnalyzeSQL(sql string) error {
// Tokenize
tkz := tokenizer.GetTokenizer()
defer tokenizer.PutTokenizer(tkz)
tokens, err := tkz.Tokenize([]byte(sql))
if err != nil {
return fmt.Errorf("tokenization failed: %w", err)
}
// Parse to AST
p := parser.NewParser()
defer p.Release()
ast, err := p.Parse(convertTokens(tokens))
if err != nil {
return fmt.Errorf("parsing failed: %w", err)
}
// Analyze AST
fmt.Printf("Statement type: %T\n", ast)
return nil
}| Guide | Description |
|---|---|
| CLI Guide | Complete CLI documentation and usage examples |
| API Reference | Complete API documentation with examples |
| Usage Guide | Detailed patterns and best practices |
| Architecture | System design and internal architecture |
| Troubleshooting | Common issues and solutions |
| Document | Purpose |
|---|---|
| Production Guide | Deployment and monitoring |
| SQL Compatibility | Dialect support matrix |
| Security Analysis | Security assessment |
| Examples | Working code examples |
GoSQLX now supports Common Table Expressions (CTEs) and Set Operations alongside complete JOIN support:
// Simple CTE
sql := `
WITH sales_summary AS (
SELECT region, SUM(amount) as total
FROM sales
GROUP BY region
)
SELECT region FROM sales_summary WHERE total > 1000
`
// Recursive CTE for hierarchical data
sql := `
WITH RECURSIVE employee_tree AS (
SELECT employee_id, manager_id, name
FROM employees
WHERE manager_id IS NULL
UNION ALL
SELECT e.employee_id, e.manager_id, e.name
FROM employees e
JOIN employee_tree et ON e.manager_id = et.employee_id
)
SELECT * FROM employee_tree
`
// Multiple CTEs in single query
sql := `
WITH regional AS (SELECT region, total FROM sales),
summary AS (SELECT region FROM regional WHERE total > 1000)
SELECT * FROM summary
`// UNION - combine results with deduplication
sql := "SELECT name FROM users UNION SELECT name FROM customers"
// UNION ALL - combine results preserving duplicates
sql := "SELECT id FROM orders UNION ALL SELECT id FROM invoices"
// EXCEPT - set difference
sql := "SELECT product FROM inventory EXCEPT SELECT product FROM discontinued"
// INTERSECT - set intersection
sql := "SELECT customer_id FROM orders INTERSECT SELECT customer_id FROM payments"
// Left-associative parsing for multiple operations
sql := "SELECT a FROM t1 UNION SELECT b FROM t2 INTERSECT SELECT c FROM t3"
// Parsed as: (SELECT a FROM t1 UNION SELECT b FROM t2) INTERSECT SELECT c FROM t3GoSQLX supports all JOIN types with proper left-associative tree logic:
// Complex JOIN query with multiple table relationships
sql := `
SELECT u.name, o.order_date, p.product_name, c.category_name
FROM users u
LEFT JOIN orders o ON u.id = o.user_id
INNER JOIN products p ON o.product_id = p.id
RIGHT JOIN categories c ON p.category_id = c.id
WHERE u.active = true
ORDER BY o.order_date DESC
`
// Parse with automatic JOIN tree construction
tkz := tokenizer.GetTokenizer()
defer tokenizer.PutTokenizer(tkz)
tokens, err := tkz.Tokenize([]byte(sql))
parser := parser.NewParser()
ast, err := parser.Parse(tokens)
// Access JOIN information
if selectStmt, ok := ast.Statements[0].(*ast.SelectStatement); ok {
fmt.Printf("Found %d JOINs:\n", len(selectStmt.Joins))
for i, join := range selectStmt.Joins {
fmt.Printf("JOIN %d: %s (left: %s, right: %s)\n",
i+1, join.Type, join.Left.Name, join.Right.Name)
}
}Supported JOIN Types:
- โ
INNER JOIN- Standard inner joins - โ
LEFT JOIN/LEFT OUTER JOIN- Left outer joins - โ
RIGHT JOIN/RIGHT OUTER JOIN- Right outer joins - โ
FULL JOIN/FULL OUTER JOIN- Full outer joins - โ
CROSS JOIN- Cartesian product joins - โ
NATURAL JOIN- Natural joins (implicit ON clause) - โ
USING (column)- Single-column using clause
// PostgreSQL with array operators
sql := `SELECT * FROM users WHERE tags @> ARRAY['admin']`
// MySQL with backticks
sql := "SELECT `user_id`, `name` FROM `users`"
// SQL Server with brackets
sql := "SELECT [user_id], [name] FROM [users]"// Japanese
sql := `SELECT "ๅๅ", "ๅนด้ฝข" FROM "ใฆใผใถใผ"`
// Russian
sql := `SELECT "ะธะผั", "ะฒะพะทัะฐัั" FROM "ะฟะพะปัะทะพะฒะฐัะตะปะธ"`
// Arabic
sql := `SELECT "ุงูุงุณู
", "ุงูุนู
ุฑ" FROM "ุงูู
ุณุชุฎุฏู
ูู"`
// Emoji support
sql := `SELECT * FROM users WHERE status = '๐'`func ProcessConcurrently(queries []string) {
var wg sync.WaitGroup
for _, sql := range queries {
wg.Add(1)
go func(query string) {
defer wg.Done()
// Each goroutine gets its own tokenizer
tkz := tokenizer.GetTokenizer()
defer tokenizer.PutTokenizer(tkz)
tokens, _ := tkz.Tokenize([]byte(query))
// Process tokens...
}(sql)
}
wg.Wait()
}| Metric | Previous | v1.0.0 | Improvement |
|---|---|---|---|
| Sustained Throughput | 2.2M ops/s | 946K+ ops/s | Production Grade โ |
| Peak Throughput | 2.2M ops/s | 1.25M+ ops/s | Enhanced โ |
| Token Processing | 8M tokens/s | 8M+ tokens/s | Maintained โ |
| Simple Query Latency | 200ns | <280ns | Optimized โ |
| Complex Query Latency | N/A | <1ฮผs (CTE/Set Ops) | New Capability โ |
| Memory Usage | Baseline | 60-80% reduction | -70% โ |
| SQL-92 Compliance | 40% | ~70% | +75% โ |
BenchmarkParserSustainedLoad-16 946,583 1,057 ns/op 1,847 B/op 23 allocs/op
BenchmarkParserThroughput-16 1,252,833 798 ns/op 1,452 B/op 18 allocs/op
BenchmarkParserSimpleSelect-16 3,571,428 279 ns/op 536 B/op 9 allocs/op
BenchmarkParserComplexSelect-16 985,221 1,014 ns/op 2,184 B/op 31 allocs/op
BenchmarkCTE/SimpleCTE-16 524,933 1,891 ns/op 3,847 B/op 52 allocs/op
BenchmarkCTE/RecursiveCTE-16 387,654 2,735 ns/op 5,293 B/op 71 allocs/op
BenchmarkSetOperations/UNION-16 445,782 2,234 ns/op 4,156 B/op 58 allocs/op
BenchmarkTokensPerSecond-16 815,439 1,378 ns/op 8,847,625 tokens/sec
| Metric | Value | Details |
|---|---|---|
| Sustained Throughput | 946K+ ops/sec | 30s load testing |
| Peak Throughput | 1.25M+ ops/sec | Concurrent goroutines |
| Token Rate | 8M+ tokens/sec | Sustained processing |
| Simple Query Latency | <280ns | Basic SELECT (p50) |
| Complex Query Latency | <1ฮผs | CTEs/Set Operations |
| Memory | 1.8KB/query | Complex SQL with pooling |
| Scaling | Linear to 128+ | Perfect concurrency |
| Pool Efficiency | 95%+ hit rate | Effective reuse |
See PERFORMANCE_REPORT.md for detailed analysis.
# Run all tests with race detection
go test -race ./...
# Run benchmarks
go test -bench=. -benchmem ./...
# Generate coverage report
go test -coverprofile=coverage.out ./...
go tool cover -html=coverage.out
# Run specific test suites
go test -v ./pkg/sql/tokenizer/
go test -v ./pkg/sql/parser/GoSQLX/
โโโ pkg/
โ โโโ models/ # Core data structures
โ โ โโโ token.go # Token definitions
โ โ โโโ location.go # Position tracking
โ โโโ sql/
โ โโโ tokenizer/ # Lexical analysis
โ โ โโโ tokenizer.go
โ โ โโโ pool.go
โ โโโ parser/ # Syntax analysis
โ โ โโโ parser.go
โ โ โโโ expressions.go
โ โโโ ast/ # Abstract syntax tree
โ โ โโโ nodes.go
โ โ โโโ statements.go
โ โโโ keywords/ # SQL keywords
โโโ examples/ # Usage examples
โ โโโ cmd/
โ โโโ example.go
โ โโโ example_test.go
โโโ docs/ # Documentation
โ โโโ API_REFERENCE.md
โ โโโ USAGE_GUIDE.md
โ โโโ ARCHITECTURE.md
โ โโโ TROUBLESHOOTING.md
โโโ tools/ # Development tools
- Go 1.19+
- Make (optional, for Makefile targets)
- golint, staticcheck (for code quality)
# Build the project
make build
# Run quality checks
make quality
# Run all tests
make test
# Clean build artifacts
make clean# Format code
go fmt ./...
# Vet code
go vet ./...
# Run linter
golint ./...
# Static analysis
staticcheck ./...We welcome contributions! Please see CONTRIBUTING.md for guidelines.
- Fork the repository
- Create a feature branch (
git checkout -b feature/amazing-feature) - Commit your changes (
git commit -m 'Add amazing feature') - Push to the branch (
git push origin feature/amazing-feature) - Open a Pull Request
- Write tests for new features
- Ensure all tests pass with race detection
- Follow Go idioms and best practices
- Update documentation for API changes
- Add benchmarks for performance-critical code
This project is licensed under the MIT License - see the LICENSE file for details.
- โ Complete JOIN support (INNER/LEFT/RIGHT/FULL OUTER/CROSS/NATURAL)
- โ Proper join tree logic with left-associative relationships
- โ USING clause parsing (single-column, multi-column planned for Phase 2)
- โ Enhanced error handling with contextual JOIN error messages
- โ Comprehensive test coverage (15+ JOIN scenarios including error cases)
- ๐๏ธ CTE foundation laid (AST structures, tokens, parser integration points)
- โ Common Table Expressions (CTEs) with RECURSIVE support
- โ Set operations (UNION/EXCEPT/INTERSECT with ALL modifier)
- โ Left-associative set operation parsing
- โ CTE column specifications and multiple CTE definitions
- โ Integration of CTEs with set operations
- โ Enhanced error handling with contextual messages
- โ ~70% SQL-92 compliance achieved
- ๐ PostgreSQL arrays, JSONB, custom types
- ๐ MySQL-specific syntax and functions
- ๐ SQL Server T-SQL extensions
- ๐ Multi-dialect parser with auto-detection
- ๐ Query optimization suggestions
- ๐ Security vulnerability detection
- ๐ Performance analysis and hints
- ๐ Schema validation
๐ Full Architectural Review & Roadmap
| Channel | Purpose | Response Time |
|---|---|---|
| ๐ Bug Reports | Report issues | Community-driven |
| ๐ก Feature Requests | Suggest improvements | Community-driven |
| ๐ฌ Discussions | Q&A, ideas, showcase | Community-driven |
| ๐ Security | Report vulnerabilities | Best effort |
We love your input! We want to make contributing as easy and transparent as possible.
- ๐ด Fork the repo
- ๐จ Make your changes
- โ
Ensure tests pass (
go test -race ./...) - ๐ Update documentation
- ๐ Submit a PR
| Industry | Use Case | Benefits |
|---|---|---|
| ๐ฆ FinTech | SQL validation & auditing | Fast validation, compliance tracking |
| ๐ Analytics | Query parsing & optimization | Real-time analysis, performance insights |
| ๐ก๏ธ Security | SQL injection detection | Pattern matching, threat prevention |
| ๐๏ธ DevTools | IDE integration & linting | Syntax highlighting, auto-completion |
| ๐ Education | SQL learning platforms | Interactive parsing, error explanation |
| ๐ Migration | Cross-database migration | Dialect conversion, compatibility check |
Using GoSQLX in production? Let us know!
graph LR
A[SQL Input] -->|946K+ ops/sec| B[Tokenizer]
B -->|8M+ tokens/sec| C[Parser]
C -->|Zero-copy| D[AST]
D -->|60-80% less memory| E[Output]
| Version | Status | Release Date | Features |
|---|---|---|---|
| v0.9.0 | โ Released | 2024-01-15 | Initial release |
| v1.0.0 | โ Released | 2024-12-01 | Production ready, +47% performance |
| v1.1.0 | โ Released | 2025-01-03 | Complete JOIN support, error handling |
| v1.2.0 | โ Released | 2025-08-15 | CTEs, set operations, ~70% SQL-92 compliance |
| v1.4.0 | ๐ Current | 2025-09-07 | Production CLI, high-performance commands, memory leak fixes |
| v1.3.0 | โ Previous | 2025-09-04 | Window functions, ~80-85% SQL-99 compliance |
| v2.0.0 | ๐ฎ Future | Q4 2025 | Dialect specialization, advanced features |
If GoSQLX helps your project, please consider:
- โญ Star this repository
- ๐ฆ Tweet about GoSQLX
- ๐ Write a blog post
- ๐ฅ Create a tutorial
- ๐ Report bugs
- ๐ก Suggest features
- ๐ง Submit PRs
This project is licensed under the MIT License - see the LICENSE file for details.
Copyright ยฉ 2024 GoSQLX. All rights reserved.