| Donate | ko-fi

Hot Fixture Tool is compound of two pieces:

• server (hfitd) installed on server with access hot db and fs data
• client (hfit) is a CLI interface to the server

graph TB
    subgraph "Production Environment"
        DB1[(Database 1)]
        DB2[(Database 2)]
        FS1[File System 1]
        FS2[File System 2]
    end
    
    subgraph "Hot Fixture Tool Server"
        HFITD[hfitd Daemon]
        API[REST API]
        AUTH[Authentication]
        REDIS[(Redis Cache)]
    end
    
    subgraph "Development Environment"
        CLI[hfit CLI]
        TESTS[Integration Tests]
        LOCAL[(Local Data and DB)]
    end
    
    DB1 --> HFITD
    DB2 --> HFITD
    FS1 --> HFITD
    FS2 --> HFITD
    
    HFITD --> API
    API --> AUTH
    HFITD --> REDIS
    
    CLI --> API
    CLI --> LOCAL
    LOCAL --> TESTS

Redis cache is used only for metadata (package export definition) and user data (public key for access challenge)

Developer workflow:

• a developer need to work on code, code that read and modify data
• developer in general has no access to data
• developer need to adopt strategy to test data read and data write, by mocking library, and mocking requests
• using mocks, the service code is not exactly the code the developer tested.
• real integration tests are often done just by CI tools, and often are hard to define.

Thanks to hfit and hfitd, developer can:

• define one or more hfit/pkg-tmpl-xxx.yaml
• automate the download of real server resource (testing server or production server)
• automate the import into local machine
• implements effective integration tests
• use integration tests to fix bug on hot real data
• keep the .yaml definitions files in the shared repo, since those does not contains real data

Thanks to hfitd, administrator can warrant that only a restrict number of developer has access to data for reading. Also hfit command can be integrated into the CI pipeline to automate the import, and the integration tests running.

Development of DataBase centric service involve unit tests and integration tests definition. Unit tests are used to keep the code clean and to lower cyclomatic and other code metrics. Integration tests are used to check the effective correctnes of the service.

This service and tool targets the integration test definition.

By using hfit, a developer can:

• define resources as DDL, db rows, and file to be ready to execute integration test locally
• run the integration tests against a real set of data
• retrieve hot data for hot fixing of bug a bug
• automate the retrievement of hot data to execute all integration tests and avoid regressions

Most of the time, running a dbms locally is not resource consuming as it is the repeated cycle of "blinded development".

The most common case is the deal with legacy application that need to be ported, refactored, adapted.

An application become "legacy" after 5 or 6 year of development, so this tool is mostly targeted at successful projects with business logic, that one want to keep alive and current, want to update to newer execution engine, compiler or interpreter, want to change the protocol exposed, etc.

• create a folder on your service repo: mkdir hfit-data; cd hfit-data
• keep your hot-export-package.yaml definition in files stored in the repo.
• run hfit to download data based on hot-export-package.yaml definition, into a target folder
• for each folder:
  • run import tool to read data from downloaded resource.
  • execute integration tests
  • cleanup (if needed)

A progressive workflow might fit better the integration test job. For this use this policy:

• define a base-export-package.yaml
• define as many hot-export-package.yaml as there are hot test cases.

Example of folder content:

└── hfit-data
    ├── .gitignore
    ├── METADATA
    │   ├── package.json
    │   ├── replacements.json
    │   └── timestamps.json
    ├── base-export
    │   ├── db1.table1.create.sql
    │   ├── db1.table2.create.sql
    │   └── db1.table3.create.sql
    ├── base-export.yaml
    ├── use-case123
    │   ├── db1.table3.data.sql
    │   └── file-import-123.txt
    └── use-case123.yaml

Importing data

To use the hfit CLI to import data, use the local-i sub-command

hfit local-i create my-docker.yaml extract-from base-export.yaml use-case123.yaml

Open a text editor and fill the required data for (write) connection to your local dbms and filesystem provider.

Then run

hfit local-i run-on my-docker.yaml base-export.yaml --no-download

With --no-download option it imports data and just use the existing data.

Use a .gitignore in hfit/ folder like this:

*
!.gitignore
!*.yaml

to avoid the checkout of hot data.

As helper there is a convenient command:

hfit prepare

This creates hfit-data/ folder and populates with .gitignore :

└── hfit-data
    └── .gitignore

Service API:

• list DBS
• list tables
• export table definition
• export table data + where filter (with streaming NDJSON support)
• list accessible folder
• list files with filter (stream LDJSON)

Database table rows now support streaming NDJSON format for optimal performance with large datasets:

Traditional JSON Response:

curl -H 'Authorization: Bearer $JWT_TOKEN' \
     'http://localhost:8080/db/mysql/mydb/table/users/rows'

Streaming NDJSON Response (Optimized):

curl -H 'Authorization: Bearer $JWT_TOKEN' \
     -H 'Accept: application/x-json-stream' \
     'http://localhost:8080/db/mysql/mydb/table/users/rows'

Advanced Filtering with SQL injection protection:

# WHERE clause filtering
curl -H 'Accept: application/x-json-stream' \
     'http://localhost:8080/db/mysql/mydb/table/users/rows?filterpart=WHERE age > 25'

# Complex queries with ORDER BY and LIMIT
curl -H 'Accept: application/x-json-stream' \
     'http://localhost:8080/db/postgres/mydb/table/orders/rows?filterpart=WHERE status = "active" ORDER BY created_at DESC LIMIT 100'

Key Benefits:

• O(1) Memory Usage: Constant memory regardless of result size
• Real-time Streaming: First rows appear immediately
• SQL Injection Protection: Safe filterpart parameter validation
• Multi-DBMS Support: Works with MySQL, PostgreSQL, and more
• Enterprise Scalability: Handles millions of rows efficiently

Client:

• interactively run all API provided by service
• define JSON for export package with parameters
• modify export JSON
• execute export package.

Features:

• service ACL per user
• user auth list
• user authentication by key-pair, challenge, and jwt
• storage by redis
• admin user- public key
• simple admin interface based on vanillaJs

The command export-package accepts a .yaml file as a template definition for exporting hot data. The format is:

hfitVersion: 1
templateName: usecase_data
projectName: project_name # typically the repository or project name
packageName: basedata_$1
prepare:
  - setVar: dataid
    from: input
    source: $1
  - setVar: usrId
    from: hot-data
    hdata:
        type: dbquery
        dbms: dbms_mysql1
        query: "SELECT usrId FROM dbname.datatable WHERE dataid=${dataid} ORDER BY utime LIMIT 1"
  - setVar: fBaseName
    from: hot-data
    hdata:
        type: volume
        volume: vol1
        glob: "*_{dataid}_{usrId}_*.txt"
        # take the first in mtime desc order:
        sort: "mtime|desc"
        # extract the first number of filename as fBaseName value
        regex_replace: "/([0-9]+).*/$1/"
exports:
  dbcreate.sql:
    type: dbcreate
    data:
      dbms: dbms_mysql1
      tablelist:
        - dbname1
        - dbname2
  tablegroup1.create.sql:
    type: table-create
    data:
      dbms: dbms_mysql1
      tablelist:
        - dbname1.table1
        - dbname2.table2
        - dbname1.tablex
      option: <dropcreate|ifnotexists>
  tabledata1.data.sql:
    type: table-data
    data:
      dbms: dbms_mysql1
      table: dbname1.table1
      filter: WHERE key<34 AND key>12
  target-filename.txt:
    type: file
    data:
      volume: datavol1
      path: relative/path/to/file

NOTES:

• order of retrievement is not important
• order of prepare is relevant

Once you defined this file, the common usage is to to store it in hfit-data/ folder of the repo. For this reason:

hfit repo-prepare

Does create an hfit-data/ folder on the root of your git repo and write there some package template example.

hfit CLI should be used to create and populate this .yaml file.

Create the .yaml file:

./hfit pkg create base-data-package.yaml basedata

Execute package download and unpack into target folder (the target folder is named as the package name)

./hfit pkg downpack base-data-package.yaml

Manipulate by adding resource (this check if resource exists):

./hfit pkg add base-data-package.yaml

Removing a resource:

./hfit pkg rm base-data-package.yaml

Edit (option/filter/tablename)

First release has no ACL control, all users read access to all resources

The fastest way to get started is using Docker:

# Start the complete stack (PostgreSQL + Redis + hfitd)
make demo

# Or manually:
docker-compose up -d
make adduser [email protected] KEY=your_public_key.pem

Visit http://localhost:8080 and see DOCKER.md for complete deployment guide.

This project is source-available - you can inspect the code, contribute improvements, and use it for personal/educational purposes. I'm actively seeking:

• Contributors to help improve the codebase
• Feedback from users who find this tool useful
• Sponsorship or commercial partnerships
• Real-world use cases and feature requests

If you find this project valuable or want to use it commercially, please reach out! I'm open to discussing licensing, sponsorship, or collaboration opportunities.

This project is licensed under a custom source-available license. You may:

• Inspect and study the source code
• Contribute improvements and bug fixes
• Use for personal, non-commercial purposes

You may NOT:

• Use commercially without permission
• Distribute without permission

See LICENSE for full details. For commercial use, please contact me to discuss licensing options.

Contributions are welcome! Please feel free to:

• Report bugs and issues
• Suggest new features
• Submit pull requests
• Provide feedback and suggestions

By contributing, you help make this tool better while allowing me to explore sustainable development models.