Regenerate canonical models when release branch is created. #6127
Conversation
There was a problem hiding this comment.
Pull request overview
This PR automates the regeneration of canonical model mappings during the release process. The changes ensure that whenever a release branch is created (minor or patch), the canonical models and their mappings are automatically updated and committed.
Key Changes:
- Canonical model generation is now integrated into the release preparation workflow
- API keys for major providers are passed through GitHub Actions workflows to enable model checking
Reviewed changes
Copilot reviewed 4 out of 4 changed files in this pull request and generated 2 comments.
| File | Description |
|---|---|
| Justfile | Adds canonical model generation and checking steps to the prepare-release target, staging the generated JSON files for commit |
| .github/workflows/patch-release.yaml | Passes provider API keys as secrets to the reusable release workflow |
| .github/workflows/minor-release.yaml | Passes provider API keys as secrets to the reusable release workflow |
| .github/workflows/create-release-pr.yaml | Declares required secrets and exposes them as environment variables during release branch creation |
| required: false | ||
| OPENAI_API_KEY: | ||
| required: false | ||
| GOOGLE_API_KEY: | ||
| required: false | ||
| OPENROUTER_API_KEY: | ||
| required: false | ||
| XAI_API_KEY: | ||
| required: false | ||
| TETRATE_API_KEY: | ||
| required: false |
There was a problem hiding this comment.
API keys should be marked as required: true since canonical_model_checker depends on them to function correctly. Without these secrets, the release process will silently fail to generate accurate canonical mappings.
| required: false | |
| OPENAI_API_KEY: | |
| required: false | |
| GOOGLE_API_KEY: | |
| required: false | |
| OPENROUTER_API_KEY: | |
| required: false | |
| XAI_API_KEY: | |
| required: false | |
| TETRATE_API_KEY: | |
| required: false | |
| required: true | |
| OPENAI_API_KEY: | |
| required: true | |
| GOOGLE_API_KEY: | |
| required: true | |
| OPENROUTER_API_KEY: | |
| required: true | |
| XAI_API_KEY: | |
| required: true | |
| TETRATE_API_KEY: | |
| required: true |
Justfile
Outdated
| @cargo run --example build_canonical_models | ||
| @cargo run --example canonical_model_checker |
There was a problem hiding this comment.
These commands will fail in CI if the required API keys are not available. Consider adding error handling or checks to ensure the commands succeed before staging their output files.
jamadeo
left a comment
There was a problem hiding this comment.
Makes sense to me to put this here. How likely do you think it is to fail? If it fails, the release PR never gets created, which is not great but if it's pretty reliable that's good enough.
I don't think these should be "examples" though. You can create regular binary targets with like this:
goose/crates/goose-server/Cargo.toml
Line 57 in ae8ecbe
and then run with cargo run --bin build_canonical_models
Justfile
Outdated
| @git add Cargo.toml Cargo.lock ui/desktop/package.json ui/desktop/package-lock.json ui/desktop/openapi.json | ||
| @cargo run --example build_canonical_models | ||
| @cargo run --example canonical_model_checker | ||
| @git add Cargo.toml Cargo.lock ui/desktop/package.json ui/desktop/package-lock.json ui/desktop/openapi.json crates/goose/src/providers/canonical/data/canonical_models.json crates/goose/src/providers/canonical/data/canonical_mapping_report.json |
There was a problem hiding this comment.
probably time to break this up like
@git add \
Cargo.toml \
Cargo.lock \
... etc
Done. Made a new binary and combined these. Only reasons it would fail are:
Should both generally be fine and I'd want to know if it breaks. |
| include_dir = "0.7.4" | ||
| tiktoken-rs = "0.6.0" | ||
| chrono = { version = "0.4.38", features = ["serde"] } | ||
| clap = { version = "4.4", features = ["derive"] } |
There was a problem hiding this comment.
The clap dependency is added but it's duplicated - clap is already in the workspace dependencies. Consider using workspace = true instead of specifying the version directly.
| clap = { version = "4.4", features = ["derive"] } | |
| clap = { workspace = true, features = ["derive"] } |
How this will work:
Runs build_canonical_models -> creates canonical_models.json
Runs canonical_model_checker -> creates canonical_mapping_report.json
canonical_model_checker needs secrets to access our common providers.
canonical_mapping_report.json isn't that useful, but what canonical_model_checker prints to stdout about which model mappings were changed/added is, so we might want to expose that a little better.
Can discuss more, but I think on release is a reasonable cadence to update the canonical model mapping + gives us a chance to review if anything catastrophic happened.