Commits on May 23, 2023

1. Update MLOps Stacks to Use Databricks Asset Bundles (#77) …

   * Remove terraform, bootstrap script, terraform related github actions
   
   Signed-off-by: Mingyu Li <[email protected]>
   
   * Remove project_name_alphanumeric. Add project_name_alphanumeric_underscore and update the project folder name to project_name_alphanumeric_underscore
   
   Signed-off-by: Mingyu Li <[email protected]>
   
   * Move the requriements file into the project folder and update mlflow version to resolve a recipe issue
   
   Signed-off-by: Mingyu Li <[email protected]>
   
   * Update the model name and experiment under different environments. The experiment folder will be removed until bundles supports managing it.
   
   Signed-off-by: Mingyu Li <[email protected]>
   
   * Add project name as prefix of github and ADO workflows
   
   Signed-off-by: Mingyu Li <[email protected]>
   
   * Update notebooks to enter notebook folder and current folder. Update format.
   
   Signed-off-by: Mingyu Li <[email protected]>
   
   * Directly do model deployment in the workflow run and avoid the callback to github. Add model name parameter to BatchInference notebook to avoid reading it from terraform output file
   
   Signed-off-by: Mingyu Li <[email protected]>
   
   * add bundle file and bundle resources. Some small fix
   
   Signed-off-by: Mingyu Li <[email protected]>
   
   * Prepare the batch inference with FS input table in CUJ prod workspace, update bundle to use the new table and fix fs batch inference
   
   Signed-off-by: Mingyu Li <[email protected]>
   
   * Fix tests
   
   Signed-off-by: Mingyu Li <[email protected]>
   
   * Support shared cluster between tasks in one job. Fix a path. Remove explicit token.
   
   Signed-off-by: Mingyu Li <[email protected]>
   
   * Add CI CD support for mlops using bundle and update notebook path according latest bundle update
   
   Signed-off-by: Mingyu Li <[email protected]>
   
   * Fix CI PR comment
   
   Signed-off-by: Mingyu Li <[email protected]>
   
   * Fix integration tests
   
   Signed-off-by: Mingyu Li <[email protected]>
   
   * Add git source info to the notebook parameters
   
   Signed-off-by: Mingyu Li <[email protected]>
   
   * Remove Bricks CLI repo token
   
   Signed-off-by: Mingyu Li <[email protected]>
   
   * Support authentication for an Azure service principal in mlops-stack bundle
   
   Signed-off-by: Mingyu Li <[email protected]>
   
   * Fix
   
   Signed-off-by: Mingyu Li <[email protected]>
   
   * Remove experiment parent dir (#9)
   
   * Remove experiment parent dir
   
   Signed-off-by: Mingyu Li <[email protected]>
   
   * Update experiment to be in the current user's directory
   
   Signed-off-by: Mingyu Li <[email protected]>
   
   * Fix test
   
   Signed-off-by: Mingyu Li <[email protected]>
   
   ---------
   
   Signed-off-by: Mingyu Li <[email protected]>
   
   * Add new bricks bundle ML resource configuration to docs and remove terraform (#5)
   
   * update docs
   
   Signed-off-by: Mingyu Li <[email protected]>
   
   * Add bundle and remove terraform from gitignore
   
   Signed-off-by: Mingyu Li <[email protected]>
   
   * Remove terraform from MLOps-stacks except ADO CICD. Fix tests and update more docs
   
   Signed-off-by: Mingyu Li <[email protected]>
   
   * Add file names to gitignore
   
   Signed-off-by: Mingyu Li <[email protected]>
   
   * More doc changes. Move the CICD set up section from ML resource README to mlops-setup doc
   
   Signed-off-by: Mingyu Li <[email protected]>
   
   * Address comments
   
   Signed-off-by: Mingyu Li <[email protected]>
   
   * Do not use hard coded aws doc for bricks CLI bundles since azure doc is also available
   
   Signed-off-by: Mingyu Li <[email protected]>
   
   ---------
   
   Signed-off-by: Mingyu Li <[email protected]>
   
   * Use variables for bundles
   
   * Add example and fix tests
   
   * Fix link
   
   * Address bug bash issues and polish stacks for private preview
   
   * Add bundle run example to docs
   
   * Fix nits
   
   * Resolve merge conflicts and add TODOs
   
   * Fix broken link
   
   * Rename bricks CLI to databricks
   
   * Add doc for generating example batch inference input (#14)
   
   Signed-off-by: Mingyu Li <[email protected]>
   
   * Fix Bug Bash Issues
   
   * Fixes
   
   * Azure DevOps integration using Databricks Asset Bundles (#12)
   
   * vscode gitignore
   
   * vscode gitignore
   
   * typo in test configs
   
   * update reqs path
   
   * update tests-ci to use bundles
   
   * add script to install bricks cli
   
   * update main -> {{cookiecutter.default_branch}}
   
   * add bundle cicd
   
   * update docs
   
   * typo
   
   * update docs to have ado
   
   * update pipelines to use cookiecutter URLs
   
   * doc updates
   
   * remove print statement
   
   * update setup script bricks -> databricks rename
   
   * update to use databricks cli and lowercase staging
   
   * lowercase standardize, databricks CLI
   
   * update conditions
   
   * update name
   
   * parameterize databricks cli version
   
   * bricks -> databricks
   
   * lowercase displayName
   
   * update pipelines
   
   * minor doc updates
   
   * revert condition
   
   * Update {{cookiecutter.root_dir__update_if_you_intend_to_use_monorepo}}/.azure/devops-pipelines/{{cookiecutter.project_name}}-tests-ci.yml
   
   Co-authored-by: Vladimir Kolovski <[email protected]>
   
   * Update {{cookiecutter.root_dir__update_if_you_intend_to_use_monorepo}}/{{cookiecutter.project_name_alphanumeric_underscore}}/databricks-resource/README.md
   
   Co-authored-by: Vladimir Kolovski <[email protected]>
   
   * Update tests/example-project-configs/aws/aws-github.yaml
   
   Co-authored-by: Pieter Noordhuis <[email protected]>
   
   * Update {{cookiecutter.root_dir__update_if_you_intend_to_use_monorepo}}/.azure/devops-pipelines/{{cookiecutter.project_name}}-bundle-cicd.yml
   
   Co-authored-by: Vladimir Kolovski <[email protected]>
   
   * Update {{cookiecutter.root_dir__update_if_you_intend_to_use_monorepo}}/.azure/devops-pipelines/{{cookiecutter.project_name}}-bundle-cicd.yml
   
   Co-authored-by: Vladimir Kolovski <[email protected]>
   
   * update url
   
   * update docs to use SPs over PATs
   
   * use SP credentials
   
   * use SP credentials
   
   * update path
   
   * standardize on Azure Pipelines
   
   * minor doc updates
   
   * typos
   
   * Fix links (#16)
   
   Signed-off-by: Mingyu Li <[email protected]>
   
   ---------
   
   Signed-off-by: Mingyu Li <[email protected]>
   Co-authored-by: Vladimir Kolovski <[email protected]>
   Co-authored-by: Pieter Noordhuis <[email protected]>
   Co-authored-by: Mingyu Li <[email protected]>
   
   * some docs fixes.
   
   * more changes
   
   * fix broken links.
   
   * fix links
   
   * add cleanup in hook
   
   * remove mlflow recipes from another place.
   
   * make changes
   
   * small fix
   
   * doc changes
   
   ---------
   
   Signed-off-by: Mingyu Li <[email protected]>
   Co-authored-by: Mingyu Li <[email protected]>
   Co-authored-by: Niall Turbitt <[email protected]>
   Co-authored-by: Vladimir Kolovski <[email protected]>
   Co-authored-by: Pieter Noordhuis <[email protected]>
   Co-authored-by: Jerry Liang <[email protected]>
   Co-authored-by: Jerry Liang <[email protected]>

   

   

   7 people authored May 23, 2023
   Configuration menu

   • View commit details
   • Copy full SHA for 49d5615
   • Browse repository at this point

   Copy the full SHA

   49d5615 View commit details

   Browse the repository at this point in the history