Tags: databricks/mlops-stacks
Tags
Update MLOps Stacks to Use Databricks Asset Bundles (#77) * Remove terraform, bootstrap script, terraform related github actions Signed-off-by: Mingyu Li <[email protected]> * Remove project_name_alphanumeric. Add project_name_alphanumeric_underscore and update the project folder name to project_name_alphanumeric_underscore Signed-off-by: Mingyu Li <[email protected]> * Move the requriements file into the project folder and update mlflow version to resolve a recipe issue Signed-off-by: Mingyu Li <[email protected]> * Update the model name and experiment under different environments. The experiment folder will be removed until bundles supports managing it. Signed-off-by: Mingyu Li <[email protected]> * Add project name as prefix of github and ADO workflows Signed-off-by: Mingyu Li <[email protected]> * Update notebooks to enter notebook folder and current folder. Update format. Signed-off-by: Mingyu Li <[email protected]> * Directly do model deployment in the workflow run and avoid the callback to github. Add model name parameter to BatchInference notebook to avoid reading it from terraform output file Signed-off-by: Mingyu Li <[email protected]> * add bundle file and bundle resources. Some small fix Signed-off-by: Mingyu Li <[email protected]> * Prepare the batch inference with FS input table in CUJ prod workspace, update bundle to use the new table and fix fs batch inference Signed-off-by: Mingyu Li <[email protected]> * Fix tests Signed-off-by: Mingyu Li <[email protected]> * Support shared cluster between tasks in one job. Fix a path. Remove explicit token. Signed-off-by: Mingyu Li <[email protected]> * Add CI CD support for mlops using bundle and update notebook path according latest bundle update Signed-off-by: Mingyu Li <[email protected]> * Fix CI PR comment Signed-off-by: Mingyu Li <[email protected]> * Fix integration tests Signed-off-by: Mingyu Li <[email protected]> * Add git source info to the notebook parameters Signed-off-by: Mingyu Li <[email protected]> * Remove Bricks CLI repo token Signed-off-by: Mingyu Li <[email protected]> * Support authentication for an Azure service principal in mlops-stack bundle Signed-off-by: Mingyu Li <[email protected]> * Fix Signed-off-by: Mingyu Li <[email protected]> * Remove experiment parent dir (#9) * Remove experiment parent dir Signed-off-by: Mingyu Li <[email protected]> * Update experiment to be in the current user's directory Signed-off-by: Mingyu Li <[email protected]> * Fix test Signed-off-by: Mingyu Li <[email protected]> --------- Signed-off-by: Mingyu Li <[email protected]> * Add new bricks bundle ML resource configuration to docs and remove terraform (#5) * update docs Signed-off-by: Mingyu Li <[email protected]> * Add bundle and remove terraform from gitignore Signed-off-by: Mingyu Li <[email protected]> * Remove terraform from MLOps-stacks except ADO CICD. Fix tests and update more docs Signed-off-by: Mingyu Li <[email protected]> * Add file names to gitignore Signed-off-by: Mingyu Li <[email protected]> * More doc changes. Move the CICD set up section from ML resource README to mlops-setup doc Signed-off-by: Mingyu Li <[email protected]> * Address comments Signed-off-by: Mingyu Li <[email protected]> * Do not use hard coded aws doc for bricks CLI bundles since azure doc is also available Signed-off-by: Mingyu Li <[email protected]> --------- Signed-off-by: Mingyu Li <[email protected]> * Use variables for bundles * Add example and fix tests * Fix link * Address bug bash issues and polish stacks for private preview * Add bundle run example to docs * Fix nits * Resolve merge conflicts and add TODOs * Fix broken link * Rename bricks CLI to databricks * Add doc for generating example batch inference input (#14) Signed-off-by: Mingyu Li <[email protected]> * Fix Bug Bash Issues * Fixes * Azure DevOps integration using Databricks Asset Bundles (#12) * vscode gitignore * vscode gitignore * typo in test configs * update reqs path * update tests-ci to use bundles * add script to install bricks cli * update main -> {{cookiecutter.default_branch}} * add bundle cicd * update docs * typo * update docs to have ado * update pipelines to use cookiecutter URLs * doc updates * remove print statement * update setup script bricks -> databricks rename * update to use databricks cli and lowercase staging * lowercase standardize, databricks CLI * update conditions * update name * parameterize databricks cli version * bricks -> databricks * lowercase displayName * update pipelines * minor doc updates * revert condition * Update {{cookiecutter.root_dir__update_if_you_intend_to_use_monorepo}}/.azure/devops-pipelines/{{cookiecutter.project_name}}-tests-ci.yml Co-authored-by: Vladimir Kolovski <[email protected]> * Update {{cookiecutter.root_dir__update_if_you_intend_to_use_monorepo}}/{{cookiecutter.project_name_alphanumeric_underscore}}/databricks-resource/README.md Co-authored-by: Vladimir Kolovski <[email protected]> * Update tests/example-project-configs/aws/aws-github.yaml Co-authored-by: Pieter Noordhuis <[email protected]> * Update {{cookiecutter.root_dir__update_if_you_intend_to_use_monorepo}}/.azure/devops-pipelines/{{cookiecutter.project_name}}-bundle-cicd.yml Co-authored-by: Vladimir Kolovski <[email protected]> * Update {{cookiecutter.root_dir__update_if_you_intend_to_use_monorepo}}/.azure/devops-pipelines/{{cookiecutter.project_name}}-bundle-cicd.yml Co-authored-by: Vladimir Kolovski <[email protected]> * update url * update docs to use SPs over PATs * use SP credentials * use SP credentials * update path * standardize on Azure Pipelines * minor doc updates * typos * Fix links (#16) Signed-off-by: Mingyu Li <[email protected]> --------- Signed-off-by: Mingyu Li <[email protected]> Co-authored-by: Vladimir Kolovski <[email protected]> Co-authored-by: Pieter Noordhuis <[email protected]> Co-authored-by: Mingyu Li <[email protected]> * some docs fixes. * more changes * fix broken links. * fix links * add cleanup in hook * remove mlflow recipes from another place. * make changes * small fix * doc changes --------- Signed-off-by: Mingyu Li <[email protected]> Co-authored-by: Mingyu Li <[email protected]> Co-authored-by: Niall Turbitt <[email protected]> Co-authored-by: Vladimir Kolovski <[email protected]> Co-authored-by: Pieter Noordhuis <[email protected]> Co-authored-by: Jerry Liang <[email protected]> Co-authored-by: Jerry Liang <[email protected]>
Unified project run modes (#72) * Unified project run modes Signed-off-by: Mingyu Li <[email protected]> * Remove model's depedency on batch_inference_job Signed-off-by: Mingyu Li <[email protected]> * Revert change of adding model name to BatchInference parameters Signed-off-by: Mingyu Li <[email protected]> * Update model name and experiment name to use 'dev' prefix when users manually run notebooks Signed-off-by: Mingyu Li <[email protected]> --------- Signed-off-by: Mingyu Li <[email protected]>