Add automl regression and classification with model evaluation #911

soheilazangeneh · 2022-08-29T15:21:48Z

This is the first draft for AutoML tabular regression model with the focus of model evolution.

If you are opening a PR for Community Notebooks under the notebooks/community folder:

This notebook has been added to the CODEOWNERS file under the Community Notebooks section, pointing to the author or the author's team.
Passes all the required formatting and linting checks. You can locally test with these instructions.

If you are opening a PR for Community Content under the community-content folder:

Make sure your main Content Directory Name is descriptive, informative, and includes some of the key products and attributes of your content, so that it is differentiable from other content
The main content directory has been added to the CODEOWNERS file under the Community Content section, pointing to the author or the author's team.
Passes all the required formatting and linting checks. You can locally test with these instructions.

review-notebook-app · 2022-08-29T15:21:53Z

Check out this pull request on

See visual diffs & provide feedback on Jupyter Notebooks.

Powered by ReviewNB

andrewferlitsch · 2022-08-29T15:58:59Z

notebooks/community/model_evaluation/automl_tabular_regression_model evaluation.ipynb

@@ -0,0 +1,1273 @@
+{


Need H1 title in this cell. Should be above the links.

Reply via ReviewNB

andrewferlitsch · 2022-08-29T15:58:59Z

notebooks/community/model_evaluation/automl_tabular_regression_model evaluation.ipynb

@@ -0,0 +1,1273 @@
+{


regression model evaluation componenet -> regression model evaluation pipeline component

Reply via ReviewNB

andrewferlitsch · 2022-08-29T15:58:59Z

notebooks/community/model_evaluation/automl_tabular_regression_model evaluation.ipynb

@@ -0,0 +1,1273 @@
+{


Replace : with . in first sentence

regression evluation component -> model evaluation pipeline component

you say 'pre-trained', but you train the model in the notebook

Add to services, Big Query

Reply via ReviewNB

andrewferlitsch · 2022-08-29T15:58:59Z

notebooks/community/model_evaluation/automl_tabular_regression_model evaluation.ipynb

@@ -0,0 +1,1273 @@
+{


Combine this with the above cell

Reply via ReviewNB

andrewferlitsch · 2022-08-29T15:58:59Z

notebooks/community/model_evaluation/automl_tabular_regression_model evaluation.ipynb

@@ -0,0 +1,1273 @@
+{


TODO?

Reply via ReviewNB

andrewferlitsch · 2022-08-29T15:59:01Z

notebooks/community/model_evaluation/automl_tabular_regression_model evaluation.ipynb

@@ -0,0 +1,1273 @@
+{


Add text cell explain using for eval data

Reply via ReviewNB

andrewferlitsch · 2022-08-29T15:59:01Z

notebooks/community/model_evaluation/automl_tabular_regression_model evaluation.ipynb

@@ -0,0 +1,1273 @@
+{


A bit of explanation on methods/params would help

Reply via ReviewNB

andrewferlitsch · 2022-08-29T15:59:01Z

notebooks/community/model_evaluation/automl_tabular_regression_model evaluation.ipynb

@@ -0,0 +1,1273 @@
+{


redundant. You already have the model from prev step

Reply via ReviewNB

Was for testing. Removed.

andrewferlitsch · 2022-08-29T15:59:01Z

notebooks/community/model_evaluation/automl_tabular_regression_model evaluation.ipynb

@@ -0,0 +1,1273 @@
+{


explain you are getting the AutoML eval metrics from training

Reply via ReviewNB

andrewferlitsch · 2022-08-29T15:59:01Z

notebooks/community/model_evaluation/automl_tabular_regression_model evaluation.ipynb

@@ -0,0 +1,1273 @@
+{


TODO?

Reply via ReviewNB

…older

…neh/vertex-ai-samples into soheilaz-model-eval

notebooks/community/model_evaluation/automl_tabular_regression_model_evaluation.ipynb

notebooks/community/model_evaluation/automl_tabular_classification_model_evaluation.ipynb

…ex-ai-samples into soheilaz-model-eval

…ge textual descriptions, add UUID

notebooks/community/model_evaluation/automl_tabular_regression_model_evaluation.ipynb

soheilazangeneh · 2022-09-05T20:56:37Z

notebooks/community/model_evaluation/automl_tabular_regression_model_evaluation.ipynb

@@ -0,0 +1,1473 @@
+{


This gives me the following error:
AttributeError: 'NoneType' object has no attribute 'artifacts'
Wondering if you could debug?
Apparently task.outputs.get('feature_attributions')returns None

Reply via ReviewNB

Cross checked again, working fine for me.
PFA Screenshot link
https://drive.google.com/file/d/1Ufrd6JsB8GbCk1ffz7WSqdnWUVfldgHY/view?usp=sharing

notebooks/community/model_evaluation/automl_tabular_classification_model_evaluation.ipynb

soheilazangeneh · 2022-09-05T20:56:41Z

notebooks/community/model_evaluation/automl_tabular_classification_model_evaluation.ipynb

@@ -0,0 +1,1427 @@
+{


This gives me the following error:
AttributeError: 'NoneType' object has no attribute 'artifacts'
Can you please run again and see if you get this error too? Is this code tested?

Reply via ReviewNB

I tested it again. It worked fine for me.

Here is a snapshot if it:

Hmm I ran it again and didn't work. Maybe we'll need to debug that. I am running it on workbench. Are these notebook tested on local env only?

Tested them on both Workbench and Colab. The above snapshot is from a Colab run.
Yes, it was tested in local env on workbench. Also, before testing, the env was installed with the requirements from .cloud-build/requirements.txt file.

KevinBNaughton · 2022-09-06T23:16:52Z

notebooks/community/model_evaluation/automl_tabular_classification_model_evaluation.ipynb

@@ -0,0 +1,1433 @@
+{


We also use "Vertex AI Model Registry"

And an additional step
"Import the Classification Metrics to the AutoML model resource"

Reply via ReviewNB

KevinBNaughton · 2022-09-06T23:16:52Z

notebooks/community/model_evaluation/automl_tabular_classification_model_evaluation.ipynb

@@ -0,0 +1,1433 @@
+{


For consistency, please capitalize all "dataflow" as "Dataflow"

Reply via ReviewNB

capitalized.

KevinBNaughton · 2022-09-06T23:16:52Z

notebooks/community/model_evaluation/automl_tabular_classification_model_evaluation.ipynb

@@ -0,0 +1,1433 @@
+{


Line #1. @kfp.dsl.pipeline(name="vertex-evaluation-automl-tabular-feature-attribution-pipeline")
Prefer to switch the name to "vertex-evaluation-automl-tabular-classification-feature-attribution", removing "pipeline"

Reply via ReviewNB

KevinBNaughton · 2022-09-06T23:16:52Z

notebooks/community/model_evaluation/automl_tabular_classification_model_evaluation.ipynb

@@ -0,0 +1,1433 @@
+{


Line #13. batch_predict_starting_replica_count: int = 5,
I think we can remove this for simplicity

Reply via ReviewNB

KevinBNaughton · 2022-09-06T23:16:52Z

notebooks/community/model_evaluation/automl_tabular_classification_model_evaluation.ipynb

@@ -0,0 +1,1433 @@
+{


Line #14. batch_predict_max_replica_count: int = 10,
Same here

Reply via ReviewNB

Add automl regression model eval first draft

1aaddd6

soheilazangeneh requested a review from a team as a code owner August 29, 2022 15:21

andrewferlitsch reviewed Aug 29, 2022

View reviewed changes

soheilazangeneh and others added 15 commits August 29, 2022 12:03

Remove extra file

c3a4a7f

Pring evaluation results

1e16e7b

adds the automl-tabular-classification notebook in model_evaluation f…

1b07079

…older

removes unnecessary imports

320f300

adjusts the imports inside the pipeline

a6c8d5c

adjusts the imports

cc88eff

Merge branch 'GoogleCloudPlatform:main' into soheilaz-model-eval

fec47f2

Merge branch 'soheilaz-model-eval' of https://github.com/soheilazange…

b6b9384

…neh/vertex-ai-samples into soheilaz-model-eval

elaborates imports inside pipeline

ce54769

modified regression notebook

293ce77

renamed pipeline displayname to resolve error

89862f5

Add automl regression model eval first draft

9343642

Remove extra file

aa60f88

Pring evaluation results

01863e8

modified some text

fa6652b

soheilazangeneh commented Sep 2, 2022

View reviewed changes

soheilazangeneh and others added 8 commits September 2, 2022 13:49

Merge branch 'soheilaz-model-eval' of github.com:soheilazangeneh/vert…

1e1973d

…ex-ai-samples into soheilaz-model-eval

Merge branch 'GoogleCloudPlatform:main' into soheilaz-model-eval

5e01921

added suggested updates from review: remove dataflow params, add/chan…

4daae03

…ge textual descriptions, add UUID

removes the output from the notebooks

1831486

removes the extra matplotlib import

ea11f9d

ran linter test

cf43980

addressed soheila's comments

2f8b659

ran linter

951eda9

soheilazangeneh commented Sep 5, 2022

View reviewed changes

Krishna Chaitanya Movva and others added 7 commits September 6, 2022 12:06

Merge branch 'GoogleCloudPlatform:main' into soheilaz-model-eval

adb6f2e

addresses the review comments

67f0f0e

ran linter test

4c782e2

removes the artifacts comment

5f0debb

ran linter test

86581cb

reviewed comments

6e2d5b7

ran linter

2f87036

soheilazangeneh changed the title ~~Add automl regression model eval first draft~~ Add automl regression and classification with model evaluation Sep 6, 2022

GoogleCloudPlatform deleted a comment from andrewferlitsch Sep 6, 2022

KevinBNaughton reviewed Sep 6, 2022

View reviewed changes

Add automl regression and classification with model evaluation #911

Add automl regression and classification with model evaluation #911

Uh oh!

Conversation

soheilazangeneh commented Aug 29, 2022

Uh oh!

review-notebook-app bot commented Aug 29, 2022

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

soheilazangeneh Sep 5, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

sudarshan-SpringML Sep 6, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

soheilazangeneh Sep 5, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

krishr2d2 Sep 6, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

krishr2d2 Sep 6, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

soheilazangeneh Sep 5, 2022 •

edited

Loading

sudarshan-SpringML Sep 6, 2022 •

edited

Loading

soheilazangeneh Sep 5, 2022 •

edited

Loading

krishr2d2 Sep 6, 2022 •

edited

Loading

krishr2d2 Sep 6, 2022 •

edited

Loading

KevinBNaughton Sep 6, 2022 •

edited

Loading

KevinBNaughton Sep 6, 2022 •

edited

Loading

KevinBNaughton Sep 6, 2022 •

edited

Loading

KevinBNaughton Sep 6, 2022 •

edited

Loading

KevinBNaughton Sep 6, 2022 •

edited

Loading