Add automl regression and classification with model evaluation #911

soheilazangeneh · 2022-08-29T15:21:48Z

This is the first draft for AutoML tabular regression model with the focus of model evolution.

If you are opening a PR for Community Notebooks under the notebooks/community folder:

This notebook has been added to the CODEOWNERS file under the Community Notebooks section, pointing to the author or the author's team.
Passes all the required formatting and linting checks. You can locally test with these instructions.

If you are opening a PR for Community Content under the community-content folder:

Make sure your main Content Directory Name is descriptive, informative, and includes some of the key products and attributes of your content, so that it is differentiable from other content
The main content directory has been added to the CODEOWNERS file under the Community Content section, pointing to the author or the author's team.
Passes all the required formatting and linting checks. You can locally test with these instructions.

review-notebook-app · 2022-08-29T15:21:53Z

Check out this pull request on

See visual diffs & provide feedback on Jupyter Notebooks.

Powered by ReviewNB

andrewferlitsch · 2022-08-29T15:58:59Z

notebooks/community/model_evaluation/automl_tabular_regression_model evaluation.ipynb

@@ -0,0 +1,1273 @@
+{


Need H1 title in this cell. Should be above the links.

Reply via ReviewNB

andrewferlitsch · 2022-08-29T15:58:59Z

notebooks/community/model_evaluation/automl_tabular_regression_model evaluation.ipynb

@@ -0,0 +1,1273 @@
+{


regression model evaluation componenet -> regression model evaluation pipeline component

Reply via ReviewNB

andrewferlitsch · 2022-08-29T15:58:59Z

notebooks/community/model_evaluation/automl_tabular_regression_model evaluation.ipynb

@@ -0,0 +1,1273 @@
+{


Replace : with . in first sentence

regression evluation component -> model evaluation pipeline component

you say 'pre-trained', but you train the model in the notebook

Add to services, Big Query

Reply via ReviewNB

andrewferlitsch · 2022-08-29T15:58:59Z

notebooks/community/model_evaluation/automl_tabular_regression_model evaluation.ipynb

@@ -0,0 +1,1273 @@
+{


Combine this with the above cell

Reply via ReviewNB

andrewferlitsch · 2022-08-29T15:58:59Z

notebooks/community/model_evaluation/automl_tabular_regression_model evaluation.ipynb

@@ -0,0 +1,1273 @@
+{


TODO?

Reply via ReviewNB

andrewferlitsch · 2022-08-29T15:59:01Z

notebooks/community/model_evaluation/automl_tabular_regression_model evaluation.ipynb

@@ -0,0 +1,1273 @@
+{


Add text cell explain using for eval data

Reply via ReviewNB

andrewferlitsch · 2022-08-29T15:59:01Z

notebooks/community/model_evaluation/automl_tabular_regression_model evaluation.ipynb

@@ -0,0 +1,1273 @@
+{


A bit of explanation on methods/params would help

Reply via ReviewNB

andrewferlitsch · 2022-08-29T15:59:01Z

notebooks/community/model_evaluation/automl_tabular_regression_model evaluation.ipynb

@@ -0,0 +1,1273 @@
+{


redundant. You already have the model from prev step

Reply via ReviewNB

Was for testing. Removed.

andrewferlitsch · 2022-08-29T15:59:01Z

notebooks/community/model_evaluation/automl_tabular_regression_model evaluation.ipynb

@@ -0,0 +1,1273 @@
+{


explain you are getting the AutoML eval metrics from training

Reply via ReviewNB

andrewferlitsch · 2022-08-29T15:59:01Z

notebooks/community/model_evaluation/automl_tabular_regression_model evaluation.ipynb

@@ -0,0 +1,1273 @@
+{


TODO?

Reply via ReviewNB

…older

…neh/vertex-ai-samples into soheilaz-model-eval

notebooks/community/model_evaluation/automl_tabular_regression_model_evaluation.ipynb

notebooks/community/model_evaluation/automl_tabular_classification_model_evaluation.ipynb

…ex-ai-samples into soheilaz-model-eval

…ge textual descriptions, add UUID

notebooks/community/model_evaluation/automl_tabular_regression_model_evaluation.ipynb

soheilazangeneh · 2022-09-05T20:56:37Z

notebooks/community/model_evaluation/automl_tabular_regression_model_evaluation.ipynb

@@ -0,0 +1,1473 @@
+{


This gives me the following error:
AttributeError: 'NoneType' object has no attribute 'artifacts'
Wondering if you could debug?
Apparently task.outputs.get('feature_attributions')returns None

Reply via ReviewNB

Cross checked again, working fine for me.
PFA Screenshot link
https://drive.google.com/file/d/1Ufrd6JsB8GbCk1ffz7WSqdnWUVfldgHY/view?usp=sharing

notebooks/community/model_evaluation/automl_tabular_classification_model_evaluation.ipynb

soheilazangeneh · 2022-09-05T20:56:41Z

notebooks/community/model_evaluation/automl_tabular_classification_model_evaluation.ipynb

@@ -0,0 +1,1427 @@
+{


This gives me the following error:
AttributeError: 'NoneType' object has no attribute 'artifacts'
Can you please run again and see if you get this error too? Is this code tested?

Reply via ReviewNB

I tested it again. It worked fine for me.

Here is a snapshot if it:

Hmm I ran it again and didn't work. Maybe we'll need to debug that. I am running it on workbench. Are these notebook tested on local env only?

Tested them on both Workbench and Colab. The above snapshot is from a Colab run.
Yes, it was tested in local env on workbench. Also, before testing, the env was installed with the requirements from .cloud-build/requirements.txt file.

KevinBNaughton · 2022-09-06T23:16:52Z

notebooks/community/model_evaluation/automl_tabular_classification_model_evaluation.ipynb

@@ -0,0 +1,1433 @@
+{


We also use "Vertex AI Model Registry"

And an additional step
"Import the Classification Metrics to the AutoML model resource"

Reply via ReviewNB

KevinBNaughton · 2022-09-06T23:16:52Z

notebooks/community/model_evaluation/automl_tabular_classification_model_evaluation.ipynb

@@ -0,0 +1,1433 @@
+{


For consistency, please capitalize all "dataflow" as "Dataflow"

Reply via ReviewNB

capitalized.

KevinBNaughton · 2022-09-06T23:16:52Z

notebooks/community/model_evaluation/automl_tabular_classification_model_evaluation.ipynb

@@ -0,0 +1,1433 @@
+{


Line #1. @kfp.dsl.pipeline(name="vertex-evaluation-automl-tabular-feature-attribution-pipeline")
Prefer to switch the name to "vertex-evaluation-automl-tabular-classification-feature-attribution", removing "pipeline"

Reply via ReviewNB

KevinBNaughton · 2022-09-06T23:16:52Z

notebooks/community/model_evaluation/automl_tabular_classification_model_evaluation.ipynb

@@ -0,0 +1,1433 @@
+{


Line #13. batch_predict_starting_replica_count: int = 5,
I think we can remove this for simplicity

Reply via ReviewNB

KevinBNaughton · 2022-09-06T23:16:52Z

notebooks/community/model_evaluation/automl_tabular_classification_model_evaluation.ipynb

@@ -0,0 +1,1433 @@
+{


Line #14. batch_predict_max_replica_count: int = 10,
Same here

Reply via ReviewNB

KevinBNaughton · 2022-09-06T23:16:52Z

notebooks/community/model_evaluation/automl_tabular_classification_model_evaluation.ipynb

@@ -0,0 +1,1433 @@
+{


Line #80. problem_type=prediction_type,
Problem type is not required since classification_metrics is provided.

Reply via ReviewNB

KevinBNaughton · 2022-09-06T23:16:52Z

notebooks/community/model_evaluation/automl_tabular_classification_model_evaluation.ipynb

@@ -0,0 +1,1433 @@
+{


batch_predict_instances_format: Format of the input instances for batch prediction. Can be "jsonl" or "bigquery".

Reply via ReviewNB

for AutoML tabular, csv is supported as well

KevinBNaughton · 2022-09-06T23:16:53Z

notebooks/community/model_evaluation/automl_tabular_regression_model_evaluation.ipynb

@@ -0,0 +1,1488 @@
+{


Line #13. batch_predict_starting_replica_count: int = 5,
Ditto comments from other notebook

Reply via ReviewNB

KevinBNaughton · 2022-09-06T23:16:53Z

notebooks/community/model_evaluation/automl_tabular_regression_model_evaluation.ipynb

@@ -0,0 +1,1488 @@
+{


Line #89. problem_type=prediction_type,
Can remove from here, and remove from pipeline inputs

Reply via ReviewNB

KevinBNaughton · 2022-09-06T23:16:53Z

notebooks/community/model_evaluation/automl_tabular_regression_model_evaluation.ipynb

@@ -0,0 +1,1488 @@
+{


Ditto comments from other notebook

Reply via ReviewNB

…eters

…neh/vertex-ai-samples into soheilaz-model-eval

karenarialin · 2022-09-08T16:45:31Z

notebooks/community/model_evaluation/automl_tabular_classification_model_evaluation.ipynb

+      "source": [
+        "## Overview\n",
+        "\n",
+        "This notebook demonstrates how to use Vertex AI classification model evaluation component to evaluate an AutoML classification model. Model evaluation helps you determine your model performance based on the evaluation metrics and improve the model if necessary. "


how to use the Vertex AI

karenarialin · 2022-09-08T16:49:22Z

notebooks/community/model_evaluation/automl_tabular_classification_model_evaluation.ipynb

+        "into the filter box, and select\n",
+        "   **Vertex AI Administrator**. Type \"Storage Object Admin\" into the filter box, and select **Storage Object Admin**.\n",
+        "\n",
+        "5. Click *Create*. A JSON file that contains your key downloads to your\n",


Create should be bolded

karenarialin · 2022-09-08T16:50:51Z

notebooks/community/model_evaluation/automl_tabular_classification_model_evaluation.ipynb

+        "id": "XoEqT2Y4DJmf"
+      },
+      "source": [
+        "### Import libraries"


Brief description for this step?

karenarialin · 2022-09-08T16:51:55Z

notebooks/community/model_evaluation/automl_tabular_classification_model_evaluation.ipynb

+        "\n",
+        "- `display_name`: The human readable name for the Vertex AI TrainingJob resource.\n",
+        "- `optimization_prediction_type`: The type of prediction the Model is to produce. Ex: regression, classification.\n",
+        "- `column_specs`(Optional): Transformations to apply to the input columns(including data-type corrections).\n",


space needed after "columns"

karenarialin · 2022-09-08T16:53:11Z

notebooks/community/model_evaluation/automl_tabular_classification_model_evaluation.ipynb

+        "- `dataset`: The TabularDataset within the same Project from which data needs to be used to train the Model.\n",
+        "- `target_column`: The name of the column values of which the Model is to predict.\n",
+        "- `model_display_name`: The display name of the Vertex AI Model that is produced as an output. \n",
+        "- `budget_milli_node_hours`(Optional): The train budget of creating this Model, expressed in milli node hours i.e. 1,000 value in this field means 1 node hour. The training cost of the model does not exceed this budget.\n",


karenarialin · 2022-09-08T16:54:25Z

notebooks/community/model_evaluation/automl_tabular_classification_model_evaluation.ipynb

+      "source": [
+        "## Create Pipeline for evaluations\n",
+        "\n",
+        "Now, you run a Vertex AI BatchPrediction job and generate evaluations and feature-attributions on its results. \n",


No hyphen between feature attributions

karenarialin · 2022-09-08T16:54:56Z

notebooks/community/model_evaluation/automl_tabular_classification_model_evaluation.ipynb

+        "\n",
+        "Now, you run a Vertex AI BatchPrediction job and generate evaluations and feature-attributions on its results. \n",
+        "\n",
+        "To do so, you create a Vertex AI pipeline using the components available from the [`google-cloud-pipeline-components`](https://google-cloud-pipeline-components.readthedocs.io/en/google-cloud-pipeline-components-1.0.17/index.html) python package.\n"


capitalize Python

karenarialin · 2022-09-08T16:58:51Z

notebooks/community/model_evaluation/automl_tabular_classification_model_evaluation.ipynb

+      "source": [
+        "In the results from last step, click on the generated link to see your run in the Cloud Console.\n",
+        "\n",
+        "In the UI, many of the pipeline DAG nodes expand or collapse when you click on them. Here is a partially-expanded view of the DAG (click image to see larger version).\n",


can you spell out this acronym? i.e. "directed acyclic graph (DAG)"

…h data-sampler task

Add automl regression model eval first draft

1aaddd6

soheilazangeneh requested a review from a team as a code owner August 29, 2022 15:21

andrewferlitsch reviewed Aug 29, 2022

View reviewed changes

soheilazangeneh and others added 15 commits August 29, 2022 12:03

Remove extra file

c3a4a7f

Pring evaluation results

1e16e7b

adds the automl-tabular-classification notebook in model_evaluation f…

1b07079

…older

removes unnecessary imports

320f300

adjusts the imports inside the pipeline

a6c8d5c

adjusts the imports

cc88eff

Merge branch 'GoogleCloudPlatform:main' into soheilaz-model-eval

fec47f2

Merge branch 'soheilaz-model-eval' of https://github.com/soheilazange…

b6b9384

…neh/vertex-ai-samples into soheilaz-model-eval

elaborates imports inside pipeline

ce54769

modified regression notebook

293ce77

renamed pipeline displayname to resolve error

89862f5

Add automl regression model eval first draft

9343642

Remove extra file

aa60f88

Pring evaluation results

01863e8

modified some text

fa6652b

soheilazangeneh commented Sep 2, 2022

View reviewed changes

soheilazangeneh and others added 8 commits September 2, 2022 13:49

Merge branch 'soheilaz-model-eval' of github.com:soheilazangeneh/vert…

1e1973d

…ex-ai-samples into soheilaz-model-eval

Merge branch 'GoogleCloudPlatform:main' into soheilaz-model-eval

5e01921

added suggested updates from review: remove dataflow params, add/chan…

4daae03

…ge textual descriptions, add UUID

removes the output from the notebooks

1831486

removes the extra matplotlib import

ea11f9d

ran linter test

cf43980

addressed soheila's comments

2f8b659

ran linter

951eda9

soheilazangeneh commented Sep 5, 2022

View reviewed changes

Krishna Chaitanya Movva and others added 7 commits September 6, 2022 12:06

Merge branch 'GoogleCloudPlatform:main' into soheilaz-model-eval

adb6f2e

addresses the review comments

67f0f0e

ran linter test

4c782e2

removes the artifacts comment

5f0debb

ran linter test

86581cb

reviewed comments

6e2d5b7

ran linter

2f87036

soheilazangeneh changed the title ~~Add automl regression model eval first draft~~ Add automl regression and classification with model evaluation Sep 6, 2022

GoogleCloudPlatform deleted a comment from andrewferlitsch Sep 6, 2022

KevinBNaughton reviewed Sep 6, 2022

View reviewed changes

Krishna Chaitanya Movva and others added 8 commits September 7, 2022 11:29

Merge branch 'GoogleCloudPlatform:main' into soheilaz-model-eval

5debadb

addresses review comments: textual updates, removes unnecessary param…

bef3840

…eters

ran linter test

8f4a865

addressed comments

4dbad08

ran linter

2b1283f

Merge branch 'soheilaz-model-eval' of https://github.com/soheilazange…

299229a

…neh/vertex-ai-samples into soheilaz-model-eval

removed unwanted variables

94d65c9

ran linter

42b8f8c

GoogleCloudPlatform deleted a comment from andrewferlitsch Sep 7, 2022

karenarialin approved these changes Sep 8, 2022

View reviewed changes

Krishna Chaitanya Movva added 3 commits September 8, 2022 23:02

Merge branch 'GoogleCloudPlatform:main' into soheilaz-model-eval

4634898

addresses the tech-writer's comments + updates the pipeline image wit…

5fbda9e

…h data-sampler task

ran linter test

ee3683a

andrewferlitsch approved these changes Sep 8, 2022

View reviewed changes

Merge branch 'main' into soheilaz-model-eval

8579a78

andrewferlitsch merged commit 9ab5f42 into GoogleCloudPlatform:main Sep 8, 2022

Add automl regression and classification with model evaluation #911

Add automl regression and classification with model evaluation #911

Uh oh!

Conversation

soheilazangeneh commented Aug 29, 2022

Uh oh!

review-notebook-app bot commented Aug 29, 2022

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

soheilazangeneh Sep 5, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

sudarshan-SpringML Sep 6, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

soheilazangeneh Sep 5, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

krishr2d2 Sep 6, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

krishr2d2 Sep 6, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

soheilazangeneh Sep 5, 2022 •

edited

Loading

sudarshan-SpringML Sep 6, 2022 •

edited

Loading

soheilazangeneh Sep 5, 2022 •

edited

Loading

krishr2d2 Sep 6, 2022 •

edited

Loading

krishr2d2 Sep 6, 2022 •

edited

Loading

KevinBNaughton Sep 6, 2022 •

edited

Loading

KevinBNaughton Sep 6, 2022 •

edited

Loading

KevinBNaughton Sep 6, 2022 •

edited

Loading

KevinBNaughton Sep 6, 2022 •

edited

Loading

KevinBNaughton Sep 6, 2022 •

edited

Loading

KevinBNaughton Sep 6, 2022 •

edited

Loading

KevinBNaughton Sep 6, 2022 •

edited

Loading