Skip to content

Add automl regression and classification with model evaluation #911

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
merged 43 commits into from
Sep 8, 2022
Merged

Add automl regression and classification with model evaluation #911

merged 43 commits into from
Sep 8, 2022

Conversation

soheilazangeneh
Copy link
Contributor


This is the first draft for AutoML tabular regression model with the focus of model evolution.


  1. If you are opening a PR for Community Notebooks under the notebooks/community folder:
  • This notebook has been added to the CODEOWNERS file under the Community Notebooks section, pointing to the author or the author's team.
  • Passes all the required formatting and linting checks. You can locally test with these instructions.

  1. If you are opening a PR for Community Content under the community-content folder:
  • Make sure your main Content Directory Name is descriptive, informative, and includes some of the key products and attributes of your content, so that it is differentiable from other content
  • The main content directory has been added to the CODEOWNERS file under the Community Content section, pointing to the author or the author's team.
  • Passes all the required formatting and linting checks. You can locally test with these instructions.

@soheilazangeneh soheilazangeneh requested a review from a team as a code owner August 29, 2022 15:21
@review-notebook-app
Copy link

Check out this pull request on  ReviewNB

See visual diffs & provide feedback on Jupyter Notebooks.

Powered by ReviewNB

andrewferlitsch
andrewferlitsch reviewed Aug 29, 2022
View reviewed changes
notebooks/community/model_evaluation/automl_tabular_regression_model evaluation.ipynb Outdated
@@ -0,0 +1,1273 @@
{
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Need H1 title in this cell. Should be above the links.


Reply via ReviewNB

notebooks/community/model_evaluation/automl_tabular_regression_model evaluation.ipynb Outdated
@@ -0,0 +1,1273 @@
{
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

regression model evaluation componenet -> regression model evaluation pipeline component


Reply via ReviewNB

notebooks/community/model_evaluation/automl_tabular_regression_model evaluation.ipynb Outdated
@@ -0,0 +1,1273 @@
{
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Replace : with . in first sentence

regression evluation component -> model evaluation pipeline component

you say 'pre-trained', but you train the model in the notebook

Add to services, Big Query


Reply via ReviewNB

notebooks/community/model_evaluation/automl_tabular_regression_model evaluation.ipynb Outdated
@@ -0,0 +1,1273 @@
{
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Combine this with the above cell


Reply via ReviewNB

notebooks/community/model_evaluation/automl_tabular_regression_model evaluation.ipynb Outdated
@@ -0,0 +1,1273 @@
{
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

TODO?


Reply via ReviewNB

notebooks/community/model_evaluation/automl_tabular_regression_model evaluation.ipynb Outdated
@@ -0,0 +1,1273 @@
{
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Add text cell explain using for eval data


Reply via ReviewNB

notebooks/community/model_evaluation/automl_tabular_regression_model evaluation.ipynb Outdated
@@ -0,0 +1,1273 @@
{
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

A bit of explanation on methods/params would help


Reply via ReviewNB

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

done.

notebooks/community/model_evaluation/automl_tabular_regression_model evaluation.ipynb Outdated
@@ -0,0 +1,1273 @@
{
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

redundant. You already have the model from prev step


Reply via ReviewNB

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Was for testing. Removed.

notebooks/community/model_evaluation/automl_tabular_regression_model evaluation.ipynb Outdated
@@ -0,0 +1,1273 @@
{
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

explain you are getting the AutoML eval metrics from training


Reply via ReviewNB

notebooks/community/model_evaluation/automl_tabular_regression_model evaluation.ipynb Outdated
@@ -0,0 +1,1273 @@
{
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

TODO?


Reply via ReviewNB

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Updated.

soheilazangeneh and others added 15 commits August 29, 2022 12:03
soheilazangeneh
soheilazangeneh commented Sep 2, 2022
View reviewed changes
soheilazangeneh
soheilazangeneh commented Sep 2, 2022
View reviewed changes
soheilazangeneh
soheilazangeneh commented Sep 5, 2022
View reviewed changes
notebooks/community/model_evaluation/automl_tabular_regression_model_evaluation.ipynb
@@ -0,0 +1,1473 @@
{
Copy link
Contributor Author

@soheilazangeneh soheilazangeneh Sep 5, 2022 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This gives me the following error:

AttributeError: 'NoneType' object has no attribute 'artifacts'

Wondering if you could debug?

Apparently task.outputs.get('feature_attributions')returns None


Reply via ReviewNB

Copy link
Contributor

@sudarshan-SpringML sudarshan-SpringML Sep 6, 2022 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Cross checked again, working fine for me.

PFA Screenshot link

https://drive.google.com/file/d/1Ufrd6JsB8GbCk1ffz7WSqdnWUVfldgHY/view?usp=sharing

soheilazangeneh
soheilazangeneh commented Sep 5, 2022
View reviewed changes
notebooks/community/model_evaluation/automl_tabular_classification_model_evaluation.ipynb
@@ -0,0 +1,1427 @@
{
Copy link
Contributor Author

@soheilazangeneh soheilazangeneh Sep 5, 2022 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This gives me the following error: 

AttributeError: 'NoneType' object has no attribute 'artifacts'

Can you please run again and see if you get this error too? Is this code tested?


Reply via ReviewNB

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I tested it again. It worked fine for me.

Copy link
Contributor

@krishr2d2 krishr2d2 Sep 6, 2022 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Here is a snapshot if it:

evaluation_print_step

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hmm I ran it again and didn't work. Maybe we'll need to debug that. I am running it on workbench. Are these notebook tested on local env only?

Copy link
Contributor

@krishr2d2 krishr2d2 Sep 6, 2022 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Tested them on both Workbench and Colab. The above snapshot is from a Colab run.
Yes, it was tested in local env on workbench. Also, before testing, the env was installed with the requirements from .cloud-build/requirements.txt file.

@soheilazangeneh soheilazangeneh changed the title Add automl regression model eval first draft Add automl regression and classification with model evaluation Sep 6, 2022
@GoogleCloudPlatform GoogleCloudPlatform deleted a comment from andrewferlitsch