Skip to content

Model validation for FS models#165

Merged
arpitjasa-db merged 12 commits into
databricks:mainfrom
aliazzzdat:dev_validation
Jul 25, 2024
Merged

Model validation for FS models#165
arpitjasa-db merged 12 commits into
databricks:mainfrom
aliazzzdat:dev_validation

Conversation

@aliazzzdat

Copy link
Copy Markdown
Contributor

FS models cannot predict as pyfunc models yet (issue #70), thus we can't use MLflow Evaluate with FS models.

However, MLflow Evaluate can take a function instead of a pyfunc model_uri.
This means we can call an FS model in the function and use this function in the Evaluate method.

Although this is a good solution, we still cannot enable baseline comparison, as we need to provide a model_uri of a pyfunc model (so this means it's not possible to compare a new FS model with a baseline FS model).

@arpitjasa-db

Copy link
Copy Markdown
Contributor

@aliazzzdat thanks for all the changes! Looks like some tests are failing, can we fix those and also do a full instantiation of Stacks to confirm everything is still working as expected? Version changes can be tricky, and want to make sure we don't break anything

Comment thread tests/example-project-configs/aws/aws-github.json Outdated
@aliazzzdat

Copy link
Copy Markdown
Contributor Author

Featurization pipeline works on DBR 15.3 and with MLflow 2.11
Screenshot 2024-07-19 at 21 43 00

Model training pipeline works on DBR 15.3 and with MLflow 2.11
Screenshot 2024-07-19 at 21 43 29

Validation works with a Feature Store model
Screenshot 2024-07-19 at 21 43 57
Screenshot 2024-07-19 at 21 44 16
Screenshot 2024-07-19 at 21 44 30

@aliazzzdat

Copy link
Copy Markdown
Contributor Author

The batch inference works as expected once we create the feature_store_inference_input table
Screenshot 2024-07-19 at 21 54 34

Comment thread .gitignore Outdated
Comment thread tests/example-project-configs/aws/aws-github-fs.json Outdated

@arpitjasa-db arpitjasa-db left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for all the changes @aliazzzdat !

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants