This article is for comments to: https://docs.greatexpectations.io/en/latest/how_to_guides/creating_and_editing_expectations/how_to_create_a_new_expectation_suite_using_suite_scaffold.html
Please comment +1 if this How to is important to you.
Is there a way to create expectation based on current data profile using scaffold in python instead of the interactive cli? Would very much want to do it as an airflow job
You should be able to just use the boilerplate code that’s in the notebook when you run suite scaffold directly in an Airflow task! suite scaffoldcreates a new suite under the hood, so you’ll have to add that step to the code:
- Grab the code to create a new suite without the CLI here: https://docs.greatexpectations.io/en/latest/guides/how_to_guides/creating_and_editing_expectations/how_to_create_a_new_expectation_suite_without_the_cli.html
- Do a “dummy” run with
great_expectations suite scaffoldto open the scaffold notebook and grab the code that runs the profiling and saves the suite. - Drop the code into an Airflow Python task
Could you report back here if that worked for you?