Snowfalke DataSource: Add BatchDefinition based on some column "x" and a filter value

GX Core Support
1

Hey,

we are currently evaluating GX for our data tests, but ran into the problem that we need to partition our table but can’t / do not know how.

We use “add_or_update_snowflake” to have GX directly connected to the database and want to select specific batches of data based on a column. So basically a dynamic WHERE filter.

How is that possible? Do we have to create our own partitioner? Is there any documentation on how to do that?

We also check add_query_asset, but that will result in all of the data being loaded to the “local” machine, which is not what we want.

We also tried using a row_condition in the expectations and ran into another problem:

validator.expect_column_values_to_not_be_null(
    "SOME_TEST_COLUMN",
    row_condition=(
        gx.expectations.row_conditions.Column("SOME_COLUMNE").is_in(["SOME_VALUE"])
    ),
)

TypeError: type=‘comparison’ column=Column(name=‘SOME_COLUMN’) operator=‘IN’ parameter=[‘SOME_VALUE’] is of type ComparisonCondition which cannot be serialized.