Datasource for Microsoft Fabric throwing error when I call get_batch() method on a batch_definition

GX Core (OSS) Support
1

Hi, I am trying to run a simple validation using Microsoft Fabric.

import great_expectations as gx
import great_expectations.expectations as gxe
import great_expectations.exceptions as exceptions
from great_expectations.core.expectation_suite import ExpectationSuite

DATA_SOURCE_NAME_FABRIC: str = "Fabric Data Source"
SEMANTIC_MODEL_NAME: str = "Retail Analysis Sample PBIX"
ASSET_TABLE_NAME: str = "District"
BATCH_DEFINITION_NAME_FABRIC: str = "District Data Batch Definition"

context = gx.get_context(mode="file")
data_source = context.data_sources.add_fabric_powerbi(name=DATA_SOURCE_NAME_FABRIC,dataset=SEMANTIC_MODEL_NAME)

try:
    table_asset = data_source.add_powerbi_table_asset(name=ASSET_TABLE_NAME,table=ASSET_TABLE_NAME)
except ValueError:                               
    table_asset = data_source.get_asset(ASSET_TABLE_NAME)

batch_definition = table_asset.add_batch_definition(name=BATCH_DEFINITION_NAME_FABRIC)

batch_definition.get_batch().head()
# I got an error whenever I call the get_batch() method.

Error:

image
image1434×627 38.3 KB

I tried using the data source pandas, but first converting using sempy then provide as batch_parameters, and it worked.

My environment:
Spark: 3.4
Great Expectations version 1.0.1

2

hello @dpires92, Welcome to our community!

thank you for bringing this to our attention. because Microsoft fabric is not a fluent supported data source, it was not initially updated for 1.0 however we do have the fix for this that will likely land in our next release. Did you say you were able to get it working by adjusting to a pandas data source?

1 Like
3

Hi @adeola. Yes I ran using pandas data source and Data asset as a whole dataframe.
I used sempy explicitly to read a table from the semantic model as a dataframe then I passed it to the get_batch() as batch parameters.
In this way worked.
However many thanks for the heads up.