New to GX can someone help?

GX OSS Support
1

I am new to GX started and I am able to write expectations and planning to use the same for one of my project, but there are several questions I am not able to clear through the documentations.

  1. what would be the file structure of GX expectations for a production ready code, I mean If I write a code in pyhton file i.e in memory is it fine or should I create a checkpoint yml and deploy the same?

  2. are expectations re-usable and how to re-use them lets say I have one datasource and expectations created for that is there a way that I can use same expectations for different datasource and a way of passing the datasource as variable parameter just like a arg to function?

I would really appreciate the help, please note: I am very new to GX

2

Hey @Ravindra ! Thanks for reaching out.

To your first question, no need to create a separate checkpoint script – take a look at this documentation showing Checkpoint creation in-code.

To your second question, you can totally re-use expectations as long as your expectations for the second datasource are the same. If they have the same structure, same content within the bounds of your expectations (for example, if you are validating data after migrating) then everything should work fine. You will need to set new expectations if your datasources are fundamentally different.

3

First of all Thank you for answering the concerns, I got myself confused across GX versions docs, I just want to be more precise.

  1. with respect to the documentation you reffered,
    checkpoint = context.add_or_update_checkpoint(
    name=“my_checkpoint”,
    validations=[
    {
    “batch_request”: batch_request,
    “expectation_suite_name”: “my_expectation_suite”,
    },
    ],
    )

does the above codeblock creates an Yaml file as a check point , which used to be in earlier version.
if yes then this yaml files should be in production? if not then this logic/codeblock is a checkpoint?

  1. lets say I have an exception for example:
    expectation_configuration = ExpectationConfiguration(
    expectation_type=“expect_column_values_to_be_in_set”,
    kwargs={
    “column”: “transaction_type”,
    “value_set”: [“purchase”, “refund”, “upgrade”],
    },
    )

can I pass value_set list values as an arg or parameters between different datasource considering datasource has exactly same dimensions but the values for which I want to check is different.

  1. do we have any example anywhere I can refer to as a production ready code for GX?

  2. Is there auto-profiling possible with GX, if so can you point me to the doc section it would be really helpful.