How to/best practice--a composite batch or custom batch?

GX OSS Support
1

tHave 1) fluent data source feeding 2) python code to manipulate data.

how or is it possible or best practice:

  • linking up in production via yml?
  • can yml call python code to create a 1) batch or 2) validator or 3) context
  • can one create a composite batch (fluent with pandas manipulation) as custom, just like an expectation in <>/plugins/expectation/*.py?

regards,

2

Hey @ddallai! Thanks for reaching out.

Can you expand on what you’re looking for with your first bullet?

To your second point, yml configs can parsed with Python to create these objects – if you see this older version of our docs, most config in that version of GX was done by passing yml configs.

To your third point, GX doesn’t support this out of the box, you would need to extend the code yourself to support – sounds like an interesting use case though!