I have a Delta Table on Databricks. I want to use GX to check if a column date has the right date format of “yyyy-MM-dd”, is there a way to achieve that?
regex like ‘^\d{4}-\d{2}-\d{2}$’ doesn’t seem a good idea since it won’t fail date like ‘0000-21-89’ or ‘2023-02-29’
What version of GX are you using?
Looks like 1.0 and above don’t have this but versions before 1.0 do have an Expectation for this.
See the documentation for that here: expect_column_values_to_be_valid_date
I’m using 1.1.0 since according to this doc: the older versions are “no longer actively maintained”
I just tried with version 1.1.0. You are right it’s not supported there. Can we have it added in the newer versions?
I think following this guidance in our documentation would be your best option. It details how to instantiate UnexpectedRowsExpectation to define a custom Expectation. In your case you could create a subclass of UnexpectedRowsExpectation and implement the SQL logic that validates whether the rows in a column conform to your expected date format.
Thanks adeola!
may i know the reason why date format check is not supported anymore? and is there a planning to add this back to the newer versions?
your welcome @lauraz! I can double-check, but I believe maintaining all the different expectations was challenging and still somewhat limited. That’s why we’re leaning towards offering full customization across as many areas of the product as possible. While it requires a bit more effort, using SQL to define your own expectations significantly expands GX’s capabilities.
Appreciate it!
agree that customization extends flexibility. Just that date check is something that’s commonly used, and it was already supported in the old version, feel like it should be in the newer versions even for backward compatibility.
after speaking with a few folks, it looks like we decided to move forward with 47 of the most widely used and best-supported Expectations, the limited data source support for the date format expectation was likely why this wasnt chosen
1 Like
Got it, thnx for checking!
1 Like