Execution Environment - Azure Synapse Analytics
Programming Tool - Python 3.11
Package - great-expectations 0.15.26
Target Database - MS-SQL Server
Issue - MS-SQL Server tempdb is getting filled up due to creation of temp tables during execution of great expectation validation suite using python. The tempdb is not getting refreshed or cleanedup after execution.
The below error highlights the issue-
ProgrammingError: (pyodbc.ProgrammingError) (‘42000’, “[42000] [Microsoft][ODBC Driver 18 for SQL Server][SQL Server]Could not allocate space for object ‘<temporary system object: 7926337660761210880>’ in database ‘tempdb’ because the ‘PRIMARY’ filegroup is full due to lack of storage space or database files reaching the maximum allowed size. Note that UNLIMITED files are still limited to 16TB. Create the necessary space by dropping objects in the filegroup, adding additional files to the filegroup, or setting autogrowth on for existing files in the filegroup. (1105) (SQLExecDirectW)”)
Methods tried to solve the issue -
Tried solution mentioned in link - Turning off creating temporary tables with ClickHouse
Even after adding the create_temp_table = False while creating the SQL Data Source , the temp table still gets created.
Please let us know if there is any resolution for this .