# Can I use Great Expectations with dask?

**URL:** https://discourse.greatexpectations.io/t/can-i-use-great-expectations-with-dask/60
**Category:** Archive
**Created:** [March 2, 2020, 3:06pm UTC](https://discourse.greatexpectations.io/t/can-i-use-great-expectations-with-dask/60 "2020-03-02T15:06:16Z")
**Posts on this page:** 2
**Page:** 1

<div class="post-metadata">

### Author: ![abegong](https://avatars.discourse-cdn.com/v4/letter/a/ecd19e/32.png) [@abegong](https://discourse.greatexpectations.io/u/abegong)
#### Post date: [March 2, 2020, 3:06pm UTC](https://discourse.greatexpectations.io/t/can-i-use-great-expectations-with-dask/60/1 "2020-03-02T15:06:16Z")

</div>

Inquiring minds want to know.

---

<div class="post-metadata">

### Author: ![jorandox](https://avatars.discourse-cdn.com/v4/letter/j/ed8c4c/32.png) [@jorandox](https://discourse.greatexpectations.io/u/jorandox)
#### Post date: [March 3, 2020, 7:15am UTC](https://discourse.greatexpectations.io/t/can-i-use-great-expectations-with-dask/60/2 "2020-03-03T07:15:36Z")

</div>

Quick copy of my post in the great expectations slack channel:

For people wondering how to use GE with dask (distributed), and dockerized, here’s a quick write-up on how I did it:

1. quick and dirty way to install great expectations on the dask workers:

```python
def install_ge():
        import os
        os.system("pip install great_expectations")
    dask_client.register_worker_callbacks(install_ge)

```

1. cast the pandas dataframes underlying the dask dataframe to GE.PandasDataset and run the validation suite

```python
def run_suite(data, data_name, suite):
    def run_partition(data_in):
        return pd.Series(PandasDataset(data_in).validate(expectation_suite=suite)["results"])
    results = data.map_partitions(run_partition).persist()
    for result in results.compute():
        # do something with the results, like aggregating them or the like
        pass

```

Fair warning:  
More complex expectations won’t really work this way, where you compare e.g. the relative amounts of things, because they might not be scattered evenly.