What it is
Dask is a flexible library for parallel computing in Python. It allows you to scale NumPy, Pandas, and Python functions to multi-core machines or distributed clusters for large-scale data processing.
Dask allows you to parallelize computations on large datasets by providing data structures like Dask Array, Dask DataFrame, and Dask Bag. It integrates seamlessly with NumPy, Pandas, and Scikit-learn, and can run on a single machine or distributed cluster.
Installation
pip install dask[complete]Getting started
The smallest useful thing you can do with it, and what each part means.
import dask.array as da
x = da.arange(1000, chunks=100)
y = x + 1
print(y.sum().compute())import dask.dataframe as dd
df = dd.read_csv('data/*.csv')
print(df.head())
print(df['column'].mean().compute())Advanced usage
Where the library earns its place over a simpler alternative.
from dask import delayed
@delayed
def inc(x):
return x + 1
@delayed
def add(x, y):
return x + y
x = inc(10)
y = inc(20)
z = add(x, y)
print(z.compute())import dask_ml.linear_model as dlm
from dask.distributed import Client
client = Client()
model = dlm.LinearRegression()
# Fit model on Dask arrays or DataFramesfrom dask.distributed import Client
client = Client('tcp://scheduler-address:8786')
# Submit tasks to the cluster
future = client.submit(pow, 2, 10)
print(future.result())Errors and fixes
The failures you are most likely to hit, and what actually resolves them.
- ValueError: Chunk size too large
- Adjust chunk sizes to fit in memory when creating Dask arrays or DataFrames.
- FileNotFoundError
- Ensure all file paths exist when using Dask’s read functions.
- dask.distributed.core.TimeoutError
- Check connectivity to the distributed cluster and adjust timeout settings.
Best practices
- Break large datasets into appropriately sized chunks to optimize parallelism.
- Use Dask collections that mimic familiar NumPy/Pandas interfaces to reduce learning curve.
- Leverage `compute()` only when necessary to trigger execution; avoid multiple small `compute()` calls.
- Use Dask’s distributed scheduler for multi-machine setups for large-scale computation.
- Monitor tasks with the Dask dashboard to detect bottlenecks and improve performance.
Background
Why it exists, and what it was reacting to.
Dask was created by Matthew Rocklin in 2014 to enable scalable analytics in Python. It provides parallel collections that mimic the interfaces of NumPy arrays, Pandas DataFrames, and Python iterators, making it easy to transition existing code to parallel or distributed computing.