Distributed scikit-learn
datatailr.joblib runs joblib parallel work on a pool of Datatailr worker containers instead of local processes. scikit-learn uses joblib for estimators and tools that take n_jobs, for example GridSearchCV, RandomizedSearchCV, cross_val_score and RandomForestClassifier. Your own joblib.Parallel calls also work.
You do not write a workflow or a service. The SDK starts the pool, sends the work to it and removes the pool when the work is done.
Example
import joblib
from sklearn.datasets import make_classification
from sklearn.ensemble import RandomForestClassifier
from sklearn.model_selection import GridSearchCV
from datatailr.joblib import register_datatailr
X, y = make_classification(n_samples=5000, n_features=20, random_state=0)
search = GridSearchCV(
RandomForestClassifier(random_state=0),
{"n_estimators": [100, 200, 400], "max_depth": [5, 10, None]},
cv=5,
)
register_datatailr()
with joblib.parallel_backend("datatailr", n_jobs=16):
search.fit(X, y)
print(search.best_params_)
register_datatailr() adds the "datatailr" backend to joblib. Call it one time in each Python process. In the with block, fit runs its 45 cross-validation fits on 16 worker containers.
Do not set n_jobs on the estimator. The estimator then uses the n_jobs of parallel_backend. If the estimator has n_jobs=-1, the pool gets 4 workers, unless you set the n_workers option.
What happens during a run
- The first
Parallelcall in thewithblock starts a pool: one small broker service and one worker container for each job. The pool starts in some tens of seconds. - joblib sends the tasks to the workers in batches. The SDK uploads each NumPy array, pandas object or bytes object of 512 KB or more one time for the pool, for example the training data. The tasks then refer to the uploaded copy.
- When the
Parallelcall ends, fails or is canceled, the SDK stops the pool and removes its jobs and its temporary files. The nextParallelcall starts a new pool.
An exception in a task goes back to your code, as with local joblib. If a worker container stops during a task, for example because it is out of memory, another worker runs the task again. After 3 attempts, the task fails with an error message.
If your Python process stops before it removes the pool, the pool stops itself after 10 minutes without contact. To stop all pools of your process earlier, call datatailr.joblib.shutdown_pools().
Options
Give the options to parallel_backend together with n_jobs:
with joblib.parallel_backend(
"datatailr",
n_jobs=16,
worker_memory="4g",
worker_cpu=2,
):
search.fit(X, y)
| Option | Default | Description |
|---|---|---|
n_workers |
n_jobs of the first Parallel call |
Number of worker containers. If n_jobs is -1, the default is 4. |
worker_memory |
"2g" |
Memory for each worker container. |
worker_cpu |
1 |
CPU cores for each worker container. A worker runs one fit for each whole core at the same time, so the pool runs n_jobs × worker_cpu fits in parallel. |
environment |
"dev" |
Datatailr environment of the pool. |
embed_modules |
none | Your own modules that are not in the workstation image. The SDK sends their code with each task. Give module objects or module names. |
share_threshold |
524288 |
Size in bytes from which the SDK uploads a NumPy array, pandas object or bytes object one time for the pool. |
idle_timeout |
600 |
Seconds without contact from your process before the pool stops itself. |
start_timeout |
600 |
Seconds to wait for the pool to start. |
initial_batch_size |
128 |
Number of joblib calls in a batch at the start. joblib then changes the batch size. |
Requirements
The workers run in the image of a Datatailr workstation. If you run your code in a workstation, the workers use the image of that workstation. On your own computer, they use the image of default-workstation. To use the SDK on your own computer, see Installing Locally.
- Install
joblib,cloudpickleandwebsocketswhere your code runs. Thedatatailrpackage does not install them. - The Python requirements of the workstation image must include
cloudpickleandwebsockets. The workers and the broker service use them. - The workstation image must contain the packages that the tasks use, for example scikit-learn and numpy. The workers do not get packages that you install only in your own environment.
- Your Python version must have the same minor version as the workstation image, for example 3.12.
When to use it
Each batch goes to the workers over the network, and each pool takes some tens of seconds to start. Small, fast fits can be slower than local joblib. The pool helps when each fit takes long enough, for example large parameter grids, large models or large data sets.
For a given number of cores, fewer workers with more cores start faster and use less memory. For example, n_jobs=4 with worker_cpu=4 gives 16 parallel fits, but the workers download NumPy training data 4 times instead of 16 times and keep 4 copies of it in memory instead of 16. The scheduler must find a node with 4 free cores for each worker, so use smaller workers if the cluster is nearly full. The fits of one worker share its memory, so increase worker_memory together with worker_cpu.
For all classes and functions of the module, see the API reference.