Distributed scikit-learn
datatailr.joblib runs joblib parallel work on a pool of Datatailr worker containers instead of local processes. scikit-learn uses joblib for estimators and tools that take n_jobs, for example GridSearchCV, RandomizedSearchCV, cross_val_score and RandomForestClassifier. Your own joblib.Parallel calls also work.
You do not write a workflow or a service. The SDK starts the pool, sends the work to it and removes the pool when the work is done.
Example
import joblib
from sklearn.datasets import make_classification
from sklearn.ensemble import RandomForestClassifier
from sklearn.model_selection import GridSearchCV
from datatailr.joblib import register_datatailr
X, y = make_classification(n_samples=5000, n_features=20, random_state=0)
search = GridSearchCV(
RandomForestClassifier(random_state=0),
{"n_estimators": [100, 200, 400], "max_depth": [5, 10, None]},
cv=5,
)
register_datatailr()
with joblib.parallel_backend("datatailr", n_jobs=16):
search.fit(X, y)
print(search.best_params_)
register_datatailr() adds the "datatailr" backend to joblib. Call it one time in each Python process. In the with block, fit runs its 45 cross-validation fits on 16 worker containers.
Do not set n_jobs on the estimator. The estimator then uses the n_jobs of parallel_backend. If the estimator has n_jobs=-1, the pool gets 4 workers, unless you set the n_workers option.
What happens during a run
- The first
Parallelcall in thewithblock starts a pool: one small broker service and one worker container for each job. The pool starts in some tens of seconds. - joblib sends the tasks to the workers in batches. The SDK uploads each NumPy array, pandas object or bytes object of 1 MiB or more one time for the pool, for example the training data. The tasks then refer to the uploaded copy.
- When the
Parallelcall ends, fails or is canceled, the SDK stops the pool and removes its jobs and its temporary files. The nextParallelcall starts a new pool.
An exception in a task goes back to your code, as with local joblib. If a worker container stops during a task, for example because it is out of memory, another worker runs the task again. After 3 attempts, the task fails with an error message.
If your Python process stops before it removes the pool, the pool stops itself after 10 minutes without contact. To stop all pools of your process earlier, call datatailr.joblib.shutdown_pools().
Options
Give the options to parallel_backend together with n_jobs:
with joblib.parallel_backend(
"datatailr",
n_jobs=16,
worker_memory="4g",
worker_cpu=2,
):
search.fit(X, y)
| Option | Default | Description |
|---|---|---|
n_workers |
n_jobs of the first Parallel call |
Number of worker containers. If n_jobs is -1, the default is 4. |
worker_memory |
"2g" |
Memory for each worker container. |
worker_cpu |
1 |
CPU cores for each worker container. |
environment |
"dev" |
Datatailr environment of the pool. |
embed_modules |
none | Your own modules that are not in the workstation image. The SDK sends their code with each task. Give module objects or module names. |
idle_timeout |
600 |
Seconds without contact from your process before the pool stops itself. |
start_timeout |
600 |
Seconds to wait for the pool to start. |
initial_batch_size |
128 |
Number of joblib calls in a batch at the start. joblib then changes the batch size. |
Requirements
The workers run in the image of a Datatailr workstation. If you run your code in a workstation, the workers use the image of that workstation. On your own computer, they use the image of default-workstation. To use the SDK on your own computer, see Installing Locally.
- Install
joblib,cloudpickleandwebsocketswhere your code runs. Thedatatailrpackage does not install them. - The Python requirements of the workstation image must include
cloudpickleandwebsockets. The workers and the broker service use them. - The workstation image must contain the packages that the tasks use, for example scikit-learn and numpy. The workers do not get packages that you install only in your own environment.
- Your Python version must have the same minor version as the workstation image, for example 3.12.
When to use it
Each batch goes to the workers over the network, and each pool takes some tens of seconds to start. Small, fast fits can be slower than local joblib. The pool helps when each fit takes long enough, for example large parameter grids, large models or large data sets.
For all classes and functions of the module, see the API reference.