Skip to content

Index

Distributed joblib on Datatailr.

Run scikit-learn (or any joblib-parallel code) on an ephemeral pool of Datatailr worker containers with no manual setup::

import joblib
from datatailr.joblib import register_datatailr

register_datatailr()
with joblib.parallel_backend("datatailr", n_jobs=16):
    search.fit(X, y)   # e.g. RandomizedSearchCV

# Optional: pack more fits per broker round-trip (default initial_batch_size=128)
# with joblib.parallel_backend("datatailr", n_jobs=16, initial_batch_size=256):
#     search.fit(X, y)

See :class:datatailr.joblib.DatatailrBackend for the available backend options (worker resources, environment, embedded modules, timeouts).