Workflows
A workflow is a DAG-based data pipeline where you define a series of tasks with dependencies between them. Datatailr automatically determines which tasks can run in parallel based on their dependencies, making your pipelines faster without extra effort.
Workflows are the most common type of job on the Datatailr platform. They are ideal for batch data processing — ETL pipelines, ML model training, report generation, and any workload that has a defined start and end.
How It Works
You define a workflow using the @workflow and @task decorators. Each function decorated with @task becomes a node in the pipeline graph. When one task's output is passed as input to another, Datatailr creates a dependency edge between them. Independent tasks run in parallel automatically.
from datatailr import workflow, task
@task()
def fetch_data():
return {"sales": [100, 200, 150]}
@task()
def calculate_total(data):
return sum(data["sales"])
@workflow(name="daily_sales_report")
def sales_pipeline():
data = fetch_data()
total = calculate_total(data)
When deployed, this creates the following execution graph:
The return value of fetch_data is used as the data argument in calculate_total, which creates a dependency between them.
Compute resources
Each task runs in its own container. You set CPU and memory on the workflow, on a task, or on one call.
from datatailr import Resources, task, workflow
@task(memory="1g", cpu=1, vram="2g")
def train(data):
return data
@workflow(name="train", resources=Resources(memory="512m", cpu=1))
def pipeline():
train(load())
memory and cpu are always written to the job JSON. A workflow defaults to 100m of memory and 1 CPU. A task that sets only one of them keeps the workflow value for the other.
vram is GPU memory. It uses the same size strings as memory, for example "256m" or "2g". Sizes are binary: k, m, g and t are KiB, MiB, GiB and TiB, and "80Gi", "80GiB" and "80GB" all mean "80g". If you do not set vram, the job JSON does not include a vram field. A task that does not set vram uses the workflow value. If the workflow also omits vram, the task definition omits it.
.set_resources(memory=..., cpu=..., vram=...) replaces the resources for one call. If that call omits vram, that task definition omits vram.
Key Features
- Automatic parallelism — Independent tasks run simultaneously without any configuration.
- Scheduling — Workflows can run on a schedule (e.g., daily, hourly) or be triggered manually.
- Local testing — Run workflows locally before deploying by passing
local_run=True. - Instant runs — Trigger a one-off run without waiting for an image build, using your workstation image.
- Fault tolerance — Failed tasks report detailed logs. You can fix and re-run without restarting the entire pipeline.
- Task aliasing — Use
.alias()to give distinct names when the same task function is invoked multiple times.
Building Complex Pipelines
More complex workflows are built using simple function composition:
@task()
def get_number():
return 42
@task()
def add(a, b):
return a + b
@task()
def reduce_list(*numbers):
return sum(numbers)
@workflow("Complex Pipeline")
def complex_pipeline():
add(get_number().alias("my_number"), 13).alias("first_addition")
total = reduce_list(*[get_number().alias(f"num_{i}") for i in range(5)])
add(total, 1).alias("add_one")
add(total, 2).alias("add_two")
add(total, 3).alias("add_three")
This generates the following graph. Since num_0 through num_4 have no dependencies between them, Datatailr runs them simultaneously — only when all five complete does reduce_list execute:
Scheduling a Workflow
To run a workflow at fixed times, give it a Schedule:
from datatailr import Schedule, workflow
@workflow(
"Weekend Report",
schedule=Schedule(at_minutes=[0], at_hours=[2], weekdays=["Sat", "Sun"]),
)
def weekend_report():
...
Schedule takes these helper fields: at_minutes, every_minute, at_hours, every_hour, weekdays, day_of_month, in_month and every_month. Datatailr compiles them into a cron expression. The example above gives 0 0 2 * * 0,6, which runs at 02:00 every Saturday and Sunday. If you do not set the minutes, the schedule runs at minute 0. If you also do not set the hours, it runs at 00:00. For example, Schedule(weekdays=["Sun"]) runs at 00:00 every Sunday.
You can also give a cron expression directly, for example Schedule(cron_expression="0 0 7 * * 1-5") for 07:00 from Monday to Friday. The expression has six fields: second, minute, hour, day of month, month and day of week. Day of week is 0 (Sunday) to 6 (Saturday). If the expression has five fields, Datatailr adds second 0 in front.
timezone takes an IANA time zone name, for example Schedule(at_hours=[7], timezone="Europe/London"). The workflow stores it and computes the run date (the rundate task argument) in this time zone. The scheduler computes the run times in UTC.
Deploying a Workflow
Run your workflow script to deploy it to Datatailr:
python my_workflow.py
Datatailr packages your code into a container image, builds it, and schedules it for execution. You can monitor the workflow from the dashboard or the CLI:
dt job ls # List all jobs
dt job get my_workflow # View workflow details
dt log read my_workflow # Read workflow logs
Setting the run date of a run
Each task can take the run date as its rundate argument. The default is the current date in the time zone of the workflow. To run the workflow for a different date, for example to fill in a day that you missed, pass run_date when you call the workflow:
my_workflow(run_date="2026-01-31")
run_date takes a datetime.date or a YYYY-MM-DD string. It applies to this run only. If the workflow has a schedule, a call with run_date runs the workflow once now and does not schedule it. Workflow.run() and Workflow.start() take the same argument. An instant run does not accept run_date.
The CLI has the same option for a workflow that is already saved:
dt job start --now --run-date 2026-01-31 my_workflow
The run date has one limit. If the call saves a version that needs a new image, the run starts after the image build and uses the current date, not run_date. This occurs when the code or the image definition of the workflow changed. In that case, wait until the build ends, and then start the run with Workflow.start(run_date=...), the CLI command above or the dashboard.
Instant Runs
An instant run starts a workflow immediately without waiting for a new image to be built. Datatailr reuses the image from your default workstation (or another workstation you specify). This is useful for one-off calculations when you care about results quickly rather than a durable deployed job.
Instant runs also work from a Jupyter notebook or a single Python script — you do not need to put the task functions in a dedicated module.
from datatailr import workflow, task
import time
@task()
def map_data(data: float) -> float:
return data ** 2
@task()
def reduce_data(*data_points: float) -> float:
return sum(data_points)
@workflow()
def sum_of_squares(N=10):
data = [map_data(x) for x in range(N)]
return reduce_data(*data)
result = sum_of_squares(instant_run=True)
status = "running"
while status != "completed":
print("Calculating", end="\r")
time.sleep(2)
status = result.status()
reduce_result = result.result("reduce_data")
map_result_of_four = result.result("map_data-4")
print("\nReady!\n")
print(f"Reduce result: {reduce_result}")
print(f"Map result of 4: {map_result_of_four}")
Calling the workflow with instant_run=True returns an InstantRunHandle. Use it to poll status and fetch per-task logs and cached results (the same primitives as on a normal deployed workflow):
handle = sum_of_squares(instant_run=True)
print(handle.status())
print(handle.task_status(task_name="reduce_data"))
print(handle.logs(task_name="reduce_data"))
print(handle.result(task_name="reduce_data"))
In a Jupyter notebook, if the handle is the last expression in a cell, it renders as a widget with live status, logs, and results for that run.
Managing in the Dashboard
Navigate to Compute Processes and select the Workflows tab to see all your workflows. The view shows the workflow name, number of tasks, current state, schedule, and a visual history of the last 10 runs along with a weekly activity heatmap.
Click on any workflow to view its DAG graph, run history, logs, and definition details.
Right-click a workflow to run it. Run starts a run with the current date as the run date. Run with custom rundate opens a dialog where you select the run date, and then runs the workflow once.
Task Arguments
On datatailr each task is executed in a container but from the perspective of the workflow function, the tasks are regular Python functions. This means that you can pass arguments to the tasks just like you would with any other Python function. The return value of the task is automatically passed to the next task in the workflow as an argument. In addition, the return values are cached and indexed for future use. There are two uses for the cached results: 1. Having the results cached allows you to re-run the workflow from a specific task without having to re-run all the tasks that came before it. 2. The results can be accessed and used in other workflows, services or apps.
Workflow Results Explorer
One particularly useful feature of the workflow results is the ability to explore them in a Jupyter notebook. This is done by using the %dt_runs magic command.
Running this command in a Jupyter notebook will open a widget that allows you to explore the results of the workflow.
%dt_runs # load the widget without a specific workflow selected
%dt_runs my-workflow # load the widget with the specific workflow selected
%dt_runs my-workflow --env prod # load the widget with the specific workflow selected in the prod environment
%dt_runs --env pre # load the widget with the dropdown showing the pre environment
The widget allows you to:
- Select a workflow from the dropdown
- Select an environment from the dropdown
- Select a date and time range to filter the runs
- Rerun a specific run from the run table using the Rerun action
- Confirm rerun options in a popup, including optional
version(integer image version) andfull(full rerun toggle) - View the results of the workflow in a graph format
- View the logs of each task run. By default only your task's own output is shown; tick Show system logs to also see the platform's setup and teardown lines
- View the return value of each task run and load it into a Jupyter notebook cell
Fetching Run Logs and Results Programmatically
Everything the explorer widget shows is also available through the Python SDK, which is useful for scripting, regression checks, or feeding task outputs into another notebook or job.
Start by getting a handle to the deployed workflow:
from datatailr import Environment
from datatailr.scheduler.batch import Workflow
wf = Workflow(name="daily_sales_report", environment=Environment.DEV, get_existing=True)
Listing runs
Workflow.runs(start_time=None, end_time=None, refresh=False) returns the run history of the workflow as a list of dictionaries. Each entry contains the run_id, state, start_time / end_time (as datetime objects), job_version (for a rerun, the version it ran with), and the original_run_id for reruns.
for run in wf.runs():
print(run["run_id"], run["state"], run["start_time"])
Pass start_time and/or end_time as datetime objects to restrict the result to runs whose start_time falls within that window. Either bound can be omitted for an open-ended range:
from datetime import datetime, timedelta, timezone
since = datetime.now(timezone.utc) - timedelta(days=1)
recent = wf.runs(start_time=since)
The first call to runs() is cached on the workflow instance; pass refresh=True to force a fresh fetch (for example, to pick up runs that started after the previous call, or when changing the time window).
To get the per-task breakdown of a single run (state and timing for every task in the DAG), use run_details(run_id):
details = wf.run_details(run_id=29)
for task in details["tasks"]:
print(task["name"], task["state"])
Fetching task logs
Workflow.logs(run_id, task_name, stderr=False) returns the captured stdout (or stderr) of a specific task within a specific run as a string. task_name is the display name of the task — the same one shown in the DAG and in run_details(...)["tasks"].
print(wf.logs(run_id=29, task_name="fetch_data"))
print(wf.logs(run_id=29, task_name="fetch_data", stderr=True))
By default the most recent 5000 lines are returned; pass lines=N to override.
When inspecting a run that was produced by an older workflow version (whose tasks may have been renamed or removed since), pass task_id instead of task_name. The id is exposed by run_details(...)["tasks"][i]["id"] and is resolved as-is, bypassing the currently-loaded definition:
details = wf.run_details(run_id=29)
for task in details["tasks"]:
print(wf.logs(run_id=29, task_id=task["id"]))
Fetching cached task results
Every task's return value is cached on the platform and can be loaded back into Python with Workflow.result(run_id, task_name):
data = wf.result(run_id=29, task_name="fetch_data")
total = wf.result(run_id=29, task_name="calculate_total")
As with logs(...), you can pass task_id (from run_details(...)["tasks"][i]["id"]) to address tasks that no longer exist in the current workflow definition:
details = wf.run_details(run_id=29)
data = wf.result(run_id=29, task_id=details["tasks"][0]["id"])
result(...) returns the original Python object (DataFrame, dict, list, …). It returns None when no result is stored — for example when the task is still running, raised an exception before returning, or its cache entry has expired.
Both logs() and result() raise a ValueError if task_name does not match any task on the workflow. They are also available on non-workflow jobs (App, Service, ExcelAddin); in that case task_name is informational and the call addresses the job itself.