Skip to content

Task

Decorator that turns a function into a workflow task.

Apply this via the public alias @task(). When the decorated function is called inside a workflow-decorated function, the call returns a Task object that becomes a node in the workflow DAG. Dependencies between tasks are inferred automatically from the data flow between calls.

Tasks are only registered for deployment when they are invoked from within the body of a @workflow-decorated function, which builds a Workflow. Calling a @task-decorated function outside that context executes it locally as a normal Python function instead.

See Task below for methods available on the returned object (for example alias() and set_resources()).

Parameters:

  • memory

    (str, default: DEFAULT_TASK_MEMORY ) –

    Memory limit for this task's container (e.g. "256m", "1g").

  • cpu

    (float, default: DEFAULT_TASK_CPU ) –

    Number of CPU cores to allocate for this task (e.g. 1, 0.5).

Returns:

  • Task ( Callable[[Callable[..., Any]], Callable[..., Any]] ) –

    The decorated function. When that function is called inside a @workflow-decorated function, the call returns a Task object that becomes a node in the workflow DAG.

Example 

# tasks.py
from datatailr import task

@task(memory="512m", cpu=2)
def heavy_computation(x, y):
    return x ** y

@task()
def prepare():
    return 2
Then to deploy the workflow, use the following code: 
# deploy.py
from datatailr import workflow
from tasks import heavy_computation, prepare

@workflow("My Workflow")
def my_workflow():
    base = prepare()
    heavy_computation(base, 10)

if __name__ == "__main__":
    my_workflow()  # deploy to Datatailr

Task

Task

Represents a job within a batch job.

This class can be extended to define specific configurations for each job in the batch.

args property writable

Returns the arguments for the Task instance.

id property

Returns the unique identifier of the Task instance.

internal_name property

Returns the internal name of the Task instance.

__call__

Allows the Task instance to be called like a function, returning itself. This is useful for chaining or functional-style programming.

alias

Set an alias for the Task instance.

:param name: The alias name to set.

run

Execute the job's entrypoint.

set_resources

Set the resources for the Task instance.

:param resources: The Resources instance to set.

to_dict

Convert the Task instance to a dictionary representation.

to_json

Convert the Task instance to a JSON string representation.

translate_dependencies

Translate the dependencies of the Task instance into a format suitable for the batch job.

update_env_vars

Update the environment variables for the Task instance.

:param env_vars: A dictionary of environment variables to update.