Skip to content

Compute Process Lifecycle

All compute processes — whether workflows, apps, services, or Excel add-ins — go through the same lifecycle stages from deployment to completion. Understanding these stages helps you troubleshoot issues and know what to expect.

Lifecycle Stages

flowchart LR subgraph Deploy["📦 Deploy"] A["Code uploaded"] --> B["Building"] end subgraph Build["🔨 Build"] B --> C{"Success?"} C -->|No| D["Created
fix & retry"] C -->|Yes| E["Ready"] end subgraph Run["🚀 Run"] E --> F["Pending"] F --> G["Starting"] G --> H["Running"] end subgraph Complete["✅ Complete"] H --> I["Completed"] end

Stage 1: Building

What happens: When you deploy code (using the Python SDK or CLI), Datatailr packages it into a Docker container image. This image includes your code, Python packages, and any system dependencies you specified.

What you see: The process shows status Building.

How long: Typically 1-3 minutes, depending on the number of dependencies.

Stage 2: Ready

What happens: The container image is built and stored in Datatailr's registry. Your compute process is ready to run.

What you see: The process shows status Ready.

For scheduled processes: If your process has a schedule (like "run daily at 9 AM"), it transitions to Scheduled and waits for the next scheduled time.

Stage 3: Pending

What happens: Datatailr is allocating compute resources. If all available machines are busy, Datatailr will automatically start a new compute node.

What you see: The process shows status Pending.

How long: Usually a few seconds. If a new node needs to start, this can take 1-2 minutes.

Stage 4: Starting

What happens: A compute node has been assigned. Datatailr is downloading your container image and preparing to run it.

What you see: The process shows status Starting.

Stage 5: Running

What happens: Your code is executing! For workflows, this is when your tasks run — in parallel where possible.

What you see: The process shows status Running. You can view real-time logs.

Stage 6: Completed

What happens: Your compute process finished successfully. For workflows, all tasks completed. Results are saved and available.

What you see: The process shows status Completed with a green indicator.

When Things Go Wrong

Sometimes compute processes don't complete successfully. Here's what the different failure states mean:

Status What It Means What To Do
Created The build failed (bad dependency, syntax error) Check build logs, fix your code, redeploy
Failed Your code threw an unhandled exception Check logs, fix the bug, re-run
Out of Memory Process exceeded its memory allocation Increase memory in config or optimize your code
Expired A scheduled process's end date has passed Update the schedule or redeploy

Reading logs

You can always check what happened by reading the logs:

  • In the UI: Click on the process and select "Logs"
  • Using CLI: dt log read <process_name>
  • In Python: Use the dt.log.read() function

Rolling Back to a Previous Version

Every deploy creates a new version of a compute process. The current version is the newest one that has not been removed. To roll back, remove the current version and start the process again:

# List versions, e.g. (1, 2, 3)
dt job versions my_process_name

# Check that the previous version still has its image; this must print true
dt registry exists my_process_name dev 2

# Remove the current version; this stops the process
dt job rm my_process_name -v 3

# Start the previous version
dt job start my_process_name

In the UI, open the process, choose Delete, select the current version, confirm, then start the process.

Removing a version does not start the previous one, and it cancels the schedule of a scheduled workflow. Starting the process runs the previous version and restores its schedule.

Image retention

Rolling back needs the previous version's image. If your administrator has enabled image retention for the environment, older images may already be deleted. dt registry exists then prints false, and if you remove the current version anyway, starting the process fails with "No image for job". To bring the process back right away, restore the version you removed (dt job restore my_process_name -v 3) and start it; its image is kept. Then deploy the previous code again instead; in Pre Production and Production, deploy it in Development and promote it. To keep rollbacks possible, ask a Datatailr administrator to keep at least 2 images in that environment.

Monitoring Your Compute Processes

From the Dashboard

The Datatailr dashboard shows all your compute processes with their current status. You can:

  • See which processes are running, pending, or completed
  • View execution history and run times
  • Access logs and output
  • Start, stop, or re-run processes

From the CLI

Use the dt job commands to manage compute processes:

# List all your compute processes
dt job ls

# Get details about a specific process
dt job get my_process_name

# View recent runs of a process
dt job runs my_process_name

# Read logs from a process
dt log read my_process_name

Next Steps

Now that you understand the lifecycle, you're ready to: