Skip to content

Environments

Datatailr uses environments to manage the lifecycle of your data workflows and applications. An environment represents a stage in the development and deployment process, such as development, staging, and production. Each environment has completely isolated resources and ensures that changes in one environment do not affect others. This isolation allows you to test and validate your workflows in a safe manner before deploying them to production.

By default, Datatailr provides three standard environments:

  • Development
  • Pre Production (staging)
  • Production

All newly created workflows and applications are deployed to the Development environment by default. Once you have tested and validated your workflows in the Development environment, you can promote them to the Pre Production environment for further testing and validation. Finally, once everything is verified, you can promote your workflows to the Production environment for live use.

All compute processes, including workflows, apps, services and excel addins are deployed as docker containers within the selected environment. This ensures consistency across different stages of the deployment process and allows for easy scaling and management of resources.

Promoting a Compute Process

Promotion copies one version of a compute process to the next environment: from Development to Pre Production, or from Pre Production to Production. Each promotion moves the version one environment up. To go from Development to Production, you promote two times. You cannot promote from Production.

Datatailr builds code only in Development. A promoted version uses a copy of the container image of the version that you promote, so the code in the next environment is the code that you tested.

Before you promote, make sure that:

  • You have the Promote permission on the compute process. See Permissions.
  • The image build of the version succeeded.
  • You deployed the version from a git repository that has no uncommitted changes and is synced with its remote. Datatailr then records the commit of the code. Without a recorded commit, the promotion fails.

Promotion stops the process in the next environment

Start the process again after you promote it. If the process already runs in the next environment, promotion stops it: a running app or service stops, and a scheduled workflow loses its scheduled runs. Promotion does not start the promoted version, so the process stays stopped until you start or enable it there.

In the dashboard

  1. Open Compute Processes and select the tab for the process type: Workflows, Apps, Services or Excel Addins.
  2. Right-click the row of the process in the environment that you promote from, and choose Promote. For a process in Production, Promote is not available.
  3. The dialog shows the current and the next environment. Under Select Version, select the version to promote.
  4. Click Promote, then click Confirm.

With the CLI or the Python SDK

dt job promote takes the job name, the version and the environment to promote from. This command promotes version 5 of my_job from Development to Pre Production:

dt job promote my_job 5 dev

To move the version on to Production, promote it again from Pre Production, for example dt job promote my_job 5 pre. dt job versions my_job lists the versions.

In Python, call promote() with the environment to promote from and the version:

from datatailr import Environment
from datatailr.scheduler import Job

job = Job(name="my_job")
success, message = job.promote(from_environment=Environment.DEV, version=5)

Give both values: the CLI command that promote() calls needs them.