Skip to content

Releases: dagster-io/dagster

1.12.2 (core) / 0.28.2 (libraries)

13 Nov 20:53

Choose a tag to compare

New

  • Dagster has dropped support for Python 3.9 (due to end of life). The minimum supported Python version is now 3.10. This is a breaking change originally intended for the 1.12.0 release.
  • Added dg api schedule {list,get} command.
  • Added new dg plus deploy configure CLI group that generates all the files necessary to get an existing project ready for deployment via Dagster+.
  • [ui] A new sidebar in the Asset catalog (behind a feature flag) provides a hierarchical view of your asset keys.
  • [ui] Improve a few parts of the "Recent updates" timeline section of the Asset overview page.
  • [dagster-census] Added CensusComponent allowing Dagster assets to sync with census connections.
  • [dagster-airbyte] New AirbyteWorkspace configurations: poll_previous_running_sync, max_items_per_page, poll_interval, poll_timeout & cancel_on_termination. (Thanks, @stevenayers and @sonianuj287!)
  • [dagster-dbt] You can now override the op_config_schema property on the DbtProjectComponent to customize how your dbt execution can be configured at runtime.

Dagster Plus

  • [ui] "Edit public catalog views" is now editable in the role permission list.

1.12.1 (core) / 0.28.1 (libraries)

07 Nov 16:53

Choose a tag to compare

New

  • Migrate SqlComponent to Pydantic v2 model_config pattern (Thanks, @LPauzies!)
  • Make dg api commands visible in dg --help output.
  • Make dg plus commands visible in the dg --help output.
  • Add Dremio kind icon. (Thanks, @maxfirman!)
  • The github actions scaffolded by dg scaffold github-actions now include commands to refresh state for StateBackedComponents.
  • Run worker health check will now tag runs with their associated ENI ids.
  • [ui] In asset sidebar, clearly indicate when a freshness policy is a legacy policy.
  • [ui] Cost metrics are now shown on the asset catalog insights page.
  • [dagster] New AutomationConditions for checking asset freshness - dg.AutomationCondition.freshness_passed(), dg.AutomationCondition.freshness_warned() and dg.AutomationCondition.freshness_failed(). (Thanks, @stevenayers!)

Bugfixes

  • [ui] Fix a sporadic race condition when loading jobs in the Dagster UI.
  • [ui] Fixed an issue where deploying multiple serverless code locations simultaneously would sometimes fail with a "the dagster package dependency was expected but not found" error.
  • [ui] Fixed a bug that would cause errors when attempting to supply config to a backfill that targeted assets with checks.
  • [ui] Fixed an issue introduced in dagster 1.11.16 where repositories using custom RepositoryData subclasses would sometimes raise an error when viewing jobs in the Dagster UI.
  • [ui] Fix sensor descriptions in the automation list.
  • [dagster-dask] Fixed an issue where Dask parquet filters configured in YAML would fail with ValueError: too many values to unpack (expected 3) when using Dask version 2022.4.2. (Thanks, @kudryk!)
  • [dagster-dbt] Fixed an issue where dagster-dbt runs targeting large numbers of models could cause the underlying dbt CLI invocation to fail from too many arguments.

Dagster Plus

  • [ui] The insights view in the Observe UI now allows you to specify a custom date range.

1.12.0 (core) / 0.28.0 (libraries)

30 Oct 18:33

Choose a tag to compare

Major changes since 1.11.0 (core) / 0.27.0 (libraries)

UI

  • Refreshed UI: The UI has been redesigned and streamlined to make it easier to find common utilities quickly. Navigation elements have been moved from a header to a collapsible sidebar to help highlight important workflows and provide more visual space throughout the product.

Components

  • Components GA: The Components framework and the dg CLI are now marked as GA (previously Release Candidate). The APIs are fully supported throughout all parts of the product and remain the recommended defaults for new Dagster projects.
  • Standardized Integrations: Integration components have been updated to have execute() and get_asset_spec() methods that can be overridden by subclasses, making it easier and more consistent to customize your components.
  • New Components:
  • State-Backed Components: Added a new StateBackedComponent abstract base class that enables components to persist and manage state separately from their YAML / Python configuration. This is particularly useful for integration components that need to fetch external data. State can be managed locally, in versioned storage, or via code server snapshots. Many integration components (AirbyteWorkspaceComponent, FivetranAccountComponent, PowerBIWorkspaceComponent, AirflowInstanceComponent, and DbtProjectComponent) now extend StateBackedComponent to provide better control over state management. You can check out the docs here!

Simplified Deployment

  • dg scaffold build-artifacts scaffolds Docker and configuration files needed to build and deploy your Dagster project to Dagster Cloud, with support for multiple container registries (ECR, DockerHub, GHCR, ACR, GCR).
  • dg scaffold github-actions generates a complete GitHub Actions CI/CD workflow for deploying to Dagster Cloud, with auto-detection of Serverless vs Hybrid agents and guided setup for required secrets.

Core Orchestration

  • FreshnessPolicies GA: The new FreshnessPolicy API introduced in 1.10.0 has stabilized and is now marked as GA (previously Preview). These supersede the LegacyFreshnessPolicy API (formerly FreshnessPolicy, deprecated in 1.10.0), as well as freshness checks.
    • FreshnessPolicy and apply_freshness_policy are now exported from the top-level dagster module (instead of dagster.preview.freshness ).
    • The build_.*_freshness_checks methods have been marked as superseded . Their functionality will remain unchanged, but we recommend using FreshnessPolicy s for new use cases.
    • The FreshnessDaemon now runs by default, rather than needing to be explicitly enabled via dagster.yaml settings. To turn it off, set:
      freshness:
        enabled: false
  • Configurable Backfills: Run config can now be supplied when launching a backfill, allowing you to specify configuration that will be applied uniformly to all runs.
  • Time-based partition exclusions: All subclasses of TimeWindowPartitionsDefinition now support an exclusions parameter, which allows you to exclude specific dates/times or recurring schedules from your partition set. This is useful for implementing custom calendars that exclude weekends, holidays, or maintenance windows. Exclusions can be specified as cron strings (e.g., "0 0 * * 6" for Saturdays) or datetime objects for specific dates.
  • Execution Dependency Options: All Executors have been updated to accept an optional step_dependency_config parameter with a require_upstream_step_success flag. When set to False via {"step_dependency_config": {"require_upstream_step_success": False}}, downstream steps can start as soon as their required upstream outputs are available, rather than waiting for the entire upstream step to complete successfully. This is particularly useful for large multi-assets where downstream assets only depend on a subset of upstream outputs.

Support & Docs

Changes since 1.11.16 (core) / 0.27.16 (libraries)

New

  • In the Dagster helm chart, images can now be specified by digest (Thanks, @pmartincalvo!)
  • The MultiprocessExecutor, DockerExecutor, and K8sJobExecutor now retry according to their configured retry policy when the step fails during resource initialization.
  • [dagster-aws] Added an optional key_prefix parameter to PipesS3ContextInjector (Thanks, @elipinska!)
  • [dagster-azure] Added a new PipesAzureMLClient.
  • [dagster-pipes] Added support for AzureBlobStorage.
  • [dagster-tableau] Added a new TableauComponent
  • [dagster-looker] Added a LookerComponent
  • [dagster-sigma] Added a new SigmaComponent.
  • [dagster-dbt] The DagsterDbtTranslator no longer has a get_freshness_policy method, nor does it automatically parse the legacy / deprecated LegacyFreshnessPolicy objects from dbt model configuration.
  • [dagster-sling] The DagsterSlingTranslator no longer has a get_freshness_policy method, nor does it automatically parse the legacy / deprecated LegacyFreshnessPolicy objects from sling configuration.

Bugfixes

  • Asset jobs that are unable to resolve their asset selection (for example, due to targeting an empty set of asset keys) will now raise a clearer exception at definition load time explaining which job failed to resolve its asset selection.
  • Fixed an issue where automatic run retries of runs that only targeted asset checks would sometimes fail with a DagsterInvalidConfigError.
  • [ui] Fixed issue causing sensor descriptions to not be displayed in the automation list.
  • [dagster-dbt] Fixed a bug causing enable_code_references to result in errors for dbt assets.

Documentation

  • Corrected several typos and inconsistencies in the helm chart documentation (Thanks, @piggybox!)

1.11.16 (core) / 0.27.16 (libraries)

23 Oct 21:12

Choose a tag to compare

New

  • The proxy GRPC server heartbeat TTL can now be configured with the DAGSTER_GRPC_PROXY_HEARTBEAT_TTL_SECONDS env var (default remains 30 seconds).

Bugfixes

  • Fixed an issue introduced in dagster 1.11.15 where code locations that previously loaded would sometimes fail to load with a gRPC Error code: RESOURCE_EXHAUSTED error.
  • Fixed an issue where defining a repository using a dictionary of job definitions with a key that did not match the name of the job would work when running dagster locally but not when using Dagster+.
  • [components] Fixed a bug that caused errors when using the DbtProjectComponent, FivetranAccountComponent, and similar state-based components in k8s deployments due to a missing StateStorage object in context.
  • [dagster-omni] Added a dependency on python-dateutil to dagster-omni. (Thanks, @bollwyvl!)

1.11.15 (core) / 0.27.15 (libraries)

16 Oct 20:47

Choose a tag to compare

New

  • All sequences are now supported in AssetKey.with_prefix. (Thanks, @aksestok!)
  • [ui] Introduce new navigation, with main navigation items previously in top navigation now in a collapsible left nav.
  • [ui] Improve loading performance of Runs page.
  • [dagster-databricks] Add support for notebook_task in PipesDatabricksClient. (Thanks, @SoerenStahlmann!)

Bugfixes

  • Fixed an issue where fetch_row_counts and fetch_column_metadata do not execute in the same working directory as the underlying dbt command.
  • Fixed a bug with AutomationCondition.execution_failed that would cause it to be evaluated as True for an unpartitioned asset in cases where the latest run failed, but the asset itself materialized successfully before that failure.
  • Unrelated resource keys are now no longer included in the run config schema for subselections of assets.
  • Ignored nodes are properly excluded when generating run config for an implicit asset job
  • Invalid UTF-8 in stderr compute logs are now handled gracefully. (Thanks, @2bxtech!)
  • [ui] Fix top nav rendering for Plus users.
  • [dagster-celery] Fix Celery executor ignoring pools for ops. (Thanks, @kkanter-asml!)
  • [dagster-dbt] Fixed issue that made custom template vars unavailable when specifying them for the cli_args: field of the DbtProjectComponent.

Documentation

Dagster Plus

  • [ui] Fix home page performance for users with large numbers of automations and jobs.
  • [ui] Fix a sporadic JavaScript error that can crash the page when loading insights charts.

1.11.14 (core) / 0.27.14 (libraries)

09 Oct 20:17

Choose a tag to compare

New

  • UnionTypes (e.g. Foo | Bar) are now supported in ConfigurableResources. (Thanks, @DominikHallab!)
  • Added an output_metadata parameter to build_output_context (Thanks, @zyd14!)
  • ResolvedAssetSpec and related resolvers now support setting the freshness_policy field.
  • dagster-dbt project prepare-and-package --components . will no longer attempt to load components outside of DbtProjectComponent, preventing errors when attempting to run this command in environments that do not have the necessary env vars set for other components.
  • click<8.2 upper bound has been removed
  • [dagster-airbyte][dagster-fivetran][dagster-powerbi][dagster-sling][dagster-dlt] The AirbyteWorkspaceComponent, FivetranAccountComponent, PowerBIWorkspaceComponent, SlingReplicationCollectionComponent, and DltLoadCollectionComponent components have been updated to include a get_asset_spec method that can be overridden by subclasses to modify translation behavior.
  • [dagster-airbyte][dagster-fivetran] The AirbyteWorkspaceComponent and FivetranAccountComponent have been updated to include an execute() method that can be overridden by subclasses to modify runtime execution behavior.
  • [dagster-airbyte] The AirbyteWorkspaceComponent no longer binds an "io_manager" or "airbyte" resource, meaning it can be used in tandem with other definitions that use those keys without causing conflicts.
  • [dagster-dbt] The DbtProjectComponent now supports overriding get_asset_spec and get_asset_check_spec methods when subclassing.
  • [dagster-fivetran] The FivetranAccountComponent no longer binds an "io_manager" or "fivetran" resource, meaning it can be used in tandem with other definitions that use those keys without causing conflicts.

Bugfixes

  • Fixed a bug that would cause errors when instantiating a TimeWindowPartitionsDefinition with a monthly schedule and the default day offset.
  • [ui] The Materialize button in the lineage view is now disabled while the view is updating to avoid inconsistencies when launching runs.
  • [ui] Fixed an issue where the "View error" link in the popup that displayed when a backfill failed to launch was very difficult to see.
  • [dagster-dbt] Fixed issue where the select and exclude parameters to @dbt_assets would be ignored when generating AssetCheckSpecs (Thanks, @nathanskone!)
  • [dagster-powerbi] Previously, assets generated for semantic models would have a kind tag with an invalid space character ("semantic model"). Now, they have the kind tag "semantic_model".
  • [dagster-sling] Resolved issue that caused the substring "INF" to be stripped from all logs.

1.11.13 (core) / 0.27.13 (libraries)

02 Oct 18:16

Choose a tag to compare

New

  • [dagster-deltalake,dagster-deltalake-polars] BREAKING CHANGE - we now support deltalake>=1.0.0 for dagster-deltalake and dagster-deltalake-polars and we will no longer support deltalake<1.0.0 moving forward. End user APIs remain the same for both libraries.
  • [dagster-databricks] Spark Python and Python Wheel tasks are now supported in PipesDatabricksServerlessClient.
  • [dagster-dbt] dagster-dbt project prepare-and-package --components . will no longer attempt to load components outside of DbtProjectComponent, preventing errors when attempting to run this command in environments that do not have the necessary env vars set for other components.
  • [dg] adds dg api secret list and dg api secret get

Bugfixes

  • Fixed a bug in the backfill daemon where an asset backfill with CANCELING​ or FAILING​ status could become permanently stuck in CANCELING​ or FAILING​ if the partitions definitions of the assets changed.
  • Fixed an issue introduced in the 1.11.12 release where auto-complete in the Launchpad for nested fields stopped working.
  • Fixed an issue where backfills would fail if a TimeWindowPartitionsDefinition's start date was changed in the middle of the backfill, even if it did not remove any of the targeted partitions.
  • [ui] Fixed the link to "View asset lineage" on runs that don't specify an asset selection.

1.11.12 (core) / 0.27.12 (libraries)

25 Sep 20:06

Choose a tag to compare

New

  • [ui] Allow searching across code locations with * wildcard in selection inputs for jobs and automations.
  • [ui] Added AutomationCondition.all_new_executed_with_tags, which allows automation conditions to be filtered to partitions that have been materialized since the last tick from runs with certain tags. This condition can be used to require or prevent certain run tags from triggering downstream declarative automation conditions.

Bugfixes

  • In dagster==1.11.1, partitioned_config was unintentionally removed from the public exports of the top-level dagster package. This has been fixed.
  • Avoid adding trailing whitespace in env vars that use dot notation in components. Thanks @edgarrmondragon!
  • [dagster-airbyte] Fix the pagination url issue for the Airbyte API. Thanks @stevenayers!
  • [dagster-dbt] Fixed an issue with the DbtCloudWorkspaceClient that would cause errors when calling trigger_job_run with no steps_override parameter.

Dagster Plus

  • [ui] Add Cost insights.
  • [ui] For users who have inherited org roles from a team, show those roles when editing the user.
  • [ui] Fix per-asset and per-job insights graphs.

1.11.11 (core) / 0.27.11 (libraries)

18 Sep 21:22

Choose a tag to compare

New

  • anthropic, mcp, and claude-code-sdk dependencies of dagster-dg-cli are now under a separate ai extra, allowing dagster-dg-cli to be installed without these dependencies.
  • Added AutomationCondition.all_new_updates_have_run_tags and AutomationCondition.any_new_update_has_run_tags, which allows automation conditions to be filtered to partitions that have been materialized since the last tick from runs with certain tags. This condition can be used to require or prevent certain run tags from triggering downstream declarative automation conditions. These conditions are similar to AutomationCondition.executed_with_tags, but look at all new runs since the most recent tick instead of just looking at the latest run.

Bugfixes

  • Fixed a bug which would cause steps downstream of an asset with skippable=True and a blocking asset check to execute as long as the asset check output was produced, even if the asset output was skipped.
  • When a backfill fails, it will now cancel all of its in-progress runs before terminating.
  • Fixed an issue that would cause trailing whitespace to be added to env vars using dot notation ({{ env.FOO }}) when listing the env vars used by a component. (Thanks, @edgarrmondragon!)
  • Fixed issue that would cause errors when using multi to single partition mappings with DbIOManagers.
  • [ui] Fixed issue with the "Report materialization" dialog for non-partitioned assets.
  • [ui] Typing large YAML documents in the launchpad when default config is present is now more performant.
  • [ui] Fixed an issue where setting a FloatMetadataValue to float('inf') or float('-inf') would cause an error when loading that metadata over graphql.
  • [ui] The "Clear" button in the dimension partition text input for multi-partitioned assets now clears invalid selections as expected.
  • [dagster-dbt] Fixed an issue with the DbtCloudWorkspaceClient that would cause errors when calling trigger_job_run with no steps_override parameter.

1.11.10 (core) / 0.27.10 (libraries)

11 Sep 17:21

Choose a tag to compare

New

  • Added inline-component command to the publicly available scaffold commands in the Dagster CLI.
  • Added a new require_upstream_step_success config param to all executors. If {"step_dependency_config": {"require_upstream_step_success": False}} is set, this will allow downstream steps to execute immediately after all required upstream outputs have finished, even if the upstream step has not completed in its entirety yet. This can be useful particularly in cases where there are large multi-assets with downstream assets that depend on only a subset of the assets in the upstream step.
  • The logsForRun​ resolvers and eventConnection​ resolvers in the Dagster GraphQL API will now apply a default limit of 1000 to the number of logs returned from a single graphql query. The cursor​ field in the response can be used to continue iterating through the logs for a given run.
  • [dagster-airbyte] @airbyte_assets and AirbyteWorkspaceComponent (previously AirbyteCloudWorkspaceComponent) now support Airbyte OSS and Enterprise.

Bugfixes

  • Fixed an issue where the dagster_dg_cli package failed to import when using Python 3.9.
  • Fixed an issue with AutomationCondition.eager() that could cause runs for materializable assets to be launched at the same time as an upstream observable source asset that had an automation condition, even if the upstream observation would not result in a new data version.
  • Fixed an issue which could, in some circumstances, cause errors during Declarative Automation evaluation after a dynamic partition was deleted.
  • Fixed an issue that could cause confusing errors when attempting to supply attributes configuration to Component subclasses that did not inherit from Resolvable.
  • Added a Matillion kind tag, thanks @RobBrownFreeAgent!
  • [ui] Fixed an issue where the "Report materialization events" dialog for partitioned assets only worked if the partition was failed or missing.
  • [ui] Fixed a browser crash which could occur in the global asset graph.
  • [ui] Fixed a bug with the sensor preview behavior that would cause run requests contianing run_keys that had already been submitted to show up in the preview result.
  • [dagster-dbt] Fixed an issue that would cause the DbtCloudWorkspace to error before yielding asset events if the associated dbt Cloud run failed. Now, it will raise the error after all relevant asset events have been produced.
  • [dagster-dbt] Added the dbt-core dependency back to dagster-dbt as it is still required for the dbt Cloud integration. If both dbt-core and dbt Fusion are installed, dagster-dbt will still prefer using dbt Fusion by default.

Documentation

  • Introduced a new "Post-processing components" guide.
  • Fixed incorrect YAML code snippets for alert policies docs page.
  • Fixed incorrect chart keys in Helm documentation. Thanks, @charlottevdscheun!
  • Fixed incorrect owner tags in Components docs. Thanks, @aaronprice00!

Dagster Plus

  • Improved the Dagster+ agent's retry behavior during when it experiences outbound connection timeouts while a code location is being deployed.