Releases: dagster-io/dagster
Releases · dagster-io/dagster
1.12.2 (core) / 0.28.2 (libraries)
New
- Dagster has dropped support for Python 3.9 (due to end of life). The minimum supported Python version is now 3.10. This is a breaking change originally intended for the 1.12.0 release.
- Added
dg api schedule {list,get}command. - Added new
dg plus deploy configureCLI group that generates all the files necessary to get an existing project ready for deployment via Dagster+. - [ui] A new sidebar in the Asset catalog (behind a feature flag) provides a hierarchical view of your asset keys.
- [ui] Improve a few parts of the "Recent updates" timeline section of the Asset overview page.
- [dagster-census] Added
CensusComponentallowing Dagster assets to sync with census connections. - [dagster-airbyte] New AirbyteWorkspace configurations:
poll_previous_running_sync,max_items_per_page,poll_interval,poll_timeout&cancel_on_termination. (Thanks, @stevenayers and @sonianuj287!) - [dagster-dbt] You can now override the
op_config_schemaproperty on theDbtProjectComponentto customize how your dbt execution can be configured at runtime.
Dagster Plus
- [ui] "Edit public catalog views" is now editable in the role permission list.
1.12.1 (core) / 0.28.1 (libraries)
New
- Migrate SqlComponent to Pydantic v2 model_config pattern (Thanks, @LPauzies!)
- Make
dg apicommands visible indg --helpoutput. - Make
dg pluscommands visible in thedg --helpoutput. - Add Dremio kind icon. (Thanks, @maxfirman!)
- The github actions scaffolded by
dg scaffold github-actionsnow include commands to refresh state forStateBackedComponents. - Run worker health check will now tag runs with their associated ENI ids.
- [ui] In asset sidebar, clearly indicate when a freshness policy is a legacy policy.
- [ui] Cost metrics are now shown on the asset catalog insights page.
- [dagster] New AutomationConditions for checking asset freshness -
dg.AutomationCondition.freshness_passed(),dg.AutomationCondition.freshness_warned()anddg.AutomationCondition.freshness_failed(). (Thanks, @stevenayers!)
Bugfixes
- [ui] Fix a sporadic race condition when loading jobs in the Dagster UI.
- [ui] Fixed an issue where deploying multiple serverless code locations simultaneously would sometimes fail with a "the dagster package dependency was expected but not found" error.
- [ui] Fixed a bug that would cause errors when attempting to supply config to a backfill that targeted assets with checks.
- [ui] Fixed an issue introduced in dagster 1.11.16 where repositories using custom RepositoryData subclasses would sometimes raise an error when viewing jobs in the Dagster UI.
- [ui] Fix sensor descriptions in the automation list.
- [dagster-dask] Fixed an issue where Dask parquet filters configured in YAML would fail with
ValueError: too many values to unpack (expected 3)when using Dask version 2022.4.2. (Thanks, @kudryk!) - [dagster-dbt] Fixed an issue where dagster-dbt runs targeting large numbers of models could cause the underlying
dbtCLI invocation to fail from too many arguments.
Dagster Plus
- [ui] The insights view in the Observe UI now allows you to specify a custom date range.
1.12.0 (core) / 0.28.0 (libraries)
Major changes since 1.11.0 (core) / 0.27.0 (libraries)
UI
- Refreshed UI: The UI has been redesigned and streamlined to make it easier to find common utilities quickly. Navigation elements have been moved from a header to a collapsible sidebar to help highlight important workflows and provide more visual space throughout the product.
Components
- Components GA: The Components framework and the
dgCLI are now marked as GA (previously Release Candidate). The APIs are fully supported throughout all parts of the product and remain the recommended defaults for new Dagster projects. - Standardized Integrations: Integration components have been updated to have
execute()andget_asset_spec()methods that can be overridden by subclasses, making it easier and more consistent to customize your components. - New Components:
- State-Backed Components: Added a new
StateBackedComponentabstract base class that enables components to persist and manage state separately from their YAML / Python configuration. This is particularly useful for integration components that need to fetch external data. State can be managed locally, in versioned storage, or via code server snapshots. Many integration components (AirbyteWorkspaceComponent,FivetranAccountComponent,PowerBIWorkspaceComponent,AirflowInstanceComponent, andDbtProjectComponent) now extendStateBackedComponentto provide better control over state management. You can check out the docs here!
Simplified Deployment
dg scaffold build-artifactsscaffolds Docker and configuration files needed to build and deploy your Dagster project to Dagster Cloud, with support for multiple container registries (ECR, DockerHub, GHCR, ACR, GCR).dg scaffold github-actionsgenerates a complete GitHub Actions CI/CD workflow for deploying to Dagster Cloud, with auto-detection of Serverless vs Hybrid agents and guided setup for required secrets.
Core Orchestration
- FreshnessPolicies GA: The new FreshnessPolicy API introduced in 1.10.0 has stabilized and is now marked as GA (previously Preview). These supersede the
LegacyFreshnessPolicyAPI (formerlyFreshnessPolicy, deprecated in 1.10.0), as well as freshness checks.FreshnessPolicyandapply_freshness_policyare now exported from the top-leveldagstermodule (instead ofdagster.preview.freshness).- The
build_.*_freshness_checksmethods have been marked assuperseded. Their functionality will remain unchanged, but we recommend usingFreshnessPolicys for new use cases. - The
FreshnessDaemonnow runs by default, rather than needing to be explicitly enabled viadagster.yamlsettings. To turn it off, set:freshness: enabled: false
- Configurable Backfills: Run config can now be supplied when launching a backfill, allowing you to specify configuration that will be applied uniformly to all runs.
- Time-based partition exclusions: All subclasses of
TimeWindowPartitionsDefinitionnow support anexclusionsparameter, which allows you to exclude specific dates/times or recurring schedules from your partition set. This is useful for implementing custom calendars that exclude weekends, holidays, or maintenance windows. Exclusions can be specified as cron strings (e.g.,"0 0 * * 6"for Saturdays) or datetime objects for specific dates. - Execution Dependency Options: All
Executors have been updated to accept an optionalstep_dependency_configparameter with arequire_upstream_step_successflag. When set toFalsevia{"step_dependency_config": {"require_upstream_step_success": False}}, downstream steps can start as soon as their required upstream outputs are available, rather than waiting for the entire upstream step to complete successfully. This is particularly useful for large multi-assets where downstream assets only depend on a subset of upstream outputs.
Support & Docs
- We launched a new Support Center with help articles on Dagster+ Hybrid and Serverless troubleshooting, and answers to customer questions previously answered by our support team.
- In our docs:
- We reorganized the Examples section for easier navigation, and added new full pipeline examples on using Dagster with DSpy and PyTorch, as well as targeted mini examples that showcase best practices for:
- We added new guides to help you:
- We added links to Dagster University, our popular Python Primer blog post series, Dagster Deep Dives, customer case studies, Dagster Open Platform, and sales demo example pipelines so you can access all the resources you need to learn about Dagster in one place.
Changes since 1.11.16 (core) / 0.27.16 (libraries)
New
- In the Dagster helm chart, images can now be specified by digest (Thanks, @pmartincalvo!)
- The
MultiprocessExecutor,DockerExecutor, andK8sJobExecutornow retry according to their configured retry policy when the step fails during resource initialization. - [dagster-aws] Added an optional
key_prefixparameter toPipesS3ContextInjector(Thanks, @elipinska!) - [dagster-azure] Added a new
PipesAzureMLClient. - [dagster-pipes] Added support for
AzureBlobStorage. - [dagster-tableau] Added a new
TableauComponent - [dagster-looker] Added a
LookerComponent - [dagster-sigma] Added a new
SigmaComponent. - [dagster-dbt] The
DagsterDbtTranslatorno longer has aget_freshness_policymethod, nor does it automatically parse the legacy / deprecatedLegacyFreshnessPolicyobjects from dbt model configuration. - [dagster-sling] The
DagsterSlingTranslatorno longer has aget_freshness_policymethod, nor does it automatically parse the legacy / deprecatedLegacyFreshnessPolicyobjects from sling configuration.
Bugfixes
- Asset jobs that are unable to resolve their asset selection (for example, due to targeting an empty set of asset keys) will now raise a clearer exception at definition load time explaining which job failed to resolve its asset selection.
- Fixed an issue where automatic run retries of runs that only targeted asset checks would sometimes fail with a
DagsterInvalidConfigError. - [ui] Fixed issue causing sensor descriptions to not be displayed in the automation list.
- [dagster-dbt] Fixed a bug causing
enable_code_referencesto result in errors for dbt assets.
Documentation
- Corrected several typos and inconsistencies in the helm chart documentation (Thanks, @piggybox!)
1.11.16 (core) / 0.27.16 (libraries)
New
- The proxy GRPC server heartbeat TTL can now be configured with the DAGSTER_GRPC_PROXY_HEARTBEAT_TTL_SECONDS env var (default remains 30 seconds).
Bugfixes
- Fixed an issue introduced in dagster 1.11.15 where code locations that previously loaded would sometimes fail to load with a
gRPC Error code: RESOURCE_EXHAUSTEDerror. - Fixed an issue where defining a repository using a dictionary of job definitions with a key that did not match the name of the job would work when running dagster locally but not when using Dagster+.
- [components] Fixed a bug that caused errors when using the
DbtProjectComponent,FivetranAccountComponent, and similar state-based components in k8s deployments due to a missingStateStorageobject in context. - [dagster-omni] Added a dependency on
python-dateutiltodagster-omni. (Thanks, @bollwyvl!)
1.11.15 (core) / 0.27.15 (libraries)
New
- All sequences are now supported in
AssetKey.with_prefix. (Thanks, @aksestok!) - [ui] Introduce new navigation, with main navigation items previously in top navigation now in a collapsible left nav.
- [ui] Improve loading performance of Runs page.
- [dagster-databricks] Add support for
notebook_taskinPipesDatabricksClient. (Thanks, @SoerenStahlmann!)
Bugfixes
- Fixed an issue where
fetch_row_countsandfetch_column_metadatado not execute in the same working directory as the underlying dbt command. - Fixed a bug with
AutomationCondition.execution_failedthat would cause it to be evaluated asTruefor an unpartitioned asset in cases where the latest run failed, but the asset itself materialized successfully before that failure. - Unrelated resource keys are now no longer included in the run config schema for subselections of assets.
- Ignored nodes are properly excluded when generating run config for an implicit asset job
- Invalid UTF-8 in stderr compute logs are now handled gracefully. (Thanks, @2bxtech!)
- [ui] Fix top nav rendering for Plus users.
- [dagster-celery] Fix Celery executor ignoring pools for ops. (Thanks, @kkanter-asml!)
- [dagster-dbt] Fixed issue that made custom template vars unavailable when specifying them for the
cli_args:field of theDbtProjectComponent.
Documentation
- Fixed broken social media link in docs. (Thanks, @MandyMeindersma!)
Dagster Plus
- [ui] Fix home page performance for users with large numbers of automations and jobs.
- [ui] Fix a sporadic JavaScript error that can crash the page when loading insights charts.
1.11.14 (core) / 0.27.14 (libraries)
New
UnionTypes(e.g.Foo | Bar) are now supported inConfigurableResources. (Thanks, @DominikHallab!)- Added an
output_metadataparameter tobuild_output_context(Thanks, @zyd14!) ResolvedAssetSpecand related resolvers now support setting thefreshness_policyfield.dagster-dbt project prepare-and-package --components .will no longer attempt to load components outside ofDbtProjectComponent, preventing errors when attempting to run this command in environments that do not have the necessary env vars set for other components.click<8.2upper bound has been removed- [dagster-airbyte][dagster-fivetran][dagster-powerbi][dagster-sling][dagster-dlt] The
AirbyteWorkspaceComponent,FivetranAccountComponent,PowerBIWorkspaceComponent,SlingReplicationCollectionComponent, andDltLoadCollectionComponentcomponents have been updated to include aget_asset_specmethod that can be overridden by subclasses to modify translation behavior. - [dagster-airbyte][dagster-fivetran] The
AirbyteWorkspaceComponentandFivetranAccountComponenthave been updated to include anexecute()method that can be overridden by subclasses to modify runtime execution behavior. - [dagster-airbyte] The
AirbyteWorkspaceComponentno longer binds an"io_manager"or"airbyte"resource, meaning it can be used in tandem with other definitions that use those keys without causing conflicts. - [dagster-dbt] The
DbtProjectComponentnow supports overridingget_asset_specandget_asset_check_specmethods when subclassing. - [dagster-fivetran] The
FivetranAccountComponentno longer binds an"io_manager"or"fivetran"resource, meaning it can be used in tandem with other definitions that use those keys without causing conflicts.
Bugfixes
- Fixed a bug that would cause errors when instantiating a
TimeWindowPartitionsDefinitionwith a monthly schedule and the default day offset. - [ui] The
Materializebutton in the lineage view is now disabled while the view is updating to avoid inconsistencies when launching runs. - [ui] Fixed an issue where the "View error" link in the popup that displayed when a backfill failed to launch was very difficult to see.
- [dagster-dbt] Fixed issue where the
selectandexcludeparameters to@dbt_assetswould be ignored when generatingAssetCheckSpecs(Thanks, @nathanskone!) - [dagster-powerbi] Previously, assets generated for semantic models would have a kind tag with an invalid space character (
"semantic model"). Now, they have the kind tag"semantic_model". - [dagster-sling] Resolved issue that caused the substring "INF" to be stripped from all logs.
1.11.13 (core) / 0.27.13 (libraries)
New
- [dagster-deltalake,dagster-deltalake-polars] BREAKING CHANGE - we now support
deltalake>=1.0.0fordagster-deltalakeanddagster-deltalake-polarsand we will no longer supportdeltalake<1.0.0moving forward. End user APIs remain the same for both libraries. - [dagster-databricks] Spark Python and Python Wheel tasks are now supported in
PipesDatabricksServerlessClient. - [dagster-dbt]
dagster-dbt project prepare-and-package --components .will no longer attempt to load components outside ofDbtProjectComponent, preventing errors when attempting to run this command in environments that do not have the necessary env vars set for other components. - [dg] adds
dg api secret listanddg api secret get
Bugfixes
- Fixed a bug in the backfill daemon where an asset backfill with CANCELING or FAILING status could become permanently stuck in CANCELING or FAILING if the partitions definitions of the assets changed.
- Fixed an issue introduced in the 1.11.12 release where auto-complete in the Launchpad for nested fields stopped working.
- Fixed an issue where backfills would fail if a TimeWindowPartitionsDefinition's start date was changed in the middle of the backfill, even if it did not remove any of the targeted partitions.
- [ui] Fixed the link to "View asset lineage" on runs that don't specify an asset selection.
1.11.12 (core) / 0.27.12 (libraries)
New
- [ui] Allow searching across code locations with
*wildcard in selection inputs for jobs and automations. - [ui] Added
AutomationCondition.all_new_executed_with_tags, which allows automation conditions to be filtered to partitions that have been materialized since the last tick from runs with certain tags. This condition can be used to require or prevent certain run tags from triggering downstream declarative automation conditions.
Bugfixes
- In
dagster==1.11.1,partitioned_configwas unintentionally removed from the public exports of the top-leveldagsterpackage. This has been fixed. - Avoid adding trailing whitespace in env vars that use dot notation in components. Thanks @edgarrmondragon!
- [dagster-airbyte] Fix the pagination url issue for the Airbyte API. Thanks @stevenayers!
- [dagster-dbt] Fixed an issue with the DbtCloudWorkspaceClient that would cause errors when calling
trigger_job_runwith no steps_override parameter.
Dagster Plus
- [ui] Add Cost insights.
- [ui] For users who have inherited org roles from a team, show those roles when editing the user.
- [ui] Fix per-asset and per-job insights graphs.
1.11.11 (core) / 0.27.11 (libraries)
New
anthropic,mcp, andclaude-code-sdkdependencies ofdagster-dg-cliare now under a separateaiextra, allowingdagster-dg-clito be installed without these dependencies.- Added
AutomationCondition.all_new_updates_have_run_tagsandAutomationCondition.any_new_update_has_run_tags, which allows automation conditions to be filtered to partitions that have been materialized since the last tick from runs with certain tags. This condition can be used to require or prevent certain run tags from triggering downstream declarative automation conditions. These conditions are similar toAutomationCondition.executed_with_tags, but look at all new runs since the most recent tick instead of just looking at the latest run.
Bugfixes
- Fixed a bug which would cause steps downstream of an asset with
skippable=Trueand a blocking asset check to execute as long as the asset check output was produced, even if the asset output was skipped. - When a backfill fails, it will now cancel all of its in-progress runs before terminating.
- Fixed an issue that would cause trailing whitespace to be added to env vars using dot notation (
{{ env.FOO }}) when listing the env vars used by a component. (Thanks, @edgarrmondragon!) - Fixed issue that would cause errors when using multi to single partition mappings with
DbIOManagers. - [ui] Fixed issue with the "Report materialization" dialog for non-partitioned assets.
- [ui] Typing large YAML documents in the launchpad when default config is present is now more performant.
- [ui] Fixed an issue where setting a FloatMetadataValue to float('inf') or float('-inf') would cause an error when loading that metadata over graphql.
- [ui] The "Clear" button in the dimension partition text input for multi-partitioned assets now clears invalid selections as expected.
- [dagster-dbt] Fixed an issue with the
DbtCloudWorkspaceClientthat would cause errors when callingtrigger_job_runwith nosteps_overrideparameter.
1.11.10 (core) / 0.27.10 (libraries)
New
- Added
inline-componentcommand to the publicly available scaffold commands in the Dagster CLI. - Added a new
require_upstream_step_successconfig param to all executors. If{"step_dependency_config": {"require_upstream_step_success": False}}is set, this will allow downstream steps to execute immediately after all required upstream outputs have finished, even if the upstream step has not completed in its entirety yet. This can be useful particularly in cases where there are large multi-assets with downstream assets that depend on only a subset of the assets in the upstream step. - The
logsForRun resolvers andeventConnection resolvers in the Dagster GraphQL API will now apply a default limit of 1000 to the number of logs returned from a single graphql query. Thecursor field in the response can be used to continue iterating through the logs for a given run. - [dagster-airbyte]
@airbyte_assetsandAirbyteWorkspaceComponent(previouslyAirbyteCloudWorkspaceComponent) now support Airbyte OSS and Enterprise.
Bugfixes
- Fixed an issue where the
dagster_dg_clipackage failed to import when using Python 3.9. - Fixed an issue with
AutomationCondition.eager()that could cause runs for materializable assets to be launched at the same time as an upstream observable source asset that had an automation condition, even if the upstream observation would not result in a new data version. - Fixed an issue which could, in some circumstances, cause errors during Declarative Automation evaluation after a dynamic partition was deleted.
- Fixed an issue that could cause confusing errors when attempting to supply
attributesconfiguration toComponentsubclasses that did not inherit fromResolvable. - Added a Matillion kind tag, thanks @RobBrownFreeAgent!
- [ui] Fixed an issue where the "Report materialization events" dialog for partitioned assets only worked if the partition was failed or missing.
- [ui] Fixed a browser crash which could occur in the global asset graph.
- [ui] Fixed a bug with the sensor preview behavior that would cause run requests contianing
run_keys that had already been submitted to show up in the preview result. - [dagster-dbt] Fixed an issue that would cause the DbtCloudWorkspace to error before yielding asset events if the associated dbt Cloud run failed. Now, it will raise the error after all relevant asset events have been produced.
- [dagster-dbt] Added the
dbt-coredependency back todagster-dbtas it is still required for the dbt Cloud integration. If bothdbt-coreanddbt Fusionare installed,dagster-dbtwill still prefer usingdbt Fusionby default.
Documentation
- Introduced a new "Post-processing components" guide.
- Fixed incorrect YAML code snippets for alert policies docs page.
- Fixed incorrect chart keys in Helm documentation. Thanks, @charlottevdscheun!
- Fixed incorrect owner tags in Components docs. Thanks, @aaronprice00!
Dagster Plus
- Improved the Dagster+ agent's retry behavior during when it experiences outbound connection timeouts while a code location is being deployed.