These release notes apply to the Cloud Composer service. Announcements and information about new or updated features, bug fixes and deprecated functionality is published on this page.
You can also:
- Follow Cloud Composer release notes in the Apache Airflow Community workspace on Slack.
- See Cloud Composer version list for more information about Cloud Composer images and support dates.
- Check the Cloud Composer known issues page to learn more about known issues and possible workarounds.
You can see the latest product updates for all of Google Cloud on the Google Cloud page, browse and filter all release notes in the Google Cloud console, or programmatically access release notes in BigQuery.
To get the latest product updates delivered to you, add the URL of this page to your feed reader, or add the feed URL directly.
October 09, 2025
Upgrade checks are now generally available (GA) in Cloud Composer 3 and Cloud Composer 2.
September 30, 2025
The GCE_METADATA_TIMEOUT environment variable is changed to reserved. This
change addresses an issue where setting a low timeout value disrupted the
environment's operations that relied on the metadata server.
DAG UI now correctly generates error messages about malformed serialized DAG.
(Airflow 2.10.5) The apache-airflow-providers-google package was upgraded
to version 17.2.0 in Cloud Composer 2 images and Cloud Composer 3 builds.
For more information about changes, see the apache-airflow-providers-google changelog from version 17.1.0 to version 17.2.0.
(Airflow 2.10.5) CloudComposerDAGRunSensor is broken in the
apache-airflow-providers-google package version 17.2.0. This package is used
by Cloud Composer versions and builds with Airflow 2.10.5 available in this
release. If your DAGs use this sensor, we recommend you postpone upgrading
until the issue is resolved.
(Airflow 2.10.5) The apache-airflow-providers-cncf-kubernetes package was
upgraded to version 10.8.0
from version 10.7.0. For changes in other packages, see the
preinstalled packages changelog.
New Airflow builds are available in Cloud Composer 3:
New images are available in Cloud Composer 2:
The following Cloud Composer versions and builds have reached their end of support period: composer-2.9.4-*, and composer-3-airflow-2.9.1 builds from build.0 to build.7.
September 11, 2025
New Airflow builds are available in Cloud Composer 3:
New images are available in Cloud Composer 2:
Fixed an issue where values of Airflow configuration options were evaluated before being set. As a result, the actual value was set to the evaluated result.
Fixed Airflow logs not exporting to Cloud Logging because of a GKE version mismatch between Airflow worker and GKE Control plane nodes.
September 03, 2025
(Cloud Composer 2) Cloud Composer's high availability infrastructure was enhanced to provide greater resilience against zonal outages. This change rolls out gradually over several releases to all regions supported by Cloud Composer 2.
(Available without upgrading) Cloud Composer 3 now supports DNS resolution for regional service endpoints. You can now reach regional service endpoints from DAGs in your environment. This change is available in Public IP environments without additional configuration. For Private IP environments, an environment must be connected to a VPC network where private endpoints are configured.
You can now check if a Cloud Composer 2 environment's configuration is compatible with Cloud Composer 3. We recommend doing this check before migrating to Cloud Composer 3.
New Airflow builds are available in Cloud Composer 3:
New images are available in Cloud Composer 2:
The following Cloud Composer versions and builds have reached their end of support period: composer-2.9.2-* and composer-2.9.3-*.
August 21, 2025
Improved the startup times of Airflow workers for environments that have a large number of custom PyPI packages installed.
This feature was announced previously and has finished gradually rolling out to all regions supported by Cloud Composer.
Airflow UI now shows the correct value of the [core]dags_folder Airflow
configuration option.
(Airflow 2.10.5) The apache-airflow-providers-google package was upgraded to version 17.1.0 in Cloud Composer 2 images and Cloud Composer 3 builds.
For more information about changes, see the apache-airflow-providers-google changelog from version 15.1.0 to version 17.1.0.
(Airflow 2.10.5) The apache-airflow-providers-cncf-kubernetes package was upgraded to version 10.7.0 from version 10.6.1. For changes in other packages, see the preinstalled packages changelog.
New Airflow builds are available in Cloud Composer 3:
New images are available in Cloud Composer 2:
The following Cloud Composer versions and builds have reached their end of support period: composer-2.9.0, composer-2.9.1, and composer-3-airflow-2.7.3 builds from build.5 to build.13.
August 05, 2025
(Cloud Composer 2) Moved the update_fab_perms option from [webserver] to
[fab] in the Airflow configuration. The corresponding deprecation warning is
no longer generated in Airflow web server logs.
(Airflow 2.10.5 only) Added task-level resource consumption Airflow metrics to Cloud Composer.
composer.googleapis.com/workflow/task/cpu_usage: percentage of CPU used by a task.composer.googleapis.com/workflow/task/cpu_usage: percentage of memory used by a task.
New Airflow builds are available in Cloud Composer 3:
New images are available in Cloud Composer 2:
Cloud Composer versions 2.8.7 and 2.8.8 have reached their end of support period.
July 28, 2025
Fixed an issue that caused unexpected restarts of Airflow component workloads in the environment's cluster.
(Cloud Composer 3) The DAGS_FOLDER reserved environment variable now
correctly points to the local directory where DAG files are stored.
New Airflow builds are available in Cloud Composer 3:
New images are available in Cloud Composer 2:
Cloud Composer version 2.8.6 has reached its end of support period.
July 24, 2025
Web server restarting is is now generally available (GA) in Cloud Composer 2 and Cloud Composer 3.
July 23, 2025
If your environment uses dag-factory package version 0.22, then you might
experience DAG failures in Cloud Composer versions that have
apache-airflow-providers-cncf-kubernetes package version 10.4.2 or later. At
the same time, upgrading the dag-factory package to version 0.23 might require
you to update your DAG code to make it compatible.
If your environment uses dag-factory version 0.22, we recommend to do the
following:
- Temporarily postpone upgrading your environment until you're ready to switch
to
dag-factoryversion 0.23. Last versions of Cloud Composer that support version 0.22 are composer-3-airflow-2.10.5-build.3, composer-3-airflow-2.9.3-build.23, composer-2.13.1-airflow-2.10.5, and composer-2.13.1-airflow-2.9.3 released on May 14, 2025. - When you are ready to upgrade, update your DAGs for compatibility with 0.23.
We recommend to do this in a development environment first.
Install
dag-factoryversion 0.23, then check that your DAGs are parsed and are working correctly, and update them if needed. After your DAGs are compatible, installdag-factoryversion 0.23 in your production environment and transfer the updated DAGs. Your environment can now be upgraded to a later version of Cloud Composer or Airflow. - If your environment is already upgraded to a later version of Cloud Composer
and you experience problems, then update
dag-factoryto version 0.23 and update your DAGs for compatibility with 0.23.
July 17, 2025
Highly resilient environments are now generally available (GA) in Cloud Composer 3.
July 14, 2025
We're changing the way we provide support dates for Airflow builds in Cloud Composer 3. Before this change, some Airflow builds had their end of support date listed as "To be announced" until a later Airflow version became available. We're deprecating this approach for all builds that are released after July 01, 2025.
We are now providing support dates that depend on a date when a particular Airflow build was released:
- We are introducing the standard support period of 12 months after the release date.
- All Airflow builds that didn't have an end of support date when this change was introduced are supported until July 1, 2026. Because of this change, some Airflow builds released before July 1, 2025 are supported for longer than the standard support period.
- All Airflow builds that had their support date shorter than 12 months are now supported for 12 months since the date of their initial release.
- All builds released after July 1, 2025 will use the standard support period.
July 09, 2025
We are gradually rolling out a change that switches the default version from Cloud Composer 2 to Cloud Composer 3 in the Cloud Composer API.
In regions where the change is rolled out, a Cloud Composer 3 environment is created by default when a version is not specified in Google Cloud CLI, Cloud Composer API, or Terraform. If you use automation scripts to provision Cloud Composer 2 environments, make sure that you explicitly specify a Cloud Composer 2 version.
In this release, the change is rolling out in the following regions: africa-south1, asia-northeast2, asia-south2, asia-southeast2, europe-southwest1, europe-west10, europe-west12, europe-west8, me-central1, me-central2, me-west1, southamerica-west1, and us-south1.
(Available without upgrading) During Cloud Composer 2 environment operations, a more informative error message is returned when an environment's web server has connectivity issues.
New Airflow builds are available in Cloud Composer 3:
New images are available in Cloud Composer 2:
Cloud Composer versions 2.8.4 and 2.8.5 have reached their end of support period.
July 02, 2025
Cloud Composer 1 and Cloud Composer 2 environments with version 2.0.x are approaching their end of life. We're planning to deprecate them in the following way:
- Starting September 15, 2025, you will no longer be able to create new Cloud Composer 1 environments.
- On September 15, 2026, all Cloud Composer 1 and Cloud Composer 2 version 2.0.x environments will reach their planned end of life, and you won't be able to use them.
Cloud Composer 2 environments with versions later than 2.1.0 and all Cloud Composer 3 environments are not affected by this deprecation.
We recommend planning migration to Cloud Composer 3 or upgrading your Cloud Composer 2 environments to a later version.
July 01, 2025
This release includes internal infrastructure improvements to Cloud Composer. There are no user-visible changes.
New Airflow builds are available in Cloud Composer 3:
New images are available in Cloud Composer 2:
June 23, 2025
Improved the startup times of Airflow workers for environments that have a large number of custom PyPI packages installed.
This feature was announced previously and is gradually rolling out over several releases. In this release, it's available in asia-east2, asia-northeast3, europe-central2, europe-west9, me-central1, me-west1, northamerica-northeast1, northamerica-northeast2, northamerica-south1, and us-west1 re
(Available without upgrading) Fixed an issue where deleting a Cloud Composer 2 environment could fail when the environment's cluster was in the process of creating a node pool.
(Airflow 2.10.5) The apache-airflow-providers-cncf-kubernetes package was upgraded to version 10.5.0 from version 10.4.2. For changes in other packages, see the preinstalled packages changelog.
New Airflow builds are available in Cloud Composer 3:
New images are available in Cloud Composer 2:
Cloud Composer version 2.8.3 has reached its end of support period.
June 16, 2025
We're planning to phase out the APIs that aren't required by Cloud Composer 3.
Starting February 27, 2026, the following APIs will become fully detachable. Deactivating these APIs won't cause the deactivation of the Cloud Composer API:
- artifactregistry.googleapis.com
- cloudbuild.googleapis.com
- container.googleapis.com
- pubsub.googleapis.com
- sqladmin.googleapis.com
Starting May 27, 2026, these APIs will no longer be enabled automatically when you enable the Cloud Composer API. To create Cloud Composer 2 environments in new projects, the group of detached APIs must be enabled manually.
Existing Cloud Composer 3 and Cloud Composer 2 environments in projects where the Cloud Composer API is already enabled will not be impacted. You can do the following:
- After February 27, 2026, if your project has only Cloud Composer 3 environments, then you can manually disable the detached APIs.
- After February 27, 2026, if your project has Cloud Composer 2 environments, then we recommend keeping these APIs enabled because disabling them might lead to environment's malfunction.
- After May 27, 2026, if you use automation scripts to provision Cloud Composer 2 environments, then make sure that the listed APIs are enabled in addition to the Cloud Composer API.
June 13, 2025
Cloud Composer pages in the Cloud Console now support the dark color theme. You can switch to the dark theme on the Appearance page in the Cloud Console.
June 10, 2025
A new Cloud Composer release has started on June 10, 2025. Get ready for upcoming changes and features as we roll out the new release to all regions. This release is in progress at the moment. Listed changes and features might not be available in some regions yet.
Fixed an issue that caused Airflow worker and scheduler Pods to be evicted when a large number of tasks was executed.
New Airflow builds are available in Cloud Composer 3:
New images are available in Cloud Composer 2:
Cloud Composer version 2.8.2 has reached its end of support period.
June 02, 2025
Improved the startup times of Airflow workers for environments that have a large number of custom PyPI packages installed.
This change rolls out gradually. In this release, it's available in asia-east1, asia-northeast2, asia-south1, europe-north1, europe-west3, us-east1, us-south1, and us-west2 regions.
(Cloud Composer 3) If a VPC network is attached to an environment, then all DNS addresses accessed by the Airflow components of the environment are resolved using the Cloud DNS configuration of the VPC network. In particular, Airflow workers that execute DAGs will resolve DNS addresses in this way.
(Cloud Composer 3) All newly created private DNS zones are immediately visible to a Cloud Composer environment. Previously, re-attaching a VPC network was required.
(Cloud Composer 3) It's now possible to use zones with cross-project binding. Before this change, cross-project bound zones weren't supported in Cloud Composer 3.
(Cloud Composer 2) In Cloud Composer versions 2.11.5 and later, log processing is switching to using OpenTelemetry instead of Fluentd.
This change was announced previously and is gradually rolling out over several releases. In this release, it's available in the following regions: asia-east1, asia-east2, asia-northeast2, asia-northeast3, asia-south2, asia-southeast2, australia-southeast1, australia-southeast2, europe-central2, europe-north1, europe-north2, europe-southwest1, europe-west, europe-west10, europe-west12, europe-west6, europe-west8, europe-west9, me-central1, me-central2, me-west1, northamerica-northeast2, northamerica-south1, southamerica-east1, southamerica-west1, us-east5, us-south1, us-west, and us-west3.
New Airflow builds are available in Cloud Composer 3:
New images are available in Cloud Composer 2:
Cloud Composer version 2.8.1 has reached its end of support period.
May 29, 2025
The Composer Local Development CLI tool is now available in Cloud Composer 3. This tool helps to streamline testing and developing by providing local Airflow environments based on Airflow builds used by Cloud Composer 3.
May 26, 2025
(Cloud Composer 3) New metrics that show the number of active Airflow components are now available for Cloud Composer 3 environments:
- The number of active schedulers
- The number of active DAG processors
- The number of active triggerers
- The number of active web servers
(Cloud Composer 3) It's now possible to override the default scopes of access tokens in all regions supported by Cloud Composer 3. This fix was announced previously and has finished gradually rolling out.
(Airflow 2.10.5 and 2.9.3) The apache-airflow-providers-cncf-kubernetes package was upgraded to version 10.4.2 from version 10.1.0.
New Airflow builds are available in Cloud Composer 3:
New images are available in Cloud Composer 2:
May 14, 2025
A new Cloud Composer release has started on May 14, 2025. Get ready for upcoming changes and features as we roll out the new release to all regions. This release is in progress at the moment. Listed changes and features might not be available in some regions yet.
It is now possible to migrate from Cloud Composer 1 to Cloud Composer 3 using snapshots in all regions supported by Cloud Composer 3. The gradual rollout of this feature is finished.
It is now possible to restart the Airflow web server through Cloud Composer API by using the RestartWebServer method.
The default value of the [scheduler]max_tis_per_query Airflow configuration option is set to 128. This change fixes the issue with DAGs remaining in the scheduled state in some cases.
Cloud Composer 3 and Cloud Composer 2 no longer support files with gzip encoding (content-type=gzip).
Before this change, all files that had gzip encoding, including .py and .zip files, were processed inconsistently by the DAG processor. After this change, the DAG processor skips all files with gzip encoding. This change doesn't apply to regular .zip archives. Airflow still decompresses and processes DAG files contained in archives that have no specified encoding.
This change is gradually rolled out to all regions supported by Cloud Composer, except us-central1, us-east4 and europe-west1. It will be rolled out to the remaining regions in one of the future releases.
(Cloud Composer 3) Fixed an issue where the Airflow scheduler failed to initialize and work properly if a custom email backend was enabled. The Airflow scheduler now supports the following email backends:
- airflow.providers.sendgrid.utils.emailer.send_email
- airflow.utils.email.send_email_smtp
This issue affected only the Airflow scheduler. Airflow workers already support these and other backends. If you use an email backend that isn't supported by the scheduler, you can still send email from Airflow workers.
(Cloud Composer 3) Improved the computation of metrics related to snapshots. This change reduces the number of calls made by Cloud Composer to the Cloud Storage API.
(Cloud Composer 2) In Cloud Composer versions 2.11.5 and later, log processing is switching to using OpenTelemetry instead of Fluentd.
This change is gradually rolled out to the asia-south1 and africa-south1 regions. It will be rolled out to more regions in future releases. New and upgraded Cloud Composer 2 environments in the listed regions will get this change.
(Airflow 2.9.3) The apache-airflow-providers-google package was upgraded to version 15.1.0 in Cloud Composer 2 images and Cloud Composer 3 builds.
For more information about changes, see the apache-airflow-providers-google changelog from version 14.0.0 to version 15.1.0.
(Airflow 2.9.3) Changes in preinstalled packages:
apache-airflow-providers-standardwas upgraded to 1.0.0 from 0.4.0.aiosqlitewas removed from preinstalled packages.json-merge-patchwas removed from preinstalled packages.time-machinewas removed from preinstalled packages.pyjwtwas downgraded to 2.9.0 from 2.10.1.
New Airflow builds are available in Cloud Composer 3:
- composer-3-airflow-2.10.5-build.3 (default)
- composer-3-airflow-2.9.3-build.23
New images are available in Cloud Composer 2:
- composer-2.13.1-airflow-2.10.5 (default)
- composer-2.13.1-airflow-2.9.3
Cloud Composer version 2.8.0 has reached its end of support period.
The following Cloud Composer versions have an extended upgrade timeline:
- composer-3-airflow-2.10.5-build.3
- composer-3-airflow-2.9.3-build.23
- composer-2.13.1-airflow-2.10.5
- composer-2.13.1-airflow-2.9.3
May 07, 2025
Data lineage in Cloud Composer now uses OpenLineage in all regions supported by Cloud Composer. For more information about this feature, see the previous announcement.
For newly created Cloud Composer 3 environments, the minimum amount of memory is changed to 2 GB.
For newly created environments, database retention policy is now enabled by default in Google Cloud CLI, API, and Terraform. Before this change, it was enabled by default only in Google Cloud Console.
Improved the environment liveness monitoring. This change addresses some cases of transient failures that caused "Liveness probe failed" warnings in the environment's logs.
(Airflow 2.10.5) The apache-airflow-providers-google package was upgraded to version 15.1.0 in Cloud Composer 2 images and Cloud Composer 3 builds.
For more information about changes, see the apache-airflow-providers-google changelog from version 14.0.0 to version 15.1.0.
(Airflow 2.10.5) Changes in preinstalled packages:
apache-airflow-providers-standardwas upgraded to 1.0.0 from 0.4.0.aiosqlitewas removed from preinstalled packages.json-merge-patchwas removed from preinstalled packages.time-machinewas removed from preinstalled packages.
The default version of Airflow is changed to 2.10.5.
Airflow 2.10.2 is no longer included in Cloud Composer images and builds.
New Airflow builds are available in Cloud Composer 3:
- composer-3-airflow-2.10.5-build.2 (default)
- composer-3-airflow-2.9.3-build.22
New images are available in Cloud Composer 2:
- composer-2.13.0-airflow-2.10.5 (default)
- composer-2.13.0-airflow-2.9.3
Cloud Composer versions 2.7.0 and 2.7.1 have reached their end of support period.
May 06, 2025
The Deployment Manager API is no longer automatically enabled when you enable Cloud Composer API because this API isn't used by the Cloud Composer service.
Environments with Cloud Composer versions 2.0.* still rely on the Deployment Manager API for updates, upgrades, and environment deletion. It won't be possible to perform these operations if this API is disabled. We recommend to upgrade your 2.0.* environments to a later version to remove this dependency.
May 05, 2025
It is now possible to migrate from Cloud Composer 1 to Cloud Composer 3 using snapshots. For more information, see the new migration guide.
This feature will gradually roll out to all regions supported by Cloud Composer 3. At the moment it is available in the africa-south1, asia-south1, me-central1, me-central2, me-west1, southamerica-east1, and southamerica-west1 regions.
April 30, 2025
Starting from 5 May, 2025, new Cloud Composer 3 environments will use 1 CPU and 4 GB of memory for the Airflow web server by default. The minimum and maximum values for these parameters will not change.
April 29, 2025
A script for migrating from Cloud Composer 2 to Cloud Composer 3 is now available on GitHub.
For instructions about migrating with the script, see the new migration guide in Cloud Composer documentation.
April 24, 2025
Starting from June 2025, the default version for new Cloud Composer environments changes from Cloud Composer 2 to Cloud Composer 3. New environments will use the latest default Airflow build (composer-3-airflow-2). Currently, the default version is composer-2-airflow-2.
April 17, 2025
Airflow 2.10.5 is available in Cloud Composer.
Database retention policy is now enabled by default in Google Cloud console and remains disabled in Google Cloud CLI, API, and Terraform.
This feature helps to maintain the Airflow database size. You can enable or disable the database retention policy or adjust the retention period for new and existing environments.
The default environment's service account setting is gradually removed in Cloud Composer. After the change, you'll need to explicitly specify a service account when you create a new Cloud Composer environment. For more information about addressing the change, see the eariler announcement of this change.
In this release, the change is rolling out to the following regions: africa-south1, asia-northeast2, asia-south2, australia-southeast2, europe-north2, europe-southwest1, europe-west8, europe-west10, europe-west12, me-central1, me-central2, me-west1, northamerica-northeast2, northamerica-south1, southamerica-west1, us-east7, and us-south1. It will be rolled out to more regions in future releases.
Cloud Composer 2 environments now always use the environment's service account for performing PyPI packages installations:
- Existing Cloud Composer 2 environments that previously used the default Cloud Build service account now use the environment's service account instead.
- Cloud Composer 2 environments created in versions 2.10.2 and later already have this change.
- Cloud Composer 3 environments already use the environment's service account, and are not affected by this change.
- This change is gradually rolled out to all regions supported by Cloud Composer 2.
Cloud Composer now detects situations when asynchronous tasks are blocked in Airflow triggerers. If a trigger's execution is blocked for more than five minutes, Cloud Composer restarts the triggerer, which solves this transient issue.
(Cloud Composer 3) Key Access Justifications now correctly works for Customer Managed Encryption Keys (CMEK).
The bucket synchronization process doesn't fail if the /plugins folder isn't available in the environment's bucket.
(Cloud Composer 3) It's now possible to override the default scopes of access tokens. Before the fix, the scope always defaulted to https://www.googleapis.com/auth/cloud-platform and https://www.googleapis.com/auth/userinfo.email. This resulted in authentication failures when accessing non-Google Cloud services.
The change is gradually rolled out to the following regions: africa-south1, asia-south2, australia-southeast2, europe-north2, europe-west3, europe-west10, europe-west12, northamerica-south1, southamerica-west1, us-east7, and us-south1. It will be rolled out to more regions in future releases.
New Airflow builds are available in Cloud Composer 3:
- composer-3-airflow-2.10.5-build.0
- composer-3-airflow-2.10.2-build.13 (default)
- composer-3-airflow-2.9.3-build.20
New images are available in Cloud Composer 2:
- composer-2.12.1-airflow-2.10.5
- composer-2.12.1-airflow-2.10.2 (default)
- composer-2.12.1-airflow-2.9.3
Support dates for previous Cloud Composer 3 builds are available. All Cloud Composer 3 builds with Airflow 2.10.2 are supported until April 17, 2026.
April 09, 2025
The Airflow web server in Cloud Composer 3 requires at least 2 GB of memory when an environment is created or updated. This might lead to longer operation times or failures to perform these operations.
As a workaround, when you create a new Cloud Composer 3 environment or upgrade an existing environment, provide at least 2 GB of memory (default value) to the Airflow web server.
April 07, 2025
All Cloud Composer environment's GKE clusters are set up with maintenance exclusions from March 27, 2025 to April 12, 2025. For more information, see Maintenance exclusions.
April 03, 2025
The unification of Cloud Composer 3 billing with BigQuery is paused until further notice. The change was previously scheduled for April 13, 2025.
In recently released Airflow builds of Cloud Composer 3, the Airflow web server requires more CPU to finish its initialization when an environment is created or updated. This might lead to longer operation times or failures to perform these operations.
As a workaround, when you create a new Cloud Composer 3 environment or upgrade an existing environment, provide at least 1 CPU to the Airflow web server.
This issue currently affects composer-3-airflow-2.10.2-build.12 and composer-3-airflow-2.9.3-build.19 Airflow builds.
March 26, 2025
Data lineage in Cloud Composer now uses OpenLineage.
Data lineage support for a specific Airflow operator is now provided by the provider package where the operator is located. See Supported classes in the apache-airflow-providers-openlineage documentation for a list of latest supported operators.
For more information about data lineage in Cloud Composer, see Data lineage with Dataplex.
This feature is gradually rolled out. It will be available in us-west1, us-south1, europe-north1, me-west1, asia-northeast2, asia-southeast2, and africa-south1 regions. We plan to provide this feature in other regions in future releases.
(Available without upgrading) Fixed an issue with updating maintenance windows when there is an upcoming Cloud Composer 3 infrastructure operation.
(Airflow 2.10.2 and 2.9.3) The apache-airflow-providers-google package was upgraded to version 14.0.0 in Cloud Composer 2 images and Cloud Composer 3 builds.
This package is a new major version where many previously deprecated Airflow operators are removed. It is not possible to use these operators in your DAGs.
Make sure that you update your DAGs to use up-to-date alternatives of the removed operators. For more information about removed and deprecated Airflow operators and their up-to-date alternatives, see Deprecated and removed Airflow operators.
For more information about changes, see the apache-airflow-providers-google changelog from version 10.26.0 to version 14.0.0.
(Airflow 2.10.2 and 2.9.3) The apache-airflow-providers-cncf-kubernetes package was upgraded to version 10.3.0 in Cloud Composer 2 images and Cloud Composer 3 builds. For more information about changes, see the apache-airflow-providers-cncf-kubernetes changelog from version 10.1.0 to version 10.3.0.
(Airflow 2.10.2 and 2.9.3) Changes in preinstalled packages:
apache-airflow-providers-postgreswas upgraded to 6.1.0 from 5.14.0.apache-airflow-providers-smtpwas upgraded to 2.0.0 from 1.9.0.types-requestswas removed from preinstalled packages.
New Airflow builds are available in Cloud Composer 3:
- composer-3-airflow-2.10.2-build.12 (default)
- composer-3-airflow-2.9.3-build.19
New images are available in Cloud Composer 2:
- composer-2.12.0-airflow-2.10.2 (default)
- composer-2.12.0-airflow-2.9.3
Cloud Composer versions 2.6.4, 2.6.5, and 2.6.6 have reached their end of support period.
March 18, 2025
After April 15, 2025 the database retention policy feature will be enabled by default in newly created Cloud Composer 3 environments.
This feature helps to maintain the Airflow database size. You can enable or disable the database retention policy or adjust the retention period for new and existing environments.
The issue with Cloud Composer 2 upgrade operations is now resolved. The upgrade operations are unblocked in all regions.
March 15, 2025
The Custom constraints with Organization Policy feature is now generally available (GA).
This feature provides more granular control over Cloud Composer environment configuration fields. You can use custom organization policies to allow or deny specific configuration values for Cloud Composer environments.
March 14, 2025
March 18, 2025 update: The issue is resolved.
Some upgrade operations for Cloud Composer 2 might lead to unhealthy environments. That is why upgrades for Cloud Composer 2 versions will be blocked until the issue is fully resolved.
March 11, 2025
(Cloud Composer 3) Validation error messages now use the correct format for image versions.
New Airflow builds are available in Cloud Composer 3:
- composer-3-airflow-2.10.2-build.11 (default)
- composer-3-airflow-2.9.3-build.18
New images are available in Cloud Composer 2:
- composer-2.11.5-airflow-2.10.2 (default)
- composer-2.11.5-airflow-2.9.3
March 05, 2025
You can now specify an order in which Airflow searches for secrets by overriding the
[secrets]backends_order Airflow configuration option.
Fixed an issue in Cloud Composer REST API that allowed some environment.patch operations to succeed when multiple update masks that aren't related to each other were passed in a request. Now operations with such masks fail with an error.
New Airflow builds are available in Cloud Composer 3:
- composer-3-airflow-2.10.2-build.10 (default)
- composer-3-airflow-2.9.3-build.17
New images are available in Cloud Composer 2:
- composer-2.11.4-airflow-2.10.2 (default)
- composer-2.11.4-airflow-2.9.3
Cloud Composer versions 2.6.2 and 2.6.3 have reached their end of support period.
March 04, 2025
Cloud Composer 3 supports Customer Managed Encryption Keys (CMEK).
February 27, 2025
Database retention policy is available in Cloud Composer 3. You can use this feature to automatically delete older records from the Airflow database, which helps to maintain the Airflow database's size.
February 25, 2025
Cloud Composer 3 is now available in Stockholm (europe-north2). The change is gradually rolling out.
February 21, 2025
Starting April 14, 2025, the Deployment Manager API won't be automatically enabled when you enable Cloud Composer API because this API isn't used by the Cloud Composer service.
February 20, 2025
(Cloud Composer 3) Fixed a problem with configuring access to SMTP servers that don't support user and password authentication. The [smtp]smtp_user and [smtp]smtp_password Airflow configuration options are no longer preconfigured with placeholder values.
Improved the forbidden APIs check. When a new environment is created, Cloud Composer now checks for forbidden APIs that are required by a specific major version of Cloud Composer.
(Airflow 2.10.2 and 2.9.3) Preinstalled packages were changed:
virtualenvwas downgraded from 20.29.1 to 20.28.1
New Airflow builds are available in Cloud Composer 3:
- composer-3-airflow-2.10.2-build.9 (default)
- composer-3-airflow-2.9.3-build.16
New images are available in Cloud Composer 2:
- composer-2.11.3-airflow-2.10.2 (default)
- composer-2.11.3-airflow-2.9.3
Cloud Composer version 2.6.1 has reached its end of support period.
February 13, 2025
(Cloud Composer 3) The GOOGLE_CLOUD_PROJECT environment variable is changed to reserved for Cloud Composer 3. This fixes an issue where creating an environment with this variable failed with a non-specific error.
(Cloud Composer 2) The dependency_permissions_check_count metric now correctly reports the number of missing permissions.
(Cloud Composer 2) The list of permissions reported by dependency_permissions_check_count metric was updated. Now the metric reports more permissions that are required for a functioning environment. Permissions that weren't necessary were replaced or removed.
(New Cloud Composer environments only) Cloud Composer now enforces SSL connections to Cloud SQL instances.
This change implements a security compliance standard in Cloud Composer and enforces SSL usage. You can adopt other security standards by following recommendations listed in Vulnerability findings. Previously, it was possible to create a non-SSL connection to the Airflow database. After the change, the connection will be refused.
(Available without upgrading) Fixed a problem that caused Terraform to unnecessarily force the replacement of cloud_composer_network_ipv4_cidr_block, web_server_ipv4_cidr_block, and cloud_sql_ipv4_cidr_block fields in some configurations of Cloud Composer 2 and Cloud Composer 1 environments.
(Airflow 2.10.2 and 2.9.3) Preinstalled packages were changed:
logbookwas removed from preinstalled packagesminimal-snowplow-trackerwas removed from preinstalled packagesmashumarowas downgraded from 3.15 to 3.14
New Airflow builds are available in Cloud Composer 3:
- composer-3-airflow-2.10.2-build.8 (default)
- composer-3-airflow-2.9.3-build.15
New images are available in Cloud Composer 2:
- composer-2.11.2-airflow-2.10.2 (default)
- composer-2.11.2-airflow-2.9.3
Cloud Composer versions 2.6.0 and 2.6.1 have reached their end of support period.
February 07, 2025
Starting March 2025, the default version for new Cloud Composer environments changes from Cloud Composer 2 to Cloud Composer 3.
New environments that are created without a specified version will use the default Airflow build of Cloud Composer 3, composer-3-airflow-2. Currently, the default version is composer-2-airflow-2.
February 06, 2025
Cloud Composer 2 is no longer available in Mexico (northamerica-south1). It is not possible to select this region when creating a new Cloud Composer 2 environment. Existing Cloud Composer 2 environments and both new and existing Cloud Composer 3 environments are not affected by this change.
February 04, 2025
All Cloud Composer environment's GKE clusters are set up with maintenance exclusions from January 21, 2025 to February 10, 2025. For more information, see Maintenance exclusions.
January 31, 2025
(Cloud Composer 3) The issue with upgrading Airflow builds is now resolved. You can now upgrade Airflow builds for Cloud Composer 3 in the asia-south1, asia-northeast2, europe-west1, europe-west3, and europe-north1 regions.
January 30, 2025
At the beginning of February 2025, we will delete inactive environments that are non-recoverable in Cloud Composer 1, Cloud Composer 2, and Cloud Composer 3. After this change, non-recoverable environments that have listed problems will be deleted automatically.
Cloud Composer 1 and Cloud Composer 2 environments that have both of the following problems present at the same time will be deleted:
- The environment's underlying GKE cluster is deleted.
- The environment is in the ERROR state for at least 60 days because of a disabled billing account or because the Cloud Composer API service was deactivated in its project.
Cloud Composer 3 environments that have the following problem will be deleted:
- The environment is in the ERROR state for at least 60 days because of a disabled billing account or because the Cloud Composer API service was deactivated in its project.
This change doesn't affect buckets of these environments. You can still recover your DAGs and other data from the environment's bucket and then delete the bucket manually. See Delete environments for information about data that is not deleted automatically together with the environment.
January 28, 2025
Java Runtime in Airflow workers and schedulers is updated from version 17 to version 21.
Fixed a problem where synchronization between the environment's bucket and Airflow components failed because incorrect objects in the bucket were not ignored.
(Cloud Composer 3) Fixed an issue with KubernetesPodOperator failing to run properly when do_xcom_push was set to True.
(Available without upgrading) Starting from Cloud Composer version 2.11.0 with Airflow 2.10.2 and 2.9.3, the enable_task_context_logger Airflow configuration option is no longer blocked.
Terminating Airflow workers now generate log messages with information about still running tasks.
Fixed an issue where log messages were incorrectly formatted for non-string content, such as dictionaries and numbers.
(Available without upgrading) In Cloud Composer 2 and Cloud Composer 1, a more meaningful error message is produced when the Composer Service Agent account doesn't have necessary permissions to access the network during the environment creation.
(Available without upgrading) In Cloud Composer 3, fixed an issue where deleting an environment with an attached VPC network failed if the project accept list of the network attachment was cleared manually.
(Cloud Composer 3) The zone in the Environment.config.node_config.location field is no longer populated in the Cloud Composer API.
(Airflow 2.10.2 and 2.9.3) The apache-airflow-providers-cncf-kubernetes package was upgraded to version 10.1.0 in Cloud Composer 2 images and Cloud Composer 3 builds. For more information about changes, see the apache-airflow-providers-cncf-kubernetes changelog from version 10.0.1 to version 10.1.0.
(Airflow 2.10.2 and 2.9.3) The following preinstalled provider packages were upgraded to new major versions:
apache-airflow-providers-apache-beamupgraded from 5.9.1 to 6.0.0apache-airflow-providers-dbt-cloudupgraded from 3.11.2 to 4.0.0apache-airflow-providers-hashicorpupgraded from 3.8.0 to 4.0.0apache-airflow-providers-httpupgraded from 4.13.3 to 5.0.0apache-airflow-providers-mysqlupgraded from 5.7.4 to 6.0.0apache-airflow-providers-sendgridupgraded from 3.6.0 to 4.0.0apache-airflow-providers-sqliteupgraded from 3.9.1 to 4.0.0apache-airflow-providers-sshupgraded from 3.14.0 to 4.0.0
(Airflow 2.10.2 and 2.9.3) The following preinstalled packages were downgraded:
certifidowngraded from 2024.12.14 to 2024.8.30.charset-normalizerdowngraded from 3.4.1 to 3.4.0.cryptographydowngraded from 44.0.0 to 43.0.3.filelockdowngraded from 3.16.1 to 3.15.4.google-authdowngraded from 2.37.0 to 2.35.0.keyringdowngraded from 25.6.0 to 25.5.0.urllib3downgraded from 2.3.0 to 2.2.3.zippdowngraded from 3.21.0 to 3.20.2.
New Airflow builds are available in Cloud Composer 3:
- composer-3-airflow-2.10.2-build.7 (default)
- composer-3-airflow-2.9.3-build.14
New images are available in Cloud Composer 2:
- composer-2.11.1-airflow-2.10.2 (default)
- composer-2.11.1-airflow-2.9.3
Cloud Composer version 2.5.5 has reached its end of support period.
January 27, 2025
January 31, 2025 update: The issue is resolved.
(Cloud Composer 3 only) We are currently experiencing an issue with upgrading Airflow builds for Cloud Composer 3 in asia-south1, asia-northeast2, europe-west1, europe-west3, europe-north1 regions.
The upgrades are temporarily disabled as we continue our work to restore the listed functionalities. We will release an additional announcement after the issue is resolved.
January 24, 2025
The following recently released Cloud Composer 3 Airflow builds and Cloud Composer 2 versions are rolled back and aren't available for creating and upgrading existing environments. We will roll out new builds in the next release.
- composer-2.11.0-airflow-2.10.2
- composer-2.11.0-airflow-2.9.3
- composer-3-airflow-2.10.2-build.6
- composer-3-airflow-2.9.3-build.13
January 15, 2025
Starting April 13, 2025, we are removing the default environment's service account setting. This change enhances security and provides greater control over your Cloud Composer environments.
- Previously, the default Compute Engine service account was used by default when a user didn't specify a service account during Cloud Composer creation.
- After the change, you'll need to explicitly specify a service account when you create a new Cloud Composer environment.
- Existing Cloud Composer environments will not be affected by this change.
To address this change:
- We recommend to create one or more user-managed service accounts for Cloud Composer environments in your project and grant them the minimum of required permissions. For more information and instructions, see Grant roles to an environment's service account.
- If you use Terraform, scripts or other automation and configuration management tools, then make sure to update them, so that an environment's service account is specified when you create an environment.
In April 2025, Cloud Composer 2 environments will always use the environment's service account for performing PyPI packages installations:
- The environment's service account will be used instead.
- Existing Cloud Composer 2 environments that previously used the default Cloud Build service account will change to using the environment's service account instead.
- Cloud Composer 2 environments created in versions 2.10.2 and later already have this change.
- Cloud Composer 3 environments already use the environment's service account, and are not impacted by this change.
January 12, 2025
April 3, 2025 update: This change is postponed.
Starting after April 13, 2025, Cloud Composer 3 will unify its billing with BigQuery. The Cloud Composer 3 standard milli DCU-hours SKU will be replaced with the new BigQuery Engine for Apache Airflow SKU, which will be calculated based on the number of BigQuery slot hours that your Cloud Composer 3 environments consume. Other SKUs will be renamed and moved under the BigQuery hierarchy. The change will be rolled out gradually to all regions supported by Cloud Composer 3. For more information about the change, see Transition to BigQuery slot-hour-based pricing.
January 10, 2025
(Cloud Composer 3) New metrics are available for Cloud Composer 3 environments:
- CPU quota limit for Cloud Composer workloads
- CPU quota usage for Cloud Composer workloads
(Cloud Composer 3) You can now use custom certificates when installing packages from your private repository. This change is gradually rolled out to all Cloud Composer 3 environments. To obtain this change earlier, upgrade the Airflow build of your environment.
The issue with automatic environment upgrades and upgrading Airflow builds in Cloud Composer 3 is resolved and these operations are working. If you think that your environment is still impacted by this issue, please reach out to the Cloud Support team.
(New Cloud Composer 2 environments only) Cloud Composer 2 environments in versions 2.10.2 and later always use the environment's service account for performing PyPI packages installations. This change applies only to newly created environments, existing environments that are upgraded to 2.10.2 and later versions will not get this change.
(Cloud Composer 3 only) The /data folder is now synchronized with Airflow triggerers.
(Available without upgrading) Improved the error message generated when the Cloud Composer Service Agent service account is missing permissions on the project or on the environment's service account.
(Cloud Composer 2) Cloud Composer 2 environments that use PSC interfaces will no longer try to allocate IP ranges for VPC peerings in the tenant project. Environments that use PSC instead of VPC peerings do not use these ranges. This fixes a problem where these ranges overlapped with ranges used for the PSC subnetwork.
The Redis persistent disk is now automatically deleted together with the environment. This persistent disk is used by the Redis queue and stores only technical data.
Fixed an issue where the user-defined Cloud DNS configuration for Google API domains would break Cloud Composer 3 environment creation and attachment of VPC networks.
The worker_autoscale Airflow configuration option is blocked in Cloud Composer 2. Previously, it was blocked only in Cloud Composer 3.
Fixed a problem with the interpolation of pip.conf file. Now the pip.conf file is not interpolated and can contain unescaped % characters.
Removed the warning log message about in-memory storage because it doesn't apply to Cloud Composer.
The importlib-resources package was removed from preinstalled packages.
New Airflow builds are available in Cloud Composer 3:
- composer-3-airflow-2.10.2-build.5 (default)
- composer-3-airflow-2.9.3-build.12
New images are available in Cloud Composer 2:
- composer-2.10.2-airflow-2.10.2 (default)
- composer-2.10.2-airflow-2.9.3
Cloud Composer 2.10.1 is a version with an extended upgrade timeline.
Cloud Composer version 2.5.4 has reached its end of support period.
January 09, 2025
January 10, 2025 update: The issue is resolved.
(Cloud Composer 3 only) We are currently experiencing an issue with automatic environment upgrades and upgrading Airflow builds for Cloud Composer 3 in asia-south1, europe-west1, and asia-northeast2 regions.
The upgrades are temporarily disabled as we continue our work to restore the listed functionalities. We will release an additional announcement after the issue is resolved.
In January 2025, we will delete inactive Cloud Composer 1 environments that are non-recoverable. Environments that have both of the following problems present at the same time will be deleted:
- The environment's underlying GKE cluster is deleted.
- The environment is in the ERROR state for at least 60 days because of a disabled billing account or because the Cloud Composer API service was deactivated in its project.
This change doesn't affect buckets of these environments. You can still recover your DAGs and other data from the environment's bucket and then delete the bucket manually. See Delete environments for information about data that is not deleted automatically together with the environment.
December 16, 2024
Cloud Composer 3 is now generally available (GA):
- All Airflow builds starting from airflow-2.9.3-build.11 and airflow-2.10.2-build.4 are supported at the GA level.
- If your environment uses an earlier Airflow build, then upgrade it to airflow-2.9.3-build.11, airflow-2.10.2-build.4, or a later build to use Cloud Composer 3 on the GA level.
December 11, 2024
In January 2025, we plan to release Cloud Composer 2 versions that will always use the environment's service account for performing PyPI packages installations:
- New Cloud Composer 2 environments created in versions 2.10.2 and later will have this change. This change applies only to newly created environments, existing environments that are upgraded to 2.10.2 and later versions will not get this change.
- Currently, Cloud Composer 2 environments use the default Cloud Build service account if it exists (and the environment's service account if it doesn't). Depending on the way Cloud Build is configured in your project, this might mean that the default Cloud Compute service account or the legacy Cloud Build service account might be used by your environment. We recommend to configure Cloud Build to adhere to the principle of least privilege.
- Make sure to check the Cloud Build default service account change page for information about changes to the default Cloud Build service account.
- Cloud Composer 3 environments already use the environment's service account, and are not impacted by this change.
(Cloud Composer 3) It is now possible to upgrade an environment if the [sentry]sentry_on Airflow configuration option is set to true.
Cloud Composer no longer adds any missing IAM permissions to the Cloud Storage bucket when it is used to create an environment with a custom environment's bucket. Make sure that the environment's service account has permissions from the Composer Worker role on the bucket.
The COMPOSER_AGENT_BUILD_SERVICE_ACCOUNT environment variable is changed to reserved. This change improves the security of Cloud Composer environments.
New Airflow builds are available in Cloud Composer 3:
- composer-3-airflow-2.10.2-build.4 (default)
- composer-3-airflow-2.9.3-build.11
December 05, 2024
New Cloud Composer 3 environments can now be created in VPC SC. This feature is gradually rolled out to all regions supported by Cloud Composer.
Long log entries now have proper task instance annotations.
(Cloud Composer 3) KubernetesPodOperator now works when the do_xcom_push parameter is set to True.
(New Cloud Composer 3 environments) Increased the maximum number of internet connections that each Airflow worker can support at the same time.
Fixed the issue in the environment's component responsible for uploading the logs of Airflow components to Cloud Logging. This bug sometimes lead to a situation where Cloud Composer-level log might be missing for an Airflow component. The same log was still available on the Kubernetes-component level.
(Airflow 2.10.2 and 2.9.3) The apache-airflow-providers-google package was upgraded to version 10.26.0 in Cloud Composer 2 images and Cloud Composer 3 builds. For more information about changes, see the apache-airflow-providers-google changelog from version 10.25.0 to version 10.26.0.
(Airflow 2.10.2 and 2.9.3) The apache-airflow-providers-cncf-kubernetes package was upgraded to version 10.0.1 in Cloud Composer 2 images and Cloud Composer 3 builds. For more information about changes, see the apache-airflow-providers-cncf-kubernetes changelog from version 9.0.1 to version 10.0.1.
(Available without upgrading) Fixed an issue where Airflow workers sometimes generated incomplete or unreadable output.
The default version of Airflow is changed to 2.10.2.
New images are available in Cloud Composer 2:
- composer-2.10.0-airflow-2.10.2 (default)
- composer-2.10.0-airflow-2.9.3
December 04, 2024
Cloud Composer 2 is now available in Mexico (northamerica-south1).
November 27, 2024
In December 2024, Google will remove the following previously deprecated Airflow operators from the apache-airflow-providers-google package.
The new version of this package will be included in one of the future releases of Cloud Composer and the change will be announced in the Release Notes. After this change, it will not be possible to use these operators in your DAGs.
Make sure that you use up-to-date alternatives of the removed operators instead. For more information about removed and deprecated Airflow operators and their up-to-date alternatives, see Deprecated and removed Airflow operators.
Operators that will be removed in December 2024: DataPipelineHook, CreateDataPipelineOperator, RunDataPipelineOperator, AutoMLDatasetLink, AutoMLDatasetListLink, AutoMLModelLink, AutoMLModelTrainLink, AutoMLModelPredictLink.
November 13, 2024
(Airflow 2.7.3) Backported #35887 to fix an issue that occurred during the DST transition. The issue affected DAGs with timezone-aware cron schedule and caused infinite loops in the Airflow scheduler.
(Airflow 2.10.2 and 2.9.3) The apache-airflow-providers-google package was upgraded to version 10.25.0 in Cloud Composer 2 images and Cloud Composer 3 builds. For more information about changes, see the apache-airflow-providers-google changelog from version 10.24.0 to version 10.25.0.
November 06, 2024
(Cloud Composer 3) Fixed an issue that affected the speed of PyPI package installation. PyPI packages are now installed slightly faster.
(Airflow 2.9.3 and 2.7.3) The docutils package was removed from preinstalled packages.
New images are available in Cloud Composer 2:
- composer-2.9.10-airflow-2.9.3 (default)
- composer-2.9.10-airflow-2.7.3
Cloud Composer 2.9.7 is a version with an extended upgrade timeline.
October 30, 2024
New Airflow builds are available in Cloud Composer 3:
- composer-3-airflow-2.9.3-build.5 (default)
- composer-3-airflow-2.7.3-build.21
October 22, 2024
(Airflow 2.9.3) The apache-airflow-providers-google package was upgraded to version 10.24.0 in Cloud Composer 2 images and Cloud Composer 3 builds. For more information about changes, see the apache-airflow-providers-google changelog from version 10.23.0 to version 10.24.0.
(Airflow 2.9.3) The apache-airflow-providers-cncf-kubernetes package was upgraded to version 9.0.0 in Cloud Composer 2 images and Cloud Composer 3 builds. For more information about changes, see the apache-airflow-providers-cncf-kubernetes changelog from version 8.4.2 to version 9.0.0.
(Airflow 2.9.3 and 2.7.3) Changes in preinstalled packages:
- The
grpciopackage was downgraded from 1.66.2 to 1.65.5. - The
js2pypackage was removed from dependencies. - The
pyjsparserpackage was removed from dependencies. - The
tzlocalpackage was removed from dependencies.
Cloud Composer 2.9.8 images are available:
- composer-2.9.8-airflow-2.9.3 (default)
- composer-2.9.8-airflow-2.7.3
October 16, 2024
New Airflow builds are available in Cloud Composer 3:
- composer-3-airflow-2.9.3-build.3 (default)
- composer-3-airflow-2.7.3-build.19
October 01, 2024
(Available without upgrading) Fixed the cause of DAG run failures for runs created from the Cloud Console when the [scheduler]allowed_run_id_pattern Airflow configuration option is set to a custom value.
(Airflow 2.9.3 and 2.9.1) The apache-airflow-providers-cncf-kubernetes package was upgraded to version 8.4.2 in Cloud Composer 2 images and Cloud Composer 3 builds. For more information about changes, see the apache-airflow-providers-cncf-kubernetes changelog from version 8.4.1 to version 8.4.2.
September 18, 2024
Airflow 2.9.3 is available in Cloud Composer images.
(Cloud Composer 2) Fixed the issue where environment create and update operations could fail in rare cases because of the scheduler probe timeouts.
(Cloud Composer 3) Fixed the issue that caused KubernetesPodOperator tasks to fail if they ran for longer than 15 minutes.
New Airflow builds are available in Cloud Composer 3:
- composer-3-airflow-2.9.3-build.0
- composer-3-airflow-2.9.1-build.7 (default)
- composer-3-airflow-2.7.3-build.16
Cloud Composer 2.9.4 images are available:
- composer-2.9.4-airflow-2.9.3
- composer-2.9.4-airflow-2.9.1 (default)
- composer-2.9.4-airflow-2.7.3
Support dates for previous Cloud Composer 3 builds are available. All Cloud Composer 3 builds with Airflow 2.9.1 are supported until September 18, 2025.
Cloud Composer versions 2.4.2 and 2.4.3 have reached their end of support period.
September 02, 2024
Added a new metric: composer.googleapis.com/workflow/task_instance/queued_duration. This metric is based on the dag.<dag_id>.<task_id>.queued_duration Airflow metric.
(Cloud Composer 3) Fixed the cause of false-positive failures of Airflow Celery workers reported by the liveness health check.
Fixed an issue where an upgrade of a PSC-based private IP environment failed leaving the environment in an inconsistent state.
The apache-airflow-providers-cncf-kubernetes package was upgraded to version 8.4.1 in Cloud Composer 2 images and Cloud Composer 3 builds with Airflow 2.9.1. For more information about changes, see the apache-airflow-providers-cncf-kubernetes changelog from version 8.3.4 to version 8.4.1.
The apache-airflow-providers-google package was upgraded to version 10.22.0 in Cloud Composer 2 images and Cloud Composer 3 builds with Airflow 2.9.1. For more information about changes, see the apache-airflow-providers-google changelog from version 10.21.0 to version 10.22.0.
New Airflow builds are available in Cloud Composer 3:
- composer-3-airflow-2.9.1-build.6 (default)
- composer-3-airflow-2.7.3-build.15
Cloud Composer 2.9.3 images are available:
- composer-2.9.3-airflow-2.9.1 (default)
- composer-2.9.3-airflow-2.7.3
August 27, 2024
(Cloud Composer 2) Starting on September 26, 2024, all Cloud Composer environments within a VPC SC perimeter will have no access to public PyPI repositories by default. For details about installing PyPI packages in the VPC SC mode, see Installing PyPI packages.
August 22, 2024
(Cloud Composer 3) Fixed the dags backfill Airflow CLI command that was failing with a connection error.
Fixed a bug where executing Airflow CLI commands produced output that was not human-readable.
The apache-airflow-providers-google package was upgraded to version 10.21.1 in Cloud Composer 2 images with Airflow 2.7.3 and 2.9.1, and in all latest Cloud Composer 3 builds. For more information about changes, see the apache-airflow-providers-google changelog from version 10.21.0 to version 10.21.1.
New Airflow builds are available in Cloud Composer 3:
- composer-3-airflow-2.9.1-build.5
- composer-3-airflow-2.7.3-build.14
Cloud Composer 2.9.2 images are available:
- composer-2.9.2-airflow-2.9.1 (default)
- composer-2.9.2-airflow-2.7.3
Cloud Composer version 2.4.1 has reached its end of support period.
August 13, 2024
(Airflow 2.7.3) Improved the reliability of connections between Airflow components and the Redis component.
New Airflow builds are available in Cloud Composer 3:
- composer-3-airflow-2.9.1-build.4
- composer-3-airflow-2.7.3-build.13
Cloud Composer 2.9.1 images are available:
- composer-2.9.1-airflow-2.9.1 (default)
- composer-2.9.1-airflow-2.7.3
August 08, 2024
The kubelet read-only port in GKE clusters (TCP port 10255) is not used by Cloud Composer. You can define appropriate firewall rules to block external traffic over TCP 10255 in your customer project as described in Modify VPC firewall rules.
Fixed a problem that caused file synchronization between the bucket and Airflow components to be stuck or progress very slowly.
(Airflow 2.9.1) Improved the reliability of connections between Airflow components and the Redis component.
The default version of Airflow is changed to 2.9.1.
New Airflow builds are available in Cloud Composer 3:
- composer-3-airflow-2.9.1-build.3
- composer-3-airflow-2.7.3-build.12
Cloud Composer 2.9.0 images are available:
- composer-2.9.0-airflow-2.9.1 (default)
- composer-2.9.0-airflow-2.7.3
Cloud Composer version 2.4.0 has reached its end of support period.
August 01, 2024
In July and August 2024, new Cloud Composer 2 environments keep switching to using GKE 1.29 and PSC as a connectivity channel to the GKE control plane. Creating new Cloud Composer 2 private IP environments will require using PSC SERVICE_PRODUCER endpoints, which might be blocked by the Disable Private Service Connect for Consumers organization policy constraint. Previously, this org policy limitation could be addressed by using VPC peerings instead of PSC, but starting from GKE 1.29, this workaround is no longer available.
(Cloud Composer 3 only) The July 24th issue with Airflow upgrade operations for Cloud Composer 3 environments is now fixed. Airflow upgrade operations are now being re-enabled, and should reach all Cloud Composer 3 environments within the next couple of days.
The initial synchronization of DAGs and plugins in Airflow workloads is performed in a separate container and utilizes full resources of the Pod.
(Cloud Composer 3) Fixed an error that happened when values in certain formats were used for Airflow configuration options.
New Airflow builds are available in Cloud Composer 3:
- composer-3-airflow-2.9.1-build.2
- composer-3-airflow-2.7.3-build.11
Cloud Composer 2.8.8 images are available:
- composer-2.8.8-airflow-2.9.1
- composer-2.8.8-airflow-2.7.3 (default)
Airflow 2.6.3 is no longer included in Cloud Composer 2 images.
July 25, 2024
Cloud Composer now verifies that the environment's service account is enabled before starting an environment creation operation. As a result, this type of error is reported sooner and the operation does not start.
New Airflow builds are available in Cloud Composer 3:
- composer-3-airflow-2.9.1-build.1
- composer-3-airflow-2.7.3-build.10
Cloud Composer 2.8.7 images are available:
- composer-2.8.7-airflow-2.9.1
- composer-2.8.7-airflow-2.7.3 (default)
- composer-2.8.7-airflow-2.6.3
Cloud Composer version 2.3.5 has reached its end of support period.
July 24, 2024
2024-08-01 Update: This issue has been resolved. For more information, see the release note from August 1st 2024.
(Cloud Composer 3 only) We are currently experiencing an issue with the Airflow upgrade operations for Cloud Composer 3 environments. Upgrading Airflow builds for Cloud Composer 3 is temporarily disabled as we continue our work to restore Airflow upgrade functionalities.
July 18, 2024
Information about excluded Cloud Storage objects in the environment's bucket is no longer logged. This change reduces the usage of the Storage API during the synchronization of DAG files, which improves the performance of Airflow components and results in fewer Airflow component restarts. The change will gradually become available in all Cloud Composer environments.
July 17, 2024
Airflow 2.9.1 is available in Cloud Composer images and builds. We recommend checking the list of changes in Apache Airflow release notes before upgrading to this version.
(Airflow 2.9.1) Task context logging is disabled, and it is not possible to enable it.
(Airflow 2.9.1) Raw HTML code in DAG docs and DAG parameter descriptions is disabled by default.
(Airflow 2.9.1) Audit log permissions are revoked from all roles except Admin.
The apache-airflow-providers-google package was upgraded to version 10.21.0 in Cloud Composer 2 images with Airflow 2.7.3 and 2.9.1, and in all latest Cloud Composer 3 builds. For more information about changes, see the apache-airflow-providers-google changelog from version 10.18.0 to version 10.21.0.
The upgraded apache-airflow-providers-google package provides new operators and sensors for triggering DAGs across Cloud Composer environments:
- CloudComposerRunAirflowCLICommandOperator makes a call to the Airflow REST API of a Cloud Composer environment.
- CloudComposerDAGRunSensor checks if a DAG Run is completed.
When installing PyPI packages, if you want your builds to run with a custom service account, you can override the
COMPOSER_AGENT_BUILD_SERVICE_ACCOUNT environment variable with the chosen service account. For more information, see Install Python dependencies.
New Airflow builds are available in Cloud Composer 3:
- composer-3-airflow-2.9.1-build.0
- composer-3-airflow-2.7.3-build.9
Cloud Composer 2.8.6 images are available:
- composer-2.8.6-airflow-2.9.1
- composer-2.8.6-airflow-2.7.3 (default)
- composer-2.8.6-airflow-2.6.3
Support dates for previous Cloud Composer 3 builds are available. All Cloud Composer 3 builds with Airflow 2.7.3 are supported until July 17, 2025.
Cloud Composer version 2.3.4 has reached its end of support period.
July 15, 2024
It is no longer possible to create Cloud Composer 1 environments in Google Cloud console. It's still possible to create Cloud Composer 1 environments through Google Cloud SDK, Terraform, and API in projects that support creating new Cloud Composer 1 environments.
July 09, 2024
A new Airflow build is available in Cloud Composer 3:
- composer-3-airflow-2.7.3-build.8
Cloud Composer 2.8.5 images are available:
- composer-2.8.5-airflow-2.7.3 (default)
- composer-2.8.5-airflow-2.6.3
July 03, 2024
New Cloud Composer 2 environments are gradually switched to using GKE 1.29 and PSC as a connectivity channel to the GKE control plane. The IP address for the PSC endpoint will be taken from the nodes IP range. This change might require using a larger IP range for the nodes when you create an environment.
July 02, 2024
2024-07-03 Update: Resolution status updated.
The CVE-2024-6387 vulnerability in the OpenSSH package issue was discovered recently. GKE clusters used by Cloud Composer environments are impacted by this vulnerability, and Cloud Composer 1 and 2 environments that use Public IP networking are especially vulnerable to the described issue. For more information about CVE-2024-6387, see Google GKE Security bulletins.
Newly created Composer environments should not be impacted by this issue any more
Composer-owned GKE clusters will be auto-upgraded to newer GKE versions including the fix for CVE-2024-6387. Other components of Composer environments using older versions of COS images will also be upgraded. These operations will be done in an expedited manner so some of the update operations might be done outside environment's regular maintenance windows.
While Google works on resolving this issue so Composer environments are immune to CVE-2024-6387, you can disallow SSH to the Cloud Composer's cluster nodes through establishing proper firewall rules on the environment's cluster as described in the Google GKE Security bulletins. Follow the steps outlined for GKE.
A new Airflow build is available in Cloud Composer 3:
- composer-3-airflow-2.7.3-build.7
Cloud Composer 2.8.4 images are available:
- composer-2.8.4-airflow-2.7.3 (default)
- composer-2.8.4-airflow-2.6.3
Cloud Composer version 2.3.3 has reached its end of support period.
June 25, 2024
Cloud Composer is now available in Johannesburg (africa-south1).
June 20, 2024
We are thrilled to announce the Public Preview launch of the new generation of Cloud Composer, Cloud Composer 3. The new version is now publicly available in all regions supported by Cloud Composer. It comes with a number of new features and characteristics:
- All infrastructure hidden in a tenant project
- Evergreen versioning
- Simplified networking configuration
- Improved performance
- More reliable DAG parsing and scheduling as DAG Processor and Schedulers are now separate components
- 10 times bigger storage for Airflow workers
As well as most functionalities already known from the previous Composer versions. To see the list of features already supported by Composer 3, see Comparison of Cloud Composer versions.
(Airflow 2.7.3) New operators for executing jobs in Google Kubernetes Engine and Kubernetes are available. For example, you can use these operators with Kueue.
Operators for Google Kubernetes Engine:
- GKEStartJobOperator
- GKEStartKueueInsideClusterOperator
- GKEDescribeJobOperator
- GKEListJobsOperator
- GKECreateCustomResourceOperator
- GKEDeleteCustomResourceOperator
- GKEStartKueueJobOperator
- GKEDeleteJobOperator
- GKESuspendJobOperator
- GKEResumeJobOperator
Operators for Kubernetes:
- KubernetesJobOperator
- KubernetesPatchJobOperator
- KubernetesDeleteJobOperator
(Airflow 2.7.3) The apache-airflow-providers-google package was upgraded to version 10.18.0. For more information about changes, see the apache-airflow-providers-google changelog from version 10.17.0 to version 10.18.0.
(Airflow 2.7.3) The apache-airflow-providers-cncf-kubernetes package was upgraded to version 8.3.1.
(Airflow 2.7.3) The apache-beam package was upgraded to version 2.56.0.
A new Airflow build is available in Cloud Composer 3:
- composer-3-airflow-2.7.3-build.6
Cloud Composer 2.8.3 images are available:
- composer-2.8.3-airflow-2.7.3 (default)
- composer-2.8.3-airflow-2.6.3
Cloud Composer versions 2.3.2, 2.3.1, and 2.3.0 have reached their end of full support period.
Cloud Composer 2.8.3 is a version with an extended upgrade timeline.
June 14, 2024
Environment upgrading is now generally available (GA)
June 05, 2024
The google-cloud-bigquery package version was downgraded from 3.23.1 to 3.20.1 because of the #39541 issue in the public version of Airflow.
The dbt-core and dbt-bigquery packages were upgraded to version 1.8.1.
Cloud Composer 2.8.2 images are available:
- composer-2.8.2-airflow-2.7.3 (default)
- composer-2.8.2-airflow-2.6.3
Cloud Composer versions 2.2.1, 2.2.0, and 2.1.15 have reached their end of full support period.
May 27, 2024
Cloud Composer 2 now supports data lineage for environments that have CMEK enabled.
Cloud Composer 2.8.1 images are available:
- composer-2.8.1-airflow-2.7.3 (default)
- composer-2.8.1-airflow-2.6.3
May 11, 2024
Starting from GKE version 1.27.5, Cloud Composer environment clusters will start using SSD disks as persistent disks. The disk quota will change from Persistent disk standard (GB) to Persistent disk SSD (GB). Please check the Persistent disk SSD (GB) quota in your project and request an increase if this quota approaches its limit.
A single environment created using a Small environment preset requires at least 600 GB SSD disk space and the SSD quota must be able to accommodate it.
Being close to the limit of the SSD quota might impact the autoscaling capabilities of Cloud Composer environments or make impossible to create new environments.
The Logs in Cloud Logging Only feature is enabled by default in new environments:
- New Cloud Composer environments now save Airflow task logs only in Cloud Logging by default.
- Existing environments are not changed. If you upgrade an existing environment to Cloud Composer 2.8.0, it keeps saving logs to the environment's bucket.
- You can enable and disable saving logs to the environment's bucket for an existing environment.
Fixed a problem where some Airflow tasks were failing because the task could not write logs to the environment's bucket.
Cloud Composer 2.8.0 images are available:
- composer-2.8.0-airflow-2.7.3 (default)
- composer-2.8.0-airflow-2.6.3
April 29, 2024
The apache-airflow-providers-google package is upgraded to version 10.17.0. For more information about changes, see the apache-airflow-providers-google changelog from version 10.16.0 to version 10.17.0.
The apache-airflow-providers-cncf-kubernetes package was upgraded to version 8.1.0.
Cloud Composer 2.7.1 images are available:
- composer-2.7.1-airflow-2.7.3 (default)
- composer-2.7.1-airflow-2.6.3
Cloud Composer version 2.1.14 has reached its end of full support period.
April 18, 2024
Python 3.11.8 is available in environments with Airflow 2.6.3 and 2.7.3:
Existing environments with Airflow 2.6.3 and 2.7.3 switch to Python 3.11.8 when upgraded.
Before upgrading, make sure that custom PyPI packages in your environment are compatible with Python 3.11.8.
Between April 16, 2024, 2:00 AM (PST) and April 17, 2024, 3:30 AM (PST), Cloud Composer service experienced problems with environment creation, upgrades, and changing the environment size. The problem is resolved and all operations are working. If you think that your environment is still impacted by this issue, please reach out to the Cloud Support team.
(New environments only) Increased the default value of the [webserver]auto_refresh_interval Airflow configuration option to 15 seconds. Pages in the Airflow UI, such as the list of DAGs, now will update every 15 seconds.
(Available without upgrading) Fixed a problem where enabling or disabling Logs in Cloud Logging Only could render the Airflow web server and workers inoperative. If your environment is affected, apply the fix by enabling or disabling this feature again.
Airflow 2.5.3 is no longer included in Cloud Composer images.
The default version of Airflow is changed to 2.7.3.
Cloud Composer 2.7.0 images are available:
- composer-2.7.0-airflow-2.7.3 (default)
- composer-2.7.0-airflow-2.6.3
Cloud Composer versions 2.1.13, 2.1.12 and 1.20.12 have reached their end of full support period.
April 11, 2024
Starting from June 15, 2024 it will not be possible to create Cloud Composer 1 environments in Google Cloud console. It will still be possible to create Cloud Composer 1 environments through Google Cloud SDK, Terraform, and API in allowlisted projects.
March 26, 2024
The Logs in Cloud Logging only feature is available in all regions:
- In newly created Cloud Composer environments, Airflow task logs are now stored only in Cloud Logging by default.
- You can disable this feature for a new or an existing environment. In this case, Airflow task logs are saved both to Cloud Logging and to the environment's bucket.
The default values of the following Airflow configuration options are changed to provide more optimized Cloud Composer environments. This change applies to new environments with Airflow 2.6.3 and 2.7.3 and to existing environments that do not use custom values for these parameters. For existing environments, the new default values will be used after an upgrade.
[scheduler]job_heartbeat_secto 30[scheduler]scheduler_health_check_thresholdto 60[scheduler]scheduler_heartbeat_secto 15
If you want to override the [scheduler]scheduler_heartbeat_sec option's value, then also adjust the [scheduler]scheduler_health_check_threshold option, as described in Cloud Composer documentation.
Cloud Composer 2.6.6 images are available:
- composer-2.6.6-airflow-2.7.3
- composer-2.6.6-airflow-2.6.3 (default)
- composer-2.6.6-airflow-2.5.3
Cloud Composer versions 2.1.11 and 1.20.11 have reached their end of full support period.
March 25, 2024
Cloud Composer 1 entered the post-maintenance mode. Google will not release any further updates to Cloud Composer 1, including new versions of Airflow, bugfixes, and security updates. We recommend planning migration to Cloud Composer 2.
In Cloud Composer versions from 2.1.0 to 2.6.4, task instances that succeeded in the past can be marked as FAILED in some cases. We recommend to upgrade to Cloud Composer version 2.6.5 or later where this issue is fixed. For more information, see the related known issue.
March 20, 2024
The Logs in Cloud Logging only feature is gradually rolled out to all regions:
- New Cloud Composer environments now save Airflow task logs only in Cloud Logging by default.
- Existing environments are not changed. If you upgrade an existing environment, it keeps saving logs to the environment's bucket.
- You can enable and disable saving logs to the environment's bucket for an existing environment.
Currently the feature is rolled out to the following regions: africa-south1, asia-east1, asia-east2, asia-northeast2, asia-south2, asia-southeast2, australia-southeast2, europe-central2, europe-southwest1, europe-west10, europe-west12, europe-west2, europe-west3, europe-west4, europe-west6, europe-west8, me-central1, me-central2, me-west1, northamerica-northeast2, southamerica-west1, us-east5, us-east7, us-south1, and us-west4.
March 19, 2024
BigQueryInsertJobOperator now correctly handles ephemeral tables created with tableDefinitions.
In BigQueryInsertJobOperator, fixed the handling of parsing errors during Lineage emission when the query is too long or deeply nested.
Airflow 2.7.3 is available in Cloud Composer images.
Fixed an issue where past Airflow task instances could be marked as failed in some cases.
Fixed an issue where Airflow task logs for the first try of a task might not be visible in Airflow UI.
The apache-airflow-providers-google package is upgraded to version 10.16.0 in images with Airflow 2.6.3, and images with Airflow 2.7.3 have this version. For more information about changes, see the apache-airflow-providers-google changelog from version 10.15.0 to version 10.16.0.
The apache-airflow-providers-cncf-kubernetes package was upgraded to version 8.0.1 in images with Airflow 2.6.3, and images with Airflow 2.7.3 have this version.
Cloud Composer 2.6.5 images are available:
- composer-2.6.5-airflow-2.7.3
- composer-2.6.5-airflow-2.6.3 (default)
- composer-2.6.5-airflow-2.5.3
Cloud Composer versions 2.1.10, 2.1.9, 1.20.10, and 1.20.9 have reached their end of full support period.
Cloud Composer 2.6.5 is a version with an extended upgrade timeline.
March 14, 2024
In Airflow 2.6.3 offered in Cloud Composer versions earlier than 2.6.2, task statuses can be deleted as a result of the #31179 Airflow issue. If you use Airflow 2.6.3, we recommend to upgrade to Cloud Composer version 2.6.2 or later where this issue is fixed.
March 13, 2024
All preview versions of Cloud Composer 2 are past their security notifications end date and are not supported. If your environment uses a preview version of Cloud Composer 2, then please upgrade this environment to a supported version or re-create it using the latest version of Cloud Composer 2.
If you see frequent scheduler restarts in your Airflow 2.6.3 environment and the [scheduler]job_heartbeat_sec Airflow configuration option is set to a non-default value, you can fix this issue either by upgrading to Cloud Composer version 2.6.4 or by removing this option's override, so that it uses the default value.
March 12, 2024
Fixed creation and upgrades in environments that have environment variables with special symbols.
Cloud Composer 2.6.4 images are available:
- composer-2.6.4-airflow-2.6.3 (default)
- composer-2.6.4-airflow-2.5.3
Cloud Composer versions 2.1.8 and 1.20.8 have reached their end of full support period.
March 05, 2024
Cloud Composer 2.6.3 release was rolled back. If you have an environment that was created with a composer-2.6.3-airflow-* image, you can later upgrade it to a newer version.
February 28, 2024
Cloud Composer 2.6.3 release started on February 28, 2024. Get ready for upcoming changes and features as we roll out the new release to all regions. This release is in progress at the moment. Listed changes and features might not be available in some regions yet
(Cloud Composer 2 in select regions) Cloud Composer shows the account selection and consent screens when opening Airflow UI for an environment.
If you have the "Don't allow users to access any third-party apps" setting enabled in Google Workspace, then configure access to the "Apache Airflow in Cloud Composer" app in Google Workspace, so that your Google Workspace users can still access Airflow UI in Cloud Composer environments. For more information, see Allow access to Airflow UI in Google Workspace.
(Cloud Composer 2 in select regions) Reduced the propagation time of the revoked Cloud IAM permission that blocks access to Airflow UI.
In new environments with Airflow 2.6.3, the default values of the following Airflow configuration options are changed to provide more optimized Cloud Composer environments:
[scheduler]job_heartbeat_secto30[scheduler]scheduler_heartbeat_secto15
Fixed a problem where the IAM policy of a custom environment's bucket is replaced when an environment is created.
The apache-airflow-providers-google package is upgraded to version 10.15.0 in images with Airflow 2.6.3. For more information about changes, see the apache-airflow-providers-google changelog from version 10.14.0 to version 10.15.0.
Cloud Composer 2.6.3 images are available:
- composer-2.6.3-airflow-2.6.3 (default)
- composer-2.6.3-airflow-2.5.3
February 26, 2024
Starting February 27, 2024, in the us-central1, europe-west1, europe-west2, europe-west3, europe-west6, us-east1, and us-east4 regions it is possible to create new Cloud Composer 1 environments only in projects that already have Cloud Composer 1 environments.
In all other existing or newly created projects in these regions, it is possible to create only Cloud Composer 2 environments. This change is a part of the preparation for Cloud Composer 1 end of support, as communicated earlier and described in the Versioning overview.
February 22, 2024
Fixed a problem where one DAG run could potentially delete task instances from other DAG runs if run_id was the same (backported #32684 from a later Airflow version).
Cloud Composer 2.6.2 images are available:
- composer-2.6.2-airflow-2.6.3 (default)
- composer-2.6.2-airflow-2.5.3
February 15, 2024
Starting February 16, 2024, in the asia-east2, asia-northeast1, asia-northeast2, asia-northeast3, asia-south1, and australia-southeast1 regions it is possible to create new Cloud Composer 1 environments only in projects that already have Cloud Composer 1 environments.
In all other existing or newly created projects in these regions, it is possible to create only Cloud Composer 2 environments. This change is a part of the preparation for Cloud Composer 1 end of support, as communicated earlier and described in the Versioning overview.
The apache-airflow-providers-google package is upgraded to version 10.14.0 in images with Airflow 2.6.3. For more information about changes, see the apache-airflow-providers-google changelog from version 10.13.1 to version 10.14.0.
Improved the reliability of syncing Airflow tasks logs to the environment bucket. This fix addresses the issue with storing Airflow task log files, which affected environments in some cases.
Improved the environment component responsible for metrics reporting (composer-monitoring) to minimize the restarts of this component.
Cloud Composer 2.6.1 images are available:
- composer-2.6.1-airflow-2.6.3 (default)
- composer-2.6.1-airflow-2.5.3
Cloud Composer versions 2.1.6 and 1.20.6 have reached their end of full support period.
February 09, 2024
Starting January 20, 2024, in the asia-southeast1, us-west3, and us-west4 regions it is possible to create new Cloud Composer 1 environments only in projects that already have Cloud Composer 1 environments.
In all other existing or newly created projects in these regions, it is possible to create only Cloud Composer 2 environments. This change is a part of the preparation for Cloud Composer 1 end of support, as communicated earlier and described in the Versioning overview.
February 08, 2024
The enabling and disabling functionality for the Logs in Cloud Logging only feature was temporarily rolled back.
At the moment, it is not possible to enable or disable this feature, and your environment will keep its current configuration. If this feature is enabled, Cloud Composer will keep saving logs to Cloud Logging only. Newly created environments save logs to Cloud Logging only and the environment's bucket.
We will announce when the issue is resolved.
February 05, 2024
Data lineage is now generally available (GA) in Cloud Composer 2.
Data lineage integration is now enabled by default in newly created environments with Cloud Composer version 2.1.2 and later, if Data Lineage API is enabled in the environment's project. Existing and upgraded environments keep their current configuration.
Python 3.11 is available in environments with Airflow 2.6.3:
New environments with Airflow 2.6.3 use Python 3.11. Python 3.8 is no longer available in new environments with Airflow 2.6.3.
Existing environments with Airflow 2.6.3 switch to Python 3.11 when upgraded. Before upgrading, make sure that custom PyPI packages in your environment are compatible with Python 3.11.
New and upgraded environments with Airflow 2.5.3 keep using Python 3.8.
Cloud Composer versions earlier than 2.6.0 keep using Python 3.8.
Airflow worker memory requirements in Python 3.11 are 10% higher compared to workers in Python 3.8. If you use custom settings for Airflow worker CPU and memory limits, then Airflow workers in your environment might enter the CrashLoopBackOff status and stop executing tasks, if resource consumption goes above the limit.
If your environment is impacted, see the related known issue for possible solutions: Workers require more memory than in previous Airflow versions.
The default worker_concurrency formula was adjusted in Airflow 2.6.3 and later versions to accommodate this change.
(Available without upgrading) The default worker_concurrency in Airflow 2.6.3 and later versions is now calculated using a different formula. For more information, see Change worker concurrency.
Connections to the Redis environment component are now additionally secured with a password. Improved the reliability of the environment component responsible for metrics reporting (airflow-monitoring).
The apache-airflow-providers-google package is upgraded to version 10.13.1 in images with Airflow 2.6.3. For more information about changes, see the apache-airflow-providers-google changelog from version 10.12.0 to version 10.13.1.
Cloud Composer 2.6.0 images are available:
- composer-2.6.0-airflow-2.6.3 (default)
- composer-2.6.0-airflow-2.5.3
Cloud Composer versions 2.1.5 and 1.20.5 have reached their end of full support period.
Cloud Composer 2.6.0 is a version with an extended upgrade timeline.
January 31, 2024
In the first half of February, 2024, Cloud Composer 2 environments with Airflow 2.6.3 will start using Python 3.11:
- New and upgraded environments with Airflow 2.6.3 will switch to Python 3.11.
- New and upgraded environments with Airflow 2.5.3 will still use Python 3.8.
- Python 3.8 will no longer be available in new versions of Cloud Composer with Airflow 2.6.3 (and later versions of Airflow).
- Existing environments with Airflow 2.6.3 will keep using Python 3.8 until they are upgraded.
- Cloud Composer versions released before this change will keep using Python 3.8.
January 25, 2024
The Logs in Cloud Logging Only feature is now generally available (GA):
- New Cloud Composer environments now save Airflow task logs only in Cloud Logging by default.
- Existing environments are not changed. If you upgrade an existing environment to Cloud Composer 2.5.5, it keeps saving logs to the environment's bucket.
- You can enable and disable saving logs to the environment's bucket for an existing environment.
Increased the amount of memory available to the Redis component in environments with Medium and Large environment sizes. This change improves the performance of Cloud Composer environments.
Sensitive Airflow configuration options are now hidden in Airflow UI by default. The default value of the [webserver]expose_config option is now set to non-sensitive-only.
(Available without upgrading) In VPC Service Controls setups, device policy attributes in access levels are now respected when accessing the Airflow web server.
(Available without upgrading) Auxiliary Cloud Pub/Sub topics and subscriptions used by save and load snapshot operations are now cleaned up if these operations fail.
Cloud Composer versions 2.1.4, 2.1.3, 1.20.4, and 1.20.3 have reached their end of full support period.
January 08, 2024
Environment snapshots now correctly work with the bring your own bucket (BYOB) feature.
(Available without upgrading) Cloud Composer now creates an audit log on a successful completion of create and update environment operations.
If an environment update operation fails, Airflow configuration overrides that were applied by this operation are rolled back.
Cloud Composer 2.5.4 images are available:
- composer-2.5.4-airflow-2.5.3
- composer-2.5.4-airflow-2.6.3 (default)
The apache-airflow-providers-google package is upgraded to version 10.12.0 in images with Airflow 2.6.3 and 2.5.3. For more information about changes, see the apache-airflow-providers-google changelog from version 10.11.1 to version 10.12.0.
Cloud Composer versions 2.1.2 and 1.20.2 have reached their end of full support period.
December 07, 2023
Cloud Composer 2.5.3 images are available:
- composer-2.5.3-airflow-2.5.3
- composer-2.5.3-airflow-2.6.3 (default)
Cloud Composer versions 2.1.1, 2.1.0, 1.20.1, and 1.20.0 have reached their end of full support period.
December 01, 2023
(Available without upgrading) The dags list-import-errors Airflow CLI command is now supported.
Improved the validation of zone values. Environment creation requests do not start the operation if the zone value is not valid.
The apache-airflow-providers-google package is upgraded to version 10.11.1 in images with Airflow 2.6.3 and 2.5.3. For more information about changes, see the apache-airflow-providers-google changelog from version 10.10.1 to version 10.11.1.
The apache-airflow-providers-cncf-kubernetes package was upgraded to version 7.9.0.
Cloud Composer 2.5.2 images are available:
- composer-2.5.2-airflow-2.5.3
- composer-2.5.2-airflow-2.6.3 (default)
Cloud Composer versions 2.0.32 and 1.19.15 have reached their end of full support period.
November 30, 2023
You can specify a preferred Cloud SQL zone for the environment's database when creating a standard resilience environment.
Quotas for snapshot operations are increased. Now it's possible to create up to 52 snapshots daily for a single environment. For scheduled snapshots taken at regular intervals, the minimum recommended interval is now 30 minutes (previously it was two hours).
November 26, 2023
Cloud Composer is now available in Dammam (me-central2).
November 21, 2023
Between January, 2024 and April, 2024 newly created Cloud Composer 2 environments will start using Python 3.11. After this change, Python 3.8 will no longer be available in new versions of Cloud Composer. If you upgrade an existing Cloud Composer 2 environment, the Python version will change to Python 3.11 as well.
The timing for Python 3.11 availability will be announced in January, 2024.
November 17, 2023
Starting December 1, 2023, in the europe-central2, northamerica-northeast1, us-west1, and us-west2 regions it will be possible to create new Cloud Composer 1 environments only in projects that already have Cloud Composer 1 environments.
In all other existing or newly created projects in these regions, it will be possible to create only Cloud Composer 2 environments. This change is a part of the preparation for Cloud Composer 1 end of support, as communicated earlier and described in the Versioning overview.
November 15, 2023
All Cloud Composer environment's GKE clusters are set up with maintenance exclusions for the following periods:
- From November 20, 2023 to November 29, 2023 (already configured)
- From December 20, 2023 to January 2, 2024 (will be configured in December)
For more information, see Maintenance exclusions.
November 06, 2023
The apache-airflow-providers-google package is upgraded to version 10.10.1 in images with Airflow 2.6.3 and 2.5.3. For more information about changes, see the apache-airflow-providers-google changelog from version 10.10.0 to version 10.10.1.
The apache-airflow-providers-cncf-kubernetes package was upgraded to version 7.6.0.
Cloud Composer 2.5.1 images are available:
- composer-2.5.1-airflow-2.5.3
- composer-2.5.1-airflow-2.6.3 (default)
Cloud Composer versions 2.0.31, 2.0.30, 1.19.14, and 1.19.13 have reached their end of full support period.
October 30, 2023
Bring your own bucket feature is now generally available (GA). You can now use a custom Cloud Storage bucket as an environment's bucket.
Directories with names ending in .py are no longer synchronized. If your
environment's bucket contains such directories, please rename them.
Fixed the cause of workers and schedulers failing when Cloud Storage objects with invalid filesystem names are synchronized.
Fixed the validation of the constraints/gcp.restrictServiceUsage Organization Policy constraint. It no longer checks the non-blockable services, such as Cloud Logging and Cloud Monitoring.
The apache-airflow-providers-google package is upgraded to version 10.10.0 in images with Airflow 2.6.3 and 2.5.3. For more information about changes, see the apache-airflow-providers-google changelog from version 10.9.0 to version 10.10.0.
In January 2024, we plan to switch newly created Cloud Composer 2 environments to stop storing task logs in the environment's bucket by default:
- Task logs will be available in Cloud Logging and Airflow UI.
- This change will not be enabled in already existing environments, including environments upgraded to a later version of Cloud Composer that supports this feature.
- It will be possible to enable and disable the synchronization of task logs to the environment's bucket for an existing environment.
This planned change will be announced in the Release Notes when it is rolled out.
The default version of Airflow is changed to 2.6.3.
Airflow 2.4.3 is no longer included in Cloud Composer images.
Cloud Composer 2.5.0 images are available:
- composer-2.5.0-airflow-2.5.3
- composer-2.5.0-airflow-2.6.3 (default)
Cloud Composer versions 2.0.29 and 1.19.12 have reached their end of full support period.
October 16, 2023
Airflow 2.6.3 is available in Cloud Composer images.
Airflow 2.6.3 consolidates the logic for handling tasks that are stuck in the queued state:
- The
[kubernetes]worker_pods_pending_timeout,[celery]stalled_task_timeout, and[celery]task_adoption_timeoutAirflow configuration options are deprecated and merged into the[scheduler]task_queued_timeoutoption. - In Cloud Composer, the default value of the
[scheduler]task_queued_timeoutoption is set to 40 minutes. - If your environment uses a custom value for any of the deprecated Airflow configuration options, please clear the overrides before upgrading. If the values are not cleared, the longest timeout of all deprecated options is selected upon upgrading.
- If required, you can override the value of the
[scheduler]task_queued_timeoutoption in your environment. - For more information about other changes between Airflow versions 2.5.3 and 2.6.3, see Airflow release notes.
Cloud Composer 2.4.6 images are available:
- composer-2.4.6-airflow-2.6.3
- composer-2.4.6-airflow-2.5.3 (default)
- composer-2.4.6-airflow-2.4.3
October 05, 2023
Cloud Composer 2 is now available in Milan (europe-west8), Berlin (europe-west10), and Turin (europe-west12).
Fixed a problem where newly-created Airflow workers ignored the SIGTERM signal, which could lead to task failures.
Cloud Composer 2.4.5 images are available:
- composer-2.4.5-airflow-2.5.3 (default)
- composer-2.4.5-airflow-2.4.3
October 03, 2023
Oozie to Airflow tool version 2.0 is available. The new version of the tool supports Airflow 2.
Oozie to Airflow tool converts Apache Oozie workflows into Apache Airflow DAGs. For more information, see the project's page in PyPI and the oozie-to-airflow repository on GitHub.
September 29, 2023
Airflow triggerer is now generally available (GA).
The cost of the environments.ExecuteAirflowCommand and environments.StopAirflowCommand operations is reduced from 100 to 25 quota units.
The apache-airflow-providers-google package is upgraded to version 10.9.0 in images with Airflow 2.5.3 and 2.4.3. For more information about changes, see the apache-airflow-providers-google changelog from version 10.7.0 to version 10.9.0.
Cloud Composer 2.4.4 images are available:
- composer-2.4.4-airflow-2.5.3 (default)
- composer-2.4.4-airflow-2.4.3
Cloud Composer versions 2.0.28, 2.0.27, 1.19.11, and 1.19.10 have reached their end of full support period.
September 18, 2023
The apache-airflow-providers-google package is upgraded to version 10.7.0 in images with Airflow 2.5.3 and 2.4.3. For more information about changes, see the apache-airflow-providers-google changelog from version 10.5.0 to version 10.7.0.
Version 10.7.0 includes new Cloud Run, Cloud Batch and Dataplex Data Quality operators:
- Cloud Run is used to build and deploy scalable containerized apps written in any language (including Go, Python, Java, Node.js, .NET, and Ruby) on a fully managed platform. The newly provided Cloud Run Operators enable creating, executing, updating, deleting and listing Cloud Run Jobs.
- Cloud Batch is a fully managed batch service to schedule, queue, and execute batch jobs on Google's infrastructure. The provided Cloud Batch Operators enable submitting, listing and deleting batch jobs as well as listing a job's tasks.
- Dataplex is an intelligent data fabric that provides unified analytics and data management across your data lakes, data warehouses, and data marts. New Dataplex operators handle creating, updating, getting, deleting and running a Data Quality scan, getting a Data Quality Scan job, creating and deleting a zone, as well as creating and deleting an asset.
Cloud Composer 2.4.3 images are available:
- composer-2.4.3-airflow-2.5.3 (default)
- composer-2.4.3-airflow-2.4.3
Cloud Composer versions 2.0.26 and 1.19.9 have reached their end of full support period.
September 14, 2023
(Cloud Composer 2 only) In the next two weeks we plan to introduce a 1 vCPU maximum limit for Airflow Triggerers in Cloud Composer. This change is a part of an upcoming adjustment to our triggerer resource allocation strategy where we want to allow for an increase in the maximum triggerer count.
After this change takes effect, if you configured more than 1 vCPU per triggerer, then the next update operation will generate an error. To address this change, adjust the triggerer configuration to meet the new 1 vCPU limit.
(Available without an upgrade) Fixed the issue with reaching the "Write requests per day" quota limit when executing Airflow CLI commands in Cloud Composer 2.
September 12, 2023
Cloud Composer 2 is now available in Doha (me-central1).
September 08, 2023
Fixed the CVE-2023-29247 vulnerability in Airflow versions 2.4.3 and 2.5.3. To apply the fix, please upgrade your environment to Cloud Composer version 2.4.2 or later. For more information about this vulnerability in relation to Cloud Composer environments, see Known issues.
Cloud Composer now correctly generates an error message when a Cloud Composer 1 environment is created using a version alias in a region where Cloud Composer 1 is not supported.
Cloud Composer now generates a FAILED_PRECONDITION error message when an environment operation fails because the environment's cluster is missing.
Cloud Composer 2.4.2 images are available:
- composer-2.4.2-airflow-2.5.3 (default)
- composer-2.4.2-airflow-2.4.3
Cloud Composer versions 2.0.25 and 1.19.8 have reached their end of full support period.
September 07, 2023
Maintenance windows configuration is now generally available (GA).
September 04, 2023
(Cloud Composer 2) An update that enforces VPC Service Controls perimeter settings for accessing Airflow UI and Airflow REST API is rolled out to all regions. As a result of this change, certain users that could previously access the Airflow UI or Airflow REST API might not be able to do it now.
If you experience problems with access to Airflow UI or Airflow REST API, update the perimeter or access level configuration for affected users. For more information on configuring VPC Service Controls see Configure VPC Service Controls.
August 26, 2023
(Cloud Composer versions 1.x.x and 2.0.x): Creating Cloud Composer environments in projects with VPC Service Controls enabled fails if you add Cloud Identity-Aware Proxy API or Identity-Aware Proxy TCP API to the security perimeter.
For more information and possible workarounds, see Environment creation fails in projects with VPC Service Controls enabled.
August 22, 2023
Cloud Composer 2 is now available in Dallas (us-south1), Santiago (southamerica-west1), Melbourne (australia-southeast2), and Madrid (europe-southwest1).
August 16, 2023
Fixed an issue where changing the environment size didn't modify the Redis component size properly.
Fixed an issue where data lineage for BigQueryInsertJobOperator was reported to the incorrect target object in cases where Airflow tasks were running in parallel.
The apache-airflow-providers-google package is upgraded to version 10.5.0 in images with Airflow 2.5.3 and 2.4.3. For more information about changes, see the apache-airflow-providers-google changelog from version 10.3.0 to version 10.5.0.
Cloud Composer 2.4.1 images are available:
- composer-2.4.1-airflow-2.5.3 (default)
- composer-2.4.1-airflow-2.4.3
Cloud Composer versions 2.0.24, 2.0.23, 1.19.7, and 1.19.6 have reached their end of full support period.
August 07, 2023
Airflow CLI commands no longer require access to the control plane of your environment's cluster.
The composer.environments.executeAirflowCommand permission is now required to run Airflow CLI commands through the gcloud environments run command:
The
composer.userandcomposer.environmentAndStorageObjectViewerroles do not have this permission and are not permitted to run Airflow CLI commands.This permission is added to the
composer.adminandcomposer.environmentAndStorageObjectAdminroles.
You can run Airflow CLI commands through Cloud Composer API.
Fixed the cause of failures when creating Qwiklabs environments in some scenarios.
GCSFuse version updated to 1.01. Cloud Composer uses GCSFuse to sync files between the environment bucket and worker pods.
Cloud Composer 2.4.0 images are available:
- composer-2.4.0-airflow-2.5.3 (default)
- composer-2.4.0-airflow-2.4.3
Cloud Composer versions 2.0.22 and 1.19.5 have reached their end of full support period.
July 31, 2023
It is now possible to enable and disable High Resilience mode for an existing environment.
July 25, 2023
The new Composer menu is now available in Airflow UI. You can use it to access environment-related resources (such as DAG files and logs) from your environment's Airflow UI.
Cloud Composer 2 is now available in Paris (europe-west9).
The apache-airflow-providers-google package is upgraded to public version 10.3.0 in images with Airflow 2.5.3 and 2.4.3. Changes compared to version 2023.6.6+composer:
- Google Ads API default version updated to v14.
scipypackage is removed from dependencies.- Several other small fixes and improvements.
For more information about changes, see the apache-airflow-providers-google changelog from version 10.1.1 to version 10.3.0.
(Cloud Composer 2) Airflow web server monitoring is now less sensitive to minor increases in response latency. This change fixes an issue where the web server status was reported as unhealthy for a short period of time.
(Cloud Composer 2) Airflow 2.5.1 is no longer included in Cloud Composer images.
(Cloud Composer 2) The default version of Airflow is changed to 2.5.3.
Cloud Composer 2.3.5 images are available:
- composer-2.3.5-airflow-2.5.3 (default)
- composer-2.3.5-airflow-2.4.3
This version is a version with an extended upgrade timeline.
Cloud Composer versions 2.0.21 and 1.19.4 have reached their end of full support period.
July 12, 2023
Airflow 2.5.3 is available in Cloud Composer images.
Fixed the retrying of transient errors in Composer Agent when creating Cloud Composer 2 environments.
(Available without upgrading) The default amount of memory available to the web server, schedulers, and workers is changed from 3.75 GB per CPU core to 4 GB per CPU core. This value is used only if you do not specify the amount of memory available to these environment components.
(Available without upgrading) Improved the performance of DAG UI in Private IP environments.
Cloud Composer 2.3.4 images are available:
- composer-2.3.4-airflow-2.5.3
- composer-2.3.4-airflow-2.5.1 (default)
- composer-2.3.4-airflow-2.4.3
Cloud Composer versions 2.0.20 and 1.19.3 have reached their end of full support period.
July 04, 2023
Cloud Composer 2 is now available in Tel Aviv (me-west1).
June 29, 2023
A new metric was added to monitor exit_codes of task runners: composer.googleapis.com/workflow/task_runner/terminated_count.
Improved the error message when environment creation fails because of issues with GKE workloads.
(Airflow 2.4.3) Logs produced in Airflow DAG callbacks are now visible in Cloud Logging under the "DAG processor manager" logs section. This feature was previously available only in images with Airflow 2.5.1.
(Airflow 2.4.3) Changed the severity of triggerer watchdog messages from error to warning and updated the message's content to be more informative. This feature was previously available only in images with Airflow 2.5.1.
Cloud Composer 2.3.3 images are available:
- composer-2.3.3-airflow-2.5.1 (default)
- composer-2.3.3-airflow-2.4.3
Cloud Composer versions 2.0.19, 2.0.18, 1.19.2, and 1.19.1 have reached their end of full support period.
June 20, 2023
(Airflow 2.5.1 only) Logs produced in Airflow DAG callbacks are now visible in Cloud Logging in the "DAG processor manager" logs section.
DataprocSubmitJobOperator now supports data lineage for Hive, SparkSQL, Presto, and Trino jobs.
(Airflow 2.5.1) Changed the severity of triggerer watchdog messages from error to warning and updated the message's content to be more informative. This change improves debugging experience for triggers.
Cloud Composer 2.3.2 images are available:
- composer-2.3.2-airflow-2.5.1 (default)
- composer-2.3.2-airflow-2.4.3
June 15, 2023
Cloud Composer 2.3.1 release started on June 15, 2023. Get ready for upcoming changes and features as we roll out the new release to all regions. This release is in progress at the moment. Listed changes and features might not be available in some regions yet.
Cloud Composer 2 is now available in Columbus (us-east5).
Cloud Composer 2.3.1 images are available:
- composer-2.3.1-airflow-2.5.1 (default)
- composer-2.3.1-airflow-2.4.3
Cloud Composer versions 2.0.17 and 1.19.0 have reached their end of full support period.
June 14, 2023
Cloud Composer 2 is now available in Finland (europe-north1), Toronto (northamerica-northeast2), and Delhi (asia-south2).
June 09, 2023
An environment can now run two Airflow triggerers. This feature was previously available only in Highly Resilient environments.
The apache-airflow-providers-google package in images with Airflow 2.5.1 and 2.4.3 was upgraded to version 2023.6.6+composer. This version is based on the public version 10.1.1, with additional fixes to some operators and upgrades to many SDK package dependencies (such as protobuf). Notable changes include:
- Google Ads default API changed from version 12 to 13.
protobuf==4.22.5is included, this is the first Cloud Composer version with protobuf version 4.x.- In this version of the provider package, the deprecated
delegate_toparameter is removed from all GCP operators, hooks, and triggers, as well as from Firestore and Gsuite transfer operators that interact with GCS. Impersonation can be achieved by utilizing theimpersonation_chainparameter instead. Thedelegate_to paramremains available only in Gsuite and marketing platform hooks and operators that don't interact with Google Cloud.
For a full list of changes in the apache-airflow-providers-google, see the changelog from version 8.9.0 to 10.1.1 on the apache-airflow-providers-google page.
(Cloud Composer 2) Fixed an issue where it was not possible to delete a Cloud Composer environment if the environment's service account was already deleted.
The google-cloud-asset package is added to images with Airflow 2.5.1 and 2.4.3.
Cloud Composer 2.3.0 images are available:
- composer-2.3.0-airflow-2.5.1 (default)
- composer-2.3.0-airflow-2.4.3
Cloud Composer versions 2.0.16, 2.0.15, 1.18.12, and 1.18.11, have reached their end of full support period.
May 30, 2023
Starting July 2023, the new composer.environments.executeAirflowCommand permission will be required to run Airflow CLI commands through the gcloud composer environments run command:
The
composer.userandcomposer.environmentAndStorageObjectViewerroles do not have this permission and will not be permitted to run Airflow CLI commands starting July 2023.This permission is already available in IAM and you can assign it in advance.
This permission is already added to the
composer.admincomposer.environmentAndStorageObjectAdminroles.This change applies only to Cloud Composer 2 environments. It will still be possible to run Airflow CLI commands on Cloud Composer 1 environments without this permission.
(Cloud Composer 2) The number of web server workers is now set dynamically based on available web server CPU and memory. This change improves Airflow web server performance and scalability by allowing it to handle more users.
These workers are internal to the gunicorn web server and are not related to workers that run tasks.
The new value is applied to the
[webserver]workersAirflow configuration option when you change the environment's configuration. To use a different value, override this Airflow configuration option.The number of web server workers is clamped between 2 and 12 workers and is calculated as the minimum of
(web_server_CPU * 2) + 1andweb_server_memory * 1.1.
(Cloud Composer 2) You can now use custom certificates when installing packages from your private repository.
(Cloud Composer 2) The deprecated [core]non_pooled_task_slot_count Airflow configuration option is replaced with the [core]default_pool_task_slot_count configuration option in the default Airflow configuration. Make sure to update your custom Airflow configuration overrides to use the new option instead of the deprecated one.
An improved error message is now displayed when a subnetwork with unsupported IPv4 ranges is used to create an environment in a shared VPC configuration.
Cloud Composer 2.2.1 images are available:
- composer-2.2.1-airflow-2.5.1 (default)
- composer-2.2.1-airflow-2.4.3
Cloud Composer versions 2.0.14, 2.0.13, 1.18.10, and 1.18.9, have reached their end of full support period.
May 19, 2023
Cloud Composer API for Highly resilient environments is available. Cloud Console UI and gcloud CLI commands for this feature will be gradually rolled out in the upcoming days. Terraform support will be rolled out in the upcoming weeks as well.
Splitting Celery logs into stdout/stderr (#30485) is now possible with the [logging]celery_stdout_stderr_separation Airflow configuration option. The default value for this option is False.
(Cloud Composer 2) Increased the default values of [core]dagbag_import_timeout to 120 seconds, and dag_file_processor_timeout to 300 seconds. The updated parameters allow the DAG Processor to parse more DAGs and provide more time to add parsed DAGs into the DAG bag.
(Cloud Composer 2) Increased the default value of [scheduler]zombie_detection_interval to 20 seconds. This change reduces the chance that Airflow skips retry attempts for a zombie task.
Cloud Composer 2.2.0 images are available:
- composer-2.2.0-airflow-2.5.1 (default)
- composer-2.2.0-airflow-2.4.3
Cloud Composer versions 2.0.12 and 1.18.8, have reached their end of full support period.
May 12, 2023
(Cloud Composer 2) The default [celery]stalled_task_timeout value is set to 1200. Tasks that stay in the queued state for more than 20 minutes (1200 seconds) are now rescheduled.
(Airflow 2.4.3 and 2.5.1) Python packages upgraded:
- Added new packages:
dbt-bigqueryandgoogle-cloud-documentai. - The
apache-airflow-providers-cncf-kubernetespackage was upgraded to version 6.0.0.
(Cloud Composer 2 with Airflow 2.5.1 and 2.4.3) Version 6.0.0 of the CNCF Kubernetes Provider package (apache-airflow-providers-cncf-kubernetes) introduces a backwards incompatible change. The kubernetes_default connection is now used by default in the KubernetesPodOperator. For more information, see CNCF Kubernetes Provider Changelog.
(Cloud Composer 2) The default version of Airflow is changed to 2.5.1.
Cloud Composer 2.1.15 images are available:
- composer-2.1.15-airflow-2.5.1 (default)
- composer-2.1.15-airflow-2.4.3
Airflow 2.3.4 is no longer included in Cloud Composer images.
Cloud Composer versions 2.0.11 and 1.18.7, have reached their end of full support period.
April 25, 2023
Starting from March 2023, Cloud Composer 1 is in maintenance mode. Maintenance releases of Cloud Composer 1 will contain only bug fixes and small improvements. Support for new Airflow versions after 2.4.3 is not planned for Cloud Composer 1.
(Cloud Composer 2) Airflow 2.5.1 is available in Cloud Composer images.
(Airflow 2.5.1 and 2.4.3) Per-folder Roles Registration now correctly reassigns permissions if a DAG file is deleted and added back.
Cloud Composer 2.1.14 and 1.20.12 images are available:
- composer-2.1.14-airflow-2.5.1
- composer-2.1.14-airflow-2.4.3 (default)
- composer-2.1.14-airflow-2.3.4
- composer-1.20.12-airflow-2.4.3
- composer-1.20.12-airflow-2.3.4
- composer-1.20.12-airflow-1.10.15
April 18, 2023
(Composer 2 only) Cloud Composer is now available in Taiwan (asia-east1), Jakarta (asia-southeast2), and Netherlands (europe-west4).
Java Runtime in Airflow workers and schedulers is updated from version 11 to version 17.
The apache-airflow-providers-google package in images with Airflow 2.3.4 and 2.4.3 was upgraded to 2023.4.13+composer. Changes compared to version 2023.3.14+composer:
- Update Google Display and Video 360 operators to use API v2.
- Update Google Campaign Manager operators to use API v4.
- Update
google-cloud-dlppackage to version3.7.1and adjust hooks and operators.
(Airflow 2.4.3 only) In environments with enabled data lineage integration, removed unnecessary warnings about deprecated operators that appeared in Airflow task logs.
The Google Display and Video 360 API v.1.1 is deprecated and its sunset date is April 27, 2023. Airflow operators that relied on API v1.1 will stop working after this date. If you use Google Display and Video 360 operators, then upgrade your environment to Cloud Composer version 2.1.13 or later. For more information about changes in operators, see Known Issues.
The Google Campaign Manager API v3.5 API is deprecated and its sunset date is May 1, 2023. Airflow operators that relied on API v3.5 will stop working after this date. If you use Google Campaign Manager operators, then upgrade your environment to Cloud Composer version 2.1.13 or later.
Cloud Composer 2.1.13 images are available:
- composer-2.1.13-airflow-2.4.3 (default)
- composer-2.1.13-airflow-2.3.4
Cloud Composer versions 2.0.10 and 1.18.6, have reached their end of full support period.
April 06, 2023
(Available without upgrading) Selected time ranges are now synchronized between the Monitoring tab and the Logs tab in Cloud Console.
(Cloud Composer 2) Fixed a problem that caused Airflow tasks to fail in some cases because a worker scheduled for deletion started a new task before it was deleted.
(Available without upgrading, Cloud Composer 1) Error messages related to the environment's cluster creation are now always propagated to error messages for Cloud Composer operations.
GCSFuse version updated to 42.3. Cloud Composer uses GCSFuse to sync files between the environment bucket and worker pods.
Cloud Composer 2.1.12 images are available:
- composer-2.1.12-airflow-2.4.3 (default)
- composer-2.1.12-airflow-2.3.4
Cloud Composer versions 2.0.9, 2.0.8, 1.18.5, and 1.18.4, have reached their end of full support period.
March 31, 2023
(Cloud Composer 2) Access Approval is now generally available (GA). See Access Approval overview and Access Approval supported services for more information.
March 27, 2023
Cloud Composer 2 now supports access with external identities through workforce identity federation.
Fixed a problem where upgrade checks were failing for some Cloud Composer 2 environments. This issue was affecting environments where Cloud Build can't be used to install PyPI packages.
The default value for the dag_dir_list_interval Airflow configuration option is changed from 30 to 120 seconds.
Increased the timeout for environment operations performed by Cloud Build to 35 minutes.
Cloud Composer 2.1.11 and 1.20.11 images are available:
- composer-2.1.11-airflow-2.4.3 (default)
- composer-2.1.11-airflow-2.3.4
- composer-1.20.11-airflow-1.10.15
- composer-1.20.11-airflow-2.4.3
- composer-1.20.11-airflow-2.3.4
March 18, 2023
If your environment interacts with Google Ads, make sure to upgrade it to Cloud Composer version 2.1.10 and 1.20.10 (or later versions) before March 29, 2023.
On this date, the Google Ads API v11 is sunset. The google-ads version 20.0.0 communicates with the supported Google Ads API v12.
(Airflow 2.3.4 and 2.4.3 only) The Google Ads Python library (google-ads) package is updated to version 20.0.0.
(Airflow 2.3.4 and 2.4.3 only) The apache-airflow-providers-google package was upgraded to version 2023.3.14+composer. This version is based on the public version 8.9.0 with the following changes:
- Use
google-ads==20.0.0
The source code for the apache-airflow-providers-google package version 2023.3.14+composer is available on GitHub.
5 new Airflow metrics are now available in Cloud Monitoring. For more information, see Monitor environments with Cloud Monitoring.
Cloud Composer 2 environments no longer create airflow.cfg and env_var.jsonfiles in the environment's bucket.
(Composer 2 only) The instance_name field for Airflow Webservers in new environments is now by default set to the Composer Environment name.
(Airflow 2) The Airflow metadata database size metric now reports more accurate values after the data is deleted from the database.
Cloud Composer 1.20.1 and 2.1.11 are versions with an extended upgrade timeline.
Cloud Composer 2.1.10 and 1.20.10 images are available:
- composer-2.1.10-airflow-2.4.3 (default)
- composer-2.1.10-airflow-2.3.4
- composer-1.20.10-airflow-1.10.15
- composer-1.20.10-airflow-2.4.3
- composer-1.20.10-airflow-2.3.4
Airflow 2.2.5 is no longer included in Cloud Composer images.
Cloud Composer versions 2.0.7 and 1.18.3 have reached their end of full support period.
March 13, 2023
PyPI package installation is now possible in network setups with maximum transmission unit (MTU) of 1280 bytes or more. This issue was affecting environments where Cloud Build cannot be used to install PyPI packages.
Fixed the issue where BigQuery tasks in the deferrable mode failed when data lineage was enabled.
Cloud Composer 2.1.9 and 1.20.9