These release notes apply to the Cloud Composer service. Announcements and information about new or updated features, bug fixes and deprecated functionality is published on this page.
You can also:
- Follow Cloud Composer release notes in the Apache Airflow Community workspace on Slack.
- See Cloud Composer version list for more information about Cloud Composer images and support dates.
- Check the Cloud Composer known issues page to learn more about known issues and possible workarounds.
You can see the latest product updates for all of Google Cloud on the Google Cloud page, browse and filter all release notes in the Google Cloud console, or programmatically access release notes in BigQuery.
To get the latest product updates delivered to you, add the URL of this page to your feed reader, or add the feed URL directly.
October 09, 2025
Upgrade checks are now generally available (GA) in Cloud Composer 3 and Cloud Composer 2.
September 30, 2025
The GCE_METADATA_TIMEOUT environment variable is changed to reserved. This
change addresses an issue where setting a low timeout value disrupted the
environment's operations that relied on the metadata server.
DAG UI now correctly generates error messages about malformed serialized DAG.
(Airflow 2.10.5) The apache-airflow-providers-google package was upgraded
to version 17.2.0 in Cloud Composer 2 images and Cloud Composer 3 builds.
For more information about changes, see the apache-airflow-providers-google changelog from version 17.1.0 to version 17.2.0.
(Airflow 2.10.5) CloudComposerDAGRunSensor is broken in the
apache-airflow-providers-google package version 17.2.0. This package is used
by Cloud Composer versions and builds with Airflow 2.10.5 available in this
release. If your DAGs use this sensor, we recommend you postpone upgrading
until the issue is resolved.
(Airflow 2.10.5) The apache-airflow-providers-cncf-kubernetes package was
upgraded to version 10.8.0
from version 10.7.0. For changes in other packages, see the
preinstalled packages changelog.
New Airflow builds are available in Cloud Composer 3:
New images are available in Cloud Composer 2:
The following Cloud Composer versions and builds have reached their end of support period: composer-2.9.4-*, and composer-3-airflow-2.9.1 builds from build.0 to build.7.
September 11, 2025
New Airflow builds are available in Cloud Composer 3:
New images are available in Cloud Composer 2:
Fixed an issue where values of Airflow configuration options were evaluated before being set. As a result, the actual value was set to the evaluated result.
Fixed Airflow logs not exporting to Cloud Logging because of a GKE version mismatch between Airflow worker and GKE Control plane nodes.
September 03, 2025
(Cloud Composer 2) Cloud Composer's high availability infrastructure was enhanced to provide greater resilience against zonal outages. This change rolls out gradually over several releases to all regions supported by Cloud Composer 2.
(Available without upgrading) Cloud Composer 3 now supports DNS resolution for regional service endpoints. You can now reach regional service endpoints from DAGs in your environment. This change is available in Public IP environments without additional configuration. For Private IP environments, an environment must be connected to a VPC network where private endpoints are configured.
You can now check if a Cloud Composer 2 environment's configuration is compatible with Cloud Composer 3. We recommend doing this check before migrating to Cloud Composer 3.
New Airflow builds are available in Cloud Composer 3:
New images are available in Cloud Composer 2:
The following Cloud Composer versions and builds have reached their end of support period: composer-2.9.2-* and composer-2.9.3-*.
August 21, 2025
Improved the startup times of Airflow workers for environments that have a large number of custom PyPI packages installed.
This feature was announced previously and has finished gradually rolling out to all regions supported by Cloud Composer.
Airflow UI now shows the correct value of the [core]dags_folder Airflow
configuration option.
(Airflow 2.10.5) The apache-airflow-providers-google package was upgraded to version 17.1.0 in Cloud Composer 2 images and Cloud Composer 3 builds.
For more information about changes, see the apache-airflow-providers-google changelog from version 15.1.0 to version 17.1.0.
(Airflow 2.10.5) The apache-airflow-providers-cncf-kubernetes package was upgraded to version 10.7.0 from version 10.6.1. For changes in other packages, see the preinstalled packages changelog.
New Airflow builds are available in Cloud Composer 3:
New images are available in Cloud Composer 2:
The following Cloud Composer versions and builds have reached their end of support period: composer-2.9.0, composer-2.9.1, and composer-3-airflow-2.7.3 builds from build.5 to build.13.
August 05, 2025
(Cloud Composer 2) Moved the update_fab_perms option from [webserver] to
[fab] in the Airflow configuration. The corresponding deprecation warning is
no longer generated in Airflow web server logs.
(Airflow 2.10.5 only) Added task-level resource consumption Airflow metrics to Cloud Composer.
composer.googleapis.com/workflow/task/cpu_usage: percentage of CPU used by a task.composer.googleapis.com/workflow/task/cpu_usage: percentage of memory used by a task.
New Airflow builds are available in Cloud Composer 3:
New images are available in Cloud Composer 2:
Cloud Composer versions 2.8.7 and 2.8.8 have reached their end of support period.
July 28, 2025
Fixed an issue that caused unexpected restarts of Airflow component workloads in the environment's cluster.
(Cloud Composer 3) The DAGS_FOLDER reserved environment variable now
correctly points to the local directory where DAG files are stored.
New Airflow builds are available in Cloud Composer 3:
New images are available in Cloud Composer 2:
Cloud Composer version 2.8.6 has reached its end of support period.
July 24, 2025
Web server restarting is is now generally available (GA) in Cloud Composer 2 and Cloud Composer 3.
July 23, 2025
If your environment uses dag-factory package version 0.22, then you might
experience DAG failures in Cloud Composer versions that have
apache-airflow-providers-cncf-kubernetes package version 10.4.2 or later. At
the same time, upgrading the dag-factory package to version 0.23 might require
you to update your DAG code to make it compatible.
If your environment uses dag-factory version 0.22, we recommend to do the
following:
- Temporarily postpone upgrading your environment until you're ready to switch
to
dag-factoryversion 0.23. Last versions of Cloud Composer that support version 0.22 are composer-3-airflow-2.10.5-build.3, composer-3-airflow-2.9.3-build.23, composer-2.13.1-airflow-2.10.5, and composer-2.13.1-airflow-2.9.3 released on May 14, 2025. - When you are ready to upgrade, update your DAGs for compatibility with 0.23.
We recommend to do this in a development environment first.
Install
dag-factoryversion 0.23, then check that your DAGs are parsed and are working correctly, and update them if needed. After your DAGs are compatible, installdag-factoryversion 0.23 in your production environment and transfer the updated DAGs. Your environment can now be upgraded to a later version of Cloud Composer or Airflow. - If your environment is already upgraded to a later version of Cloud Composer
and you experience problems, then update
dag-factoryto version 0.23 and update your DAGs for compatibility with 0.23.
July 17, 2025
Highly resilient environments are now generally available (GA) in Cloud Composer 3.
July 14, 2025
We're changing the way we provide support dates for Airflow builds in Cloud Composer 3. Before this change, some Airflow builds had their end of support date listed as "To be announced" until a later Airflow version became available. We're deprecating this approach for all builds that are released after July 01, 2025.
We are now providing support dates that depend on a date when a particular Airflow build was released:
- We are introducing the standard support period of 12 months after the release date.
- All Airflow builds that didn't have an end of support date when this change was introduced are supported until July 1, 2026. Because of this change, some Airflow builds released before July 1, 2025 are supported for longer than the standard support period.
- All Airflow builds that had their support date shorter than 12 months are now supported for 12 months since the date of their initial release.
- All builds released after July 1, 2025 will use the standard support period.
July 09, 2025
We are gradually rolling out a change that switches the default version from Cloud Composer 2 to Cloud Composer 3 in the Cloud Composer API.
In regions where the change is rolled out, a Cloud Composer 3 environment is created by default when a version is not specified in Google Cloud CLI, Cloud Composer API, or Terraform. If you use automation scripts to provision Cloud Composer 2 environments, make sure that you explicitly specify a Cloud Composer 2 version.
In this release, the change is rolling out in the following regions: africa-south1, asia-northeast2, asia-south2, asia-southeast2, europe-southwest1, europe-west10, europe-west12, europe-west8, me-central1, me-central2, me-west1, southamerica-west1, and us-south1.
(Available without upgrading) During Cloud Composer 2 environment operations, a more informative error message is returned when an environment's web server has connectivity issues.
New Airflow builds are available in Cloud Composer 3:
New images are available in Cloud Composer 2:
Cloud Composer versions 2.8.4 and 2.8.5 have reached their end of support period.
July 02, 2025
Cloud Composer 1 and Cloud Composer 2 environments with version 2.0.x are approaching their end of life. We're planning to deprecate them in the following way:
- Starting September 15, 2025, you will no longer be able to create new Cloud Composer 1 environments.
- On September 15, 2026, all Cloud Composer 1 and Cloud Composer 2 version 2.0.x environments will reach their planned end of life, and you won't be able to use them.
Cloud Composer 2 environments with versions later than 2.1.0 and all Cloud Composer 3 environments are not affected by this deprecation.
We recommend planning migration to Cloud Composer 3 or upgrading your Cloud Composer 2 environments to a later version.
July 01, 2025
This release includes internal infrastructure improvements to Cloud Composer. There are no user-visible changes.
New Airflow builds are available in Cloud Composer 3:
New images are available in Cloud Composer 2:
June 23, 2025
Improved the startup times of Airflow workers for environments that have a large number of custom PyPI packages installed.
This feature was announced previously and is gradually rolling out over several releases. In this release, it's available in asia-east2, asia-northeast3, europe-central2, europe-west9, me-central1, me-west1, northamerica-northeast1, northamerica-northeast2, northamerica-south1, and us-west1 re
(Available without upgrading) Fixed an issue where deleting a Cloud Composer 2 environment could fail when the environment's cluster was in the process of creating a node pool.
(Airflow 2.10.5) The apache-airflow-providers-cncf-kubernetes package was upgraded to version 10.5.0 from version 10.4.2. For changes in other packages, see the preinstalled packages changelog.
New Airflow builds are available in Cloud Composer 3:
New images are available in Cloud Composer 2:
Cloud Composer version 2.8.3 has reached its end of support period.
June 16, 2025
We're planning to phase out the APIs that aren't required by Cloud Composer 3.
Starting February 27, 2026, the following APIs will become fully detachable. Deactivating these APIs won't cause the deactivation of the Cloud Composer API:
- artifactregistry.googleapis.com
- cloudbuild.googleapis.com
- container.googleapis.com
- pubsub.googleapis.com
- sqladmin.googleapis.com
Starting May 27, 2026, these APIs will no longer be enabled automatically when you enable the Cloud Composer API. To create Cloud Composer 2 environments in new projects, the group of detached APIs must be enabled manually.
Existing Cloud Composer 3 and Cloud Composer 2 environments in projects where the Cloud Composer API is already enabled will not be impacted. You can do the following:
- After February 27, 2026, if your project has only Cloud Composer 3 environments, then you can manually disable the detached APIs.
- After February 27, 2026, if your project has Cloud Composer 2 environments, then we recommend keeping these APIs enabled because disabling them might lead to environment's malfunction.
- After May 27, 2026, if you use automation scripts to provision Cloud Composer 2 environments, then make sure that the listed APIs are enabled in addition to the Cloud Composer API.
June 13, 2025
Cloud Composer pages in the Cloud Console now support the dark color theme. You can switch to the dark theme on the Appearance page in the Cloud Console.
June 10, 2025
A new Cloud Composer release has started on June 10, 2025. Get ready for upcoming changes and features as we roll out the new release to all regions. This release is in progress at the moment. Listed changes and features might not be available in some regions yet.
Fixed an issue that caused Airflow worker and scheduler Pods to be evicted when a large number of tasks was executed.
New Airflow builds are available in Cloud Composer 3:
New images are available in Cloud Composer 2:
Cloud Composer version 2.8.2 has reached its end of support period.
June 02, 2025
Improved the startup times of Airflow workers for environments that have a large number of custom PyPI packages installed.
This change rolls out gradually. In this release, it's available in asia-east1, asia-northeast2, asia-south1, europe-north1, europe-west3, us-east1, us-south1, and us-west2 regions.
(Cloud Composer 3) If a VPC network is attached to an environment, then all DNS addresses accessed by the Airflow components of the environment are resolved using the Cloud DNS configuration of the VPC network. In particular, Airflow workers that execute DAGs will resolve DNS addresses in this way.
(Cloud Composer 3) All newly created private DNS zones are immediately visible to a Cloud Composer environment. Previously, re-attaching a VPC network was required.
(Cloud Composer 3) It's now possible to use zones with cross-project binding. Before this change, cross-project bound zones weren't supported in Cloud Composer 3.
(Cloud Composer 2) In Cloud Composer versions 2.11.5 and later, log processing is switching to using OpenTelemetry instead of Fluentd.
This change was announced previously and is gradually rolling out over several releases. In this release, it's available in the following regions: asia-east1, asia-east2, asia-northeast2, asia-northeast3, asia-south2, asia-southeast2, australia-southeast1, australia-southeast2, europe-central2, europe-north1, europe-north2, europe-southwest1, europe-west, europe-west10, europe-west12, europe-west6, europe-west8, europe-west9, me-central1, me-central2, me-west1, northamerica-northeast2, northamerica-south1, southamerica-east1, southamerica-west1, us-east5, us-south1, us-west, and us-west3.
New Airflow builds are available in Cloud Composer 3:
New images are available in Cloud Composer 2:
Cloud Composer version 2.8.1 has reached its end of support period.
May 29, 2025
The Composer Local Development CLI tool is now available in Cloud Composer 3. This tool helps to streamline testing and developing by providing local Airflow environments based on Airflow builds used by Cloud Composer 3.
May 26, 2025
(Cloud Composer 3) New metrics that show the number of active Airflow components are now available for Cloud Composer 3 environments:
- The number of active schedulers
- The number of active DAG processors
- The number of active triggerers
- The number of active web servers
(Cloud Composer 3) It's now possible to override the default scopes of access tokens in all regions supported by Cloud Composer 3. This fix was announced previously and has finished gradually rolling out.
(Airflow 2.10.5 and 2.9.3) The apache-airflow-providers-cncf-kubernetes package was upgraded to version 10.4.2 from version 10.1.0.
New Airflow builds are available in Cloud Composer 3:
New images are available in Cloud Composer 2:
May 14, 2025
A new Cloud Composer release has started on May 14, 2025. Get ready for upcoming changes and features as we roll out the new release to all regions. This release is in progress at the moment. Listed changes and features might not be available in some regions yet.
It is now possible to migrate from Cloud Composer 1 to Cloud Composer 3 using snapshots in all regions supported by Cloud Composer 3. The gradual rollout of this feature is finished.
It is now possible to restart the Airflow web server through Cloud Composer API by using the RestartWebServer method.
The default value of the [scheduler]max_tis_per_query Airflow configuration option is set to 128. This change fixes the issue with DAGs remaining in the scheduled state in some cases.
Cloud Composer 3 and Cloud Composer 2 no longer support files with gzip encoding (content-type=gzip).
Before this change, all files that had gzip encoding, including .py and .zip files, were processed inconsistently by the DAG processor. After this change, the DAG processor skips all files with gzip encoding. This change doesn't apply to regular .zip archives. Airflow still decompresses and processes DAG files contained in archives that have no specified encoding.
This change is gradually rolled out to all regions supported by Cloud Composer, except us-central1, us-east4 and europe-west1. It will be rolled out to the remaining regions in one of the future releases.
(Cloud Composer 3) Fixed an issue where the Airflow scheduler failed to initialize and work properly if a custom email backend was enabled. The Airflow scheduler now supports the following email backends:
- airflow.providers.sendgrid.utils.emailer.send_email
- airflow.utils.email.send_email_smtp
This issue affected only the Airflow scheduler. Airflow workers already support these and other backends. If you use an email backend that isn't supported by the scheduler, you can still send email from Airflow workers.
(Cloud Composer 3) Improved the computation of metrics related to snapshots. This change reduces the number of calls made by Cloud Composer to the Cloud Storage API.
(Cloud Composer 2) In Cloud Composer versions 2.11.5 and later, log processing is switching to using OpenTelemetry instead of Fluentd.
This change is gradually rolled out to the asia-south1 and africa-south1 regions. It will be rolled out to more regions in future releases. New and upgraded Cloud Composer 2 environments in the listed regions will get this change.
(Airflow 2.9.3) The apache-airflow-providers-google package was upgraded to version 15.1.0 in Cloud Composer 2 images and Cloud Composer 3 builds.
For more information about changes, see the apache-airflow-providers-google changelog from version 14.0.0 to version 15.1.0.
(Airflow 2.9.3) Changes in preinstalled packages:
apache-airflow-providers-standardwas upgraded to 1.0.0 from 0.4.0.aiosqlitewas removed from preinstalled packages.json-merge-patchwas removed from preinstalled packages.time-machinewas removed from preinstalled packages.pyjwtwas downgraded to 2.9.0 from 2.10.1.
New Airflow builds are available in Cloud Composer 3:
- composer-3-airflow-2.10.5-build.3 (default)
- composer-3-airflow-2.9.3-build.23
New images are available in Cloud Composer 2:
- composer-2.13.1-airflow-2.10.5 (default)
- composer-2.13.1-airflow-2.9.3
Cloud Composer version 2.8.0 has reached its end of support period.
The following Cloud Composer versions have an extended upgrade timeline:
- composer-3-airflow-2.10.5-build.3
- composer-3-airflow-2.9.3-build.23
- composer-2.13.1-airflow-2.10.5
- composer-2.13.1-airflow-2.9.3
May 07, 2025
Data lineage in Cloud Composer now uses OpenLineage in all regions supported by Cloud Composer. For more information about this feature, see the previous announcement.
For newly created Cloud Composer 3 environments, the minimum amount of memory is changed to 2 GB.
For newly created environments, database retention policy is now enabled by default in Google Cloud CLI, API, and Terraform. Before this change, it was enabled by default only in Google Cloud Console.
Improved the environment liveness monitoring. This change addresses some cases of transient failures that caused "Liveness probe failed" warnings in the environment's logs.
(Airflow 2.10.5) The apache-airflow-providers-google package was upgraded to version 15.1.0 in Cloud Composer 2 images and Cloud Composer 3 builds.
For more information about changes, see the apache-airflow-providers-google changelog from version 14.0.0 to version 15.1.0.
(Airflow 2.10.5) Changes in preinstalled packages:
apache-airflow-providers-standardwas upgraded to 1.0.0 from 0.4.0.aiosqlitewas removed from preinstalled packages.json-merge-patchwas removed from preinstalled packages.time-machinewas removed from preinstalled packages.
The default version of Airflow is changed to 2.10.5.
Airflow 2.10.2 is no longer included in Cloud Composer images and builds.
New Airflow builds are available in Cloud Composer 3:
- composer-3-airflow-2.10.5-build.2 (default)
- composer-3-airflow-2.9.3-build.22
New images are available in Cloud Composer 2:
- composer-2.13.0-airflow-2.10.5 (default)
- composer-2.13.0-airflow-2.9.3
Cloud Composer versions 2.7.0 and 2.7.1 have reached their end of support period.
May 06, 2025
The Deployment Manager API is no longer automatically enabled when you enable Cloud Composer API because this API isn't used by the Cloud Composer service.
Environments with Cloud Composer versions 2.0.* still rely on the Deployment Manager API for updates, upgrades, and environment deletion. It won't be possible to perform these operations if this API is disabled. We recommend to upgrade your 2.0.* environments to a later version to remove this dependency.
May 05, 2025
It is now possible to migrate from Cloud Composer 1 to Cloud Composer 3 using snapshots. For more information, see the new migration guide.
This feature will gradually roll out to all regions supported by Cloud Composer 3. At the moment it is available in the africa-south1, asia-south1, me-central1, me-central2, me-west1, southamerica-east1, and southamerica-west1 regions.
April 30, 2025
Starting from 5 May, 2025, new Cloud Composer 3 environments will use 1 CPU and 4 GB of memory for the Airflow web server by default. The minimum and maximum values for these parameters will not change.
April 29, 2025
A script for migrating from Cloud Composer 2 to Cloud Composer 3 is now available on GitHub.
For instructions about migrating with the script, see the new migration guide in Cloud Composer documentation.
April 24, 2025
Starting from June 2025, the default version for new Cloud Composer environments changes from Cloud Composer 2 to Cloud Composer 3. New environments will use the latest default Airflow build (composer-3-airflow-2). Currently, the default version is composer-2-airflow-2.
April 17, 2025
Airflow 2.10.5 is available in Cloud Composer.
Database retention policy is now enabled by default in Google Cloud console and remains disabled in Google Cloud CLI, API, and Terraform.
This feature helps to maintain the Airflow database size. You can enable or disable the database retention policy or adjust the retention period for new and existing environments.
The default environment's service account setting is gradually removed in Cloud Composer. After the change, you'll need to explicitly specify a service account when you create a new Cloud Composer environment. For more information about addressing the change, see the eariler announcement of this change.
In this release, the change is rolling out to the following regions: africa-south1, asia-northeast2, asia-south2, australia-southeast2, europe-north2, europe-southwest1, europe-west8, europe-west10, europe-west12, me-central1, me-central2, me-west1, northamerica-northeast2, northamerica-south1, southamerica-west1, us-east7, and us-south1. It will be rolled out to more regions in future releases.
Cloud Composer 2 environments now always use the environment's service account for performing PyPI packages installations:
- Existing Cloud Composer 2 environments that previously used the default Cloud Build service account now use the environment's service account instead.
- Cloud Composer 2 environments created in versions 2.10.2 and later already have this change.
- Cloud Composer 3 environments already use the environment's service account, and are not affected by this change.
- This change is gradually rolled out to all regions supported by Cloud Composer 2.
Cloud Composer now detects situations when asynchronous tasks are blocked in Airflow triggerers. If a trigger's execution is blocked for more than five minutes, Cloud Composer restarts the triggerer, which solves this transient issue.
(Cloud Composer 3) Key Access Justifications now correctly works for Customer Managed Encryption Keys (CMEK).
The bucket synchronization process doesn't fail if the /plugins folder isn't available in the environment's bucket.
(Cloud Composer 3) It's now possible to override the default scopes of access tokens. Before the fix, the scope always defaulted to https://www.googleapis.com/auth/cloud-platform and https://www.googleapis.com/auth/userinfo.email. This resulted in authentication failures when accessing non-Google Cloud services.
The change is gradually rolled out to the following regions: africa-south1, asia-south2, australia-southeast2, europe-north2, europe-west3, europe-west10, europe-west12, northamerica-south1, southamerica-west1, us-east7, and us-south1. It will be rolled out to more regions in future releases.
New Airflow builds are available in Cloud Composer 3:
- composer-3-airflow-2.10.5-build.0
- composer-3-airflow-2.10.2-build.13 (default)
- composer-3-airflow-2.9.3-build.20
New images are available in Cloud Composer 2:
- composer-2.12.1-airflow-2.10.5
- composer-2.12.1-airflow-2.10.2 (default)
- composer-2.12.1-airflow-2.9.3
Support dates for previous Cloud Composer 3 builds are available. All Cloud Composer 3 builds with Airflow 2.10.2 are supported until April 17, 2026.
April 09, 2025
The Airflow web server in Cloud Composer 3 requires at least 2 GB of memory when an environment is created or updated. This might lead to longer operation times or failures to perform these operations.
As a workaround, when you create a new Cloud Composer 3 environment or upgrade an existing environment, provide at least 2 GB of memory (default value) to the Airflow web server.
April 07, 2025
All Cloud Composer environment's GKE clusters are set up with maintenance exclusions from March 27, 2025 to April 12, 2025. For more information, see Maintenance exclusions.
April 03, 2025
The unification of Cloud Composer 3 billing with BigQuery is paused until further notice. The change was previously scheduled for April 13, 2025.
In recently released Airflow builds of Cloud Composer 3, the Airflow web server requires more CPU to finish its initialization when an environment is created or updated. This might lead to longer operation times or failures to perform these operations.
As a workaround, when you create a new Cloud Composer 3 environment or upgrade an existing environment, provide at least 1 CPU to the Airflow web server.
This issue currently affects composer-3-airflow-2.10.2-build.12 and composer-3-airflow-2.9.3-build.19 Airflow builds.
March 26, 2025
Data lineage in Cloud Composer now uses OpenLineage.
Data lineage support for a specific Airflow operator is now provided by the provider package where the operator is located. See Supported classes in the apache-airflow-providers-openlineage documentation for a list of latest supported operators.
For more information about data lineage in Cloud Composer, see Data lineage with Dataplex.
This feature is gradually rolled out. It will be available in us-west1, us-south1, europe-north1, me-west1, asia-northeast2, asia-southeast2, and africa-south1 regions. We plan to provide this feature in other regions in future releases.
(Available without upgrading) Fixed an issue with updating maintenance windows when there is an upcoming Cloud Composer 3 infrastructure operation.
(Airflow 2.10.2 and 2.9.3) The apache-airflow-providers-google package was upgraded to version 14.0.0 in Cloud Composer 2 images and Cloud Composer 3 builds.
This package is a new major version where many previously deprecated Airflow operators are removed. It is not possible to use these operators in your DAGs.
Make sure that you update your DAGs to use up-to-date alternatives of the removed operators. For more information about removed and deprecated Airflow operators and their up-to-date alternatives, see Deprecated and removed Airflow operators.
For more information about changes, see the apache-airflow-providers-google changelog from version 10.26.0 to version 14.0.0.
(Airflow 2.10.2 and 2.9.3) The apache-airflow-providers-cncf-kubernetes package was upgraded to version 10.3.0 in Cloud Composer 2 images and Cloud Composer 3 builds. For more information about changes, see the apache-airflow-providers-cncf-kubernetes changelog from version 10.1.0 to version 10.3.0.
(Airflow 2.10.2 and 2.9.3) Changes in preinstalled packages:
apache-airflow-providers-postgreswas upgraded to 6.1.0 from 5.14.0.apache-airflow-providers-smtpwas upgraded to 2.0.0 from 1.9.0.types-requestswas removed from preinstalled packages.
New Airflow builds are available in Cloud Composer 3:
- composer-3-airflow-2.10.2-build.12 (default)
- composer-3-airflow-2.9.3-build.19
New images are available in Cloud Composer 2:
- composer-2.12.0-airflow-2.10.2 (default)
- composer-2.12.0-airflow-2.9.3
Cloud Composer versions 2.6.4, 2.6.5, and 2.6.6 have reached their end of support period.
March 18, 2025
After April 15, 2025 the database retention policy feature will be enabled by default in newly created Cloud Composer 3 environments.
This feature helps to maintain the Airflow database size. You can enable or disable the database retention policy or adjust the retention period for new and existing environments.
The issue with Cloud Composer 2 upgrade operations is now resolved. The upgrade operations are unblocked in all regions.
March 15, 2025
The Custom constraints with Organization Policy feature is now generally available (GA).
This feature provides more granular control over Cloud Composer environment configuration fields. You can use custom organization policies to allow or deny specific configuration values for Cloud Composer environments.
March 14, 2025
March 18, 2025 update: The issue is resolved.
Some upgrade operations for Cloud Composer 2 might lead to unhealthy environments. That is why upgrades for Cloud Composer 2 versions will be blocked until the issue is fully resolved.
March 11, 2025
(Cloud Composer 3) Validation error messages now use the correct format for image versions.
New Airflow builds are available in Cloud Composer 3:
- composer-3-airflow-2.10.2-build.11 (default)
- composer-3-airflow-2.9.3-build.18
New images are available in Cloud Composer 2:
- composer-2.11.5-airflow-2.10.2 (default)
- composer-2.11.5-airflow-2.9.3
March 05, 2025
You can now specify an order in which Airflow searches for secrets by overriding the
[secrets]backends_order Airflow configuration option.
Fixed an issue in Cloud Composer REST API that allowed some environment.patch operations to succeed when multiple update masks that aren't related to each other were passed in a request. Now operations with such masks fail with an error.
New Airflow builds are available in Cloud Composer 3:
- composer-3-airflow-2.10.2-build.10 (default)
- composer-3-airflow-2.9.3-build.17
New images are available in Cloud Composer 2:
- composer-2.11.4-airflow-2.10.2 (default)
- composer-2.11.4-airflow-2.9.3
Cloud Composer versions 2.6.2 and 2.6.3 have reached their end of support period.
March 04, 2025
Cloud Composer 3 supports Customer Managed Encryption Keys (CMEK).
February 27, 2025
Database retention policy is available in Cloud Composer 3. You can use this feature to automatically delete older records from the Airflow database, which helps to maintain the Airflow database's size.
February 25, 2025
Cloud Composer 3 is now available in Stockholm (europe-north2). The change is gradually rolling out.
February 21, 2025
Starting April 14, 2025, the Deployment Manager API won't be automatically enabled when you enable Cloud Composer API because this API isn't used by the Cloud Composer service.
February 20, 2025
(Cloud Composer 3) Fixed a problem with configuring access to SMTP servers that don't support user and password authentication. The [smtp]smtp_user and [smtp]smtp_password Airflow configuration options are no longer preconfigured with placeholder values.
Improved the forbidden APIs check. When a new environment is created, Cloud Composer now checks for forbidden APIs that are required by a specific major version of Cloud Composer.
(Airflow 2.10.2 and 2.9.3) Preinstalled packages were changed:
virtualenvwas downgraded from 20.29.1 to 20.28.1
New Airflow builds are available in Cloud Composer 3:
- composer-3-airflow-2.10.2-build.9 (default)
- composer-3-airflow-2.9.3-build.16
New images are available in Cloud Composer 2:
- composer-2.11.3-airflow-2.10.2 (default)
- composer-2.11.3-airflow-2.9.3
Cloud Composer version 2.6.1 has reached its end of support period.
February 13, 2025
(Cloud Composer 3) The GOOGLE_CLOUD_PROJECT environment variable is changed to reserved for Cloud Composer 3. This fixes an issue where creating an environment with this variable failed with a non-specific error.
(Cloud Composer 2) The dependency_permissions_check_count metric now correctly reports the number of missing permissions.
(Cloud Composer 2) The list of permissions reported by dependency_permissions_check_count metric was updated. Now the metric reports more permissions that are required for a functioning environment. Permissions that weren't necessary were replaced or removed.
(New Cloud Composer environments only) Cloud Composer now enforces SSL connections to Cloud SQL instances.
This change implements a security compliance standard in Cloud Composer and enforces SSL usage. You can adopt other security standards by following recommendations listed in Vulnerability findings. Previously, it was possible to create a non-SSL connection to the Airflow database. After the change, the connection will be refused.
(Available without upgrading) Fixed a problem that caused Terraform to unnecessarily force the replacement of cloud_composer_network_ipv4_cidr_block, web_server_ipv4_cidr_block, and cloud_sql_ipv4_cidr_block fields in some configurations of Cloud Composer 2 and Cloud Composer 1 environments.
(Airflow 2.10.2 and 2.9.3) Preinstalled packages were changed:
logbookwas removed from preinstalled packagesminimal-snowplow-trackerwas removed from preinstalled packagesmashumarowas downgraded from 3.15 to 3.14
New Airflow builds are available in Cloud Composer 3:
- composer-3-airflow-2.10.2-build.8 (default)
- composer-3-airflow-2.9.3-build.15
New images are available in Cloud Composer 2:
- composer-2.11.2-airflow-2.10.2 (default)
- composer-2.11.2-airflow-2.9.3
Cloud Composer versions 2.6.0 and 2.6.1 have reached their end of support period.
February 07, 2025
Starting March 2025, the default version for new Cloud Composer environments changes from Cloud Composer 2 to Cloud Composer 3.
New environments that are created without a specified version will use the default Airflow build of Cloud Composer 3, composer-3-airflow-2. Currently, the default version is composer-2-airflow-2.
February 06, 2025
Cloud Composer 2 is no longer available in Mexico (northamerica-south1). It is not possible to select this region when creating a new Cloud Composer 2 environment. Existing Cloud Composer 2 environments and both new and existing Cloud Composer 3 environments are not affected by this change.
February 04, 2025
All Cloud Composer environment's GKE clusters are set up with maintenance exclusions from January 21, 2025 to February 10, 2025. For more information, see Maintenance exclusions.
January 31, 2025
(Cloud Composer 3) The issue with upgrading Airflow builds is now resolved. You can now upgrade Airflow builds for Cloud Composer 3 in the asia-south1, asia-northeast2, europe-west1, europe-west3, and europe-north1 regions.
January 30, 2025
At the beginning of February 2025, we will delete inactive environments that are non-recoverable in Cloud Composer 1, Cloud Composer 2, and Cloud Composer 3. After this change, non-recoverable environments that have listed problems will be deleted automatically.
Cloud Composer 1 and Cloud Composer 2 environments that have both of the following problems present at the same time will be deleted:
- The environment's underlying GKE cluster is deleted.
- The environment is in the ERROR state for at least 60 days because of a disabled billing account or because the Cloud Composer API service was deactivated in its project.
Cloud Composer 3 environments that have the following problem will be deleted:
- The environment is in the ERROR state for at least 60 days because of a disabled billing account or because the Cloud Composer API service was deactivated in its project.
This change doesn't affect buckets of these environments. You can still recover your DAGs and other data from the environment's bucket and then delete the bucket manually. See Delete environments for information about data that is not deleted automatically together with the environment.
January 28, 2025
Java Runtime in Airflow workers and schedulers is updated from version 17 to version 21.
Fixed a problem where synchronization between the environment's bucket and Airflow components failed because incorrect objects in the bucket were not ignored.
(Cloud Composer 3) Fixed an issue with KubernetesPodOperator failing to run properly when do_xcom_push was set to True.
(Available without upgrading) Starting from Cloud Composer version 2.11.0 with Airflow 2.10.2 and 2.9.3, the enable_task_context_logger Airflow configuration option is no longer blocked.
Terminating Airflow workers now generate log messages with information about still running tasks.
Fixed an issue where log messages were incorrectly formatted for non-string content, such as dictionaries and numbers.
(Available without upgrading) In Cloud Composer 2 and Cloud Composer 1, a more meaningful error message is produced when the Composer Service Agent account doesn't have necessary permissions to access the network during the environment creation.
(Available without upgrading) In Cloud Composer 3, fixed an issue where deleting an environment with an attached VPC network failed if the project accept list of the network attachment was cleared manually.
(Cloud Composer 3) The zone in the Environment.config.node_config.location field is no longer populated in the Cloud Composer API.
(Airflow 2.10.2 and 2.9.3) The apache-airflow-providers-cncf-kubernetes package was upgraded to version 10.1.0 in Cloud Composer 2 images and Cloud Composer 3 builds. For more information about changes, see the apache-airflow-providers-cncf-kubernetes changelog from version 10.0.1 to version 10.1.0.
(Airflow 2.10.2 and 2.9.3) The following preinstalled provider packages were upgraded to new major versions:
apache-airflow-providers-apache-beamupgraded from 5.9.1 to 6.0.0apache-airflow-providers-dbt-cloudupgraded from 3.11.2 to 4.0.0apache-airflow-providers-hashicorpupgraded from 3.8.0 to 4.0.0apache-airflow-providers-httpupgraded from 4.13.3 to 5.0.0apache-airflow-providers-mysqlupgraded from 5.7.4 to 6.0.0apache-airflow-providers-sendgridupgraded from 3.6.0 to 4.0.0apache-airflow-providers-sqliteupgraded from 3.9.1 to 4.0.0apache-airflow-providers-sshupgraded from 3.14.0 to 4.0.0
(Airflow 2.10.2 and 2.9.3) The following preinstalled packages were downgraded:
certifidowngraded from 2024.12.14 to 2024.8.30.charset-normalizerdowngraded from 3.4.1 to 3.4.0.cryptographydowngraded from 44.0.0 to 43.0.3.filelockdowngraded from 3.16.1 to 3.15.4.google-authdowngraded from 2.37.0 to 2.35.0.keyringdowngraded from 25.6.0 to 25.5.0.urllib3downgraded from 2.3.0 to 2.2.3.zippdowngraded from 3.21.0 to 3.20.2.
New Airflow builds are available in Cloud Composer 3:
- composer-3-airflow-2.10.2-build.7 (default)
- composer-3-airflow-2.9.3-build.14
New images are available in Cloud Composer 2:
- composer-2.11.1-airflow-2.10.2 (default)
- composer-2.11.1-airflow-2.9.3
Cloud Composer version 2.5.5 has reached its end of support period.
January 27, 2025
January 31, 2025 update: The issue is resolved.
(Cloud Composer 3 only) We are currently experiencing an issue with upgrading Airflow builds for Cloud Composer 3 in asia-south1, asia-northeast2, europe-west1, europe-west3, europe-north1 regions.
The upgrades are temporarily disabled as we continue our work to restore the listed functionalities. We will release an additional announcement after the issue is resolved.
January 24, 2025
The following recently released Cloud Composer 3 Airflow builds and Cloud Composer 2 versions are rolled back and aren't available for creating and upgrading existing environments. We will roll out new builds in the next release.
- composer-2.11.0-airflow-2.10.2
- composer-2.11.0-airflow-2.9.3
- composer-3-airflow-2.10.2-build.6
- composer-3-airflow-2.9.3-build.13
January 15, 2025
Starting April 13, 2025, we are removing the default environment's service account setting. This change enhances security and provides greater control over your Cloud Composer environments.
- Previously, the default Compute Engine service account was used by default when a user didn't specify a service account during Cloud Composer creation.
- After the change, you'll need to explicitly specify a service account when you create a new Cloud Composer environment.
- Existing Cloud Composer environments will not be affected by this change.
To address this change:
- We recommend to create one or more user-managed service accounts for Cloud Composer environments in your project and grant them the minimum of required permissions. For more information and instructions, see Grant roles to an environment's service account.
- If you use Terraform, scripts or other automation and configuration management tools, then make sure to update them, so that an environment's service account is specified when you create an environment.
In April 2025, Cloud Composer 2 environments will always use the environment's service account for performing PyPI packages installations:
- The environment's service account will be used instead.
- Existing Cloud Composer 2 environments that previously used the default Cloud Build service account will change to using the environment's service account instead.
- Cloud Composer 2 environments created in versions 2.10.2 and later already have this change.
- Cloud Composer 3 environments already use the environment's service account, and are not impacted by this change.
January 12, 2025
April 3, 2025 update: This change is postponed.
Starting after April 13, 2025, Cloud Composer 3 will unify its billing with BigQuery. The Cloud Composer 3 standard milli DCU-hours SKU will be replaced with the new BigQuery Engine for Apache Airflow SKU, which will be calculated based on the number of BigQuery slot hours that your Cloud Composer 3 environments consume. Other SKUs will be renamed and moved under the BigQuery hierarchy. The change will be rolled out gradually to all regions supported by Cloud Composer 3. For more information about the change, see Transition to BigQuery slot-hour-based pricing.
January 10, 2025
(Cloud Composer 3) New metrics are available for Cloud Composer 3 environments:
- CPU quota limit for Cloud Composer workloads
- CPU quota usage for Cloud Composer workloads
(Cloud Composer 3) You can now use custom certificates when installing packages from your private repository. This change is gradually rolled out to all Cloud Composer 3 environments. To obtain this change earlier, upgrade the Airflow build of your environment.
The issue with automatic environment upgrades and upgrading Airflow builds in Cloud Composer 3 is resolved and these operations are working. If you think that your environment is still impacted by this issue, please reach out to the Cloud Support team.
(New Cloud Composer 2 environments only) Cloud Composer 2 environments in versions 2.10.2 and later always use the environment's service account for performing PyPI packages installations. This change applies only to newly created environments, existing environments that are upgraded to 2.10.2 and later versions will not get this change.
(Cloud Composer 3 only) The /data folder is now synchronized with Airflow triggerers.
(Available without upgrading) Improved the error message generated when the Cloud Composer Service Agent service account is missing permissions on the project or on the environment's service account.
(Cloud Composer 2) Cloud Composer 2 environments that use PSC interfaces will no longer try to allocate IP ranges for VPC peerings in the tenant project. Environments that use PSC instead of VPC peerings do not use these ranges. This fixes a problem where these ranges overlapped with ranges used for the PSC subnetwork.
The Redis persistent disk is now automatically deleted together with the environment. This persistent disk is used by the Redis queue and stores only technical data.
Fixed an issue where the user-defined Cloud DNS configuration for Google API domains would break Cloud Composer 3 environment creation and attachment of VPC networks.
The worker_autoscale Airflow configuration option is blocked in Cloud Composer 2. Previously, it was blocked only in Cloud Composer 3.
Fixed a problem with the interpolation of pip.conf file. Now the pip.conf file is not interpolated and can contain unescaped % characters.
Removed the warning log message about in-memory storage because it doesn't apply to Cloud Composer.
The importlib-resources package was removed from preinstalled packages.
New Airflow builds are available in Cloud Composer 3:
- composer-3-airflow-2.10.2-build.5 (default)
- composer-3-airflow-2.9.3-build.12
New images are available in Cloud Composer 2:
- composer-2.10.2-airflow-2.10.2 (default)
- composer-2.10.2-airflow-2.9.3
Cloud Composer 2.10.1 is a version with an extended upgrade timeline.
Cloud Composer version 2.5.4 has reached its end of support period.
January 09, 2025
January 10, 2025 update: The issue is resolved.
(Cloud Composer 3 only) We are currently experiencing an issue with automatic environment upgrades and upgrading Airflow builds for Cloud Composer 3 in asia-south1, europe-west1, and asia-northeast2 regions.
The upgrades are temporarily disabled as we continue our work to restore the listed functionalities. We will release an additional announcement after the issue is resolved.
In January 2025, we will delete inactive Cloud Composer 1 environments that are non-recoverable. Environments that have both of the following problems present at the same time will be deleted:
- The environment's underlying GKE cluster is deleted.
- The environment is in the ERROR state for at least 60 days because of a disabled billing account or because the Cloud Composer API service was deactivated in its project.
This change doesn't affect buckets of these environments. You can still recover your DAGs and other data from the environment's bucket and then delete the bucket manually. See Delete environments for information about data that is not deleted automatically together with the environment.
December 16, 2024
Cloud Composer 3 is now generally available (GA):
- All Airflow builds starting from airflow-2.9.3-build.11 and airflow-2.10.2-build.4 are supported at the GA level.
- If your environment uses an earlier Airflow build, then upgrade it to airflow-2.9.3-build.11, airflow-2.10.2-build.4, or a later build to use Cloud Composer 3 on the GA level.
December 11, 2024
In January 2025, we plan to release Cloud Composer 2 versions that will always use the environment's service account for performing PyPI packages installations:
- New Cloud Composer 2 environments created in versions 2.10.2 and later will have this change. This change applies only to newly created environments, existing environments that are upgraded to 2.10.2 and later versions will not get this change.
- Currently, Cloud Composer 2 environments use the default Cloud Build service account if it exists (and the environment's service account if it doesn't). Depending on the way Cloud Build is configured in your project, this might mean that the default Cloud Compute service account or the legacy Cloud Build service account might be used by your environment. We recommend to configure Cloud Build to adhere to the principle of least privilege.
- Make sure to check the Cloud Build default service account change page for information about changes to the default Cloud Build service account.
- Cloud Composer 3 environments already use the environment's service account, and are not impacted by this change.
(Cloud Composer 3) It is now possible to upgrade an environment if the [sentry]sentry_on Airflow configuration option is set to true.
Cloud Composer no longer adds any missing IAM permissions to the Cloud Storage bucket when it is used to create an environment with a custom environment's bucket. Make sure that the environment's service account has permissions from the Composer Worker role on the bucket.
The COMPOSER_AGENT_BUILD_SERVICE_ACCOUNT environment variable is changed to reserved. This change improves the security of Cloud Composer environments.
New Airflow builds are available in Cloud Composer 3:
- composer-3-airflow-2.10.2-build.4 (default)
- composer-3-airflow-2.9.3-build.11
December 05, 2024
New Cloud Composer 3 environments can now be created in VPC SC. This feature is gradually rolled out to all regions supported by Cloud Composer.
Long log entries now have proper task instance annotations.
(Cloud Composer 3) KubernetesPodOperator now works when the do_xcom_push parameter is set to True.
(New Cloud Composer 3 environments) Increased the maximum number of internet connections that each Airflow worker can support at the same time.
Fixed the issue in the environment's component responsible for uploading the logs of Airflow components to Cloud Logging. This bug sometimes lead to a situation where Cloud Composer-level log might be missing for an Airflow component. The same log was still available on the Kubernetes-component level.
(Airflow 2.10.2 and 2.9.3) The apache-airflow-providers-google package was upgraded to version 10.26.0 in Cloud Composer 2 images and Cloud Composer 3 builds. For more information about changes, see the apache-airflow-providers-google changelog from version 10.25.0 to version 10.26.0.
(Airflow 2.10.2 and 2.9.3) The apache-airflow-providers-cncf-kubernetes package was upgraded to version 10.0.1 in Cloud Composer 2 images and Cloud Composer 3 builds. For more information about changes, see the apache-airflow-providers-cncf-kubernetes changelog from version 9.0.1 to version 10.0.1.
(Available without upgrading) Fixed an issue where Airflow workers sometimes generated incomplete or unreadable output.
The default version of Airflow is changed to 2.10.2.
New images are available in Cloud Composer 2:
- composer-2.10.0-airflow-2.10.2 (default)
- composer-2.10.0-airflow-2.9.3
December 04, 2024
Cloud Composer 2 is now available in Mexico (northamerica-south1).
November 27, 2024
In December 2024, Google will remove the following previously deprecated Airflow operators from the apache-airflow-providers-google package.
The new version of this package will be included in one of the future releases of Cloud Composer and the change will be announced in the Release Notes. After this change, it will not be possible to use these operators in your DAGs.
Make sure that you use up-to-date alternatives of the removed operators instead. For more information about removed and deprecated Airflow operators and their up-to-date alternatives, see Deprecated and removed Airflow operators.
Operators that will be removed in December 2024: DataPipelineHook, CreateDataPipelineOperator, RunDataPipelineOperator, AutoMLDatasetLink, AutoMLDatasetListLink, AutoMLModelLink, AutoMLModelTrainLink, AutoMLModelPredictLink.
November 13, 2024
(Airflow 2.7.3) Backported #35887 to fix an issue that occurred during the DST transition. The issue affected DAGs with timezone-aware cron schedule and caused infinite loops in the Airflow scheduler.
(Airflow 2.10.2 and 2.9.3) The apache-airflow-providers-google package was upgraded to version 10.25.0 in Cloud Composer 2 images and Cloud Composer 3 builds. For more information about changes, see the apache-airflow-providers-google changelog from version 10.24.0 to version 10.25.0.
November 06, 2024
(Cloud Composer 3) Fixed an issue that affected the speed of PyPI package installation. PyPI packages are now installed slightly faster.
(Airflow 2.9.3 and 2.7.3) The docutils package was removed from preinstalled packages.
New images are available in Cloud Composer 2:
- composer-2.9.10-airflow-2.9.3 (default)
- composer-2.9.10-airflow-2.7.3
Cloud Composer 2.9.7 is a version with an extended upgrade timeline.
October 30, 2024
New Airflow builds are available in Cloud Composer 3:
- composer-3-airflow-2.9.3-build.5 (default)
- composer-3-airflow-2.7.3-build.21
October 22, 2024
(Airflow 2.9.3) The apache-airflow-providers-google package was upgraded to version 10.24.0 in Cloud Composer 2 images and Cloud Composer 3 builds. For more information about changes, see the apache-airflow-providers-google changelog from version 10.23.0 to version 10.24.0.
(Airflow 2.9.3) The apache-airflow-providers-cncf-kubernetes package was upgraded to version 9.0.0 in Cloud Composer 2 images and Cloud Composer 3 builds. For more information about changes, see the apache-airflow-providers-cncf-kubernetes changelog from version 8.4.2 to version 9.0.0.
(Airflow 2.9.3 and 2.7.3) Changes in preinstalled packages:
- The
grpciopackage was downgraded from 1.66.2 to 1.65.5. - The
js2pypackage was removed from dependencies. - The
pyjsparserpackage was removed from dependencies. - The
tzlocalpackage was removed from dependencies.
Cloud Composer 2.9.8 images are available:
- composer-2.9.8-airflow-2.9.3 (default)
- composer-2.9.8-airflow-2.7.3
October 16, 2024
New Airflow builds are available in Cloud Composer 3:
- composer-3-airflow-2.9.3-build.3 (default)
- composer-3-airflow-2.7.3-build.19
October 01, 2024
(Available without upgrading) Fixed the cause of DAG run failures for runs created from the Cloud Console when the [scheduler]allowed_run_id_pattern Airflow configuration option is set to a custom value.
(Airflow 2.9.3 and 2.9.1) The apache-airflow-providers-cncf-kubernetes package was upgraded to version 8.4.2 in Cloud Composer 2 images and Cloud Composer 3 builds. For more information about changes, see the apache-airflow-providers-cncf-kubernetes changelog from version 8.4.1 to version 8.4.2.
September 18, 2024
Airflow 2.9.3 is available in Cloud Composer images.
(Cloud Composer 2) Fixed the issue where environment create and update operations could fail in rare cases because of the scheduler probe timeouts.
(Cloud Composer 3) Fixed the issue that caused KubernetesPodOperator tasks to fail if they ran for longer than 15 minutes.
New Airflow builds are available in Cloud Composer 3:
- composer-3-airflow-2.9.3-build.0
- composer-3-airflow-2.9.1-build.7 (default)
- composer-3-airflow-2.7.3-build.16
Cloud Composer 2.9.4 images are available:
- composer-2.9.4-airflow-2.9.3
- composer-2.9.4-airflow-2.9.1 (default)
- composer-2.9.4-airflow-2.7.3
Support dates for previous Cloud Composer 3 builds are available. All Cloud Composer 3 builds with Airflow 2.9.1 are supported until September 18, 2025.
Cloud Composer versions 2.4.2 and 2.4.3 have reached their end of support period.
September 02, 2024
Added a new metric: composer.googleapis.com/workflow/task_instance/queued_duration. This metric is based on the dag.<dag_id>.<task_id>.queued_duration Airflow metric.
(Cloud Composer 3) Fixed the cause of false-positive failures of Airflow Celery workers reported by the liveness health check.
Fixed an issue where an upgrade of a PSC-based private IP environment failed leaving the environment in an inconsistent state.
The apache-airflow-providers-cncf-kubernetes package was upgraded to version 8.4.1 in Cloud Composer 2 images and Cloud Composer 3 builds with Airflow 2.9.1. For more information about changes, see the apache-airflow-providers-cncf-kubernetes changelog from version 8.3.4 to version 8.4.1.
The apache-airflow-providers-google package was upgraded to version 10.22.0 in Cloud Composer 2 images and Cloud Composer 3 builds with Airflow 2.9.1. For more information about changes, see the apache-airflow-providers-google changelog from version 10.21.0 to version 10.22.0.
New Airflow builds are available in Cloud Composer 3:
- composer-3-airflow-2.9.1-build.6 (default)
- composer-3-airflow-2.7.3-build.15
Cloud Composer 2.9.3 images are available:
- composer-2.9.3-airflow-2.9.1 (default)
- composer-2.9.3-airflow-2.7.3
August 27, 2024
(Cloud Composer 2) Starting on September 26, 2024, all Cloud Composer environments within a VPC SC perimeter will have no access to public PyPI repositories by default. For details about installing PyPI packages in the VPC SC mode, see Installing PyPI packages.
August 22, 2024
(Cloud Composer 3) Fixed the dags backfill Airflow CLI command that was failing with a connection error.
Fixed a bug where executing Airflow CLI commands produced output that was not human-readable.
The apache-airflow-providers-google package was upgraded to version 10.21.1 in Cloud Composer 2 images with Airflow 2.7.3 and 2.9.1, and in all latest Cloud Composer 3 builds. For more information about changes, see the apache-airflow-providers-google changelog from version 10.21.0 to version 10.21.1.
New Airflow builds are available in Cloud Composer 3:
- composer-3-airflow-2.9.1-build.5
- composer-3-airflow-2.7.3-build.14
Cloud Composer 2.9.2 images are available:
- composer-2.9.2-airflow-2.9.1 (default)
- composer-2.9.2-airflow-2.7.3
Cloud Composer version 2.4.1 has reached its end of support period.
August 13, 2024
(Airflow 2.7.3) Improved the reliability of connections between Airflow components and the Redis component.
New Airflow builds are available in Cloud Composer 3:
- composer-3-airflow-2.9.1-build.4
- composer-3-airflow-2.7.3-build.13
Cloud Composer 2.9.1 images are available:
- composer-2.9.1-airflow-2.9.1 (default)
- composer-2.9.1-airflow-2.7.3
August 08, 2024
The kubelet read-only port in GKE clusters (TCP port 10255) is not used by Cloud Composer. You can define appropriate firewall rules to block external traffic over TCP 10255 in your customer project as described in Modify VPC firewall rules.
Fixed a problem that caused file synchronization between the bucket and Airflow components to be stuck or progress very slowly.
(Airflow 2.9.1) Improved the reliability of connections between Airflow components and the Redis component.
The default version of Airflow is changed to 2.9.1.
New Airflow builds are available in Cloud Composer 3:
- composer-3-airflow-2.9.1-build.3
- composer-3-airflow-2.7.3-build.12
Cloud Composer 2.9.0 images are available:
- composer-2.9.0-airflow-2.9.1 (default)
- composer-2.9.0-airflow-2.7.3
Cloud Composer version 2.4.0 has reached its end of support period.
August 01, 2024
In July and August 2024, new Cloud Composer 2 environments keep switching to using GKE 1.29 and PSC as a connectivity channel to the GKE control plane. Creating new Cloud Composer 2 private IP environments will require using PSC SERVICE_PRODUCER endpoints, which might be blocked by the Disable Private Service Connect for Consumers organization policy constraint. Previously, this org policy limitation could be addressed by using VPC peerings instead of PSC, but starting from GKE 1.29, this workaround is no longer available.
(Cloud Composer 3 only) The July 24th issue with Airflow upgrade operations for Cloud Composer 3 environments is now fixed. Airflow upgrade operations are now being re-enabled, and should reach all Cloud Composer 3 environments within the next couple of days.
The initial synchronization of DAGs and plugins in Airflow workloads is performed in a separate container and utilizes full resources of the Pod.
(Cloud Composer 3) Fixed an error that happened when values in certain formats were used for Airflow configuration options.
New Airflow builds are available in Cloud Composer 3:
- composer-3-airflow-2.9.1-build.2
- composer-3-airflow-2.7.3-build.11
Cloud Composer 2.8.8 images are available:
- composer-2.8.8-airflow-2.9.1
- composer-2.8.8-airflow-2.7.3 (default)
Airflow 2.6.3 is no longer included in Cloud Composer 2 images.
July 25, 2024
Cloud Composer now verifies that the environment's service account is enabled before starting an environment creation operation. As a result, this type of error is reported sooner and the operation does not start.
New Airflow builds are available in Cloud Composer 3:
- composer-3-airflow-2.9.1-build.1
- composer-3-airflow-2.7.3-build.10
Cloud Composer 2.8.7 images are available:
- composer-2.8.7-airflow-2.9.1
- composer-2.8.7-airflow-2.7.3 (default)
- composer-2.8.7-airflow-2.6.3
Cloud Composer version 2.3.5 has reached its end of support period.
July 24, 2024
2024-08-01 Update: This issue has been resolved. For more information, see the release note from August 1st 2024.
(Cloud Composer 3 only) We are currently experiencing an issue with the Airflow upgrade operations for Cloud Composer 3 environments. Upgrading Airflow builds for Cloud Composer 3 is temporarily disabled as we continue our work to restore Airflow upgrade functionalities.
July 18, 2024
Information about excluded Cloud Storage objects in the environment's bucket is no longer logged. This change reduces the usage of the Storage API during the synchronization of DAG files, which improves the performance of Airflow components and results in fewer Airflow component restarts. The change will gradually become available in all Cloud Composer environments.
July 17, 2024
Airflow 2.9.1 is available in Cloud Composer images and builds. We recommend checking the list of changes in Apache Airflow release notes before upgrading to this version.
(Airflow 2.9.1) Task context logging is disabled, and it is not possible to enable it.
(Airflow 2.9.1) Raw HTML code in DAG docs and DAG parameter descriptions is disabled by default.
(Airflow 2.9.1) Audit log permissions are revoked from all roles except Admin.
The apache-airflow-providers-google package was upgraded to version 10.21.0 in Cloud Composer 2 images with Airflow 2.7.3 and 2.9.1, and in all latest Cloud Composer 3 builds. For more information about changes, see the apache-airflow-providers-google changelog from version 10.18.0 to version 10.21.0.
The upgraded apache-airflow-providers-google package provides new operators and sensors for triggering DAGs across Cloud Composer environments:
- CloudComposerRunAirflowCLICommandOperator makes a call to the Airflow REST API of a Cloud Composer environment.
- CloudComposerDAGRunSensor checks if a DAG Run is completed.
When installing PyPI packages, if you want your builds to run with a custom service account, you can override the
COMPOSER_AGENT_BUILD_SERVICE_ACCOUNT environment variable with the chosen service account. For more information, see Install Python dependencies.
New Airflow builds are available in Cloud Composer 3:
- composer-3-airflow-2.9.1-build.0
- composer-3-airflow-2.7.3-build.9
Cloud Composer 2.8.6 images are available:
- composer-2.8.6-airflow-2.9.1
- composer-2.8.6-airflow-2.7.3 (default)
- composer-2.8.6-airflow-2.6.3
Support dates for previous Cloud Composer 3 builds are available. All Cloud Composer 3 builds with Airflow 2.7.3 are supported until July 17, 2025.
Cloud Composer version 2.3.4 has reached its end of support period.
July 15, 2024
It is no longer possible to create Cloud Composer 1 environments in Google Cloud console. It's still possible to create Cloud Composer 1 environments through Google Cloud SDK, Terraform, and API in projects that support creating new Cloud Composer 1 environments.
July 09, 2024
A new Airflow build is available in Cloud Composer 3:
- composer-3-airflow-2.7.3-build.8
Cloud Composer 2.8.5 images are available:
- composer-2.8.5-airflow-2.7.3 (default)
- composer-2.8.5-airflow-2.6.3
July 03, 2024
New Cloud Composer 2 environments are gradually switched to using GKE 1.29 and PSC as a connectivity channel to the GKE control plane. The IP address for the PSC endpoint will be taken from the nodes IP range. This change might require using a larger IP range for the nodes when you create an environment.
July 02, 2024
2024-07-03 Update: Resolution status updated.
The CVE-2024-6387 vulnerability in the OpenSSH package issue was discovered recently. GKE clusters used by Cloud Composer environments are impacted by this vulnerability, and Cloud Composer 1 and 2 environments that use Public IP networking are especially vulnerable to the described issue. For more information about CVE-2024-6387, see Google GKE Security bulletins.
Newly created Composer environments should not be impacted by this issue any more
Composer-owned GKE clusters will be auto-upgraded to newer GKE versions including the fix for CVE-2024-6387. Other components of Composer environments using older versions of COS images will also be upgraded. These operations will be done in an expedited manner so some of the update operations might be done outside environment's regular maintenance windows.
While Google works on resolving this issue so Composer environments are immune to CVE-2024-6387, you can disallow SSH to the Cloud Composer's cluster nodes through establishing proper firewall rules on the environment's cluster as described in the Google GKE Security bulletins. Follow the steps outlined for GKE.
A new Airflow build is available in Cloud Composer 3:
- composer-3-airflow-2.7.3-build.7
Cloud Composer 2.8.4 images are available:
- composer-2.8.4-airflow-2.7.3 (default)
- composer-2.8.4-airflow-2.6.3
Cloud Composer version 2.3.3 has reached its end of support period.
June 25, 2024
Cloud Composer is now available in Johannesburg (africa-south1).
June 20, 2024
We are thrilled to announce the Public Preview launch of the new generation of Cloud Composer, Cloud Composer 3. The new version is now publicly available in all regions supported by Cloud Composer. It comes with a number of new features and characteristics:
- All infrastructure hidden in a tenant project
- Evergreen versioning
- Simplified networking configuration
- Improved performance
- More reliable DAG parsing and scheduling as DAG Processor and Schedulers are now separate components
- 10 times bigger storage for Airflow workers
As well as most functionalities already known from the previous Composer versions. To see the list of features already supported by Composer 3, see Comparison of Cloud Composer versions.
(Airflow 2.7.3) New operators for executing jobs in Google Kubernetes Engine and Kubernetes are available. For example, you can use these operators with Kueue.
Operators for Google Kubernetes Engine:
- GKEStartJobOperator
- GKEStartKueueInsideClusterOperator
- GKEDescribeJobOperator
- GKEListJobsOperator
- GKECreateCustomResourceOperator
- GKEDeleteCustomResourceOperator
- GKEStartKueueJobOperator
- GKEDeleteJobOperator
- GKESuspendJobOperator
- GKEResumeJobOperator
Operators for Kubernetes:
- KubernetesJobOperator
- KubernetesPatchJobOperator
- KubernetesDeleteJobOperator
(Airflow 2.7.3) The apache-airflow-providers-google package was upgraded to version 10.18.0. For more information about changes, see the apache-airflow-providers-google changelog from version 10.17.0 to version 10.18.0.
(Airflow 2.7.3) The apache-airflow-providers-cncf-kubernetes package was upgraded to version 8.3.1.
(Airflow 2.7.3) The apache-beam package was upgraded to version 2.56.0.
A new Airflow build is available in Cloud Composer 3:
- composer-3-airflow-2.7.3-build.6
Cloud Composer 2.8.3 images are available:
- composer-2.8.3-airflow-2.7.3 (default)
- composer-2.8.3-airflow-2.6.3
Cloud Composer versions 2.3.2, 2.3.1, and 2.3.0 have reached their end of full support period.
Cloud Composer 2.8.3 is a version with an extended upgrade timeline.
June 14, 2024
Environment upgrading is now generally available (GA)
June 05, 2024
The google-cloud-bigquery package version was downgraded from 3.23.1 to 3.20.1 because of the #39541 issue in the public version of Airflow.
The dbt-core and dbt-bigquery packages were upgraded to version 1.8.1.
Cloud Composer 2.8.2 images are available:
- composer-2.8.2-airflow-2.7.3 (default)
- composer-2.8.2-airflow-2.6.3
Cloud Composer versions 2.2.1, 2.2.0, and 2.1.15 have reached their end of full support period.
May 27, 2024
Cloud Composer 2 now supports data lineage for environments that have CMEK enabled.
Cloud Composer 2.8.1 images are available:
- composer-2.8.1-airflow-2.7.3 (default)
- composer-2.8.1-airflow-2.6.3
May 11, 2024
Starting from GKE version 1.27.5, Cloud Composer environment clusters will start using SSD disks as persistent disks. The disk quota will change from Persistent disk standard (GB) to Persistent disk SSD (GB). Please check the Persistent disk SSD (GB) quota in your project and request an increase if this quota approaches its limit.
A single environment created using a Small environment preset requires at least 600 GB SSD disk space and the SSD quota must be able to accommodate it.
Being close to the limit of the SSD quota might impact the autoscaling capabilities of Cloud Composer environments or make impossible to create new environments.
The Logs in Cloud Logging Only feature is enabled by default in new environments:
- New Cloud Composer environments now save Airflow task logs only in Cloud Logging by default.
- Existing environments are not changed. If you upgrade an existing environment to Cloud Composer 2.8.0, it keeps saving logs to the environment's bucket.
- You can enable and disable saving logs to the environment's bucket for an existing environment.
Fixed a problem where some Airflow tasks were failing because the task could not write logs to the environment's bucket.
Cloud Composer 2.8.0 images are available:
- composer-2.8.0-airflow-2.7.3 (default)
- composer-2.8.0-airflow-2.6.3
April 29, 2024
The apache-airflow-providers-google package is upgraded to version 10.17.0. For more information about changes, see the apache-airflow-providers-google changelog from version 10.16.0 to version 10.17.0.
The apache-airflow-providers-cncf-kubernetes package was upgraded to version 8.1.0.
Cloud Composer 2.7.1 images are available:
- composer-2.7.1-airflow-2.7.3 (default)
- composer-2.7.1-airflow-2.6.3
Cloud Composer version 2.1.14 has reached its end of full support period.
April 18, 2024
Python 3.11.8 is available in environments with Airflow 2.6.3 and 2.7.3:
Existing environments with Airflow 2.6.3 and 2.7.3 switch to Python 3.11.8 when upgraded.
Before upgrading, make sure that custom PyPI packages in your environment are compatible with Python 3.11.8.
Between April 16, 2024, 2:00 AM (PST) and April 17, 2024, 3:30 AM (PST), Cloud Composer service experienced problems with environment creation, upgrades, and changing the environment size. The problem is resolved and all operations are working. If you think that your environment is still impacted by this issue, please reach out to the Cloud Support team.
(New environments only) Increased the default value of the [webserver]auto_refresh_interval Airflow configuration option to 15 seconds. Pages in the Airflow UI, such as the list of DAGs, now will update every 15 seconds.
(Available without upgrading) Fixed a problem where enabling or disabling Logs in Cloud Logging Only could render the Airflow web server and workers inoperative. If your environment is affected, apply the fix by enabling or disabling this feature again.
Airflow 2.5.3 is no longer included in Cloud Composer images.
The default version of Airflow is changed to 2.7.3.
Cloud Composer 2.7.0 images are available:
- composer-2.7.0-airflow-2.7.3 (default)
- composer-2.7.0-airflow-2.6.3
Cloud Composer versions 2.1.13, 2.1.12 and 1.20.12 have reached their end of full support period.
April 11, 2024
Starting from June 15, 2024 it will not be possible to create Cloud Composer 1 environments in Google Cloud console. It will still be possible to create Cloud Composer 1 environments through Google Cloud SDK, Terraform, and API in allowlisted projects.
March 26, 2024
The Logs in Cloud Logging only feature is available in all regions:
- In newly created Cloud Composer environments, Airflow task logs are now stored only in Cloud Logging by default.
- You can disable this feature for a new or an existing environment. In this case, Airflow task logs are saved both to Cloud Logging and to the environment's bucket.
The default values of the following Airflow configuration options are changed to provide more optimized Cloud Composer environments. This change applies to new environments with Airflow 2.6.3 and 2.7.3 and to existing environments that do not use custom values for these parameters. For existing environments, the new default values will be used after an upgrade.
[scheduler]job_heartbeat_secto 30[scheduler]scheduler_health_check_thresholdto 60[scheduler]scheduler_heartbeat_secto 15
If you want to override the [scheduler]scheduler_heartbeat_sec option's value, then also adjust the [scheduler]scheduler_health_check_threshold option, as described in Cloud Composer documentation.
Cloud Composer 2.6.6 images are available:
- composer-2.6.6-airflow-2.7.3
- composer-2.6.6-airflow-2.6.3 (default)
- composer-2.6.6-airflow-2.5.3
Cloud Composer versions 2.1.11 and 1.20.11 have reached their end of full support period.
March 25, 2024
Cloud Composer 1 entered the post-maintenance mode. Google will not release any further updates to Cloud Composer 1, including new versions of Airflow, bugfixes, and security updates. We recommend planning migration to Cloud Composer 2.
In Cloud Composer versions from 2.1.0 to 2.6.4, task instances that succeeded in the past can be marked as FAILED in some cases. We recommend to upgrade to Cloud Composer version 2.6.5 or later where this issue is fixed. For more information, see the related known issue.
March 20, 2024
The Logs in Cloud Logging only feature is gradually rolled out to all regions:
- New Cloud Composer environments now save Airflow task logs only in Cloud Logging by default.
- Existing environments are not changed. If you upgrade an existing environment, it keeps saving logs to the environment's bucket.
- You can enable and disable saving logs to the environment's bucket for an existing environment.
Currently the feature is rolled out to the following regions: africa-south1, asia-east1, asia-east2, asia-northeast2, asia-south2, asia-southeast2, australia-southeast2, europe-central2, europe-southwest1, europe-west10, europe-west12, europe-west2, europe-west3, europe-west4, europe-west6, europe-west8, me-central1, me-central2, me-west1, northamerica-northeast2, southamerica-west1, us-east5, us-east7, us-south1, and us-west4.
March 19, 2024
BigQueryInsertJobOperator now correctly handles ephemeral tables created with tableDefinitions.
In BigQueryInsertJobOperator, fixed the handling of parsing errors during Lineage emission when the query is too long or deeply nested.
Airflow 2.7.3 is available in Cloud Composer images.
Fixed an issue where past Airflow task instances could be marked as failed in some cases.
Fixed an issue where Airflow task logs for the first try of a task might not be visible in Airflow UI.
The apache-airflow-providers-google package is upgraded to version 10.16.0 in images with Airflow 2.6.3, and images with Airflow 2.7.3 have this version. For more information about changes, see the apache-airflow-providers-google changelog from version 10.15.0 to version 10.16.0.
The apache-airflow-providers-cncf-kubernetes package was upgraded to version 8.0.1 in images with Airflow 2.6.3, and images with Airflow 2.7.3 have this version.
Cloud Composer 2.6.5 images are available:
- composer-2.6.5-airflow-2.7.3
- composer-2.6.5-airflow-2.6.3 (default)
- composer-2.6.5-airflow-2.5.3
Cloud Composer versions 2.1.10, 2.1.9, 1.20.10, and 1.20.9 have reached their end of full support period.
Cloud Composer 2.6.5 is a version with an extended upgrade timeline.
March 14, 2024
In Airflow 2.6.3 offered in Cloud Composer versions earlier than 2.6.2, task statuses can be deleted as a result of the #31179 Airflow issue. If you use Airflow 2.6.3, we recommend to upgrade to Cloud Composer version 2.6.2 or later where this issue is fixed.
March 13, 2024
All preview versions of Cloud Composer 2 are past their security notifications end date and are not supported. If your environment uses a preview version of Cloud Composer 2, then please upgrade this environment to a supported version or re-create it using the latest version of Cloud Composer 2.
If you see frequent scheduler restarts in your Airflow 2.6.3 environment and the [scheduler]job_heartbeat_sec Airflow configuration option is set to a non-default value, you can fix this issue either by upgrading to Cloud Composer version 2.6.4 or by removing this option's override, so that it uses the default value.
March 12, 2024
Fixed creation and upgrades in environments that have environment variables with special symbols.
Cloud Composer 2.6.4 images are available:
- composer-2.6.4-airflow-2.6.3 (default)
- composer-2.6.4-airflow-2.5.3
Cloud Composer versions 2.1.8 and 1.20.8 have reached their end of full support period.
March 05, 2024
Cloud Composer 2.6.3 release was rolled back. If you have an environment that was created with a composer-2.6.3-airflow-* image, you can later upgrade it to a newer version.
February 28, 2024
Cloud Composer 2.6.3 release started on February 28, 2024. Get ready for upcoming changes and features as we roll out the new release to all regions. This release is in progress at the moment. Listed changes and features might not be available in some regions yet
(Cloud Composer 2 in select regions) Cloud Composer shows the account selection and consent screens when opening Airflow UI for an environment.
If you have the "Don't allow users to access any third-party apps" setting enabled in Google Workspace, then configure access to the "Apache Airflow in Cloud Composer" app in Google Workspace, so that your Google Workspace users can still access Airflow UI in Cloud Composer environments. For more information, see Allow access to Airflow UI in Google Workspace.
(Cloud Composer 2 in select regions) Reduced the propagation time of the revoked Cloud IAM permission that blocks access to Airflow UI.
In new environments with Airflow 2.6.3, the default values of the following Airflow configuration options are changed to provide more optimized Cloud Composer environments:
[scheduler]job_heartbeat_secto30[scheduler]scheduler_heartbeat_secto15
Fixed a problem where the IAM policy of a custom environment's bucket is replaced when an environment is created.
The apache-airflow-providers-google package is upgraded to version 10.15.0 in images with Airflow 2.6.3. For more information about changes, see the apache-airflow-providers-google changelog from version 10.14.0 to version 10.15.0.
Cloud Composer 2.6.3 images are available:
- composer-2.6.3-airflow-2.6.3 (default)
- composer-2.6.3-airflow-2.5.3
February 26, 2024
Starting February 27, 2024, in the us-central1, europe-west1, europe-west2, europe-west3, europe-west6, us-east1, and us-east4 regions it is possible to create new Cloud Composer 1 environments only in projects that already have Cloud Composer 1 environments.
In all other existing or newly created projects in these regions, it is possible to create only Cloud Composer 2 environments. This change is a part of the preparation for Cloud Composer 1 end of support, as communicated earlier and described in the Versioning overview.
February 22, 2024
Fixed a problem where one DAG run could potentially delete task instances from other DAG runs if run_id was the same (backported #32684 from a later Airflow version).
Cloud Composer 2.6.2 images are available:
- composer-2.6.2-airflow-2.6.3 (default)
- composer-2.6.2-airflow-2.5.3
February 15, 2024
Starting February 16, 2024, in the asia-east2, asia-northeast1, asia-northeast2, asia-northeast3, asia-south1, and australia-southeast1 regions it is possible to create new Cloud Composer 1 environments only in projects that already have Cloud Composer 1 environments.
In all other existing or newly created projects in these regions, it is possible to create only Cloud Composer 2 environments. This change is a part of the preparation for Cloud Composer 1 end of support, as communicated earlier and described in the Versioning overview.
The apache-airflow-providers-google package is upgraded to version 10.14.0 in images with Airflow 2.6.3. For more information about changes, see the apache-airflow-providers-google changelog from version 10.13.1 to version 10.14.0.
Improved the reliability of syncing Airflow tasks logs to the environment bucket. This fix addresses the issue with storing Airflow task log files, which affected environments in some cases.
Improved the environment component responsible for metrics reporting (composer-monitoring) to minimize the restarts of this component.
Cloud Composer 2.6.1 images are available:
- composer-2.6.1-airflow-2.6.3 (default)
- composer-2.6.1-airflow-2.5.3
Cloud Composer versions 2.1.6 and 1.20.6 have reached their end of full support period.
February 09, 2024
Starting January 20, 2024, in the asia-southeast1, us-west3, and us-west4 regions it is possible to create new Cloud Composer 1 environments only in projects that already have Cloud Composer 1 environments.
In all other existing or newly created projects in these regions, it is possible to create only Cloud Composer 2 environments. This change is a part of the preparation for Cloud Composer 1 end of support, as communicated earlier and described in the Versioning overview.
February 08, 2024
The enabling and disabling functionality for the Logs in Cloud Logging only feature was temporarily rolled back.
At the moment, it is not possible to enable or disable this feature, and your environment will keep its current configuration. If this feature is enabled, Cloud Composer will keep saving logs to Cloud Logging only. Newly created environments save logs to Cloud Logging only and the environment's bucket.
We will announce when the issue is resolved.
February 05, 2024
Data lineage is now generally available (GA) in Cloud Composer 2.
Data lineage integration is now enabled by default in newly created environments with Cloud Composer version 2.1.2 and later, if Data Lineage API is enabled in the environment's project. Existing and upgraded environments keep their current configuration.
Python 3.11 is available in environments with Airflow 2.6.3:
New environments with Airflow 2.6.3 use Python 3.11. Python 3.8 is no longer available in new environments with Airflow 2.6.3.
Existing environments with Airflow 2.6.3 switch to Python 3.11 when upgraded. Before upgrading, make sure that custom PyPI packages in your environment are compatible with Python 3.11.
New and upgraded environments with Airflow 2.5.3 keep using Python 3.8.
Cloud Composer versions earlier than 2.6.0 keep using Python 3.8.
Airflow worker memory requirements in Python 3.11 are 10% higher compared to workers in Python 3.8. If you use custom settings for Airflow worker CPU and memory limits, then Airflow workers in your environment might enter the CrashLoopBackOff status and stop executing tasks, if resource consumption goes above the limit.
If your environment is impacted, see the related known issue for possible solutions: Workers require more memory than in previous Airflow versions.
The default worker_concurrency formula was adjusted in Airflow 2.6.3 and later versions to accommodate this change.
(Available without upgrading) The default worker_concurrency in Airflow 2.6.3 and later versions is now calculated using a different formula. For more information, see Change worker concurrency.
Connections to the Redis environment component are now additionally secured with a password. Improved the reliability of the environment component responsible for metrics reporting (airflow-monitoring).
The apache-airflow-providers-google package is upgraded to version 10.13.1 in images with Airflow 2.6.3. For more information about changes, see the apache-airflow-providers-google changelog from version 10.12.0 to version 10.13.1.
Cloud Composer 2.6.0 images are available:
- composer-2.6.0-airflow-2.6.3 (default)
- composer-2.6.0-airflow-2.5.3
Cloud Composer versions 2.1.5 and 1.20.5 have reached their end of full support period.
Cloud Composer 2.6.0 is a version with an extended upgrade timeline.
January 31, 2024
In the first half of February, 2024, Cloud Composer 2 environments with Airflow 2.6.3 will start using Python 3.11:
- New and upgraded environments with Airflow 2.6.3 will switch to Python 3.11.
- New and upgraded environments with Airflow 2.5.3 will still use Python 3.8.
- Python 3.8 will no longer be available in new versions of Cloud Composer with Airflow 2.6.3 (and later versions of Airflow).
- Existing environments with Airflow 2.6.3 will keep using Python 3.8 until they are upgraded.
- Cloud Composer versions released before this change will keep using Python 3.8.
January 25, 2024
The Logs in Cloud Logging Only feature is now generally available (GA):
- New Cloud Composer environments now save Airflow task logs only in Cloud Logging by default.
- Existing environments are not changed. If you upgrade an existing environment to Cloud Composer 2.5.5, it keeps saving logs to the environment's bucket.
- You can enable and disable saving logs to the environment's bucket for an existing environment.
Increased the amount of memory available to the Redis component in environments with Medium and Large environment sizes. This change improves the performance of Cloud Composer environments.
Sensitive Airflow configuration options are now hidden in Airflow UI by default. The default value of the [webserver]expose_config option is now set to non-sensitive-only.
(Available without upgrading) In VPC Service Controls setups, device policy attributes in access levels are now respected when accessing the Airflow web server.
(Available without upgrading) Auxiliary Cloud Pub/Sub topics and subscriptions used by save and load snapshot operations are now cleaned up if these operations fail.
Cloud Composer versions 2.1.4, 2.1.3, 1.20.4, and 1.20.3 have reached their end of full support period.
January 08, 2024
Environment snapshots now correctly work with the bring your own bucket (BYOB) feature.
(Available without upgrading) Cloud Composer now creates an audit log on a successful completion of create and update environment operations.
If an environment update operation fails, Airflow configuration overrides that were applied by this operation are rolled back.
Cloud Composer 2.5.4 images are available:
- composer-2.5.4-airflow-2.5.3
- composer-2.5.4-airflow-2.6.3 (default)
The apache-airflow-providers-google package is upgraded to version 10.12.0 in images with Airflow 2.6.3 and 2.5.3. For more information about changes, see the apache-airflow-providers-google changelog from version 10.11.1 to version 10.12.0.
Cloud Composer versions 2.1.2 and 1.20.2 have reached their