In a world of fragmented data stacks, you need workflow orchestration that is innovative, portable, and extensible. Cloud Composer, our fully managed data and AI/ML workflow orchestration service, relies on Apache Airflow to provide a universal control plane. And today, we’re excited to share that Cloud Composer now supports Airflow 3.1 in preview.
Released in November, this update marks a significant milestone for the platform, and the first time a hyperscaler is offering Airflow 3.1.
Built on the foundation of Airflow 3
The new features in Airflow 3.1 are built on top of the revolutionary architecture introduced in Airflow 3.0:
-
Decoupled architecture: Robust separation between the scheduler and execution layer for better scalability and security
-
DAG versioning: Native support for automated DAG versioning, retaining the historical structure and run history even after you remove tasks or change logic
-
Powerful managed backfills: A redesigned backfill system that is now a first-class citizen, fully managed by the scheduler
-
Event-driven scheduling and data assets: Enhanced capabilities for triggering workflows based on assets as well as external events, like messages arriving in a message queue
-
and many more…
For a deeper dive into the architectural shifts of Airflow 3, you can read our previous announcement: Next Gen Data Pipelines: Airflow 3 Arrives on Google Cloud Composer.
What’s new in Airflow 3.1 on Composer?
We’ve invested heavily to ensure the new capabilities in Airflow 3.1 are robust and reliable on Google Cloud. Specifically, Airflow 3.1 introduces enhancements designed to increase oversight, improve reliability, and support global teams:
1. Human-in-the-Loop (HITL) workflows
As AI agents and automated pipelines become more complex, the need for human oversight increases. Airflow 3.1 introduces powerful Human-in-the-Loop (HITL) functionality that allows workflows to pause execution in a deferred state and wait for a person to make a decision via the Airflow UI or API call.
HITL puts you in control, whether you are approving a deployment, reviewing a generative AI model’s output, or providing feedback to steer a pipeline.
To make this process even smoother, HITL operators integrate natively with Airflow Notifiers. You can configure a notifier to automatically send a message via Slack, email, or PagerDuty the moment a task pauses for input. Using these helper methods, the notification can include a direct link to the specific approval page, allowing stakeholders to respond immediately without searching through the UI.
2. Deadline Alerts
Managing Airflow Task SLAs has evolved. Airflow 3.1 replaces legacy mechanisms with smarter, proactive Deadline Alerts. You can now set specific time-based thresholds for your Dags and tasks relative to a start time or logical date.
If a critical pipeline such as a long-running ML training job exceeds its expected duration, Airflow will proactively notify you via standard notifiers. This ensures you can identify potential delays before they impact your downstream goals.
3. Native multi-language support (Internationalization)
Many data engineering teams are global, and team members speak different languages. Airflow 3.1 brings a fully localized experience to the modern React-based UI, supporting 17 languages including Spanish, French, Polish, German, Chinese (Simplified & Traditional), Japanese, and Portuguese.
4. Modern extensibility and developer experience
Beyond the core orchestration capabilities, Airflow 3.1 introduces multiple architectural improvements designed to enhance the developer experience. These updates empower teams to extend the platform’s UI and build more responsive, synchronous applications.
-
React plugin system: Embed custom dashboards and views directly into the UI.
-
Inference execution: A new streaming API endpoint for watching synchronous DAGs until completion.
Why open orchestration wins over walled gardens
Some orchestration platforms attempt to consolidate everything into a single “walled garden” with ingestion and orchestration wrapped into a proprietary, opaque utility. But proprietary tools can suffer from feature lag as you wait for the vendor to address new use cases. Airflow has emerged as the industry standard for orchestration, so that you can be at the forefront of the market.
At Google, we are deeply committed to fostering this ecosystem — not just as a platform provider, but as active contributors to the Airflow codebase. Cloud Composer 3 represents the maturation of managed open source, balancing the security of a hardened perimeter with the near-infinite extensibility of open standards. With this approach, you get:
-
Community-led innovation: Feature velocity shouldn’t be limited by a single corporation’s R&D budget. With Airflow 3.1, you leverage the collective innovation of thousands of global contributors, providing fast access to a vast ecosystem of community-built providers.
-
Freedom from vendor lock-in: Unlike proprietary platforms where your logic is tied to a specific vendor’s ecosystem, Cloud Composer runs on standard Apache Airflow. Your orchestration code remains portable Python.
-
Extensibility: Enterprises often manage a “long tail” of legacy integrations and niche tools. Instead of waiting for a vendor to build a connector, Composer allows you to write custom Python operators to interface with any system — from proprietary internal APIs to on-prem hardware — giving you strong connectivity without the “roadmap blockage” typical of closed tools.
Get started today
The features listed above — from HITL workflows to proactive Deadline Alerts — are a testament to the innovation behind Apache Airflow. Cloud Composer 3 with Airflow 3.1 is available now in preview. We invite you to create a new environment and explore these new capabilities today.
