Google Cloud Composer

What is Cloud Composer?

Cloud Composer is a managed workflow orchestration service that is built on Apache Airflow, a workflow management platform. Developers use Cloud Composer to author, schedule and monitor software development pipelines across clouds and on-premises data centers. Also, users can create Airflow environments and use Airflow-native tools. Cloud Composer has two major versions:

• Cloud Composer 1. This version has manual scaling and its environments are zonal.
• Cloud Composer 2. This version has autoscaling environments, as well as a zonal Apache Airflow metadata database and regional Airflow scheduling and execution layer.

Within Cloud Composer, a library of connectors and multiple graphical representations are available through one-click deployment. Cloud Composer pipelines are organized as directed acyclic graphs (DAGs) to accommodate Python files and lower the entry barrier to author and schedule workflows. An Airflow DAG contains a DAG definition, operators and operator relationships.

Cloud Composer key features

Google Cloud Composer has a number of key features that are beneficial to the user:

• Fully managed. Rather than spending time provisioning resources, Cloud Composer's managed nature allows IT teams to focus on authoring, scheduling and monitoring.
• Cross-cloud capabilities. With Cloud Composer, users can create workflows that connect data, processing and services across clouds. It is also compatible with services within the Google Cloud Platform (GCP), other public clouds and on-premises environments that enable multi-cloud strategy.
• Open source. Because Cloud Composer is a managed Apache Airflow service, it is free from lock-in and portability.
• Integrated. Cloud Composer integrates with tools such as BigQuery, Dataflow, Dataproc, Datastore, Cloud Storage, Pub/Sub, AI Platform and more. This gives users the ability to orchestrate end-to-end workloads.
• Python. Python is a popular, general-purpose language. It was used to build Apache Airflow.

While these features provide many benefits, it can also lead to some drawbacks. For example, because it is open source, there is only community support and no premium paid option. There is also no long-term support guarantee. Additionally, while Airflow is a popular orchestration tool, it is technically complex and can require specialized knowledge. Also, with two different versions of the service, pricing is difficult.

Cloud Composer pricing

Google Cloud Composer charges customers based on consumption, ensuring they only pay for what they use. The service charges in 10-minute intervals. The version 1 pricing, based on the Oregon (us-west1) region, is broken down to include the following:

• Web core hours at $0.074 per vCPU hour.
• Database core hours at $0.125 per vCPU hour.
• Web and database storage at $0.273 per gigabyte, per month.
• Network Egress at $0.156 per gigabyte.

The version 2 pricing differs because of environment autoscaling.

• Cloud Composer Compute CPUs at $0.045 per 1000 mCPU hours.
• Cloud Composer Compute Memory at $0.005 per gibibyte, per hour.
• Cloud Composer Compute Storage at $0.0002 per gibibyte, per hour.
• Cloud Composer Database Storage at $0.17 per gibibyte, per month.
• Cloud Composer Environment Fee at $0.35 to $0.85 per hour, depending on the size.

Editor's note: This article was originally written in 2019. TechTarget editors revised it in 2022 to improve the reader experience.