XCom allows tasks to exchange small amounts of data by storing them in the Airflow metadata database. An XCom is essentially a key-value pair associated with a specific task instance, DAG, and execution date. The identifier for the data (e.g., filename ).
def task1(**kwargs): # Share data through XCom kwargs['ti'].xcom_push(key='customer_data', value=[1, 2, 3])
[core] xcom_backend = include.custom_xcom_backend.S3XComBackend Use code with caution. 4. Advanced XCom Patterns with TaskFlow API airflow xcom exclusive
In a downstream task, you pull the value:
In this guide, we will explore how to manage data sharing within your DAGs using XComs to ensure your pipelines remain efficient, secure, and easy to debug. What are Airflow XComs? XCom allows tasks to exchange small amounts of
The 48KB constraint means XCom is , not large datasets. Attempting to push a large JSON payload, a Pandas DataFrame, or any substantial data structure will likely trigger a database error or severely degrade performance.
Historically, Airflow allowed XCom values to be serialized using Python's pickle module, which could lead to security vulnerabilities and version incompatibilities. Modern Airflow , and pickling support is deprecated. Always ensure your XCom values are JSON‑serializable unless you have a very good reason to do otherwise. def task1(**kwargs): # Share data through XCom kwargs['ti']
Modern Airflow (2.0+) makes XCom seamless.
To get the most out of Airflow XCom exclusive, follow these best practices:
from airflow.decorators import dag, task from datetime import datetime @dag(start_date=datetime(2023,1,1), schedule=None, catchup=False) def xcom_taskflow_api(): @task def push_task(): return "secret_data_123" @task def pull_task(value): print(f"Pulled value: value") # TaskFlow automatically handles the XCom transfer pulled_value = push_task() pull_task(pulled_value) xcom_taskflow_api() Use code with caution. 3. The "Exclusive" Limitations of XComs
+--------------------+ Implicit/Explicit Return +----------------------+ | | -----------------------------------> | | | Upstream Task A | | Airflow Metadata DB | | | <----------------------------------- | (xcom table) | +--------------------+ .xcom_pull(task_ids) +----------------------+ | | .xcom_pull() v +----------------------+ | | | Downstream Task B | | | +----------------------+ Implicit vs. Explicit XComs