Airflow Xcom Exclusive -

t1 = PythonOperator( task_id='task_1', python_callable=task1, dag=dag, )

: Only pass metadata (IDs, dates, paths) via XCom. Use them as "pointers" to larger data stored elsewhere.

from airflow.decorators import dag, task from datetime import datetime @dag(start_date=datetime(2023,1,1), schedule=None, catchup=False) def xcom_taskflow_api(): @task def push_task(): return "secret_data_123" @task def pull_task(value): print(f"Pulled value: value") # TaskFlow automatically handles the XCom transfer pulled_value = push_task() pull_task(pulled_value) xcom_taskflow_api() Use code with caution. 3. The "Exclusive" Limitations of XComs airflow xcom exclusive

: Every XCom is uniquely identified by its dag_id , task_id , run_id , and a specific key .

You can manually call the xcom_push method from the task instance. dag = DAG( 'xcom_example'

dag = DAG( 'xcom_example', default_args=default_args, schedule_interval=timedelta(days=1), )

with DAG( "fraud_detection", xcom_exclusive_keys= "fetch_transactions": ["raw_txns"], "validate": ["valid_txns", "error_count"], "feature_engineering": ["features"], "fraud_model": ["score"], , xcom_backend="myapp.xcom.S3ExclusiveXCom", ) as dag: ) with DAG( "fraud_detection"

: By default, XComs are accessible by any task within the same DAG run, but they aren't meant for massive datasets (like large CSVs); for those, external storage like S3 is preferred. Best Practices for an XCom-Heavy Workflow

When using explicit XComs ( xcom_push ), avoid generic keys like data or output . Use hyper-specific, unique naming conventions ( processed_customer_count_v1 ) to prevent downstream collision issues. Final Thoughts

Use these strategies depending on your requirement: