Airflow 2.0 introduced the ability to swap the XCom backend. This changes the game regarding the "Size Limit" constraint mentioned above.
Inside your DAG, push with a unique key per execution date: airflow xcom exclusive
. This allows you to store the actual data "exclusively" in external object storage while only keeping a reference in the Airflow DB. Apache Airflow Object Storage Backend : You can configure Airflow to use Google Cloud Storage Azure Blob Storage Implementation : To build a custom one, you must subclass and override the serialize_value deserialize_value Thresholding : You can set a size threshold (e.g., xcom_objectstorage_threshold Airflow 2