Change events stream in through Kafka, land in a schema-enforced Delta lakehouse, pass a quality and governance gate, and become a searchable index that answers questions about live inventory.
Simulated event flow — the stages, the branch and the failure behaviour are the real pipeline's.
The same event, at three points in the pipeline. Nothing is thrown away — each layer adds what the next one needs and removes what it no longer should carry.
A single similarity search answers the easy questions and quietly fails the important ones.
Pick a question and watch it move through the cascade. An exact product code never reaches the model at all — and the reranker does not just rescore the shortlist, it reorders it.
Validation, PII protection and lineage run as first-class stages, ahead of the write — not as reports somebody reads the morning after.
Redacted once, before storage, logs and the search index — instead of filtered again in every consumer that touches the data.
emit_lineage("bronze_write", job_name="ingest_bronze") emit_lineage("silver_write", job_name="clean_silver") emit_lineage("gold_write", job_name="curate_gold") [OpenLineage] emitted RunEvent job=ingest_bronze state=COMPLETE [OpenLineage] emitted RunEvent job=clean_silver state=COMPLETE [OpenLineage] emitted RunEvent job=curate_gold state=COMPLETE
Every hop emits a real RunEvent to a Marquez-compatible backend, and falls back to a console log when none is reachable — observability that never becomes the reason a run fails.
The quality gate sits ahead of the lakehouse write, so a failed batch stops before it can reach silver and gold.
dag_id: ecommerce_pipeline · schedule: hourly · catchup: false