Data Engineering
Customer Transactions Data Pipeline
Built an end-to-end data engineering pipeline in Databricks using a Medallion Architecture with Bronze, Silver, and Gold layers. The pipeline includes data ingestion, transformation, data quality validation, dirty data handling, audit logging, workflow orchestration, and email notification.
- Databricks
- PySpark
- Delta Lake
- SQL
- Databricks Workflows