Back to projects
Project
DPPS — Scalable Data Processing Platform
Repository link not yet added.
Overview
Problem
Structured and semi-structured data arriving from REST APIs and relational databases needed to be processed at high throughput, with a full pipeline lifecycle that could be trusted to fail safely and alert when something broke.
Solution
Engineered a distributed ETL system on PySpark and Apache Airflow, with DAGs covering the full lifecycle — ingestion, transformation, validation, and warehouse loading — plus automated retry and alerting logic, and data quality controls built against defined SLA/SLO reliability targets.
Architecture & Data Flow
REST APIs / Relational DBs
PySpark Ingestion
Transformation
Validation (Data Quality)
Warehouse Loading
Retry & Alerting
Technologies
PySparkApache AirflowREST APIsRelational Databases
Key Engineering Decisions & Implementation
- Engineered a distributed ETL system on PySpark and Apache Airflow to process structured and semi-structured data from REST APIs and relational databases at high throughput.
- Designed Airflow DAGs covering the full pipeline lifecycle: ingestion → transformation → validation → warehouse loading, with automated retry and alerting logic.
- Implemented data quality controls achieving pipeline reliability targets aligned to defined SLA/SLO thresholds.
Key Engineering Concepts
Distributed processingAirflow DAG orchestrationData quality controlsSLA / SLO reliability targetsAutomated retry & alerting