Saurabh PatilData Engineer

Problem

Analytical workloads were running on a legacy SQL Server (RDS) database — batch-dependent, difficult to scale elastically, and slow on larger queries as analytical demand grew.

Solution

Migrated the analytical workload to Snowflake with optimized table structures, and replaced batch pipelines with an event-driven ingestion framework built on AWS Lambda, Step Functions, EventBridge, and Snowpipe. Snowflake Streams and Tasks automate incremental data capture (CDC) and scheduled transformations, and the transformation layer is a set of modular, tested dbt models version-controlled in Git.

Problem

Structured and semi-structured data arriving from REST APIs and relational databases needed to be processed at high throughput, with a full pipeline lifecycle that could be trusted to fail safely and alert when something broke.

Solution

Engineered a distributed ETL system on PySpark and Apache Airflow, with DAGs covering the full lifecycle — ingestion, transformation, validation, and warehouse loading — plus automated retry and alerting logic, and data quality controls built against defined SLA/SLO reliability targets.

Problem

Multiple educational institutions had operational data scattered across separate systems, making unified, timely reporting difficult.

Solution

Delivered end-to-end ETL pipelines that consolidate operational data from each institution into centralized SQL databases for unified reporting, using Azure Functions for event-driven ingestion so systems stay synchronized in near real time.

Problem

HR teams were manually exporting employee records to CSV every month, which was slow and left room for inconsistent, error-prone payroll data.

Solution

Built a Python-based pipeline that pulls employee records directly via the Zoho REST API, transforms the raw JSON payloads into payroll-ready datasets with Pandas, and validates the output before it reaches payroll processing.