Skip to content
Back to projects

Project

KIPP Baltimore — Cloud Data Warehouse Modernization

Repository link not yet added.

Overview

Problem

Analytical workloads were running on a legacy SQL Server (RDS) database — batch-dependent, difficult to scale elastically, and slow on larger queries as analytical demand grew.

Solution

Migrated the analytical workload to Snowflake with optimized table structures, and replaced batch pipelines with an event-driven ingestion framework built on AWS Lambda, Step Functions, EventBridge, and Snowpipe. Snowflake Streams and Tasks automate incremental data capture (CDC) and scheduled transformations, and the transformation layer is a set of modular, tested dbt models version-controlled in Git.

Architecture & Data Flow

SQL Server (RDS)
Lambda / Step Functions / EventBridge
Snowpipe Ingestion
Snowflake RAW
Streams & Tasks (CDC)
dbt Staging → Marts
BI / Analytics

Technologies

SQL Server (RDS)SnowflakeAWS LambdaAWS Step FunctionsAWS EventBridgeSnowpipedbtGitIAMVPC

Key Engineering Decisions & Implementation

  • Migrated analytical workloads from SQL Server (RDS) to Snowflake, designing optimized table structures that reduced query execution time and enabled elastic scaling for growing analytical demand.
  • Built an event-driven ingestion framework using AWS Lambda, Step Functions, EventBridge, and Snowpipe, replacing batch-dependent pipelines with resilient near-real-time orchestration.
  • Implemented Snowflake Streams and Tasks to automate incremental data capture and scheduled transformations, reducing manual intervention in pipeline execution.
  • Developed modular dbt models with automated testing suites and inline documentation, version-controlled in Git, reducing model maintenance time and accelerating onboarding for new contributors.
  • Enforced security and governance via IAM roles, VPC segmentation, and bastion host access; implemented monitoring dashboards for pipeline health and resource utilization.

Key Engineering Concepts

CDC (Change Data Capture)Incremental loadingEvent-driven orchestrationdbt modeling & testingData governance (IAM / VPC)Monitoring
Back to all projects