Mustafa Mirza

• Data Architecture Development: Spearheaded design and implementation of a scalable, modular data architecture leveraging Hudi, Open Metadata, and orchestrating workflows with Airflow coupled with Cosmos. Designed ingestion pipelines using Spark and Flink to handle high-volume, real-time data streams, ensuring seamless integration and transformation across diverse data sources.
• Data Strategy: Developed and executed long-term data strategies that prioritized stakeholder-driven use cases, addressing scalability and modularity while enabling adaptability across cloud service providers (AWS, GCP, and Azure). Achieved a potential 50% reduction in costs compared to Databricks through innovative modular designs.
• Analytics Enablement: Deployed Superset to support advanced business intelligence (BI) capabilities, empowering teams with real-time data exploration and custom dashboards for actionable insights.
• Data Drift and Schema Evolution: Addressed challenges in data drift and schema evolution using DBT and Trino, ensuring continuous alignment between dynamic business needs and data pipelines.
• Cost Optimization: Designed a modular platform strategy compatible with any cloud service provider, achieving a 50% cost reduction potential compared to Databricks through tool optimization and platform scalability.
• Data Governance: Established robust frameworks with RBAC, PII compliance, and centralized governance policies using Apache Ranger, ensuring secure, scalable, and compliant data management in alignment with SBP guidelines.
• Data Platform Enablement: Architected a data platform with high-performance compute resources and scalable storage, automating ingestion pipelines with Airflow and Cosmos, supporting real-time processing and reporting workflows.
• Leadership: Built a high-performing engineering team skilled in Spark, Flink, DBT, and Hudi, fostering innovation in areas like data analytics and platform automation.

• Spearheaded initiatives to enhance Data Quality within our Data Warehouse, using DBT for lineage maintenance and regular testing to ensure data freshness/correctness.
• Manage and Build scalable Data pipelines using Apache Hudi and Spark with a reliable underlying architecture on AWS EKS and EMR, ingesting data with volumes up to 100+ terabytes of data and Karpenter Enabled Scaling for unforeseen increases in data traffic.
• Data scheduling using Airflow and DBT to create and maintain up to 400+ Data products with Data Lineage to ensure no upstream errors affect downstream.
• Analytics platform using Apache Superset and Tableau for data exploration as well as visualization with up to 1000+ dashboards created by users enabled by the platform.
• Architecture monitoring using Prometheus and Loki to export JVM/service metrics to Grafana where we designed easy-to-read intuitive dashboards for service health and uptimes.
• Implemented Apache Ranger to ensure GDPR compliance as well as fine-grained Data governance throughout our users such that data is only exposed to those with correct clearance level.
• Implemented cost-saving measures on AWS by optimizing Spark jobs, monitoring resources with CloudWatch, leveraging Karpenter auto-scaling to optimize resource utilization, optimizing Spark jobs to reduce NAT gateway usage, and optimizing S3 operations using metadata tables.
• Achieved $12,000/month cost reduction.
• Developed a DAAS service to serve data to our applications reliably using Go-lang powered by Trino querying engine and Apache Pinot as our OLAP store.

GPA: 3.48gpa