The Senior Data Engineer will be a key contributor within the Data Products Delivery Area, responsible for designing, building, and operating scalable, secure, and high-quality data pipelines that power Business Intelligence & Analytics and AIML products. This role supports a Databricks Data Lakehouse using Medallion Architecture, built on Data Fabric principles, and operates within a release-based Agile framework.

This role requires strong hands-on development using Python, PySpark, SQL, and APIs, and experience integrating diverse enterprise and external data sources.

· Design, develop, and optimize end-to-end data pipelines using Python and PySpark.

· Build and maintain data ingestion frameworks leveraging REST APIs, streaming APIs, and batch interfaces.

· Implement Medallion Architecture (Bronze, Silver, Gold) within a Databricks Lakehouse.

· Integrate data from structured, semi-structured, and unstructured sources, including API-based and event-driven sources.

· Apply Data Fabric principles including metadata-driven ingestion, lineage, observability, and reusability.

· Collaborate with Senior Data Architects on logical and physical data models.

· Deliver data assets as releasable products

· Support downstream consumers including:

o BI&A dashboards and visualizations

o AIML feature engineering, training, and inference pipelines

o Data services and curated datasets exposed via APIs

· Decompose requirements into well-defined user stories aligned with MoSCoW prioritization.

· Implement data validation, reconciliation, and quality checks within PySpark pipelines.

· Ensure secure handling of PII and sensitive data, including encryption, masking, and access controls.

· Partner with CloudOps (DevSecOps, FinOps, InfraOps) to ensure:

o Secure API access and secrets management

o CI/CD automation for Python and PySpark workloads

o Cost-optimized compute and storage usage

· Support O&M for production data pipelines, APIs, and analytics products.

· Tune Spark jobs, SQL queries, and API integrations for performance and reliability.

· Implement monitoring, logging, and alerting for data workflows.

· Drive automation and refactoring to improve resiliency and scalability.

· Mentor junior and mid-level data engineers on Python, PySpark, and API integration best practices.

· Work closely with:

o Business Analysts & Data Architects

o Data Scientists & ML Engineers

o BI Developers

o Automation Test Engineers

· Contribute to reusable frameworks, coding standards, and engineering best practices.

· Strong hands-on experience with:

o Python for data engineering and API integration

o PySpark / Apache Spark for large-scale data processing

o SQL (advanced querying and performance tuning)

· Proven experience building API-driven data ingestion pipelines (REST, JSON, OAuth, pagination, throttling).

· Strong experience with Databricks, Delta Lake, and Lakehouse Architecture.

· Experience implementing Medallion Architecture in production environments.

· Experience working in cloud platforms (AWS, Azure, or GCP).

· Familiarity with:

o CI/CD pipelines for Python and Spark workloads

o Infrastructure as Code concepts

o Secure API authentication and secrets management

· Cost-conscious engineering mindset (FinOps exposure is a plus).

· Experience supporting BI&A workloads at scale.

· Understanding of data modeling for analytics and reporting.

· Experience enabling data consumption for AI/ML use cases.

· Proven experience working in Agile Scrum / Scrum of Scrums environments.

· Comfortable working with release-based delivery and production support.

· Experience exposing data via internal or external APIs.

· Familiarity with streaming technologies (Kinesis, Firehose, Kafka, Event Hubs, etc.).

· Experience with data governance, cataloging, and lineage tools.

· Exposure to feature stores or ML data pipelines.

· Strong analytical and problem-solving skills.

· Excellent communication and collaboration abilities.

· Ownership mindset with a focus on reliability and quality.