Responsibilities :
Key Responsibilities: Data Engineering & Delivery
Lead end-to-end development of data pipelines using ADF for orchestration and Databricks for scalable processing
Design and implement robust ETL/ELT workflows, ensuring data quality, reliability, and maintainability
Develop optimized transformations and jobs using PySpark in Databricks for batch and incremental processing
Build reusable frameworks, templates, and standards for pipeline development and deployment Architecture & Performance
Define solution architecture for ingestion, transformation, and serving layers aligned to platform best practices
Tune Spark jobs for performance and cost efficiency (partitioning, caching, shuffle optimization, file sizing)
Establish monitoring, alerting, and operational runbooks for production pipelines Leadership & Collaboration
Provide technical leadership, code reviews, and mentoring to ensure high engineering standards
Collaborate with stakeholders to translate business requirements into scalable data solutions
Drive delivery planning, estimation, and risk management for data engineering initiatives Minimum Qualifications:
BTECH, MTECH, MCA, MSC (or equivalent) in Computer Science, Engineering, or related field
7â€9 years of experience in data engineering with strong hands-on delivery ownership
Strong expertise in Azure Data Factory (ADF) for pipeline orchestration, scheduling, and integration patterns
Strong expertise in Databricks for building scalable data processing solutions
Hands-on proficiency with PySpark for building and optimizing distributed data transformations
Experience building production-grade pipelines with logging, error handling, and operational support readiness
Additional Responsibilities:
Preferred Qualifications:
Experience designing medallion/layered data architectures and implementing reusable transformation patterns in Databricks
Strong understanding of data modeling concepts and building curated datasets for analytics consumption
Experience implementing CI/CD practices for data pipelines and notebooks, including automated testing and deployment
Proven ability to lead technical discussions, mentor team members, and drive engineering best practices
Experience improving observability (metrics, alerts, dashboards) and reducing pipeline failures through proactive monitoring