Databricks Developer

About Position:

We are looking for an experienced Databricks Developer / Lead who can build, optimize, and maintain scalable data platforms and enterprise-grade data pipelines using Databricks and Spark technologies. The ideal candidate will have strong expertise in distributed data processing, performance tuning, Delta Lake architecture, and cloud-native data engineering solutions.

• Role: Databricks Developer
• Location: Pune
• Experience: 7 to 12 Years
• Job Type: Full Time Employment

What You'll Do:

• Develop and maintain scalable ETL/ELT pipelines using Databricks, PySpark, SQL, and Notebooks.
• Design and implement end-to-end data processing solutions using Delta Lake architecture.
• Build and manage Bronze, Silver, and Gold layer data pipelines for enterprise data platforms.
• Develop batch and real-time streaming solutions using Spark Structured Streaming.
• Perform Spark performance tuning, workload optimization, and resource utilization improvements.
• Optimize Databricks clusters, jobs, and infrastructure to improve performance and control operational costs.
• Collaborate with data architects, business analysts, data scientists, and stakeholders to deliver scalable data solutions.
• Ensure data quality, validation, reconciliation, and compliance with enterprise governance standards.
• Integrate Databricks solutions with Azure, AWS, or GCP cloud-native services.
• Implement and maintain CI/CD pipelines for automated deployment and release management.
• Utilize Git and version control best practices for collaborative development.
• Monitor data jobs, troubleshoot failures, and resolve cluster and environment-related issues.
• Support data lake modernization, cloud migration, and enterprise analytics initiatives.
• Participate in architecture reviews, design discussions, and technical mentoring activities.
• Drive best practices around scalability, reliability, observability, and operational excellence.

Expertise You'll Bring:

• 7 to 12 years of experience in Data Engineering, Analytics, Big Data, and Data Platform development.
• Hands-on delivery experience across 6+ Databricks implementation projects.
• Strong expertise in Databricks architecture, cluster management, and platform optimization.
• Advanced knowledge of Apache Spark, PySpark, Spark SQL, and distributed computing concepts.
• Extensive experience building enterprise-scale ETL/ELT pipelines and data integration frameworks.
• Deep understanding of Delta Lake, Lakehouse Architecture, and modern data platform design.
• Experience developing batch and streaming data pipelines using Spark Structured Streaming.
• Strong expertise in Spark performance tuning, optimization techniques, partitioning strategies, and resource management.
• Experience handling large-scale structured and unstructured datasets.
• Strong SQL and data transformation capabilities.
• Knowledge of production deployment methodologies and enterprise-level operational support.
• Expertise in monitoring, troubleshooting, and performance optimization of data workloads.
• Databricks platform features, cluster sizing, autoscaling, workload management, and optimization strategies.
• Distributed computing principles and Spark internals.
• Cloud data engineering architectures on Azure, AWS, or GCP.
• Data Lake, Lakehouse, and Enterprise Data Warehouse architectures.
• Data governance, data quality management, and data security best practices.
• CI/CD processes, DevOps methodologies, and automation frameworks.
• Git-based version control, branching strategies, and release management.
• Data orchestration, workflow automation, and scheduling frameworks.
• Data modeling, dimensional modeling, and enterprise reporting requirements.
• Analytics, AI/ML, and business intelligence data platform support.
• Agile delivery methodologies and collaborative software engineering practices.
• Core hands-on expertise in Databricks development and architecture.
• Strong experience with Databricks cluster configuration and management.
• Proven expertise in Spark performance tuning and optimization.
• Experience processing and managing large-scale enterprise datasets.
• Ability to design scalable, reliable, and cost-efficient data solutions.
• Strong problem-solving, troubleshooting, and technical leadership capabilities.

Benefits:

• Competitive salary and benefits package
• Culture focused on talent development with quarterly growth opportunities and company-sponsored higher education and certifications
• Opportunity to work with cutting-edge technologies
• Employee engagement initiatives such as project parties, flexible work hours, and Long Service awards
• Annual health check-ups
• Insurance coverage: group term life, personal accident, and Mediclaim hospitalisation for self, spouse, two children, and parents

Values-Driven, People-Centric & Inclusive Work Environment:

Persistent is dedicated to fostering diversity and inclusion in the workplace. We invite applications from all qualified individuals, including those with disabilities, and regardless of gender or gender preference. We welcome diverse candidates from all backgrounds.

• We support hybrid work and flexible hours to fit diverse lifestyles.
• Our office is accessibility-friendly, with ergonomic setups and assistive technologies to support employees with physical disabilities.
• If you are a person with disabilities and have specific requirements, please inform us during the application process or at any time during your employment.

Let's unleash your full potential at Persistent - persistent.com/careers

Persistent is an Equal Opportunity Employer and prohibits discrimination and harassment of any kind.