Job Description:
- Infosys is seeking a highly skilled Azure Databricks Engineer with 6 to 8 years of experience in designing developing and optimizing cloud based data engineering solutions
- The ideal candidate will possess strong expertise in Azure Databricks PySpark Azure Data Factory ADF Azure Data Lake Storage ADLS SQL and Data Engineering practices
Key Responsibilities:
- Design develop and maintain scalable Azure Databricks solutions for enterprise data platforms
- Build and optimize end to end ETL ELT pipelines using PySpark Spark SQL and Azure Data Factory ADF
- Develop robust data ingestion frameworks from multiple structured and unstructured data sources
- Implement and manage Delta Lake and Lakehouse Architecture for high volume data processing
- Create and optimize data models transformations and workflows to support analytical and business requirements
- Perform performance tuning and troubleshooting of Databricks workloads and Apache Spark jobs
- Work with Azure services including ADLS Gen2 Azure Synapse Analytics Azure Key Vault and Azure Data Factory
- Collaborate with business stakeholders architects and cross functional teams to deliver data solutions
- Ensure adherence to data governance security and best practices
- Participate in code reviews CI CD implementation testing deployment and production support activities
Technical Requirements:
- 6 to 8 years of experience in Data Engineering and Big Data technologies
- Strong hands on experience with Azure Databricks and PySpark
- Expertise in Apache Spark Spark SQL and distributed data processing
- Experience with Azure Data Factory ADF and building enterprise scale data pipelines
- Strong knowledge of Azure Data Lake Storage ADLS Gen2
Preferred Skills:
Technology->Cloud Integration->Azure Data Factory (ADF),Technology->Cloud Platform->Azure Core services->Azure Virtual Machines,Technology->Cloud Platform->Azure Networking Services->Azure Application Gateway,Technology->Data Engineering->Databricks