Key Responsibilities:
- 5 9 years of experience in Data Engineering Big Data development
- Strong hands on experience with Python and Apache Spark Scala PySpark
- Deep understanding of Spark architecture RDD DataFrames Spark SQL
- Experience in distributed computing and big data processing
- Strong knowledge of SQL and data modeling concepts
- Experience with data pipeline development ETL ELT
- Familiarity with Linux Unix environments
- Experience with version control tools Git
Technical Requirements:
- Primary skills Technology Big Data Data Processing Spark Technology Functional Programming Scala Technology Machine Learning Python
Additional Responsibilities:
- Experience with cloud platforms AWS Azure or GCP
- Hands on with Databricks EMR Spark clusters
- Knowledge of streaming technologies Kafka Spark Streaming Structured Streaming
- Experience with workflow orchestration tools Airflow Oozie
- Familiarity with Delta Lake Lakehouse architecture
- Exposure to NoSQL databases MongoDB Cassandra
- Knowledge of CI CD and DevOps practices
Preferred Skills:
Technology->AI-Data science->PYTHON,Technology->Big Data - Data Processing->Spark->SparkSQL,Technology->Functional Programming->Scala