Key Responsibilities:
- Big Data Spark Development
- Design and implement scalable data pipelines using Apache Spark Scala and or PySpark
- Work extensively with Spark Core Spark SQL DataFrames and Datasets
- Develop batch and real time data processing solutions using Spark Streaming Structured Streaming
- Optimize Spark jobs for performance memory management and parallel processing
- Scala Python Development
- Develop robust and efficient applications using Scala and Python
- Write reusable modular and maintainable code
- Implement business logic and transformations on large datasets
- Data Engineering ETL
- Build and maintain ETL ELT pipelines for large scale data ingestion and transformation
- Process structured and unstructured data from multiple sources
- Ensure data validation quality and consistency
- Work with file formats like Parquet ORC Avro JSON CSV
- Big Data Ecosystem
- Work with Hadoop ecosystem HDFS Hive YARN
- Integrate Spark jobs with data lakes and warehouses
- Handle large datasets with distributed computing techniques
- Cloud Integration Optional but Preferred
- Work with cloud platforms AWS Azure GCP for big data solutions
- Utilize services such as AWS EMR Glue S3 Azure Databricks Synapse
- Integrate pipelines with APIs and external systems
- Collaboration Leadership
- Collaborate with data engineers architects and business teams
- Lead technical discussions and provide guidance to junior developers
- Participate in code reviews and best practice implementation
- Work in Agile Scrum environments
Technical Requirements:
- Primary skills Domain Finacle Core Functional Finacle Core WMS Grand Master Technology Big Data Data Processing Spark Technology Java Apache
Additional Responsibilities:
- Core Skills
- 5 9 years of experience in data engineering big data development
- Strong hands on expertise in Scala mandatory for this role
- Extensive experience with Apache Spark Scala and or PySpark
- Solid understanding of ETL processes and data pipelines
- Strong proficiency in SQL and database concepts
- Technical Skills
- Deep knowledge of Spark architecture and execution model
- Experience with Spark performance tuning and optimization
- Strong data modeling and warehousing concepts
- Familiarity with version control tools Git
- Understanding of distributed computing principles
- Preferred Skills
- Experience with Spark Streaming Kafka
- Hands on with Databricks platform
- Knowledge of Airflow or workflow orchestration tools
- Familiarity with Docker Kubernetes
- Exposure to NoSQL databases Cassandra MongoDB HBase
Preferred Skills:
Technology->Big Data - Data Processing->Spark->SparkSQL,Technology->Java->Apache->Scala,Technology->Big Data - Data Processing->PySpark