Key Responsibilities:
- We are seeking a skilled Spark Scala Developer to design and build scalable big data processing applications
- The ideal candidate will have strong experience in Apache Spark Scala and distributed data processing and will work on building high performance data pipelines for analytics and data engineering use cases
- Key Responsibilities
- Big Data Development
- Develop and maintain data pipelines using Apache Spark Scala
- Process large scale datasets in distributed environments
- Implement batch and real time data processing solutions
- Spark Scala Development
- Write efficient and scalable code using Scala
- Work extensively with Spark Core Spark SQL and Spark Streaming
- Optimize Spark jobs for performance and resource utilization
- Data Engineering Integration
- Ingest data from various sources
- Databases RDBMS NoSQL
- APIs
- File systems HDFS S3
- Build ETL ELT pipelines and data transformation workflows
- Performance Optimization
- Tune Spark applications partitioning caching memory tuning
- Optimize queries using Spark SQL
- Improve job execution time and reduce resource consumption
- Collaboration Design
- Work with data engineers data scientists and business analysts
- Translate business requirements into scalable data solutions
- Participate in code reviews and architecture discussions
- Monitoring Troubleshooting
- Monitor data pipelines and Spark jobs
- Debug job failures and performance bottlenecks
- Ensure reliability and data quality in pipelines
Technical Requirements:
- Primary skills Technology Big Data Data Processing Spark Technology Functional Programming Scala
Preferred Skills:
Technology->Big Data - Data Processing->Spark,Technology->Functional Programming->Scala