Key Responsibilities:
- GCP BigQuery Development
- Design develop and optimize data warehouses using Google BigQuery
- Build and maintain data models fact dimension star snowflake schemas
- Develop complex SQL queries for analytics and reporting
- Optimize queries for cost performance and scalability
- Implement partitioning clustering and materialized views
- Python Development
- Develop data pipelines and workflows using Python
- Build reusable scripts for data ingestion transformation and automation
- Integrate Python applications with GCP services and APIs
- Ensure code quality modularity and scalability
- Data Engineering ETL
- Design and build ETL ELT pipelines using GCP tools
- Ingest data from multiple sources APIs databases streaming platforms
- Process structured and unstructured data efficiently
- Ensure data quality validation and governance
- GCP Cloud Services
- Work with GCP services such as
- BigQuery mandatory
- Cloud Storage GCS
- Dataflow Dataproc
- Pub Sub streaming
- Cloud Composer Airflow
- Design scalable and secure cloud native architectures
- Performance Optimization
- Monitor and optimize BigQuery cost and performance
- Troubleshoot data pipeline failures and bottlenecks
- Implement best practices for efficient data processing
- Collaboration Leadership
- Collaborate with data scientists analysts and business stakeholders
- Provide technical guidance and mentorship to junior engineers
- Participate in architecture and design discussions
- Work in Agile Scrum environments
Technical Requirements:
- Primary skills Technology Functional Testing Mainframe testing Proterm Technology OpenSystem Python OpenSystem
Additional Responsibilities:
- Required Skills Qualifications
- Core Skills
- 5 9 years of experience in data engineering big data development
- Strong hands on experience with GCP BigQuery mandatory
- Strong proficiency in Python mandatory
- Expertise in SQL and query optimization
- Experience in building ETL ELT pipelines
- Technical Skills
- Deep knowledge of BigQuery architecture and data modeling
- Experience with Airflow Cloud Composer for orchestration
- Strong understanding of data warehousing concepts
- Familiarity with Git and CI CD processes
- Knowledge of REST APIs and integrations
- Preferred Skills
- Experience with streaming Pub Sub Kafka equivalent
- Exposure to Dataproc Spark PySpark
- Familiarity with data lakes and lakehouse architectures
- Knowledge of Docker Kubernetes
- Experience with BI tools Looker Tableau Power BI
Preferred Skills:
Technology->Cloud Platform->GCP Database,Technology->OpenSystem->Python - OpenSystem