DescriptionThe Data Engineer is a critical role responsible for designing, building, and maintaining the data infrastructure and pipelines that power data-driven insights and applications. This role requires expertise in Google Cloud Platform (GCP) and its data services to effectively manage and process large-scale datasets.
Key Responsibilities:
- Data Ingestion and Processing: Design and implement robust data pipelines to collect, clean, transform, and store massive volumes of data using GCP services like Dataflow, Pub/Sub, and Cloud Storage.
- Data Warehousing and Analytics: Build and optimize data warehouses on Google BigQuery for efficient data analysis and reporting.
- Data Modeling: Design and implement scalable data models to support business intelligence and machine learning applications.
- ETL/ELT Development: Develop and maintain Extract, Transform, Load (ETL) or Extract, Load, Transform (ELT) processes using GCP tools.
- Performance Optimization: Continuously monitor and optimize data pipelines and queries for performance and cost-effectiveness.
- Collaboration: Work closely with data scientists, analysts, and other stakeholders to understand data requirements and deliver data solutions.
Required Skills:
- Google Cloud Platform (GCP): Strong proficiency in GCP data services, including BigQuery, Dataflow, Pub/Sub, Cloud Storage, and DataBricks.
- Data Engineering Tools: Experience with Apache Beam, Apache Airflow, or similar data pipeline orchestration tools.
- Programming Skills: Proficiency in Python or Java for data processing and pipeline development.
- Data Modeling: Expertise in data modeling techniques and schema design.
- SQL: Strong SQL skills for data querying and analysis.
- Problem Solving: Ability to analyze and solve complex data engineering challenges.
Qualifications:
- Bachelor's or Master's degree in Computer Science, Data Engineering, or a related field.
- 5+ years of experience in data engineering, with a focus on Google Cloud Platform.
- Experience with large-scale data processing and distributed systems.
Preferred Skills:
- Google Cloud Professional Data Engineer Certification.
- Experience with machine learning and data science workflows.
- Knowledge of data visualization and reporting tools.
- Strong communication and collaboration skills.
- Understanding of AI
This Data Engineer role, with its emphasis on Google Cloud Platform expertise, is ideal for individuals passionate about building scalable and efficient data solutions to drive business insights and innovation.