Data engineer for AI

Описание:

Over the past 25 years, NETCONOMY has grown from a startup into a team of 500 people across 10 European locations. We believe in the power of agile and cross-functional collaboration, bringing together people from diverse backgrounds to build outstanding digital solutions. As a Data Engineer, you'll play a key role in building modern, scalable, and high-performance data solutions on Google Cloud Platform (GCP). You'll be part of our growing Data & AI team, designing and implementing data architectures that help clients unlock the full potential of their data.

Задачи:

• Building efficient and scalable ETL/ELT processes to ingest, transform, and load data from various structured and unstructured sources (databases, APIs, streaming platforms) into BigQuery and Cloud Storage.
• Implementing data ingestion and real-time processing using Dataflow (Apache Beam) and Pub/Sub for batch and streaming workflows.
• Developing SQL transformation workflows with Dataform, including version control, testing, and automated scheduling with built-in quality assertions.
• Creating efficient, cost-optimized BigQuery queries with proper partitioning, clustering, and denormalization strategies.
• Orchestrating complex workflows using Cloud Composer (Apache Airflow) and Cloud Functions for event-driven data processing.
• Implementing centralized data governance and metadata management using Dataplex with automated cataloging and lineage tracking.
• Monitoring and optimizing data pipelines for performance, scalability, and cost using Cloud Monitoring and Cloud Logging.
• Collaborating with data scientists and analysts to understand data requirements and deliver actionable insights.
• Staying up to date with GCP advancements in data services, BigQuery features, and data engineering best practices.

Требования:

• Expert-level SQL proficiency for complex transformations, optimization, and analysis.
• Proficiency with Dataform for modular SQL-based data transformations and data pipeline management.
• Solid understanding of data warehousing principles, ETL/ELT processes, dimensional modeling, and data governance.
• Excellent communication and collaboration skills in English (min. B2 level).
• Ability to work independently and as part of an agile team.
• Google Professional Data Engineer certification.
• Knowledge of BigLake for unified access and management of structured and unstructured data.
• Familiarity with Infrastructure-as-Code (Terraform) for automating GCP resource provisioning and CI/CD pipelines.
• 3+ years of hands-on experience as a Data Engineer with proven expertise in Google Cloud Platform (GCP).
• Strong experience with BigQuery (SQL, partitioning, clustering, optimization) and Dataflow (Apache Beam).
• Strong programming skills in Python with experience in data manipulation libraries (PySpark, pandas).
• Experience integrating data from various APIs and streaming systems (Pub/Sub).
• Cloud Composer experience for workflow orchestration.
• Experience with Dataplex for managing metadata, lineage, and data governance.
• Experience with data visualization tools such as Looker, Looker Studio, or Power BI.
• Interest in or experience with machine learning workflows using Vertex AI or similar platforms.

Условия:

• Flexible Working Models: Early bird or night owl? Thanks to our flexible working models, you start your work day when and where it fits you best.
• Plus, our hybrid work option lets you choose whether you prefer working more from the office or from home.
• Career Development and Onboarding: Start your journey in NETCONOMY with a structured onboarding and mentoring phase and continue with individual training opportunities. Our People Enablement team will support you in finding the best solution for you!
• Company Summit: Exchange professional and personal experiences at our annual in-house conference! One of the highlights of our company Summit is the vibrant networking environment it cultivates.
• Social Events: Join our colleagues and build new connections at our social events such as pizza evenings, sports activities, Christmas parties or milestone celebrations.
• Snacks and Wellbeing: Fuel your productivity and taste buds with our monthly meal allowance and discounts at partner restaurants.
• Mobility Support: Choose eco friendly transportation! As we take our responsibility for the environment seriously, we support your climate-friendly transportation costs.