Lead Cloud Data Engineer - Data Mesh & AI
Overview
You will own the design and delivery of a federated data mesh platform for a central banking system, serving 5000+ internal users and AI/ML workloads at scale. You will lead a squad of 5 data engineers and collaborate with data scientists, analysts, and infrastructure teams. This contract role in San Francisco offers the chance to shape data governance and enable self-service analytics in a mission-critical environment.
What You'll Do8
- 1Design and deploy a data mesh architecture on AWS with domain-oriented data products and federated governance.
- 2Build scalable data pipelines using Spark, Airflow, and Kafka to ingest and process real-time and batch data.
- 3Implement data quality and observability frameworks using Great Expectations and DataDog.
- 4Drive adoption of Terraform for infrastructure as code and ensure reproducibility across environments.
- 5Coach 5 engineers on best practices for Python and SQL development and code reviews.
- 6Collaborate with data scientists to productionize ML models and feature stores using MLflow.
- 7Champion API-first design and publish data products into an internal marketplace.
- 8Own the end-to-end data lineage and catalog with Atlas and Amundsen.
Requirements8
- 18+ years in data engineering with a focus on cloud platforms.
- 23+ years leading teams and architecting data solutions.
- 3Expertise in Python, Spark, and SQL.
- 4Deep proficiency with AWS services: S3, Glue, Redshift, EMR, and Kinesis.
- 5Hands-on experience with Airflow, Kafka, and Terraform.
- 6Proven track record with data mesh principles and domain-driven design.
- 7Strong understanding of data governance and security in regulated environments.
- 8Bachelor's degree in Computer Science or equivalent.
Salary Insight
$90 - $105k per year
Similar open positions
Explore active roles that match your skills and interests.
ConfigUSA
VerifiedData and Cloud Solutions Architect, San Jose
You will own the design and evolution of cloud-based data architecture for a 6-month onsite engagement in San Jose. You will set the technical foundation for scalable, secure data platforms, working hands-on with AWS, Azure, and GCP ecosystems. You will collaborate with cross-functional teams to align architecture with business goals. This role stands out for its focus on hybrid cloud strategy and real-time data processing.
EXTEU-AC-CareerSite
VerifiedPrincipal Data Architect, Data Engineering & AI
You will own the enterprise data architecture and strategy, enabling scalable AI/ML, telemetry analytics, and business intelligence across a global organization. You will lead the design of cloud-native data platforms on AWS and Azure, from ingestion to insights, and set the blueprint for data storage, integration, and governance. Collaborating with Data Science and AI teams, you will industrialize ML pipelines and drive data as a strategic asset. This role combines deep technical leadership with executive influence, shaping the future of data-driven decision making.
Motion Recruitment Partners, LLC
VerifiedSr. Data Engineer, Financial Services Contract
You will own data pipelines and analytics solutions at a top-tier financial institution, serving millions of customers. The team builds high-scale platforms on AWS and Snowflake, with a stack of Python, Spark, and Airflow. You'll work alongside senior engineers and product managers in a hybrid Charlotte office. This role stands out for its long-term contract stability and direct impact on core banking systems.
US132 Guidehouse Digital, LLC
VerifiedAI & Data Lead Developer, Cloud & Analytics
You own the technical delivery of data engineering, analytics, and AI-enabled solutions for large-scale data modernization projects in the public sector. Building on 10+ years of experience, you design and operationalize data pipelines, integrate AI capabilities, and guide cross-functional teams in an Agile environment. You work with AWS and Snowflake to drive scalable architectures, partnering with architects and operations leads to ensure production-grade outcomes. This role stands apart by combining hands-on development with strategic leadership, directly shaping federal data modernization initiatives.
Digitek Software, Inc.
VerifiedData Architect, Cloud Data Platforms
You will own the end-to-end data architecture across the organization, defining storage, integration, and consumption patterns for scalable, trustworthy data. You will work with a cross-functional team of engineers and analysts, modernizing legacy pipelines into cloud-native solutions. This role sits at the center of business and technology strategy, translating requirements into robust data models. You will set the technical vision for data governance and access, directly influencing products across the company.
Robert Half
VerifiedSenior Data Engineer, Cloud Platform Modernization
You own the design and build of scalable data pipelines and cloud-based platforms for a major enterprise. 5+ years of experience with Python, SQL, and AWS underpins your work. You partner with cross-functional teams to modernize legacy data systems, driving real-time analytics and data-driven decisions. This long-term contract in Glendale, CA offers a hybrid schedule and W2 pay.