Senior Data Engineer, Databricks & Cloud Migration
Overview
Own data engineering initiatives at scale in a client environment, building robust data pipelines and migrating on-prem systems to cloud platforms. You will work with a cross functional team of data engineers, cloud engineers, and analytics partners in New York City onsite. This role emphasizes hands on Databricks development, Python, and SQL while collaborating to design scalable data architectures. What makes this role unique is leading migration projects from legacy systems to modern cloud data platforms while delivering measurable data capabilities for business users.
What You'll Do10
- 1Build and maintain production grade data pipelines using Databricks and PySpark to ingest, transform, and load data at scale
- 2Design data models and ETL/ELT processes that support analytics and reporting needs across multiple business domains
- 3Lead migration of on premise data systems to cloud platforms, including planning, execution, and validation
- 4Collaborate with data scientists and analytics teams to enable advanced analytics and ML workflows
- 5Optimize query performance and resource usage in Python and SQL based workloads
- 6Mentor junior engineers and establish best practices for coding, testing, and deployment
- 7Ship reliable data pipelines with monitoring, alerts, and automated testing to ensure data quality
- 8Drive improvements to data governance, security, and compliance during cloud migration
- 9Review and refactor legacy SQL to modern Spark SQL where needed
- 10Partner with platform engineers to ensure robust data platform infrastructure and CI/CD processes
Requirements10
- 1Minimum 5+ years of data engineering experience with a strong focus on Databricks and production-grade pipelines
- 2At least 3+ years hands-on Databricks experience in production environments
- 3Proven migration experience from on-premises data systems to cloud platforms
- 4Strong proficiency in Python, PySpark, and SQL with experience converting legacy SQL dialects to Spark SQL is a plus
- 5Cloud knowledge and familiarity with cloud services related to data platforms
- 6Solid understanding of data modeling, data warehouses, and data lake concepts
- 7Ability to work onsite in New York City and collaborate with cross functional teams
- 8Excellent communication skills and ability to translate complex technical concepts for business stakeholders
- 9Experience with data governance, security, and compliance in cloud data environments
- 10Experience in transforming and optimizing data pipelines for scale and reliability
Salary Insight
$120 - $130k per year
Location
Required Skills
Similar open positions
Explore active roles that match your skills and interests.
Techouts, Inc
VerifiedDatabricks Senior Developer, Data Engineering & Migration
Own the end-to-end data engineering stack at scale, building and migrating data systems to the cloud. You will drive production-ready Databricks pipelines and collaborate with cloud and data teams to modernize on-prem data assets. This role emphasizes hands-on development, migration know-how, and translating legacy SQL to Spark SQL. You will contribute to a strong data platform that enables analytics and insights across the organization.
WB Solutions LLC
VerifiedData Tech Lead, Java & Cloud Data Platform
Databricks and streaming data expertise to design, develop, and support scalable cloud data platforms at scale. Lead data pipelines using PySpark and Java streaming technologies and own cloud migration initiatives. Collaborate with cross functional teams, build robust CI/CD with GitLab CI/CD and drive architecture decisions. This role offers a chance to influence data platform strategy and accelerate data driven outcomes.
RMS IT Solutions Inc
VerifiedSenior Data Engineer, Databricks & Structured Finance
You own the end-to-end migration of structured finance datasets from a legacy platform to Databricks. You lead architecture decisions, data modeling, and pipeline builds. You work independently with a team of data engineers and analysts in New York. This role stands out because you design the future state while unraveling legacy business logic.
PTR Global
VerifiedDatabricks Data Engineer - New York Contract
Own the design and delivery of scalable data pipelines on Databricks in a contract role based in New York. You will partner with data scientists and analysts to build robust ETL solutions that turn raw data into business-critical insights. This position demands a hands-on engineer who can navigate complex data landscapes with Spark and Delta Lake. Your work will directly influence data-driven decisions across the organization.
Yoh - A Day & Zimmerman Company
VerifiedData Architect & Modeler with Databricks
You will own the design and delivery of data architectures for financial platforms, migrating legacy Oracle systems to Databricks on the cloud. You'll work within a cross-functional team of engineers and analysts, supporting multiple projects across the firm. This role stands out for its hands-on engineering focus and direct impact on modernizing data infrastructure.
Donato Technologies Inc
VerifiedSenior Data Engineer, Databricks & ETL Pipelines
Own the design and delivery of scalable data pipelines on Databricks for a Houston-based enterprise. You join a data engineering team of 6 inside a larger analytics organization, collaborating with architects, analysts, and application teams. Build reliable ETL/ELT solutions that move millions of records daily. This contract role demands hands-on coding in Python and Spark from day one, with direct influence on platform architecture.