Data Engineer, Python & Databricks (Azure)
Overview
Python developer building data pipelines for GenAI applications on Azure Databricks and Delta Lake. Own end-to-end ETL development and data quality frameworks. Work with a small team to scale data infrastructure. Stand out by integrating React dashboards into data workflows.
What You'll Do9
- 1Build Python-based ETL pipelines to process and transform large datasets for analytics and machine learning.
- 2Design and implement data workflows for GenAI applications, including data ingestion and preparation.
- 3Develop data quality frameworks to monitor and ensure data accuracy and consistency across pipelines.
- 4Collaborate with data scientists to deliver clean, structured datasets for model training and evaluation.
- 5Optimize and tune Databricks jobs and Delta Lake tables for performance and cost efficiency.
- 6Create interactive dashboards using React to visualize pipeline metrics and data insights.
- 7Integrate Azure services such as Data Lake and Blob Storage into the data ecosystem.
- 8Support production pipelines, debug failures, and implement monitoring and alerting.
- 9Document data engineering processes and maintain code repositories for team collaboration.
Requirements9
- 15+ years building ETL pipelines with Python and PySpark.
- 23+ years working with Azure services including Data Lake and Databricks.
- 3Hands-on experience with Delta Lake and Spark for large-scale data processing.
- 4Proficiency in SQL for querying and optimizing data.
- 5Experience developing front-end dashboards with React and JavaScript.
- 6Familiarity with GenAI and machine learning data pipelines.
- 7Strong understanding of data modeling and data warehousing concepts.
- 8Ability to work with cross-functional teams in an Agile environment.
- 9Excellent problem-solving skills and attention to detail.
Salary Insight
Salary not disclosed in listing
Location
Required Skills
Similar open positions
Explore active roles that match your skills and interests.
ePace Technologies, Inc
VerifiedDatabricks Data Engineer, ETL & Delta Lake
You will own enterprise-scale data pipelines on Databricks, driving the architecture and delivery of mission-critical data flows in cloud environments. You will join a platform team of engineers and data specialists, working with Delta Lake, Python, and SQL to process complex Sales Incentive Compensation (SIC) data. This role stands out for its focus on Delta Live Tables and high-volume, low-latency processing in a dynamic consulting environment.
Healthcare Triangle Inc
VerifiedDatabricks Lead - Data Engineering
Own the design and delivery of scalable data engineering solutions on the Databricks platform, driving value across the full data lifecycle. Lead a team of engineers building PySpark, Spark SQL, and Delta Lake pipelines. Collaborate with data architects and stakeholders to shape the lakehouse architecture. This role offers direct ownership of technical strategy and a chance to set standards for data quality and performance.
Trebecon LLC
VerifiedAzure Databricks Architect
Lead design of scalable enterprise data platforms using Azure Databricks and Lakehouse architecture. Own end-to-end data engineering initiatives to drive high-performance ETL/ELT pipelines. Implement solutions leveraging PySpark Spark SQL and Databricks Workflows. Differentiate by scaling data infrastructure for Tampa based teams.
Protos IT
VerifiedDatabricks Data Engineer, ETL & Lakehouse Architect
Design and own scalable data pipelines on Databricks and Apache Spark for a federal contractor in the DC/MD/VA area. You will modernize legacy ETL into a Delta Lake lakehouse, working cross-functionally with analytics and platform teams. This contract role stands out for its hybrid setup and direct impact on mission-critical data infrastructure.
Apex Systems
VerifiedSenior Data Engineer, Atlanta GA
Own the design and build of scalable data platforms and pipelines for Power Delivery, integrating operational systems with cloud-based lakehouse platforms. You will work with Databricks, Azure, and Python to deliver data products that drive business decisions. Join a team that values both low-code and pro-code solutions, and see your work impact real-time analytics across the organization. This role offers the chance to shape the data landscape for a major utility company.
Donato Technologies Inc
VerifiedSenior Data Engineer, Databricks & ETL Pipelines
Own the design and delivery of scalable data pipelines on Databricks for a Houston-based enterprise. You join a data engineering team of 6 inside a larger analytics organization, collaborating with architects, analysts, and application teams. Build reliable ETL/ELT solutions that move millions of records daily. This contract role demands hands-on coding in Python and Spark from day one, with direct influence on platform architecture.