Hadoop Big Data Platform Engineer SME
Overview
Design and scale Hadoop clusters powering NextGen Big Data platforms. Own cluster performance and reliability across Spark, Kafka, Impala, and HBase. Partner with engineering teams to automate deployments with Docker and Ansible. This contract role drives architecture decisions in a hybrid Chicago or Charlotte setup.
What You'll Do7
- 1Build and tune Hadoop clusters for high-throughput batch and real-time processing.
- 2Debug cluster failures and optimize Spark jobs for performance and cost.
- 3Design data pipelines using Kafka for streaming ingestion and Impala for interactive queries.
- 4Automate cluster provisioning with Docker containers and Ansible playbooks.
- 5Own capacity planning and security configuration for HBase and HDFS.
- 6Lead incident response and root-cause analysis for platform-level issues.
- 7Drive migration of legacy workloads to the Hadoop platform within the first 90 days.
Requirements6
- 15+ years hands-on experience as a Hadoop engineer or platform SME.
- 23+ years building and maintaining Spark and Kafka clusters.
- 3Deep expertise in HBase schema design and HDFS data storage.
- 42+ years automating infrastructure with Docker and Ansible.
- 5Experience tuning Impala for low-latency SQL queries.
- 6Strong troubleshooting skills in Linux environments and distributed systems.
Salary Insight
Salary not disclosed in listing
Similar open positions
Explore active roles that match your skills and interests.
BCforward
VerifiedHadoop Big Data Platform Engineer SME
Lead ownership of Hadoop and Big Data initiatives scaling across multiple sites. Drive technical excellence and mentor junior engineers. This role differs by focusing on enterprise-level platform architecture and cross-functional collaboration.
Apex Systems
VerifiedHadoop Engineer Apex Systems Chicago
We seek a Hadoop Engineer to support NextGen Platforms built around Big Data technologies. The ideal candidate will have experience in the cluster management of vendor-based Hadoop and Data Science products. This role involves participation in the full life cycle of an application within an agile development process and requires the ability to develop, engineer, and ship solutions. It is a hybrid position located in Chicago and Charlotte NC lasting three months.
K&K Global Talent Solutions
VerifiedHadoop Developer, PySpark & Big Data
Own the design and execution of scalable Big Data platforms for a Fortune 500 client in Charlotte, NC. You'll join a cross-functional team of engineers and data scientists to process 10+ TB daily. This role demands deep expertise in PySpark and Hive to transform raw logs into analytics-ready datasets. You'll directly influence the data architecture that powers real-time insights.
Wise Skulls Corp.
VerifiedData Platform Engineer, Lakehouse & Cloud Infrastructure
You will architect and operate a cloud-based data lakehouse platform serving analytics teams and downstream applications across AWS, GCP, and Azure. You will build scalable data infrastructure on open table formats like Delta Lake and Iceberg, powered by distributed compute from Spark and Trino. You will join a platform engineering squad of 8, collaborating daily with data scientists and analysts to ensure reliable, high-performance data access. This role stands out for its greenfield design of a multi-cloud lakehouse, paired with an on-prem Kubernetes migration path.
Bright Vision Technologies
VerifiedApache Spark Developer Remote Job | Bright Vision Technologies
Bright Vision Technologies seeks an Apache Spark Developer to own large-scale distributed data processing for enterprise analytics and machine learning. You will build high-performance Spark applications processing billions of records, working with cloud-native tools like Databricks and AWS EMR. Collaborate with data architects, engineers, and ML teams on real-time pipelines and lakehouse architectures. This remote role spans batch and streaming ETL, performance tuning, and cloud migration, offering direct impact on mission-critical systems.
Wise Skulls Corp.
VerifiedSoftware Engineer (C++/Java) Data Platform
Own scalable high throughput data processing platforms using C++ and Java. Lead ship of core infrastructure components. Drive performance improvements across large scale systems. Shape architecture decisions for future growth. This role differs by focusing on low latency streaming solutions.