Hadoop Developer, PySpark & Big Data
Overview
Own the design and execution of scalable Big Data platforms for a Fortune 500 client in Charlotte, NC. You'll join a cross-functional team of engineers and data scientists to process 10+ TB daily. This role demands deep expertise in PySpark and Hive to transform raw logs into analytics-ready datasets. You'll directly influence the data architecture that powers real-time insights.
What You'll Do10
- 1Design and build Hadoop clusters and Spark jobs that process 5+ TB daily.
- 2Develop PySpark scripts to parse, clean, and join 2,000+ data sources.
- 3Optimize Hive queries and Spark workloads to cut execution time by 40%.
- 4Migrate legacy MapReduce jobs to Spark for 3 critical pipelines.
- 5Implement Kafka streaming to ingest 1M events per second into the data lake.
- 6Debug and tune Java components to prevent memory bottlenecks.
- 7Collaborate with data analysts to define SQL schemas and partitioning strategies.
- 8Automate Unix shell scripts for job scheduling and data validation.
- 9Monitor cluster health and resolve YARN resource contention issues.
- 10Document data flow and maintain Airflow DAGs for end-to-end orchestration.
Requirements9
- 110+ years building data pipelines in Hadoop and Spark ecosystem.
- 2Expert-level PySpark for complex transformations and window functions.
- 3Proficient in Hive DDL, DML, and query tuning using Tez.
- 4Strong Python programming for data parsing and API integration.
- 5Hands-on SQL with 5+ years in Teradata or Oracle.
- 6Experience with Unix shell scripting and Cron job automation.
- 7Familiar with Kafka producer/consumer APIs and stream processing.
- 8Solid Java knowledge to debug and enhance Spark custom libraries.
- 9Ability to analyze YARN logs and optimize cluster resource allocation.
Salary Insight
Salary not disclosed in listing
Similar open positions
Explore active roles that match your skills and interests.
SSV Technologies Inc
VerifiedHadoop Big Data Platform Engineer SME
Design and scale Hadoop clusters powering NextGen Big Data platforms. Own cluster performance and reliability across Spark, Kafka, Impala, and HBase. Partner with engineering teams to automate deployments with Docker and Ansible. This contract role drives architecture decisions in a hybrid Chicago or Charlotte setup.
BCforward
VerifiedHadoop Big Data Platform Engineer SME
Lead ownership of Hadoop and Big Data initiatives scaling across multiple sites. Drive technical excellence and mentor junior engineers. This role differs by focusing on enterprise-level platform architecture and cross-functional collaboration.
Bright Vision Technologies
VerifiedApache Spark Developer Remote Job | Bright Vision Technologies
Bright Vision Technologies seeks an Apache Spark Developer to own large-scale distributed data processing for enterprise analytics and machine learning. You will build high-performance Spark applications processing billions of records, working with cloud-native tools like Databricks and AWS EMR. Collaborate with data architects, engineers, and ML teams on real-time pipelines and lakehouse architectures. This remote role spans batch and streaming ETL, performance tuning, and cloud migration, offering direct impact on mission-critical systems.
Apex Systems
VerifiedHadoop Engineer Apex Systems Chicago
We seek a Hadoop Engineer to support NextGen Platforms built around Big Data technologies. The ideal candidate will have experience in the cluster management of vendor-based Hadoop and Data Science products. This role involves participation in the full life cycle of an application within an agile development process and requires the ability to develop, engineer, and ship solutions. It is a hybrid position located in Chicago and Charlotte NC lasting three months.
PRIMUS Global Services Inc.
VerifiedBig Data Machine Learning Engineer, Spark & Python
You will design and build large-scale data processing pipelines and machine learning platforms using Apache Spark, Python, and SQL. You will work within a distributed systems team at PRIMUS Global Services Inc., collaborating on feature extraction and data analysis. This role requires deep experience with Hadoop, Snowflake, and related big data tools, and offers the chance to solve complex data challenges on an onsite team in Sunnyvale, CA.
Kanshe Infotech
VerifiedSenior Data Engineer Kanshe Infotech Charlotte NC
Design and maintain scalable enterprise data pipelines and analytics solutions. Lead development of robust data infrastructure. Drive optimization of data processing workflows. Ensure high availability and performance of data systems. Shape future data architecture.