QA/Test Engineer | $60-$90/hr Remote
Overview
A leading AI lab is looking for a meticulous QA/Test Engineer to help build evaluation benchmarks for cutting-edge agentic AI models. Your work ensures complex multi-step tasks are unambiguous, correctly graded, and free of shortcuts, directly impacting how frontier models are measured. You'll dive into each task, run it, probe edge cases, and debug environments, working in a tight loop with researchers and task authors. This is a fully remote, W-2 role with Cincinnatus LLC, embedded within the AI lab's extended workforce, offering a competitive hourly rate of $60–$90.
What You'll Do5
- 1Create test cases that validate each task's functionality, including tricky edge cases.
- 2Review tasks and their reference solutions before finalization, catching ambiguities and gaps early.
- 3Roll up your sleeves in Python to debug tasks or test checks when things don't behave.
- 4Help build simple, repeatable quality checklists and provide actionable feedback to task authors.
- 5Watch for shortcuts and grading gaps in AI agent runs to keep benchmark scores trustworthy.
Requirements8
- 1MSc or PhD in a STEM field, or equivalent hands-on experience in a research- or engineering-heavy role.
- 21+ years in test engineering, quality assurance, or a research/software engineering position with strong quality ownership.
- 3Proven ability to design test cases, run quality-review processes, and debug complex systems end-to-end.
- 4Working proficiency in Python and Git, comfortably navigating unfamiliar codebases and environments.
- 5Exceptional attention to detail and clear written documentation.
- 6Experience in AI training, model evaluation, or reviewing AI-generated work is a plus.
- 7A perfectionist mindset—creative in finding what others missed and able to work independently on ambiguous problems.
- 8Ability to commit reliably to roughly 35 hours per week.
Who Should Apply
You're a detail-obsessed QA engineer who loves breaking things and finding edge cases. You thrive in ambiguous, open-ended problems and aren't afraid to dig into unfamiliar code. If you have a background in research or AI evaluation and want to shape how frontier models are tested, this role is for you.
Salary Insight
$60–$90 per hour, likely depending on experience and qualifications.
Location
Required Skills
Application Tip
Highlight a specific example where you designed test cases that caught a critical bug or ambiguity in a complex system, and explain how you approached it.
Similar open positions
Explore active roles that match your skills and interests.
Cynet Systems
VerifiedAI QA Test Automation Engineer, Contract in New York
You will own the test automation strategy for the client's AI-enabled products in New York City, leading testing efforts across multiple scrum teams. You will build and maintain robust automation frameworks using Python, Selenium, and Cypress, integrating with Jenkins and Docker for continuous testing. Collaborating with developers and product managers, you will ensure high-quality releases. This contract role offers the chance to shape AI quality practices in a fast-paced financial district environment.
Mercor
VerifiedData Science & Quantitative Analysis Expert | $60-$90/hr Remote
A premier AI research organization is assembling a team of quantitative experts to help design the next generation of evaluation benchmarks for frontier models. In this role, you'll create realistic, hands-on data analysis challenges that push models to their limits — cleaning messy datasets, comparing statistical methods, and verifying whether model outputs hold up to scrutiny. This is a fully remote, W-2 position with Cincinnatus LLC, working about 35 hours per week and collaborating closely with the lab's researchers. If you're passionate about the science behind AI and want your analytical expertise to directly shape how models are measured, this is a unique opportunity.
Mercor
VerifiedSoftware Engineering Expert | $60-$90/hr Remote
This role puts you on the front lines of AI development, working with a top-tier AI lab to design the next generation of evaluation benchmarks for frontier models. As a Software Engineering Expert, you'll craft complex, multi-step engineering challenges that push the limits of today's most advanced AI coding agents. You'll work closely with researchers, using your Python skills and hands-on engineering experience to identify exactly where AI models stumble. It's a fully remote, full-time W-2 position with Cincinnatus LLC, offering the chance to shape how the best AI systems are tested and improved.
Technology Talent Network LLC
VerifiedQA Automation Engineer
Tech Talent Network LLC seeks a skilled professional to lead quality assurance automation initiatives. This role drives high-impact solutions across diverse industries while collaborating closely with cross-functional teams. The ideal candidate excels in agile environments and delivers measurable results within 90 days.
CXApp US, Inc.
VerifiedQA Engineer AI Data CXApp
QA Engineer specializing in AI and Data at CXAPP drives verification and validation of AI applications and data-driven solutions. This role focuses on ensuring quality and performance of AI systems through comprehensive testing strategies. It involves designing test plans managing test data and applying various testing methodologies to guarantee robust AI solutions.
Mercor
VerifiedSoftware Engineer, Full Stack (Python, Java, Rust, C#, C++) | $50-$65/hr Remote
This is a full-time remote contract role with a leading AI lab's GenAI team, focused on building and stress-testing software on top of frontier large language models. You'll work directly with the engineering manager to integrate pre-release model APIs, build full-stack tools, and document failure modes that shape future model development. The stack spans Python, Java, Rust, C#, C++, and TypeScript, with production experience in at least one and working proficiency in a second. What sets this apart is the chance to work on cutting-edge AI infrastructure before it ships, with a fast-paced, high-impact environment. Compensation is $50–$65 per hour, and you'll be employed via Cincinnatus LLC.