← All jobs

SWE-Bench Task Auditor (Train AI Models Part Time!)

up to $400k/year Remote Full-time
C++ DeveloperPython DeveloperSoftware Developer in TestIntegration EngineerPlatform EngineerTypescript DeveloperJava DeveloperDevOps EngineerGo DeveloperFull Stack Python DeveloperFull Stack C++ DeveloperFull Stack Go DeveloperFull Stack Java Developer

Archer is matching candidates to this role at Mercor. Create a free profile and Archer will check you against this role and every other live role, showing you exactly where you match.

Evaluate the quality, correctness, and reproducibility of software-engineering benchmark tasks used to train and evaluate a frontier AI lab's models. You'll assess repository-level tasks, reference patches, test harnesses, and grading integrity — and provide clear, rubric-based written feedback. Basic Qualifications • 3+ years professional software engineering • Real open-source contribution or maintainer experience (merged PRs, committer / maintainer roles) • Strong ability to audit reference patches, test runners, and Docker isolation, and to detect answer leakage / reward hacking • Fluency across common ecosystems (Python and at least one of Java / Go / TypeScript / C++) Preferred Qualifications • Familiarity with SWE-Bench (Verified) or similar repository benchmarks • Maintainer history on major Python OSS (Django, Flask, scikit-learn, sympy, pytest, etc.) • Prior code-review or task-grading experience

Apply knowing you're qualified

One free profile is all it takes. Archer checks you against this role and every other live role we list, and shows you exactly which requirements you meet before you apply.

More roles like this

See all Python jobs

Not quite the right role?

Archer scans thousands of live roles and surfaces the ones you genuinely match, each with a clear explanation of why. It keeps working after you apply, so you hear about roles you would never have found by searching.

Create your free profile