Senior Software Engineer, Model Evaluation
About the role
About the Role
Join an early-stage AI team building agents for manufacturing operations. You will create benchmarks and train models against real procurement and supply-chain workflows, helping ensure agents can handle customer-facing tasks reliably.
What You'll Do
Design and build evaluation suites for supplier sourcing, qualification, compliance documents, purchase orders, freight, and order tracking.
Work with procurement and manufacturing experts to scope tasks, write reference answers and rubrics, and build graders.
Train and fine-tune models, using evaluation results to guide experiments.
Identify agent failures, improve production systems, and measure whether changes work.
Help establish the team's research and evaluation practices.
What We're Looking For
At least 3 years of experience, operating at a senior or staff level.
Hands-on experience designing benchmarks and evaluations, including authoring tasks, reference solutions or rubrics, and quality checks.
Practical experience training and fine-tuning models, not only using models through APIs.
Ability to work closely with non-ML domain experts and translate their knowledge into measurable tasks.
Supply-chain, procurement, manufacturing, or early-stage startup experience is helpful.
Compensation & Benefits
Base salary is $180,000 to $250,000 annually, plus equity. Visa sponsorship details were not provided.
Location
Hybrid in San Francisco, California, United States.
About this listing
This role passed our automated spam and quality filters and was active in our feed when last checked. Joboru is an aggregator — here is how we screen listings. If anything looks off, tell us.
Similar jobs you may like
Senior / Lead / Principal Software Engineer, AI – Qualified
4 days agoSalesforce
Principal Data Scientist (Tech/Media/Entertainment)
3 weeks agoFractal
Software Engineer/Developer, Staff
1 month agoAMERICAN SYSTEMS
NAVAIR Software Engineer/Developer, Lead
1 month agoAMERICAN SYSTEMS
Sr Staff Software Engineer- Data Platform
1 month agoWEX
NAVAIR Software Engineer/Developer
1 month agoAMERICAN SYSTEMS
Staff Backend Engineer
1 day agoTatari
Staff Backend Engineer
1 day agoTatari
Software Engineer - Full Stack (Top Secret Clearance Required)
1 day agoNorth Point Technology