Evaluation Specialist

As prompt volume scales, the gap between raw output and trustworthy insight grows. We need someone who treats evaluation quality as a discipline, not a checkbox.

• Design and maintain evaluation rubrics for answer quality, citation accuracy, and positioning signals.
• Run structured human review workflows to label and audit machine-generated analysis outputs.
• Identify systematic error patterns in the current evaluation pipeline and propose remediation.
• Collaborate with research and product to translate evaluation findings into product improvements.

• Deep experience designing annotation guidelines, evaluation rubrics, or quality review workflows.
• Strong analytical instincts for identifying bias, inconsistency, and labeling noise in structured datasets.
• Comfort working with LLM outputs and an understanding of where they tend to fail in practice.
• Clear writing and the ability to articulate why a quality standard is the right one.

• Defining what high quality actually means for a product category that lacks established benchmarks.
• Building evaluation infrastructure that improves the entire product's trustworthiness.
• Working at the interface between human judgment and automated analysis.

1. 01Audit the current evaluation rubrics and identify the top gaps in coverage or consistency.
2. 02Design a structured review workflow for at least one core analysis type.
3. 03Ship a measurable improvement to inter-rater reliability on a key evaluation task.