Surge in demand for evaluation talent: from benchmark tool operators to capability analysts
In September 2026, the large model industry is undergoing a profound transformation driven by evaluation. In the past, the industry relied heavily on standard benchmark leaderboards such as MMLU and HumanEval to assess models, and companies only needed to hire algorithm engineers to run test code. However, as open-source models approach or even surpass closed-source models on multiple general benchmarks, the industry has begun to realize that a single benchmark score is no longer sufficient to reflect a model's true capabilities. Several leading AI companies have added positions such as "model evaluation specialist" and "AI capability analyst" to their job postings this month, with salaries rising about 15% to 25% year over year. These roles are no longer limited to writing evaluation scripts; they require candidates to design evaluation dimensions based on business scenarios, such as devising evaluation schemes for vertical tasks like code generation, long-document understanding, and multi-turn dialogue consistency. A team lead involved in recruiting noted, "We need people who can interpret the meaning behind the scores, not just those who upload datasets and run results."
Open-source model benchmarks approaching closed-source: evaluation system needs urgent rebuild
This month, multiple open-source large models have achieved results close to GPT-5o on Chinese benchmarks such as C-Eval and CMMLU, and some models even lead in mathematical reasoning and code generation subtasks. This trend has directly spurred a strong demand for rebuilding evaluation systems, driving a hiring frenzy for related talent. Companies are no longer satisfied with traditional evaluation sets and are instead exploring more complex assessment methods such as adversarial testing, robustness testing, and human preference alignment testing. For example, some companies have started recruiting "evaluation adversarial designers" specifically tasked with constructing edge cases to test model performance on complex instructions, sensitive topics, and long-tail scenarios. At the same time, the thriving open-source ecosystem has accelerated model iteration cycles, demanding higher timeliness in evaluation. This has given rise to the role of "automated evaluation engineer," who needs to build continuous integration/continuous deployment (CI/CD) evaluation pipelines capable of outputting multi-dimensional capability comparison reports within 24 hours of a new model release. According to recruitment data platforms, in the first three weeks of September, the number of job postings for "evaluation engineer" related positions increased by about 40% month over month, covering algorithm development, test development, and product operations.
Capability comparison drives compound talent: algorithm and product integration becomes new standard
Amid the benchmark chaos, companies are increasingly realizing that the ultimate purpose of evaluation is to serve product deployment and business decisions. Therefore, the talent market in September has shown a distinct characteristic: a pure algorithm background is no longer enough, and compound evaluation talent with product thinking and business insight has become highly sought after. Many AI companies explicitly require candidates for "evaluation product manager" roles to be able to design evaluation metrics based on user feedback and drive model iteration. For instance, an intelligent assistant application team is hiring an "AI experience evaluation lead" responsible for building user behavior tracking systems, analyzing model generation efficiency in different scenarios, and converting evaluation results into actionable optimization suggestions. Such talent needs to understand both Transformer architecture and fine-tuning principles, be able to design A/B testing experiments, and explain capability gaps in plain language to non-technical teams. A senior recruitment consultant revealed, "Candidates who can simultaneously write evaluation code, build evaluation frameworks, and produce evaluation reports now command annual salaries exceeding 800,000 RMB, but supply still falls short of demand." In addition, startups in the evaluation field are emerging, focusing on providing customized evaluation services for small and medium-sized enterprises. These teams prefer to hire talent with contributions to open-source communities or experience in leading major evaluation competitions.
From benchmark to trust: evaluation talent becomes key role in AI industry infrastructure
As large model applications penetrate high-risk sectors such as finance, healthcare, and education, evaluation is no longer just an internal tool for technical departments but a cornerstone for building user trust and regulatory compliance. In September 2026, the draft of the "Large Model Capability Assessment and Certification Measures" jointly issued by multiple ministries was opened for public comment, explicitly requiring model publishers to provide third-party evaluation reports. This has directly stimulated cross-industry mobility of evaluation talent. Not only AI companies, but also traditional consulting firms, auditing agencies, and financial institutions are setting up large model evaluation teams. For example, one of the Big Four accounting firms announced this month the formation of a dedicated AI audit team, hiring evaluation experts with natural language processing backgrounds to assess the fairness, transparency, and security of models used by clients. Under this trend, the skill set of evaluation talent has expanded from algorithms to areas such as legal compliance, ethical auditing, and risk management. Some universities have begun adjusting their curricula, planning to offer a graduate course track in "Large Model Evaluation and Governance" in spring 2027. Industry observers believe that the explosion in demand for evaluation talent signals a shift in the large model industry from "pursuing benchmarks" to "pursuing trust," and this transformation will give rise to a entirely new career track.