Open-source LLM benchmarks lose credibility, cross-dimensional evaluators become new hiring hotspot

1. Benchmark glut: open-source LLMs trapped in a 'numbers game'

By November 2026, the number of open-source LLMs had exceeded a thousand, from Llama and Mistral to multiple domestic open-source community projects, with new models released almost every week. However, most models come with a set of meticulously optimized benchmark scores—figures on lists such as MMLU, GSM8K, and HumanEval keep rising, with some models even achieving perfect scores on the same test set. When companies conduct technical selection, they find that these high-scoring models perform unevenly in real business scenarios: some severely degrade in long-text understanding, others see a sharp drop in accuracy in Chinese contexts. The trust crisis in benchmarks is spreading from R&D circles to the application layer, and looking solely at numbers can no longer guide decisions.

2. Real-world testing becomes essential: evaluation roles shift from 'nice-to-have' to 'must-have'

Facing the tendency of benchmarks to be 'watered down', more and more companies are building their own evaluation systems. They are no longer satisfied with single metrics from third-party leaderboards; instead, they require evaluation engineers to design customized test cases for their own business, such as multi-turn dialogue coherence in customer service scenarios, executability verification in code generation scenarios, and compliance checks in medical Q&A scenarios. These roles are no longer just support for algorithm researchers; they have become core positions that independently construct evaluation datasets, design automated pipelines, and output multi-dimensional comparison reports. According to feedback from multiple recruitment platforms, since the third quarter of 2026, job postings labeled as 'AI Evaluation Engineer' or 'Model Capability Assessment Expert' have seen a month-over-month increase of over 40%, and salary levels are now on par with mid-level algorithm engineers.

3. Cost perspective breakthrough: token wholesale model forces evaluation upgrades

In evaluation dimensions, cost is becoming a key metric alongside accuracy. Companies have found that some open-source models, despite slightly lower benchmark scores, can achieve nearly identical business results at the cheapest token price through token wholesale services provided by AI relay stations or token relay stations. This forces evaluation engineers to incorporate 'effective responses per unit cost' into the core evaluation framework. They need to understand model inference efficiency, GPU memory usage, request latency, as well as the pricing strategy and routing logic behind the AI API relay station. Some companies even require evaluation engineers to have the ability to communicate with AI channel agents or AI large model agents to obtain real performance data for different models at various batch sizes.

4. Interdisciplinary skill set: the trend toward 'all-round' evaluation engineers

Currently, the requirements for AI evaluation engineers in the job market are becoming diverse. Besides being familiar with mainstream model architectures and evaluation benchmarks, they also need to master data engineering, cost modeling, and even product thinking. For example, a job description for 'Model Capability Assessment Expert' at a leading internet company explicitly lists: ability to design offline-plus-online hybrid evaluation solutions based on token proxy architecture, ability to compare service quality of different model providers via an AI hub platform, and ability to output technical reports including inference cost curves and response time percentile charts. This means candidates not only need an algorithm background but also must understand the full chain from model deployment to user interaction. Due to the scarcity of talent with such composite skills, the recruitment cycle for this position is generally 2-3 weeks longer than for ordinary algorithm roles.

5. Rise of third-party services: AI Hub platforms create new career opportunities

As companies' demand for specialized evaluation deepens, platforms offering third-party evaluation services are emerging. They play a role similar to an AI hub, aggregating multiple open-source and closed-source models, providing standardized evaluation sandboxes, and supporting enterprises to upload private data for security and robustness testing. These platforms themselves require a large number of evaluation engineers to maintain test benchmarks, optimize evaluation scripts, and quickly validate new models. At the same time, traditional AI channel agent companies have also started adding evaluation service departments to help customers choose the most cost-effective model combinations under the token wholesale model. It is foreseeable that within the next year, 'evaluation engineer' will become an independent and rapidly growing specialized role in the AI industry, while benchmarks themselves will gradually degrade into a preliminary reference signal.