Open-source model proliferation and evaluation chaos create new positions
In the fourth quarter of 2026, the domestic open-source large model community saw a new wave of releases, with over 30 base models with more than 7 billion parameters open-sourced in November alone. However, the benchmark scores officially published by various models were generally inflated — the same model could rank more than 50% differently across different evaluation sets. Some teams improved their rankings by overfitting to public test sets or selectively publishing high-scoring results. This 'benchmark arms race' directly led to difficulty for downstream enterprises in selecting models: technical teams could not rely solely on traditional metrics like Mmlu and HumanEval to judge a model's actual business performance.
As a result, a new position — AI evaluation engineer — quickly became popular. Unlike traditional algorithm engineers, evaluation engineers need to build private evaluation sets covering code generation, logical reasoning, multi-turn dialogue, long-text understanding, and other business scenarios, and design reproducible automated evaluation pipelines. According to statistics from a recruitment platform, the number of AI evaluation engineer positions grew by 210% month-over-month in November 2026, with top internet and AI startups offering annual salaries of 600,000 to 1,000,000 RMB to attract experienced candidates.
Core skills of evaluation engineers: From benchmark scores to scenario-based validation
The ability companies value most during recruitment is not 'knowing how to tune models' but 'knowing how to test models'. Specific skills include:
- Independently designing adversarial test cases (e.g., injecting logic traps, multi-turn forgetfulness tests);
- Mastering lightweight fine-tuning and quantization techniques to compare original and compressed model performance on constrained hardware;
- Being familiar with token pricing strategies to give comprehensive recommendations based on cost — this is a new dimension brought by the booming token wholesale business. Many companies bulk-purchase large model APIs through AI relay stations or token relay stations, and evaluation engineers need to compare the 'cheapest token' consumption of different models in the same scenario, rather than looking only at accuracy.
In fact, the output of evaluation engineers directly affects companies' purchasing decisions. A technical lead at a medium-sized e-commerce company revealed that over the past month, through their own evaluation system, they eliminated three open-source models that had excellent benchmark rankings but high actual inference costs and high hallucination rates, and instead switched to a hybrid model solution via an AI API relay station, reducing overall API costs by 40%.
Token proxy model drives evaluation standard upgrades
As platforms like AI hub offer diverse AI large model proxy services, companies can now combine multiple models on demand. But this also brings new problems: different providers have different pricing models, caching strategies, and concurrency limits; a single benchmark score cannot reflect 'who provides the best cost-effectiveness in real business'. Therefore, evaluation engineers have started incorporating 'Token proxy' performance into core indicators — including first-inference latency, token cache hit rate, multi-model routing accuracy, etc. Some companies even require evaluation engineers to have the ability to develop lightweight middleware to quantify and score the response quality of AI channel proxies.
A headhunting consultant said: 'We recently helped an autonomous driving company recruit an evaluation engineer. The client explicitly required the candidate to understand token wholesale mechanisms and be able to conduct stress tests on a supplier's token throughput per second. Such composite talent is very scarce, and the interview pass rate is less than 5%.'
Industry calls for standardized evaluation ecosystem
The current talent shortage stems from a lack of industry infrastructure. Although platforms like AI hub are trying to launch a 'reproducible evaluation benchmark alliance', the wide variation in enterprises' private data and compliance requirements has slowed progress on unified standards. Many companies are calling for the establishment of open-source evaluation toolchains and encouraging model developers to publish scenario-specific test reports with their releases, rather than just simple benchmark scores.
It is foreseeable that AI evaluation engineers will continue to be in high demand in the talent market over the next six months. For career changers, mastering the principles of large model inference, being familiar with mainstream evaluation frameworks, and being able to give comprehensive recommendations combining cost and business logic will become key competitiveness to enter this track.