LinkedIn bypassed prompt engineering and instead used model distillation to create its next-generation AI recommender systems, according to Erran Berger, VP of product engineering at LinkedIn. Speaking on the Beyond the Pilot podcast, Berger stated that prompting was considered a "non-starter" for achieving the necessary accuracy, latency, and efficiency improvements.

Instead of relying on prompt engineering, LinkedIn developed a detailed product policy document to fine-tune a 7-billion-parameter model, which was subsequently distilled into smaller, optimized teacher and student models with hundreds of millions of parameters. This approach, known as multi-teacher distillation, has become a repeatable method used across LinkedIn's AI products.

LinkedIn has been developing AI recommender systems for over 15 years. The company found that off-the-shelf models were insufficient to meet the demands of its job-seeking platform. The move to model distillation represents a significant shift in strategy, prioritizing custom-built solutions over leveraging existing large language models (LLMs) through prompting.

Model distillation is a technique where a smaller, more efficient model (the student) is trained to mimic the behavior of a larger, more complex model (the teacher). In LinkedIn's case, multiple teacher models were used to guide the student model, enhancing its performance. This process allows for the creation of AI models that are both powerful and resource-efficient, a crucial factor for large-scale applications.

Berger emphasized the impact of this new evaluation process, stating that it would "drive substantial quality improvement of the likes we probably haven't seen in years here at LinkedIn." The company believes this approach will lead to more relevant and personalized job recommendations for its users.

The development highlights a growing trend in the AI industry: the move towards specialized, fine-tuned models designed for specific tasks. While large language models have gained significant attention, companies like LinkedIn are finding that custom solutions can provide superior performance in certain applications. The implications of this trend could lead to a more diverse AI landscape, with smaller, more efficient models playing an increasingly important role.