LinkedIn bypassed prompt engineering and instead used model distillation to create its next-generation recommendation system, according to Erran Berger, VP of product engineering at LinkedIn, speaking on the Beyond the Pilot podcast. The company, a long-time leader in AI recommender systems, found that prompt engineering was insufficient for achieving the desired levels of accuracy, latency, and efficiency required for matching job seekers with opportunities.

Berger stated that prompting was considered a "non-starter" for this specific application. Instead, LinkedIn developed a detailed product policy document to fine-tune an initial 7-billion-parameter model. This model was then distilled into smaller, more efficient teacher and student models, optimized to hundreds of millions of parameters. This process of multi-teacher distillation proved to be the key breakthrough.

Model distillation is a technique in machine learning where a smaller, more efficient model (the student) is trained to mimic the behavior of a larger, more complex model (the teacher). This allows for the deployment of AI models in resource-constrained environments without sacrificing significant accuracy. In LinkedIn's case, the initial 7-billion-parameter model served as the foundation for creating smaller, more specialized models tailored for specific recommendation tasks.

The development of this new approach has resulted in a repeatable "cookbook" that is now being applied across various AI products within LinkedIn. Berger anticipates that the adoption of this end-to-end evaluation process will lead to substantial improvements in quality, exceeding those seen in recent years.

LinkedIn's experience highlights a growing trend in the AI community: the shift towards specialized, fine-tuned models rather than relying solely on prompt engineering with large language models. While prompting has its place, it may not always be the most effective or efficient solution for complex tasks requiring high precision and low latency. The company's success with model distillation suggests that a more targeted approach, involving careful model design and training, can yield superior results in certain applications. The implications of this approach could extend beyond recommender systems, influencing the development of AI solutions in various industries.