LinkedIn bypassed prompt engineering and instead found success with smaller, fine-tuned models for its next-generation job recommendation system. Erran Berger, VP of product engineering at LinkedIn, revealed in a recent "Beyond the Pilot" podcast that the company determined prompt engineering was not viable for achieving the desired accuracy, latency, and efficiency improvements.

Instead, LinkedIn developed a detailed product policy document to fine-tune an initial 7-billion-parameter model. This model was then distilled into smaller "teacher" and "student" models, optimized to hundreds of millions of parameters. This multi-teacher distillation technique proved to be a breakthrough, creating a repeatable process now used across LinkedIn's AI products. "There was just no way we were gonna be able to do that through prompting," Berger said. "We didn't even try that for next-gen recommender systems because we realized it was a non-starter."

LinkedIn has been developing AI recommender systems for over 15 years. The company sought to move beyond off-the-shelf models to enhance its ability to connect job seekers with relevant opportunities. The move to smaller, more specialized models reflects a growing trend in AI development. While large language models (LLMs) have gained significant attention, they can be computationally expensive and inefficient for specific tasks. Fine-tuning smaller models on targeted datasets allows for greater control, improved performance, and reduced resource consumption.

The process involves creating a larger, more general model and then training smaller models to mimic its behavior on a specific task. This allows the smaller models to inherit the knowledge of the larger model while remaining more efficient and focused. The creation of a repeatable "cookbook" for AI development signifies a move towards standardized and scalable AI solutions within LinkedIn.

Berger emphasized the significant quality improvements resulting from this new approach. "Adopting this eval process end to end will drive substantial quality improvement of the likes we probably haven't seen in years here at LinkedIn," he stated. The company is now implementing this methodology across its AI product suite, suggesting a broader shift towards fine-tuned, smaller models within the organization. The success of LinkedIn's approach could influence other companies developing AI-powered recommendation systems, potentially leading to a greater emphasis on model distillation and specialized AI solutions.