all

Training implementation: Training System

As I write this post, cached input tokens are 10x cheaper in dollars per token than regular input tokens for both OpenAI and Anthropic's APIs.

ong

Large language model distillation isolates LLM performance on a specific task and mirrors its functionality in a smaller format. This lets developers get the same results they would get from an enormous model like GPT-4 at a lower cost and higher velocity—albeit only on that specific task. While rarely an endpoint, large language model (LLM) distillation lets data science teams kickstart the data development process and get to a production-ready model faster than they could with traditional approaches. LLM distillation basics Multi-billion parameter language models pre-trained on millions of documents have changed the world. Users can ask ChatGPT, Bard, or Grok any number of questions and often get useful answers. LLMs’ flexibility dazzles, but most AI problems don’t require flexibility. They require accuracy, speed, and efficiency.  LLMs, while amazing, tend to be slow and expensive. That’s where distillation comes in. What is LLM distillation? LLM distillation is when data scientists use LLMs to train smaller models. Data scientists can use distillation to jumpstart classification models or to align small-format generative AI (GenAI) models to produce better responses. How does LLM distillation work? LLM distillation positions a large generative model as a “teacher” and the smaller model as a “student.” The student model could be a simple model like logistic regression or a foundation model like BERT. In the most basic version of distillation, data scientists start with unlabeled data and ask the LLM to label it. Data scientists then use the synthetically labeled data to train the “student” model, which will mirror the “teacher” model’s performance on the task defined by the original data set. Data scientists can also use distillation to fine-tune smaller generative models. In this case, they would feed the “teacher” model prompts and capture the responses as training targets for the “student.” Why would you use LLM distillation? LLMs like GPT-4, Gemini, and Llama demonstrate incredible power, but also suffer notable drawbacks: Cost. Multi-billion parameter LLMs are expensive to host, and even more expensive to access via API. Speed. Due to the quantity of calculations necessary, full-sized LLMs can be slow. Infrastructure headaches. Hosting private versions of the largest available LLMs means wrangling and coordinating significant resources. By distilling an LLM, data science teams can build derivative models that are easier to host, cheaper to run, and much more responsive. What are the drawbacks of LLM distillation? While a powerful shortcut, LLM distillation is not a cure-all for training new models. The technique suffers from four primary challenges: The student is limited by the teacher. In the simplest version of distillation, the “student” model will mirror the performance of the “teacher” model. Generalized LLMs faced with specialized tasks typically fall short of production-grade accuracy. You still need a lot of unlabeled data. The LLM will create labels for you, but source data may be in short supply for any number of reasons. You may not be allowed to use your unlabeled data. For organizations that are limited from using client data, this may present a real hurdle. You may be limited in what LLMs you can use. While not an issue for classification tasks, the terms of service for many LLM APIs bar users from using their LLMs’ output to train potentially competitive generative models. The first two of these can be overcome using advanced techniques, as discussed below. Practical LLM distillation for classification challenges Basic distillation rarely yields production-grade models. In a Snorkel case study classifying user intents for a banking chatbot, our engineers started with labels from Google’s PaLM 2 to achieve an F1 of 50 as a baseline. That’s an impressive performance for an out-of-the-box model—especially considering the case study called for 77 classes—but it would not meet any bank’s bar for deployment. <img src="https://i.ytimg.com/vi/ID/hqdefault.jpg" alt="" width="480" height="360"><iframe title="Demo: How To Boost AI Accuracy with PaLM 2 and Snorkel Flow" width="1416" height="797" src="https://www.youtube.com/embed/Z2lI9N-5MNQ?feature=oembed" frameborder="0" allow="accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture; web-share" allowfullscreen></iframe> With a little prompt engineering (encouraging the LLM to behave as an expert in banking and giving one example per label), the team boosted the PaLM 2’s F1 score to 69. That’s much closer to production-grade performance, but not close enough. What, then, can a data scientist do with this not-quite-there dataset? Enrich it. How to enrich training data with targeted human labeling A model that achieves a 69 F1-score learned decision boundaries from broadly accurate data.  That same model can help spot questi

Generating the embedding for "Are OLAP cubes useful"

> The problem with Backpropagation is that it is a leaky abstraction.