Fine-Tuning Embeddings for Better Retrieval
In 2024, AI Engineers and leaders have seen the industry focus shift from RAG to agents and from LLMs to SLMs (Small Language Models).
The big idea behind using a smaller model is to ensure they are good at the domain-specific task you are trying to accomplish. In a recent event, we discussed methods of efficient pre-training and post-training (i.e., supervised fine-tuning and alignment), including Low-Rank Adaptation (LoRA) and Spectrum.
But what about the models that are being fine-tuned?
We can fine-tune open-source LLMs with [question, answer] pairs, for instance, but we can also fine-tune open-source embedding models with [question, retrieved context] pairs. This is extremely useful when building production RAG applications.
Perhaps the easiest way to improve the retrieval capabilities of your RAG application once you have a simple prototype set up is to fine-tune the embedding model based on the data in your vector store.
If you're an AI engineer or leader, this simple, powerful approach should be in your tool belt. If you’re using embedding models from Hugging Face, you can use Hugging Face’s Sentence Transformers tooling to fine-tune them.
In this event, we’ll walk through how to:
🗂️ Leverage open-source embedding models to build a simple RAG application
🧮 Set up data for fine-tuning of the retrieval system
⚖️ Complete fine-tuning using parameter-efficient methods
📈 Demonstrate performance improvements using the RAG ASsessment (RAGAS) framework
In short, we’ll construct a domain-adapted retrieval system for our RAG application that will perform even better than the off-the-shelf open-source model!
Join us live for a detailed concepts and code walkthrough!
📚 You’ll learn:
How to use domain-adapted retrieval to improve RAG applications
How to generate fine-tuning data for embedding models
Why fine-tuning embedding should be near the top of your production readiness checklist!
🤓 Who should attend the event:
Aspiring AI Engineers who want to improve their information retrieval systems.
AI Engineering leaders in need of low-cost ways to improve production LLM app performance.
Speakers:
Dr. Greg” Loughnane is the Co-Founder & CEO of AI Makerspace, where he is an instructor for their AI Engineering Bootcamp. Since 2021 he has built and led industry-leading Machine Learning education programs. Previously, he worked as an AI product manager, a university professor teaching AI, an AI consultant and startup advisor, and an ML researcher. He loves trail running and is based in Dayton, Ohio.
Chris “The Wiz” Alexiuk is the Co-Founder & CTO at AI Makerspace, where he is an instructor for their AI Engineering Bootcamp. During the day, he is also a Developer Advocate at NVIDIA. Previously, he was a Founding Machine Learning Engineer, Data Scientist, and ML curriculum developer and instructor. He’s a YouTube content creator YouTube who’s motto is “Build, build, build!” He loves Dungeons & Dragons and is based in Toronto, Canada.
Follow AI Makerspace on LinkedIn and YouTube to stay updated about workshops, new courses, and corporate training opportunities.