LangSmith: How to Improve LLM Apps in Production
Everyone talks about the need to quickly prototype and ship to production. This is even more important for Generative AI applications!
While the methods for developing POCs and MVPs are well-understood, best practices for operating and improving LLM applications once they are in production are much less mature.
Once deployed, you should prioritize baselining your system. Evaluating the performance of your LLMs can be different for RAG systems and fine-tuned LLMs, so you need to always focus on how your chosen metrics affect and constrain the user’s experience. Cost, latency, and token count are top-level metrics in any production environment, and it’s important to efficiently manage them using techniques like caching embeddings and prompts.
Processes for continuous improvement of these types of applications is still rapidly evolving, and the exploratory nature of trying to achieve better LLM outputs requires understanding how and when to leverage open- and closed-source tooling and what questions to ask about ensuring that your application is not just performant and scalable, but also secure and reliable.
In this event, we explore this strategy and the tactical implementation of improvement with the highly anticipated LangSmith platform. We will walk you through the monitoring, evaluation, annotation, feedback, testing, and debugging features and will highlight the key aspects of the LangChain framework that underlies the capabilities.
In this event, you'll learn:
How to evaluate LLM applications of different types.
Techniques for improving the efficiency of apps in production.
How LangSmith works off-the-shelf to streamline app improvement.
Instructors:
Dr. Greg Loughnane is the Founder & CEO of AI Makerspace, where he serves as lead instructor for their LLM Ops: LLMs in Production course. Since 2021 he has built and led industry-leading Machine Learning & AI bootcamp programs. Previously, he has worked as an AI product manager, a university professor teaching AI, an AI consultant and startup advisor, and ML researcher. He loves trail running and is based in Dayton, Ohio.
Chris Alexiuk is the Head of LLMs at AI Makerspace, where he serves as a programming instructor, curriculum developer, and thought leader for their flagship LLM Ops: LLMs in Production course. He is a solo YouTube creator Dungeons & Dragons enthusiast based in Toronto, Canada.
Follow AI Makerspace on LinkedIn & YouTube to stay updated with workshops, new courses, and opportunities for corporate training.