Events

Slow iteration cycles, inaccurate responses, and poor retrieval are just a few of the issues that currently plague the language models (LLMs) used today. So, how can you efficiently fine-tune and evaluate the performance of an LLM that powers your application?

​In this hands-on workshop, you will fine-tune a popular open source LLM (Google’s Flan-T5) at scale using Ray and HuggingFace. Additionally, you will visualize and inspect your model predictions using Arize Phoenix. 

You’ll learn about flexible and scalable approaches to fine-tuning. Then you’ll also learn how to visualize and analyze clusters of data points inside a notebook - with the ultimate goal of extracting tangible insights for fine-tuning an LLM to your own data, task, or desired response approach. 

​** Please bring your laptop to follow along with this hands-on workshop.

Note: By RSVP'ing to this event, you are opting in to receive event updates and marketing communications from Arize AI and Anyscale according to their respective privacy policies: 

Workshop: Fine-tune and Analyze LLMs

VOYAGER, the first LLM-powered embodied lifelong learning agent in Minecraft, autonomously explores the world, acquires skills, and makes discoveries without human intervention. It outperforms previous approaches, achieving exceptional proficiency in playing Minecraft and successfully applies its learned skills to solve novel tasks in different Minecraft worlds, surpassing techniques that struggle with generalization.

Community Paper Reading: VOYAGER

As more and more companies start using LLMs to build their own chatbots or search systems, poor retrieval is a common issue plaguing teams. So, how can you efficiently build and analyze the LLM that powers your search system, and how do you know how to improve it? If there isn't enough context to pull in, then the prompt doesn't have enough context to answer the question. 

Imagine you've built and deployed an LLM question-answering service that enables users to ask questions and receive answers from a knowledge base. You want to understand what kinds of questions your users are asking and whether you're providing good answers to those questions.

​​In this hands-on workshop, you will download a pre-indexed knowledge base and run a LlamaIndex application. You will download user query data and knowledge base data, including embeddings computed using the OpenAI API. Using Phoenix, you will investigate clusters of user queries with no, or limited, corresponding knowledge base entries. 

​You’ll learn how to identify if there is decent overlap between queries and context, locate where there is a density of queries without enough context,  and the next steps you can take to fine tune your model. 

​​** Please bring your laptop to follow along with this hands-on workshop.

Building and Analyzing an AI-powered Search System with Arize & Llamaindex

Community Paper Reading: HyDE

Every Wednesday Arize hosts an interactive community paper reading session to go over the latest research papers. This week we are analyzing GLoRA: a universal, parameter-efficient fine-tuning approach for diverse tasks. GLoRA enhances LoRA with a generalized prompt module, optimizing pre-trained model weights and activations. Its scalable, layer-wise structure search enables efficient parameter adaptation. GLoRA excels in transfer learning, few-shot learning, and domain generalization, outperforming previous methods on various datasets. With fewer parameters and no extra inference cost, GLoRA is a practical solution for resource-limited applications. Join us to explore GLoRA’s capabilities in this interactive community paper reading!

Community Paper Reading: GLoRA

In this hands-on virtual workshop, you’ll use Phoenix, Arize AI's open-source library for ML observability, to monitor an NLP model and identify the root-cause of a production issue. 

This workshop will explain the concept of ML observability from first principles. Through concrete examples, you will see how Phoenix makes your machine learning systems observable by visualizing your data, by detecting data quality, drift, and performance issues, and by helping you pinpoint the root-cause of problems so you can resolve them quickly. In particular, you'll follow an active learning workflow to automatically identify and export problematic production data for labeling and fine-tuning of your NLP model.

In this interactive workshop, you will learn how to:

ML Observability and Active Learning for NLP Models with Arize Phoenix

As more and more companies start using LLMs to build their own chatbots or search systems, poor retrieval is a common issue plaguing teams. So, how can you efficiently build and analyze the LLM that powers your search system, and how do you know how to improve it? If there isn't enough context to pull in, then the prompt doesn't have enough context to answer the question.

​Imagine you've built and deployed an LLM question-answering service that enables users to ask questions and receive answers from a knowledge base. You want to understand what kinds of questions your users are asking and whether you're providing good answers to those questions.

​​​In this hands-on workshop, you will download a pre-indexed knowledge base and run a LlamaIndex application. You will download user query data and knowledge base data, including embeddings computed using the OpenAI API. Using Arize Phoenix, you will investigate clusters of user queries with no, or limited, corresponding knowledge base entries.

​​You’ll learn how to identify if there is decent overlap between queries and context, locate where there is a density of queries without enough context,  and the next steps you can take to fine tune your model.

LLM Search & Retrieval Systems with Arize and Llamaindex: Powering LLMs on Your Proprietary Data

In today’s hyper-connected world, the pace of AI development and innovation is staggering, reshaping industries and redefining what’s possible at unprecedented speed. According to a recent survey, over half (53%) of data science and machine learning teams say they plan to deploy large language model (LLM) applications into production in the next 12 months or “as soon as possible” – however, nearly as many (43%) cite issues like accuracy of responses and hallucinations as a main barrier to implementation. 

In this Lunch & Learn, we will discuss how to best alleviate the challenges machine learning and data science teams face when implementing LLMs in production. 

LLM Observability 101: Common Challenges Seen in Production

During this week’s paper reading event, we are thrilled to announce that we will be joined by Frank Liu of Zilliz, who will be sharing valuable insights with us. This paper examines Position Interpolation (PI), a method extending context window sizes of LLaMA models up to 32,768 positions with minimal fine-tuning. The extended models showed strong results on tasks requiring long context and retained their quality within the original context window. PI avoids catastrophic attention score issues by linearly down-scaling input position indices. The method’s stability was demonstrated, and existing optimization and infrastructure could be reused in the extended models. 

Additionally, during the event, we will also discuss the write-up “Extending Context is Hard… But Not Impossible” available at 

Community Paper Reading: Extending the Context Window of LLaMA Models

​In today’s hyper-connected world, the pace of AI development and innovation is staggering, reshaping industries and redefining what’s possible at unprecedented speed. According to a recent survey, over half (53%) of data science and machine learning teams say they plan to deploy large language model (LLM) applications into production in the next 12 months or “as soon as possible” – however, nearly as many (43%) cite issues like accuracy of responses and hallucinations as a main barrier to implementation. 

​In this Lunch & Learn, we will discuss how to best alleviate the challenges machine learning and data science teams face when implementing LLMs in production. 

Unlocking the Power of LLMs: A Path to Optimizing, Scaling, and Maximizing ROI

Join us for an exciting hands-on workshop co-hosted by Arize Phoenix and Milvus, where we will delve into the world of querying using cutting-edge technologies like Arize Phoenix, LlamaIndex, LangChain, GPT 3.5 Turbo, NLTK, and Milvus. This workshop is designed for developers and enterprises seeking to apply large language models to common use cases.

In this workshop, we will guide you through the process of building and troubleshooting a powerful query engine. Before we jump into the code, we will provide a comprehensive overview of essential concepts to ensure everyone understands the foundation of what we are about to create.

By the end of this hands on workshop, you will have gained valuable insights into querying and acquired practical knowledge of leveraging cutting edge technologies to develop efficient language processing applications.

Join us for an eventful evening of learning followed by an evening of networking, delicious food, and refreshing drinks!

** Be sure to bring your computer to follow along! 

Building And Troubleshooting An Advanced LLM Query Engine

Join Arize Co-Founder, Jason Lopatecki, and ML Growth Lead, Amber Roberts, as they discuss “AgentBench: Evaluating LLMs as Agents”. This paper explores  AgentBench, the first benchmark designed to evaluate LLMs' ability to operate as autonomous agents in various scenarios. We'll talk through the paper's finding of a significant performance gap between leading commercial API-based LLMs and open-source alternatives, and the impact that disparity will have on the future the industry.

Community Paper Reading: Evaluating LLMs as Agents

The future of LLM engineering requires managing the context window of LLMs through prompt engineering and prompt composition. This session will review modern techniques for prompt templates both reviewing basic approaches and Microsoft Guidance. In addition, we will review generating and controlling structured output using AI functions. This sessions with leave attendees with practical approaches to managing prompts, responses and prompt window composition.

Prompt Templates, OpenAI Functions and Context Window Management

Large Language Models have the power to answer questions specific to your data. There are a few limitations, like the limited input token length and retrieval of the wrong context. Chunking is used in LLMs to break down long text documents into smaller sections. For example, you can chunk a PDF document by the headings or use a text splitter. Implementing ranking models is an effective strategy to enhance search quality. With these techniques in mind, we can use LLMs in production and interact with your specific data.

Unstructured will chunk the documents and store the objects in the Weaviate vector database. Weaviate is used to orchestrate the inference for the embeddings and the Cohere Reranking API. We will then evaluate the results in Phoenix and spot the cases where the retrieval failed.

Optimizing RAG With LLMS: Exploring Chunking Techniques and Reranking for Enhanced Results

Join us for an insightful evening dedicated to scaling LLM-powered systems from experimental stages to real-world production environments.

This event brings together developers, researchers, and technology leaders to explore groundbreaking topics essential to building the next generation of robust AI-centric solutions, including:

Engage in lively debates, practical workshops, and presentations from industry pioneers that illuminate the pathway from toy models to production-ready, scalable solutions. Whether you are an established professional or an aspiring AI engineer, this event promises to deepen your understanding of LLM-powered systems and inspire new ways to leverage its vast potential.

Let's transform ideas into reality together - reserve your spot today!

From Toy to Production: Building LLM-Powered Systems that Work in the Real World

Join us for an exciting workshop co-hosted by Arize Phoenix and Milvus, where we will delve into the world of querying using cutting-edge technologies like Arize Phoenix, LlamaIndex, LangChain, GPT 3.5 Turbo, NLTK, and Milvus. This workshop is designed for developers and enterprises seeking to apply large language models to common use cases.

Building & Troubleshooting An Advanced LLM Query Engine

Join us, September 19th and 26th for this free, 2-part, virtual workshop where participants will have hands-on experience in constructing LLM-powered AI agents . 

In Part 1 of this workshop, we will discuss agent paradigms, as well as how to leverage tools and LLM chaining to customize your agents with LangChain. In Part 2, we will dive into a code-along Google Colab notebook to tackle several types of agents that can be used to tackle a variety of language tasks. Attendees will walk away with the ability to implement a specialized agent for their use case.

Ideal for LLM developers, data scientists, ML engineers, and NLP enthusiasts. Join us for this deep dive into LLMOps and take your understanding of LLM chaining and ReAct prompting to the next level!

Agents, Tools, and Chains for LLMOps

Join Arize Co-Founder, Jason Lopatecki, and CTO, Michael Schiff, as they discuss “Explaining Grokking Through Circuit Efficiency”. This paper explores novel predictions about grokking, providing significant evidence in favour of its explanation. Most strikingly, the research conducted in this paper demonstrates two novel and surprising behaviors: ungrokking, in which a network regresses from perfect to low test accuracy, and semi-grokking, in which a network shows delayed generalisation to partial rather than perfect test accuracy.

Community Paper Reading: Explaining Grokking Through Circuit Efficiency

Join us for an exclusive evening focused on "Pushing RAG to Production and Troubleshooting LLM Apps" and innovative methodologies to evaluate and troubleshoot LLMs in a production environment. Co-hosted by Arize AI, 

, and Weaviate, this event is a must-attend for industry professionals, from startups to established enterprises, interested in the latest development paradigms and deployment strategies in generative AI and LLM Apps.

Pushing RAG to Production and Troubleshooting LLM Apps

Step into the world of LLM evaluations with a 3-part series dedicated to achieving production excellence. We’ll unpack advanced evaluation techniques and best practices formulated through rigorous testing — spanning retrieval, summarization, and hallucination — to help ensure production readiness. A must-attend for AI & ML engineers and data scientists. 

Session 1 (10/3): Benchmarking and Analyzing Retrieval Approaches

Session 2 (10/10): Statistical Analysis of Summarization LLM Evaluations

Session 3 (10/16): Statistical Analysis of Hallucination LLM Evaluations