VALIDATING THE VALIDATORS with Shreya Shanker
We're pleased to announce that our next guest on the podcast will be Shreya Shankar, a researcher working at the intersection of human-computer interaction (HCI) and AI. Shreya's research focuses on building tools and interfaces that enable collaboration between humans and AI systems, particularly in the context of large language models (LLMs).
In this live podcast recording, Hugo and Shreya will explore a range of topics related to building robust and trustworthy AI systems, with a particular focus on the challenges and opportunities of working with large language models (LLMs). They'll cover:
The importance of starting with human validation before automating the process with LLMs
Designing human-in-the-loop interfaces for AI-assisted data validation and assertion generation
Strategies for building trust in LLM-powered validators, such as clear provenance tracking and human oversight
The risks of over-automating validation processes and relying too heavily on synthetic data generation
Shreya's work on the EvalGen system for human-AI collaborative assertion generation and her upcoming project on programmatic memory over unstructured data streams
Shreya will also share her insights on the unique perspectives that HCI researchers bring to the development of AI systems, such as centering human needs and considering the long-term implications of automation.
The conversation will dive into Shreya's recent papers, "Who Validates the Validators? Aligning LLM-Assisted Evaluation of LLM Outputs with Human Preferences" and "SPADE: Synthesizing Data Quality Assertions for Large Language Model Pipelines." These works explore the challenges and opportunities in using LLMs to assist in evaluating LLM outputs and synthesizing data quality assertions for LLM pipelines.
Whether you're an AI practitioner, HCI researcher, or interested in the potential of human-AI collaboration, this episode will offer valuable insights and ideas for building more robust and trustworthy AI systems.
About Shreya Shankar
Shreya Shankar is completing her PhD in data management systems with a human-centered focus at UC Berkeley, advised by Dr. Aditya Parameswaran. Her work focuses on addressing data management challenges in production machine learning pipelines with a human-centered approach. Prior to her PhD, Shreya was the first ML engineer at Viaduct, did research engineering at Google Brain, and software engineering at Facebook. She holds a BS and MS in computer science from Stanford University.