The One About LLM Robustness

​How do we ensure LLMs remain reliable and secure as they become more prevalent? From handling out-of-knowledge queries to protecting against novel attack vectors, join us to gain practical insights into making LLMs more robust and trustworthy.

​More About the Sharings

• ​Pradyu (Data Scientist Intern, GovTech's AI practice) will share on the team's work on out-of-knowledge base robustness in RAG - the tendency of LLMs to hallucinate despite not having sufficient context. The session will cover techniques to evaluate it, with a special focus on their open-source, flexible and modular KnowOrNot framework for developers to customize their own pipelines and measure out-of-knowledge base robustness systematically. (Technical Level: 200)

• ​Brian Formento (PhD candidate, NUS) will be sharing more about "Confidence Elicitation: A New Attack Vector for Large Language Models". As LLMs become increasingly closed-source, traditional attack methods requiring access to model internals are no longer viable. Brian's work introduces a novel approach that leverages a model's expressed confidence to guide attacks, achieving state-of-the-art results on black-box systems. Through empirical demonstrations on LLaMA-3 and Mistral-7B, learn how this technique exploits calibrated confidence measures to increase misclassification likelihood, setting new benchmarks in adversarial attacks. (Technical Level: 200-300)

​More About the Speakers

• ​Pradyu is a third-year student at NUS and is an intern at GovTech's AI practice. He is passionate about developing safe and reliable AI systems. You can follow him on X at @PradyuPrasad and read more about him at PradyuPrasad.com.

• ​Brian Formento is a PhD candidate at the National University of Singapore's School of Computing, where he researches deep learning under the SINGA scholarship. Working with Professor See-Kiong Ng, Dr. Chen Zhenghua, and Dr. Chuan Sheng Foo, his research spans machine learning, natural language processing, and computer vision. Originally from Turin, Italy, and educated in the UK with a MEng in Electronic Engineering from the University of Southampton, Brian brings a diverse international perspective to his work in AI security and deep learning systems.