The One About Evals & Paradigms
How do we ensure AI systems work as intended, and what competing visions shape their development? Join us for an afternoon exploring AI evaluation frameworks and the philosophical paradigms driving the field forward.
More About the Speakers
Prof. Ramayya Krishnan (Director, AIMSEC), who'll be providing an introduction to AIMSEC (a CMU-NIST cooperative research center) using examples of research projects on model consistency to illustrate work on model evaluation and red teaming. He will be briefly discussing the role of evaluation artifacts (sandboxes/digital twins) and pathways they offer both to static and dynamic versions of data, model/systems and evaluation cards.
Prof. Rema Padman and Mr. Bhargav SriPrakash will be sharing on the Digital Vaccine technology and its evaluation via multiple randomized field trials for improving pediatric health literacy and health outcomes. The platform incorporates the science of neurocognitive training and implicit learning embedded in the technology of AI-enabled immersive gaming in a mobile health app. The evaluation studies have investigated its impact on knowledge and behavior change related to healthy lifestyle choices and the mechanisms leading to positive outcomes.
Hui Chien will trace pivotal moments that shaped various AI paradigms - from Alignment and Accelerationism to Open Source and Data Justice. As we stand at this crucial juncture in AI's development, understanding these competing visions becomes essential for technologists, industry leaders, and policymakers alike.
James Chua will discuss his work on AI Safety and how safety research is used by model developers such as OpenAI and Anthropic. He will share about his group, Truthful AI, and their research on truthfulness, explainability, and the situational awareness of language models.
More About the Speakers:
Ramayya Krishnan is the Dean Emeritus and W. W. Cooper and Ruth F. Cooper Professor of Management Science and Information Systems at the Heinz College Information Systems and Public Policy at Carnegie Mellon University. He is an expert in data and decision analytics and digital transformation. He served as President of INFORMS in 2019 and helped lead the creation of its AI strategy. He is an AAAS Fellow (section T), an INFORMS Fellow, and an elected member of the National Academy of Public Administration. He chaired the AI futures Committee of the National AI Advisory Committee to the President and the White House office of AI Initiatives office and is chair of the DOD’s RAI academic council. He directs the CMU-NIST cooperative research center on AI measurement science and engineering. Find out more information here.
Rema Padman is Trustees Professor of Management Science and Healthcare Informatics at Carnegie Mellon University and Adjunct Professor of Biomedical Informatics at the University of Pittsburgh School of Medicine. Her research broadly investigates AI in Healthcare, combining healthcare informatics, analytics and operations for data-driven decision support in clinical and consumer-facing IT interventions in healthcare delivery. These include developing and evaluating digital therapeutic approaches for health literacy with a gamified, mobile app-based Digital Vaccine initiative for children and curating healthcare videos to inform and engage adults. Her work has been published in major academic journals such as Management Science, MIS Quarterly, Information Systems Research, Journal of the American Medical Informatics Association and Journal of Medical Internet Research. She is an elected Fellow of the American Medical Informatics Association and the Inaugural Recipient of the Bufalini Prize for applications of AI in Medicine.
Bhargav Sri Prakash is the Founder & CEO of FriendsLearn and co-founder of Carnegie Mellon University's Digital Vaccine Project. As the inventor of digital vaccines and holding the world's first patent in this field, he leads breakthrough research in neurocognitive computing and health technology. His work at FriendsLearn and CMU earned global recognition from the Financial Times and IFC as the world's leading deeptech innovation in healthcare. Carnegie Mellon University features his invention as a frontier breakthrough in AI-enabled disease prevention. A serial entrepreneur, Bhargav previously founded successful ventures in simulation technologies and VR education, with exits to General Motors and a Hong Kong PE fund. He holds an MS in Automotive Engineering from the University of Michigan and multiple patents in AI, VR/AR, and simulation technologies.
Mr Loy Hui Chien is a public officer from the Ministry of Foreign Affairs (MFA). Beyond his day job, he is passionate about harnessing digital tools to enhance his effectiveness as a professional, manager, and leader. With a background in computing dating to his school days, he believes the advent of generative AI marks the most exciting era of computing in decades. Through Lorong AI, he hopes to connect with other AI changemakers in Singapore and super-charge 10x our collective AI journey.
James Chua is an Alignment Researcher at Truthful AI, an AI Safety organisation based in Berkeley. He also spends time in the Singapore AI Safety Hub. His current focus is evaluating truthfulness, situational awareness and reasoning in language models. In a past life, he was a machine learning engineer in industry.
*Do note that programming line-up is subject to changes.