​AI Benchmark Club

​tl;dr Come join us for technical talks focusing on AI benchmarks. We're a group of AI engineers, researchers, and academics.

​This session includes talks from these speakers:

• ​Pete Warden (CEO Moonshine AI, founding member of TensorFlow, and author of TinyML) and Adam Sabra (Machine Learning Engineer, Moonshine AI)

• ​Taras Sereda (Machine Learning Researcher Gimlet Labs, Research Engineer @ Whisper)

​🫵 Who should join?

​Are you a cracked engineer looking to squeeze out the last bit of latency from your AI workload? Or a researcher looking for novel optimization techniques to improve accuracy? This is a technical meetup where we dive into real benchmarks and performance-improving techniques across multiple dimensions.

​⏰ When, Where?

​📆 Date & Time: Wednesday, July 30 @ 5:30 PM
📍 Location: Hamm's Building Penthouse (Bryant & 15th, SF)

(Food and drinks provided. 🍕🍺)

​🔥 Details

This session will feature two awesome technical talks by Pete Warden / Adam Sabra, and Taras Sereda.

Agenda:

• ​5:30 - 6: Arrival, Food, Networking

• ​6 - 7: Technical Talks, Q&A

• ​7 - 7:30: Post Talk Networking

​Talk Info:

• ​🇯🇵 💬 Assessing Japanese ASR Models (When you don't speak Japanese)
  Pete Warden and Adam Sabra will discuss they they approach evaluating automatic speech recognition (ASR) systems in unfamiliar languages, and specifically the work they've done in ensuring accuracy for Japanese language models. They'll also discuss concrete steps (and tradeoffs depending on time and cost) to develop evaluation sets for domain-specific use cases.
  

• ​💥🌽 How Good is AI at Generating AI Kernels?
  Taras Sereda will discuss evaluating frontier models' ability to generate kernels for Apple's Metal framework, and comparing performance and correctness against the default PyTorch implementation. He will detail a multimodal agentic architecture for kernel generation, which even automatically screen grabs performance information from Apple's developer tools to provide live feedback to the model.

​🎤🎤 Apply to Speak at a Future Event

​Interested in sharing AI benchmarking work you've done at a future session? Tell us about it.