NEAR AI Paper Club - DiPaCo
Join us for NEAR AI’s first physical paper club where we will unpack a groundbreaking approach to distributed yet modular machine learning computation, dubbed DiPaCo, which enables efficient and scalable training of large models across poorly connected and heterogeneous workers.
Key Topics
Distributing computation by paths through shared modules and uses a Local-SGD inspired optimization to keep modules in sync with reduced communication
Enabling robustness to worker failures and preemptions, making it a promising solution for large-scale machine learning applications
Exploring experimental results, DiPaCo exceeds the performance of a 1 billion-parameter dense transformer language model while reducing wall-clock time
Event Details:
Date: Tuesday, October 22
Time: 6:30pm - 8:00pm
Format: Discussion based lead by a moderator (Alexander Skidanov, Co-Founder of NEAR AI).
Audience: Researchers and practitioners interested in machine learning, distributed systems, and large-scale computing.
Active participation is encouraged!!
Before Joining the Paper Club:
Please familiarize yourself with the research paper "DIstributed Pth Composition (DiPaCo): A Co-Designed Modular Architecture and Training Approach for Large-Scale Machine Learning" before the meeting. - https://arxiv.org/pdf/2403.10616
Come prepared to engage in a discussion, sharing your thoughts, questions, and insights about the paper.
Be respectful and open-minded, listening to diverse perspectives and opinions.
Avoid presenting your own research or promoting your own work during the discussion.
Keep the discussion focused on the paper and its implications, avoiding off-topic conversations.
About the Organizers
NEAR AI: A research organization dedicated to teaching machines to code, enabling them to conduct research that advances open-source AGI.
EDGR: A research & development foundation advancing edge intelligence. Established in early 2023, Edge was founded in response to the rapidly evolving AI and next-gen web landscape, emphasizing open, autonomous, and user-centric intelligence. It currently operates one of the largest Open Source Campus in San Francisco. Follow them at @EdgeAGI.