AI Hack Lab (Virtual via Discord): Creating an Open Source Automated Prompt Testing Suite
Who should attend?
Developers working with AI
ML Engineers bringing models to production
Prompt Engineers
What will we be working on?
It's possible to achieve 99% accuracy in responses to AI prompts through automated testing. This is especially important when you are looking for the model to call functions and act as an agent.
At our last AI Hack Lab, Paul Zabelin of Artium demonstrated how he achieves this level of accuracy with an automated testing tool. His automated prompt testing approach tests an AI prompt hundreds of times to ensure a model returns consistently correct answers and calls functions correctly.
Now Paul is going to help everyone have such a tool by assisting the AIFoundry.org community in creating a generalized open-source prompt testing engine that anyone can use for any model.
By participating in this AI Hack Lab, you'll be part of the team that lays the foundation for a new, open source prompt testing engine. We'll discuss the project's requirements, potential starting points, and design, and then we’ll start coding. This is a practical opportunity to contribute to the AI community and gain hands-on experience in a real-world project.
How to attend the event?
Date & Time: Saturday Jul 13, 2024 between 930AM PDT to 12:30 PM PDT
Where: AIFoundry.org Discord event channel
Brief agenda:
1. Vision of open source multi-model prompt testing suite - Greg
2. Goldman says people are failing at AI because they can’t use it reliably - Jesse
2. Demo of an existing tool by Paul Zabelin
3. Demo of OLMo-lamafile - an additional model for testing against - Sergey
4. Brainstorm: what would something like this look like?
5. Discussion: What do we need to build?
6. Decide on next steps
7. Mob programming
This event is a virtual gathering, making it accessible to anyone interested in AI and open source projects. For those in San Francisco, we'll host a post-event social for networking and further discussions. Details will be provided during the event.
If you new to mob programming, but want to participate, it is suggested that you sign up for an account at tuple.app.
Who is hosting?
The event is being hosted by AIFoundry.org and the following members of the team:
Paul Zabelin, Sr. Software Engineer at Artium
Roman Shaposhnik, CTO of Nekko.ai
Sergey Sergyenko, VP of Engineering at Nekko.ai
Greg Chase, Chief Marketing & Community at Nekko.ai