In-person and virtual community events of AIFoundry.org

AIFoundry.org

It's possible to achieve 99% accuracy in responses to AI prompts through automated testing. This is especially important when you are looking for the model to call functions and act as an agent.

At our last AI Hack Lab, Paul Zabelin of Artium demonstrated how he achieves this level of accuracy with an automated testing tool. His automated prompt testing approach tests an AI prompt hundreds of times to ensure a model returns consistently correct answers and calls functions correctly.

Now Paul is going to help everyone have such a tool by assisting the

 community in creating a generalized open-source prompt testing engine that anyone can use for any model.

By participating in this AI Hack Lab, you'll be part of the team that lays the foundation for a new, open source prompt testing engine. We'll discuss the project's requirements, potential starting points, and design, and then we’ll start coding. This is a practical opportunity to contribute to the AI community and gain hands-on experience in a real-world project.

​1. Vision of open source multi-model prompt testing suite - Greg

2. Goldman says people are failing at AI because they can’t use it reliably - Jesse

2. Demo of an existing tool by Paul Zabelin

3. Demo of OLMo-lamafile - an additional model for testing against - Sergey

4. Brainstorm: what would something like this look like?

This event is a virtual gathering, making it accessible to anyone interested in AI and open source projects. For those in San Francisco, we'll host a post-event social for networking and further discussions. Details will be provided during the event.

If you new to mob programming, but want to participate, it is suggested that you sign up for an account at 

AI Hack Lab (Virtual via Discord): Creating an Open Source Automated Prompt Testing Suite