Office Hours with Spare Cores

​​Discuss, showcase, learn, and share feedback with the Metaflow community!

​​​In this edition, Gergeley Daróczi at Spare Cores will show how you can magically right size the compute resources dedicated to your AI/ML workloads

​Metaflow makes it easy to manage workflows, with handy features like automatic artifact archiving and per-step cloud resource setup. But keeping an eye on actual resource usage—CPU, memory, GPU—still falls to you or whatever orchestration layer you’re using, like Kubernetes.

​To make this easier, we built an open-source Python package that tracks resource usage at both the system and process levels. Just add a @track_resources decorator to your Metaflow steps, and you’ll get interactive reports with recommendations for right-sizing your resources and picking cost-efficient cloud instances. We’ll also show how this could evolve to automatically tune resources in remote executors.

​Join us for a quick tour of the project and a live demo!

​​​We meet fortnightly on Tuesdays on Google Meet.