Spice AI and Apache Spark™ Connect Use Case for Operational AI
Please join us to learn more about the Spice AI and Apache Spark™ Connect Use Case for Operational AI 🤝
Agenda:
Welcome and Introductions
Talk 1: Extending Apache Spark for Operational AI with Spice AI and Spark Connect
Q & A
Talk Title: Extending Apache Spark for Operational AI with Spice.ai and Spark Connect
Abstract: Apache Spark is the de facto choice for large-scale data processing. For operational use cases requiring low-latency, high-concurrency data access, data-intensive and AI applications often require additional infrastructure. Spice.ai Open Source (OSS), a single-node data and AI compute engine built in Rust, bridges this gap by integrating with Apache Spark Connect to enable federated SQL queries across Spark clusters and operational data systems, delivering sub-second query performance for operational workloads.
This talk details how Spice.ai OSS extends Spark’s capabilities, leveraging Spark Connect’s decoupled architecture for scalable, secure data access and integrating with Mosaic AI models for real-time decision-making. Attendees will learn the technical motivation for integrating with Spark Connect.
A live demo will showcase Spice.ai querying a Spark-hosted dataset, achieving sub-second query times, and feeding results into a Mosaic AI model for real-time operations
What Attendees Will Learn
Spark Connect Architecture: Understand Spark Connect’s client-application model, gRPC protocol, and its role in decoupling compute and client applications for operational use cases.
Spice.ai OSS Integration: Learn how Spice.ai OSS configures Spark Connect (spark_remote: sc://<host>:15002) for federated SQL queries, including dataset definitions in spicepod.yaml and secure authentication.
Operational Use Cases: Explore how Spice.ai extends Spark to support low-latency, high-concurrency queries for data-intensive and AI applications, unifying data access across cloud, edge, and on-premises environments.
Mosaic AI Integration: Discover how Spice.ai leverages Mosaic AI model serving to process Spark query results for real-time decision-making, such as predictive analytics or recommendation systems.
Bio: Luke Kim is the Founder and CEO of Spice AI, VC-backed startup focused on data and AI infrastructure for intelligent applications. Spice AI is the creator of Spice.ai Open Source, a portal compute runtime that unifies federated data query, retrieval, and AI inference to accelerate mission-critical, operational workloads. Luke previously led engineering teams at Microsoft, most recently as the co-creator of the Azure Incubations group in the Azure CTOs Office. A vocal advocate for open-source, he drives Spice AI’s contributions across the data and AI ecosystem, including Apache Arrow, Apache DataFusion, and Apache Spark.
📅 Date: July 30 2025
⏰ Time: 9:30 AM - 10:50 AM PST
📍 Location: online