Lakehouse Days: Bengaluru
From Stream to Lakehouse
Calling all lakehouse geeks! e6data is hosting a hands-on meetup in Bengaluru for folks who live and breathe data infrastructure.
What’s on the menu?
Kafka flex: Confluent’s using Kafka to capture, process, and ship audit logs for every Confluent Cloud customer.
Iceberg angle: Same logs get parked in Iceberg (Confluent Tableflow) for long-term storage and analytics.
SageMaker Unified Studio + Lakehouse: AWS’ one-stop dashboard where data prep, ML, and Gen-AI workflows mingle, pulling straight from S3 & Redshift.
New-age Iceberg catalogs: e6data spotlights Polaris, Gravitino, Lakekeeper, Project Nessie, and Unity Catalog.
Streaming pain, streaming gain: Why Iceberg’s current streaming story hurts, and how e6data fixes it.
Speakers:
Ankit Garg, Senior Software Engineer, Confluent
Devanshu Bagadia, Software Engineer, Confluent
Topic: Real-time use case of data streaming platforms
Summary:
Real-time streaming use-case: Confluent cloud auditlogs: How Kafka is being used for capturing, processing, and delivering auditlogs across all customers of confluent cloud.
Using Apache Iceberg (Confluent Tableflow) for archival and data analytics use case.
Ravi Kompella, Analytics Specialist, AWS
Topic: Simplifying Data and AI with Amazon SageMaker Unified Studio and Amazon SageMaker Lakehouse
Summary:
Data workers often face complex tools and fragmented workflows. This talk introduces Amazon SageMaker Unified Studio and Amazon SageMaker Lakehouse, designed to dramatically simplify your data and AI journey.
Unified Studio provides a single, intuitive environment for all your analytics and ML tasks, from data prep to generative AI, eliminating context-switching.
Lakehouse unifies data access across S3 data lakes and Redshift, enabling you to build powerful AI solutions on a single, consistent data copy.
Discover how these services streamline your tasks, accelerate innovation, and unlock your data's full potential.
Ankur Rajan, Senior Software Engineer, e6data
Topic: Emerging Catalogs and Streaming Ingest Problems in Iceberg
Summary:
The competition for Apache Iceberg catalogs certainly remains fierce and relevant.
Several new catalogs have emerged in recent times, including Apache Polaris™ (incubating), Apache Gravitino, Lakekeeper, and Project Nessie, among others.
For data engineers, these developments are fascinating. But most of the developer community still banks on “classic” catalogs, such as Hadoop Catalog, Hive Metastore, and AWS Glue, but the landscape is expanding rapidly.
This session will delve into these catalogs with hands-on demos, discuss streaming ingest problems in Iceberg, and provide e6data’s solution to these issues.