Optimizing Data Pipelines and High Performance hosted by Aerospike
Join our community sponsors and friends from Aerospike for "Optimizing Data Pipelines and High Performance"
This meetup brings together industry experts from Aerospike, Pipl, and Gong to share insights on building high-performance data pipelines, optimizing transactions, and scaling streaming solutions. We’ll cover key topics like distributed transactions, the shift from batch to streaming data, and advanced Kafka batching strategies.
Whether you're an engineer, architect, or data enthusiast, this event will help you enhance your data infrastructure for speed, scalability, and reliability.
------------------------------------------------------------
Agenda:
17:30 - 18:00 - Mingling, Snacks, Beer and Prizes.
18:00 - 18:15 - Opening Session - Oshrat Ben Avi Zabludovitz, Senior Director Country Manager, Aerospike.
18:15 - 18:45 - Achieving High-Performance Transactions in Distributed Databases - Ronen Botzer, Director of Product, Aerospike.
18:45 - 19:15 - From Batch to Streaming with Aerospike - Josh Liberty, Head of Technology, Pipl.
19:15 - 19:45 - Kafka Tenant Batching on Steroids - Nadav Hoze, Division Architect, Gong.
19:45 - 20:00 - Beer and Mingling.
*The talks will be in Hebrew.
Location: Address: Hashlosha 2 Entrance D Tel Aviv, 17th Floor, “Brain Embassy”.
———————————————————————————————
// Achieving High-Performance Transactions in Distributed Databases - Ronen Botzer, Director of Product, Aerospike.
Traditional database transactions have long been a cornerstone of database management systems, but these systems weren't built with the distributed architecture in mind. The CAP theorem poses significant challenges for implementing transactions across multiple database nodes, particularly when high performance is crucial. Consequently, only a few distributed databases can offer transactions with strict serializability guarantees.
In this session, we'll dive into the intricacies of distributed transactions and explore the following:
Workarounds and Their Pitfalls: Uncover the common methods developers use to circumvent the limitations of distributed transactions, and examine the trade-offs and downsides associated with each approach.
Aerospike 8.0's Native Transaction Capabilities: Discover how Aerospike 8.0's native transaction support provides a seamless and efficient solution, eliminating the need for cumbersome workarounds and empowering developers with robust, high-performance transactions.
Join us to gain a deeper understanding of the complexities of distributed transactions and learn how Aerospike 8.0 is revolutionizing the landscape with its cutting-edge capabilities.
// From Batch to Streaming with Aerospike - Josh Liberty, Head of Technology, Pipl.
Whatever products we develop at Pipl, the underlying capability making it all possible is our data. Keeping response times low is an absolute must, which led us to build a proprietary data layer that has served us well for many years. This layer didn’t allow incremental updates to the data, which necessitated periodic batch processes and lengthened time-to-market for new data. What we needed was a streaming new data pipeline, and a database that is capable enough to provide very fast production responses for a large amount of data entities while supporting this pipeline. In this talk we’ll describe how we built that, and describe some of the challenges we faced along the way.
// Kafka Tenant Batching on Steroids - Nadav Hoze, Division Architect, Gong.
We at GONG, developed a really useful infrastructure on top of kafka to evenly distribute our data by tenant batches ending with evenly balanced batches sent to consumers for efficient processing and moreover, enabling them to scale horizontally when throughput increased.
This worked great!! Until it didn't ... in case of really high throughput it did not deliver..
causing heavy CPU and RAM usage, suddenly the infra which its whole point was to solve a bottleneck became one..
In this lecture I will show how we revisited our internal algorithm and storage layout to make our tenant batching almost 20x faster!! all with much less CPU and RAM.
Don't miss this opportunity to learn and network with fellow tech enthusiasts! 🚀