Designing I/O-Efficient Algorithms for Large-Scale Data

As datasets grow exponentially, traditional in-memory algorithms struggle when data no longer fits into RAM. To build high-performance systems, software engineers and data specialists must understand how to minimize slow storage access and optimize data movement. This course guides you through the principles of external memory and cache-oblivious algorithms. You will transition from writing standard algorithms to designing systems that optimize data transfer across the modern memory hierarchy, drastically improving performance for large-scale databases, file systems, and data pipelines. What you'll learn: - Understand the fundamental limits of the memory hierarchy, including the differences between RAM, SSDs, and modern NVMe storage. - Analyze algorithms using the external memory model to measure and minimize disk I/O operations. - Design foundational external-memory data structures, such as B-trees and buffer pools, for efficient data retrieval. - Apply cache-oblivious design principles to write algorithms that run efficiently across various cache sizes without manual tuning. - Compare modern storage paradigms, including Log-Structured Merge (LSM) trees and traditional B-trees, used in modern database engines. You will begin with foundational concepts of memory hierarchies and I/O costs before progressing to core external memory data structures, external sorting techniques, and modern database storage architectures. Through detailed written explanations and algorithmic walkthroughs, you will learn to think about computation in terms of data transfer rather than just CPU cycles. This course is designed for software developers, computer science students, and data engineers who want to understand high-performance data processing. No prior experience with external-memory algorithms is required, as we start with the absolute basics. Start mastering the algorithms that power modern, large-scale data systems today.