Log-structured

Log-structured describes a data storage method where data changes are written sequentially to a log, or a series of records, rather than overwriting existing data in place. This approach prioritizes write performance and durability. Data updates are treated as new entries in the log, with previous versions potentially remaining accessible for historical analysis or data recovery. Indexing structures are then utilized to map logical data locations to physical locations within the log. This design generally favors write-heavy workloads and offers advantages in crash recovery. The underlying storage media can range from solid-state drives (SSDs) to hard disk drives (HDDs).

Log-structured meaning with examples

• Databases like Cassandra and Kafka utilize log-structured storage. When a user updates their profile, the changes are appended to the log. This provides high write throughput, as the system rarely has to perform random writes on the storage device. A background process then eventually compacts the log, combining updates and reclaiming space. This system reduces write amplification.
• In a log-structured file system (LFS), all changes, including metadata updates, are written sequentially to the log. File modifications are appended to the log, and directory entries are updated in the log to reflect the new file locations. This allows for speedy write operations because the file system writes the entirety of the modified content, not just the changed blocks.
• Distributed storage systems often employ log-structured storage for fault tolerance. Each change is replicated across multiple nodes, creating a write-ahead log. This ensures that even if one node fails, the data can be recovered from the other replicas. The write-ahead log approach also increases data availability and increases the system's ability to process reads faster than the writes.
• Log-structured merge-trees (LSM-trees) are a specific type of log-structured storage. LSM-trees optimize read performance by merging multiple sorted data structures. Data is initially written to in-memory structures, which are then flushed to disk in sorted order and then merged to a larger persistent data structure. The merge operation then cleans up duplicates.