Delta Lake is an open table format and transaction protocol that brings ACID transactions, versioning and schema enforcement to data lakes. Originating at Databricks and now a Linux Foundation project, it keeps an ordered transaction log alongside Parquet files so that concurrent readers and writers see a consistent table.
Delta Lake
Delta Lake adds a transaction log to a directory of Parquet files, and that one addition turns a data lake into something a database engine can trust. Every change is an ordered, atomic commit; readers pin a version; schema rules are enforced on write rather than discovered on read.
- ACID transactions - Concurrent writers commit through an ordered log rather than racing on object storage.
- Time travel - Any previous version of a table can be queried by version or timestamp.
- Schema enforcement and evolution - Bad writes are rejected at the boundary instead of corrupting downstream analysis.
- Open protocol - The log format is specified and implemented outside its originating vendor.
Delta Lake sits in the same layer as Apache Iceberg and Hudi, and the three of them together explain a pattern visible in the scores: the data and analytics market standardised its storage thoroughly and its interfaces barely at all. A team can move a table between engines with confidence and still cannot read the vendor’s API without a driver and a dialect.
Referenced in API Evangelist papers
This standard shows up in my published research. These reports read the machine-readable evidence provider by provider — and put this standard in the context of a real sector.
The State of Data & Analytics APIs
One of the three open table formats that decoupled analytical storage from the query engine — settled in public, in the open, and entirely outside the API layer.
