delta lake

· opensource

delta lake, the open table format of the lakehouse · 9,000 stars.

what i did

three reports from running delta in production. the one that matters: a pyspark MERGE evolved the target table’s schema even when the merge condition did not match, so a failed merge still changed the table; schema evolution should follow a write, not an attempt. the others: a clustered spark cluster failing to write _delta_log from a notebook without data access granted to the notebook, and a request for options to disable expiration.

sources

  • issue #3706, delta table schema evolves even when merge condition fails, september 2024.
  • issue #2713, clustered spark fails to write _delta_log via a notebook, march 2024; issue #2717, options to disable expiration.