apache spark
apache spark, the most widely deployed data processing framework there is · 44,000 stars.
what i did
spark connect is how a program talks to a spark cluster without being on it, and its go client could only collect(): a whole result into memory at once, which is no use for a result measured in gigabytes. i filed spark-52780 with the apache software foundation, priority major, and the pull request that implements StreamRows and arrow record streaming: a RowIterator that hands back arrow records batch by batch, bringing the go client toward what the scala, java and python clients already had. the review ran over ten messages on the apache mail archive with martin grund, the databricks principal engineer who leads spark connect, and matt topol, an apache arrow committer. the pull request stands open against the project; the work ships today from the fork i maintain, datalake-go/spark-connect-go, which i announced on the upstream tracker.
on the way i found what the client got wrong: a gRPC stream ending mid-frame on DataFrame.Save(); CreateDataFrameFromArrow reading only the first batch of a multi-batch table, which is silent data loss; and, chasing that, the 5,000-times performance bug in apache arrow’s json reader. before any of it, databricks-connect-go, the unofficial library that let spark connect talk to databricks clusters.
sources
- apache jira, spark-52780: the feature request, priority major, component connect, affects 4.0.0.
- pull request #152, add StreamRows and arrow record streaming; the review on the apache mail archive: the jira, the review, the discussion.
- issue #149, end-of-stream mid-frame on save; issue #150, a local iterator for streaming rows; the data-loss report on
CreateDataFrameFromArrow, july 2025, on the project’s tracker. - issue #178, the notice of the maintained fork, and the fork itself, datalake-go/spark-connect-go.