apache arrow

· opensource

apache arrow, the columnar memory format under spark, pandas, duckdb and most of modern data processing · the go library.

what i did

working on spark’s go client i noticed that RecordFromJSON was absurdly slow, and measured it: ten thousand records, about twenty-nine megabytes, took nine minutes and thirty-two seconds, where NewJSONReader on the same input took a hundred and thirty-seven milliseconds. five thousand times. i filed the report with the benchmarks and the code to reproduce it, the maintainer, matt topol, labelled it a bug, and two days later his fix was merged. every go program ingesting json through arrow got the speed-up.

sources

  • issue #448, 5000x performance degradation in RecordFromJSON vs NewJSONReader, 23 july 2025.
  • pull request #449, fix(arrow/array): fix RecordFromJSON perf, merged 25 july 2025.