databricks

· opensource

databricks, the data and ai platform of the fortune 500.

what i did

the go sdk. reading a large result through the official go sdk ran at about fifty-five rows a second: a hundred thousand rows in thirty minutes. i built a prototype over arrow-odbc that read twenty-one million rows in two hundred and twelve seconds, about ninety-nine thousand rows a second, and filed the comparison as a feature request with the benchmarks and a complete, production-ready go program: buffered ndjson streaming, a sync.Pool recycling memory, zero allocations per row. a blueprint for fixing the tooling, 1,575 times faster. it is the same work that reverse-engineered the java toolchains into go at reality defender.

the terraform provider. the provider’s default aws cross-account policy lacked the network permissions a workspace needs, so a deployment that followed the documentation failed with “failed credentials validation checks: allocate address”. i filed the issue and the fix, with positive and negative unit tests; it was merged into v1.53.0 in october 2024, and every new workspace since has carried it.

also: the first databricks sql warehouse driver for golang-migrate; databricks-connect-go, spark connect over databricks clusters; a rust client for the sql warehouse; and the bug reports on the python and go sql connectors and the terraform examples.

sources