reality defender

· organisations

tech lead second to the cto, mlops and data engineering · new york · february 2024 to now.

the organisation

reality defender detects manipulated media, deepfakes in audio, video, image and text, for tier-one banks, government agencies and media companies. a y combinator company out of new york.

  • sxsw’s 2023 pitch award in ai and robotics (pr newswire).
  • the most innovative startup of 2024, rsa conference’s innovation sandbox (rsa conference).
  • jpmorganchase’s 2025 hall of innovation (pr newswire).
  • the company to beat in deepfake detection, gartner, december 2025 (reality defender).
  • a fifteen million dollar series a led by dcvc, expanded to thirty-three million, with ibm ventures, booz allen ventures and accenture (the series a, its expansion).

impact

  • we made data access 1,575 times faster than databricks’ own drivers, then shared the findings for the world to see (the issue).
  • we drove the pipeline with our own apache spark connect technology (the client we maintain), cut the redundant layers out of the medallion architecture (bronze does not need to exist), and shipped a hundred-terabyte warehouse of metadata to the ml team as rest endpoints: they hit a url, the backend did the plumbing, and it was synchronous.
  • we built the data platform the detection models train and are judged on from nothing: a hundred megabytes a second, over a hundred million rows a second.
  • for the customers, banks and governments among them: soc2, hipaa and gdpr auditing, and lineage, licence and source-poisoning tracing, so a model’s integrity can be shown, not asserted.

1,575 times faster

the official go sdk read a large result at about fifty-five rows a second: a hundred thousand rows in thirty minutes, a pipeline stalled and a warehouse billed for the wait.

we wrote a prototype in rust over arrow-odbc and read twenty-one million rows in two hundred and twelve seconds, about ninety-nine thousand rows a second, with the same queries, network and machine.

then we filed it with databricks, in january 2025, as a feature request with the benchmarks and a complete go program for anyone passing by: buffered ndjson streaming, a sync.Pool recycling memory, zero allocations per row.

the conversation on the issue is the interesting part.

  • the first update was ours: the arrow deserialisation was worth thirty to forty percent, and the orders of magnitude were the odbc driver’s.
  • the next was the fix in go, once databricks’ sql connector could fetch through cloud storage and arrow, posted so the next person would not need the rust.
  • the last, when databricks triaged it a year on, was that the rabbit hole had led to the apache arrow pmc and a few more bugs fixed there.

the pipeline

the medallion architecture wants a bronze layer, a silver and a gold, each a copy and a job. bronze does not need to exist.

with our own spark connect client for go, a fork we maintain with the streaming the upstream one lacked, the pipeline reads the warehouse of metadata, about a hundred terabytes, and serves it to the ml team through a strongly typed rest api with schema evolution.

the ml team never saw spark. they hit an endpoint for the slice they wanted and the backend did the plumbing: a spark connect session per request, the rows streamed back as arrow, the call returning when the data had.

synchronous, which is the special part. no queue, no job to poll, no table in the middle. a question and an answer.

what i did

the data lake and its ingestion; the rest api; the go drivers; ci/cd and a three-tier secured network; the mlops division upskilled on production-grade infrastructure and the c-suite advised. golang, databricks, spark connect, apache arrow, terraform, aws, delta lake, unity catalog, kubernetes.

what it meant

i lead by example. i outpace everyone, whether or not they are using bots, and my work merges because it is correct: there is no bottleneck at me. i use my experience to make sure the quality of our technology is one hundred percent right, with zero tolerance for exceptions. i raise the bar for everyone, and i am always excited to teach people to be as good as i am at my job.

the same discipline i learned at geneva is at work here: everything made completely lean, the least cumulative time spent by everyone, and nothing kept that does not earn its place.