Summary Statically hosted parquet files provide one of the easiest to use and most performant APIs for accessing bulk¹ data, and are far simpler and cheaper to provide than custom APIs. …

How unsupervised learning is used to estimate model parameters in Splink — Splink is a free probabilistic record linkage library that predicts the likelihood that two records refer to the same entity. For example, what is the probability that the following two records match?

Splink now offers support for Python and AWS Athena backends, in addition to Spark. It’s now easier to use, faster and more flexible, and can be used for close to real time linkage. — Two years ago, we introduced Splink, a Python library for data deduplication and linkage (entity resolution) at scale. Since then, Splink has been used in government, the private sector, and academia to link and deduplicate huge datasets, some in excess of 100 million records, and it’s been downloaded over 3…

The phenomenal success of data driven businesses like Google and Amazon has led to increased recognition of the value of data, and a corresponding desire for new investment. As a result, seemingly every big organisation has a new data platform that’s always just around the corner — a platform which…

Learning more about a tool that can filter and aggregate two billion rows on your laptop in two seconds — In my work as a data scientist, I’ve come across Apache Arrow in a range of seemingly-unrelated circumstances. However, I’ve always struggled to describe exactly what it is, and what it does. The official description of Arrow is: a cross-language development platform for in-memory analytics which is quite abstract —…

Fast, accurate and scalable record linkage with support for Python, PySpark and AWS Athena — Summary Splink is a Python library for probabilistic record linkage (entity resolution). It supports running record linkage workloads using the Apache Spark, AWS Athena, or DuckDB backends. Its key features are: It is extremely fast. It is capable of linking a million records on a modern laptop in under two minutes…

I once worked for a slightly terrifying senior analyst. When presenting my modelling results to her, she would often do a back-of-the-envelope calculation of her own, to estimate roughly what she expected the answer to be. If our answers differed, I knew she would challenge me to explain why. This…

How to improve the likelihood of success whilst reducing the governance burden on teams — Governance is a particularly hard problem for data improvement projects¹ because it is difficult to assess and communicate how well things are going. I suspect that the difficulty in communicating a clear picture of progress and the value that is being delivered is the key driver for the high failure…

I was inspired by a recent tweet to experiment with some of the ONS’s new beta data functionality. There’s a lot to be excited about: these new offerings are a real game changer in how government statistical data can be used in modern analytical workflows (of the kind I describe…