# assign-uprn


<!-- WARNING: THIS FILE WAS AUTOGENERATED! DO NOT EDIT! -->

## About

docs  
<https://joeldn.srht.site/assign-uprn>

issue-tracker  
<https://todo.sr.ht/~joeldn/assign-uprn>

source  
<https://git.sr.ht/~joeldn/assign-uprn>

distribution  
<https://pypi.org/project/assign-uprn/>

slides  
<https://joeldn.srht.site/uprns-a-very-british-love-affair>

license  
[AGPLV3](https://git.sr.ht/~joeldn/assign-uprn/tree/main/item/LICENSE)

## Setup

### Installation

``` sh
> pip install assign-uprn
```

### Background

#### About ASSIGN

Assign is made by Endeavour Health.

> The charity funds projects that are designed to enable new, or
> improved, healthcare services, and provides unrestricted open source
> technology for those projects.

One of these projects is the ASSIGN API, which is extremely useful for
public servants standardising, validating, and de-identifying address
data.

home  
<https://endeavourhealth.org>

docs  
<https://wiki.endeavourhealth.org>

source  
<https://github.com/endeavourhealth-discovery/ASSIGN>

#### Algorithmic address-to-UPRN matching with ASSIGN

> The algorithm, known as ASSIGN (**A**ddre**SS** Match**I**n**G** to
> Unique Property Reference **N**umbers) allocates a Unique Property
> Reference Number (UPRN) to records containing addresses.

> Every property in the UK already has a UPRN. They are allocated by
> local authorities and made nationally available by Ordnance Survey. A
> UPRN gives every address a standardised format, enabling pseudonymised
> linkage to other sources of data.

> ASSIGN compares addresses in freetext form with the Ordnance Survey’s
> “AddressBase Premium” UPRN database, one element at a time, and
> decides whether there is a match. The algorithm mirrors human pattern
> recognition, so it allows for certain character swaps, spelling
> mistakes and abbreviations. After rigorous testing and adjustments,
> [ASSIGN correctly matched 98.6% of patient
> addresses](https://ijpds.org/article/view/1674) at 38,000 records per
> minute.

#### UPRN de-identification

ASSIGN can also de-identify UPRNs into Residential Anonymised Linkage
Fields (RALFs).

RALFs are locations that are pseudo-anonymised by encrypting them using
a salt which has itself been encrypted using the OpenPseudonymiser
website:

> <https://www.openpseudonymiser.org>

Salt encryption is performed by a research governance practice which
ensures matching and linking of UPRN is conducted within the protection
of the “Five Safes” framework:

> <https://ukdataservice.ac.uk/help/secure-lab/what-is-the-five-safes-framework/>

Different datasets containing UPRNs, such as datasets from across public
services, can be safely linked whilst protecting the identity of the
data subjects, through the use of RALFs created using a shared salt.

> [!NOTE]
>
> ### On re-identification
>
> De-identified data protects information about individuals within a
> safe environment, such as the “Safe Settings” element of the “Five
> Safes” framework.
>
> Still, should the information leave the protection of the “Five
> Safes”, it can then be re-identified through links with other
> datasets. This possibility is explored in the following excerpt by
> Cory Doctorow:
>
> > <https://pluralistic.net/2024/03/08/the-fire-of-orodruin/>
>
> > …it is surprisingly easy to “re-identify” individuals in anonymous
> > data-sets. To take an obvious example: we know which two dates
> > former PM Tony Blair was given a specific treatment for a cardiac
> > emergency, because this happened while he was in office. We also
> > know Blair’s date of birth. Check any trove of NHS data that records
> > a person who matches those three facts and you’ve found Tony Blair –
> > and all the private data contained alongside those public facts is
> > now in the public domain, forever.
>
> > Not everyone has Tony Blair’s reidentification hooks, but everyone
> > has data in some kind of database, and those databases are
> > continually being breached, leaked or intentionally released. A
> > breach from a taxi service like Addison-Lee or Uber, or from
> > Transport for London, will reveal the journeys that immediately
> > preceded each prescription at each clinic or hospital in an
> > “anonymous” NHS dataset, which can then be cross-referenced to
> > databases of home addresses and workplaces. In an eyeblink, millions
> > of Britons’ records of receiving treatment for STIs or cancer can be
> > connected with named individuals – again, forever.
>
> Using de-identified UPRNs in datasets does not eliminate the
> possibility of person-level records being re-identified if the data is
> not kept within the protection of the “five safes”.

### Pre-requirements

#### License to use AddressBase Premium

AddressBase Premium usage is typically used by public service providers
under the terms of the Public Services Geospatial Mapping Agreement
(PSGA). You can check whether you are licensed to use this datas with
the following lookup:

> <https://www.ordnancesurvey.co.uk/customers/public-sector/psga-member-finder>

#### API access and authentication

Endeavour health manage access, and provide the API endpoints,
usernames, and passwords that support API usage:

> <https://endeavourhealth.org>

#### Working with environment variables and python-dotenv

You can set the environment variables for API access and authorisation
directly within the environment itself, such as in a pipeline setting,
or with a `.env` file. The default location tried is the present working
directory, but the location can be passed to the package too. Here are
the variables:

``` sh
# ASSIGN_ENDPOINT may be something like:
# https://server-root-address/uprnapi/api2
ASSIGN_ENDPOINT=api_url
ASSIGN_USER=your_username
ASSIGN_PASS=your_password
```

`.env` is explicitly excluded from version control by `.gitignore` and
keeps your authentication credentials separate from the code, making it
safer and easier to share your work.

#### De-identification salt (optional)

To obtain RALFs, your information governance service can support you to
obtain a salt they have previously encrypted with the OpenPseudonymiser
website using a salt phrase created for the work being conducted under
the “Safe Projects” principle of the “Five Safes”.

The salt is encrypted using a private key known only to the University
of Nottingham (the maintainers of OpenPseudonymiser).

## Using the API

### Single address validation

A single address can be sent for matching within a single HTTP request.
A search for `10+Downing+St,Westminster,London,SW1A2AA` would receive
the following response:

``` json
{
   "Address_format":"good",
   "Postcode_quality":"good",
   "Matched":true,
   "BestMatch":{
      "UPRN":"100023336956",
      "Qualifier":"Property",
      "LogicalStatus":"1",
      "Classification":"RD04",
      "ClassTerm":"Terraced",
      "Algorithm":"10-match1",
      "ABPAddress":{
         "Number":"10",
         "Street":"Downing Street",
         "Town":"City Of Westminster",
         "Postcode":"SW1A 2AA"
      },
      "Match_pattern":{
         "Postcode":"equivalent",
         "Street":"equivalent",
         "Number":"equivalent",
         "Building":"equivalent",
         "Flat":"equivalent"
      }
   }
}
```

### Multiple address validation

Multiple addresses can be uploaded within a text file which is processed
immediately after the file has been uploaded, and downloaded shortly
afterwards.

#### Upload

The maximum number of address candidates that you can upload in a single
file is `100,000`.

The address file to be uploaded must:

- have a .txt extension
- include no headers
- contain two columns separated by a single tab character
  - The first line must not contain any header information, only data
  - The first column is a unique numeric row id
  - The second column is the address (with commas between each address
    line)

##### Example upload file content:

``` tsv
1 ⭾ 10 Downing St,Westminster,London,SW1A2AA
3 ⭾ Bridge Street,London,SW1A 2LW
4 ⭾ 221b Baker St,Marylebone,London,NW1 6XE
5 ⭾ 3 Abbey Rd,St John's Wood,London,NW8 9AY
```

#### Download

Uploads are processed straightaway and can be downloaded by referencing
the name of the upload file in the API call. The download includes data
from AddressBase Premium (plus a RALF if you’ve previously uploaded a
`.EncryptedSalt` file):

#### Example download file content:

<div style="overflow-x: scroll">

| id | uprn | address_fmt | algorithm | classification | match_building | match_flat | match_number | match_postcode | match_street | abp_number | abp_postcode | abp_street | abp_town | qualifier | adr_candiddate | abp_building | latitude | longitude | point | x | y | ralf | classification_term | abp_flat | logical_status |
|----|----|----|----|----|----|----|----|----|----|----|----|----|----|----|----|----|----|----|----|----|----|----|----|----|----|
| 1 | 100023336956 |  | 10-match1 | RD04 | equivalent | equivalent | equivalent | equivalent | equivalent | 10 | SW1A 2AA | Downing Street | City Of Westminster | Property | 10 Downing St,Westminster,London,SW1A2AA |  | 51.5035410 | -.1276700 | 51.5035410 | 530047.00 | 179951.00 | C30921C8404087803C3687301351FF41CCB4A5E8F3691070723293C8BD654CBB | Terraced |  | 1 |
| 2 | 200002501505 |  | 550-match5a | PP | candidate field dropped | equivalent | equivalent | equivalent | equivalent |  | SW1A 2LW | Bridge Street | City Of Westminster | Property | Bridge Street,London,SW1A 2LW | Portcullis House | 51.5013476 | -.1243451 | 51.5013476 | 530284.00 | 179713.00 | 4D19E2EB66A2C12BD56B93D96CFBBE5B74525AEFC4C68329BE87B55C43EA4C36 | Property Shell |  | 1 |
| 3 | 100023071949 |  | 3200-match61A170 | CR08 | moved from Number | equivalent | moved to Building | equivalent | equivalent |  | NW1 6XE | Baker Street | London | Property | 221b Baker St,Marylebone,London,NW1 6XE | 221B | 51.5237510 | -.1585550 | 51.5237510 | 527847.00 | 182144.00 | 7727B90C7C3A744AF6FD8D5A4FEB6767B1EACBBC721B85EED6AE86EDD2B0BA9C | Shop / Showroom |  | 1 |
| 4 | 100023122909 |  | 40-match1 | CR08 | moved from Street | moved from Number | moved from Flat | equivalent | moved from Building | 3 | NW8 9AY | Abbey Road | City Of Westminster | Property | 3 Abbey Rd,St Johns Wood,London,NW8 9AY |  | 51.5321562 | -.1779541 | 51.5321562 | 526478.00 | 183045.00 | 6E479D3F8DA8A548C631622EA8640E1CE9030289C5ED4458B91A4F6C4F92C799 | Shop / Showroom |  | 1 |

</div>

#### Uploading an encrypted salt (optional)

If you wish to de-identify the UPRNs, please ensure you have a
`.EncryptedSalt` file from the OpenPseudonymiser website, and under the
guidance of the Five Safes.

You can then use this package’s `upload` function to send this to the
API.

Subsequent addresses sent to the API with the `upload` function will not
only be UPRN matched, but a RALF will be provided too (see the `ralf`
column in [Example download file
content](#example-download-file-content) in this document).

## Developer Guide

This project is made with `nbdev` which uses notebooks to create and
publish the package, tests, and documentation (using quarto). It also
makes git versioning cleaner by removing notebook metadata prior to
commits:

> <https://nbdev.fast.ai/>

### Working on the assign-uprn package

`nbdev` provides a number of commands supporting development work.

Some of the commands used during a typical development workflow are
listed below:

``` sh
# install assign_uprn as editable source code
> pip install -e '[.dev]'

# make changes to notebooks in the nbs/ directory
# ...

# runs tests to ensure functions work as expected
> nbdev_test

# remove notebook metadata to make git history cleaner
> nbdev_clean

# compile to have changes apply to assign_uprn module (also runs tests)
> nbdev_prepare

# build the static website with quarto (https://quarto.org/)
> nbdev_docs

# local preview of the website with quarto
> nbdev_preview

# increment version for pypi
> nbdev_bump_version

# publish package on pypi
> nbdev_pypi
```
