# PIDs in NFDI4Cat

> *This document is work in progress (and not yet a reference).*

Persistent identifiers (PIDs) play an important role in achieving FAIR data.
This text is about the different use cases for PIDs in NFDI4Cat and how each case is solved technically.

## PIDs for Data Portals

According to the [architecture document](https://zenodo.org/doi/10.5281/zenodo.10391090), different repositories are considered for the NFDI4Cat data infra structure:

- **Repo4Cat**, the central global repository for publishing data and collaborating on data.
- Local repositories hosted at various institutions, e.g. BasCat or LARA

Since Researchers expect DOIs for their data publications,
the data sets published in Repo4Cat should get a DOIs from DataCite as PID.

However, DOIs are not a good fit for earlier stages of research due to the metadata requirements and associated costs.
Nevertheless researchers want to share research artifacts already in the early stage with collaborators and having PIDs already at this stage would be very useful.
Therefore, NFDI4Cat will in addition to DOIs use a handle-based solution **pid4cat** (see below)
to address the need for PIDs in the early phase when data are shared in private with collaborators. Repo4Cat seeks to support this private sharing between collaborators.

NFDI4cat has selected Dataverse as software for Repo4Cat, the global data portal.
Dataverse supports two types of PIDs: DOIs and handles.
It is however not possible to use handles and DOIs at the same time in a single Dataverse installation.

To address this limitations, two possible solutions are discussed:

- [preferred] Use one Dataverse instance and pid4cat-handles as primary PID. For data sets being published DOIs will be minted (and linked to the handle).
- Use two Dataverse instances, one for published data that is configured to use DOIs and another one for early-stage data that is configured to use PID4-Cat-handles.

For the handle-PIDs in Dataverse, it is suggested

- To generate individual PIDs for all files in a dataset.

  Configuration
  - [:DataFilePIDFormat](https://guides.dataverse.org/en/latest/installation/config.html#datafilepidformat) = independent

- To generate the ID-part via a procedure stored in the DB.
  What scheme is used does not matter much, but clashes with pid4cat-handles or other Dataverse (BasCat) must be avoided.
  Via a stored procedure UUID4 (or UUID7) identifiers could be created.
  An alternative is to add a prefix to the numeric ID used by default.
  The numeric ID may be base32 encoded to shorten the ID length.
  In this case the start ID should maybe not set to 1 but to 1001 to make the numeric part stick out.
  For example:
  - r4c-9OH    (prefix and base32 encoded integer number)
    - "9OH" is 10001 using [base-32 alphabet](https://en.wikipedia.org/wiki/Base32#base-32)
  - r4c-10001  (prefix & integer number)

  Preferred is the first scheme, because it is shorter.

  It is suggested to not use RFC4648 base32 (letters A to Z and numbers 2 to 7) but another, more human-friendly alphabet, preferably z-base-32.
  The z-base-32 encoding avoids human reading problems by excluding 0, l, v, 2
  (number 0 may be confused with letter o, letter l with number 1 or letter i, letter v is close to u or r especially in handwriting, also number 2 is close to z in handwriting).
  The z-base-32 alphabet is optimized for lowercase letters so should be represented in this form.

  Configuration:
  - [:IdentifierGenerationStyle](https://guides.dataverse.org/en/latest/installation/config.html#identifiergenerationstyle) = storedProcGenerated

Related Dataverse documentation:

- [General Dataverse PID documentation](https://guides.dataverse.org/en/latest/installation/config.html#persistent-identifiers-and-publishing-datasets)
- [Dataverse handle configuration](https://guides.dataverse.org/en/latest/installation/config.html#configuring-your-dataverse-installation-for-handles)


## PIDs for Catalysis Research Resources (pid4cat)

PIDs like the well known DOIs require a certain set of metadata (see DataCite Schema).
DOIs are based on handles, which in principle allow to store additional data in the record.
However, DOI metadata are stored in a different service and have to be requested via the DataCite API
which is a different service than the handle-system used for resolving a DOI.
The reason for DataCite`s approach is performance(?).

For NFDI4Cat a simpler system that reduces complexity is more attractive.
Therefore, pid4cathandles are similar to [ePIC](http://www.pidconsortium.net/)-handles in that they use the handle record itself to store metadata.
This has the advantage that these metadata are available directly from the resolver.

pid4cat comes with a well-defined schema for how and which metadata to store in the handle record.
The schema is defined in [LinkML](https://linkml.io/linkml/) and developed in this repo ([NFDI4Cat/PID4Cat](https://github.com/nfdi4cat/pid4cat-model/)).
The schema does only require a minimum of information; it does not disclose anything about the resource except its type. Moreover, it is not required to specify an owner/creator but only a curator. Therefore, pid4cat-handles do not mandate to put resources into context if this is undesired for whatever reasons.

### Mapping of pid4cat schema to handle record

A typical handle record is shown in the table.

**Table** Example record for a pid4cat handle with the suffix "lca/dfi345", URL: https://hdl.handle.net/20.1000/lca/dfi345 <BR>  The column *PID4CatElement* is not part of the handle record; it links to the documentation of the corresponding class.

| Index | Type | Timestamp | Data | *PID4CatElement* |
|-------|------|-----------|------|-----------|
|     1 | URL  | 2024-01-01 10:47:38Z | https://pid4cat.example.org/lca/dfi345 | *landing pageURL* |
|     2 | STATUS | 2024-02-19 13:40:02Z | REGISTERED | *[status](https://nfdi4cat.github.io/pid4cat-model/dev/elements/STATUS/)* |
|     3 | SCHEMA_VER | 2024-01-01 10:47:38Z | v1.0.0 | *[pid4cat model version](https://nfdi4cat.github.io/pid4cat-model/dev/elements/SCHEMAVER/)* |
|     4 | METADATA_LICENSE | 2024-01-01 10:47:38Z | CC0-1.0 | *[license](https://nfdi4cat.github.io/pid4cat-model/dev/elements/LICENSE/)* |
|     5 | EMAIL | 2024-01-01 10:47:38Z | datafuzzi@example.org | *curation contact email* |
|     6 | RESOURCE | 2024-01-01 10:47:38Z | {json} | *[resource info](https://nfdi4cat.github.io/pid4cat-model/dev/elements/RESOURCE/)* |
|     7 | RELATED | 2024-02-19 13:40:02Z | {json} | *[related identifiers](https://nfdi4cat.github.io/pid4cat-model/dev/elements/RELATED/)* |
|     8 | CHANGES | 2024-02-19 13:40:02Z | {json} | *[change log for PID record](https://nfdi4cat.github.io/pid4cat-model/dev/elements/CHANGES/)* |

The pid4cat schema is "mapped" to the handle record by using [type designators](https://linkml.io/linkml/schemas/type-designators.html).
The TYPE column in the handle record determines ("designates") the corresponding class of the LinkML model.

In a future version, the non-standard values in the **Type**-column could be replaced by references to type declarations in a datatype registry (DTR).
Such DTRs are still under development and not yet widely used.

The METADATA_LICENSE specifies the licence for the metadata in the PID-record.
It will be fixed to CC0-1.0 in the NFDICat service to facilitate reuse.

Since pid4cat is a linkML-model we have all tools at hand to create records or an API.

The folder `examples` has a full example of creating a model instance from Python and
how the relsulting `json` looks like; the example uses all key elements of the model (all classes and slots).

Below you you can see a basic example, of using the pydantic-model created from the PID4cat schema to create a json-objects for PID records (code in `nfdi4cat_details_ex.py`):

```python
from linkml_runtime.dumpers import json_dumper
from pid4cat_model.datamodel import pid4cat_model_pydantic as p4c

p1_repr_variants = [
    p4c.RepresentationVariant(
        variant_url="https://example.org/resource",
        media_type="text/turtle",
        encoding_format="UTF-8",
        size=12345,
    ),
    # could be more than one representation variant
]

pid1_resource_info = p4c.ResourceInfo(
    label="Resource label",
    description="Resource description",
    resource_category=p4c.ResourceCategory.SAMPLE,
    representation_variants=p1_repr_variants,
)

print(json_dumper.dumps(pid1_resource_info, inject_type=False))
```

which will print the json-object to be stored under index 7 in the handle-record:

```json
{  
  "label": "Resource label",
  "description": "Resource description",
  "resource_category": "SAMPLE",
  "representation_variants": [
    {
      "url": "https://example.org/resource",
      "media_type": "text/turtle",
      "encoding_format": "UTF-8",
      "size": 12345
    }
  ]
}  
```
The pydantic model can also be used to create web APIs with other Python-based tools like django-ninja or fastapi.

### Handle-suffix design

*This text is based on the proposal "PID-suffix, Version 2" made in NFDI4Cat-openproject on 2023-12-07*

Two handle-suffix schemes are supported:

- **UUID4** (_uuid pid4cat_)
  - Can be either supplied from the PID-applicant when minting a new PID or can be generated by the gateway (both ways should be possible).
  UUID7 should also be accepted.
- **namespace_of_agent/local_ID** (_name-spaced pid4cat_)
  - The handle-gateway should only validate the uniqueness of `namespace_of_agent/local_ID`-combinations.
   It is suggested to use 3 characters from a human-friendly base32-alphabet as namespaces (e.g. **z-base-32**, see above). This gives 3**32 prefixes and if ever necessary we can switch to 4 chars.
  - The uniqueness of the `local_ID` has to be taken care of by the namespace owner.
  - The `local_ID` can be freely chosen by the namespace owner but is limited to ASCII letters [a-zA-Z], numbers [0-9] and symbols `.-`. *OK or too restrictive?*
  - The request for becoming the owner of a new namespace should be made by email.

Examples for pid4cat handles (non-resolvable):

- **UUID4 pid4cat**: `https://hdl.handle.net/21.T11978/pid4cat/7e82d892-6acf-41a8-9c91-df826f67a806`
- **name-spaced pid4cat** in the example-namespace &quot;`k3a`&quot;:
  - `https://hdl.handle.net/21.T11978/pid4cat/k3a/123-456`
  - The corresponding regex in Python syntax is:   
    `re.compile(r"21\.T11978/pid4cat/(?P<namespace>k3a)/(?P<local_ID>\d{3}-\d{3})")`<BR>
    ...which extracts "k3a" as group "namespace" and "123-456" as group "local_ID".

  > Note, the `pid4cat`-part in the above URLs/suffixes has no functional meaning.
  > It is only present to make pid4cat-handles easily recognizable.
  > We should consider registering `pid4cat` as a prefix on bioregistry.io just like we registered the [voc4cat:](https://bioregistry.io/registry/voc4cat) prefix.

### API of handle server gateway

> This section is work in progress! It does not yet reflect the final implementation.

The role of the handle-server-gateway (HSG) is to restrict and manage write access to the handle-server and to add pid4cat-specific validation for the handles.
The HSG provides an API only.
Create and updating pid4cat-handles will exclusively managed via the HSG.
Suggested URLs for the HSG are https://pid4cat.nfdi4cat.org or https://pid.nfdi4cat.org

> Note: Dataverse will not use the HSG but interface the handle-server directly.

The HSG controls which users are allowed to create and update handle records.
Except for sysadmins, HSG users will only be allowed to create and update handles for named-spaced PIDs.
All PID name-spaces will have at least one name-space admin user.
Institutions that like to manage a pid4cat-PID-namespace need to register a PID-namespace admin with NFDI4Cat.

Suggested minimal API of the HSG:

*[unedited copy from internal OpenProject]*

Here is a tentative minimal API (to be discussed).
API access is limited to special users &quot;namespace-owners&quot; (read-write), and &quot;viewers&quot; (read-only).
Anonymous users have no access.

* _Regarding permissions and the minimal API we like to get your feedback. Is this OK? Do you see gaps or need more functionality?_

#### List all PIDs records of owner [GET] or create new PID [PUT]:

`https://pid.nfdi4cat.org/<api-version>/<name-space> [GET,PUT]`

- `https://pid.nfdi4cat.org/` is the base url of the gateway server.
- `api-version` (e.g. `v1.0.0`) is a version specifier for the API (it is recommended  to version APIs)
- `name-space` is the first part of the suffix of name-spaced PID.
- Special notes on create [PUT]:
  - HSG should set "record_version" automatically on create
  - *...and some more validation & consistency checks*

#### Show PID record [GET] or update PID record [PUT]

`https://pid.nfdi4cat.org/<api-version>/<name-space>/<local-ID-of-pid> [GET,PUT]`

where `<name-space>/<local-ID-of-pid>` form the suffix of the handle to get or update. For the examples above it is `k3a/123-456` and `7e82d892-6acf-41a8-9c91-df826f67a806`, respectively.

- Special notes on updates [PUT]:
  - The data are supplied as pid4cat-schema-conform json object in the body of the request.
  - each handle record update needs an update of the changelog (append)
  - HSG should set/increment "record_version" automatically on updates
  - *...and some more validation & consistency checks*

We could allow [DELETE] but we should not perform any true delete of a handle. Instead the record should be modified to state "OBSOLETE" (see pid4catStatus enum in schema).

#### Possible API Extensions

The API may be extended to make the information in the handle record more accessible, for example

- Routes to retrieve all PIDs by category [GET]
- Routes to retrieve all PIDs by status [GET]

## Permanent IRIs for terminology or ontology terms

For linked-data applications permanent IRIs are required.
Several services exists that provide a stable base URL and
a redirect service that can be configured to redirect
to the actual resource location.

Such services should support serving different data to humans and to machines.
One way to do this is via content-negotiation. Depending on the request that
is send to the redirect service, the client is redirected to human-oriented representation of the resource of to a machine-oriented one.

NFDI4Cat has selected [w3id.org](https://w3id.org/) for this service. Comparable services are [purl.org](https://purl.org/) or [pida.org](https://purls.helmholtz-metadaten.de/) which work in the same way.

In principle, it would also be possible to use handle-based PIDs for the terms in ontologies.
However, the additional features and the more reliable redundant network of the handle-system are of minor importance for this application.
