# Data

## Files

- `raw.xlsx`: Raw dataset
- `processed.csv`: Filtering out only necessary info for training
- `{train,test}_idx.csv`: Lists entry ids part of the train/test splits

## Directories
- `./rand_cifs`: Cifs with added random noise
- `./rand_np_cifs`: Same as ./rand_cifs with rounded partial occupancies

### Private directories (removed in public repo)

- `./cifs`: Original cif files
- `./anon_cifs`: Standardized cif files (pymatgen format) --> replace with ./rand_cifs in scripts
- `./np_cifs`: Standardized cif files with rounded partial occupancies --> replace with ./rand_np_cifs in scripts

## Scripts
All scripts are in the `scripts` directory. They are meant to be run from the `data` directory.
- `preprocessing.py`: Adds paper information, process data to be compatible with DAVE
- `compare_xlsx.py`: Script to compare two excel files since the raw dataset is an excel file. Usage: `python compare_xlsx.py OLD_FILE NEW_FILE`
- `get_mp_entries.py`: script to automatically download similar cifs files from the MP
- `get_icsd_entries.py`: script to automatically download similar cifs files from the ICSD
- `analysis.py`: creates figures and stats for the paper
- `anonymize_cifs.py`: Standardize original cifs
- `randomize_cifs.py`: Add random noise to atomic positions while preserving symmetry
- `train_test_split.py`: Script used to create test and train splits. The arguments used to create the official splits are in `misc/arguments_split.yaml`

## Miscellaneous

- `unidentified_with_refs.csv`: List of 56 entries with paper info that are not in either database, sent by Homin (for automatic DOIs only)
- `checked_by_hand.csv`: A list of 94 entries for which the doi was confirmed manually (for automatic DOIs only)
- `LiIonDatabase.csv`: [The Liverpool Ionics Dataset](http://pcwww.liv.ac.uk/~msd30/lmds/LiIonDatabase.html)
- `laskowski_semi-fromatted.csv`: [Forrest A. L. Laskowski et al., Energy Environ. Sci., 16, 1264 (2023)](https://pubs.rsc.org/en/content/articlelanding/2023/ee/d2ee03499a#!). Data extracted from the pdf SI. Some of the pdf was not perfectly rendered in the csv
- `doi_lookup_table.csv`: lookup table between reference number in the SI and the doi. Extracted using `extract_doi.py`