lipd package — LiPD_Utilities documentation

# lipd package

## Module contents

`lipd.` `addEnsemble`(D, dsn, ensemble)

  Create ensemble entry and then add it to the specified LiPD dataset.

  Parameters: | D (dict) – LiPD data dsn (str) – Dataset name ensemble (list) – Nested numpy array of ensemble column data.
  Return dict D: | LiPD data

`lipd.` `collapseTs`(ts=None)

  Collapse a time series back into LiPD record form.

  Example
  1. D = lipd.readLipd()
  2. ts = lipd.extractTs(D)
  3. New_D = lipd.collapseTs(ts)

  Parameters: | ts (list) – Time series
  Return dict: | Metadata

`lipd.` `doi`()

  Update publication information using data DOIs. Updates LiPD files on disk, not in memory.

  Example
  1: lipd.readLipd()
  2: lipd.doi()

  Return none: |

`lipd.` `ensToDf`(ensemble)

  Create an ensemble data frame from some given nested numpy arrays

  Parameters: | ensemble (list) – Ensemble data
  Return obj df: | Pandas dataframe

`lipd.` `excel`()

  Convert Excel files to LiPD files. LiPD data is returned directly from this function.

  Example
  1: lipd.readExcel()
  2: D = lipd.excel()

  Return dict _d: | Metadata

`lipd.` `extractTs`(d, chron=False)

  Create a time series using LiPD data (uses paleoData by default)

  Example : paleoData
  1. D = lipd.readLipd()
  2. ts = lipd.extractTs(D)
  Example : chronData
  1. D = lipd.readLipd()
  2. ts = lipd.extractTs(D, chron=True)

  Parameters: | d (dict) – Metadata chron (bool) – Create a chronData time series
  Return list l: | Time series

`lipd.` `filterTs`(ts, expression)

  Create a new time series that only contains entries that match the given expression.

  Example:
  D = lipd.loadLipd()
  ts = lipd.extractTs(D)
  new_ts = filterTs(ts, “archiveType == marine sediment”)
  new_ts = filterTs(ts, “paleoData_variableName == sst”)

  Parameters: | expression (str) – Expression ts (list) – Time series
  Return list new_ts:
  | Filtered time series that matches the expression

`lipd.` `getCsv`(L=None)

  Get CSV from LiPD metadata

  Example
  c = lipd.getCsv(D[“Africa-ColdAirCave.Sundqvist.2013”])

  Parameters: | L (dict) – One LiPD record
  Return dict d: | CSV data

`lipd.` `getLipdNames`(D=None)

  Get a list of all LiPD names in the library

  Example
  names = lipd.getLipdNames(D)

  Return list f_list:
  | File list

`lipd.` `getMetadata`(L)

  Get metadata from a LiPD data in memory

  Example
  m = lipd.getMetadata(D[“Africa-ColdAirCave.Sundqvist.2013”])

  Parameters: | L (dict) – One LiPD record
  Return dict d: | LiPD record (metadata only)

`lipd.` `noaa`(d=None)

  Convert between NOAA and LiPD files

  Example: LiPD to NOAA converter
  1: D = lipd.readLipd()
  2: lipd.noaa(D)
  Example: NOAA to LiPD converter
  1: readNoaa()
  2: lipd.noaa()

  Return none: |

`lipd.` `queryTs`(ts, expression)

  Find the indices of the time series entries that match the given expression.

  Example:
  D = lipd.loadLipd()
  ts = lipd.extractTs(D)
  matches = queryTs(ts, “archiveType == marine sediment”)
  matches = queryTs(ts, “geo_meanElev <= 2000”)

  Parameters: | expression (str) – Expression ts (list) – Time series
  Return list _idx:
  | Indices of entries that match the criteria

`lipd.` `readAll`(usr_path='')

  Read all approved file types at once. Enter a file path, directory path, or leave args blank to trigger gui.

  Parameters: | usr_path (str) – Path to file / directory (optional)
  Return str cwd: | Current working directory

`lipd.` `readExcel`(usr_path='')

  Read Excel file(s) Enter a file path, directory path, or leave args blank to trigger gui.

  Parameters: | usr_path (str) – Path to file / directory (optional)
  Return str cwd: | Current working directory

`lipd.` `readLipd`(usr_path='')

  Read LiPD file(s). Enter a file path, directory path, or leave args blank to trigger gui.

  Parameters: | usr_path (str) – Path to file / directory (optional)
  Return dict _d: | Metadata

`lipd.` `readNoaa`(usr_path='')

  Read NOAA file(s) Enter a file path, directory path, or leave args blank to trigger gui.

  Parameters: | usr_path (str) – Path to file / directory (optional)
  Return str cwd: | Current working directory

`lipd.` `run`()

  Initialize and start objects. This is called automatically when importing the package.

  Return none: |

`lipd.` `showDfs`(d)

  Display the available data frame names in a given data frame collection

  Parameters: | d (dict) – Dataframe collection
  Return none: |

`lipd.` `showLipds`(D=None)

  Display the dataset names of a given LiPD data

  Example
  lipd.showLipds(D)

  Pararm dict D: | LiPD data
  Return none: |

`lipd.` `showMetadata`(dat)

  Display the metadata specified LiPD in pretty print

  Example
  showMetadata(D[“Africa-ColdAirCave.Sundqvist.2013”])

  Parameters: | dat (dict) – Metadata
  Return none: |

`lipd.` `tsToDf`(tso)

  Create Pandas DataFrame from TimeSeries object. Use: Must first extractTs to get a time series. Then pick one item from time series and pass it through

  Parameters: | tso (dict) – Time series entry
  Return dict dfs:
  | Pandas dataframes

`lipd.` `viewTs`(ts)

  View the contents of one time series entry in a nicely formatted way

  Example
  1. D = lipd.readLipd()
  2. ts = lipd.extractTs(D)
  3. viewTs(ts[0])

  Parameters: | ts (dict) – One time series entry
  Return none: |

`lipd.` `writeLipd`(dat, usr_path='', filename='')

  Write LiPD data to file(s)

  Parameters: | dat (dict) – Metadata usr_path (str) – Destination (optional) filename (str) – LiPD filename, for writing one specific file (optional)
  Return none: |

## Submodules

## alternates

List of alternate and synonym keys

## bag

`lipd.bag.` `create_bag`(dir_bag)

  Create a Bag out of given files. :param str dir_bag: Directory that contains csv, jsonld, and changelog files. :return obj: Bag

`lipd.bag.` `finish_bag`(dir_bag)

  Closing steps for creating a bag :param obj dir_bag: :return None:

`lipd.bag.` `open_bag`(dir_bag)

  Open Bag at the given path :param str dir_bag: Path to Bag :return obj: Bag

`lipd.bag.` `resolved_flag`(bag)

  Check DOI flag in bag.info to see if doi_resolver has been previously run :param obj bag: Bag :return bool: Flag

`lipd.bag.` `validate_md5`(bag)

  Check if Bag is valid :param obj bag: Bag :return None:

## blanks

List of empty and ignored keys

## csvs

`lipd.csvs.` `get_csv_from_metadata`(name, metadata)

  Two goals. Get all csv from metadata, and return new metadata with generated filenames to match files. :param str name: LiPD dataset name :param dict metadata: Metadata :return dict: Csv Data

`lipd.csvs.` `merge_csv_metadata`(d)

  Using the given metadata dictionary, retrieve CSV data from CSV files, and insert the CSV values into their respective metadata columns. Checks for both paleoData and chronData tables. :param dict d: Metadata :return dict: Modified metadata dictionary

`lipd.csvs.` `read_csv_from_file`(filename)

  Opens the target CSV file and creates a dictionary with one list for each CSV column. :param str filename: :return list of lists: column values

`lipd.csvs.` `write_csv_to_file`(d)

  Writes columns of data to a target CSV file. :param dict d: A dictionary containing one list for every data column. Keys: int, Values: list :return None:

## dataframes

`lipd.dataframes.` `create_dataframe`(ensemble)

  Create a data frame from given nested lists of ensemble data :param list ensemble: Ensemble data :return obj: Dataframe

`lipd.dataframes.` `get_filtered_dfs`(lib, expr)

  Main: Get all data frames that match the given expression :return dict: Filenames and data frames (filtered)

`lipd.dataframes.` `lipd_to_df`(metadata, csvs)

  Create an organized collection of data frames from LiPD data :param dict metadata: LiPD data :param dict csvs: Csv data :return dict: One data frame per table, organized in a dictionary by name

`lipd.dataframes.` `ts_to_df`(metadata)

  Create a data frame from one TimeSeries object :param dict metadata: Time Series dictionary :return dict: One data frame per table, organized in a dictionary by name

## directory

`lipd.directory.` `browse_dialog_dir`()

  Open up a GUI browse dialog window and let to user pick a target directory. :return str: Target directory path

`lipd.directory.` `browse_dialog_file`()

  Open up a GUI browse dialog window and let to user select one or more files :return str _path: Target directory path :return list _files: List of selected files

`lipd.directory.` `check_file_age`(filename, days)

  Check if the target file has an older creation date than X amount of time. i.e. One day: 60*60*24 :param str filename: Target filename :param int days: Limit in number of days :return bool: True - older than X time, False - not older than X time

`lipd.directory.` `collect_metadata_file`(full_path)

  Create the file metadata and add it to the appropriate section by file-type :param str full_path: :param dict existing_files: :return dict existing files:

`lipd.directory.` `collect_metadata_files`(cwd, new_files, existing_files)

  Collect all files from a given path. Separate by file type, and return one list for each type If ‘files’ contains specific :param str cwd: Directory w/ target files :param list new_files: Specific new files to load :param dict existing_files: Files currently loaded, separated by type :return list: All files separated by type

`lipd.directory.` `create_tmp_dir`()

  Creates tmp directory in OS temp space. :return str: Path to tmp directory

`lipd.directory.` `dir_cleanup`(dir_bag, dir_data)

  Moves JSON and csv files to bag root, then deletes all the metadata bag files. We’ll be creating a new bag with the data files, so we don’t need the other text files and such. :param str dir_bag: Path to root of Bag :param str dir_data: Path to Bag /data subdirectory :return None:

`lipd.directory.` `filename_from_path`(path)

  Extract the file name from a given file path. :param str path: File path :return str: File name with extension

`lipd.directory.` `find_files`()

  Search for the directory containing jsonld and csv files. chdir and then quit. :return none:

`lipd.directory.` `get_filenames_generated`(d, name='', csvs='')

  Get the filenames that the LiPD utilities has generated (per naming standard), as opposed to the filenames that originated in the LiPD file (that possibly don’t follow the naming standard) :param dict d: Data :param str name: LiPD dataset name to prefix :param list csvs: Filenames list to merge with :return list: Filenames

`lipd.directory.` `get_filenames_in_lipd`(path, name='')

  List all the files contained in the LiPD archive. Bagit, JSON, and CSV :param str path: Directory to be listed :param str name: LiPD dataset name, if you want to prefix it to show file hierarchy :return list: Filenames found

`lipd.directory.` `get_src_or_dst`(mode, path_type)

  User sets the path to a LiPD source location :param str mode: “read” or “write” mode :param str path_type: “directory” or “file” :return str path: dir path to files :return list files: files chosen

`lipd.directory.` `get_src_or_dst_path`(prompt, count)

  Let the user choose a path, and store the value. :return str _path: Target directory :return str count: Counter for attempted prompts

`lipd.directory.` `get_src_or_dst_prompt`(mode)

  String together the proper prompt based on the mode :param str mode: “read” or “write” :return str prompt: The prompt needed

`lipd.directory.` `list_files`(x, path='')

  Lists file(s) in given path of the X type. :param str x: File extension that we are interested in. :param str path: Path, if user would like to check a specific directory outside of the CWD :return list of str: File name(s) to be worked on

`lipd.directory.` `rm_file_if_exists`(path, filename)

  Remove a file if it exists. Useful for when we want to write a file, but it already exists in that locaiton. :param str filename: Filename :param str path: Directory :return none:

`lipd.directory.` `rm_files_in_dir`(path)

  Removes all files within a directory, but does not delete the directory :param str path: Target directory :return none:

## doi_main

`lipd.doi_main.` `doi_main`(files)

  Main function that controls the script. Take in directory containing the .lpd file(s). Loop for each file. :return None:

`lipd.doi_main.` `process_lpd`(name, dir_tmp)

  Opens up json file, invokes doi_resolver, closes file, updates changelog, cleans directory, and makes new bag. :param str name: Name of current .lpd file :param str dir_tmp: Path to tmp directory :return none:

`lipd.doi_main.` `prompt_force`()

  Ask the user if they want to force update files that were previously resolved :return bool: response

## doi_resolver

class `lipd.doi_resolver.` `DOIResolver`(dir_root, name, root_dict)

  Bases: `object`

  Use DOI id(s) to pull updated publication info from doi.org and overwrite file data.

  Input: Original publication dictionary Output: Updated publication dictionary (success), original publication dictionary (fail)

  static `compare_replace`(pub_dict, fetch_dict)

    Take in our Original Pub, and Fetched Pub. For each Fetched entry that has data, overwrite the Original entry :param pub_dict: (dict) Original pub dictionary :param fetch_dict: (dict) Fetched pub dictionary from doi.org :return: (dict) Updated pub dictionary, with fetched data taking precedence

  static `compile_authors`(authors)

    Compiles authors “Last, First” into a single list :param list authors: Raw author data retrieved from doi.org :return list: Author objects

  static `compile_date`(date_parts)

    Compiles date only using the year :param list date_parts: List of date parts retrieved from doi.org :return str: Date string or NaN

  `compile_fetch`(raw, doi_id)

    Loop over Raw and add selected items to Fetch with proper formatting :param dict raw: JSON data from doi.org :param str doi_id: :return dict:

  `find_doi`(curr_dict)

    Recursively search the file for the DOI id. More taxing, but more flexible when dictionary structuring isn’t absolute :param dict curr_dict: Current dictionary being searched :return dict bool: Recursive - Current dictionary, False flag that DOI was not found :return str bool: Final - DOI id, True flag that DOI was found

  `get_data`(doi_id, idx)

    Resolve DOI and compile all attributes into one dictionary :param str doi_id: :param int idx: Publication index :return dict: Updated publication dictionary

  `illegal_doi`(doi_string)

    DOI string did not match the regex. Determine what the data is. :param doi_string: (str) Malformed DOI string :return: None

  `main`()

    Main function that gets file(s), creates outputs, and runs all operations. :return dict: Updated or original data for jsonld file

  `noaa_citation`(doi_string)

    Special instructions for moving noaa data to the correct fields :param doi_string: (str) NOAA url :return: None

  `remove_empties`(pub)

## ensembles

`lipd.ensembles.` `create_ensemble`(ensemble)

  Add ensemble data to a LiPD object :param list ensemble: Ensemble data nested lists :return dict: Structured Ensemble data

`lipd.ensembles.` `insert_ensemble`(d, ens)

  Insert the ensemble table dictionary into the LiPD metadata :param dict d: LiPD metadata :param dict ens: Ensemble data to insert :return dict:

## excel

`lipd.excel.` `cells_dn_meta`(workbook, sheet, row, col, final_dict)

  Traverse all cells in a column moving downward. Primarily created for the metadata sheet, but may use elsewhere. Check the cell title, and switch it to. :param obj workbook: :param str sheet: :param int row: :param int col: :param dict final_dict: :return: none

`lipd.excel.` `cells_rt_meta`(workbook, sheet, row, col)

  Traverse all cells in a row. If you find new data in a cell, add it to the list. :param obj workbook: :param str sheet: :param int row: :param int col: :return list: Cell data for a specific row

`lipd.excel.` `cells_rt_meta_pub`(workbook, sheet, row, col, pub_qty)

  Publication section is special. It’s possible there’s more than one publication. :param obj workbook: :param str sheet: :param int row: :param int col: :param int pub_qty: Number of distinct publication sections in this file :return list: Cell data for a specific row

`lipd.excel.` `compile_authors`(cell)

  Split the string of author names into the BibJSON format. :param str cell: Data from author cell :return: (list of dicts) Author names

`lipd.excel.` `compile_fund`(workbook, sheet, row, col)

  Compile funding entries. Iter both rows at the same time. Keep adding entries until both cells are empty. :param obj workbook: :param str sheet: :param int row: :param int col: :return list of dict: l

`lipd.excel.` `compile_geo`(d)

  Compile top-level Geography dictionary. :param d: :return:

`lipd.excel.` `compile_geometry`(lat, lon, elev)

  Take in lists of lat and lon coordinates, and determine what geometry to create :param list lat: Latitude values :param list lon: Longitude values :param float elev: Elevation value :return dict:

`lipd.excel.` `compile_temp`(d, key, value)

  Compiles temporary dictionaries for metadata. Adds a new entry to an existing dictionary. :param dict d: :param str key: :param any value: :return dict:

`lipd.excel.` `count_chron_variables`(temp_sheet)

  Count the number of chron variables :param obj temp_sheet: :return int: variable count

`lipd.excel.` `excel_main`(file)

  Parse data from Excel spreadsheets into LiPD files. :return list: Filenames of LiPD files created

`lipd.excel.` `extract_short`(string_in)

  Extract the short name from a string that also has units. :param str string_in: :return str:

`lipd.excel.` `extract_units`(string_in)

  Extract units from parenthesis in a string. i.e. “elevation (meters)” :param str string_in: :return str:

`lipd.excel.` `geometry_linestring`(lat, lon, elev)

  GeoJSON Linestring. Latitude and Longitude have 2 values each. :param list lat: Latitude values :param list lon: Longitude values :return dict:

`lipd.excel.` `geometry_point`(lat, lon, elev)

  GeoJSON point. Latitude and Longitude only have one value each :param list lat: Latitude values :param list lon: Longitude values :param float elev: Elevation value :return dict:

`lipd.excel.` `geometry_range`(crd_range, elev, crd_type)

  Range of coordinates. (e.g. 2 latitude coordinates, and 0 longitude coordinates) :param crd_range: Latitude or Longitude values :param elev: Elevation value :param crd_type: Coordinate type, lat or lon :return dict:

`lipd.excel.` `get_chron_data`(temp_sheet, row, total_vars)

  Capture all data in for a specific chron data row (for csv output) :param obj temp_sheet: :param int row: :param int total_vars: :return list: data_row

`lipd.excel.` `get_chron_var`(temp_sheet, start_row)

  Capture all the vars in the chron sheet (for json-ld output) :param obj temp_sheet: :param int start_row: :return: (list of dict) column data

`lipd.excel.` `instance_str`(cell)

  Match data type and return string :param any cell: :return str:

`lipd.excel.` `logger_excel` = <logging.Logger object>

  VERSION: LiPD v1.2

`lipd.excel.` `name_to_jsonld`(title_in)

  Convert formal titles to camelcase json_ld text that matches our context file Keep a growing list of all titles that are being used in the json_ld context :param str title_in: :return str:

`lipd.excel.` `traverse_to_chron_data`(temp_sheet)

  Traverse down to the first row that has chron data :param obj temp_sheet: :return int: traverse_row

`lipd.excel.` `traverse_to_chron_var`(temp_sheet)

  Traverse down to the row that has the first variable :param obj temp_sheet: :return int:

## inferred_data

`lipd.inferred_data.` `get_inferred_data_table`(pc, table)

  Table level: Dive down, calculate data, then return the new table with the inferred data. :param str pc: Paleo or Chron table type :param dict table: Table data :return dict table: Table with new data

## io

## jsons

`lipd.jsons.` `get_csv_from_json`(d)

  Get CSV values when mixed into json data. Pull out the CSV data and put it into a dictionary. :param dict d: JSON with CSV values :return dict: CSV values. (i.e. { CSVFilename1: { Column1: [Values], Column2: [Values] }, CSVFilename2: … }

`lipd.jsons.` `idx_name_to_num`(d)

  Switch from index-by-name to index-by-number. :param dict d: Metadata :return dict: Modified metadata

`lipd.jsons.` `idx_num_to_name`(d)

  Switch from index-by-number to index-by-name. :param dict d: Metadata :return dict: Modified Metadata

`lipd.jsons.` `read_json_from_file`(filename)

  Import the JSON data from target file. :param str filename: Target File :return dict: JSON data

`lipd.jsons.` `read_jsonld`()

  Find jsonld file in the cwd (or within a 2 levels below cwd), and load it in. :return dict: Jsonld data

`lipd.jsons.` `remove_csv_from_json`(d)

  Remove all CSV data ‘values’ entries from paleoData table in the JSON structure. :param dict d: JSON data - old structure :return dict: Metadata dictionary without CSV values

`lipd.jsons.` `write_json_to_file`(json_data, filename='metadata')

  Write all JSON in python dictionary to a new json file. :param dict json_data: JSON data :param str filename: Target filename (defaults to ‘metadata.jsonld’) :return None:

## loggers

`lipd.loggers.` `create_benchmark`(name, log_file, level=20)

  Creates a logger for function benchmark times :param str name: Name of the logger :param str log_file: Filename :return obj: Logger

`lipd.loggers.` `create_logger`(name)

  Creates a logger with the below attributes. :param str name: Name of the logger :return obj: Logger

`lipd.loggers.` `log_benchmark`(fn, start, end)

  Log a given function and how long the function takes in seconds :param str fn: Function name :param float start: Function start time :param float end: Function end time :return none:

`lipd.loggers.` `update_changelog`()

  Create or update the changelog txt file. Prompt for update description. :return None:

## lpd_noaa

class `lipd.lpd_noaa.` `LPD_NOAA`(dir_root, name, lipd_dict)

  Bases: `object`

  Creates a NOAA object that contains all the functions needed to write out a LiPD file as a NOAA text file. Supports LiPD Version: v1.2 NOAA txt template: v3.0

  Return none: | Writes NOAA text to file in local storage

  `get_master`()

    Get the master json that has been modified :return dict: self.lipd_data

  `get_wdc_paleo_url`()

    When a NOAA file is created, it creates a URL link to where the dataset will be hosted in NOAA’s archive Retrieve and add this link to the original LiPD file, so we can trace the dataset to NOAA. :return str:

  `main`()

    Load in the template file, and run through the parser :return none:

## misc

`lipd.misc.` `cast_float`(x)

  Attempt to cleanup string or convert to number value. :param any x: :return float:

`lipd.misc.` `cast_int`(x)

  Cast unknown type into integer :param any x: :return int:

`lipd.misc.` `cast_values_csvs`(d, idx, x)

  Attempt to cast string to float. If error, keep as a string. :param dict d: Data :param int idx: Index number :param str x: Data :return any:

`lipd.misc.` `check_dsn`(name, _json)

  Get a dataSetName. If one is not provided, then insert the filename as the dataSetName. :param str name: Filename w/o extension :param dict _json: Metadata :return dict _json: Metadata

`lipd.misc.` `clean_doi`(doi_string)

  Use regex to extract all DOI ids from string (i.e. 10.1029/2005pa001215) :param str doi_string: Raw DOI string value from input file. Often not properly formatted. :return list: DOI ids. May contain 0, 1, or multiple ids.

`lipd.misc.` `fix_coordinate_decimal`(d)

  Coordinate decimal degrees calculated by an excel formula are often too long as a repeating decimal. Round them down to 5 decimals :param dict d: Metadata :return dict d: Metadata

`lipd.misc.` `generate_timestamp`(fmt=None)

  Generate a timestamp to mark when this file was last modified. :param str fmt: Special format instructions :return str: YYYY-MM-DD format, or specified format

`lipd.misc.` `generate_tsid`(size=8)

  Generate a TSid string. Use the “PYT” prefix for traceability, and 8 trailing generated characters ex: PYT9AG234GS :return:

`lipd.misc.` `get_appended_name`(name, columns)

  Append numbers to a name until it no longer conflicts with the other names in a column. Necessary to avoid overwriting columns and losing data. Loop a preset amount of times to avoid an infinite loop. There shouldn’t ever be more than two or three identical variable names in a table. :param str name: Variable name in question :param dict columns: Columns listed by variable name :return str: Appended variable name

`lipd.misc.` `get_authors_as_str`(x)

  Take author or investigator data, and convert it to a concatenated string of names. Author data structure has a few variations, so account for all. :param any x: Author data :return str: Author string

`lipd.misc.` `get_dsn`(d)

  Get the dataset name from a record :param dict d: Metadata :return str: Dataset name

`lipd.misc.` `get_ensemble_counts`(d)

  Determine if this is a 1 or 2 column ensemble. Then determine how many columns and rows it has. :param d: :return:

`lipd.misc.` `get_missing_value_key`(d)

  Get the Missing Value entry from a table of data. If none is found, try the columns. If still none found, prompt user. :param dict d: Table of data :return str: Missing Value

`lipd.misc.` `get_table_key`(key, d, fallback='')

  Try to get a table name from a data table :param str key: Key to try first :param dict d: Data table :param str fallback: (optional) If we don’t find a table name, use this as a generic name fallback. :return str: Data table name

`lipd.misc.` `get_variable_name_col`(d)

  Get the variable name from a table or column :param dict d: Metadata :return str:

`lipd.misc.` `is_ensemble`(d)

  Check if a table of data is an ensemble table. Is the first values index a list? ensemble. Int/float? not ensemble. :param dict d: Table data :return bool: Ensemble or not ensemble

`lipd.misc.` `load_fn_matches_ext`(file_path, file_type)

  Check that the file extension matches the target extension given. :param str file_path: Path to be checked :param str file_type: Target extension :return bool:

`lipd.misc.` `match_arr_lengths`(l)

  Check that all the array lengths match so that a DataFrame can be created successfully. :param list l: Nested arrays :return bool: Valid or invalid

`lipd.misc.` `match_operators`(inp, relate, cut)

  Compare two items. Match a string operator to an operator function :param str inp: Comparison item :param str relate: Comparison operator :param any cut: Comparison item :return bool: Comparison truth

`lipd.misc.` `mv_files`(src, dst)

  Move all files from one directory to another :param str src: Source directory :param str dst: Destination directory :return none:

`lipd.misc.` `normalize_name`(s)

  Remove foreign accents and characters to normalize the string. Prevents encoding errors. :param str s: :return str:

`lipd.misc.` `path_type`(path, target)

  Determine if given path is file, directory, or other. Compare with target to see if it’s the type we wanted. :param str path: Path :param str target: Target type wanted :return bool:

`lipd.misc.` `prompt_protocol`()

  Prompt user if they would like to save pickle file as a dictionary or an object. :return str: Answer

`lipd.misc.` `put_tsids`(x)

  Recursively add in TSids into any columns that do not have them. Look for “columns” keys, and then start looping and adding generated TSids to each column :param any x: Recursive, so could be any data type. :return any x: Recursive, so could be any data type.

`lipd.misc.` `rm_empty_doi`(d)

  If an “identifier” dictionary has no doi ID, then it has no use. Delete it. :param dict d: JSON Metadata :return dict: JSON Metadata

`lipd.misc.` `rm_empty_fields`(x)

  Go through N number of nested data types and remove all empty entries. Recursion :param any x: Dictionary, List, or String of data :return any: Returns a same data type as original, but without empties.

`lipd.misc.` `rm_files`(path, extension)

  Remove all files in the given directory with the given extension :param str path: Directory :param str extension: File type to remove :return none:

`lipd.misc.` `rm_keys_from_dict`(d, keys)

  Given a dictionary and a key list, remove any data in the dictionary with the given keys. :param dict d: Data :param list keys: List of key data to remove :return dict d: Data (with keys + data removed)

`lipd.misc.` `rm_missing_values_table`(d)

  Loop for each table column and remove the missingValue key & data :param dict d: Table data :return dict d: Table data

`lipd.misc.` `rm_values_fields`(x)

  (recursive) Remove all “values” fields from the metadata :param any x: Any data type :return dict: metadata without “values”

`lipd.misc.` `split_path_and_file`(s)

  Given a full path to a file, split and return a path and filename :param str s: Full path :return str str: Path, filename

`lipd.misc.` `unwrap_arrays`(l)

  Unwrap nested lists to be one “flat” list of lists. Mainly for prepping ensemble data for DataFrame() creation :param list l: Nested lists :return list: Flattened lists

## noaa

`lipd.noaa.` `lpd_to_noaa`(obj)

  Convert a LiPD format to NOAA format :param obj obj: LiPD object :return obj: LiPD object (modified)

`lipd.noaa.` `noaa_prompt`()

  Convert between NOAA and LiPD file formats. :return:

`lipd.noaa.` `noaa_to_lpd`(files)

  Convert NOAA format to LiPD format :param dict files: Files metadata :return None:

## noaa_lpd

class `lipd.noaa_lpd.` `NOAA_LPD`(dir_root, dir_tmp, name)

  Bases: `object`

  `main`()

    Convert a NOAA text file into a lipds file. CSV files will be created if chronology or data sections are available. :return dict: Metadata Dictionary

## regexes

## timeseries

`lipd.timeseries.` `collapse`(l)

  LiPD Version 1.3 Main function to initiate time series to LiPD conversion :param list l: Time series :return dict _master: LiPD data, sorted by dataset name

`lipd.timeseries.` `extract`(d, chron)

  LiPD Version 1.3 Main function to initiate LiPD to TSOs conversion. :param dict d: Metadata for one LiPD file :param bool chron: Paleo mode (default) or Chron mode :return list _ts: Time series

`lipd.timeseries.` `get_matches`(expr_lst, ts)

  Get a list of TimeSeries objects that match the given expression. :param list expr_lst: Expression :param list ts: TimeSeries :return list new_ts: Matched time series objects :return list idxs: Indices of matched objects

`lipd.timeseries.` `mode_ts`(ec, ts=None, b=None)

  Get string for the mode :param bool b: Chron boolean (for extract) :param str ec: extract or collapse :param list ts: Time series (for collapse) :return str phrase: Phrase

`lipd.timeseries.` `translate_expression`(expression)

  Check if the expression is valid, then check turn it into an expression that can be used for filtering. :return list of lists: One or more matches. Each list has 3 strings.

## validator_api

`lipd.validator_api.` `create_detailed_results`(data)

`lipd.validator_api.` `display_results`(data, detailed=False)

  Display the results from the validator in a brief or detailed way. :param dict data: Results, sorted by dataset name :param bool detailed: Detailed results on or off :return none:

`lipd.validator_api.` `get_validator_format`(data_json, data_csv, filenames)

  Format the LIPD data in the layout that the Lipd.net validator accepts. Example of one _file metadata. _file_list will contain 1 or more _file’s _file = {

  > “type”: “bagit/json/csv”, “filenameFull”: /path/to/filename.txt, “filenameShort”: filename.txt, “data”: “”, “pretty”: “”

  }

  Parameters: | data_json (dict) – Metadata data_csv (dict) – CSV data filenames (list) – All files found in LiPD archive
  Return list: | Validator-formatted data

`lipd.validator_api.` `get_validator_results`(data)

  Send LiPD data to the Lipd.net validator and get the results back. :param data: :return:

## versions

`lipd.versions.` `get_lipd_version`(d)

  Check what version of LiPD this file is using. If none is found, assume it’s using version 1.0 :param dict d: Metadata :return float:

`lipd.versions.` `update_lipd_v1_1`(d)

  Update LiPD v1.0 to v1.1 - chronData entry is a list that allows multiple tables - paleoData entry is a list that allows multiple tables - chronData now allows measurement, model, summary, modelTable, ensemble, calibratedAges tables - Added ‘lipdVersion’ key

  Parameters: | d (dict) – Metadata v1.0
  Return dict d: | Metadata v1.1

`lipd.versions.` `update_lipd_v1_2`(d)

  Update LiPD v1.1 to v1.2 - Added NOAA compatible keys : maxYear, minYear, originalDataURL, WDCPaleoURL, etc - ‘calibratedAges’ key is now ‘distribution’ - paleoData structure mirrors chronData. Allows measurement, model, summary, modelTable, ensemble,

  > distribution tables

  Parameters: | d (dict) – Metadata v1.1
  Return dict d: | Metadata v1.2

`lipd.versions.` `update_lipd_v1_3`(d)

  Update LiPD v1.2 to v1.3 - Added ‘createdBy’ key - Top-level folder inside LiPD archives are named “bag”. (No longer <datasetname>) - .jsonld file is now generically named ‘metadata.jsonld’ (No longer <datasetname>.lpd ) - All “paleo” and “chron” prefixes are removed from “paleoMeasurementTable”, “paleoModel”, etc. - Merge isotopeInterpretation and climateInterpretation into “interpretation” block - ensemble table entry is a list that allows multiple tables - summary table entry is a list that allows multiple tables :param dict d: Metadata v1.2 :return dict d: Metadata v1.3

`lipd.versions.` `update_lipd_v1_3_names`(d)

  Update the key names and merge interpretation data :param dict d: Metadata :return dict d: Metadata

`lipd.versions.` `update_lipd_v1_3_structure`(d)

  Update the structure for summary and ensemble tables :param dict d: Metadata :return dict d: Metadata

`lipd.versions.` `update_lipd_version`(d)

  Metadata is indexed by number at this step.

  Use the current version number to determine where to start updating from. Use “chain versioning” to make it modular. If a file is a few versions behind, convert to EACH version until reaching current. If a file is one version behind, it will only convert once to the newest. :param dict d: Metadata :return dict d: Metadata

## zips

`lipd.zips.` `unzipper`(filename, dir_tmp)

  Unzip .lpd file contents to tmp directory. :param str filename: filename.lpd :param str dir_tmp: Tmp folder to extract contents to :return None:

`lipd.zips.` `zipper`(root_dir='', name='', path_name_ext='')

  Zips up directory back to the original location :param str root_dir: Root directory of the archive :param str name: <datasetname>.lpd :param str path_name_ext: /path/to/filename.lpd