Adding sample and observation metadata to biom files — biom-format.org

# biom-format.org

## Adding sample and observation metadata to biom files

Frequently you’ll have an existing BIOM file and want to add sample and/or observation metadata to it. For samples, metadata is frequently environmental or technical details about your samples: the subject that a sample was collected from, the pH of the sample, the PCR primers used to amplify DNA from the samples, etc. For observations, metadata is frequently a categorization of the observation: the taxonomy of an OTU, or the EC hierarchy of a gene. You can use the `biom add-metadata` command to add this information to an existing BIOM file.

To get help with `add-metadata` you can call:

    biom add-metadata -h

This command takes a BIOM file, and corresponding sample and/or observation mapping files. The following examples are used in the commands below. You can find these files in the `biom-format/examples` directory.

Your BIOM file might look like the following:

    {
        "id":null,
        "format": "1.0.0",
        "format_url": "http://biom-format.org",
        "type": "OTU table",
        "generated_by": "some software package",
        "date": "2011-12-19T19:00:00",
        "rows":[
                {"id":"GG_OTU_1", "metadata":null},
                {"id":"GG_OTU_2", "metadata":null},
                {"id":"GG_OTU_3", "metadata":null},
                {"id":"GG_OTU_4", "metadata":null},
                {"id":"GG_OTU_5", "metadata":null}
            ],
        "columns": [
                {"id":"Sample1", "metadata":null},
                {"id":"Sample2", "metadata":null},
                {"id":"Sample3", "metadata":null},
                {"id":"Sample4", "metadata":null},
                {"id":"Sample5", "metadata":null},
                {"id":"Sample6", "metadata":null}
            ],
        "matrix_type": "sparse",
        "matrix_element_type": "int",
        "shape": [5, 6],
        "data":[[0,2,1],
                [1,0,5],
                [1,1,1],
                [1,3,2],
                [1,4,3],
                [1,5,1],
                [2,2,1],
                [2,3,4],
                [2,5,2],
                [3,0,2],
                [3,1,1],
                [3,2,1],
                [3,5,1],
                [4,1,1],
                [4,2,1]
               ]
    }

A sample metadata mapping file could then look like the following. Notice that there is an extra sample in here with respect to the above BIOM table. Any samples in the mapping file that are not in the BIOM file are ignored.

    #SampleID       BarcodeSequence DOB
    # Some optional
    # comment lines...
    Sample1 AGCACGAGCCTA    20060805
    Sample2 AACTCGTCGATG    20060216
    Sample3 ACAGACCACTCA    20060109
    Sample4 ACCAGCGACTAG    20070530
    Sample5 AGCAGCACTTGT    20070101
    Sample6 AGCAGCACAACT    20070716

An observation metadata mapping file might look like the following. Notice that there is an extra observation in here with respect to the above BIOM table. Any observations in the mapping file that are not in the BIOM file are ignored.

    #OTUID  taxonomy        confidence
    # Some optional
    # comment lines
    GG_OTU_0        Root;k__Bacteria;p__Firmicutes;c__Clostridia;o__Clostridiales;f__       0.980
    GG_OTU_1        Root;k__Bacteria;p__Firmicutes;c__Clostridia;o__Clostridiales;f__Lachnospiraceae        0.665
    GG_OTU_2        Root;k__Bacteria        0.980
    GG_OTU_3        Root;k__Bacteria;p__Firmicutes;c__Clostridia;o__Clostridiales;f__Lachnospiraceae        1.000
    GG_OTU_4        Root;k__Bacteria;p__Firmicutes;c__Clostridia;o__Clostridiales;f__Lachnospiraceae        0.842
    GG_OTU_5        Root;k__Bacteria;p__Firmicutes;c__Clostridia;o__Clostridiales;f__Lachnospiraceae        1.000

## Adding metadata

To add sample metadata to a BIOM file, you can run the following:

    biom add-metadata -i min_sparse_otu_table.biom -o table.w_smd.biom --sample-metadata-fp sam_md.txt

To add observation metadata to a BIOM file, you can run the following:

    biom add-metadata -i min_sparse_otu_table.biom -o table.w_omd.biom --observation-metadata-fp obs_md.txt

You can also combine these in a single command to add both observation and sample metadata:

    biom add-metadata -i min_sparse_otu_table.biom -o table.w_md.biom --observation-metadata-fp obs_md.txt --sample-metadata-fp sam_md.txt

In the last case, the resulting BIOM file will look like the following:

    {
        "columns": [
            {
                "id": "Sample1",
                "metadata": {
                    "BarcodeSequence": "AGCACGAGCCTA",
                    "DOB": "20060805"
                }
            },
            {
                "id": "Sample2",
                "metadata": {
                    "BarcodeSequence": "AACTCGTCGATG",
                    "DOB": "20060216"
                }
            },
            {
                "id": "Sample3",
                "metadata": {
                    "BarcodeSequence": "ACAGACCACTCA",
                    "DOB": "20060109"
                }
            },
            {
                "id": "Sample4",
                "metadata": {
                    "BarcodeSequence": "ACCAGCGACTAG",
                    "DOB": "20070530"
                }
            },
            {
                "id": "Sample5",
                "metadata": {
                    "BarcodeSequence": "AGCAGCACTTGT",
                    "DOB": "20070101"
                }
            },
            {
                "id": "Sample6",
                "metadata": {
                    "BarcodeSequence": "AGCAGCACAACT",
                    "DOB": "20070716"
                }
            }
        ],
        "data": [
            [0, 2, 1.0],
            [1, 0, 5.0],
            [1, 1, 1.0],
            [1, 3, 2.0],
            [1, 4, 3.0],
            [1, 5, 1.0],
            [2, 2, 1.0],
            [2, 3, 4.0],
            [2, 5, 2.0],
            [3, 0, 2.0],
            [3, 1, 1.0],
            [3, 2, 1.0],
            [3, 5, 1.0],
            [4, 1, 1.0],
            [4, 2, 1.0]
        ],
        "date": "2012-12-11T07:36:15.467843",
        "format": "Biological Observation Matrix 1.0.0",
        "format_url": "http://biom-format.org",
        "generated_by": "some software package",
        "id": null,
        "matrix_element_type": "float",
        "matrix_type": "sparse",
        "rows": [
            {
                "id": "GG_OTU_1",
                "metadata": {
                    "confidence": "0.665",
                    "taxonomy": "Root;k__Bacteria;p__Firmicutes;c__Clostridia;o__Clostridiales;f__Lachnospiraceae"
                }
            },
            {
                "id": "GG_OTU_2",
                "metadata": {
                    "confidence": "0.980",
                    "taxonomy": "Root;k__Bacteria"
                }
            },
            {
                "id": "GG_OTU_3",
                "metadata": {
                    "confidence": "1.000",
                    "taxonomy": "Root;k__Bacteria;p__Firmicutes;c__Clostridia;o__Clostridiales;f__Lachnospiraceae"
                }
            },
            {
                "id": "GG_OTU_4",
                "metadata": {
                    "confidence": "0.842",
                    "taxonomy": "Root;k__Bacteria;p__Firmicutes;c__Clostridia;o__Clostridiales;f__Lachnospiraceae"
                }
            },
            {
                "id": "GG_OTU_5",
                "metadata": {
                    "confidence": "1.000",
                    "taxonomy": "Root;k__Bacteria;p__Firmicutes;c__Clostridia;o__Clostridiales;f__Lachnospiraceae"
                }
            }
        ],
        "shape": [5, 6],
        "type": "OTU table"
    }

## Processing metadata while adding

There are some additional parameters you can pass to this command for more complex processing.

You can tell the command to process certain metadata column values as integers (`--int-fields`), floating point (i.e., decimal or real) numbers (`--float-fields`), or as hierarchical semicolon-delimited data (`--sc-separated`).

    biom add-metadata -i min_sparse_otu_table.biom -o table.w_md.biom --observation-metadata-fp obs_md.txt --sample-metadata-fp sam_md.txt --int-fields DOB --sc-separated taxonomy --float-fields confidence

Here your resulting BIOM file will look like the following, where `DOB` values are now integers (compare to the above: they’re not quoted now), `confidence` values are now floating point numbers (again, not quoted now), and `taxonomy` values are now lists where each entry is a taxonomy level, opposed to above where they appear as a single semi-colon-separated string.

    {
        "columns": [
            {
                "id": "Sample1",
                "metadata": {
                    "BarcodeSequence": "AGCACGAGCCTA",
                    "DOB": 20060805
                }
            },
            {
                "id": "Sample2",
                "metadata": {
                    "BarcodeSequence": "AACTCGTCGATG",
                    "DOB": 20060216
                }
            },
            {
                "id": "Sample3",
                "metadata": {
                    "BarcodeSequence": "ACAGACCACTCA",
                    "DOB": 20060109
                }
            },
            {
                "id": "Sample4",
                "metadata": {
                    "BarcodeSequence": "ACCAGCGACTAG",
                    "DOB": 20070530
                }
            },
            {
                "id": "Sample5",
                "metadata": {
                    "BarcodeSequence": "AGCAGCACTTGT",
                    "DOB": 20070101
                }
            },
            {
                "id": "Sample6",
                "metadata": {
                    "BarcodeSequence": "AGCAGCACAACT",
                    "DOB": 20070716
                }
            }
        ],
        "data": [
            [0, 2, 1.0],
            [1, 0, 5.0],
            [1, 1, 1.0],
            [1, 3, 2.0],
            [1, 4, 3.0],
            [1, 5, 1.0],
            [2, 2, 1.0],
            [2, 3, 4.0],
            [2, 5, 2.0],
            [3, 0, 2.0],
            [3, 1, 1.0],
            [3, 2, 1.0],
            [3, 5, 1.0],
            [4, 1, 1.0],
            [4, 2, 1.0]
        ],
        "date": "2012-12-11T07:30:29.870689",
        "format": "Biological Observation Matrix 1.0.0",
        "format_url": "http://biom-format.org",
        "generated_by": "some software package",
        "id": null,
        "matrix_element_type": "float",
        "matrix_type": "sparse",
        "rows": [
            {
                "id": "GG_OTU_1",
                "metadata": {
                    "confidence": 0.665,
                    "taxonomy": ["Root", "k__Bacteria", "p__Firmicutes", "c__Clostridia", "o__Clostridiales", "f__Lachnospiraceae"]
                }
            },
            {
                "id": "GG_OTU_2",
                "metadata": {
                    "confidence": 0.98,
                    "taxonomy": ["Root", "k__Bacteria"]
                }
            },
            {
                "id": "GG_OTU_3",
                "metadata": {
                    "confidence": 1.0,
                    "taxonomy": ["Root", "k__Bacteria", "p__Firmicutes", "c__Clostridia", "o__Clostridiales", "f__Lachnospiraceae"]
                }
            },
            {
                "id": "GG_OTU_4",
                "metadata": {
                    "confidence": 0.842,
                    "taxonomy": ["Root", "k__Bacteria", "p__Firmicutes", "c__Clostridia", "o__Clostridiales", "f__Lachnospiraceae"]
                }
            },
            {
                "id": "GG_OTU_5",
                "metadata": {
                    "confidence": 1.0,
                    "taxonomy": ["Root", "k__Bacteria", "p__Firmicutes", "c__Clostridia", "o__Clostridiales", "f__Lachnospiraceae"]
                }
            }
        ],
        "shape": [5, 6],
        "type": "OTU table"
    }

If you have multiple fields that you’d like processed in one of these ways, you can pass a comma-separated list of field names (e.g., `--float-fields confidence,pH`).

## Renaming (or naming) metadata columns while adding

You can also override the names of the metadata fields provided in the mapping files with the `--observation-header` and `--sample-header` parameters. This is useful if you want to rename metadata columns, or if metadata column headers aren’t present in your metadata mapping file. If you pass either of these parameters, you must name all columns in order. If there are more columns in the metadata mapping file then there are headers, extra columns will be ignored (so this is also a useful way to select only the first n columns from your mapping file). For example, if you want to rename the `DOB` column in the sample metadata mapping you could do the following:

    biom add-metadata -i min_sparse_otu_table.biom -o table.w_smd.biom --sample-metadata-fp sam_md.txt --sample-header SampleID,BarcodeSequence,DateOfBirth

If you have a mapping file without headers such as the following:

    GG_OTU_0        Root;k__Bacteria;p__Firmicutes;c__Clostridia;o__Clostridiales;f__       0.980
    GG_OTU_1        Root;k__Bacteria;p__Firmicutes;c__Clostridia;o__Clostridiales;f__Lachnospiraceae        0.665
    GG_OTU_2        Root;k__Bacteria        0.980
    GG_OTU_3        Root;k__Bacteria;p__Firmicutes;c__Clostridia;o__Clostridiales;f__Lachnospiraceae        1.000
    GG_OTU_4        Root;k__Bacteria;p__Firmicutes;c__Clostridia;o__Clostridiales;f__Lachnospiraceae        0.842
    GG_OTU_5        Root;k__Bacteria;p__Firmicutes;c__Clostridia;o__Clostridiales;f__Lachnospiraceae        1.000

you could name these while adding them as follows:

    biom add-metadata -i min_sparse_otu_table.biom -o table.w_omd.biom --observation-metadata-fp obs_md.txt --observation-header OTUID,taxonomy,confidence

As a variation on the last command, if you only want to include the `taxonomy` column and exclude the `confidence` column, you could run:

    biom add-metadata -i min_sparse_otu_table.biom -o table.w_omd.biom --observation-metadata-fp obs_md.txt --observation-header OTUID,taxonomy