---
title: Samples
jupyter: python3
---



As Starsim models are usually stochastic, for a single scenario it is often desirable to run the model multiple times with different random seeds. The role of the `Samples` class is to facilitate working with large numbers of simulations and scenarios, to ease:

- Loading large result sets
- Filtering/selecting simulation runs
- Plotting individual simulations and aggregate results
- Slicing result sets to compare scenarios

Essentially, if we think of the processed results of a model run as being

- A collection of scalar outputs (e.g., cumulative infections, total deaths)
- A dataframe of time-varying outputs (e.g., new diagnoses per day, number of people on treatment each day)

then the classes `Dataset` and `Samples` manage collections of these results. In particular, the `Samples` class manages different random samples of the same parameters, and the `Dataset` class manages a collection of `Samples`. 

<div class="alert alert-block alert-info">    
These classes are particularly designed to facilitate working with tens of thousands of simulation runs, where other approaches such as those based on the MultiSim class may not be feasible.
</div>

```{python}
import starsim as ss
ss.options(jupyter=True)
import numpy as np
import pandas as pd
from pathlib import Path
import matplotlib.pyplot as plt
import sciris as sc
```

## Obtaining simulation output

To demonstrate usage of this class, we will first consider constructing the kinds of output that the `Samples` class stores. We begin by running a basic simulation using the SIR model:

```{python}
ppl = ss.People(5000)
net = ss.ndict(ss.RandomNet(n_contacts=ss.poisson(5)))
sir = ss.SIR()
sim = ss.Sim(people=ppl, networks=net, diseases=sir, rand_seed=0)
sim.run();
```

### Dataframe output

A `Sim` instance is (in general) too large and complex to efficiently store on disk - the file size and loading time make it prohibitive to work with tens of thousands of simulations. Therefore, rather than storing entire `Sim` instances, we instead store dataframes containing just the simulation results and any other pre-processed calculated quantities. There are broadly speaking two types of outputs:

- Scalar outputs at each timepoint (e.g., daily new cases)
- Scalar outputs for each simulation (e.g., total number of deaths)

These outputs can each be produced from a `Sim` - the former has a tabular structure, and the latter has a dictionary structure (which can later be assembled into a table where the rows correspond to each simulation). The `export_df` method is a quick way to obtain a dataframe with the appropriate structure retaining all results from the `Sim`.


<div class="alert alert-block alert-info">    
In real-world use, it is often helpful to write your own function to extract a dataframe of simulation outputs, because typically some of the outputs need to be extracted from custom Analyzers.
</div>

```{python}
sim.to_df()
```

### Scalar/summary outputs

We can also consider extracting a summary dictionary of scalar values. For example:

```{python}
summary = {}
summary['seed'] = sim.pars['rand_seed']
summary['p_death'] = sim.diseases[0].pars.p_death.pars.p
summary['cum_infections'] = sum(sim.results.sir.new_infections)
summary['cum_deaths'] = sum(sim.results.new_deaths)
summary
```

<div class="alert alert-warning alert-info">    
Notice how in the example above, the summary contains both simulation inputs (seed, probability of death) as well as simulation outputs (total infections, total deaths). The simulation summary should contain sufficient information about the simulation inputs to identify the simulation. The seed should generally be present. The other inputs normally correspond to variables that scenarios are being run over. In this example, we will run scenarios comparing simulations with different probabilities of death. Therefore, we need to include the death probability in the simulation summary. 
</div>

### Running the model

For usage at scale, the steps of creating a simulation, running it and producing these outputs are usually encapsulated in functions:

```{python}
def get_sim(seed, p_death):
    ppl = ss.People(5000)
    net = ss.RandomNet(n_contacts=ss.poisson(5))
    sir = ss.SIR(p_death=p_death)
    sim = ss.Sim(people=ppl, networks=net, diseases=sir, rand_seed=seed)
    sim.init(verbose=0)
    return sim
    
def run_sim(seed, p_death):
    sim = get_sim(seed, p_death)
    sim.run(verbose=0)
    df = sim.to_df()
    
    summary = {}
    summary['seed'] = sim.pars['rand_seed']
    summary['p_death']= sim.diseases[0].pars.p_death.pars.p
    summary['cum_infections'] = sum(sim.results.sir.new_infections)
    summary['cum_deaths'] = sum(sim.results.new_deaths)
    
    return df, summary
```

<div class="alert alert-success">    
The functions above could be combined into a single function. However, in real world usage it is often convenient to be able to construct a simulation independently of running it (e.g., for diagnostic purposes or to allow running the sim in a range of different ways). The suggested structure above, with a <code>get_sim()</code> function and a <code>run_sim()</code> function are recommended as standard practice.
</div>

Now running a simulation for a given beta/seed value and returning the processed outputs can be done in a single step:

```{python}
# Scalar output
df, summary = run_sim(0, 0.2);
summary
```

We can produce all of the samples associated with a scenario by iterating over the input seed values. This is being done in a basic loop here, but could be done in more sophistical ways to leverage parallel computing (e.g., with `sc.parallelize` for single host parallelization, or with `celery` for distributed computation). 

```{python}
# Run a collection of sims
n = 20
seeds = np.arange(n)
outputs = [run_sim(seed, 0.2) for seed in seeds]
```

## Saving and loading the samples

We have now produced simulation outputs (dataframes and summary statistics) for 20 simulation runs. The `outputs` here are a list of tuples, containing the dataframe and dictionary outputs for each sample. This list can be passed to the `cvv.Samples` class to produce a single compressed file on disk:

```{python}
resultsdir = Path('results')
resultsdir.mkdir(exist_ok=True, parents=True)
ss.Samples.new(resultsdir, outputs, identifiers=["p_death"])
list(resultsdir.iterdir())
```

Notice that a list of `identifiers` should be passed to the `Samples` constructor. This is a list of keys in the simulation summary dictionaries that identifies the scenario. These would be model inputs rather than model outputs, and they should be the same for all of the outputs passed into the `Samples` object. If no file name is explicitly provided, the file will automatically be assigned a name based on the identifiers.

<div class="alert alert-success">    
The <code>Samples</code> file internally contains metadata recording the identifiers. When <code>Samples</code> are accessed using the <code>Dataset</code> class, they can be accessed via the internal metadata. Therefore for a typical workflow, the file name largely doesn't matter, and it usually doesn't need to be manually specified.
</div>

The saved file can be loaded and accessed via the `Samples` class. **Importantly, individual files can be extracted from a `.zip` file without decompressing the entire archive**. This means that loading the summary dataframe and using it to selectively load the full outputs for individual runs can be done efficiently. For example, loading retrieving a single result from a `Samples` file would take a similar amount of time regardless of whether the file contained 10 samples or 100000 samples. 

```{python}
# Load the samples
res = ss.Samples('results/0.2.zip')
res.summary
```

When the `Samples` file was created, a dictionary of scalars was provided for each result. These are automatically used to populate a 'summary' dataframe, where each identifier (and the seed) are used as the index, and the remaining keys appear as columns, as shown above. As a shortcut, columns of the summary dataframe can be accessed by indexing the `Samples` object directly, without having to access the `.summary` attribute e.g.,

```{python}
res['cum_infections']
```

Each simulation is uniquely identified by its seed, and the time series dataframe for each simulation can be accessed by indexing the `Samples` object with the seed:

```{python}
res[0]
```

The dataframes in the `Samples` object are cached, so that the dataframes don't all need to be loaded in order to start working with the file. The first time a dataframe is accessed, it will be loaded from disk. Subsequent requests for the dataframe will return a cached version instead. The cached dataframe is copied each time it is retrieved, to prevent accidentally modifying the original data. 

## Common analysis operations

Here are some examples of common analyses that can be performed using functionality in the `Samples` class

### Plotting summary quantities

Often it's useful to be able plot distributions of summary quantities, such as the total infections. This can be performed by directly indexing the `Samples` object and then using the appropriate plotting command:

```{python}
plt.hist(res['cum_infections'], density=True)

plt.xlabel('Total infections')
plt.ylabel('Probability density')
```

### Plotting time series

Time series plots can be obtained by accessing the dataframes associated with each seed, and then plotting quantities from those. For convenience, iterating over the `Samples` object will automatically iterate over all of the dataframes associated with each seed. For example:

```{python}
for df in res:
    plt.plot(df['sir_new_infections'], color='b', alpha=0.1)
```

### Other ways to access content

We have seen so far that we can use

- `res.summary` - retrieve dataframe of summary outputs
- `res[summary_column]` - retrieve a column of the summary dataframe
- `res[seed]` - retrieve the time series dataframe associated with one of the simulations
- `for df in res` - iterate over time series dataframes

Sometimes it is useful to have access to both the summary dictionary and the time series dataframe associated with a single sample. These can be accessed using the `get` method, which takes in a seed, and returns both outputs for that seed together:

```{python}
res.get(0) # Retrieve both summary quantities and dataframes
```

In the same way that it is possible to index the `Samples` object directly in order to retrieve columns from the summary dataframe, it is also possible to directly index the `Samples` object to get a column of the time series dataframe. In this case, pass a tuple of items to the `Samples` object, where the first item is the seed, and the second is a column from the time series dataframe. For example:

```{python}
res[0,'sir_n_infected'] # Equivalent to `res[0]['sir.n_infected']`
```

### Filtering results

The `.seeds` attribute contains a listing of seeds, which can be helpful for iteration:

```{python}
res.seeds
```

The seeds are drawn from the summary dataframe, which defines which seeds are accessible via the `Samples` object. Therefore, you can drop rows from the summary dataframe to filter the results. For example, suppose we only wanted to analyze simulations with over 4900 deaths. We could retrieve a copy of the summary dataframe that only contains matching simulations:

```{python}
res.summary.loc[res['cum_infections']>4900]
```

We can then make a copy of the results and write the reduced summary dataframe back to that object:

```{python}
res2 = res.copy()
res2.summary = res.summary.loc[res['cum_infections']>4900]
```

<div class="alert alert-warning">    
Unlike <code>sc.dcp()</code>, copying using the <code>.copy()</code> method only deep copies the summary dataframe. It does not duplicate the time series dataframes or the cache. For <code>Samples</code> objects, it is therefore generally preferable to use <code>.copy()</code>.
</div>

Now notice that there are fewer samples, and the seeds have been filtered:

```{python}
len(res)
```

```{python}
len(res2)
```

```{python}
res2.seeds
```

```{python}
plt.hist(res2['cum_infections'], density=True)
plt.xlabel('Total infections')
plt.ylabel('Probability density')
```

### Applying functions and transformations

Sometimes it might be necessary to calculate quantities that are derived from the time series dataframes. These could be simple scalar values, such as totals or averages that had not been computed ahead of time, or extracting values from each simulation at a particular point in time. As an alternative to writing a loop that iterates over the seeds, the `.apply()` method takes in a function and maps it to every dataframe. This makes it quick to construct lists or arrays with scalar values extracted from the time series. For example, suppose we wanted to extract the peak number of people infected from each simulation:

```{python}
peak_infections = lambda df: df['sir_n_infected'].max()
res.apply(peak_infections)
```

## Options when loading

There are two options available when loading that can change how the `Samples` class interacts with the file on disk:

- `memory_buffer` - copy the entire file into memory. This prevents the file from being locked on disk and allows scripts to be re-run and results regenerated while still running the analysis notebook. This defaults to `True` for convenience, but loading the entire file into memory can be problematic if the file is large (e.g., >1GB) in which case setting `memory_buffer=False` may be preferable
- `preload` - Populate the cache in one step. This facilitates interactive usage of the analysis notebook by making the runtime of analysis functions predictable (since all results will be retrieved from the cache) at the expense of a long initial load time


### Implementation details

If the file is loaded from a memory buffer, the `._zipfile` attribute will be populated. A helper property `.zipfile` is used to access the buffer, so if caching is not used, `.zipfile` returns the actual file on disk rather than the buffer:

```{python}
res = ss.Samples('results/0.2.zip', memory_buffer=True) # Copy the entire file into memory
print(res._zipfile)
print(res.zipfile)
```

```{python}
res = ss.Samples('results/0.2.zip', memory_buffer=False) # Copy the entire file into memory
print(res._zipfile)
print(res.zipfile)
```

The dataframes associated with the individual dataframes are cached on access, so `pd.read_csv()` only needs to be called once. The cache starts out empty:

```{python}
res._cache
```

When a dataframe is accessed, it is automatically stored in the cache:

```{python}
res[0]
res._cache.keys()
```

This means that iterating through the dataframes the first time can be slow (but in general, iterating over all dataframes is avoided in favour of either only using summary outputs, or accessing a subset of the runs):

```{python}
with sc.Timer():
    for df in res:
        continue
```

```{python}
with sc.Timer():
    for df in res:
        continue
```

The `preload` option populates the entire cache in advance. This makes creating the `Samples` object slower, but operating on the dataframes afterwards will be consistently fast. This type of usage can be useful when wanting to load large files in the background and then interactively work with them afterwards. 

```{python}
with sc.Timer():
    res = ss.Samples('results/0.2.zip', preload=True)
```

```{python}
with sc.Timer():
    for df in res:
        continue
```

```{python}
with sc.Timer():
    for df in res:
        continue
```

Together, these options provide some flexibility in terms of memory and time demands to suit analyses at various different scales.

## Running scenarios

Suppose we wanted to compare a range of different `p_death` values and `initial` values (initial number of infections). We might define these runs as:

```{python}
initials = np.arange(1,4)
p_deaths = np.arange(0,1,0.25)
```

Recall that our `run_sim()` function had an argument for `p_death`. We can extend this to include the `initial` parameter too. We can actually generalize this further by passing the parameters as keyword arguments to avoid needing to hard-code all of them. Note that we also need to add the `initial` value to the summary outputs:

```{python}
def get_sim(seed, **kwargs):
    ppl = ss.People(1000)
    net = ss.RandomNet(n_contacts=ss.poisson(5))
    sir = ss.SIR(pars=kwargs)
    sim = ss.Sim(people=ppl, networks=net, diseases=sir, rand_seed=seed, dur=ss.years(10))
    sim.init(verbose=0)
    return sim
    
def run_sim(seed, **kwargs):
    sim = get_sim(seed, **kwargs)
    sim.run(verbose=0)
    df = sim.to_df()
    sir = sim.diseases.sir
    
    summary = {}
    summary['seed'] = sim.pars.rand_seed
    summary['p_death']= sir.pars.p_death.pars.p
    summary['initial']= sir.pars.init_prev.pars.p
    summary['cum_infections'] = sum(sim.results.sir.new_infections)
    summary['cum_deaths'] = sum(sim.results.new_deaths)
    
    return df, summary
```

We can now easily run a set of scenarios with different values of `p_death` and save each one to a separate `Samples` object. Note that when we create the `Samples` objects now, we also want to specify that `'init_prev'` is one of the identifiers for the scenarios:

```{python}
# Clear the existing results
for file_path in resultsdir.glob('*'):
    file_path.unlink()
```

```{python}
# Run the sweep over initial and p_death
n = 10
seeds = np.arange(n)
for init_prev in initials:
    for p_death in p_deaths:
        outputs = [run_sim(seed, init_prev=init_prev, p_death=p_death) for seed in seeds]
        ss.Samples.new(resultsdir, outputs, ["p_death", "initial"])
```

The results folder now contains a collection of saved `Samples` objects. Notice how the automatically selected file names now contain both the `p_death` value and the `initial` value, because they were both specified as identifiers. We can load one of these objects in to see how these identifiers are stored and accessed inside the `Samples` class:

```{python}
list(resultsdir.iterdir())
```

```{python}
res = ss.Samples('results/0.25-2.zip')
```

The 'id' of a `Samples` object is a dictionary of the identifiers, which makes it easy to access the input parameters associated with a set of scenario runs:

```{python}
res.id
```

The 'identifier' is a tuple of these values, which is suitable for use as a dictionary key. This can be useful for accumulating and comparing variables across scenarios:

```{python}
res.identifier
```

### Loading multiple scenarios

We saw above that we now have a directory full of `.zip` files corresponding to the various scenario runs. These can be accessed using the `Dataset` class, which facilitates accessing multiple instances of `Samples`. We can pass the folder containing the results to the `Dataset` constructor to load them all:

```{python}
results = ss.Dataset(resultsdir)
results
```

The `.ids` attribute lists all of the values available across scenarios in the results folder:

```{python}
results.ids
```

The individual results can be accessed by indexing the `Dataset` instance using the values of the identifiers. For example:

```{python}
results[0.25,2]
```

This indexing operation is sensitive to the order in which the identifiers are specified. The `.get()` method allows you to specify them as key-value pairs:

```{python}
results.get(initial=2, p_death=0.25)
```

Iterating over the `Dataset` will iterate over the `Samples` instances contained within it:

```{python}
for res in results:
    print(res)
```

This can be used to extract and compare values across scenarios. For example, we could consider the use case of making a plot that compares total deaths across scenarios:

```{python}
labels = []
y = []
yerr = []

for res in results:
    labels.append(res.id)
    y.append(res['cum_deaths'].median())

plt.barh(np.arange(len(results)),y, tick_label=labels)
plt.xlabel('Median total deaths');
plt.ylabel('Scenario')
```

### Filtering scenarios

Often plots need to be generated for a subset of scenarios e.g., for sensitivity analysis or to otherwise compare specific scenarios. `Dataset.filter` returns a new `Dataset` containing a subset of the results:

```{python}
for res in results.filter(initial=2):
    print(res)
```

```{python}
for res in results.filter(p_death=0.25):
    print(res)
```

This is also a quick and efficient operation, so you can easily embed filtering commands inside the analysis to select subsets of the scenarios for plotting and other output generation. For instance:

```{python}
for res, color in zip(results.filter(initial=2), sc.gridcolors(4)):
    plt.plot(res[0].index, np.median([df['new_deaths'] for df in res], axis=0), color=color, label=f'p_death = {res.id["p_death"]}')
plt.legend()
plt.title('Sensitivity to p_death (initial = 2)')
plt.xlabel('Year')
plt.ylabel('New deaths')
```

```{python}
for res, color in zip(results.filter(p_death=0.25), sc.gridcolors(3)):
    plt.plot(res[0].index, np.median([df['new_deaths'] for df in res], axis=0), color=color, label=f'initial = {res.id["initial"]}')
plt.legend()
plt.title('Sensitivity to initial infections (p_death = 0.25)')
plt.xlabel('Year')
plt.ylabel('New deaths')
sc.dateformatter()
```

