---
feature_story: FS_56_43_05_79
feature_title: search diff collection
feature_status: TEST
---

# Feature description

To load data, `argrelay` server can use multiple MongoDB connections - see `MongoClientWrapper.store_envelopes`.

Single collection for all `data_envelope`-s works fine for small data sets,
but adds extra latency (observable with `mongomock`) for large data sets.

# Server config

See `class_to_collection_map` dict in server config - it maps `envelope_class` to its MongoDB collection name:

```yaml
class_to_collection_map:
    ClassCluster: ClassCluster
    ClassHost: ClassHost
    ClassService: ClassService
    AccessType: AccessType
```

# Plugin behavior

*   During the data load, `PluginType.LoaderPlugin` uses this config to select MongoDB collection name.

    Normally, if map entry is missing for one of the `envelope_class`,
    `PluginType.LoaderPlugin` falls back to use `envelope_class` name as collection name by default.

    That behavior is not mandatory, but the behavior should be synced with `PluginType.DelegatorPlugin`.

*   During requests, `PluginType.DelegatorPlugin` provides MongoDB collection name via FS_31_70_49_15 `search_control`.

# Current limitations

*   All envelopes for specific `envelope_class` can only be loaded into one MongoDB collection.
*   Different MongoDB collections can be used for different `envelope_class`-es.

# Loading behavior

Server cleans each collection completely and populates all `data_envelope`-s on start
(each `data_envelope` goes into its own collection based on `evelope_class` and `class_to_collection_map` config).

# Additional benefits

This will avoid the hardest limitation of MongoDB (not applicable for `mongomock`):
number of indexes per collection = 64
https://www.mongodb.com/docs/manual/reference/limits/#mongodb-limit-Number-of-Indexes-per-Collection
See: `test_MongoClient_index_limits.py`

This will also reduce latency in for seemingly non-important use cases with `mongomock` instead of real Mongo DB.
But `mongomock` is expected to be the default choice (majority of installations).

It also enables us to set up different handling for different collections.
Some of them might be too big to re-populate on re-start (too long).

# See also

Original issue: https://github.com/argrelay/argrelay/issues/10

[single_mongo_collection]: https://github.com/argrelay/argrelay/blob/f4c6a6fb9e5cb1226137c3744dd71693ae12c051/src/argrelay/mongo_data/MongoClientWrapper.py#L32
