## How it works

### ArangoDB concepts

stix2arango uses the following [ArangoDB data functions to do this](https://www.arangodb.com/docs/stable/data-structure.html);

1. Databases
2. Vertex Collections
3. Edge Collections

When executing the upload of a STIX object or STIX bundle they must select a;

* Database name: will create a database named as follows `<NAME>_database`. If database does not exist, one is created. Note, when entering name in the future, only the `<NAME>` part should ever be used.
* Collection name: will create a two collections, a vertex and edge, as follows `<NAME>_vertex_collection` and `<NAME>_edge_collection`. If collections do not exist, they are created. Note, when entering name in the future, only the `<NAME>` part should ever be used.

### STIX support and validation

A user must enter a valid STIX 2.1 bundle.

STIX objects in the bundle are validated against the STIX 2.1 spec. The assumption is that all objects have been generated by the STIX2 Python library and are valid.

The way STIX objects are stored depends on their type. Here is a list of all the possible STIX objects that can be ingested;

* [SDOs](https://docs.oasis-open.org/cti/stix/v2.1/os/stix-v2.1-os.html#_nrhq5e9nylke): STIX Domain Objects
	* A list of standard SDO `"type"`s is defined in the STIX specification. Can also be non-standard SDO, if an Extension Definition object is nested inside the object with `"extension_type": "new-sdo"`.
* [SCOs](https://docs.oasis-open.org/cti/stix/v2.1/os/stix-v2.1-os.html#_mlbmudhl16lr): STIX Cyber Observable Objects
	* A list of standard SCO `"type"`s is defined in the STIX specification. Can also be non-standard SDO, if an Extension Definition object is nested inside the object with `"extension_type": "new-sco"`.
* [SROs](https://docs.oasis-open.org/cti/stix/v2.1/os/stix-v2.1-os.html#_cqhkqvhnlgfh): STIX Relationship Objects
	* must be type `relationship` or `sighting`
* [SMOs](https://docs.oasis-open.org/cti/stix/v2.1/os/stix-v2.1-os.html#_mq8oo9k9rb2): STIX Meta Objects
	* must be type `language-content` or `marking-definition`
* [SBOs](https://docs.oasis-open.org/cti/stix/v2.1/os/stix-v2.1-os.html#_gms872kuzdmg) STIX Bundle Objects (wrapper of other STIX objects)
	* must be type `bundle`

Note for all standard object types, additional non-standard properties (in addition to those found in the STIX specification) can be added if there is an Extension Definition object is nested inside the object with `"extension_type": "property-extension"`.

### Bundles

Bundles can be detected where the `type` property is `bundle`. e.g.

```json
{
    "type": "bundle",
    "id": "bundle--<ID>",
    "objects": [
```

For bundles of objects, each object extracted and then treated as a separate STIX object.

### Order of bundle import / relationship generation

The next section of this document describes how STIX objects are inserted (and in some cases created). The order of insert, especially in the case of bundles is as follows;

1. STIX SMOs
2. STIX SDOs
3. STIX SCOs
4. STIX SROs
5. Creation of STIX SROs for nested references

### Default STIX objects

The following objects are automatically inserted to each vertex collection on creation.

They should have the following custom properties

* `_key`: the STIX `id` appended with `_record_modified` time
* `_stix2arango_note`: `automatically imported on collection creation`
* `_record_created`: time of collection creation
* `_record_modified`: time of collection creation
* `_record_md5_hash`: hash of object
* `_is_latest`: `true`

The are added like so;

```sql
LET default_objects = [
    {
        "_key": "<THE OBJECTS STIX ID>",
        "_stix2arango_note": "automatically imported object at script runtime",
        "_record_created": "<DATETIME OBJECT WAS INSERTED IN DB>",
        "_record_modified": "<DATETIME OBJECT WAS INSERTED IN DB>",
        "_record_md5_hash": "<HASH OF OBJECT>",
        "_is_latest": true,
        "<DEFAULT STIX OBJECT>"
    }
]
FOR default_object IN default_objects
INSERT default_object INTO <SOURCE>_vertex_collection
```

#### TLPs Marking definitions

These can be found in `templates/marking-definition.json`

#### Marking definitions

* https://raw.githubusercontent.com/muchdogesec/stix4doge/main/objects/marking-definition/stix2arango.json

#### Identities (`identity`) SDOs

* https://raw.githubusercontent.com/muchdogesec/stix4doge/main/objects/identity/stix2arango.json (identity used to create embedded relationship objects)
* https://raw.githubusercontent.com/muchdogesec/stix4doge/main/objects/identity/dogesec.json (seeded in all bundles)

#### Identify default objects in the DB

Once a collection has been created, you can identify these objects as follows;

```sql
FOR doc IN <COLLECTION>_vertex_collection
  FILTER doc._stix2arango_note == "automatically imported on collection creation"
    RETURN [doc]
```

### A note on STIX object life-cycle

ALL STIX objects all contain an `id`. An `id` denotes a single objects. Updates to an object can occur over time (e.g. property values changed), however the ID will remain the same.

The way stix2arango handles this is dependent on the object type due to their differences (e.g. SDOs have a `modified` property, but SCOs do not).

The logic for each object type of UPDATE is described later in this document.

### INSERT New Objects (all types) to Vertex Collections

All STIX 2.1 SDOs, SCOs, SMOs and Sightings that have not been added to the collection (that is, the `id` of the object is not already in the collection) are added to the ArangoDB vertex collections selected by the user as follows;

```sql
LET objects = [
    {
        "_key": "<THE OBJECTS STIX ID>",
        "_bundle_id": "<THE BUNDLE ID>",
        "_file_name": "<FILENAME OF IMPORT>",
        "_stix2arango_note": "<IF ENTERED BY USER ON IMPORT>",
        "_record_created": "<DATETIME OBJECT WAS INSERTED IN DB>",
        "_record_modified": "<DATETIME OBJECT WAS LAST MODIFIED IN DB>",
        "_record_md5_hash": "<HASH OF OBJECT>",
        "_is_latest": true,
        "<STIX SDO, SCO, SMO, Sighting PROPERTIES>"
    }
]
FOR object IN objects
INSERT object INTO <NAME>_vertex_collection
```

Note the use of [default and custom ArangoDB attributes](https://docs.arangodb.com/3.11/concepts/data-structure/documents/) (e.g. `_bundle_id`).

Where:

* `_id`: autogenerated by ArangoDB in format `COLLECTION/_KEY`
    * e.g. `sigma_rules_vertex_collection/indicator--e18ddf20-e4fe-58fb-98d1-8f2a3a1a5979+2024-06-07T07:50:17.776824`
* `_key`: the STIX `id` appended with `_record_modified` time
    * e.g. `indicator--e18ddf20-e4fe-58fb-98d1-8f2a3a1a5979+2024-06-07T07:50:17.776824`
* `_stix2arango_note`: is the value entered by user (if any) on import using `--stix2arango_note`
 `_file_name`: is the filename of the file imported by the user
    * e.g. `enterprise-attack-15_0.json`
* `_bundle_id`: this is the bundle ID
    * e.g. `bundle--1e532e16-dc33-4c7b-be64-428c93ef3b46`
* `_record_created`: the datetime the object was inserted into the database (e.g. `2020-01-01T01:00:00.000Z`)
* `_record_modified`: the datetime the object was last updated (e.g. `2020-01-01T01:00:00.000Z`). Note, for new objects this always matches the `_record_created` time. For updated object, this matches the time of update.
* `_record_md5_hash` is an MD5 hash of the STIX object properties (that is all properties that do not begin with `_*`) and the `_stix2arango_note` property. This is used to detect updates to subsequent updates to objects.
* `_is_latest`: boolean, for newly inserted objects will always be `true`. See update logic to understand why.

For example inserting an object from a bundle into a collection named `demo`;

```sql
LET objects = [
    {
        "_key": "vulnerability--00008327-8c28-5aa7-81ed-43a1f9d34e07+2019-12-31T09:15:15.347Z",
        "_bundle_id": "bundle--c4d4dc95-47ea-41cb-a24d-c9d872eb4368",
        "_file_name": "bundle--c4d4dc95-47ea-41cb-a24d-c9d872eb4368.json",
        "_stix2arango_note": "vulnerabilty_bundle_20220120",
        "_record_created": "2020-01-01T00:00:00.000Z",
        "_record_modified": "2020-01-01T00:00:00.000Z",
        "_record_md5_hash": "50843d84b5678c36789b9ebd150f24ad",
        "_is_latest": true,
        "type": "vulnerability",
	    "spec_version": "2.1",
	    "id": "vulnerability--00008327-8c28-5aa7-81ed-43a1f9d34e07",
	    "created_by_ref": "identity--7a1fea52-aa94-4f85-bc5e-c55dd4e1896b",
	    "created": "2019-12-31T09:15:15.347Z",
	    "modified": "2021-01-14T21:20:28.997Z",
	    "name": "CVE-2020-35909"
    }
]
FOR object IN objects
INSERT object INTO demo_vertex_collection
```

### UPDATE Existing Objects in Vertex Collections

All vertex objects contain an MD5 has of all the STIX properties and the `_record_md5_hash`.

If the `_record_md5_hash` changes for an object, then an update to the record is required (note, the update may be inserting an older object as well as inserting a newer object).

Depending on the STIX object type, the following behaviour happens to perform the updates;

#### 1. If objects contain a `modified` property

SDOs, SROs and some SMOs are required by the STIX spec to contain `modified` (as datetime).

Note, `modified` time alone is not always enough to detect an update. Sometimes an object exists in many versions of a framework. e.g. the same modified time of the same object might exist in MITRE ATT&CK version 14, version 15, etc.

As such, we also allow the user to define a `_stix2arango_note` (can be blank), to define additional notes, such as a framework version.

First stix2arango checks if identical object with same `id` to the one being imported already exists in the DB.

If an existing version of the object does exist, the `_record_md5_hash` (which is built from all default STIX object properties and `_stix2arango_note` property) is compared between each record.

If an identical `_record_md5_hash` value already exists for `id` (that is all default STIX properties and `_stix2arango_note` properties are the same), then no update is due and record is skipped.

If a new `_record_md5_hash` exists for `id` then an update happens. However, the type of update must be determined as follows...

##### 1.1 If update is latest version of the object

If:

* the `modified` time of the new object is greater than or equal to the current `_is_latest` copy of that object
* and the `_record_md5_hash` is different

Then the newly imported object should be considered the latest version of it.

This record should be inserted as `_is_latest==true`.

All other objects with the same `id` (could be 1 or more) should be marked as `_is_latest==false`

The MITRE ATT&CK use-case described earlier is a good example of when this logic is required.

##### 1.2 If update is older version of the object

If

* the `modified` time of the new object being imported is less than the current `_is_latest` object
* and the `_record_md5_hash` is different

Then the new object is NOT considered the latest version.

This record should be inserted as `_is_latest==false` and inserted with this value.

No other objects sharing the same `id` are modified in this case.

##### 1.3 If update represents no change at all

Note, there are occasions where the STIX objects are identical between updates (inc. having same STIX properties and `_stix2arango_note` value). For example, where user imports the same bundle twice.

If: 

* the `_record_md5_hash` is the same for the object with same `id`

Then the new object should be considered a duplicate AND NOT imported (the record is skipped for insert).

#### 2. Objects with no `modified` times

SCOs, and some SMOs never have `created` or `modified` properties. Thus, time cannot be used as a judge of update as it can for SDOs. It does however make update logic a bit simpler...

##### 2.1 If update is latest version of the object

If 

* `_record_md5_hash` is unique for object ID

Then the newly imported object should be considered the latest version of it.

This record should be inserted as `_is_latest==true`.

All other objects with the same `id` (could be 1 or more) should be marked as `_is_latest==false`

##### 2.2 If update represents no change at all

If 

* the `_record_md5_hash` is the same for the object with same `id`

Then the new object should be considered a duplicate AND NOT imported (the record is skipped for insert).

### Adding New Objects to Edge collections

#### A note on `_to` and `_from` validation

ArangoDB uses the `_from` and `_to` in edge collections to denote the source and target object in a relationship.

This means the source and target objects must exist in ArangoDB else the creation of them will result in an error.

As noted earlier, some STIX objects are automatically inserted into a collection on its creation. However, this only covers certain STIX objects (e.g TLP).

Generally if importing a STIX 2.1 bundle, all objects referenced in it are also contained inside of it.

If this is not the case, or you choose to upload objects individually you need to ensure any relationships arango2stix will create (as described below) already exist, else you will get an error.

The STIX SRO will still be imported, but it won't be properly traversable using graph queries. You can identify these objects where `_stix2arango_ref_err==true`.

#### New Relationship SROs

SROs of type `relationship` represent relationships (edges) between two STIX objects they are added to ArangoDB edge collections.

New Relationship SROs (where no other object imported already has same `id`) can be inserted to ArangoDB as follows;

```sql
LET relationships = [
    {
        "_key": "<THE OBJECTS STIX ID>",
        "_from": "<COLLECTION NAME>/<OBJECTS SOURCE_REF>",
        "_to": "<COLLECTION NAME>/<OBJECTS TARGET_REF>",
        "_bundle_id": "<THE BUNDLE ID>",
        "_file_name": "<FILENAME OF IMPORT>",
        "_stix2arango_note": "<IF ENTERED BY USER ON IMPORT>",
        "_stix2arango_ref_err": "<IF SOURCE OR TARGET NOT IN COLLECTION IS TRUE, ELSE FALSE>",
        "_record_created": "<DATETIME OBJECT WAS INSERTED IN DB>",
        "_record_modified": "<DATETIME OBJECT WAS LAST MODIFIED IN DB>",
        "_record_md5_hash": "<HASH OF OBJECT>",
        "_is_latest": true,
        "_is_ref": false,        
        "<STIX Relationship OBJECT PROPERTIES>"
    }
]
FOR relationship IN relationships
INSERT relationship INTO <NAME>_edge_collection
```

The custom properties used are identical to vertex objects with the addition of the property:

* `_is_ref`: boolean, denotes if object was created by a ref or refs property insides a STIX object (see refs section). Will always be `false` if created from relationship type objects.
* `_stix2arango_ref_err`: boolean, denotes if the `source_ref` or `target_ref` also exist in the collection. If one or both don't exist in the same collection this value will be set to `true`

For example;

```sql
LET relationships = [
    {
        "_key": "relationship--00ccf332-d848-586e-af0a-4004097dba20+2020-01-01T00:00:00.000Z",
        "_from": "demo_vertex_collection/indicator--00ccf332-d848-586e-af0a-4004097dba20",
        "_to": "demo_vertex_collection/vulnerability--00ccf332-d848-586e-af0a-4004097dba20",
        "_bundle_id": "bundle--c4d4dc95-47ea-41cb-a24d-c9d872eb4368",
        "_file_name": "bundle--c4d4dc95-47ea-41cb-a24d-c9d872eb4368.json",
        "_stix2arango_note": "vulnerabilty_bundle_20220120",
        "_stix2arango_ref_err": false,
        "_record_created": "2020-01-01T00:00:00.000Z",
        "_record_modified": "2020-01-01T00:00:00.000Z",
        "_record_md5_hash": "1cfa2a4e9ef359a410d9708c4fc15957",
        "_is_latest": true,
        "_is_ref": false,
		"type": "relationship",
		"spec_version": "2.1",
		"id": "relationship--00ccf332-d848-586e-af0a-4004097dba20",
		"created_by_ref": "identity--7a1fea52-aa94-4f85-bc5e-c55dd4e1896b",
		"created": "2008-12-12T18:30:02.843Z",
		"modified": "2021-04-09T13:52:56.360Z",
		"relationship_type": "identifies",
		"source_ref": "indicator--00ccf332-d848-586e-af0a-4004097dba20",
		"target_ref": "vulnerability--00ccf332-d848-586e-af0a-4004097dba20"
        }
    }
]
FOR relationship IN relationships
INSERT relationship INTO demo_edge_collection
```

#### Updates to SROs

Because `relationship` objects must contain `created` and `modified` properties, updates to `relationship` SROs are done in the same way as those in vertex collections.

#### Dealing with duplicate objects in a bundle

The above behaviour assumes objects are inserted in seperate bundles.

Although rare, it may be the case the the same object exists in the same bundle.

IT IS STRONGLY RECOMMENDED you split objects with the same `id` to different bundles before importing because stix2arango does not contain complete logic for handling these.

The current logic is as follow;

If:

* the bundle contains exactly the same object `id` and the `modified` times are the same OR object has no `modified` time then only one copy of the object is imported to the database (first object with duplicate `modified` time in bundle with `id` is imported)
* the bundle contain the same object (identical properties) by `id`  but the `modified` property is different in one or more instances, all versions of the objects with different `id`s are imported. Where two objects have same `modified` time in such cases, the first object with duplicate `modified` time in bundle with `id` is imported

#### New Embedded relationships (optional user setting)

All STIX 2.1 Objects (SDOs, SCOs, SMOs, SROs and SROs themselves) can also have embedded relationships found in properties ending in `_ref` (single reference to object) or `_refs` (list of references to one or more object).

e.g. 

```json
	"created_by_ref": "identity--c78cb6e5-0c4b-4611-8297-d1b8b55e40b5"
```

```json
	"object_refs": [
		"file--30038539-3eb6-44bc-a59e-d0d3fe84695a"
	]
```

Such references will not natively be created by Arango as edges, meaning you can not traverse the graph using these properties.

To solve this problem stix2arango ca be used to create STIX Relationship objects for supported attribute ending with `ref` / `_refs`.

The following default `_ref` attributes that are not be considered are the following (as they are found in relationship objects, and thus an SRO already exists);

* `source_ref`
* `target_ref`

Similarly to avoid recursion any SRO object marked with `_is_ref==true` (that is has been generated by stix2arango for an embedded relationship) does not have any embedded relationships created from it. Doing so would cause an infinite recursion of `_is_ref==true` object generation.

When a valid `_ref` or `_refs` property is detected inside an imported Vertex or Relationship STIX object and the user has selected for embedded relationships embedded relationships are created.

If:

* the object the embedded relationships are being generated is new, that is the ID does not already exist in the collection

a new relationship object is created at the end of the import job (after all other objects are imported).

This is done using the STIX2 Python library, and such relationships are generated in the following structure;

```json
{
	"type": "relationship",
	"spec_version": "2.1",
    "id": "relationship--<UUID v5>",
	"created_by_ref": "<IMPORTED STIX2ARANGO IDENTITY ID>",
	"relationship_type": "<EMBEDDED RELATIONSHIP TYPE WITHOUT REF OR REFS AND _ REPLACED WITH ->",
	"created": "<INSERT TIME OF OBJECT IN DB>",
	"modified": "<MODIFIED TIME OF OBJECT IN DB>",
    "source_ref": "<OBJECT WITH REF PROPERTY>",
    "target_ref": "<REF TARGET PROPERTY>",
    "object_marking_refs": [
        "<object_marking_refs SHOULD MATCH SOURCE_REF OBJECT>"
    ],
    "external_references": [
        {
            "source_name": "stix2arango",
            "description": "embedded-relationship"
        }
    ]
}
```

Note, to ensure the SRO `id` persists when the same relationship is described, SROs `id`s are created using UUIDv5 strings as follows:

* namespace: `72e906ce-ca1b-5d73-adcd-9ea9eb66a1b4`
* value: `<RELATIONSHIP_TYPE>+<SOURCE_REF>+<TARGET_REF>`

On first creation, created and modified times are the same. If any updates to this embedded relationship happen (id) then the new object has the same created time, but an increased modified time.

This object can then be inserted into the database as per the normal Relationship SRO import process.

```sql
LET ref_relationships = [
    {
        "_key": "<THE OBJECTS STIX ID>+<DATETIME OF INSERT>",
        "_from": "<COLLECTION NAME>/<OBJECTS SOURCE_REF>",
        "_to": "<COLLECTION NAME>/<OBJECTS TARGET_REF>",
        "_stix2arango_note": "<SAME AS SOURCE_REF VALUE>",
        "_stix2arango_ref_err": "<IF TARGET NOT IN COLLECTION IS TRUE, ELSE FALSE>"
        "_record_created": "<DATETIME OBJECT WAS INSERTED IN DB>",
        "_record_modified": "<DATETIME OBJECT WAS LAST MODIFIED IN DB>",
        "_record_md5_hash": "<HASH OF OBJECT>",
        "_is_latest": true,
        "_is_ref": true,
        "<CREATED STIX OBJECTS>"
    }
]
FOR relationship IN ref_relationships
INSERT relationship INTO <NAME>_edge_collection
```

The `_is_ref` property is always `true` for new embedded relationships allowing them to be easily filtered in Arango search.

Again all embedded relationships assume the `target_ref` object are in the same collection (hence why bundle should contain any embedded relationship STIX objects), else `_stix2arango_ref_err` will equal `true`.

#### Updating Embedded relationships (optional user setting)

If

* Vertex existing objects are updated (that is the `id` for the object already exists)

embedded relationships (`_is_ref==true`) for which that object id is a `source_ref` or `target_ref` will also need need to be updated.

##### If object being inserted is latest

If the new object being insterted (and embedded relationships are generated from) is the latest (`_is_latest==true`), firstly all Embedded relationships (`_is_ref==true`) that contain either a `source_ref` or `target_ref` property with that object are marked as `_is_latest==false`.

Then new embedded relationship are created for the object in the same way as described for New Embedded Relationships.

##### If object being inserted IS NOT latest

Sometimes old versions of an object can be inserted (`_is_latest==true`).

In which case, a new embedded relationship are created for the object in the same way as described for New Embedded Relationships HOWEVER the only difference is the `_is_latest` value is marked as false.