threat\_intel |Build Status: master|
====================================

Threat Intelligence APIs.

Supported threat intelligence feeds
-----------------------------------

The package contains API wrappers for:

-  OpenDNS Investigate API
-  VirusTotal API v2.0
-  ShadowServer API

--------------

OpenDNS Investigate API
~~~~~~~~~~~~~~~~~~~~~~~

`OpenDNS Investigate <https://investigate.opendns.com/>`__ provides an
API that allows querying for:

::

    * Domain categorization
    * Security information about a domain
    * Co-occurrences for a domain
    * Related domains for a domain
    * Domains related to an IP
    * Domain tagging dates for a domain
    * DNS RR history for a domain

To use the Investigate API wrapper import ``InvestigateApi`` class from
``threat_intel.opendns`` module:

.. code:: python

    from threat_intel.opendns import InvestigateApi

To initialize the API wrapper you need the API key:

.. code:: python

    investigate = InvestigateApi("<INVESTIGATE-API-KEY-HERE>")

You can also specify a file name where the API responses will be cached
in a JSON file, to save you the bandwidth for the multiple calls about
the same domains or IPs:

.. code:: python

    investigate = InvestigateApi("<INVESTIGATE-API-KEY-HERE>", cache_file_name="/tmp/cache.opendns.json")

Domain categorization
^^^^^^^^^^^^^^^^^^^^^

Calls ``domains/categorization/?showLabels`` Investigate API endpoint.
It takes a list (or any other Python enumerable) of domains and returns
the categories associated with this domains by OpenDNS along with a [-1,
0, 1] score, where -1 is a malicious status.

.. code:: python

    domains = ["google.com", "baidu.com", "bibikun.ru"]
    investigate.categorization(domains)

will result in:

::

    {
        "baidu.com": {"status": 1, "content_categories": ["Search Engines"], "security_categories": []},
        "google.com": {"status": 1, "content_categories": ["Search Engines"], "security_categories": []},
        "bibikun.ru": {"status": -1, "content_categories": [], "security_categories": ["Malware"]}
    }

Security information about a domain
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

Calls ``security/name/`` Investigate API endpoint. It takes any Python
enumerable with domains, e.g. list, and returns several security
parameters associated with each domain.

.. code:: python

    domains = ["google.com", "baidu.com", "bibikun.ru"]
    investigate.security(domains)

will result in:

::

    {
      "baidu.com": {
        "found": true,
        "handlings": {
          "domaintagging": 0.00032008666962131285,
          "blocked": 0.00018876906157154347,
          "whitelisted": 0.00019697641207465407,
          "expired": 2.462205150933176e-05,
          "normal": 0.9992695458052232
        },
        "dga_score": 0,
        "rip_score": 0,

        ..

      }
    }

Co-occurrences for a domain
^^^^^^^^^^^^^^^^^^^^^^^^^^^

Calls ``recommendations/name/`` Investigate API endpoint. Use this
method to find out a list of co-occurence domains (domains that are
being accessed by the same users within a small window of time) to the
one given in a list, or any other Python enumerable.

.. code:: python

    domains = ["google.com", "baidu.com", "bibikun.ru"]
    investigate.cooccurrences(domains)

will result in:

::

    {
      "baidu.com": {
        "found": true,
        "pfs2": [
          ["www.howtoforge.de", 0.14108563836506008],
        }

        ..

    }

Related domains for a domain
^^^^^^^^^^^^^^^^^^^^^^^^^^^^

Calls ``links/name/`` Investigate API endpoint. Use this method to find
out a list of related domains (domains that have been frequently seen
requested around a time window of 60 seconds, but that are not
associated with the given domain) to the one given in a list, or any
other Python enumerable.

.. code:: python

    domains = ["google.com", "baidu.com", "bibikun.ru"]
    investigate.related_domains(domains)

will result in:

::

    {
        "tb1": [
            ["t.co", 11.0],
            ]

        ..

    }

Domain tagging dates for a domain
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

Calls ``domains/name/`` Investigate API endpoint.

Use this method to get the date range when the domain being queried was
a part of the OpenDNS block list and how long a domain has been in this
list

.. code:: python

    domains = ["google.com", "baidu.com", "bibikun.ru"]
    investigate.domain_tag(domains)

will result in:

::

    {
        'category': u'Malware',
        'url': None,
        'period': {
            'begin': u'2013-09-16',
            'end': u'Current'
            }

        ..

    }

DNS RR history for a Domain
^^^^^^^^^^^^^^^^^^^^^^^^^^^

Calls ``dnsdb/name/a/`` Investigate API endpoint. Use this method to
find out related domains to domains given in a list, or any other Python
enumerable.

.. code:: python

    domains = ["google.com", "baidu.com", "bibikun.ru"]
    investigate.dns_rr(domains)

will result in:

::

    {
        'features': {
            'geo_distance_mean': 0.0,
            'locations': [
                {
                    'lat': 59.89440155029297,
                    'lon': 30.26420021057129
                }
                        ],
            'rips': 1,
            'is_subdomain': False,
            'ttls_mean': 86400.0,
            'non_routable': False,
            }

        ..

    }

DNS RR history for an IP
^^^^^^^^^^^^^^^^^^^^^^^^

Calls ``dnsdb/ip/a/`` Investigate API endpoint. Use this method to find
out related domains to the IP addresses given in a list, or any other
Python enumerable.

.. code:: python

    ips = ['8.8.8.8']
    investigate.rr_history(ips)

will result in:

::

    {
      "8.8.8.8": {
        "rrs": [
          {
            "name": "8.8.8.8",
            "type": "A",
            "class": "IN",
            "rr": "000189.com.",
            "ttl": 3600
          },
          {
            "name": "8.8.8.8",
            "type": "A",
            "class": "IN",
            "rr": "008.no-ip.net.",
            "ttl": 60
          },
        }

        ..

    }

Latest malicious domains for an IP
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

Calls ``ips/{ip}/latest_domains`` Investigate API endpoint.

Use this methods to see whether the IP address has any malicious domains
associated with it.

.. code:: python

    ips = ["8.8.8.8"]
    investigate.latest_malicious(ips)

will result in:

::

    }
        [
            '7ltd.biz',
            'co0s.ru',
            't0link.in',
        ]

        ..
    }

--------------

VirusTotal API
~~~~~~~~~~~~~~

`VirusTotal <https://www.virustotal.com/>`__ provides an
`API <https://www.virustotal.com/en/documentation/public-api/>`__ that
makes it possible to query for the reports about:

::

    * Domains
    * URLs
    * IPs
    * File hashes
    * File Upload
    * Live Feed
    * Advanced search

To use the VirusTotal API wrapper import ``VirusTotalApi`` class from
``threat_intel.virustotal`` module:

.. code:: python

    from threat_intel.virustotal import VirusTotalApi

To initialize the API wrapper you need the API key:

.. code:: python

    vt = VirusTotalApi("<VIRUSTOTAL-API-KEY-HERE>")

VirusTotal API calls allow to squeeze a list of file hashes or URLs into
a single HTTP call. Depending on the API version you are using (public
or private) you may need to tune the maximum number of the resources
(file hashes or URLs) that could be passed in a single API call. You can
do it with the ``resources_per_req`` parameter:

.. code:: python

    vt = VirusTotalApi("<VIRUSTOTAL-API-KEY-HERE>", resources_per_req=4)

When using the public API your standard request rate allows you too put
maximum 4 resources per request. With private API you are able to put up
to 25 resources per call. That is also the default value if you don't
pass the ``resources_per_req`` parameter.

Of course when calling the API wrapper methods in the ``VirusTotalApi``
class you can pass as many resources as you want and the wrapper will
take care of producing as many API calls as necessary to satisfy the
request rate.

Similarly to OpenDNS API wrapper, you can also specify the file name
where the responses will be cached:

.. code:: python

    vt = VirusTotalApi("<VIRUSTOTAL-API-KEY-HERE>", cache_file_name="/tmp/cache.virustotal.json")

Domain report endpoint
^^^^^^^^^^^^^^^^^^^^^^

Calls ``domain/report`` VirusTotal API endpoint. Pass a list or any
other Python enumerable containing the domains:

.. code:: python

    domains = ["google.com", "baidu.com", "bibikun.ru"]
    vt.get_domain_reports(domains)

will result in:

::

    {
      "baidu.com": {
        "undetected_referrer_samples": [
          {
            "positives": 0,
            "total": 56,
            "sha256": "e3c1aea1352362e4b5c008e16b03810192d12a4f1cc71245f5a75e796c719c69"
          }
        ],

        ..

        }
    }

URL report endpoint
^^^^^^^^^^^^^^^^^^^

Calls ``url/report`` VirusTotal API endpoint. Pass a list or any other
Python enumerable containing the URL addresses:

.. code:: python

    urls = ["http://www.google.com", "http://www.yelp.com"]
    vt.get_url_reports(urls)

will result in:

::

    {
      "http://www.google.com": {
        "permalink": "https://www.virustotal.com/url/dd014af5ed6b38d9130e3f466f850e46d21b951199d53a18ef29ee9341614eaf/analysis/1423344006/",
        "resource": "http://www.google.com",
        "url": "http://www.google.com/",
        "response_code": 1,
        "scan_date": "2015-02-07 21:20:06",
        "scan_id": "dd014af5ed6b38d9130e3f466f850e46d21b951199d53a18ef29ee9341614eaf-1423344006",
        "verbose_msg": "Scan finished, scan information embedded in this object",
        "filescan_id": null,
        "positives": 0,
        "total": 62,
        "scans": {
          "CLEAN MX": {
            "detected": false,
            "result": "clean site"
          },
        }
      ..

    }

URL scan endpoint
^^^^^^^^^^^^^^^^^

Calls 'url/scan' VirusTotal API endpoint. Submit a url or any other
Python enumerable containing the URL addresses:

.. code:: python

    urls = ["http://www.google.com", "http://www.yelp.com"]
    vt.post_url_report(urls)

Hash report endpoint
^^^^^^^^^^^^^^^^^^^^

Calls ``file/report`` VirusTotal API endpoint. You can request the file
reports passing a list of hashes (md5, sha1 or sha2):

.. code:: python

    file_hashes = [
        "99017f6eebbac24f351415dd410d522d",
        "88817f6eebbac24f351415dd410d522d"
    ]

    vt.get_file_reports(file_hashes)

will result in:

::

    {
      "88817f6eebbac24f351415dd410d522d": {
        "response_code": 0,
        "resource": "88817f6eebbac24f351415dd410d522d",
        "verbose_msg": "The requested resource is not among the finished, queued or pending scans"
      },
      "99017f6eebbac24f351415dd410d522d": {
        "scan_id": "52d3df0ed60c46f336c131bf2ca454f73bafdc4b04dfa2aea80746f5ba9e6d1c-1423261860",
        "sha1": "4d1740485713a2ab3a4f5822a01f645fe8387f92",
      }

     ..

    }

Hash rescan endpoint
^^^^^^^^^^^^^^^^^^^^

Calls ``file/rescan`` VirusTotal API endpoint. Use to rescan a
previously submitted file. You can request the file reports passing a
list of hashes (md5, sha1 or sha2):

Hash behaviour endpoint
^^^^^^^^^^^^^^^^^^^^^^^

Calls ``file/behaviour`` VirusTotal API endpoint. Use to get a report
about the behaviour of the file when executed in a sandboxed environment
(Cuckoo sandbox). You can request the file reports passing a list of
hashes (md5, sha1 or sha2):

.. code:: python

    file_hashes = [
        "99017f6eebbac24f351415dd410d522d",
        "88817f6eebbac24f351415dd410d522d"
    ]

    vt.get_file_behaviour(file_hashes)

Hash network-traffic endpoint
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

Calls ``file/network-traffic`` VirusTotal API endpoint. Use to get the
dump of the network traffic generated by the file when executed. You can
request the file reports passing a list of hashes (md5, sha1 or sha2):

.. code:: python

    file_hashes = [
        "99017f6eebbac24f351415dd410d522d",
        "88817f6eebbac24f351415dd410d522d"
    ]

    vt.get_file_network_traffic(file_hashes)

Hash download endpoint
^^^^^^^^^^^^^^^^^^^^^^

Calls ``file/download`` VirusTotal API endpoint. Use to download a file
by its hash. You can request the file reports passing a list of hashes
(md5, sha1 or sha2):

.. code:: python

    file_hashes = [
        "99017f6eebbac24f351415dd410d522d",
        "88817f6eebbac24f351415dd410d522d"
    ]

    vt.get_file_download(file_hashes)

IP reports endpoint
^^^^^^^^^^^^^^^^^^^

Calls ``ip-address/report`` VirusTotal API endpoint. Pass a list or any
other Python enumerable containing the IP addresses:

.. code:: python

    ips = ['90.156.201.27', '198.51.132.80']
    vt.get_ip_reports(ips)

will result in:

::

    {
      "90.156.201.27": {
        "asn": "25532",
        "country": "RU",
        "response_code": 1,
        "as_owner": ".masterhost autonomous system",
        "verbose_msg": "IP address found in dataset",
        "resolutions": [
          {
            "last_resolved": "2013-04-01 00:00:00",
            "hostname": "027.ru"
          },
          {
            "last_resolved": "2015-01-20 00:00:00",
            "hostname": "600volt.ru"
          },

          ..

        ],
        "detected_urls": [
          {
            "url": "http://shop.albione.ru/",
            "positives": 2,
            "total": 52,
            "scan_date": "2014-04-06 11:18:17"
          },
          {
            "url": "http://www.orlov.ru/",
            "positives": 3,
            "total": 52,
            "scan_date": "2014-03-05 09:13:31"
          }
        ],
      },

      "198.51.132.80": {

        ..

      }
    }

URL live feed endpoint
^^^^^^^^^^^^^^^^^^^^^^

Calls ``url/distribution`` VirusTotal API endpoint. Use to get a live a
feed with the latest URLs submitted to VirusTotal.

.. code:: python

    vt.get_url_distribution()

Hash live feed endpoint
^^^^^^^^^^^^^^^^^^^^^^^

Calls ``file/distribution`` VirusTotal API endpoint. Use to get a live a
feed with the latest Hashes submitted to VirusTotal.

.. code:: python

    vt.get_file_distribution()

Hash search endpoint
^^^^^^^^^^^^^^^^^^^^

Calls ``file/search`` VirusTotal API endpoint. Use to search for samples
that match some binary/metadata/detection criteria.

.. code:: python

    vt.get_file_search()

File date endpoint
^^^^^^^^^^^^^^^^^^

Calls ``file/clusters`` VirusTotal API endpoint. Use to list
simililarity clusters for a given time frame.

.. code:: python

    vt.get_file_clusters()

--------------

ShadowServer API
~~~~~~~~~~~~~~~~

`ShadowServer <http://shadowserver.org/>`__ provides and
`API <http://bin-test.shadowserver.org/>`__ that allows to test the
hashes against a list of known software applications.

To use the ShadowServer API wrapper import ``ShadowServerApi`` class
from ``threat_intel.shadowserver`` module:

.. code:: python

    from threat_intel.shadowserver import ShadowServerApi

To use the API wrapper simply call the ``ShadowServerApi`` initializer:

.. code:: python

    ss = ShadowServerApi()

You can also specify the file name where the API responses will be
cached:

.. code:: python

    ss = ShadowServerApi(cache_file_name="/tmp/cache.shadowserver.json")

To check whether the hashes are on the ShadowServer list of known
hashes, call ``get_bin_test`` method and pass enumerable with the hashes
you want to test:

.. code:: python

    file_hashes = [
        "99017f6eebbac24f351415dd410d522d",
        "88817f6eebbac24f351415dd410d522d"
    ]

    ss.get_bin_test(file_hashes)

--------------

Installation
------------

Install with ``pip``
~~~~~~~~~~~~~~~~~~~~

.. code:: shell

    $ pip install threat_intel

Testing
~~~~~~~

Go to town with ``make``:

.. code:: shell

    $ sudo pip install tox
    $ make test

.. |Build Status: master| image:: https://travis-ci.org/Yelp/threat_intel.svg?branch=master
   :target: https://travis-ci.org/Yelp/threat_intel
