Metadata-Version: 1.1
Name: infra-scraper
Version: 0.3
Summary: Infrastrucutre metadata scraper with support for multiple
resource providers and tools for relationship analysis and visualization.
Home-page: https://github.com/cznewt/infra-scraper/
Author: Aleš Komárek
Author-email: ales.komarek@newt.cz
License: Apache License, Version 2.0
Description-Content-Type: UNKNOWN
Description: 
        ==============================
        Infrastucture Metadata Scraper
        ==============================
        
        Get your live infrastructure topology data from your favorite resource
        providers for further processing, visualialitions, etc. Currently supported
        providers are:
        
        * Kubernetes clusters
        * OpenStack clouds
        * Amazon web services
        * SaltStack infrastructures
        * Terraform templates
        
        
        Road Map
        ========
        
        * Relational data transformations (slicing, dicing, transience)
        * Create JUJU, GCE and Azure scraping definitions
        * Enable corellation of resources and joined topologies
        * Open the way for the "Metadata as a service"
        
        
        Installation
        ============
        
        Release version of infra-scraper is currently available on `Pypi
        <https://pypi.org/project/infra-scraper/>`_, to install it, simply execute:
        
        .. code-block:: bash
        
            pip install infra-scraper
        
        To bootstrap latest development version into virtualenv, run following
        commands:
        
        .. code-block:: bash
        
            git clone git@github.com:cznewt/infra-scraper.git
            cd infra-scraper
            virtualenv venv
            source venv/bin/activate
            python setup.py install
        
        
        Configuration
        =============
        
        You provide one configuration file for all providers. The default location is
        ``/etc/infra-scraper/config.yaml`` but it can be overriden by
        ``INFRA_SCRAPER_CONFIG_PATH`` environmental variable, for example:
        
        .. code-block:: bash
        
            export INFRA_SCRAPER_CONFIG_PATH=~/scraper.yml
        
        
        ETCD hosted configuration
        -------------------------
        
        You can use ETCD as a storage backend for the configuration and scrape results. Following environmental parameters need to be set:
        
        .. code-block:: bash
        
            export INFRA_SCRAPER_CONFIG_BACKEND=etcd
            export INFRA_SCRAPER_CONFIG_PATH=/service/scraper/config
        
        
        Storage configuration
        ---------------------
        
        You can set you local filesystem path where scraped data will be saved.
        
        .. code-block:: yaml
        
            storage:
              backend: localfs
              path: /tmp/scraper
            endpoints: {}
        
        You can also set the scraping storage backend to use the ETCD service instead
        of a local filesystem backend.
        
        .. code-block:: yaml
        
            storage:
              backend: etcd
              path: /scraper
            endpoints: {}
        
        
        Endpoints configuration
        -----------------------
        
        Each endpoint kind expects a little different set of configuration. Following
        samples show the required parameters to setup individual endpoints.
        
        
        Amazon Web Services
        ~~~~~~~~~~~~~~~~~~~
        
        AWS scraping uses ``boto3`` high level AWS python SDK for accessing and
        manipulating AWS resources.
        
        
        .. code-block:: yaml
        
            endpoints:
              aws-admin:
                kind: aws
                config:
                  region: us-west-2
                  aws_access_key_id: <access_key_id>
                  aws_secret_access_key: <secret_access_key>
        
        
        Kubernetes
        ~~~~~~~~~~
        
        Kubernetes requires some information from kubeconfig file. You provide the
        parameters of the cluster and the user to the scraper. These can be found
        under corresponding keys.
        
        .. code-block:: yaml
        
            endpoints:
              k8s-admin:
                kind: kubernetes
                layouts:
                - force
                - hive
                config:
                  cluster:
                    server: https://kubernetes-api:443
                    certificate-authority-data: |
                      <ca-for-server-and-clients>
                  user:
                    client-certificate-data: |
                      <client-cert-public>
                    client-key-data: |
                      <client-cert-private>
        
        .. note::
        
            Options ``config.cluster`` and ``config.user`` can be found in your
            ``kubeconfig`` file. Just copy the config fragment with cluster parameters
            and fragment with user parameter.
        
        
        OpenStack
        ~~~~~~~~~
        
        Configurations for keystone v2 and keystone v3 clouds. Config for single
        tenant scraping.
        
        .. code-block:: yaml
        
            endpoints:
              os-v2-tenant:
                kind: openstack
                description: OpenStack (keystone v2) tenant
                scope: local
                layouts:
                - arc
                - force
                - hive
                - matrix
                config:
                  region_name: RegionOne
                  compute_api_version: '2.1'
                  auth:
                    username: user
                    password: password
                    project_name: project-name
                    domain_name: 'default'
                    auth_url: 'https://keystone-api:5000/v3'
        
        Config for scraping resources from entire cloud.
        
        .. code-block:: yaml
        
            endpoints:
              os-v2-admin:
                kind: openstack
                description: OpenStack (keystone v2) cloud
                scope: global
                layouts:
                - force
                - hive
                config:
                  region_name: RegionOne
                  auth:
                    username: admin
                    password: password
                    project_name: admin
                    auth_url:  https://keystone-api:5000/v2.0
        
        
        SaltStack
        ~~~~~~~~~
        
        Configuration for connecting to Salt API.
        
        .. code-block:: yaml
        
            endpoints:
              salt-global:
                kind: salt
                layouts:
                - force
                - hive
                config:
                  auth_url: 'http://127.0.0.1:8000'
                  username: salt-user
                  password: password
        
        
        Terraform
        ~~~~~~~~~
        
        Configuration for parsing terraform templates.
        
        .. code-block:: yaml
        
            endpoints:
              tf-aws-app:
                kind: terraform
                layouts:
                - hive
                config:
                  dir: ~/terraform/two-tier-aws
        
        Usage
        =====
        
        The application comes with several entry commands:
        
        
        Scraping commands
        -----------------
        
        **scraper_get <endpoint-name>**
        
          Scrape single endpoint once.
        
        **scraper_get_forever <endpoint-name>**
        
          Scrape single endpoint continuously.
        
        **scraper_get_all**
        
          Scrape all defined endpoints once.
        
        **scraper_get_all_forever**
        
          Scrape all defined endpoints continuously.
        
        
        UI and utility commands
        -----------------------
        
        **scraper_status**
        
          Display the service status, endpoints, scrapes, etc.
        
        **scraper_web**
        
          Start the UI with visualization samples and API that provides the scraped
          data.
        
        
        Supported Visualization Layouts
        ===============================
        
        Presented data requires different diagram visualizations. Diagrams are
        symbolic representation of information according to some visualization
        technique. Every time you need to emphasise different qualities of displayed
        resources. You can choose from several layouts to display collected data.
        
        
        Force-Directed Graph
        --------------------
        
        `Force-directed graph` drawing algorithms are used for drawing graphs in an
        aesthetically pleasing way. Their purpose is to position the nodes of a graph
        in two-dimensional or three-dimensional space so that all the edges are of
        more or less equal length and there are as few crossing edges as possible, by
        assigning forces among the set of edges and the set of nodes, based on their
        relative positions, and then using these forces either to simulate the motion
        of the edges and nodes or to minimize their energy.
        
        .. figure:: ./doc/source/static/img/force-directed-plot.png
            :width: 600px
            :figclass: align-center
        
            Force-directed plot of all OpenStack resources (cca 3000 resources)
        
        
        Hive Plot
        ---------
        
        The `hive plot` is a visualization method for drawing networks. Nodes
        are mapped to and positioned on radially distributed linear axes — this
        mapping is based on network structural properties. Edges are drawn as curved
        links. Simple and interpretable.
        
        .. figure:: ./doc/source/static/img/hive-plot.png
            :width: 600px
            :figclass: align-center
        
            Hive plot of all OpenStack resources (cca 3000 resources)
        
        
        Arc Diagram
        -----------
        
        An `arc diagram` is a style of graph drawing, in which the vertices of a graph
        are placed along a line in the Euclidean plane, with edges being drawn as
        semicircles in one of the two halfplanes bounded by the line, or as smooth
        curves formed by sequences of semicircles. In some cases, line segments of the
        line itself are also allowed as edges, as long as they connect only vertices
        that are consecutive along the line.
        
        .. figure:: ./doc/source/static/img/arc-diagram.png
            :width: 600px
            :figclass: align-center
        
            Arc diagram of OpenStack project's resources (cca 100 resources)
        
        
        Adjacency Matrix
        ----------------
        
        An adjacency matrix is a square matrix used to represent a finite graph. The
        elements of the matrix indicate whether pairs of vertices are adjacent or not
        in the graph.
        
        .. figure:: ./doc/source/static/img/adjacency-matrix.png
            :width: 600px
            :figclass: align-center
        
            Adjacency matrix of OpenStack project's resources (cca 100 resources)
        
        
        Hierarchical Edge Bundling
        --------------------------
        
        Danny Holten presents an awesome and aesthetically pleasing way of simplifying
        graphs and making tree graphs more accessible.  What makes his project so
        useful, however, is how he outlines the particular thought process that goes
        into making a visualization.
        
        .. figure:: ./doc/source/static/img/hiearchical-edge-bundling.png
            :width: 600px
            :figclass: align-center
        
            Hierarchical edge bundling of SaltStack services (cca 100 resources)
        
        
        Supported Platform Metadata
        ===========================
        
        Following outputs show available resources and relations from given domain.
        
        
        Kubernetes
        ----------
        
        .. code-block:: yaml
        
            kind: kubernetes
            name: test-kubernetes
            relations:
              k8s:deployment-k8s:namespace: 22
              k8s:deployment-k8s:replica_set: 62
              k8s:endpoint-k8s:namespace: 28
              k8s:event-k8s:namespace: 52
              k8s:persistent_volume_claim-k8s:namespace: 1
              k8s:pod-k8s:namespace: 52
              k8s:pod-k8s:node: 52
              k8s:pod-k8s:service: 52
              k8s:replica_set-k8s:namespace: 62
              k8s:replica_set-k8s:pod: 51
              k8s:replication_controller-k8s:namespace: 1
              k8s:secret-k8s:namespace: 1
              k8s:service-k8s:namespace: 30
              k8s:service_account-k8s:namespace: 1
            resources:
              k8s:deployment: 22
              k8s:endpoint: 28
              k8s:event: 52
              k8s:namespace: 4
              k8s:node: 5
              k8s:persistent_volume: 1
              k8s:persistent_volume_claim: 1
              k8s:pod: 52
              k8s:replica_set: 62
              k8s:replication_controller: 1
              k8s:secret: 1
              k8s:service: 30
              k8s:service_account: 1
            timestamp: 1508692477
        
        
        OpenStack
        ---------
        
        .. code-block:: yaml
        
            kind: openstack
            name: test-openstack
            relations:
              os:floating_ip-os:project: 617
              os:hypervisor-os:aggregate: 46
              os:network-os:project: 575
              os:port-os:hypervisor: 3183
              os:port-os:network: 3183
              os:port-os:project: 3183
              os:port-os:server: 3183
              os:router-os:project: 42
              os:server-os:flavor: 676
              os:server-os:hypervisor: 676
              os:server-os:project: 676
              os:stack-os:network: 7
              os:stack-os:port: 17
              os:stack-os:project: 2
              os:stack-os:server: 7
              os:stack-os:subnet: 7
              os:subnet-os:network: 567
              os:subnet-os:project: 567
            resources:
              os:aggregate: 13
              os:flavor: 43
              os:floating_ip: 617
              os:hypervisor: 72
              os:network: 575
              os:port: 3183
              os:resource_type: 169
              os:router: 42
              os:server: 676
              os:stack: 2
              os:subnet: 567
              os:volume: 10
            timestamp: 1508694475
        
        
        SaltStack
        ---------
        
        .. code-block:: yaml
        
            kind: salt
            name: test-salt
            relations:
              salt_job-salt_high_state: 552
              salt_job-salt_minion: 9
              salt_minion-salt_high_state: 689
              salt_service-salt_high_state: 689
              salt_service-salt_minion: 24
              salt_user-salt_job: 7
            resources:
              salt_high_state: 689
              salt_job: 7
              salt_minion: 3
              salt_service: 24
              salt_user: 2
            timestamp: 1508932328
        
Platform: UNKNOWN
Classifier: Intended Audience :: System Administrators
Classifier: License :: OSI Approved :: Apache Software License
Classifier: Operating System :: POSIX :: Linux
Classifier: Programming Language :: Python
Classifier: Programming Language :: Python :: Implementation :: CPython
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.6
