Requirements notes:
==================
arrow: will manage all datetime functionality
requests: will be used to fetch data from rest urls.


Expected format:
==============

Usable by reports scripts:

        {'client_name':
            { 'b_last'    : 'iso8601',
              'b_state'    : 'working/current',
              'b_phase' : 'phase1/phase2',
              'b_date' : 'date',
              'b_time' : 'time',
              'server' : ['server',  'list', 'in multi-agent mode']
            }
        }

*b_last:* will use http://crsmithdev.com/arrow/#arrow.arrow.Arrow.isoformat - As described in https://git.ziirish.me/ziirish/burp-ui/issues/146#note_1306
*b_date and b_time* will be converted to local time always. (It simplifies the txt module)
*server:* Is additionally added when found from api backend, it appears when the burpui is in multi-agent mode.

#### Generated by clients_reports.report_outdated:

    {'client_name':
        { --- ommited values ---,
          'b_status: 'outdated'
        }
    }

#### Required for --detail:

    {'client_name':
        { --- ommited values ---,
          'backup_report': {
              'duration': seconds
              'totsize': bytes
              'received': bytes
          }
        }
    }

#### More information that will be processed later:

```
    {'client_name':
            { 'b_number'  : 'number',
              'b_date'    : 'date',
              'b_status'  : 'outdated/ok',
              # 'b_type'    : 'finishing/working/current',
              'quota'   : 'ok/soft/hard',
              'warnings': 'warnings',
              'exclude' : 'yes/no',
              'b_phase' : 'phase1/phase2',
              'b_phase_date' : 'date',
              'b_phase_status' : 'outdated/ok',
              'b_curr_taken' : 'seconds',
              'backup_stats' : {'contents': 'all contents of backup_stats file for the client', some example:
                                'time_taken': 'seconds', 'bytes_in_backup': 'bytes', bytes_received: 'bytes'}
            }
    }
```

Testing with burpui demo server
==============================

    burp-reports --debug --burpui_apiurl https://admin:admin@demo.burp-ui.org/api/ --report outdated --detail
    burp-reports --debug --burpui_apiurl https://admin:admin@demo.burp-ui.org/api/ --detail
    burp-reports --burpui_apiurl https://admin:admin@demo.burp-ui.org/api/

Or with conf:

    burp-reports -c ~/burp-reports.conf --debug --burpui_apiurl https://admin:admin@demo.burp-ui.org/api/ --report outdated --detail


Extra resources
===============

https://www.airpair.com/python/posts/top-mistakes-python-big-data-analytics


Multiple sources
================

How to rethink about connecting burp from multiple data sources? 

Case:
-----

Take data from different ways: 

* from burpui server
* from burpui agent
* from burp -a m
* expose its own web rest service when not having burpui? 

Other notes:
------------

How to manage the possibility to connect to different source?: 

* from burpui
* from burp itself.
* from burpui_agent.

* Create one function to connect to all of sources
  * Implement inside the function the logic to know what source is it. 
  * Connect to one of the sources of data.
  * Transform/Translate into right python dict so burp_reports will read it. Or use same format as burpui?
  * Reuse same function to be able to expose as rest api. 

use python [hug](http://www.hug.rest/) module. 

* Then continue extending the burp_reports with pandas. (but optional, not as main requirements so it will be 
small for those that doesn't want these features, should create new package?)  
* Generate more features from pandas, like graphs, more exports of data, etc.
   * Pandas will be used only from webui? 
* Improve txt.py, use jinja2 templates?  
* Then release burp_reports2?

 Important note: each new line should be py.tested
 
 https://sarahleejane.github.io/learning/python/2015/08/09/simple-tables-in-webapps-using-flask-and-pandas-with-python.html
  
 Change logging options (done): 
 https://docs.python.org/3/howto/logging.html#logging-from-multiple-modules
 