Automatize_Sample_Code

# Automatise Sample Code¶

Sample Code in python notebook to use Automatise as a python library.

[download button]

Observations:

- Scripting not yet compatible with windows, only linux shell

### 1. Paths Configurations¶

You can use configured paths if you want to organize directories

In [1]:

    import sys, os

    root      = os.path.join('automatize', 'assets', 'examples', 'Example')

    # We consider this folder organization to the experimental enviromnent:
    prg_path  = os.path.join(root, 'programs')
    data_path = os.path.join(root, 'data')
    res_path  = os.path.join(root, 'results')

    # OR, you can use the .jar method files in:
    prg_path  = os.path.join('automatize', 'assets', 'method')

### 2. Pre-processing data¶

To use helpers for data pre-processing, import from package `automatise` the module `preprocessing.py`:

In [3]:

    from automatize.preprocessing import *

The preprocessing module provides some functions to work data:

Basic functions:

- `readDataset`: load datasets as pandas DataFrame (from .csv, .zip, or .ts)
- `printFeaturesJSON`: print a default JSON file descriptor for Movelets methods (version 1 or 2)
- `datasetStatistics`: calculates statistics from a datasets dataframe.

Train and Test split functions:

- `trainAndTestSplit`: split dataset (pandas DataFrame) in train / test (70/30% by default)
- `kfold_trainAndTestSplit`: split dataset (pandas DataFrame) in k-fold train / test (80/20% each fold by default)
- `stratify`: extract trajectories from the dataset, creating a subset of the data (to use when smaller datasets are needed)
- `joinTrainAndTest`: joins the train and test files into one DataFrame.

Type convertion functions:

- `convertDataset`: default format conversions. Reads the dataset files and saves in .csv and .zip formats, also do k-fold split if not present
- `zip2df`: converts .zip files and saves to DataFrame
- `zip2csv`: converts .zip files and saves to .csv files
- `df2zip`: converts DataFrame and saves to .zip files
- `zip2arf`: converts .zip and saves to .arf files
- `any2ts`: converts .zip or .csv files and saves to .ts files
- `xes2csv`: reads .xes files and converts to DataFrame

In [10]:

    #cols = ['tid','label','lat','lon','day','hour','poi','category','price','rating']

    df = joinTrainAndTest(data_path, train_file="train.csv", test_file="test.csv", class_col = 'label')
    df.head()

    Joining train and test data from... automatize/assets/examples/Example/data
    Done.
     --------------------------------------------------------------------------------

Out[10]:

| tid | lat_lon | hour | price | poi | weather | day | label
--- | --- | --- | --- | --- | --- | --- | --- | ---
0 | 12 | 0.0 6.2 | 8 | -1 | Home | Clear | Monday | Classs_False
1 | 12 | 0.8 6.2 | 9 | 2 | University | Clouds | Monday | Classs_False
2 | 12 | 3.1 11 | 12 | 2 | Restaurant | Clear | Monday | Classs_False
3 | 12 | 0.8 6.5 | 13 | 2 | University | Clear | Monday | Classs_False
4 | 12 | 0.2 6.2 | 17 | -1 | Home | Rain | Monday | Classs_False

To k-fold split a dataset into train and test:

In [11]:

    k = 3

    train, test = kfold_trainAndTestSplit(data_path, k, df, random_num=1, class_col='label')

    3-fold train and test split in... automatize/assets/examples/Example/data

    Spliting Data:   0%|          | 0/2 [00:00<?, ?it/s]

    Done.
    Writing files ... 1/3

    Writing TRAIN - ZIP|1:   0%|          | 0/7 [00:00<?, ?it/s]

    Writing TEST  - ZIP|1:   0%|          | 0/4 [00:00<?, ?it/s]

    Writing TRAIN / TEST - CSV|1

    Writing TRAIN - MAT|1:   0%|          | 0/7 [00:00<?, ?it/s]

    Writing TEST  - MAT|1:   0%|          | 0/4 [00:00<?, ?it/s]

    Writing files ... 2/3

    Writing TRAIN - ZIP|2:   0%|          | 0/7 [00:00<?, ?it/s]

    Writing TEST  - ZIP|2:   0%|          | 0/4 [00:00<?, ?it/s]

    Writing TRAIN / TEST - CSV|2

    Writing TRAIN - MAT|2:   0%|          | 0/7 [00:00<?, ?it/s]

    Writing TEST  - MAT|2:   0%|          | 0/4 [00:00<?, ?it/s]

    Writing files ... 3/3

    Writing TRAIN - ZIP|3:   0%|          | 0/8 [00:00<?, ?it/s]

    Writing TEST  - ZIP|3:   0%|          | 0/3 [00:00<?, ?it/s]

    Writing TRAIN / TEST - CSV|3

    Writing TRAIN - MAT|3:   0%|          | 0/8 [00:00<?, ?it/s]

    Writing TEST  - MAT|3:   0%|          | 0/3 [00:00<?, ?it/s]

    Done.
     --------------------------------------------------------------------------------

To convert train and test from one available format to other default formats (CSV, ZIP, MAT):

In [4]:

    convertDataset(data_path)

    Writing TRAIN - ZIP|:   0%|          | 0/14 [00:00<?, ?it/s]

    Writing TEST  - ZIP|:   0%|          | 0/14 [00:00<?, ?it/s]

    Writing TRAIN - MAT|:   0%|          | 0/14 [00:00<?, ?it/s]

    Writing TEST  - MAT|:   0%|          | 0/14 [00:00<?, ?it/s]

    All Done.

### 2. Scripting¶

To run feature extraction methods, import from package `automatise` the modules `run.py` or `script.py`:

In [5]:

    from automatize.script import *

The `gensh` function is the statring point to generate scripts for the available methods:

- `method`: method name to generate the scripts;
- `datasets`: dictionary for datasets config, with
  - key: Dataset Name + . + Descriptor JSON file prefix ()
  - value: list of subsets (the second part of JSON descriptor name)
- `params`: dictionary of configuration parameters for scripting (described later)

In [6]:

    method = 'hiper'
    datasets = {'Animals.RawTraj': ['specific']}

    params = {
        'sh_folder': 'scripts',      # where to generate script files
        'folder':    'EXP2022',      # folder prefix for result files
        'k':         5,              # number of folds - optional
        'root':      root,           # root folder of the experimental environment
        'threads':   10,             # number of threads allowed (for movelets methods) - optional
        'gig':       100,            # GB of RAM memory limit allowed (for movelets methods) - optional
        'pyname': 'python3',         # Python command - optional

        'runopts': '-TR 0.5',        # other arguments to pass to the method line (-TR is the τ for HiPerMovelets) - optional
        'timeout': '7d',             # set a timeout to methods runtime (7d limits to 7 days)
    }

    gensh(method, datasets, params)

    sh run-H-Animals-specific-10T.sh

Out[6]:

    'sh run-H-Animals-specific-10T.sh\n'

The available methods in `automatise` are declared here:

In [6]:

    from automatize.helper.script_inc import BASE_METHODS, METHODS_NAMES

    for method in BASE_METHODS:
        print(method, ''.rjust(25-len(method), ' '), METHODS_NAMES[method])

    MARC                       MARC
    npoi                       NPOI-F
    MM                         MASTERMovelets
    MM+Log                     MASTERMovelets-Log
    SM                         SUPERMovelets
    SM-2                       SUPERMovelets-λ
    SM+Log                     SUPERMovelets-Log
    SM-2+Log                   SUPERMovelets-Log-λ
    hiper                      HiPerMovelets
    hiper-pivots               HiPerMovelets-Pivots
    hiper+Log                  HiPerMovelets-Log
    hiper-pivots+Log           HiPerMovelets-Pivots-Log

Alternatively, it is possible to run methods directly from python `automatise` library

In [6]:

    from automatize.run import *

In [7]:

    prefix = 'Example'

    Movelets(data_path, res_path, prefix, 'HL-specific', 'Descriptor_hp',
             version='master', ms=False, Ms=False, prg_path=os.path.join('.'), \
             jar_name='HIPERMovelets', n_threads=1)

    Thu Jun 09 19:42:39 CEST 2022
    Starting Movelets extractor
    Configurations:
       -curpath		Datasets directory:	automatize/assets/examples/Example/data
       -respath		Results directory: 	automatize/assets/examples/Example/results/Example/HL-specific/MASTERMovelets/MASTER_Descriptor_hp_LSP_ED
       -descfile 		Description file : 	automatize/assets/examples/Example/data/Descriptor_hp.json
    +-------------+--------------------+---------------------+----------------------------------------------+
    | Option      | Description        | Value               | Help                                         |
    +-------------+--------------------+---------------------+----------------------------------------------+
    | -nt         | Allowed Threads    | 1                   |                                              |
    | -ms         | Min size           | -1                  | Any positive | -1 | Log: -2                  |
    | -Ms         | Max size           | -1                  | Any | All sizes: -1 | Log: -3 or -4          |
    | -mnf        | Max. Dimensions    | -1                  | Any | Explore dim.: -1 | Log: -2 | Other: -3 |
    | -samples    | Samples            | 1                   |                                              |
    | -sampleSize | Sample Size        | 0.5                 |                                              |
    | -q          | Quality Measure    | LSP                 |                                              |
    | -medium     | Medium             | none                |                                              |
    | -mpt        | Movelets Per Traj. | -1                  | Any | Auto: -1                               |
    | -output     | Output             | discrete (CSV,JSON) |                                              |
    |             |                    |                     |                                              |
    | -version    | Version Impl.      | master              | master, super, hiper[-pivots], random, ultra |
    |             | -- Last Prunning   | false               |                                              |
    +-------------+--------------------+---------------------+----------------------------------------------+

    [1] >> Load Input: 172 milliseconds

    Attributes and Features:
    +---+-------------+---------+-----------------+
    | # | Attribute   | Type    | Comparator      |
    +---+-------------+---------+-----------------+
    | 1 | 2 - lat_lon | space2d | euclidean/-1.0  |
    | 2 | 3 - hour    | numeric | difference/-1.0 |
    | 3 | 4 - poi     | nominal | equals/-1.0     |
    | 4 | 5 - weather | nominal | equals/-1.0     |
    | 5 | 6 - day     | nominal | equals/-1.0     |
    +---+-------------+---------+-----------------+

    Memory Usage (MiB), Memory Total: 245.5. Memory Free: 242.38372039794922. Memory Used: 3.1162796020507812.
    [2] >> Movelet Discovery: [7%] Class: User_1. Trajectory: 1. Trajectory Size: 5. Number of Candidates: 465. Total of Movelets: 4. Max Size: 5. Used Features: 5.
    [2] >> Movelet Discovery: [14%] Class: User_1. Trajectory: 2. Trajectory Size: 5. Number of Candidates: 465. Total of Movelets: 5. Max Size: 5. Used Features: 5.
    [2] >> Movelet Discovery: [21%] Class: User_1. Trajectory: 3. Trajectory Size: 5. Number of Candidates: 465. Total of Movelets: 5. Max Size: 5. Used Features: 5.
    [2] >> Movelet Discovery: [28%] Class: User_1. Trajectory: 4. Trajectory Size: 5. Number of Candidates: 465. Total of Movelets: 5. Max Size: 5. Used Features: 5.
    [2] >> Movelet Discovery: [35%] Class: User_1. Trajectory: 5. Trajectory Size: 5. Number of Candidates: 465. Total of Movelets: 5. Max Size: 5. Used Features: 5.
    [2] >> Movelet Discovery: [42%] Class: User_1. Trajectory: 6. Trajectory Size: 5. Number of Candidates: 465. Total of Movelets: 4. Max Size: 5. Used Features: 5.
    [2] >> Movelet Discovery: [50%] Class: User_1. Trajectory: 7. Trajectory Size: 5. Number of Candidates: 465. Total of Movelets: 5. Max Size: 5. Used Features: 5.
    [2] >> Movelet Discovery: [57%] Class: User_2. Trajectory: 8. Trajectory Size: 5. Number of Candidates: 465. Total of Movelets: 5. Max Size: 5. Used Features: 5.
    [2] >> Movelet Discovery: [64%] Class: User_2. Trajectory: 9. Trajectory Size: 5. Number of Candidates: 465. Total of Movelets: 4. Max Size: 5. Used Features: 5.
    [2] >> Movelet Discovery: [71%] Class: User_2. Trajectory: 10. Trajectory Size: 5. Number of Candidates: 465. Total of Movelets: 5. Max Size: 5. Used Features: 5.
    [2] >> Movelet Discovery: [78%] Class: User_2. Trajectory: 11. Trajectory Size: 5. Number of Candidates: 465. Total of Movelets: 3. Max Size: 5. Used Features: 5.
    [2] >> Movelet Discovery: [85%] Class: User_2. Trajectory: 12. Trajectory Size: 5. Number of Candidates: 465. Total of Movelets: 3. Max Size: 5. Used Features: 5.
    [2] >> Movelet Discovery: [92%] Class: User_2. Trajectory: 13. Trajectory Size: 5. Number of Candidates: 465. Total of Movelets: 5. Max Size: 5. Used Features: 5.
    [2] >> Movelet Discovery: [100%] Class: User_2. Trajectory: 14. Trajectory Size: 5. Number of Candidates: 465. Total of Movelets: 3. Max Size: 5. Used Features: 5.

    [3] >> Processing time: 937 milliseconds
    Thu Jun 09 19:42:41 CEST 2022

    Loading files - train.csv
    Writing train.csv file
    Done.
    Loading files - test.csv
    Writing test.csv file
    Done.

In [4]:

    MARC(data_path, res_path, 'Example', 'MARC-specific', train='train.csv', test='test.csv',
         EMBEDDING_SIZE=100, MERGE_TYPE='concatenate', RNN_CELL='lstm')

    python3 ./automatize/methods/marc/MARC.py "automatize/assets/examples/Example/data/train.csv" "automatize/assets/examples/Example/data/test.csv" "automatize/assets/examples/Example/results/Example/MARC-specific/MARC-specific_results.csv" "MARC-specific" 100 concatenate lstm 2>&1 | tee -a "automatize/assets/examples/Example/results/Example/MARC-specific/MARC-specific.txt"
    Done. 977.3960000000001 milliseconds
    # ---------------------------------------------------------------------------------

In [ ]:

    sequences = [1,2,3]
    features = ['poi']
    prefix = 'Example'

    POIFREQ(data_path, res_path, prefix, '', sequences, features,
            method='npoi', doclass=True)

In [38]:

    prefix = 'Example'

    Movelets(data_path, res_path, prefix, 'MML-specific', 'Descriptor', ms=False, Ms=False, \
             prg_path=os.path.join('.'), jar_name='MASTERMovelets', n_threads=1)

    Movelet Discovery   7% ### /Trajectory: 1. Trajectory Size: 5. Number of Candidates: 15. Total of Movelets: 0. Max Size: 5. Used Features: 5
    Trajectory: 2. Trajectory Size: 5. Number of Candidates: 15. Total of Movelets: 0. Max Size: 5. Used Features: 5
    Trajectory: 3. Trajectory Size: 5. Number of Candidates: 15. Total of Movelets: 0. Max Size: 5. Used Features: 5
    Trajectory: 4. Trajectory Size: 5. Number of Candidates: 15. Total of Movelets: 2. Max Size: 5. Used Features: 5
    Trajectory: 5. Trajectory Size: 5. Number of Candidates: 15. Total of Movelets: 1. Max Size: 5. Used Features: 5
    Trajectory: 6. Trajectory Size: 5. Number of Candidates: 15. Total of Movelets: 0. Max Size: 5. Used Features: 5
    Trajectory: 7. Trajectory Size: 5. Number of Candidates: 15. Total of Movelets: 2. Max Size: 5. Used Features: 5
    Trajectory: 8. Trajectory Size: 5. Number of Candidates: 15. Total of Movelets: 0. Max Size: 5. Used Features: 5
    Trajectory: 9. Trajectory Size: 5. Number of Candidates: 15. Total of Movelets: 0. Max Size: 5. Used Features: 5
    Trajectory: 10. Trajectory Size: 5. Number of Candidates: 15. Total of Movelets: 0. Max Size: 5. Used Features: 5
    Trajectory: 11. Trajectory Size: 5. Number of Candidates: 15. Total of Movelets: 2. Max Size: 5. Used Features: 5
    Trajectory: 12. Trajectory Size: 5. Number of Candidates: 15. Total of Movelets: 3. Max Size: 5. Used Features: 5
    Trajectory: 13. Trajectory Size: 5. Number of Candidates: 15. Total of Movelets: 0. Max Size: 5. Used Features: 5
    Trajectory: 14. Trajectory Size: 5. Number of Candidates: 15. Total of Movelets: 2. Max Size: 5. Used Features: 5

    Movelet Discovery   7% ### /Trajectory: 1. Trajectory Size: 5. Number of Candidates: 15. Total of Movelets: 0. Max Size: 5. Used Features: 5
    Trajectory: 2. Trajectory Size: 5. Number of Candidates: 15. Total of Movelets: 0. Max Size: 5. Used Features: 5
    Trajectory: 3. Trajectory Size: 5. Number of Candidates: 15. Total of Movelets: 0. Max Size: 5. Used Features: 5
    Trajectory: 4. Trajectory Size: 5. Number of Candidates: 15. Total of Movelets: 2. Max Size: 5. Used Features: 5
    Trajectory: 5. Trajectory Size: 5. Number of Candidates: 15. Total of Movelets: 1. Max Size: 5. Used Features: 5
    Trajectory: 6. Trajectory Size: 5. Number of Candidates: 15. Total of Movelets: 0. Max Size: 5. Used Features: 5
    Trajectory: 7. Trajectory Size: 5. Number of Candidates: 15. Total of Movelets: 2. Max Size: 5. Used Features: 5
    Trajectory: 8. Trajectory Size: 5. Number of Candidates: 15. Total of Movelets: 0. Max Size: 5. Used Features: 5
    Trajectory: 9. Trajectory Size: 5. Number of Candidates: 15. Total of Movelets: 0. Max Size: 5. Used Features: 5
    Trajectory: 10. Trajectory Size: 5. Number of Candidates: 15. Total of Movelets: 0. Max Size: 5. Used Features: 5
    Trajectory: 11. Trajectory Size: 5. Number of Candidates: 15. Total of Movelets: 2. Max Size: 5. Used Features: 5
    Trajectory: 12. Trajectory Size: 5. Number of Candidates: 15. Total of Movelets: 3. Max Size: 5. Used Features: 5
    Trajectory: 13. Trajectory Size: 5. Number of Candidates: 15. Total of Movelets: 0. Max Size: 5. Used Features: 5
    Trajectory: 14. Trajectory Size: 5. Number of Candidates: 15. Total of Movelets: 2. Max Size: 5. Used Features: 5
    {MoveletsFound=12, candidates=280, MoveletsDiscoveryTime=76, MoveletsAfterPruning=12}
    Processing time: 496 milliseconds
    Tue Jun 07 13:35:49 CEST 2022

    Loading files - train.csv
    Writing train.csv file
    Done.
    Loading files - test.csv
    Writing test.csv file
    Done.

* The subfolder `scripts` contains auxiliary runnable python files to execute in command line:

In [11]:

    # For example, to merge the result files (need for Movelets methods)
    !"MAT-MergeDatasets.py" $res_path/$prefix/HL-specific

    Loading files - train.csv
    Writing train.csv file
    Done.
    Loading files - test.csv
    Writing test.csv file
    Done.

### 3. Classification¶

To run classifiers for the HIPERMovelets results, import from package `automatise` the script `analysis.py`:

In [2]:

    from automatize.analysis import ACC4All, MLP, RF, SVM

a. To run the classifyers for each folder inside a result path prefix:

In [3]:

    save_results = True
    result_folder = 'HL-specific'
    prefix = 'Example/run1'
    ACC4All(os.path.join(res_path, prefix), result_folder, save_results)

b. To run a specific classifyer:

In [ ]:

    MLP(res_path, prefix, result_folder, save_results)

c. To run the classifyers in shell:

In [ ]:

    !MAT-Classifier-MLP_RF.py $res_path/$prefix $result_folder

### 4. Exporting Results¶

To read the results, import from package `automatise` the module `results.py`:

In [14]:

    from automatize.results import *

a. To check the results (both in python or in shell command line):

In [15]:

    check_run(res_path, True)

            OK:  HL                     ExempleDS         1     specific          [100.000][7s]

a. To check the results (both in python or in shell command line)

In [16]:

    df = results2df(res_path, prefix, result_folder)
    df

    Looking for result files in automatise/assets/examples/Example/results/ExempleDS/run1/**/HL-specific/HL-specific.txt

Out[16]:

| Dataset |  | HL-specific
--- | --- | --- | ---
0 | ExempleDS/run1 | Candidates | 1,890
1 |  | Scored | -
2 |  | Recovered | -
3 |  | Movelets | 9
4 |  | ACC (MLP) | 100.000
5 |  | ACC (RF) | 100.000
6 |  | ACC (SVM) | 100.000
7 |  | Time (Movelets) | 0.035s
8 |  | Time (MLP) | 7s
9 |  | Time (RF) | 0.270s
10 |  | Time (SVM) | -
11 |  | Trajs. Compared | 2
12 |  | Trajs. Pruned | 4

To print the dataframe result in a Latex formatted table:

In [17]:

    printLatex(df)

    \begin{table*}[!ht]
    \centering
    \resizebox{\columnwidth}{!}{
    \begin{tabular}{|c|r||r|}
    \hline
    Dataset &   & HL-specific \\

    \hline
    \hline
    \multirow{13}{2cm}{ExempleDS/run1}
    &     Candidates &     1,890 \\
    &         Scored &         - \\
    &      Recovered &         - \\
    &       Movelets &         9 \\
    &      ACC (MLP) &   100.000 \\
    &       ACC (RF) &   100.000 \\
    &      ACC (SVM) &   100.000 \\
    &Time (Movelets) &    0.035s \\
    &     Time (MLP) &        7s \\
    &      Time (RF) &    0.270s \\
    &     Time (SVM) &         - \\
    &Trajs. Compared &         2 \\
    &  Trajs. Pruned &         4 \\
    \hline
    \end{tabular}}
    \caption{Results for ExempleDS/run1 dataset.}
    \label{tab:results_ExempleDS/run1}
    \end{table*}

To export all results to DataFrame and save:

In [18]:

    df = history(res_path)
    df.to_csv('experimental_results.csv')
    df

Out[18]:

| # | timestamp | dataset | subset | subsubset | run | random | method | classifier | accuracy | runtime | cls_runtime | error | file | total_time | name | key
--- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | ---
0 | 0 | 1.649782e+09 | ExempleDS | specific | specific | 1 | 1 | HL | MLP | 100.0 | 35.0 | 7517.812 | False | automatise/assets/examples/Example/results/Exe... | 7552.812 | HL-specific-MLP | ExempleDS-specific-1

### 5. Visualizing Movelets¶

To read and visualize the resulting movelets, import from package `automatise` the module `movelets.py`:

In [19]:

    from automatise.movelets import *

In [20]:

    prefix = 'ExempleDS/run1'
    movs = read_movelets(os.path.join(res_path, prefix, 'HL-specific'))
    movs

Out[20]:

    [{'lat': '0.8', 'lon': '6.2'},
     {'precip': 10.0, 'price': -1.0},
     {'price': 2.0, 'lat': '4.3', 'lon': '16.9'},
     {'precip': 10.0, 'price': 2.0, 'weather': 'Clear'},
     {'lat': '6', 'lon': '13.1'},
     {'lat': '5.8', 'lon': '16.5'}=>{'lat': '6.3', 'lon': '13'},
     {'lat': '0.4', 'lon': '6.7'},
     {'lat': '3', 'lon': '13.5'},
     {'precip': 15.0, 'weather': 'Clouds', 'poi': 'Shopping'}]

In [21]:

    movelets_sankey(movs, attribute='lat') # or movelets_sankey(movs) -> to display all dimensions (may be confusing)

In [22]:

    movelets_markov(movs, attribute='lat')

Out[22]:
In [24]:

    tree = createTree(movs.copy())

    from anytree import RenderTree
    root_node = convert2anytree(tree)
    root_node = RenderTree(root_node)
    print(root_node)

    Node("/{'lat': '5.8', 'lon': '16.5'}=>{'lat': '6.3', 'lon': '13'} (100.00%) - 0.00")
    ├── Node("/{'lat': '5.8', 'lon': '16.5'}=>{'lat': '6.3', 'lon': '13'} (100.00%) - 0.00/{'lat': '0.8', 'lon': '6.2'} (85.71%) - 0.00")
    ├── Node("/{'lat': '5.8', 'lon': '16.5'}=>{'lat': '6.3', 'lon': '13'} (100.00%) - 0.00/{'precip': 10.0, 'price': -1.0} (85.71%) - 0.00")
    │   └── Node("/{'lat': '5.8', 'lon': '16.5'}=>{'lat': '6.3', 'lon': '13'} (100.00%) - 0.00/{'precip': 10.0, 'price': -1.0} (85.71%) - 0.00/{'precip': 10.0, 'price': 2.0, 'weather': 'Clear'} (85.71%) - 0.40")
    ├── Node("/{'lat': '5.8', 'lon': '16.5'}=>{'lat': '6.3', 'lon': '13'} (100.00%) - 0.00/{'price': 2.0, 'lat': '4.3', 'lon': '16.9'} (85.71%) - 0.00")
    ├── Node("/{'lat': '5.8', 'lon': '16.5'}=>{'lat': '6.3', 'lon': '13'} (100.00%) - 0.00/{'lat': '0.4', 'lon': '6.7'} (85.71%) - 0.00")
    ├── Node("/{'lat': '5.8', 'lon': '16.5'}=>{'lat': '6.3', 'lon': '13'} (100.00%) - 0.00/{'lat': '3', 'lon': '13.5'} (85.71%) - 0.00")
    ├── Node("/{'lat': '5.8', 'lon': '16.5'}=>{'lat': '6.3', 'lon': '13'} (100.00%) - 0.00/{'precip': 15.0, 'weather': 'Clouds', 'poi': 'Shopping'} (85.71%) - 0.00")
    └── Node("/{'lat': '5.8', 'lon': '16.5'}=>{'lat': '6.3', 'lon': '13'} (100.00%) - 0.00/{'lat': '6', 'lon': '13.1'} (80.00%) - 0.00")

In [25]:

    convert2digraph(tree)

Out[25]:

# Find in AutoMATize files:¶

Find any text in automatize code, if you want to change something.

In [1]:

    import os
    import glob2 as glob
    search_path = "automatize/"

    search_str  = "TODO" # This is all the to do's I left in the code for later implementations

    # Repeat for each file in the directory
    files = list(set(glob.glob(os.path.join(search_path, '**', '*.*'), recursive=True)))
    files.sort()
    for fname in files:
        # Open file for reading
        try:
            fo = open(fname)

            # Read the first line from the file
            line = fo.readline()

            # Initialize counter for line number
            line_no = 1
            has = False

            # Loop until EOF
            while line != '' :
                    # Search for string in line
                    index = line.find(search_str)
                    if ( index != -1) :
                        has = True
                        print('\t', "[", line_no, ",", index, "] \t", line[:-1].strip(), sep="")

                    # Read next line
                    line = fo.readline()

                    # Increment line counter
                    line_no += 1
            # Close the files
            fo.close()
            if has:
                print('^', fname, '\n')

        except:
            #print('Skip ->', fname)
            continue

    IOPub data rate exceeded.
    The Jupyter server will temporarily stop sending output
    to the client in order to avoid crashing it.
    To change this limit, set the config variable
    `--ServerApp.iopub_data_rate_limit`.

    Current values:
    ServerApp.iopub_data_rate_limit=1000000.0 (bytes/sec)
    ServerApp.rate_limit_window=3.0 (secs)

    	[14,43] 	sys.path.insert(0, os.path.abspath('.')) # TODO fix imports
    ^ automatize/methods/marc/.ipynb_checkpoints/marc_nn-checkpoint.py

    	[14,43] 	sys.path.insert(0, os.path.abspath('.')) # TODO fix imports
    ^ automatize/methods/marc/marc_nn.py

    	[50,31] 	#     adam = Adam(lr=par_lr) # TODO: check for old versions...
    	[108,42] 	#         adam = Adam(lr=lst_par_lr[k]) # TODO: check for old versions...
    	[310,31] 	#     adam = Adam(lr=par_lr) # TODO: check for old versions...
    ^ automatize/methods/movelet/.ipynb_checkpoints/moveletml-checkpoint.py

    	[50,31] 	#     adam = Adam(lr=par_lr) # TODO: check for old versions...
    	[108,42] 	#         adam = Adam(lr=lst_par_lr[k]) # TODO: check for old versions...
    	[310,31] 	#     adam = Adam(lr=par_lr) # TODO: check for old versions...
    ^ automatize/methods/movelet/moveletml.py

    	[55,6] 	# TODO - replace for pymove package version when implemented
    ^ automatize/methods/rf/.ipynb_checkpoints/randomforrest-checkpoint.py

    	[55,6] 	# TODO - replace for pymove package version when implemented
    ^ automatize/methods/rf/randomforrest.py

    	[101,30] 	#        if method == 'rf': # TODO
    	[109,32] 	#        if method == 'rfhp': # TODO
    ^ automatize/methods/tec/.ipynb_checkpoints/tec-checkpoint.py

    	[17,3] 	## TODO: Under Construction:
    ^ automatize/methods/tec/models/randomforrest.py

    	[17,3] 	## TODO: Under Construction:
    ^ automatize/methods/tec/models/randomforresthp.py

    	[101,30] 	#        if method == 'rf': # TODO
    	[109,32] 	#        if method == 'rfhp': # TODO
    ^ automatize/methods/tec/tec.py

    	[133,77] 	#tids = tids[from_traj: to_traj if len(tids) > to_traj else len(tids)] # TODO
    	[554,61] 	#stats = stats.sort_values('Variance', ascending=False) #TODO
    ^ automatize/movelets.py

    	[42,73] 	#    elif '.mat' in url or os.path.exists(url.replace('.csv', '.mat')): #TODO
    	[525,2] 	# TODO fix stratify:
    ^ automatize/preprocessing.py

    	[446,42] 	n_ds = len(df['Dataset'].unique())-1 #TODO ?
    	[1017,19] 	# OLDER FUNCTIONS: TODO Remake
    	[1201,45] 	elif os.path.exists(file+'-r2df.csv'): # TODO: format
    ^ automatize/results.py

    	[338,45] 	for i in range(1, num_runs+1): # TODO: set a different random number in python
    ^ automatize/run.py

    	[98,2] 	# TODO: Configuring environment
    ^ automatize/scripts/.ipynb_checkpoints/MARC-checkpoint.py

    	[50,34] 	random       = config["random"] # TODO
    ^ automatize/scripts/.ipynb_checkpoints/MAT-MC-checkpoint.py

    	[67,2] 	# TODO:
    ^ automatize/scripts/.ipynb_checkpoints/POIS-checkpoint.py

    	[98,2] 	# TODO: Configuring environment
    ^ automatize/scripts/MARC.py

    	[50,34] 	random       = config["random"] # TODO
    ^ automatize/scripts/MAT-MC.py

    	[67,2] 	# TODO:
    ^ automatize/scripts/POIS.py

    	[19,2] 	# TODO: def_random_seed(random_num=1, seed_num=1)
    ^ automatize/scripts/deprecated/MAT-MC_MLP.py

    	[21,2] 	# TODO: def_random_seed(random_num=1, seed_num=1)
    ^ automatize/scripts/deprecated/MAT-MC_MLP_RF.py

    	[154,42] 	#         adam = Adam(lr=lst_par_lr[k]) # TODO: check for old versions...
    ^ automatize/scripts/deprecated/MAT-MoveletsCLS.py

# By Tarlis Portela (2023)