# randan
_A python package for gaining social and financial data and their analysis_

_Current version: 1.5.4_

_Documentation: https://randan.readthedocs.io/en/latest/_

_If you want to contribute or report a bug, do not hesitate to [open an issue](https://github.com/LanaLob/randan/issues) on this page or contact us: alexey.n.rotmistrov@gmail.com (Alexey Rotmistrov), lana_lob@mail.ru (Svetlana Zhuchkova)._ 

## Overview
randan is a python package that aims to help social scientists, statisticians and financiers. For the former ones it provides twelve analytical modules that emulate the most popular options presented in SPSS. Unlike the other python packages for data analysis, it has three main features, which make it attractive for social scientists:
1. it provides the results of the analysis in a readable and understandable form, similar to SPSS
2. it provides information about statistical significance of the parameters whenever possible
3. it meets the most popular analytical needs of social scientists; so, the switching among different packages and software stays in the past

As we emphasize the importance of the way that the output looks like, we highly recommend using `randan` in Anaconda or CoLab and store data in `pandas` DataFrames.

A new -- thirteenth and forteenth -- modules provide data from YouTube and VK respectively literally by couple of clicks.

> _**N.B.:** You should understand that this project is under development now, which means it is constantly updating. But you can use all the modules and classes presented in the last release._ 

## Installation
You can easily install the package from the PyPi by running:

```
pip install randan
```
If something goes wrong during the installation, consider using this code:

```
pip install --user randan
```

To upgrade package's version, run this code:
```
pip install --upgrade randan
```

Once you install the package, you can import it as any python package:

```python
# like this
import randan

# or like this
from randan.tree import CHAIDRegressor

# etc.
```

## Structure
### Statistical modules
By now, **twelve statistical** modules have been included in the package. These modules correspond to the SPSS functions as follows:

| Module | Class or function | Corresponding SPSS option | Description |
|--------|-------------------|-----------------------------|-------------|
| descriptive_statistics | NominalStatistics | Analyze -> Descriptive statistics -> Frequencies, Descriptives, Explore | Descriptive statistics relevant for nominal variables |
| descriptive_statistics | OrdinalStatistics | Analyze -> Descriptive statistics -> Frequencies, Descriptives, Explore | Descriptive statistics relevant for ordinal variables |
| descriptive_statistics | ScaleStatistics | Analyze -> Descriptive statistics -> Frequencies, Descriptives, Explore | Descriptive statistics relevant for scale (interval) variables |
| bivariate_association | Crosstab | Analyze -> Descriptive statistics -> Crosstabs | Analysis of contingency tables |
| bivariate_association | Correlation | Analyze -> Correlate -> Bivariate | Correlation coefficients |
| comparison_of_central_tendency | ANOVA | Analyze -> Compare means -> One-Way ANOVA | Analysis of variance |
| clustering | KMeans | Analyze -> Classify -> K-Means Cluster | Cluster analysis with k-means algorithm |
| dimension_reduction | CA | Analyze -> Dimension Reduction -> Correspondence Analysis | Correspondence analysis |
| dimension_reduction | PCA | Analyze -> Dimension Reduction -> Factor (extraction method: principal components) | Principal component analysis |
| regression | LinearRegression | Analyze -> Regression -> Linear | OLS regression |
| regression | BinaryLogisticRegression | Analyze -> Regression -> Binary Logistic | Binary logistic regression |
| tree | CHAIDRegressor, CHAIDClassifier | Analyze -> Classify -> Tree -> CHAID | CHAID decision tree for scale and categorical dependent variables, respectively |

The module **scrapingYouTube** aims to gain data from YouTube by means of seven methods of its API. These methods are: search, playlists & playlistItems, videos, commentThreads & comments, channels. In this module, complicated iterative implementing of these methods allows to maximize output volume. The iterative implementing includes resorting the output and segmenting it by years. The module automatically stores its output in Excel and JSON files logically organized by relevant folders. The module provides a user with a default scenario of gaining YouTube data. This scenario is arranged as a simple dialog interface. Guiding by this interface and pressing Enter on the keyboard, a user has no need for coding. Meanwhile, an advanced user might customize both the scenario and the module's code itself if needs.

The module **scrapingVK** aims to gain data from Russian popular social medium VK by means of its API method news.search. In this module, iterative implementing of this method allows to maximize output volume. The iterative implementing includes segmenting output by years and months. The module automatically stores its output in Excel and JSON files logically organized by relevant folders. The module provides a user with a default scenario of gaining VK data. This scenario is arranged as a simple dialog interface. Guiding by this interface and pressing Enter on the keyboard, a user has no need for coding. Meanwhile, an advanced user might customize both the scenario and the module's code itself if needs.

## Quick start
Although statistical modules of `randan` are built to be similar to SPSS, they reproduces the fit-predict and fit-transform approach, which is now being used in the most popular machine learning python packages. This approach means that you should, firstly, initialize your model and then, secondly, fit it to your data (i.e., use the `fit` function) if necessary. 
> 1. If the method you use belongs to the *unsupervised methods* (i.e., you *do not have* a dependent variable in your data), you can then use `transform` function to get values of the obtained, hidden, dependent variable such as cluster membership, factor scores etc. 
> 2. If the method you use belongs to the *supervised methods* (i.e., you *have* a dependent variable in your data), you can then use `predict` function to get values of the given dependent variable. 
> 3. If the method does not assume to estimate new values for your data (such methods are crosstabs, t-tests etc.), then it does not require to use `fit` and `transform` / `predict` functions. 

If you want to see the full list of the availiable functions associated with some class, please visit our [documentation page](https://randareadthedocs.io/en/latest/) or literally ask for help:
```python
from randan.bivariate_association import Crosstab
help(Crosstab)
```

#### Module `bivariate_association`
This module aggregates methods devoted to searching for statistical relationships between two variables. These methods do not require to use `fit` function, i.e. you only need to call the necessary class:
```python
from randan.bivariate_association import Crosstab

# with this code, you will immediately see the results
ctab = Crosstab(data, row='genre', column='age_ord')

# however, if you want to somehow use separate statistics, you can call them this way
print(ctab.chi_square, ctab.pvalue, ctab.n_cells)
```

#### Module `comparison_of_central_tendency`
This module contains both parametric and non-parametric methods for comparison of central tendency statistics. These methods do not require to use `fit` function, i.e. you only need to call the necessary class:
```python
from randan.comparison_of_central_tendency import ANOVA

# with this code, you will immediately see the results
anv = ANOVA(data, dependent_variables='kinopoisk_rate', independent_variable='genre')

# however, if you want to somehow use separate statistics, you can call them this way
print(anv.F, anv.pvalue, anv.SSt)
```
#### Module `clustering`
This module includes two main clustering methods: k-means and hierarchical (agglomerative) clustering. 

Clustering methods belong to unsupervised learning, which means you should use the `fit` function after calling the appropriate class, and then, if necessary, the `transform` function to acquire cluster membership (and / or distances to each center in case of k-means).
```python
from randan.clustering import KMeans

# with this code, you will immediately see the results, including visualization of clusters
km = KMeans(2).fit(data, ['year', 'time', 'kinopoisk_rate_count'])

# this is how you can predict the cluster membership, 
# and the distances from each observation to each cluster's center
clusters = km.transform(distance_to_centers=True)
```
> If you experience troubles with visualization and see captions like <Figure size 800x500 with 1 Axes> instead of plots, just re-run the code that produces them.
#### Module `dimension_reduction`
This module unites methods for factorization of nominal and scale variables: correspondence analysis (class `CA`) and principal component analysis (class `PCA`). 

Factorization methods belong to unsupervised learning, which means you should use the `fit` function after calling the appropriate class, and then, if necessary, the `transform` function to acquire so-called factor scores.
```python
from randan.dimension_reduction import PCA 
 
vars_ = ['trstprl', 'trstlgl', 'trstplc', 'ppltrst', 'pplfair', 'pplhlp']

# with this code, you will immediately see the results
pca = PCA(n_components=2, rotation='varimax').fit(data, variables=vars_)

# this is how you can predict the factor scores
f_scores = pca.transform()
```
#### Module `regression`
This module consists of two classical regression models: linear regression and binary logistic regression. This group of methods belongs to supervised learning, which means you should use the `fit` function after calling the appropriate class, and then, if necessary, the `predict` function to acquire predictions.
```python
from randan.regression import LinearRegression

# with this code, you will immediately see the results
formula = 'kinopoisk_rate = time + year + genre + genre*type'

regr = LinearRegression().fit(
    data, 
    formula=formula,
    categorical_variables=['genre', 'type'],
    collinearity_statistics=True
)

# this is how you can predict values of the dependent variable for the given data... 
predictions = regr.predict()

# ... save various types of residuals ...
residuals = regr.save_residuals(unstardandized=False, studentized=True)

# ... and even save values of independent variables 
# if you didn't create them manually (e.g. dummies and interactions) ...
indep_vars = regr.save_independent_variables()
```
#### Module `tree`
This module includes various methods of building decision trees. If you have a categorical dependent variable, please use those methods that contain `Classifier` part in their names. Otherwise, if you have a scale dependent variable, please use the methods that contain `Regressor` part in their names.

This group of methods belongs to supervised learning, which means you should use the `fit` function after calling the appropriate class, and then, if necessary, the `predict` function to acquire predictions.
```python
from randan.tree import CHAIDRegressor

# with this code, you will immediately see the results, including the plot of your tree
chaid = CHAIDRegressor().fit(
    data,
    dependent_variable='kinopoisk_rate',
    independent_variables=['genre', 'age_ord', 'year', 'time', 'type', 'kinopoisk_rate_count'],
    scale_variables=['year', 'time', 'kinopoisk_rate_count'],
    ordinal_variables=['age_ord']
    )

# this is how you can predict values of the dependent variable, the node membership, 
# and the description of the node in terms of interactions for the given data 
predictions = chaid.predict(node=True, interaction=True)
```
---

### Module `randanTopic`
This module performs topic modeling on a collection of texts. It takes `DataFrame` with texts and optional metadata, extracts topics, and returns `DataFrame`s with topic loadings, representative tokens, and document assignments. The results will be saved as a facilitating interpretation form and, optionally, returned as `pandas` `DataFramee` for further analysis.

Basic usage (minimal example):

```python
import pandas as pd
from randan.topicModeling.randanTopic import randanTopic

# Assuming you have a DataFrame with a column of lemmatized texts
df = pd.read_csv('my_texts.csv') # texts
matrix_df = pd.read_csv(matrix_df.csv') # bag-of-words

# Run topic modeling with default parameters
randanTopic(df=df,
            matrix_df= matrix_df,
            textFull_lemmatized=' textFull_lemmatized ',
            topicsCount=10)
```

Advanced usage:

You can customize the modeling process by providing additional parameters. The function returns DataFrames with results when `returnDfs=True`.

```python
from randan.topicModeling.randanTopic import randanTopic

dfs = randanTopic(
    df=df,
    matrix_df=matrix_df,                               # precomputed document-term matrix
    loadingsThreshold=0.75,                            # minimum absolute loading to consider a token
    docsLimit=10,                                      # max documents per topic pole for interpretation
    returnDfs=True,                                    # returns `pandas` `DataFrame`s containing fragments of documents (snippets) and their metadata + document–topic scores and auxiliary variables
    rowsNumerator='document_id',                       # column to use as row labels
    supplementarieS=['author', 'date'],                # additional columns to keep in output
    textFull_lemmatized='textFull_lemmatized',         # column with both cleaned, and lemmatized texts without lemmatization and stop word removal
    textFull_simbolsCleaned='textFull_simbolsCleaned', # column with cleaned but not lemmatized texts without lemmatization and stop word removal
    tokensLimit=20,                                    # max tokens per topic for interpretation
    topicsCount=15                                     # number of topics to extract
)

# dfs is a tuple: (docs_snippets_df, scores_df)
docs_snippets, scores = dfs
display(docs_snippets.head())
```

#### Parameters:
- `df` : `pandas.DataFrame` — Input `DataFrame` containing the texts and any auxiliary variables (e.g., metadata). Must include at least the column specified in textFull_lemmatized.
- `matrix_df` : `pandas.DataFrame` — A precomputed document-term matrix (bag-of-words).
- `docsLimit` : `int`, default `5` — Maximum number of documents to show per topic pole. Used for selecting representative documents.
- `loadingsThreshold` : `float`, default `0.5` — Threshold for the average token–topic loading. Tokens with loadings below this value are filtered out when interpreting topics. This indirectly limits the number of tokens shown per topic.
- `returnDfs` : `bool`, default `False` — If `True`, the function returns a tuple of two `DataFrame`s: (1) `docs_snippets_df` — contains fragments of documents (snippets) and their metadata and (2) `scores_df` — contains document–topic scores and auxiliary variables. If `False`, the function returns `None` and only saves the results to disk in the folder where the module performs.
- `rowsNumerator` : `str`, optional — Name of a column in df whose values will be used as row labels (index) in the output `DataFrame`s. If not provided, a default integer index is used.
- `supplementarieS` : list, optional — List of additional column names from df to include in the output `DataFrame`s. These can be used for later interpretation (e.g., document author, date, category).
- `textFull_lemmatized` : `str` — Name of the column in `df` that contains lemmatized texts (stop words not removed). This is used to build the document-term matrix.
- `textFull_simbolsCleaned` : `str` — Name of the column in `df` that contains texts after cleaning (removal of special characters, punctuation, etc.) but without lemmatization and stop word removal. This is used for generating readable snippets.
- `tokensLimit` : `int` — Maximum number of tokens to display per topic pole (positive/negative side). Helps keep interpretations concise.
- `topicsCount` : `int` — The number of topics to extract. This is a key parameter for the topic modeling algorithm.

#### Notes:
- The module automatically saves output files (Excel and JSON) to the folder where the module performs, regardless of the returnDfs setting.
- The document‑term matrix is constructed using the lemmatized texts; make sure that column contains space‑separated tokens.
- If you already have a document‑term matrix (e.g., from another preprocessing step), you can pass it via matrix_df to save computation time.
- The user_words parameter is optional and only effective if the underlying algorithm supports seeded topic modeling.

---

### Module `scrapingVK`
This module wraps the VK API method newsfeed.search. It automatically saves the results to ./output/scrapingVK/ as JSON (raw API response) and Excel (flattened table) files.

Basic usage (Interactive mode):
```python
from randan.scrapingVK import newsFeedSearch

# Launches a step-by-step dialog. Press Enter to proceed with default values.
newsFeedSearch.newsFeedSearch()
```

Advanced usage (Expirienced mode):

You can pass parameters directly to the function. All parameters correspond to the official VK API newsfeed.search method, with the addition of params and returnDfs.
```python
from randan.scrapingVK import newsFeedSearch

# Example 1: Simple search for posts containing "python"
newsFeedSearch.newsFeedSearch(
    q="python",
    count=50,
    returnDfs=False   # only saves files, does not return DataFrame
)

# Example 2: Search with time range and specific fields, returning DataFrame
df = newsFeedSearch.newsFeedSearch(
    access_token="YOUR_TOKEN",
    q="data science",
    start_time=1609459200,      # 2021-01-01 00:00:00 UTC
    end_time=1612137600,         # 2021-02-01 00:00:00 UTC
    fields=["likes", "reposts"],
    count=200,
    returnDfs=True
)

print(df[['text', 'likes', 'date']].head())
```

#### Parameters:
- `access_token` : `str` — VK API access token (service token).
- `q` : `str` — Search query string. Supports VK search syntax (e.g., quotes for exact phrases).
- `count` : `int`, default `200` — Number of posts to return per request. The function automatically handles pagination, so you can effectively retrieve more than 200 posts.
- `start_time` : `int` — Unix timestamp (in seconds). Restricts results to posts published after this time.
- `end_time` : `int` — Unix timestamp (in seconds). Restricts results to posts published before this time.
- `latitude` : `float` — Latitude for location-based search. Must be used together with `longitude`.
- `longitude` : `float` — Longitude for location-based search.
- `fields` : `list` of `str` — Additional fields to include in the response. For example: `["city", "sex"]`. Refer to the official VK documentation for the full list of available fields.
- `params` : `dict` — A pre‑built dictionary of parameters for the VK API request. If this argument is provided, all other individual parameters (like `q`, `count`, `start_time`, etc.) are ignored (except `returnDfs`). This is useful when you want to reuse a complex query configuration.
- `returnDfs` : `bool`, default `False` — If `True`, the function returns a pandas DataFrame containing the posts and their metadata. If `False` (default), it returns `None` and only saves the data to disk.

#### Notes:
- The function automatically handles pagination — you do not need to manage the offset parameter manually.
- All parameters are optional; if none are provided (or if you run the function without arguments), the interactive mode will guide you through the available options.
- When returnDfs=True, the returned DataFrame includes standard VK post fields such as id, date, text, likes, reposts, views, plus any additional fields requested via the fields parameter.
- The module saves output files in the ./output/scrapingVK/ directory. The files are named with a timestamp to avoid overwriting.

Finally, the third way of usage is to take the module's code manually and and alter it

---

### Module `scrapingYouTube`
This module wraps several YouTube Data API v3 methods (search, playlists, videos, comments, channels). It automatically saves results to ./output/scrapingYouTube/ as JSON and Excel files.

Basic usage (Interactive mode):
```python
from randan.scrapingYouTube import searchByText

# Launches a step-by-step dialog. Press Enter to proceed with default values.
searchByText.searchByText()
```

Advanced usage (Expirienced mode):

You can pass parameters directly to the function. Parameters correspond to the YouTube Search.list API parameters.
```python
from randan.scrapingYouTube import searchByText

dfs = searchByText.searchByText(
    q="Python programming",
    publishedAfter="2023-01-01T00:00:00Z",
    videoDuration="long",
    regionCode="US",
    returnDfs=True  # Returns a tuple of 5 DataFrames
)

# Unpack results
search_df, playlists_df, videos_df, comments_df, channels_df = dfs

print(f"Found {len(videos_df)} videos")
```

#### Parameters:
- `access_token` (`str`): Your YouTube Data API key.
- `q` (`str`, optional): Search query string.
- `publishedAfter` (`str`, optional): RFC 3339 formatted timestamp (e.g., `2023-01-01T00:00:00Z`).
- `videoDuration` (`str`, optional): Filter by duration (`any`, `short`, `medium`, `long`).
- `returnDfs` (`bool`, default=False): If `True`, returns a tuple of five pandas DataFrames: (search, playlists, videos, comments, channels). If `False`, returns `None` (data is only saved to disk).
- ... (and all other standard YouTube API parameters)

#### Notes:
- The function automatically handles pagination — you do not need to manage the offset parameter manually.
- All parameters are optional; if none are provided (or if you run the function without arguments), the interactive mode will guide you through the available options.
- When returnDfs=True, the returned DataFrame includes standard VK post fields such as id, date, text, likes, reposts, views, plus any additional fields requested via the fields parameter.
- The module saves output files in the ./output/scrapingVK/ directory. The files are named with a timestamp to avoid overwriting.
- The parameter `channelIdForSearch` is used to restrict the search to a specific channel (or a specific playlist).

Finally, the third way of usage is to take the module's code manually and and alter it
