# Ripandtear 

### An asynchronous file archival program

The intention of Ripandtear is to make it easy to save/update content uploaded by content creators online. 
What makes Ripandtear unique is that it stores all information on a user in a .rat file to help condense all the information about a content creator into one location
(don't worry, a .rat is just a .json file with a different extension name). 
From what usernames they use across different websites, to already downloaded URLs and even file names with their MD5 hash to remove duplicates, it's all stored in the .rat for easy convience when adding new information from the command line. 
You can store all the information about a content creator, archive their content, remove duplicates and sort the files all with one command!

By using the .rat file you eliminate the need for redownloading the same content and creating duplicate files.
All the previously downloaded Url's are tracked in the .rat file.
If the file has already been downloaded it is skipped to save you time/data/bandwith and make fewer requests to servers.
This also makes it convient if you share a folder that has a .rat file with someone else.
They can pick up where you left off without having to download the same content all over again!

## Installation

heree

## Current Supported Sites

**Bunkr.su**

**Cyberdrop.me**

**Gfycat.com** (only direct links and image pages. Not profiles)

**Gofile.io** (can take passwords if seperated by a `~`. `https://gofile.io/d/Qjakte~yiAjdwl`)

**Imgur.com**

**Jpg.fish**

**Reddit.com**

**Redgifs.com**

### Store Names

Tell Ripandtear to save names to the below categories with the specific flags. 
If you want to print the names to the screen put a 'p' in front of the flag. 
If syncing is availible add a 's' in front of the flag to download all new content from the site (you cannot print and sync in the same flag: -psr)

reddit - `-r`, `-pr` (to print), `-sr` (to sync)

redgifs - `-R`, `-pR` (to print), `-sR` (to sync)

onlyfans - `-o`, `-po` (to print)

fansly - `-f`, `-pf` (to print)

pornhub - `-p`, `-pp` (to print)

twitter - `-t`, `-pt` (to print)

instagram - `-i`, `-pi` (to print)

youtube - `-y`, `-py` (to print)

tiktok - `-T`, `-pT` (to print)

twitch - `-twitch`, `-ptwitch` (to print)

patreon - `-P`, `-pP` (to print)

tumblr - `-tum`, `-ptum` (to print)

myfreecams - `-mfc`, `-pmfc` (to print)

chaturbate - `-cb`, `-pcb` (to print)

### Store Links

Store links to the websites below with the following flags

coomer.party - `-c`, `-pc` (to print)

simpcity - `-s`, `-ps` (to print)

### Save Urls

Save urls to download and already downloaded urls with the following flags

add url to download - `-u`, `-pu` (to print)

add already downloaded url - `-U`, `-pU` (to print)

## Examples

### Download a link

Ripandtear has extractors to download content from many different content providers (see ***Supported Sites***). The simplest use case is to download content from a supported url


`ripandtear -d 'https://www.supportedsite.com/<content-id>,https://www.differentsite.io/<content-id>.jpg'`

`ripandtear` - the name of the program

`-d` - `-d` stands for download. If Ripandtear recognizes the url it will download all the content it can find into the current directory. You can download multiple links at once by seperating them with commas (,) or even adding multiple `-d` flags in the same command

### Creating a new user

Make a new directory and move into it:

`mkdir 'Username' && cd 'Username'`

Run Ripandtear with the following flags:

`ripandtear -r 'random_reddit_username' -R 'redgifs_username' -sr -sR -H -S`

With those two commands you have created a new directory, recorded a reddit and redgifs username in a .rat file, downloaded all the content from each website, found the hashes of the files to remove duplicates and sorted the files into distinct directories based on their type.

Here is what the result of the command:


    Username (<- the directory)
    ├── Username.rat
    ├── pics
    │   └── reddit-2022-01-01-example_pic.png
    └── vids
        └── redgifs-2023-04-12-example_vid.mp4


Inside the directory you created you now have a .rat file with the same name as the directory (the .rat must match have the same name as the directory. Don't worry it is created automatically).
The .rat now contains the usernames you set for reddit and redgifs.
Ripandtear has also downloaded all the content off of reddit and redgifs, hashed the files to remove duplicates, sorted the files into folders respective of their content type and recorded all information into the .rat

Lets walk through the previous command to explain what each flag does:

`ripandtear` - the name of the program

`-r 'random_reddit_username'` - the `-r` flag adds the reddit username to the .rat file.

`-R 'redgifs_username'` - the `-R` flag adds the redgifs username to the .rat file.

`-sr` - `-sr` stands for "Sync Reddit". Running this command will look up all the Reddit names saved in the .rat file, then download all content that has not been downloaded yet from the users Reddit profile

`-sR` - `-sR` stands for "Sync Redgifs". Running this command will look up all the Redgifs names saved in the .rat file, then download all content that has not been downloaded yet from the users Redgifs profile

`-H` - `-H` stands for "Hash files". This command hashes all the files in the current directory and removes duplicates. 
If it is run in the same directory with a .rat file it looks in the .rat file to see if newly downloaded content matches older downloaded content. 
If there is a match Ripandtear deletes the file with the shorter filename and keeps the file with the longer username

`-S` - `-S` stand for "Sort files". It sorts the files in the current directory into either pics, vids, audio and text directories depending on the file type. 
If a file does not fit in any of those folders it is kept in the current directory

### Adding another username

You find out that the same content creator uses multiple different reddit names. 
You want to add the new name you found.
Doing that is extreamly easy. 
All you have to do is run the following command:

`ripandtear -r 'new_example_reddit_username,plus_another_username' -r 'even_another_name'`

Ripandtear looks in the current directory for a .rat file. If a .rat doesn't exist is creates one (naming itself after the current directory) and adds the names. 
If it finds a .rat it adds the new username(s) to the already existing .rat file. 
You can even add multiple names at once by separting them with commas (,) and/or multiple identical flags. 
Don't worry about accidentally adding the same name multiple times. 
Ripandtear makes sure that each name stored in the .rat is unique and that there are no duplicates. 
This approach applies for all username categories. See ***stored names*** to see which usernames you can store and the flags to use to set them.

### Adding a website url to download later

Ripandtear has the ability to download content from many different hosting sites. 
See the ***Supported Sites*** section for compatible websites. 
If you want to add a url to the .rat file to be downloaded later you can do it with the following command.

`ripandtear -u 'https://www.examplesite.com/<content-id>.jpg'`

`-u` - `-u` adds a url to be downloaded later. 
It stores raw url's so any url can be stored. 
Saved urls can be downloaded with Ripandtear (if Ripandtear supports the link) with `-su` (sync urls). 
Ripandtear looks at all the stored urls in the .rat, If the url matches an extractor Ripandtear has, it will attempt to download all content.
If the url doesn't match, it will be kept in the .rat for later, either to help you archive links you want to manually download later, or for when Ripandtear adds an extractor to download the content. 
Feel free to add whatever url's you want!


### Syncing errors

Sometimes when downloading, things go wrong.
A server could be down, you could be making so many requests that you were blocked, maybe the content is temporarily unavailible.
Ripandtear keeps track of all urls and information relating to the url if an error occures when attempting a download and saves it for later.
If you want to reattempt failed downloads you can run the command:

`ripandtear -se`

`-se` - `-se` stands for "Sync Errors". Ripandtear will look for any saved errors and attempt them again. If the url works, the error will be removed.
If there is another error Ripandtear will continue to save the url to attempt later.
After 5 attempts Ripandtear assumes getting the content will not be possible to download and move the bad url into the downloaded urls category within the .rat to prevent more attempts in the future.
The errors saved in the .rat can be cleared with the `-ee` (erase errors) and printed with `-pe` (print errors)

### Sync All

If you want to update a user and get all of their new content you can run the following command

`ripandtear -sa -HS` 

`-sa` - `-sa` stand for "Sync All". It syncs all supported sites (see ***supported names***) and stored urls (`-su`). It does not sync errors though (`-se`)

`-SH` - This is a combination of `-S` and `-H`. The order of `-S` and `-H` does not matter when combining them

### Print Urls

Maybe you want to see all availible url's Ripandtear can find. 
Maybe you want to take those url's and save them to a file to be used with a different downloader.
Whatever you want to do, if you want to print all files Ripandtear can find to the screen from a supported site without downloading them, you can run the following command

`ripandtear -g 'https://www.url.com/'`

`-g` - `-g` stands for "Get urls". It prints all the found urls to the screen instead of downloading them. 
On a OS like Linux you can then pipe the output into another command or file

### Add Already Downloaded Urls

If you are coming from another downloader, or have a list of already downloaded urls, you can add them to the .rat with the following command so Ripandtear will skip downloading the url again

`ripandtear -U 'https://url.com/content.jpg'`

`-U` - `-U` adds the url to the already downloaded urls section of the .rat file. This is the information Ripandtear checks before preceding with a download.
If a url to be downloaded is found within this category, it is skipped.
If you are a more advanced Linux user you could run a command like this to move all the urls from a text file into the .rat

`cat location/to/urls.txt | xargs -i ripandtear -U {}`

Just remember that Ripandtear works off the current directory it is in.
Make sure you are located where you want the .rat file to be saved/updated

### Logging to the screen

`ripandtear -l 2 -SH`

By default Ripandtear trys to show as little information as possible to keep the downloading experience very minimal.
Sometimes you might want more information to make sure the program is working, or incase you need to post information online to help with trouble shooting.
You can do this by adding the `-l 2` flag to the command you are executing. 
There are five levels (1-5).
2 is the recommended level for just tracking that Ripandtear is working and see what is happening.
1 is for debugging and will print a LOT of information that won't be useful if you are doing multiple downloads at once, but exists for extreme situations where you need help trouble shooting a single url

# FAQ

> What do the url colors mean?

Ripandtear uses colors to minimize the amount of information to the screen.
The color lets you know what has happend with each url

- Green: The file is downloading/downloaded

- Blue: The file has already been downloaded and the url is stored in the .rat (it was skipped)

- Cyan: No .rat file was found, but a file with the same name and file size was located in the current directory.
The file was skipped

- Red: There was a problem with the url. The majority of the time it means the file is a 404 (not found), deleted, or the url does not match an extractor for Ripandtear.
If you are curious what the specific case is you can run a command like `ripandtear -l 2 -d 'suspect_url'` to log and see what the exact problem is.
Any urls that should normally be downloaded, but encountered an error, are saved in the errors section within the .rat.
They can be attempted again with `-se` (sync errors) or printed out with `-pe` (print errors).
Look in the example section for more information



> Ripandtear seems to be hanging/has long pauses while downloading. Has it frozen?

Ripandtear tries to keep the information printed to the screen as minimal as possible.
While Ripandtear is trying to find all the downloadable links, it won't print anything to the screen.
If Ripandtear is finding a lot of content this is probably the reason for it to "hang" or seem frozen.
In most cases it is just doing a lot of work in the background so just let it run and it should be fine.
If you are really worried that something might be wrong, run it with the `-l 2` flag.
This will print logging information to the screen and you can get more information about what is going on behind the screen.
Some websites have specific quirks so if you notice it having more problems more often on specific sites check out the ***Website Quirks*** section for more potential causes from that site

> WTF?! I ran Ripandtear, it paused and then just shut down without downloading or printing anything

If you are syncing a users profile Ripandtear only downloads content that has not been downloaded yet.
If the user has not uploaded anything new between the last time you did a sync, then Ripandtear won't download anything.
It not printing anything to the screen just means it didn't find any new content

# Website Quirks

Every website is a unique and beautiful website. Because of this they might have their own little oddities and special behavior.
Here are some potential symptoms you might notice and what is going on behind the scene

### Bunkr
Bunkr file format:

`<content_name>.<extension>`

Bunkr is a special use case with Ripandtear. 
They use a lot of javascript and are constantly changing the layout/backend information. 
Ripandtear's goal is to focus on accuracy and making sure every availible url is found, instead of trying to guess what the download url is, like other downloaders do (not intended as a diss). 
Ripandtear uses Playwright to find all the content located from Bunkr urls. 
Ripandtear also uses Playwright to download files and videos from the appropriate pages. 
Because of this Ripandtear is unable to show a progress bar like it does with other websites.
If you are downloading a video/file from bunkr you will first see the filename that is being downloaded printed in white.
When the same filename is printed again, but in green it means the file was successfully downloaded.
Bunkr images and direct links to video files (that beginning with 'media-files') will be downloaded with a progress bar.

If downloading extreamly large videos you might see a warning pop up about not enough event listeners. 
This is a warning from Playwright. 
In my experience I have yet to have a problem with the memory leak it warns you about so it is most likely a not issue (again in my experience)

Also to be kinder on Bunkr's servers, Ripandtear's Bunkr extractor incorporates random delays.
Because Ripandtear is an asynchronous downloader it can easily slam Bunkrs servers with too many requests at the same time. 
Instead Ripandtear limits downloads to 4 videos at a time (which are taken from a queue. Don't think you can only give it 4 videos at a time or that it won't work if an album has more than 4 videos) and waits at random intervals to prevent 429 error codes (Too Many Requests).
Bunkr videos will probably download slower than expected, but should be more consistent.
Currently if Ripandtear is going to have a problem with a site, it is going to be Bunkr.
With that being said it is still very useable

### Cyberdrop
Cyberdrop file format:

`<content_name>.<extension>`

### Gfycat
Gfycat file format:

`gfycat-<year>-<month>-<day>-<gfycat-id>-<reddit post tile: if sent from reddit>.<extension>`

### Gofile
Gofile file format:

`<content_name>.<extension>`

You can add gofile urls with passwords via the `-d` or `-u` flags if you separte the url and password with a `~`

Example: `https://gofile.io/d/Qjakte~yiAjdwl`

### Imgur
Imgur file format:

`imgur-<year>-<month>-<day>-<album hash: if applicable>-<order: if applicable>-<image hash>-<reddit post title: if sent from reddit>.<extension>`

### Jpg.fish|church
Jpg file format:

`<content_id>.<extension>`

### Reddit

Reddit file format: 

`reddit-<year>-<month>-<day>-<gallery name: if applicable>-<order: if applicable>-<reddit post id>-<post title>.<extension>`

Reddit asks that anybody using their API to download content follows certain rules they have. 
One of the rules is limiting the amount of downloads a user makes in a five minute period (they limit it to 300 requests).
Because Ripandtear is an asynchronous downloader it can easily hit that 300 requests limit in just a minute or two.
To be a respectful downloader (and prevent your IP address and Ripandtear the program from being banned) Ripandtear follows their API rules and times itself out if it hits that requests limit in the 5 minute window.
This means there might be times when downloading a Reddit users profile where Ripandtear will pause until it hits the next 5 minute increment.
For example at 12:02 PM Ripandtear pauses while downloading and "stops working". 
It should start back up again at 12:05 PM when it hits the next 5 minute window.
In some extreme cases it might need to wait 10 minutes.
If Ripandtear has paused for more than 10 minutes then something might be wrong. 
The only time this will really be a problem is if you are doing an initial download of a users profile and they have a lot of content.
After the first download Ripandtear compares all availible links with what has been already downloaded in the .rat and only downloads the new content.
If you are syncing reddit profiles on a regular basis then you shouldn't hit the API limit often (unless the user posts a LOT)

The Reddit extractor also uses yt-dlp to download Reddit videos

### Redgifs
Redgifs file format:

`redgifs-<year>-<month>-<day>-<redgifs_id>-<reddit post title: if passed from reddit>.<extension>`
