# Soup Stars :stew: :star: :boom:

[![Build Status](https://travis-ci.org/soupstars/client.svg?branch=master)](https://travis-ci.org/soupstars/client)
<!-- [![Coverage Status](https://coveralls.io/repos/github/tjwaterman99/soupstars/badge.svg?branch=master)](https://coveralls.io/github/tjwaterman99/soupstars?branch=master) -->
<!-- [![Docs](https://readthedocs.org/projects/soupstars/badge/?version=latest)](https://soupstars.readthedocs.io/en/latest/?badge=latest) -->
[![Version](https://badge.fury.io/py/soupstars.svg)](https://badge.fury.io/py/soupstars)
[![Python](https://img.shields.io/pypi/pyversions/soupstars.svg)](https://pypi.org/project/soupstars/)

Soup Stars is a framework for building web parsers with Python. It is designed to make building, deploying, and scheduling web parsers easier by simplifying what you need to get started.

## Quickstart

### Installation

Install it with pip.

```
pip install soupstars
```

The client is also available as a docker image.

```
docker pull soupstars/client
```

### Building a parser

Create a new parser using the `soupstars` command. The `create` command will use a template parser.

```
soupstars create -m myparser.py
```

Parsers are simple python modules.

```
cat myparser.py
```

Notice that the only set up required is the special `parse` decorator and a variable named `url` for the web page you want to parse.

```python
from soupstars import parse

url = "https://corbettanalytics.com/"

@parse
def h1(soup):
    return soup.h1.text
```

You can test that the parser functions correctly.

```
soupstars test -m myparser.py
```

The output is a json object.

```json
{
  "data": {
    "h1": "Level up your analytics"
  },
  "errors": {},
  "status": 200,
  "url": "https://corbettanalytics.com/"
}
```

Use `soupstars --help` to see a full list of available commands.

```
Usage: soupstars [OPTIONS] COMMAND [ARGS]...

  CLI to interact with SoupStars cloud.

Options:
  --help  Show this message and exit.

Commands:
  config    Print the configuration used by the client
  create    Create a new parser from a template
  debug     Open a python prompt with a parser result
  health    Print the status of the SoupStars api
  login     Log in with an existing email
  ls        Show the parsers uploaded to SoupStars cloud
  pull      Pull a parser from SoupStars cloud into a local module
  push      Push a parser to SoupStars cloud
  register  Register a new account on SoupStars cloud
  results   Print results of a parser
  run       Run a parser on SoupStars cloud
  show      Show the contents of a parser on SoupStars cloud
  test      Test running a parser locally
  whoami    Print the email address of the current user
```

### Deploying to soupstars.cloud

You can deploy your parsers to be ran on our service.

Use the CLI to create an account. You'll be prompted for a username and password.

```
soupstars register
```

Upload your parser.

```
soupstars push -m myparser.py
```

You can now run the parser from our service.

```
soupstars run -m myparser.py
```

The output is a json object.

```json
{
  "data": {
    "data": {
      "h1": "Level up your analytics"
    },
    "errors": {},
    "status": 200,
    "url": "https://corbettanalytics.com/"
  },
  "id": "2bd746f6-ae14-4057-8af8-f92aa5d304ca",
  "parser_id": "cb25aa3b-375d-4d55-966b-99cfef6e4015",
  "status_code": 200,
  "user_id": 1
}
```

## Development

Create a virtual environment with python3.6

```
virtualenv venv --python=python3.6
```

Install the package in development mode.

```
venv/bin/pip3 install --requirement requirements.txt
venv/bin/pip3 install --editable .
```

Run the tests.

```
venv/bin/pytest -v
venv/bin/flake soupstars examples
```

## Releasing

New tags that pass on CI will automatically be pushed to PyPI and docker hub.
