Metadata-Version: 2.1
Name: opennem
Version: 0.3.0
Summary: opennem engine agent
Home-page: https://opennem.org.au
License: MIT
Keywords: opennem,analysis,australia
Author: Dylan McConnell
Author-email: dylan.mcconnell@unimelb.edu.au
Requires-Python: >=3.7,<4.0
Classifier: Development Status :: 4 - Beta
Classifier: Environment :: Console
Classifier: Framework :: Scrapy
Classifier: License :: OSI Approved :: MIT License
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.7
Classifier: Programming Language :: Python :: 3.8
Requires-Dist: alembic (>=1.4.2,<2.0.0)
Requires-Dist: click (>=7.1.2,<8.0.0)
Requires-Dist: psycopg2 (>=2.8.5,<3.0.0)
Requires-Dist: pydantic (>=1.5.1,<2.0.0)
Requires-Dist: python-dotenv (>=0.13.0,<0.14.0)
Requires-Dist: pyyaml (>=5.3.1,<6.0.0)
Requires-Dist: requests (>=2.23.0,<3.0.0)
Requires-Dist: scrapy (>=2.1.0,<3.0.0)
Requires-Dist: sentry-sdk (>=0.14.4,<0.15.0)
Requires-Dist: smart_open (>=2.0.0,<3.0.0)
Requires-Dist: sqlalchemy (>=1.3.17,<2.0.0)
Project-URL: Repository, https://github.com/opennem/opennem
Description-Content-Type: text/markdown

# OpenNEM Energy Market Data Access

The OpenNEM project aims to make the wealth of public National Electricity Market (NEM) data more accessible to a wider audience.

This toolkit enables downloading, mirroring and accessing energy data from various networks

Project homepage at https://opennem.org.au

Available on Docker at https://hub.docker.com/r/opennem/opennem

Currently supports:

- Australian NEM: https://www.nemweb.com.au/
- West Australia Energy Market: http://data.wa.aemo.com.au/


## Install

You can install this project with python `pip`:

```sh
$ pip install opennem
```

Or alternatively with docker:

```
$ docker pull opennem/opennem
```

Bundled with sqlite support. Other database drivers are optional and not installed by default. Install a supported database driver:

Postgres:

```sh
$ pip install psycopg2
```

## Usage

List the crawlers

```sh
$ scrapy list
```

Crawl

```sh
$ scrapy crawl au.nem.current.dispatch_scada
```

## Development

Setting up a virtual environment and installing requiements using Poetry:

```sh
$ poetry install
$ source .venv/bin/activate
```

Settings are read from environment variables. Environment variables can be read from a `.env` file in the root of the folder. Setup the environment by copying the `.env.sample` file to `.env`. The defaults in the sample file map to the settings in `docker-compose.yml`

There is a `docker-compose` file that will bring a local database:

```sh
$ docker-compose up -d
```

Bring up the database migrations using alembic:

```sh
$ alembic upgrade head
```

Run scrapy in the root folder for options:

```sh
$ scrapy
```

The `opennem` cli interface provides other options and settings:

```sh
$ opennem -h
```

Settings for Visual Studio Code are stored in `.vscode`. Code is kept formatted and linted using `pylint`, `black` and `isort` with settings defined in `pyproject.toml`

## Architecture overview

This project uses [Scrapy](https://scrapy.org/) to obtain data from supported energy markets and [SQLAlchemy](https://www.sqlalchemy.org/) to store data, and [Alembic](https://alembic.sqlalchemy.org/en/latest/) for database migrations. Database storage has been tested with sqlite, postgres and mysql.

Overview of scrapy architecture:

![](https://docs.scrapy.org/en/latest/_images/scrapy_architecture_02.png)

## Code Navigation

* Spider definitions in `opennem/spiders`
* Processing pipelines for crawls in `opennem/pipelines`
* Database models for supported energy markets are stored in `opennem/db/models`

