# Onionprobe

![](assets/logo.jpg "Onionprobe")

Onionprobe is a tool for testing and monitoring the status of
[Tor Onion Services](https://community.torproject.org/onion-services/) sites.

It can run a single time or continuously to probe a set of onion services
endpoints and paths, optionally exporting to [Prometheus](https://prometheus.io).

## Requirements

Onionprobe requires the following software:

* [Python 3](https://www.python.org)
* [Stem Tor Control library](https://stem.torproject.org)
* [Prometheus Python client](https://github.com/prometheus/client_python)
* [PyYAML](https://pyyaml.org)
* [Requests](https://docs.python-requests.org)
* [PySocks](https://github.com/Anorov/PySocks)
* [Tor daemon](https://gitlab.torproject.org/tpo/core/tor)

On [Debian](https://debian.org), they can be installed using

    sudo apt install python3 python3-prometheus-client \
                     python3-stem python3-cryptography \
                     python3-yaml python3-requests     \
                     python3-socks tor

## Installation

Onionprobe is [available on PyPI](https://pypi.org/project/keyring/):

    pip install onionprobe

It's also possible to run it directly from the Git repository:

    git clone https://gitlab.torproject.org/tpo/onion-services/onionprobe
    cd onionprobe

## Usage

Simply ask Onionprobe to try an Onion Service site:

    onionprobe -e http://2gzyxa5ihm7nsggfxnu52rck2vv4rvmdlkiu3zzui5du4xyclen53wid.onion

It's possible to supply multiple addresses:

    onionprobe -e <onion-address1> <onionaddress2> ...

Onionprobe also accepts a configuration file with a list of .onion endpoints
and options. A [detailed sample config](configs/tor.yaml) is provided and can
be invoked with:

    onionprobe -c configs/tor.yaml

By default, Onionprobe starts it's own Tor daemon instance, so the `tor` binary
must be available in the system.

See the [manual
page](https://gitlab.torproject.org/tpo/onion-services/onionprobe/-/blob/main/docs/man/onionprobe.1.md)
for the complete list of options and available metrics.

## Testing

Onionprobe comes with a working test environment with the [sample
configuration](configs/tor.yaml) and based on [Docker
Compose](https://docs.docker.com/compose/), which can be started using

    docker-compose up

Then point your browser to:

* The built-in Prometheus dashboard: https://localhost:9090
* The built-in Onionprobe Prometheus exporter: https://localhost:9935

## Compiled configurations

Besides the [sample config](configs/tor.yaml) containing sites listed at
https://onion.torproject.org, Onionprobe comes also with other example configs:

1. [Real-World Onion Sites](https://github.com/alecmuffett/real-world-onion-sites) .onions at
   [real-world-onion-sites.yaml](configs/real-world-onion-sites.yaml), generated by the
   [real-world-onion-sites.py](packages/real-world-onion-sites.py) script.
2. [The SecureDrop API](https://securedrop.org/api/v1/directory/) .onions at
   [securedrop.yaml](configs/securedrop.yaml), generated by the
   [securedrop.py](packages/securedrop.py) script.

You can build your own configuration compiler by using the
[OnionprobeConfigCompiler](onionprobes/configs.py) class.

## Folder structure and files

Relevant folders and files in this repository:

* `assets`: logos and other stuff.
* `configs`: miscelaneous configurations.
* `containers`: container configurations.
* `debian`: debian packaging.
* `docs`: documentation.
* `packages`: python packages codebase.
* `scripts`: provisioning and other configuration scripts.
* `tests`: test procedures.
* `vendors`: other third-party libraries and helpers.
* `kvmxfile`: please ignore this if you're not a [KVMX](https://kvmx.fluxo.info) user.
* `.env`: should be manually created if you plan to use custom configuration with Docker Composer.
* `docker-compose.yml`: service container configuration.

## Tasks

Check the [issue tracker](https://gitlab.torproject.org/tpo/onion-services/onionprobe/-/issues).

## Acknowledgements

Thanks to:

* @irl for the idea/specs/tasks.
* @hiro for suggestions.
* @arma and @juga for references.
* @anarcat and @georg for Python and Debian packaging guidance and review.

## Alternatives

* [OnionScan](https://onionscan.org/)
* [Webmon](https://webmon.dev.akito.ooo/) has support for Onion Services
  monitoring if used along with [Orbot](https://guardianproject.info/apps/org.torproject.android/).
* [BrassHornCommunications/OnionWatch: A GoLang daemon for notifying Tor Relay and Hidden Service admins of status changes](https://github.com/BrassHornCommunications/OnionWatch)
* [systemli/prometheus-onion-service-exporter: Prometheus Exporter for Tor Onion Services](https://github.com/systemli/prometheus-onion-service-exporter)
* [prometheus/blackbox_exporter: Blackbox prober
  exporter](https://github.com/prometheus/blackbox_exporter), which could be
  configured using `proxy_url` pointing to a [Privoxy](http://www.privoxy.org/)
  instance relaying traffic to `tor` daemon. See [this
  issue](https://github.com/prometheus/blackbox_exporter/issues/264) for details.

## Known issues

From Stem:

* [Python 3.9 warning · Issue #105 · torproject/stem](https://github.com/torproject/stem/issues/105)
* [noisy log: stem: INFO: Error while receiving a control message (SocketClosed): received exception "peek of closed file" · Issue #112 · torproject/stem · GitHub](https://github.com/torproject/stem/issues/112)

## References

Related software and libraries with useful routines:

* [onbasca](https://gitlab.torproject.org/tpo/network-health/onbasca)
* [sbws](https://gitlab.torproject.org/tpo/network-health/sbws)
* [Stem](https://stem.torproject.org/)
* [txtorcon](https://txtorcon.readthedocs.io/en/latest/)
* [Onionbalance](https://onionbalance.readthedocs.io/en/latest/)
* [hs-health](https://gitlab.com/hs-health/hs-health)

Relevant issues:

* [Write a hidden service hsdir health measurer](https://gitlab.torproject.org/tpo/network-health/metrics/analysis/-/issues/13209)
* [Write tool for onion service health assessment](https://gitlab.torproject.org/tpo/core/tor/-/issues/28841)

Research questions:

* [When an onion service lookup has failed at the first k HSDirs we tried, what are the chances it will still succeed?](https://gitlab.torproject.org/tpo/network-health/analysis/-/issues/28)
* [What's the average number of hsdir fetches before we get the hsdesc?](https://gitlab.torproject.org/tpo/core/tor/-/issues/13208)
