This file is an overview of availabe sources along with descriptions and some guidance.

* Extra dependencies
Some sources require additional dependencies, which you can install if needed.

#+begin_src python :dir .. :exports output :results output drawer
print('\n') # fix github org-mode issue with drawers

import setup
for (name, description), _ in setup.DEPS_SOURCES.items():
    # fuck org-ruby. promnesia[name] should be in quotes, but then it doesn't render as code. ugh.
    # https://github.com/wallyqs/org-ruby/issues/45
    print(f"- ~pip3 install --user promnesia[{name}]~")
    print(f'   ')
    print(f'   {description}')
#+end_src

#+RESULTS:
:results:


- ~pip3 install --user promnesia[optional]~

   dependencies that bring some bells & whistles
- ~pip3 install --user promnesia[HPI]~

   dependencies for [[https://github.com/karlicoss/HPI][HPI]]
- ~pip3 install --user promnesia[html]~

   dependencies for sources.html
- ~pip3 install --user promnesia[markdown]~

   dependencies for sources.markdown
- ~pip3 install --user promnesia[org]~

   dependencies for sources.org
- ~pip3 install --user promnesia[telegram]~

   dependencies for sources.telegram
:end:

Alternatively, you can just install all of them in bulk: ~pip3 install --user promnesia[all]~.


* Sources

These are included with the current Promnesia distribution:

#+begin_src python :python "with_my python3" :dir ../src :exports output :results output drawer
print('\n') # fix github org-mode issue with drawers

from pathlib import Path
import pkgutil
import importlib
import inspect

indent = lambda s: ''.join('  ' + l for l in s.splitlines(keepends=True))

git_root = Path('.').absolute().parent

from promnesia.common import Results

import promnesia.sources as pkg
for importer, name, ispkg in sorted(pkgutil.walk_packages(
        path=pkg.__path__,
        prefix=pkg.__name__+'.'
), key=lambda x: x[1]):
    if name in {
            # TODO damn, these modules need depednencies...
            'promnesia.sources.browser',
            'promnesia.sources.markdown',
            'promnesia.sources.org',
            'promnesia.sources.plaintext',
    }:
        continue
    m = importlib.import_module(name)
    public = [(k, v) for k, v in inspect.getmembers(m) if not k.startswith('_')]
    indexers = [(k, v) for k, v in public if getattr(v, '__annotations__', {}).get('return') == Results]
    assert len(indexers) > 0, name
    for k, i in indexers:
        # print(inspect.signature(i))
        link = '../' + str(Path(m.__file__).relative_to(git_root))
        print(f'- [[file:{link}][{name}]]')
        d = m.__doc__
        if d is not None:
            print(indent(d))
#+end_src

#+RESULTS:
:results:


- [[file:../src/promnesia/sources/auto.py][promnesia.sources.auto]]

  - discovers files recursively
  - guesses the format (orgmode/markdown/json/etc) by the extension/MIME type

- [[file:../src/promnesia/sources/fbmessenger.py][promnesia.sources.fbmessenger]]

  Uses [[https://github.com/karlicoss/HPI][HPI]] for the messages data.

- [[file:../src/promnesia/sources/guess.py][promnesia.sources.guess]]
- [[file:../src/promnesia/sources/html.py][promnesia.sources.html]]

  Extracts links from HTML files

- [[file:../src/promnesia/sources/hypothesis.py][promnesia.sources.hypothesis]]

  Uses HPI [[https://github.com/karlicoss/HPI/blob/master/doc/MODULES.org#myhypothesis][hypothesis]] module

- [[file:../src/promnesia/sources/instapaper.py][promnesia.sources.instapaper]]

  Uses HPI [[https://github.com/karlicoss/HPI/blob/master/doc/MODULES.org#myinstapaper][instapaper]] module

- [[file:../src/promnesia/sources/pocket.py][promnesia.sources.pocket]]

  Uses [[https://github.com/karlicoss/HPI][HPI]] for Pocket highlights & bookmarks

- [[file:../src/promnesia/sources/reddit.py][promnesia.sources.reddit]]

  Uses HPI [[https://github.com/karlicoss/HPI/blob/master/doc/MODULES.org#myreddit][reddit]] module

- [[file:../src/promnesia/sources/roamresearch.py][promnesia.sources.roamresearch]]

  Uses [[https://github.com/karlicoss/HPI][HPI]] for Roam Research data

- [[file:../src/promnesia/sources/rss.py][promnesia.sources.rss]]
- [[file:../src/promnesia/sources/shellcmd.py][promnesia.sources.shellcmd]]
- [[file:../src/promnesia/sources/takeout.py][promnesia.sources.takeout]]

  Uses HPI [[https://github.com/karlicoss/HPI/blob/master/doc/MODULES.org#mygoogletakeoutpaths][google.takeout]] module

- [[file:../src/promnesia/sources/telegram.py][promnesia.sources.telegram]]

  Uses [[https://github.com/fabianonline/telegram_backup#readme][telegram_backup]] database for messages data

- [[file:../src/promnesia/sources/twitter.py][promnesia.sources.twitter]]

  Uses [[https://github.com/karlicoss/HPI][HPI]] for Twitter data.

- [[file:../src/promnesia/sources/vcs.py][promnesia.sources.vcs]]

  Clones & indexes Git repositories (via sources.auto)

- [[file:../src/promnesia/sources/website.py][promnesia.sources.website]]

  Clones a website with wget and indexes via sources.auto

:end:


* Extending
Adding your own data sources is very easy!
One day I'll add some proper documentation, but the easiest at the moment is to use existing simple modules as a reference, for example:

- [[https://github.com/karlicoss/promnesia/blob/master/src/promnesia/sources/pocket.py][pocket]], 25 lines of code
- [[https://github.com/karlicoss/promnesia/blob/master/src/promnesia/sources/fbmessenger.py][fbmessenger]], 30 lines of code

After that, you can simply import your custom source in =config.py=, and use it, same way as any builtin source.
