#+OPTIONS: num:nil

TOC: [[#motivation][Motivation]] | [[#demos][*Demos*]] | [[#install][*Install*]] | [[#try-it-out][Try it out]] | [[#develop][Develop]]

Promnesia is a browser extension ([[https://chrome.google.com/webstore/detail/promnesia/kdmegllpofldcpaclldkopnnjjljoiio][Chrome]]/[[https://addons.mozilla.org/en-US/firefox/addon/promnesia][Firefox]]/Firefox mobile) which serves as a web surfing copilot, enhancing your browsing history and web exploration experience.

*TLDR*: it lets you explore your browsing history *in context*: where you encountered it, in chat, on Twitter, on Reddit, or just in one of the text files on your computer.
This is unlike most modern browsers, where you can only see when you visited the link.

It allows you to answer different questions about the current web page:

- have I been here before? When? [[https://raw.githubusercontent.com/karlicoss/promnesia-demos/master/child-visits.webm][demo (30 s)]] [[https://raw.githubusercontent.com/karlicoss/promnesia-demos/master/child-visits-2.webm][demo (40s)]]
- how did I get on it? Which page has led to it? [[https://raw.githubusercontent.com/karlicoss/promnesia-demos/master/how_did_i_get_here.webm][demo (40s)]]
- why have I bookmarked it? who sent me this link? Can I just jump to the message? [[https://raw.githubusercontent.com/karlicoss/promnesia-demos/master/watch_later.webm][demo (30s)]]
- which links on this page have I already explored? [[https://raw.githubusercontent.com/karlicoss/promnesia-demos/master/mark-visited.webm][demo (30s)]],
- which posts from this blog page have I already read? [[https://raw.githubusercontent.com/karlicoss/promnesia-demos/master/mark-visited-2.webm][demo (20s)]]
- have I annotated it? [[https://raw.githubusercontent.com/karlicoss/promnesia-demos/master/highlights.webm][demo (1m)]]

You can jump straight to the [[#demos][Demos]] and [[#install][Install]] sections if you're already overexcited.

* Motivation

- Has it ever occurred to you that you were reading an old bookmark or some lengthy blog post and suddenly realized you had read it already before?
  It would be fairly easy to search in the browser history, however, it is only stored locally for three months.
  # TODO link?
- Or perhaps you even have a habit of annotating and making notes elsewhere? And you wanna know quickly if you have the current page annotated and display the annotations.
  However, most services want you to use their web apps even for viewing highglights. You don't know if you have highlights, unless you waste time checking every page.
- Or you have this feeling that someone sent you this link ages ago, but you don't remember who and where.
- Or you finally got to watch that thing in your 'Watch later' youtube playlist, that's been there for three years, and now you want to know why did you add it in the first place.

Then this tool is for you.

It consists of two parts:

- the extension: talks to the backend and neatly displays the history in a sidebar and shows you whether you visited a page
- the backend. It can run locally as a simple Python program (you don't need to host it, although you can).

The backend regularly processes various *data sources* like:

- data exports from online services: Reddit/Twitter/Hackernews/Telegram/Messenger/Hypothesis/Pocket/Instapaper, etc.
- Markdown/org-mode/HTML or any other plaintext on your disk
- Google Takeout/Activity backups
- in general, anything that can be parsed, it's easily extensible
  # TODO link to extension guide
  # TODO link to individual modules?
# TODO mentioned some motivation for the backend

#+html: <div id="demo"><div>
* Demos
  :PROPERTIES:
  :CUSTOM_ID: demos
  :END:

- [[https://user-images.githubusercontent.com/291333/69828210-3755ac80-121b-11ea-9d1e-e5086cc9feda.png]]

  # TODO might be good to explain what I mean by context and visit..
  - a green eye means that the link was visited before and has some contexts, associated with it
    when you click on it, the sidebar opens with more information
  - You can see that I've sent the link to someone on Telegram (1st sidebar item )
  - I've annotated the link on instapaper and highlights (2nd and 3rd) are shown inline on the page (yellow)
  - I've clipped some content to my personal notes at some point (2nd sidebar item), the selected text was matched and highlighted as well
    If I click the link, it will cause my Emacs to open and jump straight to the clipping within the file.

- [[https://user-images.githubusercontent.com/291333/64424146-2bd16a00-d0a0-11e9-80d2-73cf3b2b60df.PNG]]

  - I chatted with some of my friends and sent them the link at some point (2nd sidebar item)
    Clicking on the link will open Telegram and jump straight to the message where the link was mentioned.
    So you can reply to it without having to search or scroll over the whole chat history.
    # Json is clearly not the most convenient way to go through conversations with friends, but that's a matter of representing chats in a plaintext form. The benefit though is that once you have any sort of grepable source it's super easy to feed it into the plugin.
  - I've tweeted about this link before (4th item)
    Similarly, clicking would jump straight to my tweet.
  - I also have this link annotated via [[https://hypothes.is][Hypothesis]] (1st, 3rd and 5th items)

- More:

  - You can find more short screencasts, demonstrating various features [[https://github.com/karlicoss/promnesia-demos][here]]
  - There are more screenshots [[https://github.com/karlicoss/promnesia/issues/5#issuecomment-619365708][here]]
  # TODO roam?

* Install
  :PROPERTIES:
  :CUSTOM_ID: install
  :END:
  
- extension:

  - [[https://chrome.google.com/webstore/detail/promnesia/kdmegllpofldcpaclldkopnnjjljoiio][Chrome]]
  - [[https://addons.mozilla.org/en-US/firefox/addon/promnesia][Firefox]]
  - Firefox mobile: not released in the store yet, but you can build it yourself
   
- backend: available on [[https://pypi.org/project/promnesia][PyPi]]: =pip3 install --user promnesia=

  Alternatively, you can clone this repository and run it as ~scripts/promnesia~
  
* Try it out
The easies is to try out Promnesia is a demo mode, it can give you a sense of what Promnesia is doing with almost no configuration.

# TODO use smth different
1. [[#install][Install]] the extension and the backend in case you haven't already
2. Run ~promnesia demo --port 16789 https://github.com/karlicoss/exobrain~

   This clones the repository, ([[https://github.com/karlicoss/exobrain][my personal wiki]] in this case), extracts the URLs, and runs on the port =16789=

   You can also use a path on your local filesystem, or a website URL.

3. In the extension settings, set "Host" to =http://localhost:16789= and save.

4. After that, visit https://www.gwern.net . If you press the extension icon, you will see the pages from my blog where I link to articles on Gwern's site.
  
* Extended backend setup
# TODO mention where they get the database  
I'm working on improving the documentation on setting the backend up, so please feel free to ask if something is unclear!

- create the config

  You can start from the example config (=cp config.py.example config.py=), and then edit it.
  You can also find a more complete configuration example here: [[file:doc/config.py]].
 
  The only required settings are:

    - =OUTPUT_DIR=

      This is the directory for the indexer's output. At the moment it's only the =promnesia.sqlite= file.
      You can keep it as =/tmp=, but you might prefer something else.

    - =SOURCES=

      These are the data sources, that will be processed and indexed by Promnesia.
      I'm working on proper documentation for available sources. For now:

      - see the example configuration for now.
      - you can browse the code [[file:src/promnesia/sources/][here]].

      # TODO put up a list of sources automatically from the code?

- [optional] do a quick test

  This is useful to quickly check your config: run =promnesia demo --port 13131 --config /path/to/config.py=.

  This will index the data and launch the backend immediately, so you can check that everything works as expected in your browser.

- run the indexer: =promnesia index --config /path/to/config.py=

  At the moment, indexing is *periodic, not realtime*. The best is to run it via cron/systemd once or several times a day.

- run the server: =promnesia serve --db OUTPUT_DIR/promnesia.sqite=.

  You only have to start it once, it will automatically detect index changes.
- [optional] autostart the server with =promnesia install-server=

  This sets it up to start via Systemd, which should be present in most common *Linux* distributions, so it won't work on Mac/Windows at the moment.

  I don't have a Mac nearby, so if you can contribute a Launchd script, or point me to a setup guide, I'll be very grateful!
  I /think/ you can also use cron with =@reboot= attribute, or just create a manual autostart entry.

# TODO Frontend -- mention what settings are possible?

* Extension permissions
  
- =file/http/https=: the extension is meant to work on any page, so 
  In the future this might be changed to an optional permission and requested on demand if people feel it's worth it
- =activeTab=: getting current tab info and adding the sidebar
- =tabs=: making the extension work without an explicit user action
  It's meant to be an assistant first of all.
- =webNavigation=: watching page state changes (to trigger the extension on page load)
- =storage=: for settings
- =history=: to use local browsing history
  Might be optional in future
- =contextMenus=: Promnesia context menu
- =notifications=: showing notifications

# TODO tabs could be optional in the future?  

* Develop
I'm in the process of putting up better developer's documentation, but you should be able to figure it out from the CI pipelines:
[[file:.github/workflows/main.yml][Github Actions]] and [[file:.circleci/config.yml][CircleCI]].

** Run backend  
   During development, the easiest way to run =promnesia= as =scripts/promnesia=. Alternatively, you can use =pip install -e .=.

   # TODO reuse doc from pymplate?

** Build extension
   
   : cd extension
   : ./build --firefox # you can also use --chrome and --firefox-mobile
   :         --release # [optional], pass to build in the release mode (with optimizations)
   :         --publish # [optional], pass to release to the Chrome Web Store/Mozilla addons
   
   You'll find the result in =dist/firefox=. After that, you can load it in your browser and develop.
   
** Running end-to-end tests

   You need:

   - firefox dev edition (standard wouldn't let you install unsigned extensions) and =geckodriver=
   - TODO chromedriver?
   - =pip3 install pyautogui= for keyboard automation
   
   Example of running a test:

   : PYTHONPATH=src python3 -m pytest -s 'tests/end2end_test.py::test_add_to_blacklist[chrome]'



* WIP on automatic sources summary :noexport:
#+begin_src python :python "with_my python3" :dir src :results output
import pkgutil
import importlib
import inspect


from promnesia.common import Extraction
from typing import Iterable
IndexerType = Iterable[Extraction] # TODO move to promnesia? also careful with List etc...

import promnesia.sources as pkg
for importer, name, ispkg in pkgutil.walk_packages(
        path=pkg.__path__,
        prefix=pkg.__name__+'.'
):
    if name in {'promnesia.sources.plaintext'}:
        continue
    # TODO remove
    # if name not in {'promnesia.sources.vcs', 'promnesia.sources.roamresearch'}:
    #     continue
    # TODO get by output type??
    m = importlib.import_module(name)
    public = [(k, v) for k, v in inspect.getmembers(m) if not k.startswith('_')]
    indexers = [(k, v) for k, v in public if getattr(v, '__annotations__', {}).get('return') == IndexerType]
    assert len(indexers) > 0, name
    for k, i in indexers:
        # print(inspect.signature(i))
        print(f'{name}.{k}', i.__doc__ or 'NO DOCUMENTATION!!!')
#+end_src

