* Development Guide

** Getting Set Up

*** Obtaining the source code

For the source code:

#+BEGIN_SRC bash
git clone https://github.com/salotz/jubeo
cd jubeo
#+END_SRC

*** Tooling

To make things easier for developers we provide a set automation
scripts implemented in ~invoke~ (similar to a makefile if you are
familiar with that).

So you will want to install invoke somehow to use this tooling. So
make sure you are in a suitable python environment (i.e. not the
systems python) and install the tools in the ~tools.requirements.txt~
file.

#+begin_src bash
pip install -r tools.requirements.txt
#+end_src

*** Virtual Environments

This is for managing environments which are just for the purpose of
developing this project, and not necessarily just for running it (as a
user). See [[*Managing Dependencies][Managing Dependencies]] for details on managing dependencies
of the installable project.

We also require a few shell environmental variables which are exported
in the ~env.bash~ file. Go ahead and source this before doing anything
else:

#+begin_src bash
source env.bash
#+end_src

To create an env called ~jubeo.dev~ just run the ~env~ target from
~invoke~:

#+begin_src bash
inv env
#+end_src

Then follow the activation instructions that are printed as different
projects might use different backends.

If you ever have problems with an environment just rerun this to get a
clean one. A practice we encourage to do frequently so that developers
don't diverge in their envs with local modifications. So while you can
make your env, try to use this one unless you have problems.

We maintain a number of preconfigured environments in the ~envs~
directory which are used for different purposes. Calling ~inv env -n
dev~ is the same as ~inv dev~ since it is the default, but any other
environment can be created by passing the matching name. For instance
there is an environment that mimics the user's installation
environment so that we can test experiences upon install, to make sure
we haven't accidentally depended on something in the dev env. For
instance:

#+begin_src bash
inv env -n base
#+end_src

If you want to make another environment it is straightforward to copy
the examples in the ~envs~ dir.


** Maintenance Tasks

*** Managing Dependencies


Reminder that there are two separate goals of managing dependencies
and where they are managed:

- Python Libraries :: These dependencies are managed in ~setup.py~ and
  in PyPI or other indices.
- Python Applications/Deployments :: These are dependencies managed in
  ~requirements.in~ and ~requirements.txt~ and used for developer
  environments and deployment environments.

So for the library aspect we use abstract requirements. These should
essentially be the same as ~requirements.in~.

For the deployment side of things we use ~requirements.txt~. Don't
manually edit this. We use ~pip-tools~ to "compile" dependencies for
this.

# TODO: figure out high level and pinned conda version files

To initially pin an environment or when you add requirements run this
target:

#+begin_src bash
inv deps-pin
#+end_src

To update it (should be accompanied by a reason why):

#+begin_src bash
inv deps-pin-update
#+end_src

*** Documentation and Website

**** Editing and Building Docs

To compile and build the docs just run:

#+begin_src bash
inv docs-build
#+end_src

Which will output them to a temporary build directory ~_build/html~.

You can clean this build with:

#+begin_src bash
inv clean-docs
#+end_src


To view how the docs would look as a website you can point your
browser at the ~_build/html~ folder or run a python http web server
with this target:

#+begin_src bash
inv docs-serve
#+end_src


**** COMMENT TODO: WIP: Building and testing the website

The website is still a work in progress and is located in the ~jekyll~
folder.

The website uses jekyll and so you must have ~ruby~, ~bundler~, and
~jekyll~ installed.

On ubuntu and debian:

#+begin_src bash
sudo apt install -y ruby-full build-essential zlib1g-dev
#+end_src

And then on whichever distro with ~GEM_HOME~ on your ~PATH~:

#+begin_src bash
gem install jekyll bundler
#+end_src


Then you just need to run this command:

#+begin_src bash
inv website-deploy-local
#+end_src


**** COMMENT Deploying the website

We are using github pages. To avoid having to keep the entire built
website in the main tree we use the alternate ~gh-pages~ branch. To
make this process easy to deploy we have a script ~sphinx/deploy.sh~
that checks the ~gh-pages~ branch out, does some necessary cleaning
up, and copies the built website to the necesary folder (which is the
toplevel), commits the changes and pushes to github, and then returns
to your working branch.

The invoke target is:

#+begin_src bash
inv website-deploy
#+end_src


*** Testing

*** Code Quality Metrics

Just run the end target:

#+begin_src bash
inv quality
#+end_src

This will write files to ~metrics~.

*** Releases

**** Choosing a version number

***** Primordial Epoch

In the pre-covenant releases (think "1.0" release) we just release
based on the date. So just make the date string and add any release
candidate, dev, or postrelease additions.

***** Covenental Epoch

After the primordial epoch we have a more semantic meaning behind
version numbers.

Following PEP 440 epochs these should all be prefixed by '1!' for the
second epoch.

Otherwise versions should be "canonical" strings following the
guidelines in PEP 440.

That means we support roughly:

- semver-ish major, minor, and patch parts
- release candidates
- dev
- post release "patches"

However, we don't prescribe explicit semantics to any of the parts as
this is highly variable depending on the needs of a project and the
forums by which it is published (e.g. PyPI, Github, etc.).


**** Writing and/or Generating the Changelog and Announcement
**** Changing the version number

You can check the current version number with this command:

#+begin_src bash
inv py.version-which
#+end_src

Increase the version number we currently do it by hand (although an
automatic way would be nice and ~bumpversion~ wasn't working for
us). So go in and manually edit them. For reference see PEP 440 for
valid ones.

The target locations for changing versions are in the following files:

- [ ] ~setup.py~
- [ ] ~src/jubeo/__init__.py~
- [ ] ~sphinx/conf.py~
- [ ] ~tasks/config.py~

# IDEA: would like to have this done automatically with some
# replacement but I need a robust way to do this. Preferrably not
# using regexs, and still with some interactive intervention and
# confirmation of correctness

**** Making the release official
To make a release do some changes and make sure they are fully tested
and functional and commit them in version control. At this point you
will also want to do any rebasing or cleaning up the actual commits if
this wasn't already done in the feature branch.

If this is a 'dev' release and you just want to run a version control
tag triggered CI pipeline go ahead and change the version numbers and
commit. Then tag the 'dev' release.

If you intend to make a non-dev release you will first want to test it
out a little bit with a release-candidate prerelease.

So do all the following bookeeping steps in a single but separate git
commit from the actual changes to the code:

- [ ] write the changelog
- [ ] write the announcement (optional)
- [ ] change the version numbers
- [ ] build to test it out ~inv build~

To test a build go ahead and run:

#+begin_src bash
inv py.build
#+end_src

# TODO: add a test builds target

And then try to install it from an empty environment:

#+begin_src bash
inv env -n test_install
#+end_src

Activate the environment e.g. (make sure if using ~pyenv-virtualenv~
you are not in a virtualenv):

#+begin_src bash
source _venv/test_install/bin/activate
#+end_src

or

#+begin_src bash
conda activate _conda_envs/test_install
#+end_src

then run it for each build, e.g.:

#+begin_src bash
pip install dist/BUILD.tar.gz
#+end_src

They should all succeed.


When you change the version number put a 'rc0' or 'dev0' at the end of
the new intended (semantic) number.

Once you have built it and nothing is wrong go ahead and publish it to
the test indexes (if available):

#+begin_src bash
inv py.publish-test
#+end_src

You can test that it works from the index using the same
~test_install~ environment above.

And install the package from the test repo with no dependencies:

#+begin_src bash
inv env -n test_install
pip install --index-url https://test.pypi.org/simple/ --no-deps jubeo
#+end_src

# QUEST: should this message be the release message we want for the VCS
# repos to show or should we just point them to the changelog?

Then go ahead and commit the changes after that works. The message
should follow a fixed form like 

#+begin_src fundamental
1.0.0rc0 release preparation
#+end_src

Then you can tag the release in the ~git~ commit history:

#+begin_src bash
inv git.release
#+end_src

Publishing the results will vary but you can start with publishing the
package to PyPI and the VCS hosts with the real publish target:

#+begin_src bash
inv py.publish
inv git.publish
#+end_src


** Initializing this repository

These are tasks that should only be done once at the inception of the
project but are described for posterity and completeness.

*** Version Control

First we need to initialize the version control system (~git~):

#+begin_src bash
inv vcs-init
#+end_src

If you want to go ahead and add the remote repositories for this
project. We don't manage this explicitly since ~git~ is treated mostly
as first class for these kinds of tasks and is better left to special
purpose tools which are well integrated and developed.

*** Compiling Dependencies

Then add any extra dependencies you want to the development
environment [[file:../envs/dev/requirements.in][requirements.in]] file and then compile and pin them:

#+begin_src bash
inv deps-pin
#+end_src

*** Creating Environments

Then just create the virtual environment. For portability we use the
builin ~venv~ package, but this is customizable.

#+begin_src bash
inv env
#+end_src

Then you can activate it with the instructions printed to the screen.

*** Website Admin

We use Github Pages by default since it is pretty easy. Because we
don't want to clutter up the master branch with website build
artifacts we use the ~gh-pages~ branch approach.

If you just run the ~inv website-deploy~ target this will idempotently
take care of setting this up for you.

However, you will need to create it and push it before you can set
this in the github settings for the page.
