\section{Quick start}

We illustrate Pylightnix principles by showing how to use it for making a mock
data science task. We will follow the
\href{https://docs.scipy.org/doc/scipy/reference/generated/scipy.optimize.dual_annealing.html}{SciPy
annealing example} but imagine that we additionally want to save some intermediate results
and also that there are several people working on this task.

We start by importing the required Python modules.

\begin{pythontexcode}
import numpy as np
import scipy.optimize as o
import matplotlib.pyplot as plt

from pylightnix import *
\end{pythontexcode}

\subsection{The problem}

The problem begins with defining a parametric function \texttt{f} for which we
need to find an approximate minimum by running the simulated annealing
algorithm.

\begin{pythontexcode}
def f(z, *params):
    x, y = z
    a, b, c, d, e, f, g, h, i, j, k, l, scale = params
    return (a * x**2 + b * x * y + c * y**2 + d*x + e*y + f) + \
           (-g*np.exp(-((x-h)**2 + (y-i)**2) / scale)) + \
           (-j*np.exp(-((x-k)**2 + (y-l)**2) / scale))
\end{pythontexcode}

\subsection{Stages basics}

Suppose we are to get the \texttt{f}'s parameters from elsewhere and
that we want to get the following results: (a) the parameters themselves (b)
the result of the annealing simulation and (c) a visual plot of the result.

\subsubsection{Parameter stage}

Pylightnix \pref{Stage}{pylightnix.types.Stage} is a Python function that
registers a user-defined action in the \pref{Registry}{pylightnix.types.Registry}
(passing None means asking Pylightnix to use the default one) and returns a
\pref{DRef}{pylightnix.types.DRef} reference stating that the action is
accepted.

\begin{pythontexcode}
def stage_params(r:Optional[Registry]=None)->DRef:
  def _config():
    name = 'params'
    out = [selfref, "params.npy"]
    return locals()
  def _make(b:Build):
    np.save(mklens(b).out.syspath, (2, 3, 7, 8, 9, 10, 44, -1, 2, 26, 1, -2, 0.5))
  return mkdrv(mkconfig(_config()), match_only(), build_wrapper(_make), r)
\end{pythontexcode}

Here, a user-defind action \texttt{\_make} goes to Pylightnix by calling the
\pref{mkdrv}{pylightnix.core.mkdrv} with the Registry and the following three
arguments: (a) the set of \pref{prerequisites}{pylightnix.types.Config} for the
action; (b) the \pref{Matcher}{pylightnix.types.Matcher} callback function; (c)
the action itself that takes a \pref{Build}{pylightnix.types.Build} context as
an argument. In this example we are pretending to receive input parameters from
a third-party. We could just as well have downloaded them from the Internet using
a pre-defined stage \pref{fetchurl}{pylightnix.stages.fetchurl2}.

\hfill \break \noindent
A few notes on the above code:

\begin{itemize}
  \item We define configuration object by reading a local variables of a helper
    function named \texttt{\_config()}. The resulting dictionary should match
    the \pref{Config}{pylightnix.types.Config} requirements. As is often the
    case in Pylightnix, we use a standalone
    \pref{mkconfig}{pylightnix.core.mkconfig} constructor rather than create the
    object directly.
  \item We promise to produce an artifact named \texttt{out} by specifying its
    \texttt{selfref} path. Pylightnix will throw an error if it does not find
    such an artifact after the build action completes.
  \item When inside the build action, we refer to the pylightnix config items by
    using the \pref{mklens}{pylightnix.lens.mklens} swiss knife function.
    \texttt{mklens} returns the \pref{Lens}{pylightnix.lens.Lens} object which
    has an overloaded dot "\texttt{.}" operator. The middle part of the lens
    expression encodes the path in the configuration and the last word
    "\texttt{syspath}" encodes the format of the expression result.
  \item We pass a \pref{match\_only()}{pylightnix.core.match\_only} matcher in
    the third argument of \texttt{mkdrv}. By this we say that we expect our
    action to return a deterministic result. Later we will see how to deal with
    non-deterministic actions.
  \end{itemize}

\subsubsection{Annealing stage}

\begin{pythontexcode}
def stage_anneal(ref_params:DRef, r:Optional[Registry]=None)->DRef:
  def _config():
    name = 'anneal'
    nonlocal ref_params
    trace_xs = [selfref, 'tracex.npy']
    trace_fs = [selfref, 'tracef.npy']
    out = [selfref, 'result.npy']
    return locals()
  def _make(b:Build):
    params = np.load(mklens(b).ref_params.out.syspath)
    xs = []; fs = []
    def _trace(x,f,ctx):
      nonlocal xs,fs
      xs.append(x.tolist())
      fs.append(f)
    res = o.dual_annealing(f, [[-10,10],[-10,10]],
                         x0=[2.,2.],args=params,
                         maxiter=500, callback=_trace)
    np.save(mklens(b).trace_xs.syspath, np.array(xs))
    np.save(mklens(b).trace_fs.syspath, np.array(fs))
    np.save(mklens(b).out.syspath, res['x'])
  return mkdrv(mkconfig(_config()), match_only(), build_wrapper(_make), r)
\end{pythontexcode}

\subsubsection{Plotting stage}

\begin{pythontexcode}
def stage_plot(ref_anneal:DRef, r:Optional[Registry]=None)->DRef:
  def _config():
    name = 'plot'
    nonlocal ref_anneal
    out = [selfref, 'plot.png']
    return locals()
  def _make(b:Build):
    xs=np.load(mklens(b).ref_anneal.trace_xs.syspath)
    fs=np.load(mklens(b).ref_anneal.trace_fs.syspath)
    res=np.load(mklens(b).ref_anneal.out.syspath)
    plt.figure()
    plt.title(f"Min {fs[-1]}, found at {res}")
    plt.plot(range(len(fs)),fs)
    plt.grid(True)
    plt.savefig(mklens(b).out.syspath)
  return mkdrv(mkconfig(_config()), match_latest(), build_wrapper(_make), r)
\end{pythontexcode}

\subsubsection{Running the experiment}

To run the experiment we register all the stages in the internal manager by
calling \texttt{instantiate} and ask Pylightnix to provide us with the
\pref{realization reference}{pylightnix.types.RRef} by calling
\texttt{realize1}.

\begin{pythontexcode}
with current_registry(Registry()):
  ds=stage_params()
  cl=stage_anneal(ds)
  vis=stage_plot(cl)
  rref=realize1(instantiate(vis))
  print(rref)
\end{pythontexcode}

\mystdout

\subsection{Nested stages}

In the previous sections we used a global
\pref{Registry}{pylightnix.types.Registry} object to register stages. Another way
of combining stages together is to wrap them with an umbrella stage covering the
whole experiment.

\begin{pythontexcode}
def stage_all(r:Registry):
  ds=stage_params(r)
  cl=stage_anneal(ds,r)
  vis=stage_plot(cl,r)
  return vis
\end{pythontexcode}

We may pass the top-level stage directly to \texttt{instantiate}
which would call it with a local disposable Registry.

\begin{pythontexcode}
rref_all=realize1(instantiate(stage_all))
assert rref_all==rref
print(rref_all)
\end{pythontexcode}

\mystdout


\subsection{Collaborative work}

Alice and Bob runs the same experiment using their machines. We model
this situation by running Pylightnix with different storage settings.

\begin{pythontexcode}
Sa=mkSS('_storageA')
Sb=mkSS('_storageB')
fsinit(Sa,remove_existing=True)
fsinit(Sb,remove_existing=True)
rrefA=realize1(instantiate(stage_all, S=Sa))
rrefB=realize1(instantiate(stage_all, S=Sb))
print(rrefA)
print(rrefB)
\end{pythontexcode}

Our annealing problem is stochastic by nature so the results naturally differ.

\subsection{Synchronization}

Alice sends her results to Bob, so he faces a problem of picking the best
realizaion.

\begin{pythontexcode}
print("Bob's storage before the sync:")
for rref in allrrefs(S=Sb):
  print(rref)
arch=Path('archive.zip')
spack([rrefA], arch, S=Sa)
# .. Alice transfers the archive to Bob using her favorite pigeon post service.
sunpack(arch, S=Sb)
print("Bob's storage after the sync:")
for rref in allrrefs(S=Sb):
  print(rref)
\end{pythontexcode}

\mystdout


Bob writes a custom matcher that selects the realization which has the best
annealing result.

\begin{pythontexcode}
def match_min(S, rrefs:List[RRef])->List[RRef]:
  avail=[np.load(mklens(rref,S=S).trace_fs.syspath)[-1] for rref in rrefs]
  best=sorted(zip(avail,rrefs))[0]
  if best[1] in allrrefs(Sa):
    print(f"Picking Alice ({best[0]}) out of {avail}")
  else:
    print(f"Picking Bob ({best[0]}) out of {avail}")
  return [best[1]]

def stage_all2(r:Registry):
  ds=stage_params(r)
  cl=redefine(stage_anneal, new_matcher=match_min)(ds,r=r)
  vis=stage_plot(cl,r)
  return vis

rref_best=realize1(instantiate(stage_all2,S=Sb))
print(rref_best)
\end{pythontexcode}

\mystdout
