% options draft, arxiv, submit, target
\documentclass[target,proc,reqno]{ml-gen}

\usepackage{upref,mathrsfs,enumerate} % maybe amscd

%Footnotes
%\usepackage[symbol]{footmisc}
%\usepackage{perpage}
%\MakePerPage[5]{footnote}

\usepackage[colorlinks=true,linkcolor=blue,citecolor=blue,backref=page]{hyperref}
\usepackage{local}

%% DRAFT ONLY
\opt{draft}{
\usepackage{showlabels}
\renewcommand{\showlabelfont}{\scriptsize\ttfamily}
}

%% nicer fonts
\renewcommand{\rmdefault}{pplj}
\usepackage{euler}
\usepackage{noweb}
\usepackage{shortvrb}
\MakeShortVerb{\|}
\usepackage{listings}

%\setcounter{tocdepth}{1}
%\numberwithin{equation}{subsection}

\tolerance=2000 
\emergencystretch=20pt 
%}}}
% TITLE AND ABSTRACT %{{{
\begin{document}
\lstset{%
language=Python,
basicstyle=\small,
%commentstyle=\ttfamily,
%identifierstyle=\ttfamily,
%showspaces=true,
%stringstyle=\ttfamily,
columns=fullflexible}

\title[\texttt{pnw}]{\texttt{pnw} -- A text manipulation and literate
programming tool written in \texttt{python}}

\author{Matthias Lesch}
\address{Mathematisches Institut,
Universit\"at Bonn,
Endenicher Allee 60,
53115 Bonn,
Germany}

\email{ml@matthiaslesch.de, lesch@math.uni-bonn.de}
\urladdr{www.matthiaslesch.de, www.math.uni-bonn.de/people/lesch}
%
\thanks{Partially supported by the 
        Hausdorff Center for Mathematics}

\subjclass[2010]{Primary XXX; Secondary XXX}

% draft, submit only
\opt{draft,submit}{
\date{\today}
}
\begin{abstract}
\lstinline!pnw! is a text processing tool with a wide range of
applications. It can be used to compile chunks of text from various
sources into a new text file. The advantage over cut-and-paste
techniques is that changes in the source file are automatically
followed. Secondly, we provide a thin wrapper around the
\lstinline!pandoc! document converter. Therefore, chunks of text can be
converted from and to a wide range of lightweight markup languages as
well as \lstinline!html! and \lstinline!tex/latex!. Although literate
programming has become a little out of fashion, \lstinline!pnw! can be
used as a literate programming tool. Not surprisingly, \lstinline!pnw!
itself and its documentation is written using \lstinline!pnw!.
\lstinline!pnw! is remotely inspired by \lstinline!noweb! and heavily
inspired by (even borrowing part of the syntax from)
\href{https://pypi.python.org/pypi/antiweb/0.2.2}{antiweb}. An important
feature of pnw, which is not present in \lstinline!antiweb!, is the
support of namespaces for block. This allows to create reusable
\emph{block libraries} which imported and addressed in a
\emph{dot-notation syntax} similar to python modules.

A \lstinline!pnw! file is a text file with tags defining named blocks of
text. In a way this is similar to an xml file. Tags may be hidden behind
comment characters, such that the file remains processable by another
program. The blocks of text from various \lstinline!pnw! files can be
processed and rendered by another \lstinline!pnw! file.

\end{abstract}

\maketitle
\tableofcontents
%\listoffigures

\newpage
%}}}

% SECTION 1 
\section{\pnw\ language syntax}
\subsection{pnw tags}

A line is recognized as a pnw tag if it is of the form

\begin{lstlisting}
<whitespace><one non-word character>@tagname(args,kwargs)
\end{lstlisting}
The indent (length of the whitespace) is significant for some tags. The
non-word character is completely arbitrary. Typically one might use a
language specific comment character (\% for latex, \# for python) here.
Here is a quick summary of the available \lstinline!pnw! \# tags:

\begin{description}
\item[\textbf{\#@block(name,args,kwargs)}]
Starts block name; one may also write \lstinline!chunk! instead of
\lstinline!block!, they are synonymous. A block ends at

\begin{itemize}
\item
  the start of another block or an \lstinline!#@end! with a smaller or
  equal indent,
\item
  another \lstinline!#@! tag with a smaller indent,
\item
  the end of the file.
\end{itemize}
Several blocks may have the same name if they have the same indent. Then
these blocks are combined into a \emph{chunk} of that name and the
individual subblocks are named name1, name2 etc.

\item[\textbf{\#@end}]
Explicit end of a block, indent is significant.

\item[\textbf{\#@import(fname,nspace)}]
Imports file fname into nspace. This means that the blocks and chunks in
fname can be addressed as nspace.name. Within fname blocks/chunks can
address each other without nspace prefix. This is analogous to the
python import statement.

\item[\textbf{\#@include(name)}]
Includes the block/chunk name instead of this line. The indent is
significant: the indent of the \textbar{}\#@include\textbar{} statement
is added when rendering the content of block/chunk name.

\end{description}
All other lines outside of blocks are ignored. The content of each
namespace nspace is collected in the block (resp. chunk if several files
are imported into the same namespacefile fname) nspace.


\subsection{options, markup languages}

The syntax for the |block|/|chunk| tag is similar to the syntax for |python|
functions. In detail
\begin{lstlisting}
    chunk(arg1,arg2,...,argn,kwarg1=value1,...,kwargm=valuem)
\end{lstlisting}
So far the following options are available
\begin{description}
    \item[remove] Removes block completely from its parent block.
\end{description}        
Furthermore, the following keyword arguments are observed:
\begin{description}
    \item[format=\{pandoc format\}] Specifies the formatting of the following
        block. Any format which |pandoc| can handle is allowed. E.g. if a
        block is |markdown| formatted and it includes another block using the
        include directive, the program tries to convert that block into
        |markdown| as well.
\end{description}        


\section{Usage}
\subsection{CLI}
Command line usage is as follows
\begin{lstlisting}
pnw <options> <file>
\end{lstlisting}
where |<file>| is the pnw file. The following options are available
\begin{description}
    \item[-R, --chunk |<chname>|] Extracts chunk |chname| and renders it.
    \item[-F] Renders whole file as one chunk.
    \item[-f, --format] Output format, can be any format supported by
        |pandoc|.     
    \item[-t, --tree] Show document tree.
    \item[-d, --depth] Traverses the tree up to depth d. Nodes of depths
        d are indicated as |<<name>>|, as in noweb, and not expanded any further.
\end{description}        

\subsubsection{Example}
The command
\begin{lstlisting}
pnw -Rmoduledoc -flatex pnw.pnw > pnw.tex
\end{lstlisting}
renders the chunk |moduledoc| from the source file |pnw.pnw| in latex
format and writes the result to the file |pnw.tex|.

\subsection{\lstinline!python! module}

The public interface consists of the following functions and classes,

\begin{lstlisting}
class Block():
    def __init__(self,text,fname,lineno,parent_block = None, parent_chunk = None):
class Chunk(Block):
    def __init__(self,name,block):
def registerblock(name,block):
def pnwimport(fname,nspace,line=None):
\end{lstlisting}
We have the following global variables:

\begin{description}
\item[\textbf{paths}]
Search path for files

\item[\textbf{imports}]
Dictionary mapping imported filenames to their associated namespaces

\item[\textbf{chunks}]
Dictionary of Chunks and Blocks

\item[\textbf{roots}]
Set of Root Nodes

\item[\textbf{rootchunks}]
Set of Root Chunks

\end{description}
\subsubsection{\lstinline!function pnwimport!}

This is the main function for using pnw as a python module. The file
\lstinline!fname! is parsed and its blocks/chunks are read into the
namespace \lstinline!nspace!. The argument \lstinline!line! is used
internally for conveying error information and it can be ignored by
external user.

Indent is taken care of in chunk defs, not in imports, so the
indentation of import statement is irrelevant.

\subsubsection{\lstinline!class Block!}

This is the base class for nodes of the document tree. For each line of
a pnw document an instance of \lstinline!Block! is created. A
\lstinline!Block! needs the following default information:

\begin{description}
\item[\textbf{text}]
The (line of) text
\item[\textbf{fname}]
The name of the file
\item[\textbf{lineno}]
Line number of the text line
\end{description}
\textbf{Important attributes:}

\textbf{parent\_ block} : the parent block in the document tree

\textbf{parent\_chunk} : if several blocks have the same name they are
combined into a \lstinline!Chunk!. This attribute points to the parent
\lstinline!Chunk!

\textbf{children} : list of child nodes

\textbf{args,kwards} : list of args, resp. dictionary of keyword args
found in () after the tag

\textbf{tag} : name of the tag

\textbf{indent} : indentation (int) of the block/chunk

\%end of descr list \textbf{Public methods:}

\begin{lstlisting}
def expand(self,indent=0,depth=0,toformat=None):
def render(self,indent=0,depth=0,toformat=None):
\end{lstlisting}
\lstinline!expand! returns a list of lines, render just joins them into
a string. \lstinline!indent! and \lstinline!toformat! are hopefully
self-explanatory. \lstinline!depth! is the recursion level up to which
\lstinline!includes! are expanded. Beyond that level
\lstinline!include(foo)! is rendered as \lstinline!<<foo>>! with the
correct indentation.

\subsubsection{\lstinline!class Chunk(Block)!}

A Chunk is a node which is outside the normal document tree. A
\lstinline!Chunk! \lstinline!cname! is created when a second
\lstinline!Block! \lstinline!cname! is found. All \lstinline!n! Blocks
of name \lstinline!cname! become children of the newly created
\lstinline!Chunk!. The Blocks are renamed \lstinline!cnamei! where
\lstinline!i! runs from \lstinline!1! to \lstinline!n!.

\subsubsection{\lstinline!function registerblock!}

Fills the dictionary \lstinline!chunks! and takes care of the tree
structure. If the name does not yet exist, the block is added to the
dictionary \lstinline!chunks! with key \lstinline!name!. Otherwise, if
necessary, a new chunk \lstinline!name! is created and the block is
appended to its list of children.

\subsubsection{Example}

Assuming that pnw is installed, from the main directory of the pnw
package open an interactive python session, I recommend to use
\lstinline!ipython!, and write

\begin{lstlisting}
import pnw
pnw.pnwimport('pnw.pnw','')
\end{lstlisting}
Now you can explore the pnw document tree of the \lstinline!pnw.pnw!
source file by inspecting the dictionary

\begin{lstlisting}
pnw.chunks
\end{lstlisting}
To see the structure of a block, just print its repr

\begin{lstlisting}
pnw.chunks['moduledoc']
\end{lstlisting}
To render it, try

\begin{lstlisting}
print pnw.chunks['moduledoc'].render(indent=0,depth=2,toformat='rst')
\end{lstlisting}
The whole pnw documentation is written using pnw. To see it in action,
explore the files \lstinline!pnw-doc.pnw!, \lstinline!md-blocks!,
\lstinline!latex-blocks.tex! in the \lstinline!doc! folder as well as
\lstinline!pnw.pnw! in the main folder.

\section{Implementation details}

\subsection{pnwimport}

\begin{lstlisting}
def pnwimport(fname,nspace,line=None):
    if line is None:
        line = Block(u'Command Line','CLI',0)
    imports[fname] = nspace
    f = None
    for path in paths:
        tmpname = os.path.join(path,fname)
        try:
             f = open(tmpname,"r")
        except IOError:
            continue
        break
    if f is None:
        raise PNWError(line,"Could not open file %s"%fname)
    else:
        with f:
            linegen=enumerate(f.read().split('\n'),1)
            rootblock = Block(u'#@chunk(%s)'%nspace,fname,lineno = 0)
            rootblock.indent = -1
            registerblock(nspace,rootblock)
            parse_block(linegen,rootblock)
\end{lstlisting}
\subsection{parse\_block}

\begin{lstlisting}
def parse_block(linegen,node): 
    lineno,text = linegen.next()
    line = Block(text,node.fname,lineno,node)
    while True:
        if line.tag == IMPORT:
            imp_fname,imp_nspace = line.args[0:2]
            if imp_fname in imports:
                raise PNWError(line, "pnw does dot support multiple \
                        imports of the same file, file already imported")
            pnwimport(imp_fname,imp_nspace,line)
        elif (line.tag in keywords and line.indent < node.indent) or\
                (line.tag in [END,BLOCK,CHUNK] and line.indent <= node.indent): # our block ends here
            return (line,lineno)
        elif line.tag in [BLOCK,CHUNK]:
            registerblock(line.args[0],line)
            #import pdb;pdb.set_trace()
            if REMOVE not in line.args:
               node.append(line) # we need to append this node
            line,lineno=parse_block(linegen,line)
            continue
        elif (line.tag in [EMPTY,TEXT]) and line.indent < node.indent:
                #import pdb;pdb.set_trace()
                raise PNWError(line,u'indentation error')
        else:
            node.append(line)
        if line.tag in [INCLUDE]: 
            # is here, because node must get appended, list here those directives which need nspace adjustment
            line.args[0] = format_nspace(line.nspace,line.args[0])
        # if we are here, we need to get a new line or close up
        try:
            #import pdb;pdb.set_trace()
            lineno,text = linegen.next()
            line = Block(text,node.fname,lineno,node)
        except StopIteration:
            # end of file reached, add a #@end with correct indentaton
            lineno+=1
            line = Block(u'#@end',node.fname,lineno,node)
            line.indent = -1
            return (line,lineno)
\end{lstlisting}
\subsection{Class Block}

\begin{lstlisting}
class Block():
    def __init__(self,text,fname,lineno,parent_block = None, parent_chunk = None):
        #self.__id__ = BLOCK # is needed by registerblock, avoid isinstance
        self.text = text
        self.name = u''
        self.fname = fname
        self.lineno = lineno
        self.parent_block = parent_block
        self.parent_chunk = parent_chunk
        self.children = []
        self.parse()

    @property
    def nspace(self):
        return imports[self.fname]

    @property
    def format(self):
        return self.kwargs.get('format')

    def append(self,block):
        block.parent_block = self
        self.children.append(block)

    def expand(self,indent=0,depth=0,toformat=None):
        #Baustelle
        if self.indent < 0:
            selfind = 0
        else:
            selfind = self.indent
        res = []
        if self.children and depth > 0:
            for child in self.children:
                res.extend(child.expand(indent+child.indent-selfind,depth-1,toformat=self.format))
        else:        
            if self.tag == TEXT: # return verbatim
                res.append(u' ' * indent + self.text[selfind:])
            elif self.tag == EMPTY:
                res.append(self.text)
            elif self.tag == INCLUDE: # must be include, don't mix up insert and import!!
                # ch gets expanded *instead* of self, thus depth is same
                ch = chunks[self.args[0]]
                #if self.depth:
                #    tmpdepth = self.depth
                #else:
                #    tmpdepth = depth
                # possible shortcut: self.depth or depth    
                res.extend(ch.expand(indent,self.depth or\
                    depth,toformat=toformat))
            if self.tag in [BLOCK,CHUNK]:
                res.append(u' ' * (indent) +'<<'+self.name+'>>\n')   # was self.args[0]
        tmp = u'\n'.join(res)
        if self.tag in [BLOCK,CHUNK] and toformat and self.format:
            tmp = convert(tmp,toformat,self.format)
            res = tmp.split('\n')
        return res 

    def render(self,indent=0,depth=0,toformat=None):
        return u'\n'.join(self.expand(indent,depth,toformat))

    def parse(self):
        #m=re.match("(\s*)#@(\w*)(\((.*)\)){0,1}",self.text)
        m=re.match("(?P<indent>\s*)(%|#)@(?P<tag>\w*)(\((?P<args>.*)\)){0,1}",self.text)
        if m:
            d = m.groupdict()
            tag = d['tag'] #m.group(2)
            if tag not in tags:
                logger.error(errmsg(self,"Unknown Tag %s"%tag))
            self.tag = tag
            if d['args']:
                args,kwargs = parseargs(d['args'])
            else: 
                args = [u'',u'']
                kwargs = dict()
            self.indent = len(d['indent'])
            self.args = args
            self.kwargs = kwargs
            while len(self.args) < 2:
                self.args.append(u'')
            if self.tag in [BLOCK,CHUNK]:
                self.name = args[0]
                self.tag = BLOCK
            # tags we process just here on the fly
            if self.tag == PATH:
                paths.insert(0,args[0])
            if self.tag == INCLUDE:
                if self.args[1]:
                    try:
                        self.depth = int(self.args[1])
                    except:
                        raise PNWError(self,"optional argument of\
                        include tag must be a nonnegative integer") 
                else:
                    self.depth = None
            return (tag,args,self.indent)
        elif m is None:
            # empty lines get type None
            if self.text.strip() == u'':
                self.tag = EMPTY
                self.indent = 256
            else:    
                self.tag = TEXT
                self.indent = len(self.text) - len(self.text.lstrip())
            return None

    def _repr(self,depth=256,indent = 0):
        res = []
        if self.tag not in [TEXT,EMPTY]:
            try:
               res.append(u' '*indent+u'--- <BLOCK %s:%s:%i:%s: %s '%\
                       (self.name,self.fname,self.lineno,self.tag,\
                       self.text.rstrip("\n").strip())+u'>')
            except:
                #strace()
                pass
        if depth > 0:
            for child in self.children:
                res.extend(child._repr(depth-1,indent+4))
        return res

    def __repr__(self):
        if self.tag in keywords:
            return u'\n'.join(self._repr())
        else:
            return u'--- <TEXT %s:%i:%s: %s'%(self.fname,self.lineno,\
                    self.tag,self.text.rstrip("\n").strip())+u'>'
\end{lstlisting}
\subsection{Clas Chunk}

\begin{lstlisting}
class Chunk(Block):
    def __init__(self,name,block):
        Block.__init__(self,block.text,block.fname,block.lineno)
        self.__id__ = CHUNK
        self.indent = block.indent # cannot be parsed from block.text, 
                                   # since we play around with indent -1
        self.name = name
        self.tag  = CHUNK
        self.append(block)

    def append(self,block):
        if block.indent != self.indent:
            raise PNWError(block,"All blocks in a chunk must have the same\
                            indentation level")
        block.parent_chunk = self
        self.children.append(block)

    def _repr(self,depth=256):
        res = []
        res.append(u'<Chunk %s:%s:%i:%s: %s'%(self.name,self.fname,\
                self.lineno,self.tag,self.text.rstrip('\n'))+u'>')
        for child in self.children:
            res.extend(child._repr(depth = depth))
        return res    

    def __repr__(self):
        return u'\n'.join(self._repr())
\end{lstlisting}



%}}} 
\bibliography{localbib}
\bibliographystyle{amsalpha-lmp}

% CUT AND PASTE MATERIAL
%\section{Cut- and paste material}
\end{document}


