Coopr Developer Guide 3.1
=========================
:doctype: book

//include::book-start.txt[]

== Overview ==

This overview chapter outlines the book.


== Coopr Library API ==

=== Introduction ===

This chapter describes the API of Coopr's library and functions
and classes.  This API is focused on the functions and classes that
support the transformation and analysis of Coopr models.

A key aspect of Coopr's library are functors, which are function
objects that can be executed like other functions.  Coopr's functors
provide a unifying interface for Coopr's API, and they support the
following functionality:

* Functor registration standardizes function definition, which
facilitates plug-and-play of the functions in the API.

* The functor docstrings are parsed to support error checking for
functor inputs and outputs.

* Functors are globally registered, which allows functors to be created
on the fly and enables documentation/enumeration of all functors in
the API.

* Functors can be integrated into formal workflows (using components from +pyutilib.workflow+)

Thus, functor
declarations allow the Coopr API to be more than a simple library of function
calls.

=== Declaring Coopr Functors ===

Functors in the Coopr API are declared using the +coopr_api+ decorator
with a Python function.  The following example illustrates the use
of this decorator:
[source,python]
----
include::examples/example1@main.py[]
----
The +f1+ function is a normal Python declaration, and the +coopr_api+
decorator transforms this function into a functor.  This functor can be 
executed as if it was a function.  For example:
[source,python]
----
include::examples/example1@exec.py[]
----

Coopr functors are required to have an argument that is a container of labeled data,
which is treated specially. The
container is required to be a +dict+ or +CooprAPIData+ class.  The +CooprAPIData+ class 
generalizes the Python +dict+ class in several ways.  Most notably,
an attribute added to an +CooprAPIData+ object is also added to the
underlying dictionary.  In the previous example, the +CooprAPIData+
object is passed in as the first argument.  In fact, Coopr functors
are only allowed to have one non-keyword argument.  Alternatively,
a functor can be declared with a +data+ keyword argument, in which
case it has no non-keyword arguments.  For example, the functor can be 
declared as follows
[source,python]
----
include::examples/example2@main.py[]
----
and it is evaluated as follows
[source,python]
----
include::examples/example2@exec.py[]
----
In fact, this functor can also be evaluated as before:
[source,python]
----
include::examples/example1@exec.py[]
----
Although this creates a syntactic difference between the declaration and formulation of
functors, this allows functors to be used with a common API.

If a +dict+ is passed into a functor to provide the container of
labeled data, then the functor converts it to a +CooprAPIData+ object
before executing the function.  Since +CooprAPIData+ objects are subclasses
of +dict+, this change may be transparent to the user.  However,
this is important in contexts where features of +CooprAPIData+ are used.

The return value of a Coopr functor is an +CooprAPIData+ object.  However,
the return value of the function used to declare the constructor
may be either +None+, a +dict+ object, or an +CooprAPIData+ object.  If the function returns
+None+ or the +data+ object is returned, then the functor creates
an +CooprAPIData+ object with an element with key +data+ whose value is
the +CooprAPIData+ object passed into the functor.  Otherwise, if an
+CooprAPIData+ object is returned then the functor adds an element with
key +data+ if it does not already exist.  If a +dict+ object is returned, then it is converted to an +CooprAPIData+ object and processed in the same manner. Consequently, the return
value of a Coopr function is an +CooprAPIData+ object that is guaranteed
to contain a container of labeled data with key +data+.

The docstring comments used in these examples are needed to fully
specify the API of the Coopr functor.  These docstrings are needed
to properly execute Coopr functors.  The +Required+, +CooprAPIData+,
and +Return+ keywords declare blocks where keyword arguments and
return values are described.  Although all keywords are declared
with a default value, these values are only used in a functor for
the optional arguments.  An exception is generated if a required
argument to a functor is omitted.  The labeled return values specify
the possible outputs of a functor.  An exception is generated if a
unexpected label for a return value is specified.  When a functor
returns without defining a defined return label, then its value is
+None+.

The +Required+ block can also be used to validate the existence of
data that is nested inside of the functor arguments. Consider the
following example, which validates the existence of values in the
+data+ and +x+ arguments:
[source,python]
----
include::examples/example3@main.py[]
----
Note that the nested values are assumed to be simple nested attributes
of the form +a.b.c.d+.  General purpose tests are not supported for 
checking the validity of data, and the test for a nested value simply verifies
that it exists and that it is not equal to +None+.

These requirements on functors enforce a uniform API for the input
and output values.  Input values consist of a container of labeled data
along with keyword arguments, and output values have the same form.
This consistency facilitates the use of functors in a larger
computational workflows.  The incorporation of the container object
into the Coopr functor API allows keyword arguments to be added in
an extensible manner.  For example, this feature enables the
incorporation of data that is used by subsequent functors in the
computation without requiring an extension of a the APIs of the
preceding functors.

Note: The +CooprAPIData+ class supports a container for labeled data
that generalizes the dictionary used for Python nonformal keyword
arguments, which are specified with the syntax +**kwd+.  Nonformal
keyword arguments are used in the APIs of other Python packages
(e.g. MatplotLib).  The use of +CooprAPIData+ is motivated by the use
of functors in formal computational workflows.


=== The Functor Registry ===

Declarations of Coopr functors automatically populate a global
registry of the Coopr API.  This registry allows functor objects
to be _created_ on the fly.  For example:
[source,python]
----
include::examples/example4@create.py[]
----
This example illustrates how the functor object +g+ can be created
from the registered functor +f1+.  The functor +g+ acts exactly
like +f1+, and in practice there is little difference between using
+g+ and +f1+.  However, functors can be created by name using the
factory +CooprAPIFactory+, and thus the user does not need to know
the specific Coopr library in which a functor is defined.

To help organize functors, a +namespace+ option can be specified
when declaring the functor.  This allows functors to be defined
with the same name in different packages, while distinguishing how
they are registered.  For example:
[source,python]
----
include::examples/example5@main.py[]
----
The functor +f1+ is declared within the +utility+ namespace.  The +f1+ object can be used within
the python module containing this declaration.  However, this functor is registered as +utility.f1+ in the
registry, so the functor is created as follows:
[source,python]
----
include::examples/example5@create.py[]
----

Finally, the functor registry allows for the automatic generation of documentation for the 
Coopr API.  The +coopr+ command supports the +api+ subcommand, which generates a simple summary of all
functor namespaces and their corresponding functors.  This output looks something like the following:

----
include::examples/coopr_api.txt[]
----
Additionally, the +--asciidoc+ option can be specified to generate a detailed description of the Coopr API, which is used to generate this document (see below).



=== Functors and Workflow ===

TODO: Coopr functors can be tied together into formal workflows that can be executed in an arbitrary manner


=== API Notes ===

The following notes describe further action items that we've outlined for the Coopr API:

* Tasks/WF objects
** Parallelization
** Reuse?
** Push/Pull input/output ports
** Pre/Post events
*** Global
*** Task-specific  (extension point or events?)
** Tasks support extension points
*** Use 'implements()' to define the EP that is supported

* API code review
** Document code review along-side API functions?

* Case studies (and tests)

* Pyomo model transformations
** In-place vs. copies
** Need to clarify semantics
** Active/Frozen models
** Flattening structured models
*** Utilities for walking a non-flattened model (iterator?)
** EP's supported by new components (?)

* Script generation
** Record sequence of calls
** Tracing logic OR auto-scripting

// Input an auto-generated summary of the Coopr API.
include::coopr_api.txt[]

== How To Run Tests ==

=== Running Smoke Tests ===

These commands run the Python Nose utility (+nosetests+) for each package within the named argument.

To test all of coopr, use the test.coopr script. From the coopr directory, use the command:
----
bin/test.coopr
----

Or from any sub-directory of coopr, use
----
lbin test.coopr
----

To test pyutilib, use
----
bin/test.pyutilib
----

To test just pyomo, from any coopr sub-directory, use
----
lbin test.coopr coopr.pyomo
----

There are options for tests (e.g., +--cat=all+), to see a list of options use the +-h+ or +--help+ option.

=== Interpreting Output ===

There are two modes of "not-pass:" FAIL and ERROR.
FAIL is an assertion failure. ERROR is a crash.

// vim: set syntax=asciidoc:

