# Princeton University licenses this file to You under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.  You may obtain a copy of the License at:
#     http://www.apache.org/licenses/LICENSE-2.0
# Unless required by applicable law or agreed to in writing, software distributed under the License is distributed
# on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and limitations under the License.


# *******************************************  LearningMechanism *******************************************************

"""
.. _LearningMechanism_Overview:

Overview
--------

A LearningMechanism is an `AdaptiveMechanism <AdaptiveMechanism>` that modifies the `matrix <MappingProjection.matrix>`
parameter of one or more `MappingProjections <MappingProjection>`.  Its function takes one or more `error_signals
<LearningMechanism_Input_Error_Signal>` (usually the output of a `ComparatorMechanism` or one or more other
`LearningMechanisms <LearningMechanism>`), as well as information about the `MappingProjection(s) and activity
<LearningMechanism_Additional_Attributes>` that generated the error(s), and calculates a `learning_signal
<LearningMechanism.learning_signal>` that is used to modify the MappingProjection(s) by way of its
`LearningProjection(s) <LearningProjection>`.  Typically, a LearningMechanism is used to "train" a single
MappingProjection (its `primary_learned_projection`), using the output of the Mechanism to which that
MappingProjection projects (its `output_source`) as the source of the error it attempts to reduce.  A
LearningMechanism can be used to train multiple MappingProjections, by assigning it `additional LearningProjections
<LearningMechanism_Multiple_LearningSignals>`; however, these will all use the same `learning_signal
<LearningMechanism.learning_signal>`, generated by the `primary_learned_projection` and its associated `output_source`.

All of the MappingProjection(s) modified by a LearningMechanism must project from one `ProcessingMechanism
<ProcessingMechanism>` to another in the same `System` or `Process`. The learning components of a System can be
displayed using the System's `show_graph <System.show_graph>` method with its **show_learning** argument assigned
`True` or *ALL*. LearningMechanisms are executed after all of the ProcessingMechanisms in a System or Process have
been executed, and before any `ControlMechanisms <ControlMechanism>` of the System have been executed (see `System
Execution <System_Execution>`).

.. _LearningMechanism_Note

*A Note about the Implementation of Learning*
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

The implementation of learning in PsyNeuLink was designed for flexibility and "transparency" rather than efficiency.
Unlike its implementation in most other environments -- where the learning algorithm is tightly integrated with the
elements of processing that it modifies --  PsyNeuLink separates it into three constituent components:  An
`ObjectiveMechanism` used to evaluate the most proximal source of error, a `LearningMechanism` that uses that error
(or one derived from it by another LearningMechanism) to calculate a learning signal;  and `LearningProjection(s)
<LearningProjection>` that use that learning signal to modify the weight `matrix <MappingProjection.matrix>` of the
`MappingProjection(s) <MappingProjection>` being learned.  This has the advantage of isolating and exposing the
constituent computations, making it clearer to students what these are and how they operate, and also making each
individually accessible for reconfiguration. However, it comes at the cost of efficiency.  Therefore, in its present
form, PsyNeuLink is not suitable for conducting large scale learning models (e.g, deep learning networks). However,
once trained, the parameters of such models can easily be incorporated into a PsyNeuLink model for use as part of a
`System`.  Future extensions of PsyNeuLink will provide an interface for the direct construction and integration
within the PsyNeuLink environment of objects created in other environments (e.g.,
`TensorFlow <https://www.tensorflow.org/>`_, `PyTorch <https://pytorch.org>`_ or
`Emergent <https://grey.colorado.edu/emergent/index.php/Main_Page>`_).


.. _LearningMechanism_Creation:

Creating a LearningMechanism
----------------------------

A LearningMechanism can be created in any of the ways used to `create Mechanisms <Mechanism_Creation>`.
More commonly, however, LearningMechanisms are created automatically.

.. LearningMechanism_Automatic_Creation:

*Automatic Creation*
~~~~~~~~~~~~~~~~~~~~

A LearningMechanism is created automatically when:

    * the **learning** argument is specified in the constructor for a `System <System_Execution_Learning>` or `Process
      <Process_Learning_Sequence>`;
    ..
    * a `tuple specification <MappingProjection_Learning_Tuple_Specification>` is used to specify learning for a
      MappingProjection;
    ..
    * a `LearningProjection` is created without specifying its `sender <LearningProjection.sender>` attribute.

In these instances, a `ComparatorMechanism`, `LearningSignal`, `LearningProjection`, and any additional Projections
required to implement learning that do not already exist are also instantiated.  This is described below, under
`Learning Configurations <LearningMechanism_Learning_Configurations>`.


.. _LearningMechanism_Explicit_Creation

*Explicit Creation*
~~~~~~~~~~~~~~~~~~~

If a LearningMechanism is created explicitly (using its constructor), then its **variable** and **error_sources**
arguments must be specified.  The **variable** must have at leaset three items that are compatible (in number and type)
with the `value <InputState.value>` of the LearningMechanism's `InputStates <LearningMechanism_InputStates>`.  Each
item in **error_sources** must be one of the following: a `ComparatorMechanism`, for `single layer learning
<LearningMechanism_Single_Layer_Learning>` or for the last `MappingProjection` in a learning sequence for `multilayer
learning <LearningMechanism_Multilayer_Learning>`; or a `LearningMechanism`.

.. _LearningMechanism_Learning_Signals:

When a LearningMechanism is created explicitly, it can also be assigned existing LearningSignals and/or specified to
create these, as well as `LearningProjections <LearningProjection>` from these to specified MappingProjections.  These
are specified in the **learning_signals** argument of the LearningMechanism's constructor, using any of the forms
allowed for `specifying a LearningSignal <LearningSignal_Specification>`.

.. _LearningMechanism_Structure:

Structure
---------

A LearningMechanism has three types of `InputStates <InputState>`, a learning `function <LearningMechanism.function>`,
and two types of `OutputStates <OutputState>`. These are used, respectively, to receive, compute, and transmit the
information needed to modify the MappingProjection(s) for which the LearningMechanism is responsible.  In addition,
it has several attributes that govern and provide access to its operation.  These are described below.

.. _LearningMechanism_InputStates:

*InputStates*
~~~~~~~~~~~~~

These receive the information required by the LearningMechanism's `function <LearningMechanism.function>`.  They are
listed in the LearningMechanism's `input_states <LearningMechanism.input_states>` attribute.  They have the following
names and roles (shown in the `figure <LearningMechanism_Single_Layer_Learning_Figure>` below):

.. _LearningMechanism_Activation_Input:

* *ACTIVATION_INPUT* - receives the value of the input to the `primary_learned_projection`; that is, the
  `value <MappingProjection.value>` of that MappingProjection's `sender <MappingProjection.sender>`.
  The value is assigned as the first item of the LearningMechanism's `variable <LearningMechanism.variable>` attribute.

.. _LearningMechanism_Activation_Output:

* *ACTIVATION_OUTPUT* - receives the value of the LearningMechanism's `output_source <LearningMechanism.output_source>`;
  that is, the `value <OutputState.value>` of the `OutputState` of the *ProcessingMechanism* to which the
  `primary_learned_projection` projects.  By default, the `output_source <LearningMechanism.output_source>`'s
  `primary OutputState <OutputState_Primary>` is used.  However, a different OutputState can be designated in
  the constructor for the `output_source <LearningMechanism.output_source>`, by assigning a `parameter specification
  dictionary <ParameterState_Specification>` to the **params** argument of its constructor, with an entry that uses
  *MONITOR_FOR_LEARNING* as its key and a list containing the desired OutputState(s) as its value. The `value
  <InputState.value>` of the *ACTIVATION_OUTPUT* InputState is assigned as the second item of the LearningMechanism's
  `variable <LearningMechanism.variable>` attribute.

.. _LearningMechanism_Input_Error_Signal:

* *ERROR_SIGNAL* - this receives the `value <OutputState.value>` from the *OUTCOME* `OutputState
  <ComparatorMechanism_Structure>` of a `ComparatorMechanism`, or of the *ERROR_SIGNAL* OutputState of another
  `LearningMechanisms <LearningMechanism_Output_Error_Signal>`. If the `primary_learned_projection` projects
  to the `TERMINAL` Mechanism of the Process or System being learned, or is not part of a `multilayer learning
  sequence <LearningMechanism_Multilayer_Learning>`, then the LearningMechanism has a single *ERROR_SIGNAL* InputState,
  that receives its input from a ComparatorMechanism. If the `primary_learned_projection` is part of a `multilayer
  learning sequence <LearningMechanism_Multilayer_Learning>`, then the LearningMechanism will have one or more
  *ERROR_SIGNAL* InputStates, that receive their input from the next LearningMechanism(s) in the sequence;  that is,
  the one(s) associated with the `efferents <OutputState.efferents>` (outgoing Projections) of its `output_source`,
  with one *ERROR_SIGNAL* InputState for each of those Projections.  The `value <InputState.value>`\\s of the
  *ERROR_SIGNAL* InputStates are summed by the LearningMechanism's `function <LearningMechanism.function>` to
  calculate the `learning_signal <LearningMechanism.learning_signal>` (see `below <LearningMechanism_Function>`);
  note that the value of the *ERROR_SIGNAL* InputState may not be the same as that of the LearningMechanism's
  `error_signal <LearningMechanism.error_signal>` attribute or *ERROR_SIGNAL* `OutputState
  <LearningMechanism_Output_Error_Signal>` (see `note <LearningMechanism_Error_Signal>` below).  If a LearningMechanism
  has more than one *ERROR_SIGNAL* InputState, their names are suffixed with a hyphenated index, that is incremented for
  each additional InputState (e.g., ``error_signal-1``, ``error_signal-2``, etc.).  These are listed in the
  LearningMechanism's `error_signal_input_states` attribute, and the `value <InputState.value>` of each is assigned
  as an item of the LearningMechanism's `variable <LearningMechanism.variable>` attribute, beginning with its third
  item (i.e., following the `value <InputState.value>` of the *ACTIVATION_INPUT* and *ACTIVATION_VALUE* InputStates).

The Mechanisms from the which the `value <InputState.values>`\\s above are received are listed in the
LearningMechanism's `input_source <LearningMechanism.input_source>`, `output_source <LearningMechanism.output_source>`,
and `error_sources <LearningMechanism.error_sources>` attributes, respectively (see
`LearningMechanism_Additional_Attributes` for additional details).

.. _LearningMechanism_Function:

*Learning Function*
~~~~~~~~~~~~~~~~~~~

The `function <LearningMechanism.function>` of a LearningMechanism uses the values received by the Mechanism's
InputStates (described `above <LearningMechanism_InputStates>`) to calculate the value of its `learning_signal
<LearningMechanism.learning_signal>` and `error_signal <LearningMechanism.error_signal>` attributes.

.. _LearningMechanism_Learning_Signal:

* `learning_signal` - the set of changes to the `matrix <MappingProjection.matrix>` parameter of the
  `MappingProjections <MappingProjection>` being learned, calculated to reduce the summed value of the
  LearningMechanism's *ERROR_SIGNAL* `InputState(s) <LearningMechanism_Input_Error_Signal>`.

.. _LearningMechanism_Error_Signal:

* `error_signal <LearningMechanism.error_signal>` - the contribution made by the `primary_learned_projection` to the
  error_signal(s) received by the LearningMechanism's *ERROR_SIGNAL* `InputState(s)
  <LearningMechanism_Input_Error_Signal>`. It is used by the LearningMechanism's `function <LearningMechanism.function>`
  to calculate the `learning_signal <LearningMechanism.learning_signal>`. Depending upon the context and specific
  `LearningFunction` used, it may also take account of the `value <Mechanism_Base.value>` of its `output_source`,
  as well as the `matrix <MappingProjection.matrix>` parameter of any of the `output_source`'s outgoing Projections
  that are also being learned (these are listed in the LearningMechanism's `error_matrices
  <LearningMechanism.error_matrices>` attribute).  The value of the `error_signal <LearningMechanism.error_signal>`
  is assigned as the value of the LearningMechanism's *ERROR_SIGNAL* `OutputState
  <LearningMechanism_Output_Error_Signal>`.

  .. _LearningMechanism_Error_Signal_Note:

  .. note::

     A LearningMechanism's *ERROR_SIGNAL* `InputState(s) <LearningMechanism_Input_Error_Signal>` and its
     *ERROR_SIGNAL* `OutputState <LearningMechanism_Output_Error_Signal>` may not have the same value.
     The former are the error signal(s) received from a `ComparatorMechanism` or one or more `LearningMechanisms
     <LearningMechanism>`, while the latter is the contribution made to those errors by the `primary_learned_projection`
     and the `output_source`, as calculated by the LearningMechanism's `function <LearningMechanism.function>`
     (see `error_signal <LearningMechanism_Error_Signal>` above).

The default `function <LearningMechanism.function>` of a LearningMechanism is `BackPropagation` (also known as the
*Generalized Delta Rule*; see
`Rumelhart et al., 1986 <http://www.nature.com/nature/journal/v323/n6088/abs/323533a0.html>`_).  However, it can be
assigned any other PsyNeuLink `LearningFunction`, or any other Python function that takes as its input a list or
np.array containing three lists or 1d np.arrays of numeric values, and returns two lists or 1d np.arrays.  The two
values it returns are assigned to the LearningMechanism's `learning_signal <LearningMechanism.learning_signal>` and
`error_signal <LearningSignal.error_signal>` attributes, respectively, as well as to its two OutputStates,
as described below.

.. _LearningMechanism_OutputStates:

*OutputStates*
~~~~~~~~~~~~~~

By default, a LearningMechanism has two `OutputStates <OutputState>`, the first of which is named *ERROR_SIGNAL* and
is assigned the value of the `error_signal <LearningMechanism.error_signal>` returned by the LearningMechanism's
`function <LearningMechanism.function>`, and the second of which is a `LearningSignal` and is assigned the value of the
`learning_signal <LearningMechanism.learning_signal>` returned by the `function <LearningMechanism.function>`.
They are each described below:

.. _LearningMechanism_Output_Error_Signal:

* *ERROR_SIGNAL* - this is the `primary OutputState <OutputState_Primary>` of a LearningMechanism, and  receives the
  value of the `error_signal <LearningMechanism.error_signal>` used to calculate the `learning_signal
  <LearningMechanism.learning_signal>`.  Its value is assigned as the first item of the LearningMechanism's
  `output_values <LearningMechanism.output_values>` attribute.  If the LearningMechanism is part of a `multilayer
  learning sequence <LearningMechanism_Multilayer_Learning>`, the *ERROR_SIGNAL* OutputState is assigned a Projection
  to the LearningMechanism for the preceding MappingProjection in the sequence being learned - see `figure
  <LearningMechanism_Multilayer_Learning_Figure>` below).  Note that the `value <OutputState.value>` of the
  *ERROR_SIGNAL* OutputState may not be the same as that of the LearningMechanism's *ERROR_SIGNAL* `InputStates
  <LearningMechanism_Input_Error_Signal>` (see `error_signal <LearningMechanism_Error_Signal>`).

.. _LearningMechanism_LearningSignal:

* `LearningSignal(s) <LearningSignal>` - by default, a LearningMechanism has a single LearningSignal, which is a
  special type of OutputState that receives the `learning_signal <LearningMechanism.learning_signal>` generated by the
  LearningMechanism's `function <LearningMechanism.function>`, and used to modify the `matrix
  <MappingProjection.matrix>` parameter of the `primary_learned_projection`.  The LearningSignal is assigned as the
  second item in the list of the LearningMechanism's OutputStates (i.e., of its `output_states
  <LearningMechanism.output_states>` attribute), and its `value <LearningSignal.value>` is assigned as the second
  item of the LearningMechanism's `output_values <LearningMechanism.output_values>` attribute.

.. _LearningMechanism_Multiple_LearningSignals:

  **Multiple LearningSignals and LearningProjections.** Though not common, it is possible for a LearningMechanism to
  be assigned more than one `LearningSignal`, and/or more than one `LearningProjection` to its LearningSignal(s). This
  allows multiple MappingProjections to be trained by a single LearningMechanism. Note, however, that all of the
  LearningSignals of a LearningMechanism (and therefore its LearningProjections) use the same
  `learning_signal <LearningMechanism.learning_signal>`, that is calculated based on the LearningMechanism's
  `primary_learned_projection` and its associated `output_source`.  This can be useful in some settings, such as for
  certain forms of `convolutional neural networks <https://en.wikipedia.org/wiki/Convolutional_neural_network>`_.

  If all of the LearningProjections are used to implement the same form of `modulation <ModulatorySignal_Modulation>`,
  (determined by their LearningSignals' `modulation <LearningSignal.modulation>` attribute), then they should be
  assigned to a single LearningSignal. Multiple LearningProjections can be assigned to a LearningSignal by specifying
  them in the **projections** argument of its constructor, or the *PROJECTIONS* entry of a dictionary assigned to its
  **params** argument); however, the `matrix <MappingProjection.matrix>` parameter for the MappingProjection to which
  they project must have the same shape for all of them. If different forms of modulation are required to train
  different MappingProjections, then multiple LearningSignals should be assigned to the LearningMechanism, each
  specified for one of the required types of modulation, and then LearningProjections assigned to them for the
  corresponding MappingProjections. Multiple LearningSignals can be specified for a LearningMechanism by including
  them in a list assigned to the **learning_signals** argument of the LearningMechanism's constructor.

  The `learning_rate <LearningSignal.learning_rate>` for each LearningSignal, and the `learning_rate
  <LearningProjection.learning_rate>` for each of its `LearningProjections <LearningProjection>`, can all be assigned
  different values (with the latter taking precedence over the former).  If none of these are specified, the
  `learning_rate <LearningMechanism.learning_rate>` of the LearningMechanism is used (see `below
  <LearningMechanism_Learning_Rate>`).

  All of the LearningSignals of a LearningMechanism are listed in its `learning_signals` attribute.  Because these
  are `OutputStates <OutputState>`, they are also listed in the `output_states <LearningMechanism.output_states>`
  attribute, after the *ERROR_SIGNAL* OutputState.  All of the LearningMechanism's LearningProjections (that is, those
  belonging to all of its LearningSignals) are listed in its `learning_projections` attribute.

.. _LearningMechanism_Additional_Attributes:

*Additional Attributes*
~~~~~~~~~~~~~~~~~~~~~~~

In addition to its `InputStates <LearningMechanism_InputStates>`, `function <LearningMechanism_Function>` and
`OutputStates <LearningMechanism_OutputStates>`, a LearningMechanism has the following attributes that
refer to the Components being learned and/or its operation:

.. _LearningMechanism_Primary_Learned_Projection:

* `primary_learned_projection` - the `MappingProjection` with the `matrix <MappingProjection.matrix>` parameter for
  which the `learning_signal <LearningMechanism.learning_signal>` is computed.  This is always the first Projection
  listed first in LearningMechanism's `learned_projections` attribute.
..
* `learned_projections` - a list of all of the MappingProjections to which the LearningMechanism sends a
  `LearningProjection`, listed in the order of the `LearningSignal(s) <LearningSignal>` to which they belong,
  as those are listed in the LearningMechanism's `learning_signals <LearningMechanism.learning_signals>` attribute.
..
* `learning_enabled <LearningMechanism.learning_enabled>` - if set to `False`, learning is disabled for all of its
  `LearningProjections <LearningProjection>`;  however, the LearningMechanism is still executed during the learning
  phase of execution of a `Process <Process_Execution>` or `System  <System_Execution_Learning>`, so that the error
  signals it calculates can be passed to any other LearningMechanism(s) to which it projects (see
  `LearningMechanism_Multilayer_Learning`).
..
* `input_source` - the `Mechanism <Mechanism>` that sends the `primary_learned_projection`, and projects to the
  LearningMechanism's *ACTIVATION_INPUT* `InputState <LearningMechanism_Activation_Input>`.
..
* `output_source` - the `Mechanism <Mechanism>` that receives the `primary_learned_projection`, and  provides the
  input to the LearningMechanism's *ACTIVATION_OUTPUT* `InputState <LearningMechanism_Activation_Output>`.
..
* `error_sources` - a `ComparatorMechanism`, `LearningMechanism`, or list of them that calculate the error signal(s)
  provided to the LearningMechanism's *ERROR_SIGNAL(s)* `InputState(s) <LearningMechanism_Input_Error_Signal>`.
..
* `error_matrices` - the `matrix <MappingProjection.matrix>` parameters of the Projections associated with the
  `error_sources <LearningMechanism.error_sources>`;  that is, of any of the `output_source
  <LearningMechanism.output_source>`'s `efferents <OutputStates.efferents>` that are also being learned.
..
* `modulation` - the default value used for the `modulation <LearningSignal.modulation>` attribute of
  LearningMechanism's `LearningSignals <LearningSignal>` (i.e. those for which it is not explicitly specified).
  This determines the way in which the `learning_signal <LearningMechanism.learning_signal>` is used to modify the
  `matrix <MappingProjection.matrix>` parameter of the `learned_projections`.  By default its value is
  Modulation.ADD, which causes the weight changes in the `learning_signal` to be added to the current value of the
  `matrix <MappingProjection.matrix>` parameter (see `LearningMechanism_Execution` for a description of how the
  modifications are executed).

.. _LearningMechanism_Learning_Rate:

* `learning_rate <LearningMechanism.learning_rate>` - specifies the :keyword:`learning_rate` parameter used by the
  LearningMechanism's `function <LearningMechanism.function>`, which uses it to multiply the weight change matrix
  before returning it as the `learning_signal <LearningMechanism.learning_signal>`.  This can be specified in the
  **learning_rate** argument of the LearningMechanism's constructor (or the constructor for its `function
  <LearningMechanism.function>`;  doing so supersedes specification of the `learning_rate <Process.learning_rate>`
  for any `Process`, and/or the `learning_rate <System.learning_rate>` for any `System` to which the
  LearningMechanism belongs.  The default value for a LearningMechanism's `learning_rate
  <LearningMechanism.learning_rate>` attribute is `None`, in which case the LearningMechanism (and its `function
  <LearningMechanism.function>`) inherit the specification of the `learning_rate <Process.learning_rate>` for
  the `Process` in which the LearningMechanism is executed. If that is `None`, then it inherits the `learning_rate
  <System.learning_rate>` specification of the `System` in which it is executed.  If that is also `None`, then it
  uses the value of the `default_learning_rate <LearningFunction.default_learning_rate>` parameter of its `function
  <LearningMechanism.function>`. A :keyword:`learning_rate` parameter can also be specified for individual
  `LearningSignals <LearningSignal>` and/or their associated `LearningProjections <LearningProjection>`.  Those have
  a direct multiplicative effect on the `learning_signal <LearningProjection.learning_signal>` of the LearningSignal
  and/or it LearningProjections (see `LearningSignal learning_rate <LearningSignal_Learning_Rate>` for additional
  details).

.. _LearningMechanism_Learning_Configurations:

COMMENT:
@@@ THE FOLLOWING SECTIONS SHOULD BE MOVED TO THE "USER'S MANUAL" WHEN THAT IS WRITTEN
COMMENT

*Learning Configurations*
~~~~~~~~~~~~~~~~~~~~~~~~~

When learning is specified for a `MappingProjection <Mapping_Matrix_Specification>`, a `Process
<Process_Learning_Sequence>`, or a `System <System_Execution_Learning>`, all of the Components required for learning are
created automatically. The types of Components that are generated depend on the `LearningFunction <LearningFunction>`
specified, and the configuration of the `Composition <Composition>`, as described below.  If learning is `specified for
individual Projections <Process_Learning_Specification>` in the `pathway <Process.pathway>` of a Process, it takes
effect only if that Process is executed on its own (i.e., using the Process' `execute <Process.execute>` or `run
<Process.run>` methods.  For the learning Components associated with a Process to be implemented by a System,
learning must be `specified for the entire Process <Process_Learning_Specification>`.  All of the learning Components
of a System can be displayed using the System's `show_graph` method with its **show_learning** argument assigned `True`
or *ALL*.

.. _LearningMechanism_Single_Layer_Learning:

Single layer learning
^^^^^^^^^^^^^^^^^^^^^

This configuration occurs when only a single `MappingProjection` is `specified for learning <Projection_Specification>`,
or the LearningMechanism's `function <LearningMechanism.function>` only considers the error on its `output_source`
(and not any additional sources of error) when calculating the `learning_signal <LearningMechanism.learning_signal>`
(e.g., for `Reinforcement` learning).  In this case, a single `ComparatorMechanism` and LearningMechanism are created
(if they do not already exist);  in addition, the following MappingProjections are created:

* from an `OutputState` of the LearningMechanism's `output_source` to the ComparatorMechanism's *SAMPLE* `InputState
  <ComparatorMechanism_Structure>`.  By default, the `primary OutputState <OutputState_Primary>` of the
  `output_source` is used; however, this can be modified by specifying its *MONITOR_FOR_LEARNING* parameter
  (see `above <LearningMechanism_Activation_Output>`).
..
* from the Process or System to the ComparatorMechanism's *TARGET* `InputState <ComparatorMechanism_Structure>`;
..
* from the ComparatorMechanism's *OUTCOME* `OutputState <ComparatorMechanism_Structure>` to the
  LearningMechanism's *ERROR_SIGNAL* `InputState <LearningMechanism_Activation_Input>`.

In addition, a `LearningProjection` is created from the `LearningSignal<LearningMechanism_LearningSignal>` for the
`primary_learned_projection` to the `ParameterState` for the `matrix <MappingProjection.matrix>` of the
`primary_learned_projection`.  Because this configuration involves only a single layer of learning, *no* Projection
is created or assigned to the LearningMechanism's *ERROR_SIGNAL* `OutputState <LearningMechanism_Output_Error_Signal>`.

.. _LearningMechanism_Single_Layer_Learning_Learning_Figure:

    **Components for Single Layer Learning**

    .. figure:: _static/LearningMechanism_Single_Layer_Learning_fig.svg
       :alt: Schematic of Mechanisms and Projections involved in learning for a single MappingProjection
       :scale: 150 %

       ComparatorMechanism, LearningMechanism and associated Projections created for the `primary_learned_projection`
       and `output_source`.  Each Mechanism is labeled by its type (upper line, in bold) and its designated
       status in the Process and/or System to which it belongs (lower line, capitalized).  Italicized label beside a
       component indicates the attribute of the LearningMechanism with which it is associated.


.. _LearningMechanism_Multilayer_Learning:

Multilayer learning
^^^^^^^^^^^^^^^^^^^

This configuration occurs when a set of `MappingProjections <MappingProjection>` being learned are in a sequence (such
as the `pathway <Process.pathway>` of a `Process`); that is, in which each projects to a `ProcessingMechanism
<ProcessingMechanism>` that is the `sender <MappingProjection.sender>` for the next MappingProjection(s) in the sequence
(see the `figure <LearningMechanism_Multilayer_Learning_Figure>` below). This requires the use of a learning function
that can calculate the influence that each MappingProjection and its output have on the error that the LearningMechanism
receives from the next one(s) in the sequence (e.g., `BackPropagation`). In multilayer learning, the Components created
depend on the position of the `primary_learned_projection` and `output_source <LearningMechanism.output_source>` in
the sequence.  If these are the last ones in the sequence, they are treated in the same way as `single layer
learning <LearningMechanism_Single_Layer_Learning>`.  This is the case if the `output_source` is a standalone Mechanism
(one not in a `Process` or `System`), the `TERMINAL` Mechanism of a standalone Process (i.e., one not in a System),
or the `TERMINAL` of all of the Processes to which it belongs in a System (and therefore a `TERMINAL` for the
System).  In these cases, as for single layer learning, a `ComparatorMechanism` is created that receives the output
of the `output_source` as well as the target for learning (see `TARGET Mechanisms <LearningMechanism_Targets>` below),
and projects to a LearningMechanism that is created for the `primary_learned_projection`.  For all other
MappingProjections being learned in the sequence, the following additional MappingProjections are created for
learning (shown in the `figure <LearningMechanism_Multilayer_Learning_Figure>` below):

* from the `input_source <LearningMechanism.input_source>` to the LearningMechanism's *ACTIVATION_INPUT* `InputState
  <LearningMechanism_Activation_Input>`.
..
* from the `output_source <LearningMechanism.output_source>` to the LearningMechanism's *ACTIVATION_OUTPUT* `InputState
  <LearningMechanism_Activation_Output>`.
..
* from the *ERROR_SIGNAL* `OutputState <LearningMechanism_Output_Error_Signal>` of each of the LearningMechanism's
  `error_sources <LearningMechanisms.error_sources>` to each of its corresponding *ERROR_SIGNAL* `InputState(s)
  <LearningMechanism_Input_Error_Signal>`.

In addition, a `LearningProjection` is created from the `LearningSignal <LearningMechanism_LearningSignal>` for the
`primary_learned_projection` of each LearningMechanism in the sequence, to the `ParameterState` for the `matrix
<MappingProjection.matrix>` of the `primary_learned_projection`.  If the `primary_learned_projection` is the first in
the sequence, then *no* Projection is created or assigned to its LearningMechanism's *ERROR_SIGNAL* `OutputState
<LearningMechanism_Output_Error_Signal>`.

.. _LearningMechanism_Multilayer_Learning_Figure:

    **Components for Multilayer Learning**

    .. figure:: _static/LearningMechanism_Multilayer_Learning_fig.svg
       :alt: Schematic of Mechanisms and Projections involved in learning for a sequence of MappingProjections
       :scale: 50%

       ComparatorMechanism and LearningMechanisms and associated Projections created for a sequence of two
       MappingProjections specified for learning.  Each Mechanism is labeled by its type (upper line, in bold) and
       its designated status in the Process and/or System to which it belongs (lower line, capitalized).  Italicized
       label beside a component indicates the attribute of the LearningMechanism with which it is associated.

.. _LearningMechanism_Targets:

**TARGET Mechanisms**. When a `LearningFunction` is specified for a LearningMechanism that requires a target (e.g.,
`BackPropagation` or `Reinforcement`), a `ComparatorMechanism` must be specified to receive the target.  For `multilayer
learning <LearningMechanism_Multilayer_Learning>`, this is the `error_source <LearningMechanism.error_sources>` for the
last MappingProjection in each learning sequence.  When learning is specified for a `Composition <Composition>` (i.e.,
a `Process <Process_Learning_Sequence>` or a `System <System_Execution_Learning>`), the `ComparatorMechanism(s)
<ComparatorMechanism>` that receive the `targets <Run_Targets>`  are identified and designated as `TARGET` Mechanisms,
and are listed in the Composition's `target_nodes` attribute. If a `TERMINAL` Mechanism of a Composition receives a
MappingProjection that is specified for learning, then it always projects to a `TARGET` Mechanism in that Composition.
It is important to note, in this context, the status of a Mechanism in a System takes precedence over its status in any
of the Processes to which it belongs. This means that even if a Mechanism is the `TERMINAL` of a particular Process, if
that Process is combined with others in a System, and the Mechanism appears in any of those other Processes, and it is
not the `TERMINAL` of *all* of them, then it will *not* be the `TERMINAL` for the System.  As a consequence, although it
will project to a `TARGET` Mechanism in the Process for which it is the `TERMINAL`, it will not do so in the System,
and its LearningMechanism will receive its error_signal from another LearningMechanism rather than a
ComparatorMechanism (see `figure below <LearningProjection_Target_vs_Terminal_Figure>` for an example).  Finally,
if a Mechanisms is the `TERMINAL` for more than one Process used to create a System (that is, the pathways for those
Processes converge on that Mechanism), only one ComparatorMechanism will be created for it in the System.

.. _LearningProjection_Target_vs_Terminal_Figure:

    **TERMINAL** and **TARGET** Mechanisms in Learning

    .. figure:: _static/LearningMechanism_TERMINAL_vs_TARGET_fig.svg
       :alt: Schematic of Mechanisms and Projections involved in learning
       :scale: 50 %

       Mechanism 3 is the `TERMINAL` Mechanism for Process A, However, it is also an `INTERNAL` Mechanism of Process B.
       Therefore, Mechanism 3 is designated as an `INTERNAL` Mechanism for the System, and Mechanism 4 is its `TERMINAL`
       Mechanism. As a consequence, if `BackPropagation` is used for learning, then Mechanism 4 will project to a
       `TARGET` Mechanism of the System, while Mechanism 3 will not.

.. _LearningMechanism_Execution:

Execution
---------

LearningMechanisms are executed after all of the `ProcessingMechanisms <ProcessingMechanism>` in the `Process` or
`System` to which they belong have been executed.  When a LearningMechanism is executed, it uses the `value
<InputState.value>`(s) of its *ERROR_SIGNAL* `InputState(s) <LearningMechanism_Input_Error_Signal>` to calculate changes
to the weights of the `matrix <MappingProjection.MappingProjection.matrix>` parameter of its `primary_learned_projection
<LearningMechanism.primary_learned_projection>` and any of its other `learned_projections
<LearningMechanis.learned_projections>`. Those weight changes are assigned as the LearningMechanism's
`learning_signal <LearningMechanism.learning_signal>` attribute, the `value <LearningSignal.value>` of each of its
`LearningSignals <LearningMechanism_LearningSignal>`, and as the `value <LearningProjection.value>` of each of their
LearningProjections.  That value is used, in turn, to modify the `value <ParameterState.value>` of the *MATRIX*
`ParameterState` of each of the MappingProjections being learned (listed in the LearningMechanism's
`learned_projections <LearningMechanism.learned_projections>` attribute).

Each ParameterState uses the value it receives from the `LearningProjection` that projects to it to modify the
parameter of its `function <ParameterState.function>`, in a manner specified by the `modulation
<LearningSignal.modulation>` attribute of the `LearningSignal` from which it receives the LearningProjection (see
`Modulation <ModulatorySignal_Modulation>` for a description of how modulation operates). By default, the
`modulation <LearningSignal.modulation>` attribute of a LearningSignal is `ModulationParam.ADDITIVE`,
the `function <ParameterState.function>` of a *MATRIX* ParameterState for a MappingProjection is
`AccumulatorIntegrator`, and the parameter it uses for additive modulation is its `increment
<AccumulatorIntegrator.increment>` parameter.  These assignments cause the value of the LearningProjection to be added
to the previous value of the *MATRIX* ParameterState, thus incrementing the weights by an amount specified by the
LearningMechanism's `learning_signal <LearningMechanism.learning_signal>`. Note, that the changes to the `matrix
<MappingProjection.MappingProjection.matrix>` parameter itself do not take effect
until the next time the `learned_projection` is executed (see :ref:`Lazy Evaluation <LINK>` for an explanation of
"lazy" updating).

A LearningMechanism's `function <LearningMechanism.function>` also computes an `error_signal
<LearningMechanism.error_signal>` that is assigned as the `value <OutputState.value>` of its *ERROR_SIGNAL*
`OutputState <LearningMechanism_Output_Error_Signal>`;  in a `multilayer learning configuration
<LearningMechanism_Multilayer_Learning>`, that value is provided to the *ERROR_SIGNAL* `InputState(s)
<LearningMechanism_Input_Error_Signal>` of the LearningMechanism(s) for the preceding MappingProjection(s)
being learned in the sequence.

.. _LearningMechanism_Class_Reference:

Class Reference
---------------

"""

import numpy as np
import typecheck as tc

from enum import Enum

from psyneulink.core.components.component import parameter_keywords
from psyneulink.core.components.functions.function import ModulationParam, _is_modulation_param, is_function_type
from psyneulink.core.components.functions.learningfunctions import BackPropagation
from psyneulink.core.components.mechanisms.adaptive.adaptivemechanism import AdaptiveMechanism_Base
from psyneulink.core.components.mechanisms.mechanism import Mechanism_Base
from psyneulink.core.components.mechanisms.processing.objectivemechanism import ObjectiveMechanism
from psyneulink.core.components.shellclasses import Mechanism
from psyneulink.core.components.states.inputstate import InputState
from psyneulink.core.components.states.modulatorysignals.learningsignal import LearningSignal
from psyneulink.core.components.states.parameterstate import ParameterState
from psyneulink.core.globals.context import ContextFlags
from psyneulink.core.globals.keywords import ASSERT, CONTROL_PROJECTIONS, ENABLED, INPUT_STATES, LEARNED_PARAM, LEARNING, LEARNING_MECHANISM, LEARNING_PROJECTION, LEARNING_SIGNAL, LEARNING_SIGNALS, MATRIX, NAME, OUTPUT_STATE, OUTPUT_STATES, OWNER_VALUE, PARAMS, PROJECTIONS, SAMPLE, STATE_TYPE, VARIABLE
from psyneulink.core.globals.parameters import Parameter
from psyneulink.core.globals.preferences.componentpreferenceset import is_pref_set
from psyneulink.core.globals.preferences.preferenceset import PreferenceLevel
from psyneulink.core.globals.utilities import ContentAddressableList, is_numeric, parameter_spec

__all__ = [
    'ACTIVATION_INPUT', 'ACTIVATION_INPUT_INDEX', 'ACTIVATION_OUTPUT', 'ACTIVATION_OUTPUT_INDEX',
    'DefaultTrainingMechanism', 'ERROR_OUTPUT_INDEX', 'ERROR_SIGNAL', 'ERROR_SIGNAL_INDEX', 'ERROR_SOURCES',
    'LearningMechanism', 'LearningMechanismError', 'input_state_names', 'output_state_names'
]


def _is_learning_spec(spec, include_matrix_spec=True):
    """Evaluate whether spec is a valid learning specification

    Return `True` if spec is LEARNING or a valid projection_spec (see Projection_Base._is_projection_spec)
    Otherwise, return `False`

    """
    # MODIFIED 11/28/17 OLD:
    from psyneulink.core.components.projections.projection import _is_projection_spec

    try:
        if spec in {LEARNING, ENABLED}:
            return True
        else:
            return _is_projection_spec(spec=spec,
                                       type=LEARNING_PROJECTION,
                                       include_matrix_spec=include_matrix_spec)
    except:
        return False

class LearningType(Enum):
    """
        Denotes whether LearningMechanism requires a target input.

    Attributes
    ----------

    UNSUPERVISED
        implements (and requires a Projection to) a *ERROR_SIGNAL* InputState.

    SUPERVISED
        does not implement a *ERROR_SIGNAL* InputState.

    """
    UNSUPERVISED = 0
    SUPERVISED = 1


class LearningTiming(Enum):
    """
        Denotes

    Attributes
    ----------

    EXECUTION_PHASE
        LearningMechanism (and associated `LearningProjections(s) <LearningProjection>`) executed during the
        `execution phase <System_Execution>` of the System to which they belong, usually immediately after execution of
        the `Mechanism` that receives the `primary_learned_projection`

    LEARNING_PHASE
        LearningMechanism (and associated `LearningProjections(s) <LearningProjection>`) executed during the
        `learning phase <System_Execution>` of the System to which they belong.

    """
    EXECUTION_PHASE = 0
    LEARNING_PHASE = 1


# Parameters:

parameter_keywords.update({LEARNING_PROJECTION, LEARNING})

LEARNING_TYPE = 'learning_type'
LEARNING_TIMING = 'learning_timing'

# Used to index variable:
ACTIVATION_INPUT_INDEX = 0
ACTIVATION_OUTPUT_INDEX = 1
ERROR_OUTPUT_INDEX = 2
ERROR_SIGNAL_INDEX = 3

# Used to name input_states and output_states:
ACTIVATION_INPUT = 'activation_input'     # InputState
ACTIVATION_OUTPUT = 'activation_output'   # InputState
ERROR_SIGNAL = 'error_signal'
input_state_names =  [ACTIVATION_INPUT, ACTIVATION_OUTPUT, ERROR_SIGNAL]
output_state_names = [LEARNING_SIGNAL, ERROR_SIGNAL]

ERROR_SOURCES = 'error_sources'

DefaultTrainingMechanism = ObjectiveMechanism


class LearningMechanismError(Exception):
    def __init__(self, error_value):
        self.error_value = error_value

    def __str__(self):
        return repr(self.error_value)


def _learning_signal_getter(owning_component=None, execution_id=None):
    try:
        return owning_component.parameters.value.get(execution_id)[0]
    except (TypeError, IndexError):
        return None


def _error_signal_getter(owning_component=None, execution_id=None):
    try:
        return owning_component.parameters.value.get(execution_id)[1]
    except (TypeError, IndexError):
        return None


class LearningMechanism(AdaptiveMechanism_Base):
    """
    LearningMechanism(                             \
        variable,                                  \
        error_sources,                             \
        function=BackPropagation,                  \
        learning_rate=None,                        \
        learning_signals=LEARNING_SIGNAL,          \
        modulation=ModulationParam.ADDITIVE,       \
        params=None,                               \
        name=None,                                 \
        prefs=None)

    Implements a Mechanism that modifies the `matrix <MappingProjection.matrix>` parameter of a `MappingProjection`.

    COMMENT:
        Description:
            LearningMechanism is a subtype of the AdaptiveMechanism Type of the Mechanism Category of Component
            It implements a Mechanism that calculates changes to a Projection's parameters.
            Its function takes the output of an ObjectiveMechanism and generates a
            learning_signal (ndarray of parameter changes) to be used by the recipient of a LearningProjection
            that projects from the LearningMechanism to a MappingProjection.

        # DOCUMENT: ??NOT SURE WHETHER THIS IS STILL RELEVANT
        #    IF objective_mechanism IS None, IT IS LEFT UNSPECIFIED (FOR FURTHER IMPLEMENTATION BY COMPOSITION)
        #    THESE ARE HANDLED BY A MODULE METHOD _instantiate_objective_mechanism (AS PER OBJECTIVE MECHANISM):
        #        IF objective_mechanism IS SPECIFIED AS ObjectiveMechanism, AN OBJECTIVE MECHANISM IS CREATED FOR IT
        #        IF objective_mechanism IS SPECIFIED AS A MECHANISM OR OUTPUTSTATE,
        #               a MappingProjection WITH AN IDENTITY MATRIX IS IMPLEMENTED FROM IT TO THE LearningMechanism

        Learning function:
            Generalized delta rule:
            dE/dW  =          learning_rate   *    dE/dA          *       dA/dW             *    I
            weight = weight + (learning_rate  * error_derivative  *  activation_derivative  *  input)
            for sumSquared error fct =        (target - output)
            for logistic activation fct =                           output * (1-output)
            where:
                output = activity of output (target) units (higher layer)
                input = activity of sending units (lower layer)
            Needs:
            - activation_derivative:  get from FUNCTION of sample_activation_mechanism/receiver_mech
                                      assumes derivative of Logistic unless otherwise specified
            - error_derivative:  get from FUNCTION of error_sources/next_level_mech;  but handled in ObjectiveMechanism

        Class attributes:
            + className = LEARNING_MECHANISM
            + componentType = ADAPTIVE_MECHANISM
            + paramClassDefaults (dict):
                + FUNCTION (Function): (default: BP)
                + FUNCTION_PARAMS:
                    + LEARNING_RATE (value): (default: 1)
            + classPreference (PreferenceSet): LearningSignalPreferenceSet, instantiated in __init__()
            + classPreferenceLevel (PreferenceLevel): PreferenceLevel.TYPE

        Class methods:
            None

        MechanismRegistry:
            All instances of LearningMechanism are registered in MechanismRegistry, which maintains an
              entry for the subclass, a count for all instances of it, and a dictionary of those instances
    COMMENT

    Arguments
    ---------

    variable : List or 2d np.array
        it must have three items that correspond to the three values required by the LearningMechanism's `function
        <LearningMechanism.function>`;  they must each be compatible (in number and type) with the `value
        <InputState.value>` of the corresponding `InputState <LearningMechanism_InputStates>` (see `variable
        <LearningMechanism.variable>` for additional details).

    error_sources : ComparatorMechanism, LearningMechanism or list of them
        specifies the source(s) of the error signal(s) used by the LearningMechanism's `function
        <LearningMechanism.function>`.  Each must be a `ComparatorMechanism` for `single layer learning
        <LearningMechanism_Single_Layer_Learning>`, or for the last `MappingProjection` in a learning sequence in
        `multilayer learning <LearningMechanism_Multilayer_Learning>`;  otherwise they must be a `LearningMechanism`.

    learning_signals : List[parameter of Projection, ParameterState, Projection, tuple[str, Projection] or dict] :
    default *LEARNING_SIGNAL*
        specifies the parameter(s) to be learned (see `learning_signals <LearningMechanism.learning_signals>` for
        details).

    modulation : ModulationParam : default ModulationParam.ADDITIVE
        specifies the default form of modulation used by the LearningMechanism's LearningSignals,
        unless they are `individually specified <LearningSignal_Specification>`.

    function : LearningFunction or function : default BackPropagation
        specifies the function used to calculate the LearningMechanism's `learning_signal
        <LearningMechanism.learning_signal>` and `error_signal <LearningMechanism.error_signal>` attributes.  It's
        `variable <Function_Base.variable>` must have three items, each of which must be a list or 1d array of
        numeric values, corresponding to values provided by the LearningMechanism's *ACTIVATION_INPUT*,
        *ACTIVATION_OUTPUT*, and *ERROR_SOURCES* InputStates, respectively (see `LearningMechanism_InputStates
        `LearningMechanism_Function` and `LearningMechanism_InputStates` for additional details).

    learning_rate : float : default None
        specifies the learning rate for the LearningMechanism (see `learning_rate <LearningMechanism.learning_rate>`
        for details).

    params : Dict[param keyword: param value] : default None
        a `parameter dictionary <ParameterState_Specification>` that specifies the parameters for the
        Projection, its function, and/or a custom function and its parameters. By default, it contains an entry for
        the Projection's default `function <LearningProjection.function>` and parameter assignments.  Values specified
        for parameters in the dictionary override any assigned to those parameters in arguments of the constructor.

    name : str : default see `name <LearningMechanism.name>`
        specifies the name of the LearningMechanism.

    prefs : PreferenceSet or specification dict : default Mechanism.classPreferences
        specifies the `PreferenceSet` for the LearningMechanism; see `prefs <LearningMechanism.prefs>` for details.


    Attributes
    ----------

    COMMENT:
        componentType : LEARNING_MECHANISM
    COMMENT

    variable : 2d np.array
        has three items that serve as the template for the three inputs required by the LearningMechanism's `function
        <LearningMechanism.function>` (corresponding to its three `InputStates <LearningMechanism_InputStates>`:
        the input to the `primary_learned_projection` (from `input_source`), the output of the Mechanism to which
        that projects (i.e., of `output_source`); and the error signal (from `LearningMechanism.error_sources`).

    input_states : ContentAddressableList[OutputState]
        list containing the LearningMechanism's three `InputStates <LearningMechanism_InputStates>`:
        *ACTIVATION_INPUT*,  *ACTIVATION_OUTPUT*, and *ERROR_SIGNAL*.

    error_signal_input_states : list[InputStates]
        list of InputStates that receive error_signals from the LearningMechanism's `error_sources
        <LearningMechanism.error_sources>`.

    input_source : ProcessingMechanism
        the Mechanism that sends the `primary_learned_projection`, and projects to the
        LearningMechanism's *ACTIVATION_INPUT* `InputState <LearningMechanism_Activation_Input>`.

    output_source : ProcessingMechanism
        the Mechanism that receives the `primary_learned_projection`, and  projects to the
        LearningMechanism's *ACTIVATION_OUTPUT* `InputState <LearningMechanism_Activation_Output>`.

    error_sources : list[ComparatorMechanism or LearningMechanism]
        the Mechanism(s) that calculate the error signal(s) provided to the
        LearningMechanism's *ERROR_SIGNAL(s)* `InputState(s) <LearningMechanism_Input_Error_Signal>`.

    error_matrices : list[ParameterState]
        the matrices of the Projections associated with the `error_sources <LearningMechanism.error_sources>`,
        (i.e., for the next Projection(s) in the learning_sequence, or to the `ComparatorMechanism`);
        note: these are *not* for the LearningMechanism's `learned_projections <LearningMechanism.learned_projections>`.

    primary_learned_projection : MappingProjection
        the Projection with the `matrix <MappingProjection.matrix>` parameter used to generate the
        LearningMechanism's `error_signal <LearningMechanism.error_signal>` and `learning_signal
        <LearningMechanism.learning_signal>` attributes.  It is always the first Projection listed in the
        LearningMechanism's `learned_projections <LearningMechanism.learned_projections>` attribute.

    learned_projections : List[MappingProjection]
        all of the MappingProjections modified by the LearningMechanism;  the first item in the list is always the
        `primary_learned_projection <LearningMechanism.primary_learned_projection>`.

    learning_enabled : bool : True
        if set to False, learning is disabled for all of its LearningProjections; however, the LearningMechanism is
        still executed during the learning phase of execution of a `Process <Process_Execution>` or `System
        <System_Execution_Learning>`, so that the error signals it calculates can be passed to any other
        LearningMechanism(s) to which it projects (see `LearningMechanism_Multilayer_Learning`).

    function : LearningFunction or function : default BackPropagation
        specifies the function used to calculate the `learning_signal <LearningMechanism.learning_signal>` (assigned
        to the LearningMechanism's `LearningSignal(s) <LearningMechanism_LearningSignal>`), and the `error_signal
        <LearningMechanism.error_signal>` (passed to the next LearningMechanism in a learning sequence for
        `multilayer learning <LearningMechanism_Multilayer_Learning>`).  It takes the following three arguments,
        each of which must be a list or 1d array: **input**,  **output**, and **error** (see
        `LearningMechanism_Function` for additional details).

    learning_rate : float : None
        determines the learning rate for the LearningMechanism.  It is used to specify the :keyword:`learning_rate`
        parameter for the LearningMechanism's `learning function <LearningMechanism.function>`
        (see description of `learning_rate <LearningMechanism_Learning_Rate>` for additional details).

    error_signal : 1d np.array
        one of two values returned by the LearningMechanism's `function <LearningMechanism.function>`.  For
        `single layer learning <LearningMechanism_Single_Layer_Learning>`, this is the same as the value received in
        the LearningMechanism's *ERROR_SIGNAL* `InputState <LearningMechanism_Input_Error_Signal>`;  for `multilayer
        learning <LearningMechanism_Multilayer_Learning>`, it is a modified version of the value received, that takes
        account of the contribution made by the learned_projection and its input to the error signal received. This
        is assigned as the `value <OutputState.value>` of the LearningMechanism's *ERROR_SIGNAL* `OutputState
        <LearningMechanism_Output_Error_Signal>`.

    learning_signal : number, ndarray or matrix
        one of two values returned by the LearningMechanism's `function <LearningMechanism.function>`, that specifies
        the changes to the weights of the `matrix <MappingProjection.matrix>` parameter for the LearningMechanism's
        `learned_projections <LearningMechanism.learned_projections>`;  it is calculated to reduce the error signal
        associated with the `primary_learned_projection <LearningMechanism.primary_learned_projection>` and received
        from the LearningMechanism's `error_sources`.  It is assigned as the value of the LearningMechanism's
        `LearningSignal(s) <LearningMechanism_LearningSignal>` and, in turn, its LearningProjection(s).

    learning_signals : ContentAddressableList[LearningSignal]
        list of all of the `LearningSignals <LearningSignal>` for the LearningMechanism, each of which sends one or
        more `LearningProjections <LearningProjection>` to the `ParameterState(s) <ParameterState>` for the `matrix
        <MappingProjection.matrix>` parameter of the `MappingProjection(s) <MappingProjection>` trained by the
        LearningMechanism.  The `value <LearningSignal>` of each LearningSignal is the LearningMechanism's
        `learning_signal <LearningMechanism.learning_signal>` attribute. Since LearningSignals are `OutputStates
        <OutputState>`, they are also listed in the LearningMechanism's `output_states
        <LearningMechanism.output_states>` attribute, after it *ERROR_SIGNAL* `OutputState
        <LearningMechanism_Output_Error_Signal>`.

    learning_projections : List[LearningProjection]
        list of all of the LearningProjections <LearningProject>` from the LearningMechanism, listed in the order of
        the `LearningSignals <LearningSignal>` to which they belong (that is, in the order they are listed in
        the `learning_signals <LearningMechanism>` attribute).

    output_states : ContentAddressableList[OutputState]
        list of the LearningMechanism's `OutputStates <OutputState>`, including its *ERROR_SIGNAL* `OutputState
        <LearningMechanism_Output_Error_Signal>`, followed by its `LearningSignal(s)
        <LearningMechanism_LearningSignal>`, and then any additional (user-specified) `OutputStates <OutputState>`.

    COMMENT:
       #  FIX: THIS MAY NEED TO BE A 3d array (TO ACCOMDATE 2d array (MATRICES) AS ENTRIES)\
    COMMENT

    output_values : 2d np.array
        the first item is the `value <OutputState.value>` of the LearningMechanism's *ERROR_SIGNAL* `OutputState
        <LearningMechanism_Output_Error_Signal>`, followed by the `value <LearningSignal.value>` \\(s) of its
        `LearningSignal(s) <LearningMechanism_LearningSignal>`, and then those of any additional (user-specified)
        `OutputStates <OutputState>`.

    modulation : ModulationParam
        the default form of modulation used by the LearningMechanism's `LearningSignal(s)
        <LearningMechanism_LearningSignal>`, unless they are `individually specified <LearningSignal_Specification>`.

    name : str
        the name of the LearningMechanism; if it is not specified in the **name** argument of the constructor, a
        default is assigned by MechanismRegistry (see `Naming` for conventions used for default and duplicate names).

    prefs : PreferenceSet or specification dict
        the `PreferenceSet` for the LearningMechanism; if it is not specified in the **prefs** argument of the
        constructor, a default is assigned using `classPreferences` defined in __init__.py (see :doc:`PreferenceSet
        <LINK>` for details).

    """

    componentType = LEARNING_MECHANISM
    className = componentType
    suffix = " " + className

    outputStateType = LearningSignal

    stateListAttr = Mechanism_Base.stateListAttr.copy()
    stateListAttr.update({LearningSignal:LEARNING_SIGNALS})

    classPreferenceLevel = PreferenceLevel.TYPE

    class Parameters(AdaptiveMechanism_Base.Parameters):
        """
            Attributes
            ----------

                error_matrix
                    see `error_matrix <LearningMechanism.error_matrix>`

                    :default value: None
                    :type:

                error_signal
                    see `error_signal <LearningMechanism.error_signal>`

                    :default value: None
                    :type:
                    :read only: True

                function
                    see `function <LearningMechanism.function>`

                    :default value: `BackPropagation`
                    :type: `Function`

                learning_enabled
                    see `learning_enabled <LearningMechanism.learning_enabled>`

                    :default value: True
                    :type: bool

                learning_signal
                    see `learning_signal <LearningMechanism.learning_signal>`

                    :default value: None
                    :type:
                    :read only: True

        """
        function = Parameter(BackPropagation, stateful=False, loggable=False)
        error_matrix = Parameter(None, modulable=True)

        learning_signal = Parameter(None, read_only=True, getter=_learning_signal_getter)
        error_signal = Parameter(None, read_only=True, getter=_error_signal_getter)

        learning_enabled = True

    paramClassDefaults = AdaptiveMechanism_Base.paramClassDefaults.copy()
    paramClassDefaults.update({
        CONTROL_PROJECTIONS: None,
        INPUT_STATES:input_state_names,
        OUTPUT_STATES:[{NAME:ERROR_SIGNAL,
                        STATE_TYPE:OUTPUT_STATE,
                        VARIABLE: (OWNER_VALUE, 1)},
                       {NAME:LEARNING_SIGNAL,  # NOTE: This is the default, but is overridden by any LearningSignal arg
                        VARIABLE: (OWNER_VALUE, 0)}
                       ]})

    @tc.typecheck
    def __init__(self,
                 # default_variable:tc.any(list, np.ndarray),
                 default_variable=None,
                 size=None,
                 error_sources:tc.optional(tc.any(Mechanism, list))=None,
                 function=None,
                 learning_signals:tc.optional(list) = None,
                 modulation:tc.optional(_is_modulation_param)=ModulationParam.ADDITIVE,
                 learning_rate:tc.optional(parameter_spec)=None,
                 params=None,
                 name=None,
                 prefs:is_pref_set=None,
                 context=None):

        # IMPLEMENTATION NOTE: THIS SHOULD BE MOVED TO ABC WHEN CREATED
        if context is ContextFlags.CONSTRUCTOR:
            self._check_type_and_timing()
        else:
            self.learning_type = LearningType.SUPERVISED
            self.learning_timing = LearningTiming.LEARNING_PHASE

        if error_sources and not isinstance(error_sources, list):
            error_sources = [error_sources]

        # Assign args to params and functionParams dicts (kwConstants must == arg names)
        params = self._assign_args_to_param_dicts(error_sources=error_sources,
                                                  function=function,
                                                  learning_signals=learning_signals,
                                                  params=params)

        # # USE FOR IMPLEMENTATION OF deferred_init()
        # # Store args for deferred initialization
        # self.init_args = locals().copy()
        # self.init_args['context'] = self
        # self.init_args['name'] = name
        # delete self.init_args[ERROR_SOURCES]

        # # Flag for deferred initialization
        # self.parameters.context.get(execution_id).initialization_status = ContextFlags.DEFERRED_INIT
        # self.initialization_status = ContextFlags.DEFERRED_INIT

        self._learning_rate = learning_rate

        super().__init__(default_variable=default_variable,
                         size=size,
                         modulation=modulation,
                         function=function,
                         params=params,
                         name=name,
                         prefs=prefs,
                         context=ContextFlags.CONSTRUCTOR)

    def _check_type_and_timing(self):
        try:
            self.learning_type
        except:
            raise LearningMechanismError("{} subclass of {} must implement {} attribute".
                                         format(self.__class__.__name__, LearningMechanism.__name__,
                                                repr(LEARNING_TYPE)))
        try:
            self.learning_timing
        except:
            raise LearningMechanismError("{} subclass of {} must implement {} attribute".
                                         format(self.__class__.__name__, LearningMechanism.__name__,
                                                repr(LEARNING_TIMING)))

    def _parse_function_variable(self, variable, execution_id=None, context=None):
        function_variable = np.zeros_like(
            variable[np.array([ACTIVATION_INPUT_INDEX, ACTIVATION_OUTPUT_INDEX, ERROR_OUTPUT_INDEX])]
        )
        function_variable[ACTIVATION_INPUT_INDEX] = variable[ACTIVATION_INPUT_INDEX]
        function_variable[ACTIVATION_OUTPUT_INDEX] = variable[ACTIVATION_OUTPUT_INDEX]
        function_variable[ERROR_OUTPUT_INDEX] = variable[ERROR_OUTPUT_INDEX]

        return function_variable

    def _validate_variable(self, variable, context=None):
        """Validate that variable has exactly three items: activation_input, activation_output and error_signal
        """

        variable = super()._validate_variable(variable, context)

        if len(variable) < 3:
            raise LearningMechanismError("Variable for {} ({}) must have at least three items ({}, {}, and {}{})".
                                         format(self.name, variable,
                                                ACTIVATION_INPUT,
                                                ACTIVATION_OUTPUT,
                                                ERROR_SIGNAL,"(s)"))

        # Validate that activation_input, activation_output are numeric and lists or 1d np.ndarrays
        #    and that there is the correct number of error_signal_input_states and and error_matrices:
        #    (which should be the number of items for error_signals in variable)

        assert ASSERT, "ADD TEST FOR LEN OF VARIABLE AGAINST NUMBER OF ERROR_SIGNALS AND ERROR_MATRICES"

        for i in range(len(variable)):
            item_num_string = "Item {} ".format(i)
            try:
                item_name = self.input_states.names[i]
            except:
                item_name = input_state_names[i]
            if not np.array(variable[i]).ndim == 1:
                raise LearningMechanismError("{} of variable for {} ({}:{}) is not a list or 1d np.array".
                                              format(item_num_string, self.name, item_name, variable[i]))
            if not (is_numeric(variable[i])):
                raise LearningMechanismError("{} of variable for {} ({}:{}) is not numeric".
                                              format(item_num_string, self.name, item_name, variable[i]))
        return variable

    def _validate_params(self, request_set, target_set=None, context=None):
        """Validate error_sources

        `error_sources` argument must be an `ObjectiveMechanism`, another `LearningMechanism`, or a list of them,
        and there must be the same number as there are ERROR_SIGNAL InputStates.

        """

        super()._validate_params(request_set=request_set, target_set=target_set,context=context)

        from psyneulink.core.components.states.state import _parse_state_spec
        from psyneulink.core.components.states.modulatorysignals.learningsignal import LearningSignal
        from psyneulink.core.components.projections.pathway.mappingprojection import MappingProjection
        from psyneulink.core.components.projections.projection import _validate_receiver

        if ERROR_SOURCES in target_set and target_set[ERROR_SOURCES] is not None:
            error_sources = target_set[ERROR_SOURCES]
            if not isinstance(error_sources, list):
                error_sources = [error_sources]

            if not len(error_sources) == len(self.error_signal_input_states):
                raise LearningMechanismError("Number of items specified in \'{}\' arg for {} ({}) "
                                             "must equal the number of its \'{}\' {}s ()".
                                             format(ERROR_SOURCES, self.name, len(error_sources),
                                                    InputState.__name__, ERROR_SIGNAL.upper(),
                                                    len(self.error_signal_input_states)))

            for error_source in error_sources:
                if not isinstance(error_source, (ObjectiveMechanism, LearningMechanism)):
                    raise LearningMechanismError("{} arg for {} ({}) must be an ObjectiveMechanism, another "
                                                 "LearningMechanism, or list of them".
                                                 format(ERROR_SOURCES, self.name, error_source))

        if LEARNING_SIGNALS in target_set and target_set[LEARNING_SIGNALS]:

            if not isinstance(target_set[LEARNING_SIGNALS], list):
                raise LearningMechanismError("{} arg of {} must be list".
                                           format(LEARNING_SIGNAL, self.name))

            for spec in target_set[LEARNING_SIGNALS]:
                learning_signal = _parse_state_spec(state_type=LearningSignal, owner=self, state_spec=spec)

                # Validate that the receiver of the LearningProjection (if specified)
                #     is a MappingProjection and in the same System as self (if specified)
                if learning_signal[PARAMS] and PROJECTIONS in learning_signal[PARAMS]:
                    for learning_projection in  learning_signal[PARAMS][PROJECTIONS]:
                        _validate_receiver(sender_mech=self,
                                           projection=learning_projection,
                                           expected_owner_type=MappingProjection,
                                           spec_type=LEARNING_SIGNAL,
                                           context=context)
                else:
                    pass

    def _instantiate_attributes_before_function(self, function=None, context=None):
        """Instantiates MappingProjection(s) from error_sources (if specified) to LearningMechanism

        Also determines and assigns `error_matrices` from the `error_sources`, identified as the matrix for the
            Projection with which each error_source is associated.
            :param function:
        """
        from psyneulink.core.components.mechanisms.adaptive.learning.learningauxiliary \
            import _instantiate_error_signal_projection

        super()._instantiate_attributes_before_function(function=function, context=context)

        self.error_matrices = None
        if self.error_sources:
            self.error_matrices = [None] * len(self.error_sources)
            for i, error_source in enumerate(self.error_sources):
                self.error_signal_projection = _instantiate_error_signal_projection(sender=error_source, receiver=self)
                if isinstance(error_source, ObjectiveMechanism):
                    self.error_matrices[i] = np.identity(len(error_source.input_states[SAMPLE].value))
                else:
                    # IMPLEMENTATION NOTE:
                    #     This assumes that error_source has only one LearningSignal or,
                    #     if it has more, that they are all equivalent
                    self.error_matrices[i] = error_source.primary_learned_projection.parameter_states[MATRIX]

    def _instantiate_output_states(self, context=None):

        from psyneulink.core.globals.registry import register_category
        from psyneulink.core.components.states.modulatorysignals.learningsignal import LearningSignal
        from psyneulink.core.components.states.state import State_Base, _instantiate_state

        # Create registry for LearningSignals (to manage names)
        register_category(entry=LearningSignal,
                          base_class=State_Base,
                          registry=self._stateRegistry,
                          context=context)

        # Instantiate LearningSignals if they are specified, and assign to self._output_states
        # Notes:
        #    - if any LearningSignals are specified they will replace the default LEARNING_SIGNAL OutputState
        #        in the OUTPUT_STATES entry of paramClassDefaults;
        #    - the LearningSignals are appended to _output_states, leaving ERROR_SIGNAL as the first entry.

        # Get default LearningSignal
        default_learning_signal = next((item for item in self._output_states
                                        if NAME in item and item[NAME] is LEARNING_SIGNAL),None)
        if default_learning_signal is None:
            raise LearningMechanismError("PROGRAM ERROR: Can't find default {} for {}".
                                         format(LearningSignal.__name__, self.name))

        # Assign default if user didn't specify any
        if self.learning_signals is None:
            self.learning_signals = [default_learning_signal]

        # Either way, delete default LearningSignal
        del self._output_states[self._output_states.index(default_learning_signal)]

        # Instantiate LearningSignals and assign to self._output_states
        for learning_signal in self.learning_signals:
            # Instantiate LearningSignal

            params = {LEARNED_PARAM: MATRIX}

            # Parses learning_signal specifications (in call to State._parse_state_spec)
            #    and any embedded Projection specifications (in call to <State>._instantiate_projections)
            learning_signal = _instantiate_state(state_type=LearningSignal,
                                                 owner=self,
                                                 variable=(OWNER_VALUE,0),
                                                 params=params,
                                                 reference_value=self.parameters.learning_signal.get(),
                                                 modulation=self.modulation,
                                                 # state_spec=self.learning_signal)
                                                 state_spec=learning_signal,
                                                 context=context)
            # Add LearningSignal to output_states list
            self._output_states.append(learning_signal)

        # Assign LEARNING_SIGNAL as the name of the 1st LearningSignal; the names of any others can be user-defined
        first_learning_signal = next(state for state in self.output_states if isinstance(state, LearningSignal))
        first_learning_signal.name = LEARNING_SIGNAL

        super()._instantiate_output_states(context=context)

        # Reassign learning_signals to capture any user_defined LearningSignals instantiated in call to super
        #   and assign them to a ContentAddressableList
        self._learning_signals = ContentAddressableList(component_type=LearningSignal,
                                                        list=[state for state in self.output_states if
                                                                  isinstance(state, LearningSignal)])

        # Initialize _error_signals;  this is assigned for efficiency (rather than just using the property)
        #    since it is used by the execute method
        self._error_signal_input_states = self.error_signal_input_states

    def add_states(self, states, context=None):
        """Add error_source and error_matrix for each InputState added"""

        if context is None:
            context = ContextFlags.COMMAND_LINE

        states = super().add_states(states=states)
        for input_state in states[INPUT_STATES]:
            error_source = input_state.path_afferents[0].sender.owner
            self.error_sources.append(error_source)
            self.error_matrices.append(error_source.primary_learned_projection.parameter_states[MATRIX])
            if ERROR_SIGNAL in input_state.name:
                self._error_signal_input_states.append(input_state)

    def _execute(
        self,
        variable=None,
        execution_id=None,
        runtime_params=None,
        context=None
    ):
        """Execute LearningMechanism function and return learning_signal

        Identify error_signals received from LearningMechanisms currently being executed
        Assign them, and the corresponding error_matrices to a pair of arrays
        Execute function for each error_signal, error_matrix pair
        Sum the learning_signal and error_signal values received from each execution

        Returns
        -------

        List[ndarray, ndarray] : summed learning_signal, summed error_signal

        """

        # Get error_signals (from ERROR_SIGNAL InputStates) and error_matrices relevant for the current execution:
        current_error_signal_inputs = self.error_signal_input_states
        curr_indices = [self.input_states.index(s) for s in current_error_signal_inputs]
        error_signal_inputs = variable[curr_indices]
        error_matrices = np.array(self.error_matrices)[np.array([c - ERROR_OUTPUT_INDEX for c in curr_indices])]
        for i, matrix in enumerate(error_matrices):
            if isinstance(error_matrices[i], ParameterState):
                error_matrices[i] = error_matrices[i].parameters.value.get(execution_id)

        summed_learning_signal = 0
        summed_error_signal = 0

        # Compute learning_signal for each error_signal (and corresponding error-Matrix:
        for error_signal_input, error_matrix in zip(error_signal_inputs, error_matrices):

            variable[ERROR_OUTPUT_INDEX] = error_signal_input
            learning_signal, error_signal = super()._execute(
                variable=variable,
                execution_id=execution_id,
                error_matrix=error_matrix,
                runtime_params=runtime_params,
                context=context
            )
            # Sum learning_signals and error_signals
            summed_learning_signal += learning_signal
            summed_error_signal += error_signal

        if self.parameters.context.get(execution_id).initialization_status != ContextFlags.INITIALIZING and self.reportOutputPref:
            print("\n{} weight change matrix: \n{}\n".format(self.name, summed_learning_signal))

        return [summed_learning_signal, summed_error_signal]

    @property
    def learning_enabled(self):
        try:
            return self._learning_enabled
        except AttributeError:
            self._learning_enabled = True
            return self._learning_enabled

    @learning_enabled.setter
    def learning_enabled(self, assignment):
        self._learning_enabled = assignment

    @property
    def learning_rate(self):
        return self.function.learning_rate

    @learning_rate.setter
    def learning_rate(self, assignment):
        self.function.learning_rate = assignment

    @property
    def input_source(self):
        try:
            return self.input_states[ACTIVATION_INPUT].path_afferents[0].sender.owner
        except IndexError:
            return None

    @property
    def output_source(self):
        try:
            return self.input_states[ACTIVATION_OUTPUT].path_afferents[0].sender.owner
        except IndexError:
            return None

    @property
    def error_signal_input_states(self):
        try:
            # This is maintained for efficiency (since it is called by execute method)
            return self._error_signal_input_states
        except AttributeError:
            try:
                return [s for s in self.input_states if ERROR_SIGNAL in s.name]
            except:
                return [s for s in self.input_states if ERROR_SIGNAL in s]

    @property
    def primary_learned_projection(self):
        return self.learned_projections[0]

    @property
    def learned_projections(self):
        return [lp.receiver.owner for ls in self.learning_signals for lp in ls.efferents]
