# Princeton University licenses this file to You under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.  You may obtain a copy of the License at:
#     http://www.apache.org/licenses/LICENSE-2.0
# Unless required by applicable law or agreed to in writing, software distributed under the License is distributed
# on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and limitations under the License.


# *******************************************  LearningMechanism *******************************************************

"""
.. _LearningMechanism_Overview:

Overview
--------

A LearningMechanism is an `AdaptiveMechanism <AdaptiveMechanism>` that modifies the `matrix <MappingProjection.matrix>`
of one or more `MappingProjections <MappingProjection>`.  Its function takes an `error_signal
<LearningMechanism_Input_Error_Signal>` (usually the output of a `ComparatorMechanism` or another `LearningMechanism`),
as well as information about the `MappingProjection and activity <LearningMechanism_Additional_Attributes>` that
generated the error, and calculates a `learning_signal <LearningMechanism.learning_signal>` that is used to modify
the MappingProjection(s) by way of `LearningProjection(s) <LearningProjection>`.  Typically, a LearningMechanism is
used to "train" a single MappingProjection (its `primary_learned_projection`), using the output of the Mechanism to
which that MappingProjection projects (its `output_source`) as the source of the error it attempts to reduce.  A
LearningMechanism can be used to train multiple MappingProjections, by assigning it `additional LearningProjections
<LearningMechanism_Multiple_LearningSignals>`; however, these will all use the same `learning_signal
<LearningMechanism.learning_signal>`, generated by the `primary_learned_projection` and its associated `output_source`.

All of the MappingProjection(s) modified by a LearningMechanism must project from one `ProcessingMechanism
<ProcessingMechanism>` to another in the same `System` or `Process`. The learning components of a System can be
displayed using the System's `show_graph <System.show_graph>` method with its **show_learning** argument assigned
`True`. LearningMechanisms are executed after all of the ProcessingMechanisms in a System or Process have been executed,
and before any `ControlMechanisms <ControlMechanism>` of the System have been executed (see `System Execution
<System_Execution>`).

.. _LearningMechanism_Note

A Note about the Implementation of Learning
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

The implementation of learning in PsyNeuLink was designed for flexibility and "transparency" rather than efficiency.
Unlike its implementation in most other environments -- where the learning algorithm is tightly integrated with the
elements of processing that it modifies --  PsyNeuLink separates it into three constituent components:  An
`ObjectiveMechanism` used to evaluate the most proximal source of error, a `LearningMechanism` that uses that error
(or one derived from it by another LearningMechanism) to calculate a learning signal;  and the `MappingProjection`
that uses that learning signal to modify its weight matrix.  This has the advantage of isolating and exposing the
constituent computations, making it clearer to students what these are and how they operate, and also making each
individually accessible for reconfiguration.  However, it comes at the cost of efficiency.  Therefore, in its present
form, PsyNeuLink is not suitable for conducting large scale learning models (e.g, deep learning networks).
However, once trained, the parameters of such models can easily be incorporated into a PsyNeuLink model for use as
part of a `System`.  Future extensions of PsyNeuLink will provide an interface for the direct construction and
integration within the PsyNeuLink environment of objects created in other environments (e.g., `TensorFlow
<https://www.tensorflow.org/>`_ or `Emergent <https://grey.colorado.edu/emergent/index.php/Main_Page>`_).


.. _LearningMechanism_Creation:

Creating a LearningMechanism
----------------------------

A LearningMechanism can be created in any of the ways used to `create Mechanisms <Mechanism_Creation>`.
More commonly, however, LearningMechanisms are created automatically.

.. LearningMechanism_Automatic_Creation:

Automatic Creation
~~~~~~~~~~~~~~~~~~

A LearningMechanism is created automatically when:

    * the learning attribute is specified for a `System <System_Execution_Learning>` or `Process
      <Process_Learning_Sequence>`;
    ..
    * a `tuple specification <MappingProjection_Learning_Tuple_Specification>` is used to specify learning for a
      MappingProjection;
    ..
    * a `LearningProjection` is created without specifying its `sender <LearningProjection.sender>` attribute.

In these instances, a `ComparatorMechanism`, `LearningSignal`, `LearningProjection`, and any additional Projections
required to implement learning that do not already exist are also instantiated.  This is described below, under
`Learning Configurations <LearningMechanism_Learning_Configurations>`.


.. _LearningMechanism_Explicit_Creation

Explicit Creation
~~~~~~~~~~~~~~~~~

If a LearningMechanism is created explicitly (using its constructor), then it **variable** and **error_source**
arguments must be specified.  The **variable** must have three items that are compatible (in number and type) with the
`value <InputState.value>` of the the LearningMechanism's three `InputStates <LearningMechanism_InputStates>`.  The
**error_source** must be a `ComparatorMechanism` for `single layer learning <LearningMechanism_Single_Layer_Learning>`
or for the last `MappingProjection` in a learning sequence for `multilayer learning
<LearningMechanism_Multilayer_Learning>`;  otherwise it must be a `LearningProjection`.

.. _LearningMechanism_Learning_Signals:

When a LearningMechanism is created explicitly, it can also be assigned existing LearningSignals and/or specified to
create these, as well as `LearningProjections <LearningProjection>` from these to specified MappingProjections.  These
are specified in the **learning_signals** argument of the LearningMechanism's constructor, using any of forms allowed
for `specifying a LearningSignal <LearningSignal_Specification>`.

.. _LearningMechanism_Structure:

Structure
---------

A LearningMechanism has three `InputStates <InputState>`, a learning `function <LearningMechanism.function>`,
and two types of `OutputStates <OutputState>` that are used, respectively, to receive, compute, and transmit the
information needed to modify the MappingProjection(s) for which it is responsible.  In addition, it has several
attributes that govern its operation.  These are all described below.

.. _LearningMechanism_InputStates:

InputStates
~~~~~~~~~~~

These receive the information required by the LearningMechanism's `function <LearningMechanism.function>`.  They are
listed in the LearningMechanism's `input_states <LearningMechanism.input_states>` attribute.  They have the following
names and roles (shown in the `figure <LearningMechanism_Single_Layer_Learning_Figure>` below):

.. _LearningMechanism_Activation_Input:

* *ACTIVATION_INPUT* - receives the value of the input to the `primary_learned_projection`; that is, the
  `value <MappingProjection.value>` of that MappingProjection's `sender <MappingProjection.sender>`).
  The value is assigned as the first item of the LearningMechanism's `variable <LearningMechanism.variable>` attribute.

.. _LearningMechanism_Activation_Output:

* *ACTIVATION_OUTPUT* - receives the `value <OutputState.value>` of the LearningMechanism's `output_source
  <LearningMechanism_Additional_Attributes>` (that is, the output of the *ProcessingMechanism* to which
  the `primary_learned_projection` projects).  By default, the `primary OutputState <OutputState_Primary>`
  of the `output_source` is used.  However, a different OutputState can be designated in the constructor for the
  `output_source`, by assigning a `parameter specification dictionary <ParameterState_Specification>` to the **params**
  argument of its constructor, with an entry that uses *MONITOR_FOR_LEARNING* as its key and a list containing the
  desired OutputState(s) as its value. The `value <InputState.value>` of the *ACTIVATION_OUTPUT* InputState is
  assigned as the second item of the LearningMechanism's `variable <LearningMechanism.variable>` attribute.

.. _LearningMechanism_Input_Error_Signal:

* *ERROR_SIGNAL* - receives the `value <OutputState.value>` from the *OUTCOME* `OutputState
  <ComparatorMechanism_Structure>` of a `ComparatorMechanism` or the *ERROR_SIGNAL* OutputState of another
  `LearningMechanism <LearningMechanism_Output_Error_Signal>`. If the
  `primary_learned_projection` projects to the `TERMINAL` Mechanism of the Process or System being learned,
  or is not part of a `multilayer learning sequence <LearningMechanism_Multilayer_Learning>`,
  then the `error_signal` comes from a ComparatorMechanism. If the `primary_learned_projection` is part of a
  `multilayer learning sequence <LearningMechanism_Multilayer_Learning>`, then the `error_signal` comes from the next
  LearningMechanism in the sequence (i.e., the one associated with the `output_source`).  It is used by the
  LearningMechanism's `function <LearningMechanism.function>` to calculate the `learning_signal
  <LearningMechanism.learning_signal>`;  note that the value of the *ERROR_SIGNAL* InputState may not be the same as
  that of the `error_signal <LearningMechanism.error_signal>` attribute or *ERROR_SIGNAL* `OutputState
  <LearningMechanism_Output_Error_Signal>` (see `note <LearningMechanism_Error_Signal>` below). The `value
  <InputState.value>` of the *ERROR_SIGNAL* InputState is assigned as the third item of the LearningMechanism's
  `variable <LearningMechanism.variable>` attribute.

The Mechanism from the which the `value <InputState.values>`\\s above are received are listed in the
LearningMechanism's `input_source <LearningMechanism.input_source>`, `output_source <LearningMechanism.output_source>`,
and `error_source <LearningMechanism.error_source>` attributes, respectively (see
`LearningMechanism_Additional_Attributes` for additional details).

.. _LearningMechanism_Function:

Learning Function
~~~~~~~~~~~~~~~~~

The `function <LearningMechanism.function>` of a LearningMechanism uses the three values received by the Mechanism's
InputStates (described `above <LearningMechanism_InputStates>`) to calculate the value of its `learning_signal
<LearningMechanism.learning_signal>` and `error_signal <LearningMechanism.error_signal>` attributes.

.. _LearningMechanism_Learning_Signal:

* `learning_signal` - the set of changes to the `matrix <MappingProjection.matrix>` parameter of the
  `MappingProjections <MappingProjection>` being learned, calculated to reduce the value of the LearningMechanism's
  *ERROR_SIGNAL* `InputState <LearningMechanism_Input_Error_Signal>`.

.. _LearningMechanism_Error_Signal:

* `error_signal <LearningMechanism.error_signal>` - the contribution made by the `primary_learned_projection` to the
  error signal received by the LearningMechanism's *ERROR_SIGNAL* `InputState <LearningMechanism_Input_Error_Signal>`.
  It is used by the LearningMechanism's `function <LearningMechanism.function>`, possibly weighted by the
  `primary_learned_projection` and the output of the `ProcessingMechanism <ProcessingMechanism>` to which it projects
  (the LearningMechanism's `output_source` attribute), to calculate the `learning_signal
  <LearningMechanism.learning_signal>`. Its value is assigned as the value of the LearningMechanism's *ERROR_SIGNAL*
  `OutputState <LearningMechanism_Output_Error_Signal>`.

  .. _LearningMechanism_Error_Signal_Note:

  .. note::

     A LearningMechanism's *ERROR_SIGNAL* `InputState <LearningMechanism_Input_Error_Signal>` and its
     *ERROR_SIGNAL* `OutputState <LearningMechanism_Output_Error_Signal>` may not have the same value.
     The former is the error signal received from a `ComparatorMechanism` or another `LearningMechanism`,
     while the latter is the contribution made to that error by the `primary_learned_projection` and the
     `output_source`, as calculated by the LearningMechanism's `function <LearningMechanism.function>`
     (see `error_signal <LearningMechanism_Error_Signal>` above).

The default `function <LearningMechanism.function>` of a LearningMechanism is `BackPropagation` (also known as the
*Generalized Delta Rule*; see
`Rumelhart et al., 1986 <http://www.nature.com/nature/journal/v323/n6088/abs/323533a0.html>`_).  However, it can be
assigned any other PsyNeuLink `LearningFunction`, or any other Python function that takes as its input a list or
np.array containing three lists or 1d np.arrays of numeric values, and returns two lists or 1d np.arrays.  The two
values it returns are assigned to the LearningMechanism's `learning_signal <LearningMechanism.learning_signal>` and
`error_signal <LearningSignal.error_signal>` attributes, respectively, as well as to its two OutputStates,
as described below.


.. _LearningMechanism_OutputStates:

OutputStates
~~~~~~~~~~~~

By default, a LearningMechanism has two `OutputStates <OutputState>`, the first of which is named *ERROR_SIGNAL* and
is assigned the value of the `error_signal <LearningMechanism.error_signal>` returned by the LearningMechanism's
`function <LearningMechanism.function>`, and the second of which is a `LearningSignal` and is assigned the value of the
`learning_signal <LearningMechanism.learning_signal>` returned by the `function <LearningMechanism.function>`.
They are each described below:

.. _LearningMechanism_Output_Error_Signal:

* *ERROR_SIGNAL* - this is the `primary OutputState <OutputState_Primary>` of a LearningMechanism, and  receives the
  value of the `error_signal <LearningMechanism.error_signal>` used to calculate the `learning_signal
  <LearningMechanism.learning_signal>`.  Its value is assigned as the first item of the LearningMechanism's
  `output_values <LearningMechanism.output_values>` attribute.  If the LearningMechanism is part of a `multilayer
  learning sequence <LearningMechanism_Multilayer_Learning>`, the *ERROR_SIGNAL* OutputState is assigned a Projection
  to the LearningMechanism for the preceding MappingProjection in the sequence being learned - see `figure
  <LearningMechanism_Multilayer_Learning_Figure>` below).  Note that the value of the *ERROR_SIGNAL* OutputState
  value may not be the same as that of the *ERROR_SIGNAL* `InputState <LearningMechanism_Input_Error_Signal>`
  (see `error_signal <LearningMechanism_Error_Signal>`).

.. _LearningMechanism_LearningSignal:

* `LearningSignal(s) <LearningSignal>` - by default, a LearningMechanism has a single LearningSignal, which is a
  special type of OutputState that receives the `learning_signal <LearningMechanism.learning_signal>` generated by the
  LearningMechanism's `function <LearningMechanism.function>`, and used to modify the `matrix
  <MappingProjection.matrix>` parameter of the `primary_learned_projection`.  The LearningSignal is assigned as the
  second item in the list of the LearningMechanism's OutputStates (i.e., of its `output_states
  <LearningMechanism.output_states>` attribute), and its `value <LearningSignal.value>` is assigned as the second
  item of the LearningMechanism's `output_values <LearningMechanism.output_values>` attribute.

.. _LearningMechanism_Multiple_LearningSignals:

  **Multiple LearningSignals and LearningProjections.** Though not common, it is possible for a LearningMechanism to
  be assigned more than one `LearningSignal`, and/or more than one `LearningProjection` to its LearningSignal(s). This
  allows multiple MappingProjections to be trained by a single LearningMechanism. Note, however, that all of the
  LearningSignals of a LearningMechanism (and therefore its LearningProjections) use the same
  `learning_signal <LearningMechanism.learning_signal>`, that is calculated based on the LearningMechanism's
  `primary_learned_projection` and its associated `output_source`.  This can be useful in some settings, such as for
  certain forms of `convolutional neural networks <https://en.wikipedia.org/wiki/Convolutional_neural_network>`_.

  If all of the LearningProjections are used to implement the same form of `modulation <ModulatorySignal_Modulation>`,
  (determined by their LearningSignals' `modulation <LearningSignal.modulation>` attribute), then they should be
  assigned to a single LearningSignal. Multiple LearningProjections can be assigned to a LearningSignal by specifying
  them in the **projections** argument of its constructor, or the *PROJECTIONS* entry of a dictionary assigned to its
  **params** argument); however, the `matrix <MappingProjection.matrix>` parameter for the MappingProjection to which
  they project must have the same shape for all of them. If different forms of modulation are required to train
  different MappingProjections, then multiple LearningSignals should be assigned to the LearningMechanism, each
  specified for one of the required types of modulation, and then LearningProjections assigned to them for the
  corresponding MappingProjections. Multiple LearningSignals can be specified for a LearningMechanism by including
  them in a list assigned to the **learning_signals** argument of the LearningMechanism's constructor.

  The `learning_rate <LearningSignal.learning_rate>` for each LearningSignal, and the `learning_rate
  <LearningProjection.learning_rate>` for each of its `LearningProjections <LearningProjection>`, can all be assigned
  different values (with the latter taking precedence over the former).  If none of these are specified, the
  `learning_rate <LearningMechanism.learning_rate>` of the LearningMechanism is used (see `below
  <LearningMechanism_Learning_Rate>`).

  All of the LearningSignals of a LearningMechanism are listed in its `learning_signals` attribute.  Because these
  are `OutputStates <OutputState>`, they are also listed in the `output_states <LearningMechanism.output_states>`
  attribute, after the *ERROR_SIGNAL* OutputState.  All of the LearningMechanism's LearningProjections (that is, those
  belonging to all of its LearningSignals) are listed in its `learning_projections` attribute.

.. _LearningMechanism_Additional_Attributes:

Additional Attributes
~~~~~~~~~~~~~~~~~~~~~

In addition to its `InputStates <LearningMechanism_InputStates>`, `function <LearningMechanism_Function>` and
`OutputStates <LearningMechanism_OutputStates>`, a LearningMechanism has the following attributes that
refer to the Components being learned and/or its operation:

.. _LearningMechanism_Primary_Learned_Projection:

* `primary_learned_projection` - the `MappingProjection` with the `matrix <MappingProjection.matrix>` parameter for
  which the `learning_signal <LearningMechanism.learning_signal>` is computed.  This is always the first Projection
  listed first in LearningMechanism's `learned_projections` attribute.
..
* `learned_projections` - a list of all of the MappingProjections to which the LearningMechanism sends a
  `LearningProjection`, listed in the order of the `LearningSignal(s) <LearningSignal>` to which they belong,
  as those are listed in the LearningMechanism's `learning_signals <LearningMechanism.learning_signals>` attribute.
..
* `learning_enabled <LearningMechanism.learning_enabled>` - if set to `False`, learning is disabled for all of its
  `LearningProjections <LearningProjection>`;  however, the LearningMechanism is still executed during the learning
  phase of execution of a `Process <Process_Execution>` or `System  <System_Execution_Learning>`, so that the error
  signals it calculates can be passed to any other LearningMechanism(s) to which it projects (see
  `LearningMechanism_Multilayer_Learning`).
..
* `input_source` - the `Mechanism <Mechanism>` that sends the `primary_learned_projection`, and projects to the
  LearningMechanism's *ACTIVATION_INPUT* `InputState <LearningMechanism_Activation_Input>`.
..
* `output_source` - the `Mechanism <Mechanism>` that receives the `primary_learned_projection`, and  provides the
  input to the LearningMechanism's *ACTIVATION_OUTPUT* `InputState <LearningMechanism_Activation_Output>`.
..
* `error_source` - the `ComparatorMechanism` or `LearningMechanism` that calculates the error signal provided to the
  LearningMechanism's *ERROR_SIGNAL* `InputState <LearningMechanism_Input_Error_Signal>`.
..
* `modulation` - the default value used for the `modulation <LearningSignal.modulation>` attribute of
  LearningMechanism's `LearningSignals <LearningSignal>` (i.e. those for which it is not explicitly specified).
  This determines the way in which the `learning_signal <LearningMechanism.learning_signal>` is used to modify the
  `matrix <MappingProjection.matrix>` parameter of the `learned_projections`.  By default its value is
  Modulation.ADD, which causes the weight changes in the `learning_signal` to be added to the current value of the
  `matrix <MappingProjection.matrix>` parameter (see `LearningMechanism_Execution` for a description of how the
  modifications are executed).

.. _LearningMechanism_Learning_Rate:

* `learning_rate <LearningMechanism.learning_rate>` - specifies the :keyword:`learning_rate` parameter used by the
  LearningMechanism's `function <LearningMechanism.function>`, which uses it to multiply the weight change matrix
  before returning it as the `learning_signal <LearningMechanism.learning_signal>`.  This can be specified in the
  **learning_rate** argument of the LearningMechanism's constructor (or the constructor for its `function
  <LearningMechanism.function>`;  doing so supersedes specification of the `learning_rate <Process.learning_rate>`
  for any `Process`, and/or the `learning_rate <System.learning_rate>` for any `System` to which the
  LearningMechanism belongs.  The default value for a LearningMechanism's `learning_rate
  <LearningMechanism.learning_rate>` attribute is `None`, in which case the LearningMechanism (and its `function
  <LearningMechanism.function>`) inherit the specification of the `learning_rate <Process.learning_rate>` for
  the `Process` in which the LearningMechanism is executed. If that is `None`, then it inherits the `learning_rate
  <System.learning_rate>` specification of the `System` in which it is executed.  If that is also `None`, then it
  uses the value of the `default_learning_rate <LearningFunction.default_learning_rate>` parameter of its `function
  <LearningMechanism.function>`. A :keyword:`learning_rate` parameter can also be specified for individual
  `LearningSignals <LearningSignal>` and/or their associated `LearningProjections <LearningProjection>`.  Those have
  a direct multiplicative effect on the `learning_signal <LearningProjection.learning_signal>` of the LearningSignal
  and/or it LearningProjections (see `LearningSignal learning_rate <LearningSignal_Learning_Rate>` for additional
  details).

.. _LearningMechanism_Learning_Configurations:

COMMENT:
@@@ THE FOLLOWING SECTIONS SHOULD BE MOVED TO THE "USER'S MANUAL" WHEN THAT IS WRITTEN
COMMENT

Learning Configurations
~~~~~~~~~~~~~~~~~~~~~~~

When learning is specified for a `MappingProjection <Mapping_Matrix_Specification>`, a `Process
<Process_Learning_Sequence>`, or a `System <System_Execution_Learning>`, all of the Components required for learning are
created automatically. The types of Components that are generated depend on the `LearningFunction <LearningFunction>`
specified, and the configuration of the `Composition <Composition>`, as described below.  If learning is `specified for
individual Projections <Process_Learning_Specification>` in the `pathway <Process.pathway>` of a Process, it takes
effect only if that Process is executed on its own (i.e., using the Process' `execute <Process.execute>` or `run
<Process.run>` methods.  For the learning Components associated with a Process to be implemented by a System,
learning must be `specified for the entire Process <Process_Learning_Specification>`.  All of the learning Components
of a System can be displayed using the System's `show_graph` method with its **show_learning** argument assigned `True`.

.. _LearningMechanism_Single_Layer_Learning:

Single layer learning
^^^^^^^^^^^^^^^^^^^^^

This configuration occurs when only a single `MappingProjection` is `specified for learning <Projection_Specification>`,
or the LearningMechanism's `function <LearningMechanism.function>` only considers the error on its `output_source`
(and not any additional sources of error) when calculating the `learning_signal <LearningMechanism.learning_signal>`
(e.g., for `Reinforcement` learning).  In this case, a single `ComparatorMechanism` and LearningMechanism are created
(if they do not already exist);  in addition, the following MappingProjections are created:

* from an `OutputState` of the LearningMechanism's `output_source` to the ComparatorMechanism's *SAMPLE* `InputState
  <ComparatorMechanism_Structure>`.  By default, the `primary OutputState <OutputState_Primary>` of the
  `output_source` is used; however, this can be modified by specifying its *MONITOR_FOR_LEARNING* parameter
  (see `above <LearningMechanism_Activation_Output>`).
..
* from the Process or System to the ComparatorMechanism's *TARGET* `InputState <ComparatorMechanism_Structure>`;
..
* from the ComparatorMechanism's *OUTCOME* `OutputState <ComparatorMechanism_Structure>` to the
  LearningMechanism's *ERROR_SIGNAL* `InputState <LearningMechanism_Activation_Input>`.

In addition, a `LearningProjection` is created from the `LearningSignal<LearningMechanism_LearningSignal>` for the
`primary_learned_projection` to the `ParameterState` for the `matrix <MappingProjection.matrix>` of the
`primary_learned_projection`.  Because this configuration involves only a single layer of learning, *no* Projection
is created or assigned to the LearningMechanism's *ERROR_SIGNAL* `OutputState <LearningMechanism_Output_Error_Signal>`.

.. _LearningMechanism_Single_Layer_Learning_Learning_Figure:

    **Components for Single Layer Learning**

    .. figure:: _static/LearningMechanism_Single_Layer_Learning_fig.svg
       :alt: Schematic of Mechanisms and Projections involved in learning for a single MappingProjection
       :scale: 50%

       ComparatorMechanism, LearningMechanism and associated Projections created for the `primary_learned_projection`
       and `output_source`.  Each Mechanism is labeled by its type (upper line, in bold) and its designated
       status in the Process and/or System to which it belongs (lower line, capitalized).  Italicized label beside a
       component indicates the attribute of the LearningMechanism with which it is associated.


.. _LearningMechanism_Multilayer_Learning:

Multilayer learning
^^^^^^^^^^^^^^^^^^^

This configuration occurs when a set of `MappingProjections <MappingProjection>` being learned are in a sequence (such
as the `pathway <Process.pathway>` of a `Process`); that is, in which each projects to a `ProcessingMechanism
<ProcessingMechanism>` that is the `sender <MappingProjection.sender>` for the next MappingProjection in the sequence
(see the `figure <LearningMechanism_Multilayer_Learning_Figure>` below). This requires the use of a learning function
that can calculate the influence that each MappingProjection and its output have on the error that the LearningMechanism
receives from the next one in the sequence (e.g., `BackPropagation`). In multilayer learning, the Components created
depend on the position of the `primary_learned_projection` and `output_source <LearningMechanism.output_source>` in
the sequence.  If these are the last ones in the sequence, they are treated in the same way as `single layer
learning <LearningMechanism_Single_Layer_Learning>`.  This is the case if the `output_source` is a standalone Mechanism
(one not in a `Process` or `System`), the `TERMINAL` Mechanism of a standalone Process (i.e., one not in a System),
or the `TERMINAL` of all of the Processes to which it belongs in a System (and therefore a `TERMINAL` for the
System).  In these cases, as for single layer learning, a `ComparatorMechanism` is created that receives the output
of the `output_source` as well as the target for learning (see `TARGET Mechanisms <LearningMechanism_Targets>` below),
and projects to a LearningMechanism that is created for the `primary_learned_projection`.  For all other
MappingProjections being learned in the sequence, the following additional MappingProjections are created for
learning (shown in the `figure <LearningMechanism_Multilayer_Learning_Figure>` below):

* from the `input_source <LearningMechanism.input_source>` to the LearningMechanism's *ACTIVATION_INPUT* `InputState
  <LearningMechanism_Activation_Input>`.
..
* from the `output_source <LearningMechanism.output_source>` to the LearningMechanism's *ACTIVATION_OUTPUT* `InputState
  <LearningMechanism_Activation_Output>`.
..
* from the *ERROR_SIGNAL* `OutputState <LearningMechanism_Output_Error_Signal>` of the LearningMechanism for the
  next MappingProjection in the sequence (i.e., the one to which the `output_source <LearningMechanism.output_source>`
  projects) to the LearningMechanism's *ERROR_SIGNAL* `InputState <LearningMechanism_Input_Error_Signal>`.

In addition, a `LearningProjection` is created from the `LearningSignal <LearningMechanism_LearningSignal>` for the
`primary_learned_projection` of each LearningMechanism in the sequence, to the `ParameterState` for the `matrix
<MappingProjection.matrix>` of the `primary_learned_projection`.  If the `primary_learned_projection` is the first in
the sequence, then *no* Projection is created or assigned to its LearningMechanism's *ERROR_SIGNAL* `OutputState
<LearningMechanism_Output_Error_Signal>`.

.. _LearningMechanism_Multilayer_Learning_Figure:

    **Components for Multilayer Learning**

    .. figure:: _static/LearningMechanism_Multilayer_Learning_fig.svg
       :alt: Schematic of Mechanisms and Projections involved in learning for a sequence of MappingProjections
       :scale: 50%

       ComparaotrMechanism and LearningMechanisms and associated Projections created for a sequence of two
       MappingProjections specified for learning.  Each Mechanism is labeled by its type (uppler line, in bold) and
       its designated status in the Process and/or System to which it belongs (lower line, capitalized).  Italicized
       label beside a component indicates the attribute of the LearningMechanism with which it is associated.

.. _LearningMechanism_Targets:

**TARGET Mechanisms**. When a learning function is specified for a LearningMechanism that requires a target (e.g.,
`BackPropagation` or `Reinforcement`), a `ComparatorMechanism` must be specified to receive the target.  For `multilayer
learning <LearningMechanism_Multilayer_Learning>`, this is the `error_source <LearningMechanism.error_source>` for the
last MappingProjection in each learning sequence.  When learning is specified for a `Composition <Composition>` (i.e.,
a `Process <Process_Learning_Sequence>` or a `System <System_Execution_Learning>`), the `ComparatorMechanism(s)
<ComparatorMechanism>` that receive the `targets <Run_Targets>`  are identified and designated as `TARGET` Mechanisms,
and are listed in the Composition's `target_mechanisms` attribute. If a `TERMINAL` Mechanism of a Composition receives a
MappingProjection that is specified for learning, then it always projects to a `TARGET` Mechanism in that Composition.
It is important to note, in this context, the status of a Mechanism in a System takes precedence over its status in any
of the Processes to which it belongs. This means that even if a Mechanism is the `TERMINAL` of a particular Process, if
that Process is combined with others in a System, and the Mechanism appears in any of those other Processes, and it is
not the `TERMINAL` of *all* of them, then it will *not* be the `TERMINAL` for the System.  As a consequence, although it
will project to a `TARGET` Mechanism in the Process for which it is the `TERMINAL`, it will not do so in the System (see
`figure below <LearningProjection_Target_vs_Terminal_Figure>` for an example).  Finally, if a Mechanisms is the
`TERMINAL` for more than one Process used to create a System (that is, the pathways for those Processes converge on that
Mechanism), only one ComparatorMechanism will be created for it in the System.

.. _LearningProjection_Target_vs_Terminal_Figure:

    **TERMINAL** and **TARGET** Mechanisms in Learning

    .. figure:: _static/LearningMechanism_TERMINAL_vs_TARGET_fig.svg
       :alt: Schematic of Mechanisms and Projections involved in learning
       :scale: 50 %

       Mechanism 3 is the `TERMINAL` Mechanism for Process A, However, it is also an `INTERNAL` Mechanism of Process B.
       Therefore, Mechanism 3 is designated as an `INTERNAL` Mechanism for the System, and Mechanism 4 is its `TERMINAL`
       Mechanism. As a consequence, if `BackPropagation` is used for learning, then Mechanism 4 will project to a
       `TARGET` Mechanism of the System, while Mechanism 3 will not.

.. _LearningMechanism_Execution:

Execution
---------

LearningMechanisms are executed after all of the `ProcessingMechanisms <ProcessingMechanism>` in the `Process` or
`System` to which they belong have been executed.  When a LearningMechanism is executed, it uses the `value
<InputState.value>` of its *ERROR_SIGNAL* `InputState <LearningMechanism_Input_Error_Signal>` to calculate changes to
the weights of the `matrix <MappingProjection.MappingProjection.matrix>` parameter of the `learned_projections`. Those
weight changes are assigned as the LearningMechanism's `learning_signal <LearningMechanism.learning_signal>` attribute,
the `value <LearningSignal.value>` of each of its `LearningSignals <LearningMechanism_LearningSignal>`, and as the
`value <LearningProjection.value>` of each of their LearningProjections.  That value is used, in turn, to modify the
`value <ParameterState.value>` of the *MATRIX* `ParameterState` of each of the MappingProjections being learned (listed
in the LearningMechanism's `learned_projections` attribute).

Each ParameterState uses the value it receives from the `LearningProjection` that projects to it to modify the
parameter of its `function <ParameterState.function>`, in a manner specified by the `modulation
<LearningSignal.modulation>` attribute of the `LearningSignal` from which it receives the LearningProjection (see
`Modulation <ModulatorySignal_Modulation>` for a description of how modulation operates). By default, the
`modulation <LearningSignal.modulation>` attribute of a LearningSignal is `ModulationParam.ADDITIVE`,
the `function <ParameterState.function>` of a *MATRIX* ParameterState for a MappingProjection is
`AccumulatorIntegrator`, and the parameter it uses for additive modulation is its `increment
<AccumulatorIntegrator.increment>` parameter.  These assignments cause the value of the LearningProjection to be added
to the previous value of the *MATRIX* ParameterState, thus incrementing the weights by an amount specified by the
LearningMechanism's `learning_signal <LearningMechanism.learning_signal>`. Note, that the changes to the `matrix
<MappingProjection.MappingProjection.matrix>` parameter itself do not take effect
until the next time the `learned_projection` is executed (see :ref:`Lazy Evaluation <LINK>` for an explanation of
"lazy" updating).

A LearningMechanism's `function <LearningMechanism.function>` also computes an `error_signal
<LearningMechanism.error_signal>` that is assigned as the `value <OutputState.value>` of its *ERROR_SIGNAL*
`OutputState <LearningMechanism_Output_Error_Signal>`;  in a `multilayer learning configuration
<LearningMechanism_Multilayer_Learning>`, that value is provided to the *ERROR_SIGNAL* `InputState
<LearningMechanism_Input_Error_Signal>` of the LearningMechanism for the preceding MappingProjection
being learned in the sequence.

.. _LearningMechanism_Class_Reference:

Class Reference
---------------

"""

import numpy as np
import typecheck as tc

from psyneulink.components.component import InitStatus, parameter_keywords
from psyneulink.components.functions.function import BackPropagation, ModulationParam, _is_modulation_param, \
    is_function_type
from psyneulink.components.mechanisms.mechanism import Mechanism_Base
from psyneulink.components.mechanisms.adaptive.adaptivemechanism import AdaptiveMechanism_Base
from psyneulink.components.mechanisms.processing.objectivemechanism import OUTCOME, ObjectiveMechanism
from psyneulink.components.shellclasses import Mechanism
from psyneulink.components.states.modulatorysignals.learningsignal import LearningSignal
from psyneulink.globals.keywords import CONTROL_PROJECTIONS, IDENTITY_MATRIX, INDEX, INITIALIZING, \
    INPUT_STATES, LEARNED_PARAM, LEARNING, LEARNING_MECHANISM, LEARNING_PROJECTION, LEARNING_SIGNAL, \
    LEARNING_SIGNALS, MATRIX, NAME, PARAMS, OUTPUT_STATES, PROJECTIONS, ENABLED, MATRIX_KEYWORD_SET
from psyneulink.globals.preferences.componentpreferenceset import is_pref_set
from psyneulink.globals.preferences.preferenceset import PreferenceLevel
from psyneulink.globals.utilities import is_numeric, parameter_spec, ContentAddressableList
from psyneulink.scheduling.timescale import CentralClock, TimeScale

__all__ = [
    'ACTIVATION_INPUT', 'ACTIVATION_INPUT_INDEX', 'ACTIVATION_OUTPUT', 'ACTIVATION_OUTPUT_INDEX',
    'DefaultTrainingMechanism', 'ERROR_OUTPUT_INDEX', 'ERROR_SIGNAL', 'ERROR_SIGNAL_INDEX', 'ERROR_SOURCE',
    'LearningMechanism', 'LearningMechanismError', 'input_state_names', 'output_state_names'
]

# Params:

parameter_keywords.update({LEARNING_PROJECTION, LEARNING})


def _is_learning_spec(spec, include_matrix_spec=True):
    """Evaluate whether spec is a valid learning specification

    Return `True` if spec is LEARNING or a valid projection_spec (see Projection_Base._is_projection_spec)
    Otherwise, return `False`

    """
    # MODIFIED 11/28/17 OLD:
    from psyneulink.components.projections.projection import _is_projection_spec

    try:
        if spec in {LEARNING, ENABLED}:
            return True
        else:
            return _is_projection_spec(spec=spec,
                                       type=LEARNING_PROJECTION,
                                       include_matrix_spec=include_matrix_spec)
    except:
        return False
    # # MODIFIED 11/28/17 NEW:
    # from psyneulink.components.projections.modulatory.learningprojection import LearningProjection
    # if isinstance(spec, tuple):
    #     return _is_learning_spec(spec[1])
    # elif isinstance(spec, (LearningMechanism, LearningSignal, LearningProjection)):
    #     return True
    # elif isinstance(spec, type) and issubclass(spec, LearningSignal):
    #     return True
    # elif isinstance(spec, str) and spec in {LEARNING, LEARNING_PROJECTION, LEARNING_SIGNAL}:
    #     return True
    # elif include_matrix_spec:
    #     if isinstance(spec, str) and spec in MATRIX_KEYWORD_SET:
    #         return True
    #     from psyneulink.components.functions.function import get_matrix
    #     if get_matrix(spec) is not None:
    #         return True
    # else:
    #     return False
    # # MODIFIED 11/28/17 END:


# Used to index variable:
ACTIVATION_INPUT_INDEX = 0
ACTIVATION_OUTPUT_INDEX = 1
ERROR_OUTPUT_INDEX = 2
ERROR_SIGNAL_INDEX = 3

# Used to name input_states and output_states:
ACTIVATION_INPUT = 'activation_input'     # InputState
ACTIVATION_OUTPUT = 'activation_output'   # InputState
ERROR_SIGNAL = 'error_signal'
input_state_names =  [ACTIVATION_INPUT, ACTIVATION_OUTPUT, ERROR_SIGNAL]
output_state_names = [ERROR_SIGNAL, LEARNING_SIGNAL]

ERROR_SOURCE = 'error_source'

DefaultTrainingMechanism = ObjectiveMechanism

class LearningMechanismError(Exception):
    def __init__(self, error_value):
        self.error_value = error_value

    def __str__(self):
        return repr(self.error_value)


class LearningMechanism(AdaptiveMechanism_Base):
    """
    LearningMechanism(                             \
        variable,                                  \
        error_source,                              \
        function=BackPropagation,                  \
        learning_rate=None,                        \
        learning_signals=LEARNING_SIGNAL,          \
        modulation=ModulationParam.MULTIPLICATIVE, \
        params=None,                               \
        name=None,                                 \
        prefs=None)

    Implements a Mechanism that modifies the `matrix <MappingProjection.matrix>` parameter of a `MappingProjection`.

    COMMENT:
        Description:
            LearningMechanism is a subtype of the AdaptiveMechanism Type of the Mechanism Category of Component
            It implements a Mechanism that calculates changes to a Projection's parameters.
            Its function takes the output of an ObjectiveMechanism and generates a
            learning_signal (ndarray of parameter changes) to be used by the recipient of a LearningProjection
            that projects from the LearningMechanism to a MappingProjection.

        # DOCUMENT: ??NOT SURE WHETHER THIS IS STILL RELEVANT
        #    IF objective_mechanism IS None, IT IS LEFT UNSPECIFIED (FOR FURTHER IMPLEMENTATION BY COMPOSITION)
        #    THESE ARE HANDLED BY A MODULE METHOD _instantiate_objective_mechanism (AS PER OBJECTIVE MECHANISM):
        #        IF objective_mechanism IS SPECIFIED AS ObjectiveMechanism, AN OBJECTIVE MECHANISM IS CREATED FOR IT
        #        IF objective_mechanism IS SPECIFIED AS A MECHANISM OR OUTPUTSTATE,
        #               a MappingProjection WITH AN IDENTITY MATRIX IS IMPLEMENTED FROM IT TO THE LearningMechanism

        Learning function:
            Generalized delta rule:
            dE/dW  =          learning_rate   *    dE/dA          *       dA/dW             *    I
            weight = weight + (learning_rate  * error_derivative  *  activation_derivative  *  input)
            for sumSquared error fct =        (target - output)
            for logistic activation fct =                           output * (1-output)
            where:
                output = activity of output (target) units (higher layer)
                input = activity of sending units (lower layer)
            Needs:
            - activation_derivative:  get from FUNCTION of sample_activation_mechanism/receiver_mech
                                      assumes derivative of Logistic unless otherwise specified
            - error_derivative:  get from FUNCTION of error_source/next_level_mech;  but handled in ObjectiveMechanism

        Class attributes:
            + className = LEARNING_MECHANISM
            + componentType = ADAPTIVE_MECHANISM
            + paramClassDefaults (dict):
                + FUNCTION (Function): (default: BP)
                + FUNCTION_PARAMS:
                    + LEARNING_RATE (value): (default: 1)
            + classPreference (PreferenceSet): LearningSignalPreferenceSet, instantiated in __init__()
            + classPreferenceLevel (PreferenceLevel): PreferenceLevel.TYPE

        Class methods:
            None

        MechanismRegistry:
            All instances of LearningMechanism are registered in MechanismRegistry, which maintains an
              entry for the subclass, a count for all instances of it, and a dictionary of those instances
    COMMENT

    Arguments
    ---------

    variable : List or 2d np.array
        it must have three items that correspond to the three values required by the LearningMechanism's `function
        <LearningMechanism.function>`;  they must each be compatible (in number and type) with the `value
        <InputState.value>` of the corresponding `InputState <LearningMechanism_InputStates>` (see `variable
        <LearningMechanism.variable>` for additional details).

    error_source : ComparatorMechanism or LearningMechanism
        specifies the source of the error signal used by the LearningMechanism's `function
        <LearningMechanism.function>`.  It must be a `ComparatorMechanism` for `single layer learning
        <LearningMechanism_Single_Layer_Learning>`, or for the last `MappingProjection` in a learning sequence in
        `multilayer learning <LearningMechanism_Multilayer_Learning>`;  otherwise it must be a `LearningProjection`.

    learning_signals : List[parameter of Projection, ParameterState, Projection, tuple[str, Projection] or dict]
        specifies the parameter(s) to be learned (see `learning_signals <LearningMechanism.learning_signals>` for
        details).

    modulation : ModulationParam : ModulationParam.ADDITIVE
        specifies the default form of modulation used by the LearningMechanism's LearningSignals,
        unless they are `individually specified <LearningSignal_Specification>`.

    function : LearningFunction or function
        specifies the function used to calculate the LearningMechanism's `learning_signal
        <LearningMechanism.learning_signal>` and `error_signal <LearningMechanism.error_signal>` attributes.  It's
        `variable <Function_Base.variable>` must have three items, each of which must be a list or 1d array of
        numeric values, corresponding to values provided by the LearningMechanism's *ACTIVATION_INPUT*,
        *ACTIVATION_OUTPUT*, and *ERROR_SOURCE* InputStates, respectively (see `LearningMechanism_InputStates
        `LearningMechanism_Function` and `LearningMechanism_InputStates` for additional details).

    learning_rate : float
        specifies the learning rate for the LearningMechanism (see `learning_rate <LearningMechanism.learning_rate>`
        for details).

    params : Dict[param keyword, param value] : default None
        a `parameter dictionary <ParameterState_Specification>` that specifies the parameters for the
        Projection, its function, and/or a custom function and its parameters. By default, it contains an entry for
        the Projection's default `function <LearningProjection.function>` and parameter assignments.  Values specified
        for parameters in the dictionary override any assigned to those parameters in arguments of the constructor.

    name : str : default see `name <LearningMechanism.name>`
        specifies the name of the LearningMechanism.

    prefs : PreferenceSet or specification dict : default Mechanism.classPreferences
        specifies the `PreferenceSet` for the LearningMechanism; see `prefs <LearningMechanism.prefs>` for details.


    Attributes
    ----------

    COMMENT:
        componentType : LEARNING_MECHANISM
    COMMENT

    variable : 2d np.array
        has three items that serve as the template for the three inputs required by the LearningMechanism's `function
        <LearningMechanism.function>` (corresponding to its three `InputStates <LearningMechanism_InputStates>`:
        the input to the `primary_learned_projection` (from `input_source`), the output of the Mechanism to which
        that projects (i.e., of `output_source`); and the error signal (from `error_source`).

    input_states : ContentAddressableList[OutputState]
        list containing the LearningMechanism's three `InputStates <LearningMechanism_InputStates>`:
        *ACTIVATION_INPUT*,  *ACTIVATION_OUTPUT*, and *ERROR_SIGNAL*.

    input_source : ProcessingMechanism
        the Mechanism that sends the `primary_learned_projection`, and projects to the
        LearningMechanism's *ACTIVATION_INPUT* `InputState <LearningMechanism_Activation_Input>`.

    output_source : ProcessingMechanism
        the Mechanism that receives the `primary_learned_projection`, and  projects to the
        LearningMechanism's *ACTIVATION_OUTPUT* `InputState <LearningMechanism_Activation_Output>`.

    error_source : ComparatorMechanism or LearningMechanism
        the Mechanism that calculates the error signal provided to the
        LearningMechanism's *ERROR_SIGNAL* `InputState <LearningMechanism_Input_Error_Signal>`.

    primary_learned_projection : MappingProjection
        the Projection with the `matrix <MappingProjection.matrix>` parameter used to generate the
        LearningMechanism's `error_signal <LearningMechanism.error_signal>` and `learning_signal
        <LearningMechanism.learning_signal>` attributes.  It is always the first Projection listed in the
        LearningMechanism's `learned_projections <LearningMechanism.learned_projections>` attribute.

    learned_projections : List[MappingProjection]
        all of the MappingProjections modified by the LearningMechanism;  the first item in the list is always the
        `primary_learned_projection <LearningMechanism.primary_learned_projection>`.

    learning_enabled : bool : True
        if set to False, learning is disabled for all of its LearningProjections; however, the LearningMechanism is
        still executed during the learning phase of execution of a `Process <Process_Execution>` or `System
        <System_Execution_Learning>`, so that the error signals it calculates can be passed to any other
        LearningMechanism(s) to which it projects (see `LearningMechanism_Multilayer_Learning`).

    function : LearningFunction or function : default BackPropagation
        specifies the function used to calculate the `learning_signal <LearningMechanism.learning_signal>` (assigned
        to the LearningMechanism's `LearningSignal(s) <LearningMechanism_LearningSignal>`), and the `error_signal
        <LearningMechanism.error_signal>` (passed to the next LearningMechanism in a learning sequence for
        `multilayer learning <LearningMechanism_Multilayer_Learning>`).  It takes the following three arguments,
        each of which must be a list or 1d array: **input**,  **output**, and **error** (see
        `LearningMechanism_Function` for additional details).

    learning_rate : float : None
        determines the learning rate for the LearningMechanism.  It is used to specify the :keyword:`learning_rate`
        parameter for the LearningMechanism's `learning function <LearningMechanism.function>`
        (see description of `learning_rate <LearningMechanism_Learning_Rate>` for additional details).

    error_signal : 1d np.array
        one of two values returned by the LearningMechanism's `function <LearningMechanism.function>`.  For
        `single layer learning <LearningMechanism_Single_Layer_Learning>`, this is the same as the value received in
        the LearningMechanism's *ERROR_SIGNAL* `InputState <LearningMechanism_Input_Error_Signal>`;  for `multilayer
        learning <LearningMechanism_Multilayer_Learning>`, it is a modified version of the value received, that takes
        account of the contribution made by the learned_projection and its input to the error signal received. This
        is assigned as the `value <OutputState.value>` of the LearningMechanism's *ERROR_SIGNAL* `OutputState
        <LearningMechanism_Output_Error_Signal>`.

    learning_signal : number, ndarray or matrix
        one of two values returned by the LearningMechanism's `function <LearningMechanism.function>`, that specifies
        the changes to the weights of the `matrix <MappingProjection.matrix>` parameter for the LearningMechanism's
        `learned_projections <LearningMechanism.learned_projections>`;  it is calculated to reduce the error signal
        associated with the `primary_learned_projection <LearningMechanism.primary_learned_projection>` and received
        from the LearningMechanism's `error_source`.  It is assigned as the value of the LearningMechanism's
        `LearningSignal(s) <LearningMechanism_LearningSignal>` and, in turn, its LearningProjection(s).

    learning_signals : ContentAddressableList[LearningSignal]
        list of all of the `LearningSignals <LearningSignal>` for the LearningMechanism, each of which sends one or
        more `LearningProjections <LearningProjection>` to the `ParameterState(s) <ParameterState>` for the `matrix
        <MappingProjection.matrix>` parameter of the `MappingProjection(s) <MappingProjection>` trained by the
        LearningMechanism.  The `value <LearningSignal>` of each LearningSignal is the LearningMechanism's
        `learning_signal <LearningMechanism.learning_signal>` attribute. Since LearningSignals are `OutputStates
        <OutputState>`, they are also listed in the LearningMechanism's `output_states
        <LearningMechanism.output_states>` attribute, after it *ERROR_SIGNAL* `OutputState
        <LearningMechanism_Output_Error_Signal>`.

    learning_projections : List[LearningProjection]
        list of all of the LearningProjections <LearningProject>` from the LearningMechanism, listed in the order of
        the `LearningSignals <LearningSignal>` to which they belong (that is, in the order they are listed in
        the `learning_signals <LearningMechanism>` attribute).

    output_states : ContentAddressableList[OutputState]
        list of the LearningMechanism's `OutputStates <OutputState>`, including its *ERROR_SIGNAL* `OutputState
        <LearningMechanism_Output_Error_Signal>`, followed by its `LearningSignal(s)
        <LearningMechanism_LearningSignal>`, and then any additional (user-specified) `OutputStates <OutputState>`.

    COMMENT:
       #  FIX: THIS MAY NEED TO BE A 3d array (TO ACCOMDATE 2d array (MATRICES) AS ENTRIES)\
    COMMENT

    output_values : 2d np.array
        the first item is the `value <OutputState.value>` of the LearningMechanism's *ERROR_SIGNAL* `OutputState
        <LearningMechanism_Output_Error_Signal>`, followed by the `value <LearningSignal.value>` \\(s) of its
        `LearningSignal(s) <LearningMechanism_LearningSignal>`, and then those of any additional (user-specified)
        `OutputStates <OutputState>`.

    modulation : ModulationParam
        the default form of modulation used by the LearningMechanism's `LearningSignal(s)
        <LearningMechanism_LearningSignal>`, unless they are `individually specified <LearningSignal_Specification>`.

    name : str
        the name of the LearningMechanism; if it is not specified in the **name** argument of the constructor, a
        default is assigned by MechanismRegistry (see `Naming` for conventions used for default and duplicate names).

    prefs : PreferenceSet or specification dict
        the `PreferenceSet` for the LearningMechanism; if it is not specified in the **prefs** argument of the
        constructor, a default is assigned using `classPreferences` defined in __init__.py (see :doc:`PreferenceSet
        <LINK>` for details).

    """

    componentType = LEARNING_MECHANISM
    className = componentType
    suffix = " " + className

    outputStateType = LearningSignal

    stateListAttr = Mechanism_Base.stateListAttr.copy()
    stateListAttr.update({LearningSignal:LEARNING_SIGNALS})

    classPreferenceLevel = PreferenceLevel.TYPE

    # ClassDefaults.variable = None

    paramClassDefaults = AdaptiveMechanism_Base.paramClassDefaults.copy()
    paramClassDefaults.update({
        CONTROL_PROJECTIONS: None,
        INPUT_STATES:input_state_names,
        OUTPUT_STATES:[{NAME:ERROR_SIGNAL,
                        INDEX:1},
                       {NAME:LEARNING_SIGNAL,  # NOTE: This is the default, but is overridden by any LearningSignal arg
                        INDEX:0}
                       ]})

    @tc.typecheck
    def __init__(self,
                 variable:tc.any(list, np.ndarray),
                 size=None,
                 error_source:tc.optional(Mechanism)=None,
                 function:is_function_type=BackPropagation,
                 learning_signals:tc.optional(list) = None,
                 modulation:tc.optional(_is_modulation_param)=ModulationParam.ADDITIVE,
                 learning_rate:tc.optional(parameter_spec)=None,
                 params=None,
                 name=None,
                 prefs:is_pref_set=None,
                 context=None):

        # Assign args to params and functionParams dicts (kwConstants must == arg names)
        params = self._assign_args_to_param_dicts(error_source=error_source,
                                                  function=function,
                                                  learning_signals=learning_signals,
                                                  params=params)

        # # USE FOR IMPLEMENTATION OF deferred_init()
        # # Store args for deferred initialization
        # self.init_args = locals().copy()
        # self.init_args['context'] = self
        # self.init_args['name'] = name
        # delete self.init_args[ERROR_SOURCE]

        # # Flag for deferred initialization
        # self.init_status = InitStatus.DEFERRED_INITIALIZATION

        self._learning_rate = learning_rate

        super().__init__(variable=variable,
                         size=size,
                         modulation=modulation,
                         params=params,
                         name=name,
                         prefs=prefs,
                         context=self)

    def _validate_variable(self, variable, context=None):
        """Validate that variable has exactly three items: activation_input, activation_output and error_signal
        """

        variable = self._update_variable(super()._validate_variable(variable, context))

        if len(variable) != 3:
            raise LearningMechanismError("Variable for {} ({}) must have three items ({}, {}, and {})".
                                format(self.name, variable,
                                       ACTIVATION_INPUT,
                                       ACTIVATION_OUTPUT,
                                       ERROR_SIGNAL))

        # Validate that activation_input, activation_output, and error_signal are numeric and lists or 1d np.ndarrays
        for i in range(len(variable)):
            item_num_string = ['first', 'second', 'third'][i]
            item_name = input_state_names[i]
            if not np.array(variable[i]).ndim == 1:
                raise LearningMechanismError("The {} item of variable for {} ({}:{}) is not a list or 1d np.array".
                                              format(item_num_string, self.name, item_name, variable[i]))
            if not (is_numeric(variable[i])):
                raise LearningMechanismError("The {} item of variable for {} ({}:{}) is not numeric".
                                              format(item_num_string, self.name, item_name, variable[i]))
        return variable

    def _validate_params(self, request_set, target_set=None, context=None):
        """Validate error_source as an Objective Mechanism or another LearningMechanism
        """

        super()._validate_params(request_set=request_set, target_set=target_set,context=context)

        from psyneulink.components.states.state import _parse_state_spec
        from psyneulink.components.states.modulatorysignals.learningsignal import LearningSignal
        from psyneulink.components.projections.pathway.mappingprojection import MappingProjection
        from psyneulink.components.projections.projection import _validate_receiver

        if ERROR_SOURCE in target_set and target_set[ERROR_SOURCE] is not None:
            if not isinstance(target_set[ERROR_SOURCE], (ObjectiveMechanism, LearningMechanism)):
                raise LearningMechanismError("{} arg for {} ({}) must be an ObjectiveMechanism or another "
                                             "LearningMechanism".
                                             format(ERROR_SOURCE, self.name, target_set[ERROR_SOURCE]))

        if LEARNING_SIGNALS in target_set and target_set[LEARNING_SIGNALS]:

            if not isinstance(target_set[LEARNING_SIGNALS], list):
                raise LearningMechanismError("{} arg of {} must be list".
                                           format(LEARNING_SIGNAL, self.name))

            for spec in target_set[LEARNING_SIGNALS]:
                learning_signal = _parse_state_spec(state_type=LearningSignal, owner=self, state_spec=spec)

                # learning_proj = None  # Projection from LearningSignal to MappingProjection
                # mapping_proj = None   # MappingProjection that receives Projection from LearningSignal
                #
                # # Specification is for a LearningSignal
                # if isinstance(spec, LearningSignal):
                #     #  Check that any LearningProjections it has
                #     #    are to MappingProjections to Mechanisms in the same System
                #     for learning_proj in spec.efferents:
                #         _validate_receiver(self,learning_proj, MappingProjection, LEARNING_SIGNAL, context)
                #     continue
                #
                # # Specification is for a ParameterState
                # elif isinstance(spec, ParameterState):
                #     param_name = spec.name
                #     mapping_proj = spec.owner
                #
                # # Specification is for a Projection
                # elif isinstance(spec, Projection):
                #     if isinstance(spec, LearningProjection):
                #         param_name = spec.receiver.name
                #         learning_proj = spec
                #         mapping_proj = learning_proj.receiver.owner
                #     elif isinstance(spec, MappingProjection):
                #         param_name = MATRIX
                #         mapping_proj = spec
                #     else:
                #         raise LearningMechanismError("The {} specified in the {} arg for {} ({}) must be a {}".
                #                                      format(PROJECTION,
                #                                             LEARNING_SIGNALS,
                #                                             self.name,
                #                                             spec.name,
                #                                             MAPPING_PROJECTION))
                #
                # # Specification is for a tuple (str, MappingProjection):
                # elif isinstance(spec, tuple):
                #     param_name = spec[0]
                #     mapping_proj = spec[1]
                #     # Check that 1st item is a str (presumably the name of the learned Projection's attribute
                #     #    for the param to be learned; e.g., 'MATRIX' for MappingProjection)
                #     if not isinstance(param_name, str):
                #         raise LearningMechanismError("1st item of tuple in specification of {} for {} ({}) "
                #                                      "must be a string".format(LEARNING_SIGNAL, self.name, param_name))
                #     # Check that 2nd item is a MappingProjection
                #     if not isinstance(mapping_proj, MappingProjection):
                #         raise LearningMechanismError("2nd item of tuple in specification of {} for {} ({}) "
                #                                      "must be a {}".
                #                                      format(LEARNING_SIGNAL,
                #                                             self.name,
                #                                             mapping_proj,
                #                                             MAPPING_PROJECTION))
                #
                # # LearningSignal specification dictionary, must have the following entries:
                # #    NAME:str - must be the name of an attribute of PROJECTION
                # #    PROJECTION:Projection - must be a MappingProjection
                # #                            and have an attribute and corresponding ParameterState named NAME
                # #    PARAMS:dict - entries must be valid LearningSignal parameters (e.g., LEARNING_RATE)
                # elif isinstance(spec, dict):
                #     if not NAME in spec:
                #         raise LearningMechanismError("Specification dict for {} of {} must have a {} entry".
                #                                     format(LEARNING_SIGNAL, self.name, NAME))
                #     param_name = spec[NAME]
                #     if not PROJECTION in spec:
                #         raise LearningMechanismError("Specification dict for {} of {} must have a {} entry".
                #                                     format(LEARNING_SIGNAL, self.name, PROJECTION))
                #     mapping_proj = spec[PROJECTION]
                #     if not isinstance(mapping_proj, MappingProjection):
                #         raise LearningMechanismError("{} entry of specification dict for {} of {} must be a {}".
                #                                     format(PROJECTION, LEARNING_SIGNAL, self.name, MAPPING_PROJECTION))
                #     # Check that all of the other entries in the specification dictionary
                #     #    are valid LearningSignal params
                #     for param in spec:
                #         if param in {NAME, PROJECTION}:
                #             continue
                #         if not hasattr(LearningSignal, param):
                #             raise LearningMechanismError("Entry in specification dictionary for {} arg of {} ({}) "
                #                                        "is not a valid {} parameter".
                #                                        format(LEARNING_SIGNAL, self.name, param,
                #                                               LearningSignal.__class__.__name__))
                # else:
                #     raise LearningMechanismError("PROGRAM ERROR: unrecognized specification for {} arg of {} ({})".
                #                                 format(LEARNING_SIGNALS, self.name, spec))
                #     # raise LearningMechanismError("Specification of {} for {} ({}) must be a "
                #     #                             "ParameterState, Projection, a tuple specifying a parameter and "
                #     #                              "Projection, a LearningSignal specification dictionary, "
                #     #                              "or an existing LearningSignal".
                #     #                             format(CONTROL_SIGNAL, self.name, spec))

                # Validate that the receiver of the LearningProjection (if specified)
                #     is a MappingProjection and in the same System as self (if specified)
                try:
                    for learning_projection in  learning_signal[PARAMS][PROJECTIONS]:
                        _validate_receiver(sender_mech=self,
                                           projection=learning_projection,
                                           expected_owner_type=MappingProjection,
                                           spec_type=LEARNING_SIGNAL,
                                           context=context)
                except KeyError:
                    pass

                # # IMPLEMENTATION NOTE: the tests below allow for the possibility that the MappingProjection
                # #                      may not yet be fully implemented (e.g., this can occur if the
                # #                      LearningMechanism being implemented here is as part of a LearningProjection
                # #                      specification for the MappingProjection's matrix param)
                # # Check that param_name is the name of a parameter of the MappingProjection to be learned
                # if not param_name in (set(mapping_proj.user_params) | set(mapping_proj.user_params[FUNCTION_PARAMS])):
                #     raise LearningMechanismError("{} (in specification of {} for {}) is not an "
                #                                 "attribute of {} or its function"
                #                                 .format(param_name, LEARNING_SIGNAL, self.name, mapping_proj))
                # # Check that the MappingProjection to be learned has a ParameterState for the param
                # if mapping_proj._parameter_states and not param_name in mapping_proj._parameter_states.names:
                #     raise LearningMechanismError("There is no ParameterState for the parameter ({}) of {} "
                #                                 "specified in {} for {}".
                #                                 format(param_name, mapping_proj.name, LEARNING_SIGNAL, self.name))

    def _instantiate_attributes_before_function(self, context=None):
        """Instantiates MappingProjection from error_source (if specified) to the LearningMechanism

        Also assigns learned_projection attribute (to MappingProjection being learned)
        """

        super()._instantiate_attributes_before_function(context=context)

        if self.error_source:
            _instantiate_error_signal_projection(sender=self.error_source, receiver=self)

    def _instantiate_attributes_after_function(self, context=None):

        if self._learning_rate is not None:
            self.learning_rate = self._learning_rate

        super()._instantiate_attributes_after_function(context=context)

    def _instantiate_output_states(self, context=None):

        from psyneulink.globals.registry import register_category
        from psyneulink.components.states.modulatorysignals.learningsignal import LearningSignal
        from psyneulink.components.states.state import State_Base, _instantiate_state

        # Create registry for LearningSignals (to manage names)
        register_category(entry=LearningSignal,
                          base_class=State_Base,
                          registry=self._stateRegistry,
                          context=context)

        # Instantiate LearningSignals if they are specified, and assign to self._output_states
        # Note: if any LearningSignals are specified they will replace the default LEARNING_SIGNAL OutputState
        #          in the OUTPUT_STATES entry of paramClassDefaults;
        #       the LearningSignals are appended to _output_states, leaving ERROR_SIGNAL as the first entry.
        if self.learning_signals:

            # Delete default LEARNING_SIGNAL item in output_states
            del self._output_states[1]
            for learning_signal in self.learning_signals:
                # Instantiate LearningSignal

                params = {LEARNED_PARAM:MATRIX}

                # Parses learning_signal specifications (in call to State._parse_state_spec)
                #    and any embedded Projection specifications (in call to <State>._instantiate_projections)
                learning_signal = _instantiate_state(state_type=LearningSignal,
                                                     owner=self,
                                                     params=params,
                                                     reference_value=self.learning_signal,
                                                     modulation=self.modulation,
                                                     # state_spec=self.learning_signal)
                                                     state_spec=learning_signal)
                # Add LearningSignal to output_states list
                self._output_states.append(learning_signal)

            # Assign LEARNING_SIGNAL as the name of the 1st LearningSignal; the names of any others can be user-defined
            first_learning_signal = next(state for state in self.output_states if isinstance(state, LearningSignal))
            first_learning_signal.name = LEARNING_SIGNAL

        super()._instantiate_output_states(context=context)

        # Reassign learning_signals to capture any user_defined LearningSignals instantiated by in call to super
        #   and to assign to a ContentAddressableList
        self._learning_signals = ContentAddressableList(component_type=LearningSignal,
                                                        list=[state for state in self.output_states if
                                                                  isinstance(state, LearningSignal)])

    def _execute(self,
                variable=None,
                runtime_params=None,
                clock=CentralClock,
                time_scale = TimeScale.TRIAL,
                context=None):
        """Execute LearningMechanism function and return learning_signal

        :return: (2D np.array) self.learning_signal
        """

        # COMPUTE LEARNING SIGNAL (dE/dW):
        self.learning_signal, self.error_signal = self.function(variable=variable,
                                                                params=runtime_params,
                                                                context=context)

        if not INITIALIZING in context and self.reportOutputPref:
            print("\n{} weight change matrix: \n{}\n".format(self.name, self.learning_signal))

        self.value = [self.learning_signal, self.error_signal]
        return self.value

    @property
    def learning_enabled(self):
        try:
            return self._learning_enabled
        except AttributeError:
            self._learning_enabled = True
            return self._learning_enabled

    @learning_enabled.setter
    def learning_enabled(self, assignment):
        self._learning_enabled = assignment

    @property
    def learning_rate(self):
        return self.function_object.learning_rate

    @learning_rate.setter
    def learning_rate(self, assignment):
        self.function_object.learning_rate = assignment

    @property
    def input_source(self):
        try:
            return self.input_states[ACTIVATION_INPUT].path_afferents[0].sender.owner
        except IndexError:
            return None

    @property
    def output_source(self):
        try:
            return self.input_states[ACTIVATION_OUTPUT].path_afferents[0].sender.owner
        except IndexError:
            return None

    @property
    def primary_learned_projection(self):
        return self.learned_projection[0]

    @property
    def learned_projections(self):
        return [lp.receiver.owner for lp in self.learning_projections]


# IMPLEMENTATION NOTE:  THIS SHOULD BE MOVED TO COMPOSITION ONCE THAT IS IMPLEMENTED
def _instantiate_error_signal_projection(sender, receiver):
    """Instantiate a MappingProjection to carry an error_signal to a LearningMechanism

    Can take as the sender an `ObjectiveMechanism` or a `LearningMechanism`.
    If the sender is an ObjectiveMechanism, uses its `primary OutputState <OutputState_Primary>`.
    If the sender is a LearningMechanism, uses its `ERROR_SIGNAL <LearningMechanism.output_states>` OutputState.
    The receiver must be a LearningMechanism; its `ERROR_SIGNAL <LearningMechanism.input_states>` InputState is used.
    Uses and IDENTITY_MATRIX for the MappingProjection, so requires that the sender be the same length as the receiver.

    """
    from psyneulink.components.projections.pathway.mappingprojection import MappingProjection

    if isinstance(sender, ObjectiveMechanism):
        sender = sender.output_states[OUTCOME]
    elif isinstance(sender, LearningMechanism):
        sender = sender.output_states[ERROR_SIGNAL]
    else:
        raise LearningMechanismError("Sender of the error signal Projection {} must be either "
                                     "an ObjectiveMechanism or a LearningMechanism".
                                     format(sender))

    if isinstance(receiver, LearningMechanism):
        receiver = receiver.input_states[ERROR_SIGNAL]
    else:
        raise LearningMechanismError("Receiver of the error signal Projection {} must be a LearningMechanism".
                                     format(receiver))

    if len(sender.value) != len(receiver.value):
        raise LearningMechanismError("The length of the OutputState ({}) for the sender ({}) of "
                                     "the error signal Projection does not match "
                                     "the length of the InputState ({}) for the receiver ({})".
                                     format(len(sender.value), sender.owner.name,
                                            len(receiver.value),receiver.owner.name))

    return MappingProjection(sender=sender,
                             receiver=receiver,
                             matrix=IDENTITY_MATRIX,
                             name = sender.owner.name + ' ' + OUTCOME)
