# Princeton University licenses this file to You under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.  You may obtain a copy of the License at:
#     http://www.apache.org/licenses/LICENSE-2.0
# Unless required by applicable law or agreed to in writing, software distributed under the License is distributed
# on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and limitations under the License.


# *******************************************  LearningMechanism *******************************************************

"""

Contents
--------

  * `LearningMechanism_Overview`
  * `LearningMechanism_Creation`
      - `LearningMechanism_Automatic_Creation`
      - `LearningMechanism_Explicit_Creation`
  * `LearningMechanism_Structure`
      - `LearningMechanism_InputPorts`
      - `LearningMechanism_Function`
      - `LearningMechanism_OutputPorts`
      - `LearningMechanism_Additional_Attributes`
      - `LearningMechanism_Learning_Configurations`
  * `LearningMechanism_Execution`
  * `LearningMechanism_Class_Reference`


.. _LearningMechanism_Overview:

Overview
--------

A LearningMechanism is a `ModulatoryMechanism <ModulatoryMechanism>` that modifies the `matrix <MappingProjection.matrix>`
parameter of one or more `MappingProjections <MappingProjection>`.  Its function takes one or more `error_signals
<LearningMechanism_Input_Error_Signal>` (usually the output of a `ComparatorMechanism` or one or more other
`LearningMechanisms <LearningMechanism>`), as well as information about the `MappingProjection(s) and activity
<LearningMechanism_Additional_Attributes>` that generated the error(s), and calculates a `learning_signal
<LearningMechanism.learning_signal>` that is used to modify the MappingProjection(s) by way of its
`LearningProjection(s) <LearningProjection>`.  Typically, a LearningMechanism is used to "train" a single
MappingProjection (its `primary_learned_projection`), using the output of the Mechanism to which that
MappingProjection projects (its `output_source`) as the source of the error it attempts to reduce.  A
LearningMechanism can be used to train multiple MappingProjections, by assigning it `additional LearningProjections
<LearningMechanism_Multiple_LearningSignals>`; however, these will all use the same `learning_signal
<LearningMechanism.learning_signal>`, generated by the `primary_learned_projection` and its associated `output_source`.
All of the MappingProjection(s) modified by a LearningMechanism must project from one `ProcessingMechanism
<ProcessingMechanism>` to another in the same `Composition`. The learning components of a Composition can be
displayed using the Composition's `show_graph`show_graph <ShowGraph.show_graph>` method with its
**show_learning** argument assigned `True` or *ALL*.

.. _LearningMechanism_Note

*A Note about the Implementation of Learning*
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

The implementation of learning in PsyNeuLink was designed for exposition rather than efficiency. Unlike its
implementation in most other environments -- where the learning algorithm is tightly integrated with the
elements of processing that it modifies --  PsyNeuLink separates it into three constituent components, often including:
an `ObjectiveMechanism` used to evaluate the most proximal source of error; a `LearningMechanism` that uses that error
(or one derived from it by another LearningMechanism) to calculate a learning signal;  and one or more
`LearningProjections <LearningProjection>` that use that learning signal to modify the weight `matrix
<MappingProjection.matrix>` of the `MappingProjection(s) <MappingProjection>` being learned.  This has the advantage
of isolating and exposing the constituent computations, making it clearer what these are and how they operate, and
also making each individually accessible for reconfiguration. However, it comes at the cost of efficiency.  For
efficient execution of supervised forms of learning (e.g., reinforcement learning and backpropagation),
an `AutodiffComposition` can be used, which allows the model to be specified using PsyNeuLink, but actually executes
learning either in a compiled form or using `PyTorch <https://pytorch.org>` (see `Compilation` for additional details).

.. _LearningMechanism_Creation:

Creating a LearningMechanism
----------------------------

A LearningMechanism can be created in any of the ways used to `create Mechanisms <Mechanism_Creation>`.
More commonly, however, LearningMechanisms are created automatically.

.. LearningMechanism_Automatic_Creation:

*Automatic Creation*
~~~~~~~~~~~~~~~~~~~~

A LearningMechanism is created automatically when a Composition's `learning method <Composition_Learning_Methods>` is
called.  In that case, a `LearningSignal`, `LearningProjection`, a `ComparatorMechanism` (in the case of `supervised
learning <>`), and any additional Projections required to implement learning that do not already exist are also
instantiated.  This is described below, under `Learning Configurations <LearningMechanism_Learning_Configurations>`.
A LearningMechanism is also created automatically when either the `tuple specification
<MappingProjection_Learning_Tuple_Specification>` is used to specify learning for a MappingProjection, or a
`LearningProjection` is created without specifying its `sender <LearningProjection.sender>` attribute. However, this
is not advised, and should only used in special circumstances, as properly configuring learning generally requires
the instantiation of several other closely related Components, as described below.

.. _LearningMechanism_Explicit_Creation:

*Explicit Creation*
~~~~~~~~~~~~~~~~~~~

If a LearningMechanism is created explicitly (using its constructor), then its **variable** and **error_sources**
arguments must be specified.  The **variable** must have at leaset three items that are compatible (in number and type)
with the `value <InputPort.value>` of the LearningMechanism's `InputPorts <LearningMechanism_InputPorts>`.  Each
item in **error_sources** must be one of the following: a `ComparatorMechanism`, for `single layer learning
<LearningMechanism_Single_Layer_Learning>` or for the last `MappingProjection` in a `learning Pathway
<Composition_Learning_Pathway>` for `multilayer learning <LearningMechanism_Multilayer_Learning>`; or a
`LearningMechanism`.

.. _LearningMechanism_Learning_Signals:

When a LearningMechanism is created explicitly, it can also be assigned existing LearningSignals and/or specified to
create these, as well as `LearningProjections <LearningProjection>` from these to specified MappingProjections.  These
are specified in the **learning_signals** argument of the LearningMechanism's constructor, using any of the forms
allowed for `specifying a LearningSignal <LearningSignal_Specification>`.

.. _LearningMechanism_Structure:

Structure
---------

A LearningMechanism has three types of `InputPorts <InputPort>`, a learning `function <LearningMechanism.function>`,
and two types of `OutputPorts <OutputPort>`. These are used, respectively, to receive, compute, and transmit the
information needed to modify the MappingProjection(s) for which the LearningMechanism is responsible.  In addition,
it has several attributes that govern and provide access to its operation.  These are described below.

.. _LearningMechanism_InputPorts:

*InputPorts*
~~~~~~~~~~~~~

These receive the information required by the LearningMechanism's `function <LearningMechanism.function>`.  They are
listed in the LearningMechanism's `input_ports <LearningMechanism.input_ports>` attribute.  They have the following
names and roles (shown in the `figure <LearningMechanism_Single_Layer_Learning_Figure>` below):

.. _LearningMechanism_Activation_Input:

* *ACTIVATION_INPUT* - receives the value of the input to the `primary_learned_projection`; that is, the `value
  <Projection_Base.value>` of that MappingProjection's `sender <MappingProjection.sender>`. The value is assigned
  as the first item of the LearningMechanism's `variable <LearningMechanism.variable>` attribute.

.. _LearningMechanism_Activation_Output:

* *ACTIVATION_OUTPUT* - receives the value of the LearningMechanism's `output_source <LearningMechanism.output_source>`;
  that is, the `value <OutputPort.value>` of the `OutputPort` of the *ProcessingMechanism* to which the
  `primary_learned_projection` projects.  By default, the `output_source <LearningMechanism.output_source>`'s
  `primary OutputPort <OutputPort_Primary>` is used.  However, a different OutputPort can be designated in
  the constructor for the `output_source <LearningMechanism.output_source>`, by assigning a `parameter specification
  dictionary <ParameterPort_Specification>` to the **params** argument of its constructor, with an entry that uses
  *MONITOR_FOR_LEARNING* as its key and a list containing the desired OutputPort(s) as its value. The `value
  <InputPort.value>` of the *ACTIVATION_OUTPUT* InputPort is assigned as the second item of the LearningMechanism's
  `variable <LearningMechanism.variable>` attribute.

.. _LearningMechanism_Input_Error_Signal:

* *ERROR_SIGNAL* - this receives the `value <OutputPort.value>` from the *OUTCOME* `OutputPort
  <ComparatorMechanism_Structure>` of a `ComparatorMechanism`, or of the *ERROR_SIGNAL* OutputPort of another
  `LearningMechanisms <LearningMechanism_Output_Error_Signal>`. If the `primary_learned_projection` projects
  to the `TERMINAL` Mechanism of the `Composition` to which it belongs, or is not part of a `multilayer learning
  sequence <LearningMechanism_Multilayer_Learning>`, then the LearningMechanism has a single *ERROR_SIGNAL* InputPort,
  that receives its input from a ComparatorMechanism. If the `primary_learned_projection` is part of a `multilayer
  learning pathway <LearningMechanism_Multilayer_Learning>`, then the LearningMechanism will have one or more
  *ERROR_SIGNAL* InputPorts, that receive their input from the next LearningMechanism(s) in the sequence;  that is,
  the one(s) associated with the `efferents <Port_Base.efferents>` (outgoing Projections) of its `output_source`,
  with one *ERROR_SIGNAL* InputPort for each of those Projections.  The `value <InputPort.value>`\\s of the
  *ERROR_SIGNAL* InputPorts are summed by the LearningMechanism's `function <LearningMechanism.function>` to
  calculate the `learning_signal <LearningMechanism.learning_signal>` (see `below <LearningMechanism_Function>`);
  note that the value of the *ERROR_SIGNAL* InputPort may not be the same as that of the LearningMechanism's
  `error_signal <LearningMechanism.error_signal>` attribute or *ERROR_SIGNAL* `OutputPort
  <LearningMechanism_Output_Error_Signal>` (see `note <LearningMechanism_Error_Signal>` below).  If a LearningMechanism
  has more than one *ERROR_SIGNAL* InputPort, their names are suffixed with a hyphenated index, that is incremented for
  each additional InputPort (e.g., ``error_signal-1``, ``error_signal-2``, etc.).  These are listed in the
  LearningMechanism's `error_signal_input_ports` attribute, and the `value <InputPort.value>` of each is assigned
  as an item of the LearningMechanism's `variable <LearningMechanism.variable>` attribute, beginning with its third
  item (i.e., following the `value <InputPort.value>` of the *ACTIVATION_INPUT* and *ACTIVATION_VALUE* InputPorts).

.. _LearningMechanism_Covariates:

* *COVARIATES* - one or more InputPorts that receive the value of the inputs to `output_source (i.e., the
  *ProcessingMechanism* to which the `primary_learned_projection` projects) other than `primary_learned_projection
  <LearningMechanism_Primary_Learned_Projection>`, if the `output_source <LearningMechanism.output_source>`'s
  `function <Mechanism_Base.function>` takes more than one argument and they affect its derivative. These are
  assigned as a list to the fourth item in the LearningMechanism's `variable <LearningMechanism.variable>`
  attribute, and are used to calculate the `learning_signal <LearningMechanism.learning_signal>` (see `below
  <LearningMechanism_Function>`).

  .. note::
     The LearningMechanism has a separate InputPort for each covariate, however their
     values are combined into a single item of the LearningMechanism's `variable <LearningMechanism.variable>`
     that is provided to its `function <LearningMechanism.function>` at the fourth position.

  .. technical_note::
     Although the *COVARIATES* InputPorts receive Projections from the same senders as the corresponding
     Projections to the `output_source <LearningMechanism.output_source>`, these are for display only,
     and are not to actually compute the LearningSignal, since they are not subject to learning and therefore
     do not provide the same value as the actual Projections they shadow.  Rather, those values are accessed
     directly by the LearningMechanism from the InputPorts of the `output_source <LearningMechanism.output_source>`,
     in a manner similar to its access of the `error_matrices <LearningMechanism.error_matrices>` from the
     `ParameterPort`s of the `error_sources <LearningMechanism.error_sources>`.

The Mechanisms from the which the `value <InputPort.values>`\\s above are received are listed in the
LearningMechanism's `input_source <LearningMechanism.input_source>`, `output_source <LearningMechanism.output_source>`,
and `error_sources <LearningMechanism.error_sources>` attributes, respectively (see
`LearningMechanism_Additional_Attributes` for additional details).


.. _LearningMechanism_Function:

*Learning Function*
~~~~~~~~~~~~~~~~~~~

The `function <LearningMechanism.function>` of a LearningMechanism uses the values received by the Mechanism's
InputPorts (described `above <LearningMechanism_InputPorts>`) to calculate the value of its `learning_signal
<LearningMechanism.learning_signal>` and `error_signal <LearningMechanism.error_signal>` attributes.

.. _LearningMechanism_Learning_Signal:

* `learning_signal` - the set of changes to the `matrix <MappingProjection.matrix>` parameter of the
  `MappingProjections <MappingProjection>` being learned, calculated to reduce the summed value of the
  LearningMechanism's *ERROR_SIGNAL* `InputPort(s) <LearningMechanism_Input_Error_Signal>`.

.. _LearningMechanism_Error_Signal:

* `error_signal <LearningMechanism.error_signal>` - the contribution made by the `primary_learned_projection` to the
  error_signal(s) received by the LearningMechanism's *ERROR_SIGNAL* `InputPort(s)
  <LearningMechanism_Input_Error_Signal>`. It is used by the LearningMechanism's `function <LearningMechanism.function>`
  to calculate the `learning_signal <LearningMechanism.learning_signal>`. Depending upon the context and specific
  `LearningFunction <LearningFunctions>` used, it may also take account of the `value <Mechanism_Base.value>` of its
  `output_source`, as well as the `matrix <MappingProjection.matrix>` parameter of any of the `output_source`'s
  outgoing Projections that are also being learned (these are listed in the LearningMechanism's `error_matrices
  <LearningMechanism.error_matrices>` attribute).  The value of the `error_signal <LearningMechanism.error_signal>`
  is assigned as the value of the LearningMechanism's *ERROR_SIGNAL* `OutputPort
  <LearningMechanism_Output_Error_Signal>`.

  .. _LearningMechanism_Error_Signal_Note:

  .. note::

     A LearningMechanism's *ERROR_SIGNAL* `InputPort(s) <LearningMechanism_Input_Error_Signal>` and its
     *ERROR_SIGNAL* `OutputPort <LearningMechanism_Output_Error_Signal>` may not have the same value.
     The former are the error signal(s) received from a `ComparatorMechanism` or one or more `LearningMechanisms
     <LearningMechanism>`, while the latter is the contribution made to those errors by the `primary_learned_projection`
     and the `output_source`, as calculated by the LearningMechanism's `function <LearningMechanism.function>`
     (see `error_signal <LearningMechanism_Error_Signal>` above).

The default `function <LearningMechanism.function>` of a LearningMechanism is `BackPropagation` (also known as the
*Generalized Delta Rule*; see
`Rumelhart et al., 1986 <http://www.nature.com/nature/journal/v323/n6088/abs/323533a0.html>`_).  However, it can be
assigned any other PsyNeuLink `LearningFunction <LearningFunctions>`, or any other Python function that takes as its
input a list or np.array containing three lists or 1d np.arrays of numeric values, and returns two lists or 1d
np.arrays.  The two values it returns are assigned to the LearningMechanism's `learning_signal
<LearningMechanism.learning_signal>` and `error_signal <LearningSignal.error_signal>` attributes, respectively,
as well as to its two OutputPorts, as described below.

.. _LearningMechanism_OutputPorts:

*OutputPorts*
~~~~~~~~~~~~~~

By default, a LearningMechanism has two `OutputPorts <OutputPort>`, the first of which is named *ERROR_SIGNAL* and
is assigned the value of the `error_signal <LearningMechanism.error_signal>` returned by the LearningMechanism's
`function <LearningMechanism.function>`, and the second of which is a `LearningSignal` and is assigned the value of the
`learning_signal <LearningMechanism.learning_signal>` returned by the `function <LearningMechanism.function>`.
They are each described below:

.. _LearningMechanism_Output_Error_Signal:

* *ERROR_SIGNAL* - this is the `primary OutputPort <OutputPort_Primary>` of a LearningMechanism, and  receives the
  value of the `error_signal <LearningMechanism.error_signal>` used to calculate the `learning_signal
  <LearningMechanism.learning_signal>`.  Its value is assigned as the first item of the LearningMechanism's
  `output_values <LearningMechanism.output_values>` attribute.  If the LearningMechanism is part of a `multilayer
  learning pathway <LearningMechanism_Multilayer_Learning>`, the *ERROR_SIGNAL* OutputPort is assigned a Projection
  to the LearningMechanism for the preceding MappingProjection in the sequence being learned - see `figure
  <LearningMechanism_Multilayer_Learning_Figure>` below).  Note that the `value <OutputPort.value>` of the
  *ERROR_SIGNAL* OutputPort may not be the same as that of the LearningMechanism's *ERROR_SIGNAL* `InputPorts
  <LearningMechanism_Input_Error_Signal>` (see `error_signal <LearningMechanism_Error_Signal>`).

.. _LearningMechanism_LearningSignal:

* `LearningSignal(s) <LearningSignal>` - by default, a LearningMechanism has a single LearningSignal, which is a
  special type of OutputPort that receives the `learning_signal <LearningMechanism.learning_signal>` generated by the
  LearningMechanism's `function <LearningMechanism.function>`, and used to modify the `matrix
  <MappingProjection.matrix>` parameter of the `primary_learned_projection`.  The LearningSignal is assigned as the
  second item in the list of the LearningMechanism's OutputPorts (i.e., of its `output_ports
  <LearningMechanism.output_ports>` attribute), and its `value <LearningSignal.value>` is assigned as the second
  item of the LearningMechanism's `output_values <LearningMechanism.output_values>` attribute.

.. _LearningMechanism_Multiple_LearningSignals:

  **Multiple LearningSignals and LearningProjections.** Though not common, it is possible for a LearningMechanism to
  be assigned more than one `LearningSignal`, and/or more than one `LearningProjection` to its LearningSignal(s). This
  allows multiple MappingProjections to be trained by a single LearningMechanism. Note, however, that all of the
  LearningSignals of a LearningMechanism (and therefore its LearningProjections) use the same
  `learning_signal <LearningMechanism.learning_signal>`, that is calculated based on the LearningMechanism's
  `primary_learned_projection` and its associated `output_source`.  This can be useful in some settings, such as for
  certain forms of `convolutional neural networks <https://en.wikipedia.org/wiki/Convolutional_neural_network>`_.

  If all of the LearningProjections are used to implement the same form of `modulation <ModulatorySignal_Modulation>`,
  (determined by their LearningSignals' `modulation <LearningSignal.modulation>` attribute), then they should be
  assigned to a single LearningSignal. Multiple LearningProjections can be assigned to a LearningSignal by specifying
  them in the **projections** argument of its constructor, or the *PROJECTIONS* entry of a dictionary assigned to its
  **params** argument); however, the `matrix <MappingProjection.matrix>` parameter for the MappingProjection to which
  they project must have the same shape for all of them. If different forms of modulation are required to train
  different MappingProjections, then multiple LearningSignals should be assigned to the LearningMechanism, each
  specified for one of the required types of modulation, and then LearningProjections assigned to them for the
  corresponding MappingProjections. Multiple LearningSignals can be specified for a LearningMechanism by including
  them in a list assigned to the **learning_signals** argument of the LearningMechanism's constructor.

  COMMENT:
  The `learning_rate <LearningSignal.learning_rate>` for each LearningSignal, and the `learning_rate
  <LearningProjection.learning_rate>` for each of its `LearningProjections <LearningProjection>`, can all be assigned
  different values (with the latter taking precedence over the former).  If none of these are specified, the
  `learning_rate <LearningMechanism.learning_rate>` of the LearningMechanism is used (see `below
  <LearningMechanism_Learning_Rate>`).
  COMMENT

  All of the LearningSignals of a LearningMechanism are listed in its `learning_signals` attribute.  Because these
  are `OutputPorts <OutputPort>`, they are also listed in the `output_ports <LearningMechanism.output_ports>`
  attribute, after the *ERROR_SIGNAL* OutputPort.  All of the LearningMechanism's LearningProjections (that is, those
  belonging to all of its LearningSignals) are listed in its `learning_projections` attribute.

.. _LearningMechanism_Additional_Attributes:

*Additional Attributes*
~~~~~~~~~~~~~~~~~~~~~~~

In addition to its `InputPorts <LearningMechanism_InputPorts>`, `function <LearningMechanism_Function>` and
`OutputPorts <LearningMechanism_OutputPorts>`, a LearningMechanism has the following attributes that
refer to the Components being learned and/or its operation:

.. _LearningMechanism_Primary_Learned_Projection:

* `primary_learned_projection` - the `MappingProjection` with the `matrix <MappingProjection.matrix>` parameter for
  which the `learning_signal <LearningMechanism.learning_signal>` is computed.  This is always the first Projection
  listed first in LearningMechanism's `learned_projections` attribute.
..
* `learned_projections` - a list of all of the MappingProjections to which the LearningMechanism sends a
  `LearningProjection`, listed in the order of the `LearningSignal(s) <LearningSignal>` to which they belong,
  as those are listed in the LearningMechanism's `learning_signals <LearningMechanism.learning_signals>` attribute.
..
* `learning_enabled <LearningMechanism.learning_enabled>` - determines whether and when the LearningMechanism's
  `learning_projections <LearningMechanism.learning_priojections>` are executed (see `learning_enabled
  <LearningMechanism.learning_enabled>` for additional details).
..
* `input_source` - the `Mechanism <Mechanism>` that sends the `primary_learned_projection`, and projects to the
  LearningMechanism's *ACTIVATION_INPUT* `InputPort <LearningMechanism_Activation_Input>`.
..
* `output_source` - the `Mechanism <Mechanism>` that receives the `primary_learned_projection`, and  provides the
  input to the LearningMechanism's *ACTIVATION_OUTPUT* `InputPort <LearningMechanism_Activation_Output>`.
..
* `error_sources` - a `ComparatorMechanism`, `LearningMechanism`, or list of them that calculate the error signal(s)
  provided to the LearningMechanism's *ERROR_SIGNAL(s)* `InputPort(s) <LearningMechanism_Input_Error_Signal>`.
..
* `error_matrices` - the `matrix <MappingProjection.matrix>` parameters of the Projections associated with the
  `error_sources <LearningMechanism.error_sources>`;  that is, of any of the `output_source
  <LearningMechanism.output_source>`'s `efferents <Port_Base.efferents>` that are also being learned.
..
* `covariates_sources` - the `InputPort`s of `Mechanism`(s) that provide covariates used in calculating the derivative
  of the `output_source <LearningMechanism.output_source>`'s `function <Mechanism_Base.function>` (see `above
  `LearningMechanism_Covariates`), and project to its *COVARIATES* `InputPort <LearningMechanism_Covariates>`.
..
* `modulation` - the default value used for the `modulation <LearningSignal.modulation>` attribute of
  LearningMechanism's `LearningSignals <LearningSignal>` (i.e. those for which it is not explicitly specified).
  This determines the way in which the `learning_signal <LearningMechanism.learning_signal>` is used to modify the
  `matrix <MappingProjection.matrix>` parameter of the `learned_projections`.  By default its value is
  Modulation.ADD, which causes the weight changes in the `learning_signal` to be added to the current value of the
  `matrix <MappingProjection.matrix>` parameter (see `LearningMechanism_Execution` for a description of how the
  modifications are executed).

.. _LearningMechanism_Learning_Rate:

* `learning_rate <LearningMechanism.learning_rate>` - specifies the `learning_rate <LearningFunction.learning_rate>`
  parameter used by the LearningMechanism's `function <LearningMechanism.function>` which, for most forms of learning,
  uses it to multiply the weight change matrix before returning it as the `learning_signal
  <LearningMechanism.learning_signal>`.  The value can be specified in the **learning_rate** argument of the
  LearningMechanism's constructor, or in the constructor for its `function <LearningMechanism.function>`.  If both
  are specified, the specification for the function takes precedence; in either case, the value of the learning_rate
  parameter is always the same for the LearningMechanism and its function.  If neither is specified, then it inherits
  its value from any specifications made for the `Composition` or a `learning Pathway <Composition_Learning_Pathway>`
  of the Composition to which it belongs (see `Composition_Learning_Rate` for additional details). However, after the
  Composition and/or its `learning pathways <Composition_Learning_Pathway>` have been constructed, specifying the
  `learning_rate <LearningMechanism.learning_rate>` for a LearningMechanism (or its function) overrides any
  specifications made for the Composition or its learning pathways, including in calls the Composition's `learn
  <Composition.learn>` method.  In this way, individual LearningMechanisms can be assigned specific learning rates
  that apply whenever those are executed. If the learning_rate is not explicitly specified anywhere, the `default
  value <Parameter_Defaults>` for the LearningMechanism's `function <LearningMechanism.function>` is used.

  COMMENT: TBI
  A `learning_rate` parameter can also be specified for individual `LearningSignals <LearningSignal>` and/or their
  associated `LearningProjections <LearningProjection>`. Those have a direct multiplicative effect on the
  `learning_signal <LearningProjection.learning_signal>` of the LearningSignal and/or it LearningProjections (see
  `LearningSignal learning_rate <LearningSignal_Learning_Rate>` for additional details).
  A `learning_rate` parameter can also be specified for individual `LearningProjections <LearningProjection>`.
  Those have a direct multiplicative effect on the `learning_signal <LearningProjection.learning_signal>` of the
  LearningSignal (see `LearningProjection_Function_and_Learning_Rate>` for additional details).
  COMMENT

.. _LearningMechanism_Learning_Configurations:

COMMENT:
@@@ THE FOLLOWING SECTIONS SHOULD BE MOVED TO THE "USER'S MANUAL" WHEN THAT IS WRITTEN
COMMENT

*Learning Configurations*
~~~~~~~~~~~~~~~~~~~~~~~~~

When learning is enabled for a `RecurrentTransferMechanism <RecurrentTransferMechanism_Learning>` (for `unsupervised
learning <Composition_Learning_Unsupervised>`) or using the `learning method <Composition_Learning_Methods>` of a
Composition, all of the Components required for learning are created automatically. The types of Components that are
generated depend on the type of learning specified and the configuration of the `Composition <Composition>`, as
described below. All of the learning Components of a Composition can be displayed using its `show_graph
<ShowGraph.show_graph` method with the **show_learning** argument assigned `True` or *ALL*.

.. _LearningMechanism_Single_Layer_Learning:

Single layer learning
^^^^^^^^^^^^^^^^^^^^^

This configuration occurs when `unsupervised learning <Composition_Learning_Unsupervised>` is used, or `supervised
learning <Composition_Learning_Supervised>` is used for a pathway in a Composition with only two Mechanisms (i.e.,
for the Projection between them).  In this case, a single `ComparatorMechanism` and LearningMechanism are created
(if they do not already exist) as well as the following MappingProjections:

* from an `OutputPort` of the LearningMechanism's `output_source` to the ComparatorMechanism's *SAMPLE* `InputPort
  <ComparatorMechanism_Structure>`.  By default, the `primary OutputPort <OutputPort_Primary>` of the
  `output_source` is used; however, this can be modified by specifying its *MONITOR_FOR_LEARNING* parameter
  (see `above <LearningMechanism_Activation_Output>`).
..
* from the `TARGET_MECHANISM <Composition_Learning_Components>` in the Composition to the ComparatorMechanism's
  *TARGET* `InputPort <ComparatorMechanism_Structure>`;
..
* from the ComparatorMechanism's *OUTCOME* `OutputPort <ComparatorMechanism_Structure>` to the
  LearningMechanism's *ERROR_SIGNAL* `InputPort <LearningMechanism_Activation_Input>`.

In addition, a `LearningProjection` is created from the `LearningSignal<LearningMechanism_LearningSignal>` for the
`primary_learned_projection` to the `ParameterPort` for the `matrix <MappingProjection.matrix>` of the
`primary_learned_projection`.  Because this configuration involves only a single layer of learning, *no* Projection
is created or assigned to the LearningMechanism's *ERROR_SIGNAL* `OutputPort <LearningMechanism_Output_Error_Signal>`.

.. _LearningMechanism_Single_Layer_Learning_Figure.svg:

    **Components for Single Layer Learning**

    .. figure:: _static/LearningMechanism_Single_Layer_Learning_fig.svg
       :alt: Schematic of Mechanisms and Projections involved in learning for a single MappingProjection

       Components generated by a call to a `learning method <Composition_Learning_Methods>` of a Composition
       (e.g., ``add_backpropagaption_learning_pathway(pathway=[X, Y]``), labeled by type of Component (in **bold**,
       camelCase), NodeRole(s) assigned (*italics*), learning component type used as key in dictionary returned by
       the learning method (*UPPER_CASE* italics, outside of object) and, where relevant, the name of the attribute
       of the LearningMechanism with which it is associated (*italicized* lower case, outside of object). The
       ComparatorMechanism and LearningMechanism are shown with their InputPorts, OutputPorts and functions
       diagrammed.

.. _LearningMechanism_Multilayer_Learning:

Multilayer learning
^^^^^^^^^^^^^^^^^^^
This refers to learning in a pathway that has three or more Mechanisms in a sequence, and therefore two or more
MappingProjections that are being learned.  The learning components for such a configuration are created
automatically when a `learning method <Composition_Learning_Methods>` of a Composition is used that supports
multilayer learning (at present, this is only `add_backpropagation_learning_pathway`).  If the learning components
are `constructed explicitly <LearningMechanism_Explicit_Creation>` for a multilayer learning configuration,
then each LearningMechanism must use a `LearningFunction <LearningFunctions>` that can calculate the influence
that each MappingProjection and its output have on the error that the LearningMechanism receives from the previous
one(s) in the sequence (e.g., `BackPropagation`).  Furthermore, the construction of the other `learning-related
Components <Composition_Learning_Components>` associated with each LearningMechanism depend on the position of its
`primary_learned_projection` and `output_source <LearningMechanism.output_source>` in the sequence.  If it is the
last one in the sequence, it is treated in the same way the LearningMechanism in a `single layer learning configuration
configuration <LearningMechanism_Single_Layer_Learning>`.  This is the  case if the `output_source` is a standalone
Mechanism or the `TERMINAL` Mechanism of a Composition. In these cases, as for single layer learning, a
`ComparatorMechanism` is created that receives the output of the `output_source` as well as the target for learning
(see `TARGET Mechanisms <LearningMechanism_Targets>` below), and projects to a LearningMechanism that is created for
the `primary_learned_projection`.  For all other MappingProjections being learned in the
sequence, the following additional MappingProjections are created for learning (shown in the `figure
<LearningMechanism_Multilayer_Learning_Figure>` below):

* from the `input_source <LearningMechanism.input_source>` to the LearningMechanism's *ACTIVATION_INPUT* `InputPort
  <LearningMechanism_Activation_Input>`;
..
* from the `output_source <LearningMechanism.output_source>` to the LearningMechanism's *ACTIVATION_OUTPUT* `InputPort
  <LearningMechanism_Activation_Output>`;
..
* from the *ERROR_SIGNAL* `OutputPort <LearningMechanism_Output_Error_Signal>` of each of the LearningMechanism's
  `error_sources <LearningMechanisms.error_sources>` to each of its corresponding *ERROR_SIGNAL* `InputPort(s)
  <LearningMechanism_Input_Error_Signal>`;

* from the `covariates_source <LearningMechanism.covariates_source>` to the LearningMechanism's *COVARIATES*
  `InputPort`(s)  if the `function <Mechaism_Base.function>` of the `output_source <LearningMechanism.output_source>`
  takes more than one argument and they affect its derivative (see `above <LearningMechanism_Covariates>`).

In addition, a `LearningProjection` is created from the `LearningSignal <LearningMechanism_LearningSignal>` for the
`primary_learned_projection` of each LearningMechanism in the sequence, to the `ParameterPort` for the `matrix
<MappingProjection.matrix>` of the `primary_learned_projection`.  If the `primary_learned_projection` is the first in
the sequence, then *no* Projection is created or assigned to its LearningMechanism's *ERROR_SIGNAL* `OutputPort
<LearningMechanism_Output_Error_Signal>`.

.. _LearningMechanism_Multilayer_Learning_Figure:

    **Components for Multilayer Learning**

    .. figure:: _static/LearningMechanism_Multilayer_Learning_fig.svg
       :alt: Schematic of Mechanisms and Projections involved in learning for a sequence of MappingProjections

       Components generated by a call to a `learning method <Composition_Learning_Methods>` of a Composition for a
       sequence with three ProcessingMechanisms (and therefore two MappingProjections to be learned; e.g.,
       ``add_backpropagaption_learning_pathway(pathway=[X, Y, Z]``).  Components are labeled by: type of Component (in
       **bold**, camelCase); NodeRole(s) assigned (*italics*) to it; name of learning component type (used as key in
       dictionary returned by the learning method; *UPPER_CASE* italics, outside of object); and, where relevant,
       the name of the attribute of the LearningMechanism with which it is associated (*italicized* lower case,
       outside of object). The ComparatorMechanism and LearningMechanism are shown with their InputPorts, OutputPorts
       and functions diagrammed.

.. _LearningMechanism_Targets:

`TARGET` and `OBJECTIVE` Mechanisms**.  When `supervised learning <Composition_Learning_Supervised>` is
implemented using one of a Composition's `learning methods <Composition_Learning_Methods>`, it automatically creates a
number of  `learning-related Components <Composition_Learning_Components>`.  This includes a `TARGET_MECHANISM`, that
receives the target stimulus specifed in the **inputs** argument of the Composition's `learn <Composition.learn>`
method; and a `OBJECTIVE_MECHANISM`, that computes the error_signal for the sequence.  The output of the
`OBJECTIVE_MECHANISM` is the `error_source <LearningMechanism.error_sources>` for the last MappingProjection in the
learning pathway.  If a multilayer learning pathway is implemented explicitly, it must include these Components.

.. _LearningMechanism_Execution:

Execution
---------

LearningMechanisms that implement `unsupervised learning <Composition_Learning_Unsupervised>`
(`AutoassociativeLearningMechanisms <AutoAssociativeLearningMechanism>`) execute when the `RecurrentTransferMechanism`
with which they are associated executes.  LearningMechanisms that are part of a `supervised learning pathway
<Composition_Learning_Supervised>` execute after all of the ProcessingMechanisms in the `Composition` to which they
belong have executed.  When a LearningMechanism is executed, it uses the `value <InputPort.value>`(s) of its
*ERROR_SIGNAL* `InputPort(s) <LearningMechanism_Input_Error_Signal>` to calculate changes to the weights of the
`matrix <MappingProjection.MappingProjection.matrix>` parameter of its `primary_learned_projection
<LearningMechanism.primary_learned_projection>` and any of its other `learned_projections
<LearningMechanis.learned_projections>`. Those weight changes are assigned as the LearningMechanism's
`learning_signal <LearningMechanism.learning_signal>` attribute, the `value <LearningSignal.value>` of each of its
`LearningSignals <LearningMechanism_LearningSignal>`, and as the `value <LearningProjection.value>` of each of their
LearningProjections.  That value is used, in turn, to modify the `value <ParameterPort.value>` of the *MATRIX*
`ParameterPort` of each of the MappingProjections being learned (listed in the LearningMechanism's
`learned_projections <LearningMechanism.learned_projections>` attribute).

Each ParameterPort uses the value it receives from the `LearningProjection` that projects to it to modify the
parameter of its `function <ParameterPort.function>`, in a manner specified by the `modulation
<LearningSignal.modulation>` attribute of the `LearningSignal` from which it receives the LearningProjection (see
`Modulation <ModulatorySignal_Modulation>` for a description of how modulation operates). By default, the
`modulation <LearningSignal.modulation>` attribute of a LearningSignal is `ADDITIVE`, the `function
<ParameterPort.function>` of a *MATRIX* ParameterPort for a MappingProjection is `AccumulatorIntegrator`,
and the parameter it uses for additive modulation is its `increment <AccumulatorIntegrator.increment>` parameter.
These assignments cause the value of the LearningProjection to be added to the previous value of the *MATRIX*
ParameterPort, thus incrementing the weights by an amount specified by the LearningMechanism's `learning_signal
<LearningMechanism.learning_signal>`. Note, that the changes to the `matrix
<MappingProjection.MappingProjection.matrix>` parameter itself do not take effect until the next time the
`learned_projection` is executed (see `Lazy Evaluation <Component_Lazy_Updating>` for an explanation of
"lazy" updating).

A LearningMechanism's `function <LearningMechanism.function>` also computes an `error_signal
<LearningMechanism.error_signal>` that is assigned as the `value <OutputPort.value>` of its *ERROR_SIGNAL*
`OutputPort <LearningMechanism_Output_Error_Signal>`;  in a `multilayer learning configuration
<LearningMechanism_Multilayer_Learning>`, that value is provided to the *ERROR_SIGNAL* `InputPort(s)
<LearningMechanism_Input_Error_Signal>` of the LearningMechanism(s) for the preceding MappingProjection(s)
being learned in the sequence.

.. _LearningMechanism_Class_Reference:

Class Reference
---------------

"""

import warnings
from enum import Enum
from collections import defaultdict

import numpy as np
from beartype import beartype

from psyneulink._typing import Optional, Union, Literal, Type

from psyneulink.core.components.shellclasses import Port
from psyneulink.core.components.component import parameter_keywords
from psyneulink.core.components.functions.nonstateful.learningfunctions import BackPropagation
from psyneulink.core.components.mechanisms.mechanism import Mechanism_Base, MechanismError
from psyneulink.core.components.mechanisms.modulatory.modulatorymechanism import ModulatoryMechanism_Base
from psyneulink.core.components.mechanisms.processing.objectivemechanism import ObjectiveMechanism
from psyneulink.core.components.ports.inputport import InputPort
from psyneulink.core.components.ports.parameterport import ParameterPort
from psyneulink.core.components.ports.modulatorysignals.learningsignal import LearningSignal
from psyneulink.core.components.shellclasses import Mechanism
from psyneulink.core.globals.context import Context, ContextFlags, handle_external_context
from psyneulink.core.globals.keywords import \
    ADDITIVE, ASSERT, ENABLED, INPUT_PORTS, \
    LEARNING, LEARNING_MECHANISM, LEARNING_PROJECTION, LEARNING_SIGNAL, LEARNING_SIGNALS, MATRIX, \
    MODULATION, NAME, OUTPUT_PORT, OWNER_VALUE, PARAMS, PROJECTIONS, REFERENCE_VALUE, SAMPLE, PORT_TYPE, VARIABLE
from psyneulink.core.globals.parameters import FunctionParameter, Parameter, check_user_specified
from psyneulink.core.globals.preferences.basepreferenceset import ValidPrefSet
from psyneulink.core.globals.preferences.preferenceset import PreferenceLevel
from psyneulink.core.globals.utilities import ContentAddressableList, convert_all_elements_to_np_array, convert_to_np_array, is_numeric, ValidParamSpecType, \
    convert_to_list

__all__ = [
    'ACTIVATION_INPUT', 'ACTIVATION_INPUT_INDEX', 'ACTIVATION_OUTPUT', 'ACTIVATION_OUTPUT_INDEX',
    'COVARIATES', 'COVARIATES_SOURCES', 'DefaultTrainingMechanism',
    'ERROR_SIGNAL', 'ERROR_SIGNAL_INDEX', 'ERROR_SOURCES',
    'LearningMechanism', 'LearningMechanismError', 'input_port_names', 'output_port_names'
]


def _is_learning_spec(spec, include_matrix_spec=True):
    """Evaluate whether spec is a valid learning specification

    Return `True` if spec is LEARNING or a valid projection_spec (see Projection_Base._is_projection_spec)
    Otherwise, return `False`

    """
    # MODIFIED 11/28/17 OLD:
    from psyneulink.core.components.projections.projection import _is_projection_spec

    try:
        if spec in {LEARNING, ENABLED}:
            return True
        else:
            return _is_projection_spec(spec=spec,
                                       type=LEARNING_PROJECTION,
                                       include_matrix_spec=include_matrix_spec)
    except:
        return False


ValidLearningSpecType = Union[Literal['LEARNING', 'ENABLED'], Port, Type[Port]]


class LearningType(Enum):
    """
        Denotes whether LearningMechanism requires a target input.

    Attributes
    ----------

    UNSUPERVISED
        implements (and requires a Projection to) a *ERROR_SIGNAL* InputPort.

    SUPERVISED
        does not implement a *ERROR_SIGNAL* InputPort.

    """
    UNSUPERVISED = 0
    SUPERVISED = 1


class LearningTiming(Enum):
    """
        Denotes

    Attributes
    ----------

    EXECUTION_PHASE
        LearningMechanism (and associated `LearningProjections(s) <LearningProjection>`) executed during the
        `execution phase <Composition_Execution>` of the Composition to which they belong, usually immediately after
        execution of of the `Mechanism <Mechanism>` that receives the `primary_learned_projection`

    LEARNING_PHASE
        LearningMechanism (and associated `LearningProjections(s) <LearningProjection>`) executed during the
        `learning phase <Composition_Execution>` of the Composition to which they belong.

    """
    EXECUTION_PHASE = 0
    LEARNING_PHASE = 1


# Parameters:

parameter_keywords.update({LEARNING_PROJECTION, LEARNING})

LEARNING_TYPE = 'learning_type'
LEARNING_TIMING = 'learning_timing'

# Used to index variable:
ACTIVATION_INPUT_INDEX = 0
ACTIVATION_OUTPUT_INDEX = 1
ERROR_SIGNAL_INDEX = 2

# Used to name input_ports and output_ports:
ACTIVATION_INPUT = 'activation_input'     # InputPort
ACTIVATION_OUTPUT = 'activation_output'   # InputPort
ERROR_SIGNAL = 'error_signal'
COVARIATES = 'covariate'                  # Basename for one or more InputPorts
input_port_names = [ACTIVATION_INPUT, ACTIVATION_OUTPUT]
output_port_names = [LEARNING_SIGNAL, ERROR_SIGNAL]

ERROR_SOURCES = 'error_sources'
COVARIATES_SOURCES = 'covariates_sources'

DefaultTrainingMechanism = ObjectiveMechanism


class LearningMechanismError(MechanismError):
    pass


def _learning_signal_getter(owning_component=None, context=None):
    try:
        return owning_component.parameters.value._get(context)[0]
    except (TypeError, IndexError):
        return None

def _error_signal_getter(owning_component=None, context=None):
    try:
        return owning_component.parameters.value._get(context)[1]
    except (TypeError, IndexError):
        return None


class LearningMechanism(ModulatoryMechanism_Base):
    """
    LearningMechanism(                    \
        variable,                         \
        covariates_sources,               \
        error_sources,                    \
        function=BackPropagation,         \
        learning_rate=None,               \
        learning_signals=LEARNING_SIGNAL, \
        modulation=ADDITIVE,              \
        learning_enabled=True)

    Subclass of ModulatoryMechanism that modifies the `matrix <MappingProjection.matrix>` parameter of a
    `MappingProjection`. See `Mechanism <Mechanism_Class_Reference>` for additional arguments and attributes.

    COMMENT:
        Description:
            LearningMechanism is a subtype of the ModulatoryMechanism Type of the Mechanism Category of Component
            It implements a Mechanism that calculates changes to a Projection's parameters.
            Its function takes the output of an ObjectiveMechanism and generates a
            learning_signal (ndarray of parameter changes) to be used by the recipient of a LearningProjection
            that projects from the LearningMechanism to a MappingProjection.

        # DOCUMENT: ??NOT SURE WHETHER THIS IS STILL RELEVANT
        #    IF objective_mechanism IS None, IT IS LEFT UNSPECIFIED (FOR FURTHER IMPLEMENTATION BY COMPOSITION)
        #    THESE ARE HANDLED BY A MODULE METHOD _instantiate_objective_mechanism (AS PER OBJECTIVE MECHANISM):
        #        IF objective_mechanism IS SPECIFIED AS ObjectiveMechanism, AN OBJECTIVE MECHANISM IS CREATED FOR IT
        #        IF objective_mechanism IS SPECIFIED AS A MECHANISM OR OutputPort,
        #               a MappingProjection WITH AN IDENTITY MATRIX IS IMPLEMENTED FROM IT TO THE LearningMechanism

        Learning function:
            Generalized delta rule:
            dE/dW  =          learning_rate   *    dE/dA          *       dA/dW             *    I
            weight = weight + (learning_rate  * error_derivative  *  activation_derivative  *  input)
            for sumSquared error fct =        (target - output)
            for logistic activation fct =                           output * (1-output)
            where:
                output = activity of output (target) units (higher layer)
                input = activity of sending units (lower layer)
            Needs:
            - activation_derivative:  get from FUNCTION of sample_activation_mechanism/receiver_mech
                                      assumes derivative of Logistic unless otherwise specified
            - error_derivative:  get from FUNCTION of error_sources/next_level_mech;  but handled in ObjectiveMechanism
    COMMENT

    Arguments
    ---------

    variable : List or 2d np.array
        it must have three items that correspond to the three values required by the LearningMechanism's `function
        <LearningMechanism.function>`;  they must each be compatible (in number and type) with the `value
        <InputPort.value>` of the corresponding `InputPort <LearningMechanism_InputPorts>` (see `variable
        <LearningMechanism.variable>` for additional details).

    covariates_sources : InputPort or list of them : default None
        specifies the `InputPort`(s) of the LearningMechanism's `output_source <LearningMechanism.output_source>`
        other than the one to which the `primary_learned_projection` projects. (see `LearningMechanism_Covariates`
        for additional details).

    error_sources : ComparatorMechanism, LearningMechanism, OutputPort or list of them
        specifies the source(s) of the error signal(s) used by the LearningMechanism's `function
        <LearningMechanism.function>`.  Each must be a `ComparatorMechanism` for `single layer learning
        <LearningMechanism_Single_Layer_Learning>`, or for the last `MappingProjection` in a learning pathway in
        `multilayer learning <LearningMechanism_Multilayer_Learning>`;  otherwise they must be a `LearningMechanism`
        or the *ERROR_SIGNAL* OutputPort of one.

    function : LearningFunction or function : default BackPropagation
        specifies the function used to calculate the LearningMechanism's `learning_signal
        <LearningMechanism.learning_signal>` and `error_signal <LearningMechanism.error_signal>` attributes.  It's
        `variable <Function_Base.variable>` must have three items, each of which must be a list or 1d array of
        numeric values, corresponding to values provided by the LearningMechanism's *ACTIVATION_INPUT*,
        *ACTIVATION_OUTPUT*, and *ERROR_SOURCES* InputPorts, respectively (see `LearningMechanism_InputPorts
        `LearningMechanism_Function` and `LearningMechanism_InputPorts` for additional details). If supports an
        activation function that takes more than one argument that impact its derivative, then it must also accept a
        potential fourth keyword argument (``covariates``), that is a list or 1d array of numeric values provided by
        the LearningMechanism's *COVARIATES* InputPort(s) (see `LearningMechanism_Covariates` for additional details).

    learning_rate : float : default None
        specifies the learning rate for the LearningMechanism (see `learning_rate <LearningMechanism.learning_rate>`
        for details).

    learning_signals : List[parameter of Projection, ParameterPort, Projection, tuple[str, Projection] or dict] :
        default *LEARNING_SIGNAL* specifies the parameter(s) to be learned (see `learning_signals
        <LearningMechanism.learning_signals>` for details).

    modulation : str : default ADDITIVE
        specifies the default form of modulation used by the LearningMechanism's LearningSignals,
        unless they are `individually specified <LearningSignal_Specification>`.

    learning_enabled : bool or Enum[ONLINE|AFTER] : True
        specifies whether and when the LearningMechanism's `LearningProjections <LearningProjection>` are executed
        (see `learning_enabled <LearningMechanism.learning_enabled>` for additional details).


    Attributes
    ----------

    COMMENT:
        componentType : LEARNING_MECHANISM
    COMMENT

    variable : 2d np.array
        has three items that serve as the template for the three inputs required by the LearningMechanism's `function
        <LearningMechanism.function>` (corresponding to its three `InputPorts <LearningMechanism_InputPorts>`:
        the input to the `primary_learned_projection` (from `input_source`), the output of the Mechanism to which
        that projects (i.e., of `output_source`); and the error signal (from `LearningMechanism.error_sources`).

    input_ports : ContentAddressableList[OutputPort]
        list containing the LearningMechanism's three `InputPorts <LearningMechanism_InputPorts>`:
        *ACTIVATION_INPUT*, *ACTIVATION_OUTPUT*, *ERROR_SIGNAL*, and possible its *COVARIATES*.

    error_signal_input_ports : list[InputPorts]
        list of InputPorts that receive error_signals from the LearningMechanism's `error_sources
        <LearningMechanism.error_sources>`.

    input_source : ProcessingMechanism
        the Mechanism that sends the `primary_learned_projection`, and projects to the
        LearningMechanism's *ACTIVATION_INPUT* `InputPort <LearningMechanism_Activation_Input>`.

    output_source : ProcessingMechanism
        the Mechanism that receives the `primary_learned_projection`, and  projects to the
        LearningMechanism's *ACTIVATION_OUTPUT* `InputPort <LearningMechanism_Activation_Output>`.

    covariates_sources : List[InputPort]
        the `InputPort`(s) of the LearningMechanism's `output_source <LearningMechanism.output_source>`
        other than the one to which the `primary_learned_projection` projects;  these are used as
        covariates in the calculation of the derivative of `output_source <LearningMechanism.output_source>`\\'s
        `function <LearningMechanism.function>` with respect to `value <MappingProjection.value>` of the
        `primary_learned_projection` (see `LearningMechanism_Covariates` for additional details).

    covariates_values : List[1d np.array]
        the values of the InputPorts to which the `covariates_sources <LearningMechanism.covariates_sources>` project;
        passed to the LearningMechanism's `function <LearningMechanism.function>` as the *COVARIATES* item of its
        `variable <LearningMechanism.variable>`, and assigned as the `value <InputPort.value>` of the LearningMechanism's
        *COVARIATES* `InputPorts <LearningMechanism_Covariates>`.

    error_sources : list[ComparatorMechanism or LearningMechanism]
        the Mechanism(s) that calculate the error signal(s) provided to the
        LearningMechanism's *ERROR_SIGNAL(s)* `InputPort(s) <LearningMechanism_Input_Error_Signal>`.

    error_matrices : list[ParameterPort]
        the matrices of the Projections associated with the `error_sources <LearningMechanism.error_sources>`,
        (i.e., for the next Projection(s) in the learning_sequence, or to the `ComparatorMechanism`);
        note: these are *not* for the LearningMechanism's `learned_projections <LearningMechanism.learned_projections>`.

    primary_learned_projection : MappingProjection
        the Projection with the `matrix <MappingProjection.matrix>` parameter used to generate the
        LearningMechanism's `error_signal <LearningMechanism.error_signal>` and `learning_signal
        <LearningMechanism.learning_signal>` attributes.  It is always the first Projection listed in the
        LearningMechanism's `learned_projections <LearningMechanism.learned_projections>` attribute.

    learned_projections : List[MappingProjection]
        all of the MappingProjections modified by the LearningMechanism;  the first item in the list is always the
        `primary_learned_projection <LearningMechanism.primary_learned_projection>`.

    function : LearningFunction or function : default BackPropagation
        specifies the function used to calculate the `learning_signal <LearningMechanism.learning_signal>` (assigned
        to the LearningMechanism's `LearningSignal(s) <LearningMechanism_LearningSignal>`), and the `error_signal
        <LearningMechanism.error_signal>` (passed to the LearningMechanism for the preceding `MappingProjection` in a
        `multilayer learning pathway <LearningMechanism_Multilayer_Learning>`).  It takes the following
        three arguments, each of which must be a list or 1d array: **input**,  **output**, and **error** (see
        `LearningMechanism_Function` for additional details).

    learning_rate : float : None
        determines the learning rate for the LearningMechanism.  It is used to specify the :keyword:`learning_rate`
        parameter for the LearningMechanism's `learning function <LearningMechanism.function>`
        (see description of `learning_rate <LearningMechanism_Learning_Rate>` for additional details).

    error_signal : 1d np.array
        one of two values returned by the LearningMechanism's `function <LearningMechanism.function>`.  For
        `single layer learning <LearningMechanism_Single_Layer_Learning>`, this is the same as the value received in
        the LearningMechanism's *ERROR_SIGNAL* `InputPort <LearningMechanism_Input_Error_Signal>`;  for `multilayer
        learning <LearningMechanism_Multilayer_Learning>`, it is a modified version of the value received, that takes
        account of the contribution made by the learned_projection and its input to the error signal received. This
        is assigned as the `value <OutputPort.value>` of the LearningMechanism's *ERROR_SIGNAL* `OutputPort
        <LearningMechanism_Output_Error_Signal>`.

    learning_signal : number or ndarray
        one of two values returned by the LearningMechanism's `function <LearningMechanism.function>`, that specifies
        the changes to the weights of the `matrix <MappingProjection.matrix>` parameter for the LearningMechanism's
        `learned_projections <LearningMechanism.learned_projections>`;  it is calculated to reduce the error signal
        associated with the `primary_learned_projection <LearningMechanism.primary_learned_projection>` and received
        from the LearningMechanism's `error_sources`.  It is assigned as the value of the LearningMechanism's
        `LearningSignal(s) <LearningMechanism_LearningSignal>` and, in turn, its LearningProjection(s).

    learning_signals : ContentAddressableList[LearningSignal]
        list of all of the `LearningSignals <LearningSignal>` for the LearningMechanism, each of which sends one or
        more `LearningProjections <LearningProjection>` to the `ParameterPort(s) <ParameterPort>` for the `matrix
        <MappingProjection.matrix>` parameter of the `MappingProjection(s) <MappingProjection>` trained by the
        LearningMechanism.  The `value <LearningSignal>` of each LearningSignal is the LearningMechanism's
        `learning_signal <LearningMechanism.learning_signal>` attribute. Since LearningSignals are `OutputPorts
        <OutputPort>`, they are also listed in the LearningMechanism's `output_ports
        <LearningMechanism.output_ports>` attribute, after it *ERROR_SIGNAL* `OutputPort
        <LearningMechanism_Output_Error_Signal>`.

    learning_projections : List[LearningProjection]
        list of all of the LearningProjections <LearningProject>` from the LearningMechanism, listed in the order of
        the `LearningSignals <LearningSignal>` to which they belong (that is, in the order they are listed in
        the `learning_signals <LearningMechanism>` attribute).

    learning_enabled : bool or Enum[ONLINE|AFTER]
        determines whether and when the `learning_projections <LearningMechanism.learning_projections>` are executed.
        If set to False, they are never updated;  however, the LearningMechanism is still executed in any `Composition`
        to which it belongs, so that the error signals it calculates can be passed to any other LearningMechanism(s)
        to which it projects (see `LearningMechanism_Multilayer_Learning`).  If set to True or `ONLINE`,
        `learning_projections <LearningMechanism.learning_projections>` are updated when the LearningMechanism
        executes.  If set to `AFTER`, `learning_projections <LearningMechanism.learning_projections>` are updated at the
        end of each `TRIAL <TimeScale.TRIAL>` of execution of the Composition to which the LearningMechanism belongs.

        .. note::
           the `learning_enabled <LearningMechanism.learning_enabled>` attribute of a LearningMechanism determines the
           default behavior of its `learning_projections <LearningMechanism.learning_projections>`.  However, this
           can be overridden for individual `LearningProjections <LearningProjection>` by assigning their
           `learning_enabled <LearningProjection.learning_enabled>` attributes either at or after construction.

    output_ports : ContentAddressableList[OutputPort]
        list of the LearningMechanism's `OutputPorts <OutputPort>`, including its *ERROR_SIGNAL* `OutputPort
        <LearningMechanism_Output_Error_Signal>`, followed by its `LearningSignal(s)
        <LearningMechanism_LearningSignal>`, and then any additional (user-specified) `OutputPorts <OutputPort>`.

    COMMENT:
       #  FIX: THIS MAY NEED TO BE A 3d array (TO ACCOMDATE 2d array (MATRICES) AS ENTRIES)\
    COMMENT

    output_values : 2d np.array
        the first item is the `value <OutputPort.value>` of the LearningMechanism's *ERROR_SIGNAL* `OutputPort
        <LearningMechanism_Output_Error_Signal>`, followed by the `value <LearningSignal.value>` \\(s) of its
        `LearningSignal(s) <LearningMechanism_LearningSignal>`, and then those of any additional (user-specified)
        `OutputPorts <OutputPort>`.

    modulation : str
        the default form of modulation used by the LearningMechanism's `LearningSignal(s)
        <LearningMechanism_LearningSignal>`, unless they are `individually specified <LearningSignal_Specification>`.

    """

    componentType = LEARNING_MECHANISM
    className = componentType
    suffix = " " + className

    outputPortTypes = LearningSignal

    portListAttr = Mechanism_Base.portListAttr.copy()
    portListAttr.update({LearningSignal:LEARNING_SIGNALS})

    classPreferenceLevel = PreferenceLevel.TYPE

    class Parameters(ModulatoryMechanism_Base.Parameters):
        """
            Attributes
            ----------

                variable
                    see `variable <LearningMechanism.variable>`

                    :default value: numpy.array([[0],[0],[0]])
                    :type: numpy.ndarray

                covariates_sources
                    see `covariates_source <LearningMechanism.covariates_source>`

                    :default value: None
                    :type: list[InputPort]

                error_matrix
                    see `error_matrix <LearningMechanism.error_matrix>`

                    :default value: None
                    :type:

                error_signal
                    see `error_signal <LearningMechanism_Error_Signal>`

                    :default value: None
                    :type:
                    :read only: True

                error_sources
                    see `error_source <LearningMechanism_Error_Source>`

                    :default value: None
                    :type: list[Mechanism, OutputPort, tuple[str, Mechanism, OutputPort]]
                    :read only: True

                function
                    see `function <LearningMechanism_Function>`

                    :default value: `BackPropagation`
                    :type: `Function`

                input_ports
                    see `input_ports <LearningMechanism.input_ports>`

                    :default value: [`ACTIVATION_INPUT`, `ACTIVATION_OUTPUT`, `ERROR_SIGNAL`, `COVARIATES`]
                    :type: ``list``
                    :read only: True

                learning_enabled
                    see `learning_enabled <LearningMechanism.learning_enabled>`

                    :default value: True
                    :type: ``bool``

                learning_rate
                    see `learning_rate <LearningMechanism_Learning_Rate>`

                    :default value: None
                    :type:

                learning_signal
                    see `learning_signal <LearningMechanism_Learning_Signal>`

                    :default value: None
                    :type:
                    :read only: True

                learning_signals
                    see `learning_signals <LearningMechanism_Learning_Signals>`

                    :default value: ["{name: LearningSignal, variable: (OWNER_VALUE, 0)}"]
                    :type: ``list``
                    :read only: True

                modulation
                    see `modulation <LearningMechanism.modulation>`

                    :default value: `ADDITIVE_PARAM`
                    :type: ``str``

                output_ports
                    see `output_ports <LearningMechanism.output_ports>`

                    :default value: ["{name: error_signal, port_type: OutputPort, variable: (OWNER_VALUE, 1)}"]
                    :type: ``list``
                    :read only: True
        """
        variable = Parameter(np.array([[0],[0]]),
                             pnl_internal=True,
                             constructor_argument='default_variable')
        function = Parameter(BackPropagation, stateful=False, loggable=False)
        covariates_sources = Parameter(None, stateful=False, structural=True, read_only=True)
        error_sources = Parameter(None, stateful=False, structural=True, read_only=True)
        error_matrix = Parameter(None, modulable=True)
        learning_signal = Parameter(None, read_only=True, getter=_learning_signal_getter)
        error_signal = Parameter(None, read_only=True, getter=_error_signal_getter)
        learning_rate = FunctionParameter(None)
        learning_enabled = True
        modulation = ADDITIVE
        learning_type = LearningType.SUPERVISED

        input_ports = Parameter([ACTIVATION_INPUT, ACTIVATION_OUTPUT],
                                stateful=False,
                                loggable=False,
                                read_only=True,
                                structural=True,
                                parse_spec=True)
        output_ports = Parameter([{NAME: ERROR_SIGNAL,
                                   PORT_TYPE: OUTPUT_PORT,
                                   VARIABLE: (OWNER_VALUE, 1)}],
                                 stateful=False,
                                 loggable=False,
                                 read_only=True,
                                 structural=True)
        learning_signals = Parameter([{NAME: LEARNING_SIGNAL,
                                       VARIABLE: (OWNER_VALUE, 0)}],
                                     stateful=False,
                                     loggable=False,
                                     read_only=True,
                                     structural=True)
        def _parse_covariates_sources(self, covariates_sources):
            if covariates_sources:
                return convert_to_list(covariates_sources)

        def _parse_error_sources(self, error_sources):
            if error_sources:
                return convert_to_list(error_sources)

    @check_user_specified
    @beartype
    def __init__(self,
                 # default_variable:Union[list, np.ndarray],
                 default_variable=None,
                 input_shapes=None,
                 covariates_sources: Optional[Union[InputPort, list]] = None,
                 error_sources: Optional[Union[Mechanism, list]] = None,
                 function=None,
                 learning_signals: Optional[list] = None,
                 output_ports=None,
                 modulation: Optional[str] = None,
                 learning_rate: Optional[ValidParamSpecType] = None,
                 learning_enabled: Optional[Union[bool, Literal['online', 'after']]] = None,
                 in_composition=False,
                 params=None,
                 name=None,
                 prefs: Optional[ValidPrefSet] = None,
                 **kwargs
                 ):

        self.in_composition = in_composition

        # # USE FOR IMPLEMENTATION OF deferred_init()
        # # Store args for deferred initialization
        # self._init_args = locals().copy()
        # self._init_args['context'] = self
        # self._init_args['name'] = name
        # delete self._init_args[ERROR_SOURCES]

        # # Flag for deferred initialization
        # self.initialization_status = ContextFlags.DEFERRED_INIT

        super().__init__(
            default_variable=default_variable,
            input_shapes=input_shapes,
            covariates_sources=covariates_sources,
            error_sources=error_sources,
            function=function,
            output_ports=output_ports,
            modulation=modulation,
            learning_rate=learning_rate,
            learning_enabled=learning_enabled,
            learning_signals=learning_signals,
            params=params,
            name=name,
            prefs=prefs,
            **kwargs
        )
        # If default to value assigned to its function, so any later assignments can be detected
        #    (e.g., in CompositionRunner.run_learning)
        self.defaults.learning_rate = self.parameters.learning_rate.get()

    def _check_type_and_timing(self):
        try:
            self.learning_type
        except:
            raise LearningMechanismError("{} subclass of {} must implement {} attribute".
                                         format(self.__class__.__name__, LearningMechanism.__name__,
                                                repr(LEARNING_TYPE)))
        try:
            self.learning_timing
        except:
            raise LearningMechanismError("{} subclass of {} must implement {} attribute".
                                         format(self.__class__.__name__, LearningMechanism.__name__,
                                                repr(LEARNING_TIMING)))

    def _parse_function_variable(self, variable, context=None):
        # # MODIFIED 9/11/23 OLD:
        # Return values of ACTIVATION_INPUT_INDEX, ACTIVATION_OUTPUT_INDEX, and first ERROR_SIGNAL_INDEX InputPorts
        #   in variable; remaining inputs (additional error signals and/or COVARITES) are passed in kwargs)
        return variable[range(min(len(self.input_ports),3))]
        # MODIFIED 9/11/23 NEW:
        # Return values of ACTIVATION_INPUT, ACTIVATION_OUTPUT, and ERROR_SIGNAL InputPorts in variable;
        #   remaining inputs (additional error signals and/or COVARITES) are passed in kwargs)
        # return np.array(self.input_values, dtype=object)
        # FIX: SHOULD EXTRA ERROR_SIGNAL (AND ERROR_MATRIX) BE PUT IN params? CF WHAT HAPPENS WITH RUMELHART NETWORK
        # return variable[range(2 + len(self.error_signal_input_ports))]
        # MODIFIED 9/11/23 END

    def _validate_variable(self, variable, context=None):
        """Validate that variable has exactly three items: activation_input, activation_output and error_signal
        """

        variable = super()._validate_variable(variable, context)

        # if len(variable) < 3:
        num_input_ports = len(self.input_ports)
        error_signals_msg = f", and 'ERROR_SIGNAL(s)'" if num_input_ports > 2 else ""
        if len(variable) < num_input_ports:
            raise LearningMechanismError(f"Variable for {self.name} ({variable}) must have at least {num_input_ports} "
                                         f"items: 'ACTIVATION_INPUT', 'ACTIVATION_INPUT'{error_signals_msg}")

        # Validate that activation_input, activation_output are numeric and lists or 1d arrays and that
        # there is the correct number of items beyond those for the number of error_sources and covariates_sources
        for i in range(len(variable)):
            item_num_string = "Item {i+1}"

            # Ensure that activation_input and activation_output are numeric and lists or 1d np.ndarrays
            # (harder to do this for error_sources and covariates_sources, since they may have different shapes
            if i < 3:
                if not np.array(variable[i]).ndim == 1:
                    raise LearningMechanismError(f"{item_num_string} of variable for '{self.name}' "
                                                 f"({variable[i]}) is not a list of numeric values or 1d np.array.")
                if not (is_numeric(variable[i])):
                    raise LearningMechanismError(f"{item_num_string} of variable for '{self.name}' "
                                                 f"({variable[i]}) is not numeric.")

            # Any input_ports for error_sources > 1 (the default) and/or ones for covariates haven't yet been assigned
            #    so length of variable should be short that number
            elif self.is_initializing:
                num_error_sources = len(self.error_sources) # 1 is already in input_ports
                num_covariates_sources = len(self.covariates_sources) if self.covariates_sources else 0
                if len(variable) != len(self.input_ports) + num_error_sources + num_covariates_sources:
                    assert False, f"Number of items ({len(variable)}) in variable for '{self.name}' doesn't match " \
                                  f"the number of expected InputPorts ({len(self.input_ports)}) during initialization."

            # All input_ports have been assigned, so number of items in variable should match number of input_ports
            elif len(variable) != len(self.input_ports):
                # All input_ports have been assigned, so number of items in variable should match number of input_ports
                assert False, f"Number of items ({len(variable)}) in variable for '{self.name}' doesn't match " \
                              f"the number of its assigned InputPorts ({len(self.input_ports)})"

        return variable

    def _validate_params(self, request_set, target_set=None, context=None):
        """Validate error_sources

        `error_sources` argument must be an `ObjectiveMechanism`, another `LearningMechanism`, an *ERROR_SIGNAL*
        OutputPort of a LearningMechanism, or a list of these, and there must be the same number as there are
        ERROR_SIGNAL InputPorts.

        """

        super()._validate_params(request_set=request_set, target_set=target_set,context=context)

        from psyneulink.core.components.ports.port import _parse_port_spec
        from psyneulink.core.components.ports.outputport import OutputPort
        from psyneulink.core.components.ports.modulatorysignals.learningsignal import LearningSignal
        from psyneulink.core.components.projections.pathway.mappingprojection import MappingProjection
        from psyneulink.core.components.projections.projection import _validate_receiver

        if COVARIATES_SOURCES in target_set and target_set[COVARIATES_SOURCES]:
            if not all(isinstance(item, InputPort) for item in self.covariates_sources):
                raise LearningMechanismError(f"Each item in {repr(COVARIATES_SOURCES)} arg for '{self.name}' "
                                             f"must be an {InputPort.__name__}.")

        if ERROR_SOURCES in target_set and target_set[ERROR_SOURCES]:
            error_sources = self.parameters.error_sources.get()
            num_error_sources = len(error_sources)
            if not num_error_sources == \
                   len(self.defaults.variable[ERROR_SIGNAL_INDEX:ERROR_SIGNAL_INDEX + num_error_sources]):
                raise LearningMechanismError(f"Number of items specified in {repr(ERROR_SOURCES)} arg "
                                             f"for '{self.name}' ({len(error_sources)}) must equal the number "
                                             f"of its {InputPort.__name__} {ERROR_SIGNAL.upper()}(S) "
                                             f"({len(self.error_signal_input_ports)}).")

            for error_source in error_sources:
                if (not isinstance(error_source, (ObjectiveMechanism, LearningMechanism, OutputPort))
                        or (isinstance(error_source, OutputPort)
                            and error_source not in error_source.owner.output_ports[ERROR_SIGNAL])):
                    raise LearningMechanismError(f"{repr(ERROR_SOURCES)} arg for {self.name} ({error_source}) "
                                                 f"must be an {ObjectiveMechanism.__name__}, "
                                                 f"another {LearningMechanism.__name__}, an {repr(ERROR_SIGNAL)} "
                                                 f"{OutputPort.__name__} of one, or list of any of these.")

        if LEARNING_SIGNALS in target_set and target_set[LEARNING_SIGNALS]:

            if not isinstance(target_set[LEARNING_SIGNALS], list):
                raise LearningMechanismError("{} arg of {} must be list".
                                           format(LEARNING_SIGNAL, self.name))

            for spec in target_set[LEARNING_SIGNALS]:
                learning_signal = _parse_port_spec(port_type=LearningSignal, owner=self, port_spec=spec,
                                                   context=Context(string='LearningMechanism.validate_params'))

                # Validate that the receiver of the LearningProjection (if specified)
                #     is a MappingProjection and in the same Composition as self (if specified)
                if learning_signal[PARAMS] and PROJECTIONS in learning_signal[PARAMS]:
                    for learning_projection in learning_signal[PARAMS][PROJECTIONS]:
                        _validate_receiver(sender_mech=self,
                                           projection=learning_projection,
                                           expected_owner_type=MappingProjection,
                                           spec_type=LEARNING_SIGNAL,
                                           context=context)
                else:
                    pass


    def _instantiate_input_ports(self, input_ports=None, reference_value=None, context=None):
        """Instantiate COVARIATES InputPorts if there are any covariate_sources"""

        input_ports = self.input_ports

        # if self.error_sources:
        #     input_ports += [ERROR_SIGNAL] * (len(self.error_sources) - 1)  # 1 ERROR_SIGNAL InputPort is assigned by default
        # num_error_sources = len(self.error_sources) if self.error_sources else 0
        num_error_sources = len(self.error_sources) if self.error_sources else 0
        input_ports += [ERROR_SIGNAL] * num_error_sources

        if self.covariates_sources:
            num_covariates = len(self.covariates_sources)
            if num_covariates == 1:
                input_ports += [COVARIATES]
            else:
                for i in range(num_covariates):
                    input_ports += [f'{COVARIATES}_{i}']
                    # input_ports[i].reference_value = reference_value[i]

        super()._instantiate_input_ports(input_ports=input_ports, reference_value=reference_value, context=context)

        self.error_signal_input_ports = self.input_ports[ERROR_SIGNAL_INDEX:ERROR_SIGNAL_INDEX + num_error_sources]
        self.covariates_input_ports = self.input_ports[ERROR_SIGNAL_INDEX + num_error_sources:]


    def _instantiate_attributes_before_function(self, function=None, context=None):
        """Instantiates MappingProjection(s) from error_sources (if specified) to LearningMechanism

        Also determines and assigns `error_matrices` from the `error_sources`, identified as the matrix for the
            Projection with which each error_source is associated.
            :param function:
        """
        # error_sources = self.parameters.error_sources.get(context)
        # if error_sources:
        #     self.parameters.input_ports._set(self.input_ports[:2] + [ERROR_SIGNAL] * len(error_sources),context)

        super()._instantiate_attributes_before_function(function=function, context=context)

        self.error_matrices = None
        # error_sources = self.parameters.error_sources.get()
        if self.error_sources:
            self.error_matrices = [None] * len(self.error_sources)
            for i, error_source in enumerate(self.error_sources):
                if not self.in_composition:
                    # IMPLEMENTATION NOTE:
                    #    _create_terminal_backprop_sequence_components and _create_multilayer_backprop_components
                    #    in Composition take care of creating projections from _error_sources to LearningMechanisms
                    warnings.warn("Instantiation of a LearningMechanism outside of a Composition is tricky!")
                if isinstance(error_source, ObjectiveMechanism):
                    self.error_matrices[i] = np.identity(len(error_source.input_ports[SAMPLE].value))
                else:
                    # IMPLEMENTATION NOTE:
                    #     This assumes that error_source has only one LearningSignal or,
                    #     if it has more, that they are all equivalent
                    self.error_matrices[i] = error_source.primary_learned_projection.parameter_ports[MATRIX]

    def _instantiate_output_ports(self, context=None):

        from psyneulink.core.globals.registry import register_category
        from psyneulink.core.components.ports.modulatorysignals.learningsignal import LearningSignal
        from psyneulink.core.components.ports.port import Port_Base, _instantiate_port

        # Create registry for LearningSignals (to manage names)
        register_category(entry=LearningSignal,
                          base_class=Port_Base,
                          registry=self._portRegistry,
                          )

        # Instantiate LearningSignals if they are specified, and assign to self.output_ports
        # Notes:
        #    - if any LearningSignals are specified they will replace the default LEARNING_SIGNAL OutputPort
        #    - the LearningSignals are appended to _output_ports, leaving ERROR_SIGNAL as the first entry.

        # Instantiate LearningSignals and assign to self.output_ports
        for learning_signal in self.learning_signals:
            # If it is already a dict, use that (e.g., in case varialbe or modulation is specified by user)
            variable = (OWNER_VALUE, 0)
            modulation = self.defaults.modulation
            reference_value = self.parameters.learning_signal._get(context)
            if isinstance(learning_signal, dict):
                variable = defaultdict(dict,learning_signal)[VARIABLE] or variable
                modulation = defaultdict(dict,learning_signal)[MODULATION] or modulation
                reference_value = defaultdict(dict,learning_signal)[REFERENCE_VALUE] \
                    if REFERENCE_VALUE in learning_signal else reference_value
            # Parse learning_signal specifications (in call to Port._parse_port_spec)
            #    and any embedded Projection specifications (in call to <Port>._instantiate_projections)
            learning_signal = _instantiate_port(port_type=LearningSignal,
                                                 owner=self,
                                                 variable=variable,
                                                 reference_value=reference_value,
                                                 modulation=modulation,
                                                 # port_spec=self.learning_signal)
                                                 port_spec=learning_signal,
                                                 context=context)
            # Add LearningSignal to output_ports list
            self.output_ports.append(learning_signal)

        # # Assign LEARNING_SIGNAL as the name of the 1st LearningSignal; the names of any others can be user-defined
        # first_learning_signal = next(port for port in self.output_ports if isinstance(port, LearningSignal))
        # first_learning_signal.name = LEARNING_SIGNAL

        super()._instantiate_output_ports(context=context)

        # Reassign learning_signals to capture any user_defined LearningSignals instantiated in call to super
        #   and assign them to a ContentAddressableList
        self.parameters.learning_signals._set(
            ContentAddressableList(
                component_type=LearningSignal,
                list=[port for port in self.output_ports if isinstance(port, LearningSignal)]
            ),
            context
        )

        # # Initialize _error_signals;  this is assigned for efficiency (rather than just using the property)
        # #    since it is used by the execute method
        # self._error_signal_input_ports = self.error_signal_input_ports

    @handle_external_context()
    def add_ports(self, error_sources, context=None):
        """Add error_source and error_matrix for each InputPort added"""

        ports = super().add_ports(ports=error_sources, update_variable=False, context=context)
        instantiated_input_ports = []
        for input_port in ports[INPUT_PORTS]:
            error_source = input_port.path_afferents[0].sender.owner
            self.error_matrices.append(error_source.primary_learned_projection.parameter_ports[MATRIX])
            if ERROR_SIGNAL in input_port.name:
                self.error_signal_input_ports.append(input_port)
            instantiated_input_ports.append(input_port)

        assert True

        # TODO: enable this. fails because LearningMechanism does not have a
        # consistent _parse_function_variable
        # self._update_default_variable(np.asarray(self.input_values, dtype=int), context)

        return instantiated_input_ports

    # FIX 7/28/19 [JDC]:  REMOVE THIS ONCE error_input_ports HAS SETTER OR IS OTHERWISE REFACTORED
    def remove_ports(self, ports):
        """Keep error_signal_input_ports and error_matrices in sych with error_signals in input_ports"""
        ports = convert_to_list(ports)
        for i, port in enumerate([s for s in ports if s in self.error_signal_input_ports]):
            del self.error_matrices[i]
        super().remove_ports(ports=ports)
        self.error_signal_input_ports = [s for s in self.input_ports if ERROR_SIGNAL in s.name]

    def _execute(
            self,
            variable=None,
            context=None,
            runtime_params=None)->list:
        """Execute LearningMechanism function and return learning_signal

        Identify error_signals received from LearningMechanisms currently being executed
        Assign them, and the corresponding error_matrices to a pair of arrays
        Execute function for each error_signal, error_matrix pair
        Sum the learning_signal and error_signal values received from each execution

        Returns
        -------

        List[ndarray, ndarray] : summed learning_signal, summed error_signal
        """

        # Get error_signals (from ERROR_SIGNAL InputPorts) and error_matrices relevant for the current execution:
        error_signal_indices = [self.input_ports.index(s) for s in self.error_signal_input_ports]
        error_signal_inputs = variable[error_signal_indices]
        if self.error_matrices is None:
            error_matrices = self._init_error_matrices(error_signal_inputs, error_signal_indices)
        else:
            error_matrices = np.array(self.error_matrices)[range(len(self.error_signal_input_ports))]

        for i, matrix in enumerate(error_matrices):
            if isinstance(error_matrices[i], ParameterPort):
                error_matrices[i] = error_matrices[i].parameters.value._get(context)

        summed_learning_signal = 0
        summed_error_signal = 0

        # Compute learning_signal for each error_signal (and corresponding error-Matrix):
        for error_signal_input, error_matrix in zip(error_signal_inputs, error_matrices):
            function_variable = convert_to_np_array([variable[ACTIVATION_INPUT_INDEX],
                                                     variable[ACTIVATION_OUTPUT_INDEX],
                                                     error_signal_input])
            # Get covariates_values and pass in covariates arg
            if self.covariates_sources:
                covariates_values = [source.value for source in self.covariates_sources]
            else:
                covariates_values = None
            learning_signal, error_signal = super()._execute(variable=function_variable,
                                                             context=context,
                                                             error_matrix=error_matrix,
                                                             covariates=covariates_values,
                                                             runtime_params=runtime_params)
            # Assign covariates values to their input_ports (not needed for computation; make available for reference)
            for i, covariate_input_port in enumerate(self.covariates_input_ports):
                covariate_input_port.parameters.value._set(covariates_values[i], context, override=True)

            # Sum learning_signals and error_signals
            summed_learning_signal += learning_signal
            summed_error_signal += error_signal

        from psyneulink.core.compositions.report import ReportOutput
        if (self.reportOutputPref is not ReportOutput.OFF
                and self.initialization_status != ContextFlags.INITIALIZING):
            print("\n{} weight change matrix: \n{}\n".format(self.name, summed_learning_signal))

        # During initialization return zeros so that the first "real" trial for Backprop does not start
        # with the error computed during initialization
        if (self.in_composition and
                isinstance(self.function, BackPropagation) and
                self.initialization_status == ContextFlags.INITIALIZING):
            return convert_all_elements_to_np_array(
                [np.zeros(summed_learning_signal.shape), np.zeros(summed_error_signal.shape)]
            )

        return convert_all_elements_to_np_array([summed_learning_signal, summed_error_signal])

    @property
    def input_source(self):
        try:
            return self.input_ports[ACTIVATION_INPUT].path_afferents[0].sender.owner
        except IndexError:
            return None

    @property
    def output_source(self):
        try:
            return self.input_ports[ACTIVATION_OUTPUT].path_afferents[0].sender.owner
        except IndexError:
            return None

    @property
    def primary_learned_projection(self):
        return self.learned_projections[0]

    @property
    def learned_projections(self):
        return [lp.receiver.owner for ls in self.learning_signals for lp in ls.efferents]

    @property
    def validate_error_signal_and_covariate_sources(self):
        # FIX: 8/1/23 - NEEDS ERROR MESSAGES, AND TO BE CALLED SOMEWHERE
        #               (MAYBE BY COMPOSITION ONCE ALL NODES AND PROJECTIONS HAVE BEEN INSTANTIATED)
        assert set(input_port.path_afferents[0].sender.owner
                   for input_port in self.error_signal_input_ports) == set(self.error_sources)
        assert set(input_port.path_afferents[0].sender.owner
                   for input_port in self.covariates_input_ports) == set(self.covariates_sources)

    def _init_error_matrices(self, error_signal_inputs, error_signal_indices):
        # KAM 6/28/19 Hack to get the correct shape and contents for initial error matrix in backprop
        if self.function is BackPropagation or isinstance(self.function, BackPropagation):
            mat = []
            for i in range(len(error_signal_inputs[0])):
                row = []
                for j in range(len(error_signal_inputs[0])):
                    if i == j:
                        row.append(1.)
                    else:
                        row.append(0.)
                mat.append(row)
            self.error_matrices = mat
            error_matrices = mat
        else:
            self.error_matrices = [[0.]]
            error_matrices = \
                np.array(self.error_matrices)[np.array([c - ERROR_SIGNAL_INDEX for c in error_signal_indices])]
        return error_matrices
