{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Stock Price Prediction using Echo State Networks\n",
    "\n",
    "## Introduction\n",
    "\n",
    "In this notebook, the impact of different hyper-parameters of an ESN are explained. The notebook depends on just a small variety of packages: numpy, matplotlib, IPython and pyrcn."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "\n",
    "from matplotlib import pyplot as plt\n",
    "plt.rcParams['image.cmap'] = 'jet'\n",
    "plt.rcParams['pdf.fonttype'] = 42\n",
    "plt.rcParams['ps.fonttype'] = 42\n",
    "%matplotlib inline\n",
    "\n",
    "from IPython.display import set_matplotlib_formats\n",
    "set_matplotlib_formats('png', 'pdf')\n",
    "\n",
    "from pyrcn.echo_state_network import ESNRegressor"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In this notebook, we feed an example impulse through the ESN, treating it as a non-linear filter. The unit impuls starts at n=5."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [
    {
     "data": {
      "application/pdf": "[encoded data, 510564 characters]\n",
      "image/png": "[encoded data, 53544 characters]\n",
      "text/plain": [
       "<Figure size 2850x660 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "X = np.zeros(shape=(100, 1), dtype=int)\n",
    "X[5] = 1\n",
    "y = X[:, 0]\n",
    "\n",
    "_, ax = plt.subplots(figsize=(4.75, 1.1), dpi=600)\n",
    "ax.plot(X)\n",
    "ax.set_xlim([0, 100])\n",
    "ax.set_ylim([0, 1])\n",
    "ax.set_xlabel('n')\n",
    "ax.set_ylabel('X[n]')\n",
    "plt.grid()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "pycharm": {
     "name": "#%% md\n"
    }
   },
   "source": [
    "At first, show the impact of different input scaling factors.\n",
    "\n",
    "Therefore, we neutralize the other hyper-parameters, i.e., no recurrent connections ($\\rho = 0$), no bias ($\\alpha_{\\mathrm{b}} = 0$) and no leakage ($\\lambda = 1$). "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [],
   "source": [
    "esn = ESNRegressor(k_in=1, input_scaling=0.1, spectral_radius=0.0, bias=0.0, ext_bias=False, leakage=1.0,\n",
    "                   reservoir_size=50, k_res=10, reservoir_activation='tanh', bi_directional=False,\n",
    "                   teacher_scaling=1.0, teacher_shift=0.0, solver='ridge', beta=1e-6, random_state=10)\n",
    "\n",
    "esn.fit(X=X, y=y, n_jobs=0)\n",
    "_ = esn.predict(X=X, keep_reservoir_state=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Visualizing this, we can see exactly what we might expect. We have chosen an input scaling factor of 0.1. Thus, the reservoir state is non-zero for exactly one sample. We can see that all reservoir states are zero all the times except for $n=5$, when the impulse is fed into the ESN. \n",
    "\n",
    "The absolute values of the reservoir states lie between 0 and 0.1."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [
    {
     "data": {
      "application/pdf": "[encoded data, 514200 characters]\n",
      "image/png": "[encoded data, 35048 characters]\n",
      "text/plain": [
       "<Figure size 2850x660 with 2 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "_, ax = plt.subplots(figsize=(4.75, 1.1), dpi=600)\n",
    "im = ax.imshow(np.abs(esn.reservoir_state[:50, 1:].T),vmin=0, vmax=1)\n",
    "ax.set_xlim([0, esn.reservoir_state[:50, 1:].shape[0]])\n",
    "ax.set_ylim([0, esn.reservoir_state.shape[1] - 1])\n",
    "ax.set_xlabel('n')\n",
    "ax.set_ylabel('R[n]')\n",
    "plt.colorbar(im, ax=ax)\n",
    "plt.grid()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "What happens if we increase the input scaling factor?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [],
   "source": [
    "esn = ESNRegressor(k_in=1, input_scaling=1.0, spectral_radius=0.0, bias=0.0, ext_bias=False, leakage=1.0,\n",
    "                   reservoir_size=50, k_res=10, reservoir_activation='tanh', bi_directional=False,\n",
    "                   teacher_scaling=1.0, teacher_shift=0.0, solver='ridge', beta=1e-6, random_state=10)\n",
    "\n",
    "esn.fit(X=X, y=y, n_jobs=0)\n",
    "_ = esn.predict(X=X, keep_reservoir_state=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Each reservoir state still has only one non-zero value at $n=5$ as before, just with higher activations up to 0.8. The $\\tanh$ non-linearity is damping the reservoir states so that they cannot reach 1.  "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [
    {
     "data": {
      "application/pdf": "[encoded data, 519928 characters]\n",
      "image/png": "[encoded data, 82688 characters]\n",
      "text/plain": [
       "<Figure size 5700x1320 with 2 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "_, ax = plt.subplots(figsize=(4.75, 1.1), dpi=1200)\n",
    "im = ax.imshow(np.abs(esn.reservoir_state[:50, 1:].T),vmin=0, vmax=1)\n",
    "ax.set_xlim([0, esn.reservoir_state[:50, 1:].shape[0]])\n",
    "ax.set_ylim([0, esn.reservoir_state.shape[1] - 1])\n",
    "ax.set_xlabel('n')\n",
    "ax.set_ylabel('R[n]')\n",
    "plt.colorbar(im, ax=ax)\n",
    "plt.grid()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's keep the input scaling factor fixed to 1.0 for the next experiments. Echo State Networks have two ways to\n",
    "incorporate past information.\n",
    "\n",
    "Next, let us analyze the spectral radius. Therefore, we set it to 0.3 in the following example."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [],
   "source": [
    "esn = ESNRegressor(k_in=1, input_scaling=1.0, spectral_radius=0.3, bias=0.0, ext_bias=False, leakage=1.0,\n",
    "                   reservoir_size=50, k_res=10, reservoir_activation='tanh', bi_directional=False,\n",
    "                   teacher_scaling=1.0, teacher_shift=0.0, solver='ridge', beta=1e-6, random_state=10)\n",
    "\n",
    "esn.fit(X=X, y=y, n_jobs=0)\n",
    "_ = esn.predict(X=X, keep_reservoir_state=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can observe that the impulse responses are starting at n=5 and decaying until reaching zero after a short time. Obviously, the reservoir states are decaying rather fast, because the recurrent connections are small compared to the input scaling. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [
    {
     "data": {
      "application/pdf": "[encoded data, 524456 characters]\n",
      "image/png": "[encoded data, 96020 characters]\n",
      "text/plain": [
       "<Figure size 5700x1320 with 2 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "_, ax = plt.subplots(figsize=(4.75, 1.1), dpi=1200)\n",
    "im = ax.imshow(np.abs(esn.reservoir_state[:, 1:].T),vmin=0, vmax=1)\n",
    "ax.set_xlim([0, esn.reservoir_state.shape[0]])\n",
    "ax.set_ylim([0, esn.reservoir_state.shape[1] - 1])\n",
    "ax.set_xlabel('n')\n",
    "ax.set_ylabel('R[n]')\n",
    "plt.colorbar(im, ax=ax)\n",
    "plt.grid()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "It is interesting to increase the spectral radius. Therefore, we set it to 0.9."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [],
   "source": [
    "esn = ESNRegressor(k_in=1, input_scaling=1.0, spectral_radius=0.9, bias=0.0, ext_bias=False, leakage=1.0,\n",
    "                   reservoir_size=50, k_res=10, reservoir_activation='tanh', bi_directional=False,\n",
    "                   teacher_scaling=1.0, teacher_shift=0.0, solver='ridge', beta=1e-6, random_state=10)\n",
    "\n",
    "esn.fit(X=X, y=y, n_jobs=0)\n",
    "_ = esn.predict(X=X, keep_reservoir_state=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The values are still bounded between [0.8, 0.8], but the reservoir states are active over a longer time now ($n\\approx 30$)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [
    {
     "data": {
      "application/pdf": "[encoded data, 528864 characters]\n",
      "image/png": "[encoded data, 101260 characters]\n",
      "text/plain": [
       "<Figure size 5700x1320 with 2 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "_, ax = plt.subplots(figsize=(4.75, 1.1), dpi=1200)\n",
    "im = ax.imshow(np.abs(esn.reservoir_state[:, 1:].T),vmin=0, vmax=1)\n",
    "ax.set_xlim([0, esn.reservoir_state.shape[0]])\n",
    "ax.set_ylim([0, esn.reservoir_state.shape[1] - 1])\n",
    "ax.set_xlabel('n')\n",
    "ax.set_ylabel('R[n]')\n",
    "plt.colorbar(im, ax=ax)\n",
    "plt.grid()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can try to further increase the spectral radius. Therefore, we set it to 1.0."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [],
   "source": [
    "esn = ESNRegressor(k_in=1, input_scaling=1.0, spectral_radius=1.0, bias=0.0, ext_bias=False, leakage=1.0,\n",
    "                   reservoir_size=50, k_res=10, reservoir_activation='tanh', bi_directional=False,\n",
    "                   teacher_scaling=1.0, teacher_shift=0.0, solver='ridge', beta=1e-6, random_state=10)\n",
    "\n",
    "esn.fit(X=X, y=y, n_jobs=0)\n",
    "_ = esn.predict(X=X, keep_reservoir_state=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "As we expected, the values are getting slightly larger, but are still bounded because of the tanh()-non-linearities.\n",
    "Interestingly, the reservoir states are not completely decaying, although the spectral radius was just increased by 0.1.\n",
    "\n",
    "We can see that the reservoir states are decaying very slowly, and they are oscillating with a resonance frequency. For many tasks, it is indeed necessary to preserve the echo state property of reservoir and keep $\\rho < 1$. However in some cases, such as time-series prediction in this paper, the spectral radius can be larger than 1. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [
    {
     "data": {
      "application/pdf": "[encoded data, 536668 characters]\n",
      "image/png": "[encoded data, 112272 characters]\n",
      "text/plain": [
       "<Figure size 5700x1320 with 2 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "_, ax = plt.subplots(figsize=(4.75, 1.1), dpi=1200)\n",
    "im = ax.imshow(np.abs(esn.reservoir_state[:, 1:].T),vmin=0, vmax=1)\n",
    "ax.set_xlim([0, esn.reservoir_state.shape[0]])\n",
    "ax.set_ylim([0, esn.reservoir_state.shape[1] - 1])\n",
    "ax.set_xlabel('n')\n",
    "ax.set_ylabel('R[n]')\n",
    "plt.colorbar(im, ax=ax)\n",
    "plt.grid()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "For some use cases, the reservoir should behave more non-linear. Therefore, we can play around with the bias. Here, we\n",
    "will show the impact of the bias scaling factor just for one example and increase it to 0.2.\n",
    "\n",
    "The spectral radius is decreased to 0.9 in order to fulfil the Echo State Property."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [],
   "source": [
    "esn = ESNRegressor(k_in=1, input_scaling=1.0, spectral_radius=0.9, bias=0.2, ext_bias=False, leakage=1.0,\n",
    "                   reservoir_size=50, k_res=10, reservoir_activation='tanh', bi_directional=False,\n",
    "                   teacher_scaling=1.0, teacher_shift=0.0, solver='ridge', beta=1e-6, random_state=10)\n",
    "\n",
    "esn.fit(X=X, y=y, n_jobs=0)\n",
    "_ = esn.predict(X=X, keep_reservoir_state=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Two impacts of the bias scaling can be mainly observed: (1) The absolute value of the stable states of the reservoir neurons is approximately distributed from 0 to 0.2 and each neuron has its own stable state. When new information from the input is passed to the reservoir neurons, this is the excitation point. (2) Before the impulse arrives in the reservoir ($n=5$), the states are approaching their stable state. Due to the spectral radius, each reservoir neuron is connected to other neurons and thus feeds the constant bias through the network, until each neuron has reached its final state."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [
    {
     "data": {
      "application/pdf": "[encoded data, 530804 characters]\n",
      "image/png": "[encoded data, 105524 characters]\n",
      "text/plain": [
       "<Figure size 5700x1320 with 2 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "_, ax = plt.subplots(figsize=(4.75, 1.1), dpi=1200)\n",
    "im = ax.imshow(np.abs(esn.reservoir_state[:, 1:].T),vmin=0, vmax=1)\n",
    "ax.set_xlim([0, esn.reservoir_state.shape[0]])\n",
    "ax.set_ylim([0, esn.reservoir_state.shape[1] - 1])\n",
    "ax.set_xlabel('n')\n",
    "ax.set_ylabel('R[n]')\n",
    "plt.colorbar(im, ax=ax)\n",
    "plt.grid()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Finally, let us observe the impact of the leakage. Leaky integration is the other way to incorporate past information\n",
    "into the reservoir. This works by keeping the previous reservoir states over a longer time."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [],
   "source": [
    "esn = ESNRegressor(k_in=1, input_scaling=1.0, spectral_radius=0.9, bias=0.0, ext_bias=False, leakage=0.3,\n",
    "                   reservoir_size=50, k_res=10, reservoir_activation='tanh', bi_directional=False,\n",
    "                   teacher_scaling=1.0, teacher_shift=0.0, solver='ridge', beta=1e-6, random_state=10)\n",
    "\n",
    "esn.fit(X=X, y=y, n_jobs=0)\n",
    "_ = esn.predict(X=X, keep_reservoir_state=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The leakage behaves in the same way for all nodes in the reservoir and acts like a low-pass filter. The magnitude is strongly damped, and all reservoir states are decaying exponentially over a longer time. Due to the spectral radius, all neurons have individual decaying times."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [
    {
     "data": {
      "application/pdf": "[encoded data, 526116 characters]\n",
      "image/png": "[encoded data, 98200 characters]\n",
      "text/plain": [
       "<Figure size 5700x1320 with 2 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "_, ax = plt.subplots(figsize=(4.75, 1.1), dpi=1200)\n",
    "im = ax.imshow(np.abs(esn.reservoir_state[:, 1:].T),vmin=0, vmax=1)\n",
    "ax.set_xlim([0, esn.reservoir_state.shape[0]])\n",
    "ax.set_ylim([0, esn.reservoir_state.shape[1] - 1])\n",
    "ax.set_xlabel('n')\n",
    "ax.set_ylabel('R[n]')\n",
    "plt.colorbar(im, ax=ax)\n",
    "plt.grid()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "If we would like to incorporate future information into the reservoir, we can pass our input samples for- and backward\n",
    "through the reservoir. Therefore, we set bidirectional to True."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [],
   "source": [
    "esn = ESNRegressor(k_in=1, input_scaling=1.0, spectral_radius=0.9, bias=0.2, ext_bias=False, leakage=0.3,\n",
    "                   reservoir_size=50, k_res=10, reservoir_activation='tanh', bi_directional=True,\n",
    "                   teacher_scaling=1.0, teacher_shift=0.0, solver='ridge', beta=1e-6, random_state=10)\n",
    "\n",
    "esn.fit(X=X, y=y, n_jobs=0)\n",
    "_ = esn.predict(X=X, keep_reservoir_state=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can see the clear effect of usind bidirectional ESNs:\n",
    "\n",
    "Because of the additional backward-pass, the number of reservoir states is doubled, and we can see that they decay in\n",
    "both, forward and backward direction. This is especially useful for some classification tasks."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [
    {
     "data": {
      "application/pdf": "[encoded data, 526468 characters]\n",
      "image/png": "[encoded data, 90724 characters]\n",
      "text/plain": [
       "<Figure size 5700x1320 with 2 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "_, ax = plt.subplots(figsize=(4.75, 1.1), dpi=1200)\n",
    "im = ax.imshow(np.abs(esn.reservoir_state[:, 1:].T),vmin=0, vmax=1)\n",
    "ax.set_xlim([0, esn.reservoir_state.shape[0]])\n",
    "ax.set_ylim([0, esn.reservoir_state.shape[1] - 1])\n",
    "ax.set_xlabel('n')\n",
    "ax.set_ylabel('R[n]')\n",
    "plt.colorbar(im, ax=ax)\n",
    "plt.grid()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The ESN from the Mackey-Glass dataset with a reduced number of neurons"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [],
   "source": [
    "esn = ESNRegressor(k_in=1, input_scaling=1.0, spectral_radius=1.2, bias=0.0, ext_bias=False, leakage=1.0,\n",
    "                   reservoir_size=50, k_res=10, wash_out=0, reservoir_activation='tanh', bi_directional=False,\n",
    "                   teacher_scaling=1., teacher_shift=0., solver='ridge', beta=1e-4, random_state=0)\n",
    "\n",
    "esn.fit(X=X, y=y, n_jobs=0)\n",
    "_ = esn.predict(X=X, keep_reservoir_state=True)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Visualization of the Mackey-Glass-ESN"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [
    {
     "data": {
      "application/pdf": "[encoded data, 550400 characters]\n",
      "image/png": "[encoded data, 129508 characters]\n",
      "text/plain": [
       "<Figure size 5700x1320 with 2 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "_, ax = plt.subplots(figsize=(4.75, 1.1), dpi=1200)\n",
    "im = ax.imshow(np.abs(esn.reservoir_state[:, 1:].T),vmin=0, vmax=1)\n",
    "ax.set_xlim([0, esn.reservoir_state.shape[0]])\n",
    "ax.set_ylim([0, esn.reservoir_state.shape[1] - 1])\n",
    "ax.set_xlabel('n')\n",
    "ax.set_ylabel('R[n]')\n",
    "plt.colorbar(im, ax=ax)\n",
    "plt.grid()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The ESN from the Stock-Price dataset with a reduced number of neurons"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [],
   "source": [
    "esn = ESNRegressor(k_in=1, input_scaling=0.6, spectral_radius=0.9, bias=0.0, ext_bias=False, leakage=1.0,\n",
    "                   reservoir_size=50, k_res=10, reservoir_activation='tanh', bi_directional=False,\n",
    "                   teacher_scaling=1., teacher_shift=0., solver='ridge', beta=1e-8, random_state=0)\n",
    "\n",
    "esn.fit(X=X, y=y, n_jobs=0)\n",
    "_ = esn.predict(X=X, keep_reservoir_state=True)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Visualization of the Stock-Price-ESN"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [
    {
     "data": {
      "application/pdf": "[encoded data, 527608 characters]\n",
      "image/png": "[encoded data, 99932 characters]\n",
      "text/plain": [
       "<Figure size 5700x1320 with 2 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "_, ax = plt.subplots(figsize=(4.75, 1.1), dpi=1200)\n",
    "im = ax.imshow(np.abs(esn.reservoir_state[:, 1:].T),vmin=0, vmax=1)\n",
    "ax.set_xlim([0, esn.reservoir_state.shape[0]])\n",
    "ax.set_ylim([0, esn.reservoir_state.shape[1] - 1])\n",
    "ax.set_xlabel('n')\n",
    "ax.set_ylabel('R[n]')\n",
    "plt.colorbar(im, ax=ax)\n",
    "plt.grid()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The ESN from the Multipitch dataset with a reduced number of neurons"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [],
   "source": [
    "esn = ESNRegressor(k_in=1, input_scaling=0.6, spectral_radius=0.2, bias=0.7, ext_bias=False, leakage=0.3,\n",
    "                   reservoir_size=50, k_res=10, reservoir_activation='tanh', bi_directional=False,\n",
    "                   teacher_scaling=1., teacher_shift=0., solver='ridge', beta=1e-8, random_state=0)\n",
    "\n",
    "esn.fit(X=X, y=y, n_jobs=0)\n",
    "_ = esn.predict(X=X, keep_reservoir_state=True)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Visualization of the Multipitch-ESN"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [
    {
     "data": {
      "application/pdf": "[encoded data, 526460 characters]\n",
      "image/png": "[encoded data, 100172 characters]\n",
      "text/plain": [
       "<Figure size 5700x1320 with 2 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "_, ax = plt.subplots(figsize=(4.75, 1.1), dpi=1200)\n",
    "im = ax.imshow(np.abs(esn.reservoir_state[:, 1:].T),vmin=0, vmax=1)\n",
    "ax.set_xlim([0, esn.reservoir_state.shape[0]])\n",
    "ax.set_ylim([0, esn.reservoir_state.shape[1] - 1])\n",
    "ax.set_xlabel('n')\n",
    "ax.set_ylabel('R[n]')\n",
    "plt.colorbar(im, ax=ax)\n",
    "plt.grid()"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.5"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 1
}
