Madalina Ciortan

# Gentle introduction to Echo State Networks

This post will address the following questions:

- What are Echo State Networks?

- Why and when should you use an Echo State Network?

- Simple implementation example in python

The figure below is a simplification of the paper __Reservoir computing approaches for representation and classification of multivariate time series__ but it captures well the gist of ESNs. Each component will be detailed in the following sections.

Echo state network is a type of Recurrent Neural Network, part of the *reservoir computing* framework, which has the following particularities:

- the weights between the input -the hidden layer ( the ‘reservoir’) : *Win* and also the weights of the ‘reservoir’: *Wr* are **randomly assigned** and **not trainable**

- the weights of the output neurons (the ‘readout’ layer) are **trainable **and can be learned so that the network can reproduce specific temporal patterns

- the hidden layer (or the ‘reservoir’) is very sparsely connected (typically < 10% connectivity)

- the reservoir architecture creates a recurrent **non linear** embedding (H on the image below) of the input which can be then connected to the desired output and these final weights will be trainable

- it is possible to connect the embedding to a different predictive model (a trainable NN or a ridge regressor/SVM for classification problems)

## Reservoir Computing

Reservoir computing is an extension of neural networks in which the input signal is connected to a fixed (non-trainable) and random dynamical system (the reservoir), thus creating a higher dimension representation (embedding). This embedding is then connected to the desired output via trainable units.

The non-recurrent equivalent of reservoir computing is the __Extreme Learning Machine__ and consists only of feed forward networks having only the readout layer trainable.

## Workflow

For an input of shape N, T, V, where N is the number of observations, T is the number of time steps and V is the number of variables we will:

- choose the size of the reservoir R and other parameters governing the level of sparsity of connection, if we want to model a leakage, the ideal number of components after the dimensionality reduction, etc

- generate (V, R) input weights *Win* by sampling from a random binomial distribution

- generate (R, R) reservoir weights *Wr *by sampling from an uniform distribution with a given density, parameter which sets the level of sparsity

- calculate the high dimensional state representation H as a non linear function (typically tanh) of the input at the current time step (N, V) multiplied by the internal weights plus the previous state multiplied by the reservoir matrix (R, R)

- optionally we can run a dimensionality reduction algorithm such as PCA to D components, which brings H to (N, T, D)

-create an input representation either by using for example the entire reservoir and training a regressor to map states t to t+1: one representation could be the matrix of all calculated slopes and intercepts. Another option could be to use the mean or the last value of H

-connect this embedding to the desired output, either by using a NN structure which will be trainable or by using other types of predictors. The above mentioned paper suggest the use of Ridge regression

## Why and when should you use Echo State Networks?

-Traditional NN architectures suffer from the vanishing/exploding gradient problem and as such, the parameters in the hidden layers either don’t change that much or they lead to numeric instability and chaotic behavior. Echo state networks don’t suffer from this problem

-Traditional NN architectures are computationally expensive, Echo State Networks are very fast as there is no back propagation phase on the reservoir

-Traditional NN can be disrupted by __bifurcations__

-ESN are well adapted for handling chaotic time series

## Implementation

The paper comes with a cleanly written and well documented implementation of a pipeline proposing various approaches for each step (weight initialization scheme, dimensionality reduction, input representation, readout layer). I have created more simplified version by cherry-picking steps such as PCA, reservoir representation based on ridge regression and linear regression connection to output as these options were best performing on my input. I made this version available __here__.