|
Abstract:
An autoencoder consists of an encoder network that converts the input into a low-dimensional code and a decoder network that converts the code back into a reconstruction of the input. If the encoder and decoder networks both contain hidden layers, an autoencoder ought to be a powerful, non-linear generalization of principal components. Highly non-linear autoencoders have a long and disappointing history. We want the similarities between codes to be a better reflection of the underlying similarities in the data than the raw input vectors themselves, but autoencoders fail to achieve this. I shall explain why they fail and show how they can be fixed by using a regularizer that encourages the autoencoder to use similar codes for similar input vectors. Then I shall show how a biologically unrealistic regularizer could be replaced by a much more realistic one. Finally I shall show how low-dimensional codes can be learned by large populations of neurons that do not need prior knowledge of the appropriate dimensionality.