How Autoencoders works ?

Last Updated : 01 Mar, 2025

Autoencoders is a type of neural network used for unsupervised learning particularly for tasks like dimensionality reduction, anomaly detection and feature extraction. It consists of two main parts: an encoder and a decoder. The goal of an autoencoder is to learn a more efficient representation of the data by minimizing the difference between the input and its reconstructed version.

In this article we will learn how autoencoders work, their cost function and optimization techniques and how to implement a deep convolutional autoencoder.

Understanding Working of AutoEncoders

Autoencoder is made up of two main parts: the encoder and the decoder.

Encoder: The encoder takes an input sample [Tex] x_i[/Tex] and compresses it into a lower-dimensional representation often called the latent space or latent code denoted by [Tex]z_i = e(x_i)[/Tex]
Decoder: The decoder then reconstructs the original input from this compressed representation producing an output denoted by [Tex]\hat{x}_i = d(z_i)[/Tex]

The goal of the autoencoder is to learn a mapping from the input space to a lower-dimensional space and back to the original space. This is done by training the network to minimize the difference between the input and its reconstruction.

Cost Function and Optimization

In the context of an autoencoder the cost function finds how well the autoencoder is performing. The objective is to minimize the difference between the input sample [Tex]x_i[/Tex] and the reconstructed output[Tex] \hat{x}_i[/Tex] typically measured using the Mean Squared Error (MSE). The MSE is defined as:

[Tex]MSE = \frac{1}{n} \sum_{i=1}^{n} (x_i – \hat{x}_i)^2[/Tex]

Where:

[Tex]n[/Tex] is the number of samples,
[Tex]x_i[/Tex] is the input sample,
[Tex]\hat{x}_i[/Tex] is the reconstructed output.

The goal is to minimize this error through optimization techniques such as gradient descent or the Adam optimizer which adjusts the weights of the encoder and decoder networks to reduce the reconstruction error.

Variational Autoencoder

A Variational Autoencoder (VAE) is an extension of the basic autoencoder that models the data distribution probabilistically. Instead of encoding the input into a single point in latent space, the encoder outputs a probabilistic distribution [Tex]q(z)[/Tex] over the latent code [Tex]z[/Tex]. The decoder then learns to map this distribution back to the original data space.

The main objective in a VAE is to learn a distribution that approximates the true data distribution [Tex]p_{\text{data}}(x)[/Tex]. To measure how close the learned distribution [Tex]q(z) [/Tex]is to the true distribution we use the Kullback-Leibler (KL) Divergence:

[Tex]KL(p_{\text{data}} \parallel q) = \sum p_{\text{data}}(x) \log\left( \frac{p_{\text{data}}(x)}{q(z)} \right)[/Tex]

KL divergence finds how much the learned distribution away from the true data distribution. In VAE this term is added to the loss function to encourage the model to learn a latent distribution that is as close as possible to the true distribution of the data.

The optimization process aims to minimize both the reconstruction error (MSE) and the KL divergence, which results in the combined objective function:

[Tex]\mathcal{L} = \text{MSE}(x_i, \hat{x}_i) + \lambda \cdot KL(p_{\text{data}} \parallel q)[/Tex]

where [Tex]\lambda[/Tex] is a regularization parameter that controls the relative importance of the reconstruction loss and the KL divergence.

Equivalence of MSE and KL Divergence

In cases where both the data distribution [Tex]p_{\text{data}}(x)[/Tex] and the latent distribution [Tex]q(z)[/Tex] are Gaussian distributions the KL become equivalent. This is because minimizing the KL divergence between two Gaussian distributions is mathematically the same as minimizing the MSE between their means and variances.

Thus when working with Gaussian distributions the VAE’s objective function simplifies to the standard MSE making it easier to compute and optimize.

Bernoulli Distribution and Binary Data

For datasets where the data is binary or normalized between 0 and 1 (for example pixel values in images) the appropriate model is to use a Bernoulli distribution for both [Tex]p_{\text{data}}(x)[/Tex] and [Tex]q(z)[/Tex]. The output of the decoder is typically modeled using a sigmoid activation function ensuring that the output values are constrained to the range [0, 1] which can be interpreted as probabilities.

In this case the cost function changes to Binary Cross-Entropy which is more suitable for binary or probabilistic data:

[Tex]\text{Binary Cross-Entropy} = – \sum_{i=1}^{n} \left[ x_i \log(\hat{x}_i) + (1 – x_i) \log(1 – \hat{x}_i) \right][/Tex]

Where:

[Tex]x_i[/Tex] is the binary input data (0 or 1),
[Tex]\hat{x}_i[/Tex] is the predicted probability output by the decoder (after applying the sigmoid function).

This cost function is used when reconstructing binary or probabilistic data such as tasks involving binary classification or image generation with binary-valued pixels.

Implementing a deep convolutional autoencoder

This guide explains how to build a deep convolutional autoencoder using TensorFlow. We’ll train it on the Olivetti Faces dataset which contains 400 grayscale images (64×64 pixels each). The autoencoder will compress and reconstruct these images.

Step 1: Load the Dataset

We will use the Olivetti Faces dataset which contains grayscale face images. The Olivetti Faces dataset is commonly used in face recognition tasks.

We first load the dataset using fetch_olivetti_faces from sklearn.datasets.

fetch_olivetti_faces(): Downloads the dataset.
shuffle=True: Mixes the data to avoid patterns during training.
random_state=1000: Ensures the shuffle order is always the same for reproducibility.

Python

import tensorflow as tf import numpy as np import matplotlib.pyplot as plt from tensorflow import keras from tensorflow.keras import layers from sklearn.datasets import fetch_olivetti_faces  faces = fetch_olivetti_faces(shuffle=True, random_state=1000) X_train = faces['images']

Step 2: Resize the Images

Now to increase the speed of our computation we will resize them to 32 × 32. This will also help avoid any memory issues. We may lose a minor visual precision. Note that you can skip this if you have high computational resources.

Python

width, height = 32, 32 X_train = tf.image.resize(X_train[..., np.newaxis], (width, height))

Step 3: Define Constants

Now let’s setting up training parameters

nb_epochs: More epochs give better performance but take more time.
batch_size: Smaller batches need less memory but may take longer to converge.
code_length: Size of the compressed data representing the image.

Python

nb_epochs = 100 batch_size = 50 code_length = 256

Step 4: Adding Noise to Images

To make the autoencoder more robust we add Gaussian noise to the images. This will be the input to the autoencoder during training.

tf.random.normal(): Creates random noise.
noise_factor: Controls how much noise is added.
tf.clip_by_value(): Keeps pixel values between 0 and 1 (valid grayscale range).

Python

def add_noise(images, noise_factor=0.2):     noisy_images = images + noise_factor * tf.random.normal(shape=images.shape)     return tf.clip_by_value(noisy_images, 0.0, 1.0)

Step 5: Define the Autoencoder

Conv2D: Extracts features from the image.
Flatten: Converts the image into a 1D vector.
Dense: Fully connected layer for compression.
Conv2DTranspose: Up sampling layers to reconstruct the image.
sigmoid: Keeps pixel values between 0 and 1.

Python

def build_autoencoder(input_shape, code_length=256):     input_img = keras.Input(shape=input_shape)      x = layers.Conv2D(16, (3, 3), strides=(2, 2), activation='relu', padding='same')(input_img)     x = layers.Conv2D(32, (3, 3), strides=(1, 1), activation='relu', padding='same')(x)     x = layers.Conv2D(64, (3, 3), strides=(1, 1), activation='relu', padding='same')(x)     x = layers.Conv2D(128, (3, 3), strides=(1, 1), activation='relu', padding='same')(x)     x = layers.Flatten()(x)     code_layer = layers.Dense(code_length, activation='sigmoid')(x)      x = layers.Dense((width // 2) * (height // 2) * 128, activation='relu')(code_layer)     x = layers.Reshape((width // 2, height // 2, 128))(x)     x = layers.Conv2DTranspose(128, (3, 3), strides=(2, 2), activation='relu', padding='same')(x)     x = layers.Conv2DTranspose(64, (3, 3), strides=(1, 1), activation='relu', padding='same')(x)     x = layers.Conv2DTranspose(32, (3, 3), strides=(1, 1), activation='relu', padding='same')(x)     output_img = layers.Conv2DTranspose(1, (3, 3), strides=(1, 1), activation='sigmoid', padding='same')(x)      autoencoder = keras.Model(input_img, output_img)     return autoencoder

Step 6: Define and Compile the Model

Now we define the loss function (MSE) and the optimizer (Adam).

Python

input_shape = (width, height, 1) autoencoder = build_autoencoder(input_shape)  autoencoder.compile(optimizer=keras.optimizers.Adam(learning_rate=0.001), loss='mse')

Step 7: Train the Model

We train the model to reconstruct noisy images and the model learns how to remove noise from images during training.

Python

X_train_noisy = add_noise(X_train) autoencoder.fit(X_train_noisy, X_train, epochs=nb_epochs, batch_size=batch_size, shuffle=True)

Output:

Epoch 1/100 8/8 ━━━━━━━━━━━━━━━━━━━━ 23s 2s/step – loss: 0.0268
Epoch 2/100 8/8 ━━━━━━━━━━━━━━━━━━━━ 22s 2s/step – loss: 0.0212
Epoch 3/100 8/8 ━━━━━━━━━━━━━━━━━━━━ 19s 2s/step – loss: 0.0206
…
Epoch 100/100 8/8 ━━━━━━━━━━━━━━━━━━━━ 11s 1s/step – loss: 0.0165

Step 8: Show Original and Reconstructed Images

Finally we display the original and reconstructed images

Python

def show_images(original, reconstructed, num=5):     plt.figure(figsize=(10, 4))     for i in range(num):         plt.subplot(2, num, i + 1)         plt.imshow(original[i].squeeze(), cmap='gray')         plt.axis('off')         plt.title("Original")          plt.subplot(2, num, num + i + 1)         plt.imshow(reconstructed[i].squeeze(), cmap='gray')         plt.axis('off')         plt.title("Reconstructed")      plt.show()  X_train_noisy = X_train_noisy.numpy() if isinstance(X_train_noisy, tf.Tensor) else X_train_noisy  reconstructed_images = autoencoder.predict(X_train_noisy[:5]) show_images(X_train_noisy, reconstructed_images)

Output:

Initially the reconstructed images appear blurred and noisy indicating that the model is still learning and hasn’t fully captured the necessary features of the images. However as the model progresses in training the reconstructions improve becoming clearer and more similar to the original images.

Variational AutoEncoders

muktik

Improve

Article Tags :

How Autoencoders works ?

Understanding Working of AutoEncoders

Cost Function and Optimization

Variational Autoencoder

Equivalence of MSE and KL Divergence

Bernoulli Distribution and Binary Data

Implementing a deep convolutional autoencoder

Step 1: Load the Dataset

Step 2: Resize the Images

Step 3: Define Constants

Step 4: Adding Noise to Images

Step 5: Define the Autoencoder

Step 6: Define and Compile the Model

Step 7: Train the Model

Step 8: Show Original and Reconstructed Images

Similar Reads

Introduction to Deep Learning

Basic Neural Network

Activation Functions

Artificial Neural Network

Classification

Regression

Hyperparameter tuning

Introduction to Convolution Neural Network

Recurrent Neural Network