What is "with torch no_grad" in PyTorch?

Last Updated : 05 Jun, 2022

In this article, we will discuss what does with a torch.no_grad() method do in PyTorch.

torch.no_grad() method

With torch.no_grad() method is like a loop in which every tensor in that loop will have a requires_grad set to False. It means that the tensors with gradients currently attached to the current computational graph are now detached from the current graph and no longer we will be able to compute the gradients with respect to that tensor. Until the tensor is within the loop it is detached from the current graph. As soon as the tensor defined with gradient is out of the loop, it is again attached to the current graph. This method disables the gradient calculation which reduces the memory consumption for computations.

Example 1

In this example, we will define a tensor A with requires_grad=true and then we will define a function B inside the torch.no_grad() using the tensor A. Now tensor A is within the loop so the requires_grad is set to false.

Python3

# import necessary libraries import torch  # define a tensor A = torch.tensor(1., requires_grad=True) print("Tensor-A:", A)  # define a function using A tensor  # inside loop with torch.no_grad():     B = A + 1 print("B:-", B)  # check gradient print("B.requires_grad=", B.requires_grad)

Output

Tensor-A: tensor(1., requires_grad=True)  B:- tensor(2.)  B.requires_grad= False

Example 2

In this example, we will define two functions, one should be inside the loop, and the other will be outside of it after that we will check the difference between the requires_grad parameter value for the two functions. In the output, we will see that the requires_grad is set to False because the function y is inside the loop which generally disables the gradient calculation, whereas requires_grad is set to True for function x which is outside of the loop.

Python3

# import necessary libraries import torch  # define  tensors A = torch.tensor(1., requires_grad=False) print("Tensor-A:", A) B = torch.tensor(2.2, requires_grad=True) print("Tensor-B:", B)  # define a function x outside loop and  # check gradient x = A+B print("x:-", x) print("x.requires_grad=", x.requires_grad)  # define a function y inside loop and  # check gradient with torch.no_grad():     y = B - A print("y:-", y) print("y.requires_grad=", y.requires_grad)

Output

Tensor-A: tensor(1.)  Tensor-B: tensor(2.2000, requires_grad=True)  x:- tensor(3.2000, grad_fn=<AddBackward0>)  x.requires_grad= True  y:- tensor(1.2000)  y.requires_grad= False

Example 3

In this example, we are just defining a tensor A using the tensor method the requires_grad is set to be True and by defining a function x using the tensor A inside the no_grad() method it checked the requires_grad value for the function x. In the output the function x has requires_grad is set to False because it is specified in the loop, which generally disables the gradient calculation.

Python3

# import necessary libraries import torch  # define a tensor A = torch.tensor(5., requires_grad=True) print("Tensor-A:", A)  # define a function x inside loop and  # check gradient with torch.no_grad():     x = A**2 print("x:-", x) print("x.requires_grad=", x.requires_grad)

Output

Tensor-A: tensor(5., requires_grad=True)  x:- tensor(25.)  x.requires_grad= False

What are Torch Scripts in PyTorch?

akhilvasabhaktula03

Improve

Article Tags :

Practice Tags :

python

Similar Reads

What are Torch Scripts in PyTorch?

TorchScript is a powerful feature in PyTorch that allows developers to create serializable and optimizable models from PyTorch code. It serves as an intermediate representation of a PyTorch model that can be run in high-performance environments, such as C++, without the need for a Python runtime. Th

5 min read

Why Do We Need to Call zero_grad() in PyTorch?

When training neural networks in PyTorch, managing gradients is a fundamental part of the optimization process. One crucial step often included in training loops is calling optimizer.zero_grad() before performing the backward pass. In this article, we will explore the importance of this step, how it

6 min read

Way to Copy a Tensor in PyTorch

In deep learning, PyTorch has become a popular framework for building and training neural networks. At the heart of PyTorch is the tensorâ€”a multi-dimensional array that serves as the fundamental building block for all operations in the framework. There are many scenarios where you might need to copy

5 min read

PyTorch Lightning with TensorBoard

Pytorch-Lightning is a popular deep learning framework. It basically works with PyTorch models to simplify the training and testing of the models. This library is useful for distributed training as one can train the model seamlessly without much complex codes. Now to get the metrics in an user inter

5 min read

What Does model.train() Do in PyTorch?

A crucial aspect of training a model in PyTorch involves setting the model to the correct mode, either training or evaluation. This article delves into the purpose and functionality of the model.train() method in PyTorch, explaining its significance in the training process and how it interacts with

4 min read

What is PyTorch Ignite?

PyTorch Ignite is a high-level library designed to simplify the process of training and evaluating neural networks using PyTorch. It provides a flexible and transparent framework that allows developers to focus on building models rather than dealing with the complexities of the training process. Thi

7 min read

Difference Between detach() and with torch.no_grad() in PyTorch

In PyTorch, managing gradients is crucial for optimizing models and ensuring efficient computations. Two commonly used methods to control gradient tracking are detach() and with torch.no_grad(). Understanding the differences between these two approaches is essential for effectively managing computat

6 min read

What Is the Relationship Between PyTorch and Torch?

The landscape of deep learning frameworks has evolved significantly over the years, with various libraries emerging to cater to different needs and preferences. Two prominent frameworks in this domain are PyTorch and Torch, which, despite their similarities in name, have distinct origins, functional

6 min read

How to Write Distributed Applications with Pytorch?

Distributed computing has become essential in the era of big data and large-scale machine learning models. PyTorch, one of the most popular deep learning frameworks, offers robust support for distributed computing, enabling developers to train models on multiple GPUs and machines. This article will

6 min read

Two-Dimensional Tensors in Pytorch

PyTorch is a python library developed by Facebook to run and train machine learning and deep learning models. In PyTorch everything is based on tensor operations. Two-dimensional tensors are nothing but matrices or vectors of two-dimension with specific datatype, of n rows and n columns. Representat

3 min read