Pytorch is a deep learning framework, i.e. set of functions and libraries which allow you to do higher-order programming designed for Python programming language based on Torch, which is an open-source machine learning package based on the programming language Lua. It is primarily developed by Facebook’s artificial-intelligence research group and Uber’s Pyro probabilistic programming language software is built on it.
PyTorch is more pythonic and has a more consistent API. It also has native ONNX model exports, which can be used to speed up inference. Also, PyTorch shares many commands with numpy, which helps in learning the framework with ease.
At its core, PyTorch provides two main features:
- An n-dimensional Tensor, similar to Numpy but can run on GPUs
- Automatic differentiation for building and training neural networks
If you’re using anaconda distribution, you can install the Pytorch by running the below command in the anaconda prompt
conda install pytorch-cpu torchvision-cpu -c pytorch
Rest of the article is structured as follows:
- What is Colab, Anyway?
- Setting up GPU in Colab
- Pytorch Tensors
- Simple Tensor Operations
- Pytorch to Numpy Bridge
- CUDA Support
- Automatic Differentiation
- Conclusion
Colab – Colaboratory
Google Colab is a research tool for machine learning education and research. It’s a Jupyter notebook environment that requires no setup to use. Colab offers a free GPU cloud service hosted by Google to encourage collaboration in the field of Machine Learning, without worrying about the hardware requirements. Colab was released to the public by Google in October 2017
Getting Started with Colab
- Go to Google Colab
- Sign in with your Google Account
- Create a new notebook via File -> New Python 3 notebook or New Python 2 notebook
You can also create a notebook in Colab via Google Drive
- Go to Google Drive
- Create a folder of any name in the drive to save the project
- Create a new notebook via Right click > More > Colaboratory
To rename the notebook, just click on the file name present at the top of the notebook.
Setting up GPU in Colab
In Colab, you will get 12 hours of execution time but the session will be disconnected if you are idle for more than 60 minutes. It means that for every 12 hours Disk, RAM, CPU Cache and the Data that is on our allocated virtual machine will get erased.
To enable GPU hardware accelerator, just go to Runtime -> Change runtime type -> Hardware accelerator -> GPU
Pytorch – Tensors
Numpy based operations are not optimized to utilize GPUs to accelerate its numerical computations. For modern deep neural networks, GPUs often provide speedups of 50x or greater. So, unfortunately, numpy won’t be enough for modern deep learning. This where Pytorch introduces the concept of Tensor. A Pytorch Tensor is conceptually identical to an n-dimensional numpy array. Unlike the numpy, PyTorch Tensors can utilize GPUs to accelerate their numeric computations
Let’s see how you can create a Pytorch Tensor. First, we will import the required libraries. Remember that torch, numpy and matplotlib are pre-installed in Colab’s virtual machine.
import torch import numpy import matplotlib.pyplot as plt
The default tensor type in PyTorch is a float tensor defined as torch.FloatTensor. We can create tensors by using the inbuilt functions present inside the torch package.
## creating a tensor of 3 rows and 2 columns consisting of ones
>> x = torch.ones(3,2)
>> print(x)
tensor([[1., 1.],
[1., 1.],
[1., 1.]])
## creating a tensor of 3 rows and 2 columns consisting of zeros
>> x = torch.zeros(3,2)
>> print(x)
tensor([[0., 0.],
[0., 0.],
[0., 0.]])Creating a tensor by random initialization
To increase the reproducibility, we often set the random seed to a specific value first. >> torch.manual_seed(2) #generating tensor randomly >> x = torch.rand(3, 2) >> print(x) #generating tensor randomly from normal distribution >> x = torch.randn(3,3) >> print(x)
Simple Tensor Operations
Slicing of Tensors
You can slice PyTorch tensors the same way you slice ndarrays
#create a tensor
>> x = torch.tensor([[1, 2],
[3, 4],
[5, 6]])
>> print(x[:, 1]) # Every row, only the last column
>> print(x[0, :]) # Every column in first row
>> y = x[1, 1] # take the element in first row and first column and create a another tensor
>> print(y)
Reshape Tensor
Reshape a Tensor to different shape
>> x = torch.tensor([[1, 2],
[3, 4],
[5, 6]]) #(3 rows and 2 columns)
>> y = x.view(2, 3) #reshaping to 2 rows and 3 columns
Use of -1 to reshape the tensors.
-1 indicates that the shape will be inferred from previous dimensions. In the below code snippet x.view(6,-1) will result in a tensor of shape 6x1 because we have fixed the size of rows to be 6, Pytorch will now infer the best possible dimension for the column such that it will be able to accommodate all the values present in the tensor.
>> x = torch.tensor([[1, 2],
[3, 4],
[5, 6]]) #(3 rows and 2 columns)
>> y = x.view(6,-1) #y shape will be 6x1
Mathematical Operations
#Create two tensors >> x = torch.ones([3, 2]) >> y = torch.ones([3, 2]) #adding two tensors >> z = x + y #method 1 >> z = torch.add(x,y) #method 2 #subtracting two tensors >> z = x - y #method 1 >> torch.sub(x,y) #method 2
Inplace Operations
In Pytorch all operations on the tensor that operate in-place on it will have an _ postfix. For example, add is the out-of-place version, and add_ is the in-place version.
>> y.add_(x) #tensor y added with x and result will be stored in y
Pytorch to Numpy Bridge
Converting an Pytorch tensor to numpy ndarray is very useful sometimes. By using .numpy() on a tensor, we can easily convert tensor to ndarray.
>> x = torch.linspace(0 , 1, steps = 5) #creating a tensor using linspace >> x_np = x.numpy() #convert tensor to numpy >> print(type(x), type(x_np)) #check the types <class 'torch.Tensor'> <class 'numpy.ndarray'>
To convert numpy ndarray to pytorch tensor, we can use .from_numpy() to convert ndarray to tensor
>> a = np.random.randn(5) #generate a random numpy array >> a_pt = torch.from_numpy(a) #convert numpy array to a tensor >> print(type(a), type(a_pt)) <class 'numpy.ndarray'> <class 'torch.Tensor'>
During the conversion, Pytorch tensor and numpy ndarray will share their underlying memory locations and changing one will change the other.
CUDA Support
To check how many CUDA supported GPU’s are connected to the machine, you can use below code snippet. If you are executing the code in Colab you will get 1, that means that the Colab virtual machine is connected to one GPU. torch.cuda is used to set up and run CUDA operations. It keeps track of the currently selected GPU.
>> print(torch.cuda.device_count()) 1
If you want to get the name of the GPU Card connected to the machine,
>> print(torch.cuda.get_device_name(0)) Tesla T4
The important thing to note is that we can reference this CUDA supported GPU card to a variable and use this variable for any Pytorch Operations. All CUDA tensors you allocate will be created on that device. The selected GPU device can be changed with a torch.cuda.device context manager.
#Assign cuda GPU located at location '0' to a variable
>> cuda0 = torch.device('cuda:0')
#Performing the addition on GPU
>> a = torch.ones(3, 2, device=cuda0) #creating a tensor 'a' on GPU
>> b = torch.ones(3, 2, device=cuda0) #creating a tensor 'b' on GPU
>> c = a + b
>> print(c)
tensor([[2., 2.],
[2., 2.],
[2., 2.]], device='cuda:0')
As you can see from the above code snippet the tensors are created on GPU and any operation you do on these tensors will be done on GPU. If you want to move the result to CPU you just have to do .cpu()
#moving the result to cpu
>> c = c.cpu()
>> print(c)
tensor([[2., 2.],
[2., 2.],
[2., 2.]])
Automatic Differentiation
In this section, we will discuss the important package called automatic differentiation or autograd in Pytorch. The autograd package gives us the ability to perform automatic differentiation or automatic gradient computation for all operations on tensors. It is a define-by-run framework, which means that your back-propagation is defined by how your code is run.
Let’s see how to perform automatic differentiation by using a simple example. First, we create a tensor with requires_grad parameter set to True because we want to track all the operations performing on that tensor.
#create a tensor with requires_grad = True
>> x = torch.ones([3,2], requires_grad = True)
>> print(x)
tensor([[1., 1.],
[1., 1.],
[1., 1.]], requires_grad=True)
Perform a simple tensor addition operation
>> y = x + 5 #tensor addition
>> print(y) #check the result
tensor([[6., 6.],
[6., 6.],
[6., 6.]], grad_fn=<AddBackward0>)
Because y was created as a result of an operation on x, so it has a grad_fn. Perform more operations on y and create a new tensor z.
>> z = y*y + 1
>> print(z)
tensor([[37., 37.],
[37., 37.],
[37., 37.]], grad_fn=<AddBackward0>)
>> t = torch.sum(z) #adding all the values in z
>> print(t)
tensor(222., grad_fn=<SumBackward0>)
Back-Propagation
To perform back-propagation, you can just call t.backward()
>> t.backward() #peform backpropagation but pytorch will not print any output.
Print gradients d(t)/dx.
>> print(x.grad)
tensor([[12., 12.],
[12., 12.],
[12., 12.]])
x.grad will give you the partial derivative of t with respect to x. If you are able to figure out how we got a tensor with all the values equal to 12, then you have understood the automatic differentiation. If not don’t worry just follow along, when we execute t.backward() we are calculating the partial derivate of t with respect to x. Remember that t is a function of z, which in turn is a function of x.
d(t)/dx = 2y + 1 at x = 1 and y = 6, where y = x + 5
The important point to note is that the value of the derivative is calculated at the point where we initialized the tensor x. Since we initialized x at a value equal to one, we get an output tensor with all the values equal to 12.
Conclusion
In this post, we briefly looked at the Pytorch & Google Colab and we also saw how to enable GPU hardware accelerator in Colab. Then we have seen how to create tensors in Pytorch and perform some basic operations on those tensors by utilizing CUDA supported GPU. After that, we discussed the autograd
Note: This is a guest post, and opinion in this article is of the guest writer. If you have any issues with any of the articles posted at www.marktechpost.com please contact at asif@marktechpost.com
Niranjan Kumar is working as a Senior Consultant Data Science at Allstate India. He is passionate about Deep Learning and Artificial Intelligence. He writes about the latest tools and technologies in the field of Deep Learning. He is one of the top writers in Artificial Intelligence at Medium. A Graduate of Praxis Business School, Niranjan Kumar holds a degree in Data Science. Feel free to contact him via LinkedIn for collaboration on projects
