84
edits
Changes
→Progress Report
3. Novell Rasam
== Introduction to Neural Networks ==
return t
== Parallelization Methods Data Parallelism == This section details a way to parallelize your NN .
device = torch.device("cuda:0")
Then, you can copy all your tensors to the GPU:
mytensor = my_tensor.to(device)
However, PyTorch will only use one GPU by default. In order to run on multiple GPUs you need to use <code>DataParallel</code>:
device = torch.device("cuda:0" if torch.cuda.is_available() else "cpu")
==== Dummy DataSet ====
You can make a random dummy dataset:
class RandomDataset(Dataset):
def __init__(self, size, length):
self.len = length
self.data = torch.randn(length, size)
def __getitem__(self, index):
return self.data[index]
def __len__(self):
return self.len
rand_loader = DataLoader(dataset=RandomDataset(input_size, data_size),
batch_size=batch_size, shuffle=True)
==== Simple Model ====
Here is a simple linear model definition, but <code>DataParallel</code> can be used any model (CNN, RNN, etc).
class Model(nn.Module):
# Our model
def __init__(self, input_size, output_size):
super(Model, self).__init__()
self.fc = nn.Linear(input_size, output_size)
def forward(self, input):
output = self.fc(input)
print("\tIn Model: input size", input.size(),
"output size", output.size())
return output
==== Create Model and DataParallel ====
Now that everything is defined, we need create an instance of the model and check if we have multiple GPUs.
model =Model(input_size, output_size) if torch.cuda.device_count() > 1: print("Let's use", torch.cuda.device_count(), "GPUs!") # dim == Single0 [30, xxx] -Machine Model ==> [10, ...], [10, ...], [10, ...] on 3 GPUs model =nn.DataParallel(model) # wrap model using nn.DataParallel
==== Basic Usage Run the Model ====
The print statement will let us see the input and output sizes. for data in rand_loader: input = data.to(device) output = model(input) print("Outside: input size", input.size(), "output_size", output.size()) ==== Results ==== If you have no GPU or only one GPU, the input and output size will match the batch size, so no parallelization. But if you have 2 GPUs, you'll get these results: # on 2 GPUs Let's use 2 GPUs! In Model: input size torch.Size([15, 5]) output size torch.Size([15, 2]) In Model: input size torch.Size([15, 5]) output size torch.Size([15, 2]) Outside: input size torch.Size([30, 5]) output_size torch.Size([30, 2]) In Model: input size torch.Size([15, 5]) output size torch.Size([15, 2]) In Model: input size torch.Size([15, 5]) output size torch.Size([15, 2]) Outside: input size torch.Size([30, 5]) output_size torch.Size([30, 2]) In Model: input size torch.Size([15, 5]) output size torch.Size([15, 2]) In Model: input size torch.Size([15, 5]) output size torch.Size([15, 2]) Outside: input size torch.Size([30, 5]) output_size torch.Size([30, 2]) In Model: input size torch.Size([5, 5]) output size torch.Size([5, 2]) In Model: input size torch.Size([5, 5]) output size torch.Size([5, 2]) Outside: input size torch.Size([10, 5]) output_size torch.Size([10, 2]) With 2 GPUs, the input and output sizes are half of 30, 15. === A More Intermediate Example === Here is a toy model that contains two linear layers. Each linear layer is designed to run a separate GPU[https://pytorch.org/tutorials/intermediate/model_parallel_tutorial.html].
import torch
pip install torch===1.7.0 torchvision===0.8.1 torchaudio===0.7.0 -f https://download.pytorch.org/whl/torch_stable.html
== Progress Report ==
*Update 1: Friday, November 27, 2020 - Started on Introduction to Neural Networks Section
*Update 2: Friday, November 27, 2020 - Installation and Configuration of Jupyter Lab
*Update 3: Saturday, November 28, 2020 - Practiced Working With and Learning About Jupyter Lab
*Update 4: Saturday, November 28, 2020 - Created a 4 layer ANN on Jupyter Lab
*Update 5: Saturday, November 28, 2020 - Initiated Training of the ANN and Verified Digit Recognition Capabilities
*Update 6: Sunday, November 29, 2020 - Finished Introduction to Neural Networks
*Update 7: Sunday, November 29, 2020 - Implemented a basic CNN Based on Previous Implementation of ANN
*Update 8: Monday, November 30, 2020 - Added Section on Data Parallel
==References==
*[1] Khan, Faisa. “Infographics Digest - Vol. 3.” Medium, [https://medium.com/datadriveninvestor/infographics-digest-vol-3-da67e69d71ce]*[2][5][8] “ANN vs CNN vs RNN | Types of Neural Networks.” Analytics Vidhya, 17 Feb. 2020, [https://www.analyticsvidhya.com/blog/2020/02/cnn-vs-rnn-vs-mlp-analyzing-3-types-of-neural-networks-in-deep-learning/.]*[3][4][6] 3Blue1Brown. But What Is a Neural Network? | Deep Learning, Chapter 1. 2017. YouTube, [https://www.youtube.com/watch?v=aircAruvnKk.]*[7] What Is Backpropagation Really Doing? | Deep Learning, Chapter 3. 2017. YouTube, [https://www.youtube.com/watch?v=Ilg3gGewQ5U&list=PLZHQObOWTQDNU6R1_67000Dx_ZCJB-3pi&index=3.]*PyTorch Tutorials: beginner Data Parallel [https://pytorch.org/tutorials/beginner/blitz/data_parallel_tutorial.html]*PyTorch Tutorials: Intermediate Model Parallel [https://pytorch.org/tutorials/intermediate/model_parallel_tutorial.html]*Building our Neural Network - Deep Learning and Neural Networks with Python and Pytorch p.3, YouTube, [https://www.youtube.com/watch?v=ixathu7U-LQ] *Training Model - Deep Learning and Neural Networks with Python and Pytorch p.4, YouTube, [https://www.youtube.com/watch?v=9j-_dOze4IM]*Deep Learning with PyTorch: Building a Simple Neural Network| packtpub.com, YouTube, [https://www.youtube.com/watch?v=VZyTt1FvmfU&list=LL&index=4]*Github PyTorch Neural Network Module Implementation [https://github.com/pytorch/pytorch/tree/master/torch/nn/modules]*Project Jupyter Official Documentation [https://jupyter.org/documentation]*PyTorch: Get Started [https://pytorch.org/get-started/locally/]