== Parallelization Methods Data Parallelism ==  This section details a way to parallelize your NN .  As image recognition is graphical in nature, multiple GPUs are the best way to parallelize dataset training. <code>DataParallel</code> is a single-machine parallel model, that uses multiple GPUs [https://pytorch.org/tutorials/beginner/blitz/data_parallel_tutorial.html]. It is more convenient than a multi-machine, distributed training model. You can easily put your model on a GPU by writing:  device = torch.device("cuda:0") model.to(device) Then, you can copy all your tensors to the GPU:  mytensor = my_tensor.to(device) However, PyTorch will only use one GPU by default. In order to run on multiple GPUs you need to use <code>DataParallel</code>:  model = nn.DataParallel(model) ==== Imports and Parameters ==== Import the following modules and define your parameters:  import torch import torch.nn as nn from torch.utils.data import Dataset, DataLoader  # Parameters and DataLoaders input_size = 5 output_size = 2  batch_size = 30 data_size = 100 Device:  device = torch.device("cuda:0" if torch.cuda.is_available() else "cpu") ==== Dummy DataSet ==== You can make a random dummy dataset:  class RandomDataset(Dataset):  def __init__(self, size, length): self.len = length self.data = torch.randn(length, size)  def __getitem__(self, index): return self.data[index]  def __len__(self): return self.len  rand_loader = DataLoader(dataset=RandomDataset(input_size, data_size), batch_size=batch_size, shuffle=True) ==== Simple Model ==== Here is a simple linear model definition, but <code>DataParallel</code> can be used any model (CNN, RNN, etc).  class Model(nn.Module): # Our model  def __init__(self, input_size, output_size): super(Model, self).__init__() self.fc = nn.Linear(input_size, output_size)  def forward(self, input): output = self.fc(input) print("\tIn Model: input size", input.size(), "output size", output.size())  return output ==== Create Model and DataParallel ==== Now that everything is defined, we need create an instance of the model and check if we have multiple GPUs.

This section details the ways to parallelize your NN model = Model(input_size, output_size) if torch.cuda.device_count() > 1: print("Let's use", torch.cuda.device_count(), "GPUs!") # dim = 0 [30, xxx] -> [10, ...], [10, ... As image recognition is graphical in nature], [10, multiple ...] on 3 GPUs are model = nn.DataParallel(model) # wrap model using nn.DataParallel  model.to(device) ==== Run the best way Model ==== The print statement will let us see the input and output sizes.  for data in rand_loader: input = data.to parallelize dataset training(device) output = model(input) print("Outside: input size", input.size(), "output_size", output.size()) === Single-Machine Model = Results ==== If you have no GPU or only one GPU, the input and output size will match the batch size, so no parallelization. But if you have 2 GPUs, you'll get these results:

The Single-Machine # on 2 GPUs Let's use 2 GPUs! In Model: input size torch.Size([15, 5]) output size torch.Size([15, 2]) In Model is a parallel model that is done on one machine: input size torch.Size([15, 5]) output size torch.Size([15, 2]) Outside: input size torch.Size([30, 5]) output_size torch.Size([30, 2]) In Model: input size torch.Size([15, 5]) output size torch.Size([15, 2]) In Model: input size torch.Size([15, 5]) output size torch.Size([15, 2]) Outside: input size torch.Size([30, 5]) output_size torch.Size([30, 2]) In Model: input size torch.Size([15, 5]) output size torch.Size([15, 2]) In Model: input size torch.Size([15, 5]) output size torch.Size([15, 2]) Outside: input size torch.Size([30, 5]) output_size torch.Size([30, 2]) In Model: input size torch.Size([5, 5]) output size torch.Size([5, 2]) In Model: input size torch.Size([5, 5]) output size torch.Size([5, with multiple GPUs2]) Outside: input size torch. It is more convenient than a distributed training model with multiple machinesSize([10, 5]) output_size torch.Size([10, 2])

 The code is very similar to a single GPU implementation, except for the ''.to('cuda:x')'' calls, where ''cuda:0'' and ''cuda:1'' are each their own GPU[https://pytorch.org/tutorials/intermediate/model_parallel_tutorial.html]. 

 The backward() and torch.optim will automatically take care of gradients as if the model is on one GPU. You only need to make sure that the labels are on the same device as the outputs when calling the loss function[https://pytorch.org/tutorials/intermediate/model_parallel_tutorial.html].

*[1] Khan, Faisa. “Infographics Digest - Vol. 3.” Medium, [https://medium.com/datadriveninvestor/infographics-digest-vol-3-da67e69d71ce]*[2][5][8] “ANN vs CNN vs RNN | Types of Neural Networks.” Analytics Vidhya, 17 Feb. 2020, [https://www.analyticsvidhya.com/blog/2020/02/cnn-vs-rnn-vs-mlp-analyzing-3-types-of-neural-networks-in-deep-learning/.]*[3][4][6] 3Blue1Brown. But What Is a Neural Network? | Deep Learning, Chapter 1. 2017. YouTube, [https://www.youtube.com/watch?v=aircAruvnKk.]*[7] What Is Backpropagation Really Doing? | Deep Learning, Chapter 3. 2017. YouTube, [https://www.youtube.com/watch?v=Ilg3gGewQ5U&list=PLZHQObOWTQDNU6R1_67000Dx_ZCJB-3pi&index=3.]*PyTorch Tutorials: beginner Data Parallel [https://pytorch.org/tutorials/beginner/blitz/data_parallel_tutorial.html]*PyTorch Tutorials: Intermediate Model Parallel [https://pytorch.org/tutorials/intermediate/model_parallel_tutorial.html]*Building our Neural Network - Deep Learning and Neural Networks with Python and Pytorch p.3, YouTube, [https://www.youtube.com/watch?v=ixathu7U-LQ] *Training Model - Deep Learning and Neural Networks with Python and Pytorch p.4, YouTube, [https://www.youtube.com/watch?v=9j-_dOze4IM]*Deep Learning with PyTorch: Building a Simple Neural Network| packtpub.com, YouTube, [https://www.youtube.com/watch?v=VZyTt1FvmfU&list=LL&index=4]*Github PyTorch Neural Network Module Implementation [https://github.com/pytorch/pytorch/tree/master/torch/nn/modules]*Project Jupyter Official Documentation [https://jupyter.org/documentation]*PyTorch: Get Started [https://pytorch.org/get-started/locally/]