== Parallelization Methods Data Parallelism ==  This section details a way to parallelize your NN .  As image recognition is graphical in nature, multiple GPUs are the best way to parallelize dataset training. <code>DataParallel</code> is a single-machine parallel model, that uses multiple GPUs [https://pytorch.org/tutorials/beginner/blitz/data_parallel_tutorial.html]. It is more convenient than a multi-machine, distributed training model. You can easily put your model on a GPU by writing:  device = torch.device("cuda:0") model.to(device) Then, you can copy all your tensors to the GPU:  mytensor = my_tensor.to(device) However, PyTorch will only use one GPU by default. In order to run on multiple GPUs you need to use <code>DataParallel</code>:  model = nn.DataParallel(model) ==== Imports and Parameters ==== Import the following modules and define your parameters:  import torch import torch.nn as nn from torch.utils.data import Dataset, DataLoader  # Parameters and DataLoaders input_size = 5 output_size = 2

This section details the ways to parallelize your NN batch_size = 30 data_size = 100 Device:  device = torch.device("cuda:0" if torch.cuda.is_available() else "cpu") ==== Dummy DataSet ==== You can make a random dummy dataset:  class RandomDataset(Dataset):  def __init__(self, size, length): self.len = length self.data = torch.randn(length, size)  def __getitem__(self, index): return self.data[index]  def __len__(self): return self.len  rand_loader = DataLoader(dataset=RandomDataset(input_size, data_size), batch_size=batch_size, shuffle=True) ==== Simple Model ==== Here is a simple linear model definition, but <code>DataParallel</code> can be used any model (CNN, RNN, etc).  class Model(nn.Module): # Our model  def __init__(self, input_size, output_size): super(Model, self).__init__() self.fc = nn.Linear(input_size, output_size)  def forward(self, input): output = self.fc(input) print("\tIn Model: input size", input.size(), "output size", output. As image recognition size())  return output ==== Create Model and DataParallel ==== Now that everything is graphical in naturedefined, we need create an instance of the model and check if we have multiple GPUs are .  model = Model(input_size, output_size) if torch.cuda.device_count() > 1: print("Let's use", torch.cuda.device_count(), "GPUs!") # dim = 0 [30, xxx] -> [10, ...], [10, ...], [10, ...] on 3 GPUs model = nn.DataParallel(model) # wrap model using nn.DataParallel  model.to(device) ==== Run the best way Model ==== The print statement will let us see the input and output sizes.  for data in rand_loader: input = data.to parallelize dataset training(device) output = model(input) print("Outside: input size", input.size(), "output_size", output.size()) === Single-Machine Model = Results ==== If you have no GPU or only one GPU, the input and output size will match the batch size, so no parallelization. But if you have 2 GPUs, you'll get these results:  # on 2 GPUs Let's use 2 GPUs! In Model: input size torch.Size([15, 5]) output size torch.Size([15, 2]) In Model: input size torch.Size([15, 5]) output size torch.Size([15, 2]) Outside: input size torch.Size([30, 5]) output_size torch.Size([30, 2]) In Model: input size torch.Size([15, 5]) output size torch.Size([15, 2]) In Model: input size torch.Size([15, 5]) output size torch.Size([15, 2]) Outside: input size torch.Size([30, 5]) output_size torch.Size([30, 2]) In Model: input size torch.Size([15, 5]) output size torch.Size([15, 2]) In Model: input size torch.Size([15, 5]) output size torch.Size([15, 2]) Outside: input size torch.Size([30, 5]) output_size torch.Size([30, 2]) In Model: input size torch.Size([5, 5]) output size torch.Size([5, 2]) In Model: input size torch.Size([5, 5]) output size torch.Size([5, 2]) Outside: input size torch.Size([10, 5]) output_size torch.Size([10, 2])

The Single-Machine Model is a parallel model that is done on one machineWith 2 GPUs, with multiple GPUs. It is more convenient than a distributed training model with multiple machinesthe input and output sizes are half of 30, 15.

==== Basic Usage =A More Intermediate Example ===

Here is a toy model that contains two linear layers. Each linear layer is designed to run a separate GPU[https://pytorch.org/tutorials/intermediate/model_parallel_tutorial.html].

The code is very similar to a single GPU implementation, except for the ''.to('cuda:x')'' calls, where ''cuda:0'' and ''cuda:1'' are each their own GPU[https://pytorch.org/tutorials/intermediate/model_parallel_tutorial.html].

The backward() and torch.optim will automatically take care of gradients as if the model is on one GPU. You only need to make sure that the labels are on the same device as the outputs when calling the loss function[https://pytorch.org/tutorials/intermediate/model_parallel_tutorial.html].

*[1] Khan, Faisa. “Infographics Digest - Vol. 3.” Medium, [https://medium.com/datadriveninvestor/infographics-digest-vol-3-da67e69d71ce]*[2][5][8] “ANN vs CNN vs RNN | Types of Neural Networks.” Analytics Vidhya, 17 Feb. 2020, [https://www.analyticsvidhya.com/blog/2020/02/cnn-vs-rnn-vs-mlp-analyzing-3-types-of-neural-networks-in-deep-learning/.]*[3][4][6] 3Blue1Brown. But What Is a Neural Network? | Deep Learning, Chapter 1. 2017. YouTube, [https://www.youtube.com/watch?v=aircAruvnKk.]*[7] What Is Backpropagation Really Doing? | Deep Learning, Chapter 3. 2017. YouTube, [https://www.youtube.com/watch?v=Ilg3gGewQ5U&list=PLZHQObOWTQDNU6R1_67000Dx_ZCJB-3pi&index=3.]*PyTorch Tutorials: beginner Data Parallel [https://pytorch.org/tutorials/beginner/blitz/data_parallel_tutorial.html]*PyTorch Tutorials: Intermediate Model Parallel [https://pytorch.org/tutorials/intermediate/model_parallel_tutorial.html]*Building our Neural Network - Deep Learning and Neural Networks with Python and Pytorch p.3, YouTube, [https://www.youtube.com/watch?v=ixathu7U-LQ] *Training Model - Deep Learning and Neural Networks with Python and Pytorch p.4, YouTube, [https://www.youtube.com/watch?v=9j-_dOze4IM]*Deep Learning with PyTorch: Building a Simple Neural Network| packtpub.com, YouTube, [https://www.youtube.com/watch?v=VZyTt1FvmfU&list=LL&index=4]*Github PyTorch Neural Network Module Implementation [https://github.com/pytorch/pytorch/tree/master/torch/nn/modules]*Project Jupyter Official Documentation [https://jupyter.org/documentation]*PyTorch: Get Started [https://pytorch.org/get-started/locally/]