Getting error while producing latency through yolo.py

[jaskiratsingh2000]

❔Question

Getting error while producing latency through yolo.py

Additional context

I want to view the latency output as discussed in issue #2898 but I feel like this is an issue that should go independently. I want to produce this output:

YOLOv5 🚀 v5.0-39-g1849916 torch 1.8.1+cu101 CUDA:0 (Tesla P100-PCIE-16GB, 16280.875MB)


                 from  n    params  module                                  arguments                     
  0                -1  1      3520  models.common.Focus                     [3, 32, 3]                    
  1                -1  1     18560  models.common.Conv                      [32, 64, 3, 2]                
  2                -1  1     18816  models.common.C3                        [64, 64, 1]                   
  3                -1  1     73984  models.common.Conv                      [64, 128, 3, 2]               
  4                -1  1    156928  models.common.C3                        [128, 128, 3]                 
  5                -1  1    295424  models.common.Conv                      [128, 256, 3, 2]              
  6                -1  1    625152  models.common.C3                        [256, 256, 3]                 
  7                -1  1   1180672  models.common.Conv                      [256, 512, 3, 2]              
  8                -1  1    656896  models.common.SPP                       [512, 512, [5, 9, 13]]        
  9                -1  1   1182720  models.common.C3                        [512, 512, 1, False]          
 10                -1  1    131584  models.common.Conv                      [512, 256, 1, 1]              
 11                -1  1         0  torch.nn.modules.upsampling.Upsample    [None, 2, 'nearest']          
 12           [-1, 6]  1         0  models.common.Concat                    [1]                           
 13                -1  1    361984  models.common.C3                        [512, 256, 1, False]          
 14                -1  1     33024  models.common.Conv                      [256, 128, 1, 1]              
 15                -1  1         0  torch.nn.modules.upsampling.Upsample    [None, 2, 'nearest']          
 16           [-1, 4]  1         0  models.common.Concat                    [1]                           
 17                -1  1     90880  models.common.C3                        [256, 128, 1, False]          
 18                -1  1    147712  models.common.Conv                      [128, 128, 3, 2]              
 19          [-1, 14]  1         0  models.common.Concat                    [1]                           
 20                -1  1    296448  models.common.C3                        [256, 256, 1, False]          
 21                -1  1    590336  models.common.Conv                      [256, 256, 3, 2]              
 22          [-1, 10]  1         0  models.common.Concat                    [1]                           
 23                -1  1   1182720  models.common.C3                        [512, 512, 1, False]          
 24      [17, 20, 23]  1    229245  Detect                                  [80, [[10, 13, 16, 30, 33, 23], [30, 61, 62, 45, 59, 119], [116, 90, 156, 198, 373, 326]], [128, 256, 512]]
Model Summary: 283 layers, 7276605 parameters, 7276605 gradients, 17.1 GFLOPS

 time (ms)     GFLOPS     params  module
      0.93       1.44       3520  models.common.Focus
      0.52       1.90      18560  models.common.Conv
      1.31       1.93      18816  models.common.C3
      0.38       1.89      73984  models.common.Conv
      1.58       4.02     156928  models.common.C3
      0.37       1.89     295424  models.common.Conv
      1.48       4.00     625152  models.common.C3
      0.51       1.89    1180672  models.common.Conv
      0.79       1.05     656896  models.common.SPP
      0.98       1.89    1182720  models.common.C3
      0.18       0.21     131584  models.common.Conv
      0.03       0.00          0  torch.nn.modules.upsampling.Upsample
      0.05       0.00          0  models.common.Concat
      0.87       2.32     361984  models.common.C3
      0.16       0.21      33024  models.common.Conv
      0.05       0.00          0  torch.nn.modules.upsampling.Upsample
      0.09       0.00          0  models.common.Concat
      0.81       2.33      90880  models.common.C3
      0.35       0.95     147712  models.common.Conv
      0.03       0.00          0  models.common.Concat
      0.80       1.90     296448  models.common.C3
      0.35       0.94     590336  models.common.Conv
      0.03       0.00          0  models.common.Concat
      1.04       1.89    1182720  models.common.C3
      0.69       1.47     229245  Detect
14.4 ms total

I was able to produce the parameter output by running the following commands

!git clone https://github.com/jaskiratsingh2000/yolov5.git  # clone repo
!pip install -r yolov5/requirements.txt  # install dependencies

%cd yolov5
!python models/yolo.py

@glenn-jocher mentioned that I can get the output of the latency of layer by layer which is the following

time (ms)     GFLOPS     params  module
      0.93       1.44       3520  models.common.Focus
      0.52       1.90      18560  models.common.Conv
      1.31       1.93      18816  models.common.C3
      0.38       1.89      73984  models.common.Conv
      1.58       4.02     156928  models.common.C3
      0.37       1.89     295424  models.common.Conv
      1.48       4.00     625152  models.common.C3
      0.51       1.89    1180672  models.common.Conv
      0.79       1.05     656896  models.common.SPP
      0.98       1.89    1182720  models.common.C3
      0.18       0.21     131584  models.common.Conv
      0.03       0.00          0  torch.nn.modules.upsampling.Upsample
      0.05       0.00          0  models.common.Concat
      0.87       2.32     361984  models.common.C3
      0.16       0.21      33024  models.common.Conv
      0.05       0.00          0  torch.nn.modules.upsampling.Upsample
      0.09       0.00          0  models.common.Concat
      0.81       2.33      90880  models.common.C3
      0.35       0.95     147712  models.common.Conv
      0.03       0.00          0  models.common.Concat
      0.80       1.90     296448  models.common.C3
      0.35       0.94     590336  models.common.Conv
      0.03       0.00          0  models.common.Concat
      1.04       1.89    1182720  models.common.C3
      0.69       1.47     229245  Detect
14.4 ms total

by running the model here #2898 (comment)
whereas, I am unable to do that because as I run this command to produce the above out I get the error.

command run:

img = torch.rand(8 if torch.cuda.is_available() else 1, 3, 320, 320).to(device) 
y = model(img, profile=True)

Error got:

---------------------------------------------------------------------------
NameError                                 Traceback (most recent call last)
<ipython-input-4-f21874f58c27> in <module>()
----> 1 img = torch.rand(8 if torch.cuda.is_available() else 1, 3, 320, 320).to(device)
      2 y = model(img, profile=True)

NameError: name 'device' is not defined

[jaskiratsingh2000]

@glenn-jocher Your response and a bit of help would really be appreciable. Can you please let me know?

[glenn-jocher]

@jaskiratsingh2000 I already answered this question previously in #2898 (comment)

[jaskiratsingh2000]

@glenn-jocher Things were not clear for me.
when I run the following command here https://colab.research.google.com/drive/1PRjPmQ8Nk7oM0KUcGfMBLVWSyNQ29lih?usp=sharing

You mentioned running this following command with profiling made to true:

img = torch.rand(8 if torch.cuda.is_available() else 1, 3, 320, 320).to(device) 
y = model(img, profile=True)

but I am getting this error:

NameError                                 Traceback (most recent call last)
<ipython-input-4-f21874f58c27> in <module>()
----> 1 img = torch.rand(8 if torch.cuda.is_available() else 1, 3, 320, 320).to(device)
      2 y = model(img, profile=True)

NameError: name 'device' is not defined

If I change the "device" to 'cuda' I still get lots of runtime errors. Can you please check and let me know exactly here? @glenn-jocher

[jaskiratsingh2000]

@glenn-jocher I tried creating the device by following the command:

device = torch.device("cuda:0")
and then creating the model:

model = torch.hub.load('ultralytics/yolov5', 'yolov5s', force_reload=True)

then on running the profiling commands I get the following out put and runtime errors.

time (ms)     GFLOPS     params  module
      3.71       1.44       3520  models.common.Focus
      2.87       1.90      18560  models.common.Conv
      7.84       1.93      18816  models.common.C3
      1.71       1.89      73984  models.common.Conv
      6.98       4.02     156928  models.common.C3
      1.42       1.89     295424  models.common.Conv
      4.65       4.00     625152  models.common.C3
      1.54       1.89    1180672  models.common.Conv
      2.15       1.05     656896  models.common.SPP
      2.54       1.89    1182720  models.common.C3
      0.29       0.21     131584  models.common.Conv
      0.10       0.00          0  torch.nn.modules.upsampling.Upsample
      0.29       0.00          0  models.common.Concat
      2.33       2.32     361984  models.common.C3
      0.29       0.21      33024  models.common.Conv
      0.15       0.00          0  torch.nn.modules.upsampling.Upsample
      0.25       0.00          0  models.common.Concat
      3.15       2.33      90880  models.common.C3
      0.73       0.95     147712  models.common.Conv
      0.16       0.00          0  models.common.Concat
      2.05       1.90     296448  models.common.C3
      0.92       0.94     590336  models.common.Conv
      0.07       0.00          0  models.common.Concat
      2.30       1.89    1182720  models.common.C3
---------------------------------------------------------------------------
RuntimeError                              Traceback (most recent call last)
<ipython-input-7-f21874f58c27> in <module>()
      1 img = torch.rand(8 if torch.cuda.is_available() else 1, 3, 320, 320).to(device)
----> 2 y = model(img, profile=True)

10 frames
/usr/local/lib/python3.7/dist-packages/torch/nn/modules/module.py in _call_impl(self, *input, **kwargs)
    887             result = self._slow_forward(*input, **kwargs)
    888         else:
--> 889             result = self.forward(*input, **kwargs)
    890         for hook in itertools.chain(
    891                 _global_forward_hooks.values(),

/usr/local/lib/python3.7/dist-packages/torch/autograd/grad_mode.py in decorate_context(*args, **kwargs)
     25         def decorate_context(*args, **kwargs):
     26             with self.__class__():
---> 27                 return func(*args, **kwargs)
     28         return cast(F, decorate_context)
     29 

/root/.cache/torch/hub/ultralytics_yolov5_master/models/common.py in forward(self, imgs, size, augment, profile)
    253         if isinstance(imgs, torch.Tensor):  # torch
    254             with amp.autocast(enabled=p.device.type != 'cpu'):
--> 255                 return self.model(imgs.to(p.device).type_as(p), augment, profile)  # inference
    256 
    257         # Pre-process

/usr/local/lib/python3.7/dist-packages/torch/nn/modules/module.py in _call_impl(self, *input, **kwargs)
    887             result = self._slow_forward(*input, **kwargs)
    888         else:
--> 889             result = self.forward(*input, **kwargs)
    890         for hook in itertools.chain(
    891                 _global_forward_hooks.values(),

/root/.cache/torch/hub/ultralytics_yolov5_master/models/yolo.py in forward(self, x, augment, profile)
    122             return torch.cat(y, 1), None  # augmented inference, train
    123         else:
--> 124             return self.forward_once(x, profile)  # single-scale inference, train
    125 
    126     def forward_once(self, x, profile=False):

/root/.cache/torch/hub/ultralytics_yolov5_master/models/yolo.py in forward_once(self, x, profile)
    134                 t = time_synchronized()
    135                 for _ in range(10):
--> 136                     _ = m(x)
    137                 dt.append((time_synchronized() - t) * 100)
    138                 if m == self.model[0]:

/usr/local/lib/python3.7/dist-packages/torch/nn/modules/module.py in _call_impl(self, *input, **kwargs)
    887             result = self._slow_forward(*input, **kwargs)
    888         else:
--> 889             result = self.forward(*input, **kwargs)
    890         for hook in itertools.chain(
    891                 _global_forward_hooks.values(),

/root/.cache/torch/hub/ultralytics_yolov5_master/models/yolo.py in forward(self, x)
     44         self.training |= self.export
     45         for i in range(self.nl):
---> 46             x[i] = self.m[i](x[i])  # conv
     47             bs, _, ny, nx = x[i].shape  # x(bs,255,20,20) to x(bs,3,20,20,85)
     48             x[i] = x[i].view(bs, self.na, self.no, ny, nx).permute(0, 1, 3, 4, 2).contiguous()

/usr/local/lib/python3.7/dist-packages/torch/nn/modules/module.py in _call_impl(self, *input, **kwargs)
    887             result = self._slow_forward(*input, **kwargs)
    888         else:
--> 889             result = self.forward(*input, **kwargs)
    890         for hook in itertools.chain(
    891                 _global_forward_hooks.values(),

/usr/local/lib/python3.7/dist-packages/torch/nn/modules/conv.py in forward(self, input)
    397 
    398     def forward(self, input: Tensor) -> Tensor:
--> 399         return self._conv_forward(input, self.weight, self.bias)
    400 
    401 class Conv3d(_ConvNd):

/usr/local/lib/python3.7/dist-packages/torch/nn/modules/conv.py in _conv_forward(self, input, weight, bias)
    394                             _pair(0), self.dilation, self.groups)
    395         return F.conv2d(input, weight, bias, self.stride,
--> 396                         self.padding, self.dilation, self.groups)
    397 
    398     def forward(self, input: Tensor) -> Tensor:

RuntimeError: Expected 4-dimensional input for 4-dimensional weight [255, 128, 1, 1], but got 5-dimensional input of size [8, 3, 40, 40, 85] instead

[jaskiratsingh2000]

Hey, @glenn-jocher Did you get a chance to check on the above? I am still not able to produce the output you mentioned. Please help me out in knowing about this. It would be highly appreciative of your response and help.
Thanks

[glenn-jocher]

@jaskiratsingh2000 my previous response answers your question: #2898 (comment)

[jaskiratsingh2000]

@glenn-jocher It does answer my question but things are not clear to me exactly what I have to do. What you mentioned in this comment I ran the model that way but I am getting the error as stated above that's why I am pinging you that I am stuck here please help me with that if possible. I would really be grateful to you.

@glenn-jocher you told me to run directly this and when I run that I get the above runtime error.

img = torch.rand(8 if torch.cuda.is_available() else 1, 3, 320, 320).to(device) 
y = model(img, profile=True)

[glenn-jocher]

If you follow #2898 (comment) and you are seeing errors then either you have modified your code, your code is out of date, or you have environment problems. See below.

👋 Hello, thank you for your interest in our work! This issue seems to lack the minimum requirements for a proper response, or is insufficiently detailed for us to help you. Please note that most technical problems are due to:

• Your modified or out-of-date code. If your issue is not reproducible in a new git clone version of this repo we can not debug it. Before going further run this code and verify your issue persists:

$ git clone https://github.com/ultralytics/yolov5 yolov5_new  # clone latest
$ cd yolov5_new
$ python detect.py  # verify detection

# CODE TO REPRODUCE YOUR ISSUE HERE

• Your custom data. If your issue is not reproducible in one of our 3 common datasets (COCO, COCO128, or VOC) we can not debug it. Visit our Custom Training Tutorial for guidelines on training your custom data. Examine train_batch0.jpg and test_batch0.jpg for a sanity check of your labels and images.

• Your environment. If your issue is not reproducible in one of the verified environments below we can not debug it. If you are running YOLOv5 locally, verify your environment meets all of the requirements.txt dependencies specified below. If in doubt, download Python 3.8.0 from https://www.python.org/, create a new venv, and install requirements.

If none of these apply to you, we suggest you close this issue and raise a new one using the 🐛 Bug Report template, providing screenshots and a minimum reproducible example of your issue. Thank you!

Requirements

Python 3.8 or later with all requirements.txt dependencies installed, including torch>=1.7. To install run:

$ pip install -r requirements.txt

Environments

YOLOv5 may be run in any of the following up-to-date verified environments (with all dependencies including CUDA/CUDNN, Python and PyTorch preinstalled):

• Google Colab and Kaggle notebooks with free GPU:
• Google Cloud Deep Learning VM. See GCP Quickstart Guide
• Amazon Deep Learning AMI. See AWS Quickstart Guide
• Docker Image. See Docker Quickstart Guide

Status

If this badge is green, all YOLOv5 GitHub Actions Continuous Integration (CI) tests are passing. These tests evaluate proper operation of basic YOLOv5 functionality, including training (train.py), testing (test.py), inference (detect.py) and export (export.py) on MacOS, Windows, and Ubuntu.

[jaskiratsingh2000]

I am not sure if you are able to get me. I ran the steps you mentioned got an error then raised above. I am just asking about the way you produce it. What all things and commands you ran

[glenn-jocher]

Follow #2898 (comment)

[pravastacaraka]

@jaskiratsingh2000 The variable 'device' is already listed in the main function on yolo.py.

yolov5/models/yolo.py

Lines 255 to 270 in 1849916

	if __name__ == '__main__':
	parser = argparse.ArgumentParser()
	parser.add_argument('--cfg', type=str, default='yolov5s.yaml', help='model.yaml')
	parser.add_argument('--device', default='', help='cuda device, i.e. 0 or 0,1,2,3 or cpu')
	opt = parser.parse_args()
	opt.cfg = check_file(opt.cfg) # check file
	set_logging()
	device = select_device(opt.device)
	# Create model
	model = Model(opt.cfg).to(device)
	model.train()
	# Profile
	# img = torch.rand(8 if torch.cuda.is_available() else 1, 3, 320, 320).to(device)
	# y = model(img, profile=True)

See line 262, so you don't need to write another like you doing in the #2951 (comment).
You just uncomment in line 269-270. And this line

yolov5/models/yolo.py

Line 41 in 1849916

# x = x.copy() # for profiling

Then you just run !python yolo.py in your colab.