Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Grad-Cam for yolov5-5.0 #5863

Closed
2 tasks done
xiaowk5516 opened this issue Dec 3, 2021 · 81 comments
Closed
2 tasks done

Grad-Cam for yolov5-5.0 #5863

xiaowk5516 opened this issue Dec 3, 2021 · 81 comments
Labels
enhancement New feature or request Stale

Comments

@xiaowk5516
Copy link
Contributor

Search before asking

  • I have searched the YOLOv5 issues and found no similar feature requests.

Description

I visualize where the model concerns using grad-cam.
detection results:
zidane
layer 17
17
layer 20
20
layer 23
23

detection results:
bus
layer 17
17
layer 20
20
layer 23
23

my code:
in detect.py
1.add a function

def cam_show_img(img, feature_map, grads, out_name):
    H, W, _ = img.shape
    cam = np.zeros(feature_map.shape[1:], dtype=np.float32)		
    grads = grads.reshape([grads.shape[0],-1])					
    weights = np.mean(grads, axis=1)							
    for i, w in enumerate(weights):
        cam += w * feature_map[i, :, :]							
    cam = np.maximum(cam, 0)
    cam = cam / cam.max()
    cam = cv2.resize(cam, (W, H))

    heatmap = cv2.applyColorMap(np.uint8(255 * cam), cv2.COLORMAP_JET)
    cam_img = 0.3 * heatmap + 0.7 * img

    cv2.imwrite(out_name, cam_img)

change:

yolov5/detect.py

Lines 59 to 61 in f5b8f7d

# Run inference
if device.type != 'cpu':
model(torch.zeros(1, 3, imgsz, imgsz).to(device).type_as(next(model.parameters()))) # run once

to:

if visualize:
    # require grad
    for k, v in model.named_parameters():
        v.requires_grad = True  # train all layers
    compute_loss = ComputeLoss(model)

# Run inference
if pt and device.type != 'cpu':
    model(torch.zeros(1, 3, *imgsz).to(device).type_as(next(model.parameters())))  # run once

change:

yolov5/detect.py

Lines 71 to 72 in f5b8f7d

t1 = time_synchronized()
pred = model(img, augment=opt.augment)[0]

to

visualize = increment_path(save_dir / Path(path).stem, mkdir=True) if visualize else False
# pred = model(img, augment=augment, visualize=visualize)[0]

if visualize:
    # grad-cam
    pred = model(img, augment=augment, visualize=visualize)
    
    model.zero_grad()
    targets = torch.zeros(2, 6)
    loss, loss_items = compute_loss(pred[1], targets.to(device))
    loss.requires_grad_(True)
    loss.backward()

    _grads = model.grads_list
    _grads.reverse()
    _features = model.features_list

    # for g, f in zip(_grads, _features):
    #     print('grad', type(g), g.shape)
    #     print('feature', type(f), f.shape)
    
    for i in [17, 20, 23]:
        out_name = str(save_dir / f"{i}.jpg")
        cam_show_img(im0s, _features[i].cpu().detach().numpy()[0], _grads[i].cpu().detach().numpy()[0], out_name)

    pred = pred[0]
else:
    pred = model(img, augment=augment, visualize=visualize)[0]

in yolo.py
1.add grad_list and feature_list to store grad and feature

yolov5/models/yolo.py

Lines 66 to 67 in f5b8f7d

class Model(nn.Module):
def __init__(self, cfg='yolov5s.yaml', ch=3, nc=None, anchors=None): # model, input channels, number of classes

class Model(nn.Module):

    grads_list = []
    features_list = []

    def __init__(self, cfg='yolov5s.yaml', ch=3, nc=None, anchors=None):  # model, input channels, number of classes

2.add hook function in class Model

    def save_gradient(self, grad):
        # print('grad shape:', grad.shape)
        self.grads_list.append(grad)

3.store grad and feature in Model.forward_once()

            y.append(x if m.i in self.save else None)  # save output
            
            # save grad
            if visualize:
                if 'Detect' not in m.type:
                    x.register_hook(self.save_gradient)
                    self.features_list.append(x)
                else:
                    if isinstance(x, list):
                        for _x in x:
                            _x.register_hook(self.save_gradient)
                            self.features_list.append(_x)
                    elif isinstance(x, tuple):
                        for _x in x[1]:
                            _x.register_hook(self.save_gradient)
                            self.features_list.append(_x)

            if visualize:
                feature_visualization(x, m.type, m.i, save_dir=visualize)

Use case

a piece of code about Grad-Cam, which can visualize where the model concerns.

Additional

No response

Are you willing to submit a PR?

  • Yes I'd like to help by submitting a PR!
@xiaowk5516 xiaowk5516 added the enhancement New feature or request label Dec 3, 2021
@sakurasakura1996
Copy link

@xiaowk5516 hi, Thank you for your work, I would like to ask if you can show ComputeLoss function in your change code

@xiaowk5516
Copy link
Contributor Author

you can find ComputeLoss in loss.py.(I remove the bbox loss.)

@sakurasakura1996
Copy link

Yes, I found ComputeLoss, but I encountered another error:
RuntimeError: cannot register a hook on a tensor that doesn't require gradient. It occurs at
if 'Detect' not in m.type: x.register_hook(self.save_gradient) self.features_list.append(x)

@1160404405
Copy link

Excuse me, could you please tell me what should be changed in ComputeLoss,Because the direct call will report an error。There is no hyp parameter

@xiaowk5516
Copy link
Contributor Author

Yes, I found ComputeLoss, but I encountered another error: RuntimeError: cannot register a hook on a tensor that doesn't require gradient. It occurs at if 'Detect' not in m.type: x.register_hook(self.save_gradient) self.features_list.append(x)

comment out @torch.grad()

@xiaowk5516
Copy link
Contributor Author

Excuse me, could you please tell me what should be changed in ComputeLoss,Because the direct call will report an error。There is no hyp parameter

just set the bbox loss to 0.

@1160404405
Copy link

sorry,I don't understand what you mean, can you publish the code of your ComputeLoss

@xiaowk5516
Copy link
Contributor Author

xiaowk5516 commented Dec 8, 2021

I have only one class.(you can use the origin ComputeLoss and dont change anything)
Just for yolov5-5.0.

lbox *= self.hyp['box']
        lobj *= self.hyp['obj']
        lcls *= self.hyp['cls']
        bs = tobj.shape[0]  # batch size

        # grad_cam = True
        # if grad_cam:
        #     return lobj * bs, torch.cat((lbox, lobj, lcls)).detach()

        return (lbox + lobj + lcls) * bs, torch.cat((lbox, lobj, lcls)).detach()

@xiaowk5516
Copy link
Contributor Author

xiaowk5516 commented Dec 8, 2021

Thank you.But my problem is that when I run detect.py, this code in ComputeLoss is wrong -- (h = model.hyp),because I didn't give this parameter in detect.py. Maybe my version is different from yours. Could you please email me your project code.I would be most grateful. I have been working on this feature for a long time and I am very tired. ------------------ 原始邮件 ------------------ 发件人: "ultralytics/yolov5" @.>; 发送时间: 2021年12月8日(星期三) 下午3:26 @.>; @.@.>; 主题: Re: [ultralytics/yolov5] Grad-Cam for yolov5-5.0 (Issue #5863) I have only one class.(you can use the origin ComputeLoss) Just for yolov5-5.0. lbox *= self.hyp['box'] lobj *= self.hyp['obj'] lcls *= self.hyp['cls'] bs = tobj.shape[0] # batch size # grad_cam = True # if grad_cam: # return lobj * bs, torch.cat((lbox, lobj, lcls)).detach() return (lbox + lobj + lcls) * bs, torch.cat((lbox, lobj, lcls)).detach() — You are receiving this because you commented. Reply to this email directly, view it on GitHub, or unsubscribe. Triage notifications on the go with GitHub Mobile for iOS or Android.

sorry..., I will submit a PR later.
you can git clone yolov5-5.0 and any weights for yolov5-5.0. try to add this codes.(yolov5-6.0 has DetectMultiBackend layer, it is different from yolov5-5.0, cannot use it directly)

@1160404405
Copy link

1160404405 commented Dec 8, 2021 via email

@1160404405
Copy link

I'm using version 5.0 and I have the same problem with the man up there:cannot register a hook on a tensor that doesn't require gradient.Your explanation is comment out @torch.grad().But this code does not exist in version 5.0, it only exists in version 6.0.

@xiaowk5516
Copy link
Contributor Author

I'm using version 5.0 and I have the same problem with the man up there:cannot register a hook on a tensor that doesn't require gradient.Your explanation is comment out @torch.grad().But this code does not exist in version 5.0, it only exists in version 6.0.
comment out torch.no_grad()

yolov5/detect.py

Lines 172 to 178 in f5b8f7d

with torch.no_grad():
if opt.update: # update all models (to fix SourceChangeWarning)
for opt.weights in ['yolov5s.pt', 'yolov5m.pt', 'yolov5l.pt', 'yolov5x.pt']:
detect()
strip_optimizer(opt.weights)
else:
detect()

@1160404405
Copy link

Thank you very much for your patient guidance.I hope I can help you some day.

@xiaowk5516
Copy link
Contributor Author

Thank you very much for your patient guidance.I hope I can help you some day.

😄

@MrReeed
Copy link

MrReeed commented Dec 13, 2021

Hello, thank you very much for writing this article. I have a question: I implemented it exactly according to your method. Although there were no problems with the above friends, the program said "device = next(model.parameters()).device # get model device " of "loss.py" does not work when executing "compute_loss = ComputeLoss(model)" of "detect.py" , and then an error StopIteration is reported. Do you know how to solve this situation? thank you😫

@1160404405
Copy link

@MrReeed Are you using version 6 ?

@MrReeed
Copy link

MrReeed commented Dec 13, 2021

@MrReeed Are you using version 6 ?

yeah, so, will your version 6 work properly? Neither my version 5 nor version 6 can run successfully😭
Now there is another such mistake:
model(torch.zeros(1, 3, *imgsz).to(device).type_as(next(model.parameters()))) # run once
TypeError: Value after * must be an iterable, not int I think there's no problem. I'm stuck now

@xiaowk5516
Copy link
Contributor Author

what`s the type of imgsz? imgsz should be (640, 640) or somethings like this.

@MrReeed
Copy link

MrReeed commented Dec 13, 2021

@xiaowk5516

what`s the type of imgsz? imgsz should be (640, 640) or somethings like this.

Yes, except for the code you specified changed, the rest of the code is the original default V5 version (v6 was also tried)

@xiaowk5516
Copy link
Contributor Author

xiaowk5516 commented Dec 14, 2021

@xiaowk5516

what`s the type of imgsz? imgsz should be (640, 640) or somethings like this.

Yes, except for the code you specified changed, the rest of the code is the original default V5 version (v6 was also tried)

the error message shows that imgsz is int not iterable, how about checking your code?

@1160404405
Copy link

@xiaowk5516 model(torch.zeros(1, 3, *imgsz).to(device).type_as(next(model.parameters())))model(torch.zeros(1, 3, *imgsz).to(device).type_as(next(model.parameters()))),Only available in version 6,

@xiaowk5516
Copy link
Contributor Author

@xiaowk5516 model(torch.zeros(1, 3, *imgsz).to(device).type_as(next(model.parameters())))model(torch.zeros(1, 3, *imgsz).to(device).type_as(next(model.parameters()))),Only available in version 6,

yes, indeed

@zxq309
Copy link

zxq309 commented Dec 15, 2021

I encountered this error what is going on? please help me
Traceback (most recent call last):
File "detect-grad-cam.py", line 240, in
detect(opt=opt)
File "detect-grad-cam.py", line 75, in detect
if visualize:
UnboundLocalError: local variable 'visualize' referenced before assignment

@1160404405
Copy link

@XiangqingZhang need you add variable 'visualize' in the arguments to the function

@MrReeed
Copy link

MrReeed commented Dec 15, 2021

I encountered this error what is going on? please help me Traceback (most recent call last): File "detect-grad-cam.py", line 240, in detect(opt=opt) File "detect-grad-cam.py", line 75, in detect if visualize: UnboundLocalError: local variable 'visualize' referenced before assignment

u can set visualize = True before the code

@zxq309
Copy link

zxq309 commented Dec 15, 2021

Thank you for your prompt reply, but I have another problem?
Traceback (most recent call last):
File "detect-grad-cam.py", line 241, in
detect(opt=opt)
File "detect-grad-cam.py", line 80, in detect
compute_loss = ComputeLoss(model)
NameError: name 'ComputeLoss' is not defined

@O2andsAnoxia
Copy link

@xiaowk5516 你好,首先感谢你的工作,但是对于你的实现我有以下几点疑惑:
①请问能否贴下你的CoputeLoss的实现细节,做了哪些修改,如何将bbox的loss置零
②targets = torch.zeros(2, 6)有什么具体含义吗,如果仅仅这样作为target似乎并不能通过computeloss中的build_targets()
③你似乎只是对输出x作hook的注册,而不是对最后一个卷积层,因此并没有使用到register_forward_hook和register_backward_hook,这和grad-cam似乎不太相同

@xiaowk5516
Copy link
Contributor Author

xiaowk5516 commented Dec 17, 2021

@xiaowk5516 你好,首先感谢你的工作,但是对于你的实现我有以下几点疑惑: ①请问能否贴下你的CoputeLoss的实现细节,做了哪些修改,如何将bbox的loss置零 ②targets = torch.zeros(2, 6)有什么具体含义吗,如果仅仅这样作为target似乎并不能通过computeloss中的build_targets() ③你似乎只是对输出x作hook的注册,而不是对最后一个卷积层,因此并没有使用到register_forward_hook和register_backward_hook,这和grad-cam似乎不太相同

1.#5863 (comment)
2.version of your code
3. https://pytorch.org/docs/stable/generated/torch.Tensor.register_hook.html

@1160404405
Copy link

I have a question about whether your visualizations's results focusing on the results of center, even if the center of the target is hollow.

@wangt89123
Copy link

Hello, I am running on yolov5 version 6.0 and the error message is as follows:

Traceback (most recent call last):
File "detect.py", line 306, in
main(opt)
File "detect.py", line 301, in main
run(**vars(opt))
File "C:\yolov5\lib\site-packages\torch\autograd\grad_mode.py", line 28, in decorate_context
return func(*args, **kwargs)
File "detect.py", line 136, in run
compute_loss = ComputeLoss(model)
File "C:\yolov5-master\utils\loss.py", line 96, in init
h = model.hyp # hyperparameters
File "C:\yolov5\lib\site-packages\torch\nn\modules\module.py", line 1177, in getattr
raise AttributeError("'{}' object has no attribute '{}'".format(
AttributeError: 'DetectMultiBackend' object has no attribute 'hyp'

Is there any solution please? thank you very much!

@MrReeed
Copy link

MrReeed commented Mar 5, 2022 via email

@jojo0513
Copy link

Hi,
Is there any way I could run CAM for multiple images? It works fine on single image but when there are multiple the CAM seems to be the same for all images. My code is show as follow.
!python /content/yolov5/detect.py --weights /content/yolov5/YOLO_v5l.pt --img 448 --conf 0.5 --source /content/yolov5/Fish-V3-11/valid/0 --visualize

@1160404405
Copy link

Hi, Is there any way I could run CAM for multiple images? It works fine on single image but when there are multiple the CAM seems to be the same for all images. My code is show as follow. !python /content/yolov5/detect.py --weights /content/yolov5/YOLO_v5l.pt --img 448 --conf 0.5 --source /content/yolov5/Fish-V3-11/valid/0 --visualize

This is related to the name of the saved image file, because every CAM image have the same name, so it is overwritten, you can change the name regulation.

@hjfdsssdg
Copy link

Hello, may I ask that the representation of my thermal map on all images is the same

@MrReeed
Copy link

MrReeed commented Apr 15, 2022 via email

@hjfdsssdg
Copy link

yolov5 的 grad-cam
https://github.com/xiaowk5516/yolov5/tree/grad-cam

@xiaowk5516

我的结果很奇怪。所有图像都有相同的激活区域。我无法理解。请帮助我。我完全按照您提供的代码和我训练有素的权重文件。

Have you solved the problem that all diagrams are the same active region?

@xiaowk5516
Copy link
Contributor Author

yolov5 的 grad-cam
https://github.com/xiaowk5516/yolov5/tree/grad-cam

@xiaowk5516
我的结果很奇怪。所有图像都有相同的激活区域。我无法理解。请帮助我。我完全按照您提供的代码和我训练有素的权重文件。

Have you solved the problem that all diagrams are the same active region?

you can read this https://github.com/ultralytics/yolov5/issues/5863#issuecomment-1081437756.

@tienndm
Copy link

tienndm commented Apr 18, 2022

Have you ever testing on edge devices. I have implement it in edge devices and it compute to slow(about 1 minute per image)
. How can I make it run faster

@yjw-1998
Copy link

The problem I have now is RuntimeError: Can't call numpy() on Tensor that requires grad. Use tensor.detach().numpy() instead.

I have same problem as you. Have you solve it?

It's already giving you a prompt,Use tensor.detach().numpy() instead.

May i please ask how to choose the

I tried the output of other layers, but it did not enable the thermal map to cover the entire target as described in the Grad-Cam paper

I have a question about whether your visualizations's results focusing on the results of center, even if the center of the target is hollow.

for layer 17,20,23, it will. you can print other layers. wait for your results.

I tried the output of other layers, but it did not enable the thermal map to cover the entire target as described in the Grad-Cam paper

May i please ask how to designate the output heat-map of other layers?

@xiaowk5516
Copy link
Contributor Author

Have you ever testing on edge devices. I have implement it in edge devices and it compute to slow(about 1 minute per image) . How can I make it run faster

No. I dont test it on any edge devices. do you test the fps of yolov5 with commenting out gradcam. you can output time-comusing of all steps and optimez them.

@xiaowk5516
Copy link
Contributor Author

for i in [17, 20, 23]:
        out_name = str(save_dir / f"{i}.jpg")
        cam_show_img(im0s, _features[i].cpu().detach().numpy()[0], _grads[i].cpu().detach().numpy()[0], out_name)

which means output the 17, 20 and 23 layer. you can change it according the structure of yolov5.

@yjw-1998
Copy link

yjw-1998 commented Apr 29, 2022

for i in [17, 20, 23]:
        out_name = str(save_dir / f"{i}.jpg")
        cam_show_img(im0s, _features[i].cpu().detach().numpy()[0], _grads[i].cpu().detach().numpy()[0], out_name)

which means output the 17, 20 and 23 layer. you can change it according the structure of yolov5.
Thank u, I have got the wanted results. I find the code cannot be applied in yolov5-5.0 version, it can work in yolov5-6.0.

@github-actions
Copy link
Contributor

github-actions bot commented Jun 1, 2022

👋 Hello, this issue has been automatically marked as stale because it has not had recent activity. Please note it will be closed if no further activity occurs.

Access additional YOLOv5 🚀 resources:

Access additional Ultralytics ⚡ resources:

Feel free to inform us of any other issues you discover or feature requests that come to mind in the future. Pull Requests (PRs) are also always welcomed!

Thank you for your contributions to YOLOv5 🚀 and Vision AI ⭐!

@github-actions github-actions bot added the Stale label Jun 1, 2022
@MrReeed
Copy link

MrReeed commented Jun 1, 2022 via email

@github-actions github-actions bot removed the Stale label Jun 2, 2022
@github-actions
Copy link
Contributor

github-actions bot commented Jul 3, 2022

👋 Hello, this issue has been automatically marked as stale because it has not had recent activity. Please note it will be closed if no further activity occurs.

Access additional YOLOv5 🚀 resources:

Access additional Ultralytics ⚡ resources:

Feel free to inform us of any other issues you discover or feature requests that come to mind in the future. Pull Requests (PRs) are also always welcomed!

Thank you for your contributions to YOLOv5 🚀 and Vision AI ⭐!

@MrReeed
Copy link

MrReeed commented Jul 14, 2022 via email

1 similar comment
@MrReeed
Copy link

MrReeed commented Jul 30, 2022 via email

@noreenanwar
Copy link

grad-cam for yolov5

https://github.com/xiaowk5516/yolov5/tree/grad-cam

when I tried to run your code I did not get any heatmap

@MrReeed
Copy link

MrReeed commented Aug 24, 2022 via email

@MrReeed
Copy link

MrReeed commented Oct 11, 2022 via email

@glenn-jocher
Copy link
Member

@MrReeed hi there! I'm glad to hear that you received the email. If you have any further questions or need assistance, feel free to ask!

@MrReeed
Copy link

MrReeed commented Nov 15, 2023 via email

@glenn-jocher
Copy link
Member

@MrReeed 您好!如果您有任何疑问或需要帮助,请随时告诉我!🙂

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request Stale
Projects
None yet
Development

No branches or pull requests