Detect.py on screen input

[gpierard]

I am using the following in order to grab screenshots from an open application in realtime. How can I run detect.py on screen or scr without saving to disk and running os.system(python detect.py ...) for every frame? Thanks

hwnd = win32gui.FindWindow(None, 'Calculator')
left_x, top_y, right_x, bottom_y = win32gui.GetWindowRect(hwnd)
bbox = {'top': top_y, 'left': left_x, 'width': right_x-left_x, 'height':bottom_y-top_y }
sct.grab(bbox)
scr = np.array(screen)
cv2.imshow('window', scr)

(PS more details on my SO question)

[glenn-jocher]

For standalone inference in 3rd party projects or repos importing your model into the python workspace with PyTorch Hub is the recommended method. See YOLOv5 PyTorch Hub tutorial here, specifically the section on loading custom models.
https://github.com/ultralytics/yolov5#tutorials

Custom Models

This example loads a custom 20-class VOC-trained YOLOv5s model 'yolov5s_voc_best.pt' with PyTorch Hub.

import torch

model = torch.hub.load('ultralytics/yolov5', 'custom', path_or_model='yolov5s_voc_best.pt')
model = model.autoshape()  # for PIL/cv2/np inputs and NMS

Then once the model is loaded:

from PIL import Image

# Images
img1 = Image.open('zidane.jpg')
img2 = Image.open('bus.jpg')
imgs = [img1, img2]  # batched list of images

# Inference
result = model(imgs, size=640)  # includes NMS
result.print()

[gpierard]

Thanks for your reply. When evaluating with PyTorch hub, the performance of detection drastically decreases. Might this be due to the fact that result.show() shows an image with different color channel than the actual image?

Do you know how I can display the result model image with bounding boxes in a single window like with cv2.imshow() (result.show opens a new process instance for each frame) ? How can I save this result image and find more documentation on how to interact with model in general?

screen = sct.grab(bbox)
scr = np.array(screen)    
result = model(scr, size=400)  # includes NMS
result.save("test.png") #this gives a TypeError: save() takes 1 positional argument but 2 were given
scr = cv2.imread("test.png")

Cheers

PS: again, more details on a new SO question

[hhankj2u]

@gpierard
You can try this:

result.display(render=True)
cv2.imshow('', result.imgs[0])

Source code:

yolov5/models/common.py

Lines 340 to 341 in e8c5237

	if render:
	self.imgs[i] = np.asarray(im)

[glenn-jocher]

@gpierard results.save() accepts no arguments. See tutorial for proper usage, or simply view the source:

yolov5/models/common.py

Lines 233 to 235 in 49abc72

	def save(self):
	self.display(save=True) # save results

[gpierard]

Okay, but do you know how I can save the result image to a specific file, and how can I use this image as an input to cv2.imshow without saving it? thanks

EDIT result.display(save=True, show=False, pprint= False) saves a jpg image, but again the wrong color channel seems to be used?

[glenn-jocher]

@gpierard results.show() will show images with predictions. See hub for details:
https://pytorch.org/hub/ultralytics_yolov5/

[glenn-jocher]

EDIT result.display(save=True, show=False, pprint= False) saves a jpg image, but again the wrong color channel seems to be used?

If you spot a bug please submit a bug report using the bug report template with code to reproduce. Thank you!

[gpierard]

The following worked: result = model(cv2.cvtColor(scr, cv2.COLOR_BGR2RGB), size=400)

Cheers

[glenn-jocher]

@gpierard YOLOv5 models require RGB inputs. The tutorial shows proper usage on cv2 inputs, suggest you review the tutorial, or even the hub webpage:
https://pytorch.org/hub/ultralytics_yolov5/

[gpierard]

Yes, I saw. Thought this might be useful anyway because the proposed method img2 = cv2.imread('bus.jpg')[:, :, ::-1] # OpenCV image (BGR to RGB) does not seem to correctly convert to RGB with my particular use case e.g. model(scr[:, :, ::-1]) does not correctly convert to RGB.

With images grabbed using screen = sct.grab(bbox), what works for me is model(cv2.cvtColor(np.array(screen), cv2.COLOR_BGR2RGB)

[glenn-jocher]

Oh that’s odd. Can you raise a bug report with example image and code to reproduce your user?

On Fri, Dec 18, 2020 at 1:47 PM gpierard ***@***.***> wrote: Yes, I saw. Thought this might be useful anyway because the proposed method img2 = cv2.imread('bus.jpg')[:, :, ::-1] # OpenCV image (BGR to RGB) does not work with my particular use case. — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#1697 (reply in thread)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AGMXEGJZBHZH55SRIO2636TSVPEVNANCNFSM4U46EWVA> .

-- <https://www.ultralytics.com/> *Glenn Jocher*Founder & CEO, Ultralytics LLC +1 301 237 6695 <https://www.facebook.com/ultralytics> <https://www.twitter.com/ultralytics> <https://www.youtube.com/ultralytics> <https://www.github.com/ultralytics> <https://www.linkedin.com/company/ultralytics> <https://www.instagram.com/ultralytics> <https://contact.ultralytics.com/>

[gpierard]

sure, see here

Cheers

[glenn-jocher]

Thanks! Will check it out.

On Fri, Dec 18, 2020 at 5:23 PM gpierard ***@***.***> wrote: sure, see here <#1735 (comment)> Cheers — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#1697 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AGMXEGM7O3ZOTBEWBJHSQOLSVP6BTANCNFSM4U46EWVA> .

-- <https://www.ultralytics.com/> *Glenn Jocher*Founder & CEO, Ultralytics LLC +1 301 237 6695 <https://www.facebook.com/ultralytics> <https://www.twitter.com/ultralytics> <https://www.youtube.com/ultralytics> <https://www.github.com/ultralytics> <https://www.linkedin.com/company/ultralytics> <https://www.instagram.com/ultralytics> <https://contact.ultralytics.com/>

[iam-dante]

How to edit the cv2 window when open the webcam, need to resize the --source 0 (webcam) window

[glenn-jocher]

@iam-dante cv2 imshow window is a function of the image size I believe. Check cv2 docs.

[iam-dante]

Am talking about the window that opens when using webcam as an input

[wisplite]

I’m quite late to this discussion, but you can use the cv2.resize function, and just display the resized frame with cv2.imshow().

[wb-08]

Try:
```

result` = model(screen, size=640)
cv2.imshow('Screen', result.render()[0])

[BHUVAN-RJ]

@glenn-jocher I wanted to access results to check whether there is some detection (lets say fire). If my model has detected fire I want to execute some function. How can i do this?

[glenn-jocher]

👋 Hello! Thanks for asking about handling inference results. YOLOv5 🚀 PyTorch Hub models allow for simple model loading and inference in a pure python environment without using detect.py.

Simple Inference Example

This example loads a pretrained YOLOv5s model from PyTorch Hub as model and passes an image for inference. 'yolov5s' is the YOLOv5 'small' model. For details on all available models please see the README. Custom models can also be loaded, including custom trained PyTorch models and their exported variants, i.e. ONNX, TensorRT, TensorFlow, OpenVINO YOLOv5 models.

import torch

# Model
model = torch.hub.load('ultralytics/yolov5', 'yolov5s')  # yolov5n - yolov5x6 official model
#                                            'custom', 'path/to/best.pt')  # custom model

# Images
im = 'https://ultralytics.com/images/zidane.jpg'  # or file, Path, URL, PIL, OpenCV, numpy, list

# Inference
results = model(im)

# Results
results.print()  # or .show(), .save(), .crop(), .pandas(), etc.
results.xyxy[0]  # im predictions (tensor)

results.pandas().xyxy[0]  # im predictions (pandas)
#      xmin    ymin    xmax   ymax  confidence  class    name
# 0  749.50   43.50  1148.0  704.5    0.874023      0  person
# 2  114.75  195.75  1095.0  708.0    0.624512      0  person
# 3  986.00  304.00  1028.0  420.0    0.286865     27     tie

results.pandas().xyxy[0].value_counts('name')  # class counts (pandas)
# person    2
# tie       1

See YOLOv5 PyTorch Hub Tutorial for details.

Good luck 🍀 and let us know if you have any other questions!

[BHUVAN-RJ]

Thanks a lot @glenn-jocher , made my work a lot easier. Thanks a lot.