guides/instance-segmentation-and-tracking/

[giscus[bot]]

guides/instance-segmentation-and-tracking/

Instance Segmentation with Object Tracking using Ultralytics YOLOv8

https://docs.ultralytics.com/guides/instance-segmentation-and-tracking/

[yakumoagito]

Can the number of pixels in the target mask be directly obtained after obtaining it

[pderrenger]

@yakumoagito hey there! It seems like there's a bit of confusion with accessing the .numpy() method directly after .cpu() on the Masks object. The correct way to convert the masks to a NumPy array involves a slightly different approach. Let's adjust your code snippet to correctly sum the pixels in each mask:

from ultralytics import YOLO

# Load your model and run inference to get results
model = YOLO('yolov8n-seg.pt')
results = model('./test.jpg')

# Check if masks are present
if results[0].masks is not None:
    # Convert masks to NumPy array
    masks_numpy = results[0].masks.cpu().numpy()  # This line seems to be the issue

    # Assuming masks_numpy is a list of numpy arrays now
    # You'll need to iterate through each mask to sum its pixels
    pixel_counts = [mask.sum() for mask in masks_numpy]

    # Now pixel_counts contains the number of pixels for each mask
    print(f"Pixel counts for each mask: {pixel_counts}")
else:
    print("No masks found in the results.")

If you're still encountering the same error, it's possible that the conversion method or the structure of results[0].masks is not as expected. Since results[0].masks should give you a list of masks, you might need to adjust how you're accessing and converting these masks to NumPy arrays.

If results[0].masks.cpu().numpy() is not working as expected, it might be due to a misunderstanding of the Masks object structure or a change in the API. In that case, double-checking the documentation or reaching out for further clarification would be the best course of action.

Hope this helps! If you have more questions or need further assistance, feel free to ask. 😊

[yakumoagito]

Thank you for the reply,I am trying to run this new code but this new code still has the same error

[pderrenger]

Hey there! 😊 It looks like you're encountering an issue with the .numpy() method on the Masks object. Based on the error message, it seems there's a misunderstanding in how to access the mask data for processing.

The Masks object doesn't directly support the .sum() method because it's not a NumPy array itself but contains methods to convert it to one. After calling .cpu().numpy() on the Masks object, you should have a NumPy array that you can work with. If you're still facing issues, it might be due to how you're handling the conversion.

Here's a quick fix to ensure you're accessing the masks correctly:

from ultralytics import YOLO

# Load your model and run inference to get results
model = YOLO('yolov8n-seg.pt')
results = model('./test.jpg')

# Check if masks are present
if results[0].masks is not None:
    # Convert masks to NumPy array correctly
    masks_numpy = results[0].masks.cpu().numpy()  # Make sure this conversion is correct

    # Assuming you now have a list of numpy arrays for each mask
    # You'll need to process each mask individually
    pixel_counts = [mask.sum() for mask in masks_numpy]  # This should work now

    print(f"Pixel counts for each mask: {pixel_counts}")
else:
    print("No masks found in the results.")

Make sure that results[0].masks.cpu().numpy() is correctly converting the masks to a NumPy array. If the error persists, please share more details about the structure of results[0].masks after calling .cpu().numpy() so we can help you better! 🚀

If you have further questions or need more assistance, feel free to reach out. Happy coding!

[yakumoagito]

Thank you for the reply,I changed my code and it ran successfully this time

masks = results[0].masks[0] # Get the first mask
masks_numpy = results[0].masks.cpu().numpy()
# print(masks_numpy.data)

    pixel_counts = masks_numpy.data.sum(axis=(1, 2))
    print(f"Target pixel count: {pixel_counts}")

[pderrenger]

@yakumoagito hey there! 🎉 I'm glad to hear your code is running successfully now. It looks like you've made great progress. Just a quick note, when working with the .sum() method on NumPy arrays derived from the Masks object, you're on the right track!

For future reference or if you need to tweak your approach, remember that after converting the masks to a NumPy array with .cpu().numpy(), you can directly perform operations like .sum() on that array. Your current method is spot on for summing up the pixels in each mask:

masks_numpy = results[0].masks.cpu().numpy()  # Convert masks to NumPy array
pixel_counts = masks_numpy.sum(axis=(1, 2))  # Sum the pixels in each mask
print(f"Target pixel count: {pixel_counts}")

If you have any more questions or dive into more complex tasks, don't hesitate to reach out. Happy coding! 😄👨‍💻

[teengineer]

Hello, can i use 'Instance Segmentation with Object Tracking' process for images like 'Instance Segmentation with Object Detection'? And if the answer is yes, how?

[glenn-jocher]

Hey there! 👋 Yes, you can definitely use the 'Instance Segmentation with Object Tracking' process for images, similar to how you would with 'Instance Segmentation with Object Detection'. The main difference is that tracking typically involves video frames where objects are tracked over time. However, for static images, you can still perform instance segmentation to identify and outline individual objects.

Here's a quick example using a static image:

from ultralytics import YOLO

# Load the segmentation model
model = YOLO("yolov8n-seg.pt")

# Run instance segmentation on an image
results = model('path/to/your/image.jpg')

# Visualize the segmentation
annotated_image = results[0].plot()

This will give you the segmented output for your image. If you have any more questions or need further assistance, feel free to ask! 😊

[Elvis-Kiilu2]

Hello, how do you implement instance segmentation with object tracking on a custom dataset? I have done instance segmentation on a custom dataset without tracking but I am having trouble doing it with tracking.:
%cd {HOME}

!yolo task=segment mode=train model=yolov8m-seg.pt data={dataset.location}/data.yaml epochs=3 imgsz=1024

[glenn-jocher]

@Elvis-Kiilu2 hello! 😊 For your scenario where you're combining instance segmentation and object tracking, you typically only need to export the segmentation model since the tracking is handled at runtime using the segmentation results.

Here's a concise approach:

1. Export your segmentation model (which includes detection capabilities) using the format that suits your deployment environment. For example, if deploying in a web environment, TensorFlow.js might be appropriate:

   yolo export model=path/to/your/seg_model.pt format=tfjs

2. Integrate the exported model into your application. The tracking logic, such as using ByteTrack, can be implemented in the application layer where the video is processed.

This way, you only manage one model export, and all tracking logic is applied dynamically to the video input in your application. If you need further assistance or have more questions, feel free to ask! 🚀

[Elvis-Kiilu2]

Hey Glen! 😂 First off, big thanks for the reply! I really appreciate the help, even if it did arrive fashionably late – must've been traveling at the speed of snail-mail. But seriously, thank you for the insightful recommendations.

I actually did what you suggested before your advice arrived the only difference is that instead of using TensorFlow.js, I went the Flask route.

Currently, I'm working on reading text and performing OCR from the detected boxes—specifically, reading license plate numbers from detected license plates. That probably means I have to retrain a detection-only model instead of a segmentation one. Or can I do it with the segmentation model? Not too sure about that. Any thoughts?

After reading the license plate numbers, I plan to add them to a database. Any tips on the best way to approach this?

Thanks again, @glenn-jocher

[glenn-jocher]

Hey @Elvis-Kiilu2! 😄 Thanks for the kind words and for diving into the OCR for license plates—that's an exciting application!

For reading text from detected boxes, your segmentation model can indeed be helpful, especially if it's accurately segmenting the license plates. You can use the segmented plates as inputs to an OCR system to extract the text. This way, you don't necessarily need to switch to a detection-only model unless you find the segmentation model isn't focusing well on the plates.

As for adding the read license plate numbers to a database, here's a simple approach:

• Use an OCR library like Tesseract to extract text from the segmented plates.
• Connect to your database (e.g., SQLite for simplicity or PostgreSQL for more robust applications) using an ORM like SQLAlchemy or a direct database driver.
• Insert the extracted plate numbers into the database with appropriate error handling and data validation to ensure data integrity.

This setup should allow you to efficiently store and manage the data extracted from your video inputs. If you need more detailed code examples or run into any specific issues, feel free to ask. Happy coding! 🚀

[Elvis-Kiilu2]

Thanks @glenn-jocher .As always very helpful. Tweets on the Ultralytics X page are also really insightful. Happy coding! 🚀

[glenn-jocher]

Thanks for the shout-out, @Elvis-Kiilu2! 🚀 Glad to hear the tweets and support have been helpful. If you dive deeper into instance segmentation or have any more questions, just drop them in the discussion section. Happy coding! 😊

[Anandukc]

Hi, how can I do object counting inn and out with instance segmentation tracking.

[glenn-jocher]

Hey there! 👋 For object counting with instance segmentation and tracking, you can use the YOLOv8 instance segmentation model to detect and segment objects, and then track their movement to count entries and exits.

Here's a simple example using the provided code snippets. You'll need to maintain counters for objects entering and exiting the scene:

import cv2
from ultralytics import YOLO

# Initialize counters
in_counter = 0
out_counter = 0

# Load the segmentation model with tracking
model = YOLO("yolov8n-seg.pt")
cap = cv2.VideoCapture("path/to/video/file.mp4")

while True:
    ret, frame = cap.read()
    if not ret:
        break

    # Perform tracking and get results
    results = model.track(frame, persist=True)
    
    # Update counters based on tracking IDs and your defined entry/exit zones
    # This part of the code depends on your specific use case

    # Visualize the frame with annotations
    annotated_frame = results[0].plot()
    cv2.imshow("Tracking", annotated_frame)

    if cv2.waitKey(1) & 0xFF == ord('q'):
        break

cap.release()
cv2.destroyAllWindows()

print(f"Objects entered: {in_counter}")
print(f"Objects exited: {out_counter}")

You'll need to define the logic for when an object is considered "in" or "out" based on your scene's layout. If you have any more questions or need further assistance, feel free to ask in the Ultralytics Issue Section. Happy coding! 😊

[Anandukc]

Can I use yolov8n-seg model directly in the object counting code of Ultralytics to do instance segmentation tracking with counting?

[glenn-jocher]

@Anandukc absolutely! You can use the yolov8n-seg model directly in your object counting code. Just make sure to track the objects across frames to count them accurately. Here's a quick example to get you started with tracking and counting:

from ultralytics import YOLO
from collections import defaultdict

# Initialize the model
model = YOLO("yolov8n-seg.pt")

# Initialize tracking history
track_history = defaultdict(int)

# Your video capture logic here
# ...

while True:
    # Your frame capture logic here
    # ...

    # Perform tracking
    results = model.track(frame, persist=True)

    # Update your counters based on tracking IDs
    for track_id in results[0].boxes.id.int().cpu().tolist():
        track_history[track_id] += 1  # Increment counter for each track ID

# Your display and release logic here
# ...

# Now you have the counts in track_history

Remember to adapt the counting logic to your specific entry/exit zones. If you have more questions or need further assistance, don't hesitate to reach out in the Ultralytics Issue Section. Happy coding! 🔥

[iskewedI]

Hi! Great article.
I have a question, it could be harder than I think but a little bit of guidance on what things should I investigate to accomplish this could help a lot.
The question: is it possible to maintain the detected object ID even after it went out of screen or covered by another object for a few frames?
I did some testing and every time some object reappears on the screen, a new ID is assigned to it (I can understand why), but could it be possible to assign an specific ID to an specific object and only that object?

[glenn-jocher]

Hey there! Thanks for reaching out and for the kind words. 😊

Maintaining the ID of a detected object even after it temporarily leaves the screen or gets occluded is indeed a challenging task, primarily due to the way most tracking algorithms assign IDs based on the presence of objects in consecutive frames. However, it's not impossible and involves advanced tracking techniques.

One approach to investigate is re-identification (Re-ID) combined with your tracking algorithm. Re-ID attempts to recognize previously detected objects based on their features, even after they've disappeared and reappeared. This involves creating a robust feature descriptor for each object, which can be matched against new detections to reassign the correct ID.

For Ultralytics YOLOv8, you might consider enhancing the tracking logic in the model.track() method to include a Re-ID component. This could involve:

1. Extracting feature vectors for each detected object using the model's intermediate layers.
2. Storing these features along with the object IDs.
3. When an object reappears, compare its features with the stored ones using a similarity metric (e.g., cosine similarity).
4. Reassign the ID based on the highest similarity score.

Here's a simplified code snippet to give you an idea:

# Assuming 'features' is a dictionary storing object IDs and their feature vectors
# and 'new_features' is the feature vector of a newly detected object.

from scipy.spatial.distance import cosine

def find_closest_id(new_features, features):
    min_distance = float('inf')
    closest_id = None
    for obj_id, obj_features in features.items():
        distance = cosine(new_features, obj_features)
        if distance < min_distance:
            min_distance = distance
            closest_id = obj_id
    return closest_id

Keep in mind, this is a high-level overview, and implementing a robust Re-ID system can get quite complex, involving deep learning for feature extraction.

For a deeper dive, I recommend exploring literature on object tracking and Re-ID in computer vision. Also, checking out existing Re-ID frameworks or libraries could provide some ready-to-use solutions or inspiration.

Hope this helps! If you have more questions or need further assistance, feel free to ask. 🚀

[Enryuwashere]

Hello if I have two objects for examples car and truck, Can we change the id start from 1 for each class? for example car =1 , truck =1 Thank you

[pderrenger]

Hello! 😊

For assigning unique IDs starting from 1 for each class (e.g., car=1, truck=1), the tracking functionality in YOLOv8 assigns unique track IDs across all classes rather than resetting for each class. This approach ensures each object has a distinct ID for tracking purposes across frames.

However, if you're looking to customize this behavior for instance segmentation and tracking, you might consider post-processing the tracking results. You could map the global track IDs to class-specific IDs in your application logic.

Here's a simple example of how you might approach this in Python:

from collections import defaultdict

# Assuming `results` is your tracking output
track_ids = results[0].boxes.id.int().cpu().tolist()
class_ids = results[0].boxes.cls.int().cpu().tolist()

# Mapping global track IDs to class-specific IDs
class_specific_ids = defaultdict(lambda: defaultdict(lambda: None))
class_counters = defaultdict(int)

for track_id, class_id in zip(track_ids, class_ids):
    if class_specific_ids[class_id][track_id] is None:
        class_counters[class_id] += 1
        class_specific_ids[class_id][track_id] = class_counters[class_id]

# Now, class_specific_ids contains class-specific IDs starting from 1

This script assigns a new, incremental ID for each track within its class. It's a starting point and might need adjustments based on your specific requirements.

If you have further questions or need more assistance, feel free to ask! 🚀

[elliosam]

Is there a way to filter the kind of object this code detects? For example, I'd like to create a version of this that only detects cars. What parts of this code would I have to change to make that possible?

[pderrenger]

@elliosam absolutely, you can filter detections to specific object classes like cars by modifying the class names in your dataset's YAML file or by filtering the results after prediction. Here's a quick example on how to filter for cars post-prediction:

from ultralytics import YOLO

# Load your model
model = YOLO('yolov8n.pt')  # or your custom model

# Run prediction
results = model('path/to/your/image.jpg')

# Filter for cars (assuming 'car' is class index 2)
car_detections = results[0].pandas().xyxy[0][results[0].pandas().xyxy[0]['class'] == 2]

print(car_detections)

Make sure to replace 2 with the correct class index for cars in your model's class list. You can find this index in the model.names attribute. Happy coding! 🚗

[elliosam]

This is great info! I'm running into some issues with that filter line though. Here's the terminal output for whenever I run the code. The code seems to run fine, but I haven't been able to fully test it quite yet.

0: 384x640 (no detections), 100.8ms
Speed: 2.5ms preprocess, 100.8ms inference, 2.0ms postprocess per image at shape (1, 3, 384, 640)
Traceback (most recent call last):
  File "c:\Users\ellio\Documents\senior d\ALRIGHTMANN\mainprotov3.py", line 46, in <module>
    car_detections = results[0].pandas().xyxy[0][results[0].pandas().xyxy[0]['class'] == 2]
                     ^^^^^^^^^^^^^^^^^
  File "C:\Users\ellio\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.12_qbz5n2kfra8p0\LocalCache\local-packages\Python312\site-packages\ultralytics\utils\__init__.py", line 156, in __getattr__
    raise AttributeError(f"'{name}' object has no attribute '{attr}'. See valid attributes below.\n{self.__doc__}")
AttributeError: 'Results' object has no attribute 'pandas'. See valid attributes below.

    A class for storing and manipulating inference results.

    Attributes:
        orig_img (numpy.ndarray): Original image as a numpy array.
        orig_shape (tuple): Original image shape in (height, width) format.
        boxes (Boxes, optional): Object containing detection bounding boxes.
        masks (Masks, optional): Object containing detection masks.
        probs (Probs, optional): Object containing class probabilities for classification tasks.
        keypoints (Keypoints, optional): Object containing detected keypoints for each object.
        speed (dict): Dictionary of preprocess, inference, and postprocess speeds (ms/image).
        names (dict): Dictionary of class names.
        path (str): Path to the image file.

    Methods:
        update(boxes=None, masks=None, probs=None, obb=None): Updates object attributes with new detection results.
        cpu(): Returns a copy of the Results object with all tensors on CPU memory.
        numpy(): Returns a copy of the Results object with all tensors as numpy arrays.
        cuda(): Returns a copy of the Results object with all tensors on GPU memory.
        to(*args, **kwargs): Returns a copy of the Results object with tensors on a specified device and dtype.
        new(): Returns a new Results object with the same image, path, and names.
        plot(...): Plots detection results on an input image, returning an annotated image.
        show(): Show annotated results to screen.
        save(filename): Save annotated results to file.
        verbose(): Returns a log string for each task, detailing detections and classifications.
        save_txt(txt_file, save_conf=False): Saves detection results to a text file.
        save_crop(save_dir, file_name=Path("im.jpg")): Saves cropped detection images.
        tojson(normalize=False): Converts detection results to JSON format.

Here's my code:

import cv2 as cv
import numpy as np
import pickle
import pandas
from ultralytics import YOLO
from ultralytics.utils.plotting import Annotator, colors
from PIL import Image, ImageDraw

# background image
bg = cv.imread('Photos/parkingTestImg.png')

# read in rhombus dictionaries from pickle
# each dictionary has 4 sets of x & y coords labeled as 'x1', 'y1', etc
with open('parkingPositions', 'rb') as f:
    shapes = pickle.load(f)

# load AI model (in this case, YOLOv8 nano)
# change the 'n' to s, m, or l for larger models. could help with accuracy
model = YOLO('yolov8n.pt')

# take rhombus verticies from pickle file and draw the shapes again
for d in shapes:
    cv.circle(bg, (d['x1'], d['y1']), 2, (0,0,255), 2, cv.FILLED)
    cv.circle(bg, (d['x2'], d['y2']), 2, (0,0,255), 2, cv.FILLED)
    cv.circle(bg, (d['x3'], d['y3']), 2, (0,0,255), 2, cv.FILLED)
    cv.circle(bg, (d['x4'], d['y4']), 2, (0,0,255), 2, cv.FILLED)
    cv.line(bg, (d['x1'], d['y1']), (d['x2'], d['y2']), (255,255,0), 2, cv.LINE_AA)
    cv.line(bg, (d['x2'], d['y2']), (d['x3'], d['y3']), (255,255,0), 2, cv.LINE_AA)
    cv.line(bg, (d['x3'], d['y3']), (d['x4'], d['y4']), (255,255,0), 2, cv.LINE_AA)
    cv.line(bg, (d['x4'], d['y4']), (d['x1'], d['y1']), (255,255,0), 2, cv.LINE_AA)
    cv.imshow('Parking Spots', bg)

# initialize variables to create masks of each polygon
img2 = bg
img3 = Image.new('L', (bg.shape[1], bg.shape[0]),0)
i = 0

# run through each polygon and create a mask of it individually
while i<len(shapes):
    polygon = [(shapes[i]['x1'], shapes[i]['y1']), (shapes[i]['x2'], shapes[i]['y2']), (shapes[i]['x3'], shapes[i]['y3']), (shapes[i]['x4'], shapes[i]['y4'])]
    ImageDraw.Draw(img3).polygon(polygon, outline=1, fill=1)
    mask = np.array(img3)
    masked_img = cv.bitwise_and(img2, img2, mask = mask)
    cv.imshow('d',masked_img)
    
    # run the mask through YOLO
    results = model(masked_img)
    # filter results to just cars
    car_detections = results[0].pandas().xyxy[0][results[0].pandas().xyxy[0]['class'] == 2]
    print(car_detections)
    i += 1
    
cv.waitKey(0)

[pderrenger]

Hey there! 🚗 It looks like the issue is due to the .pandas() method not being available in the Results object directly. Instead, you can access the detections and then filter for cars like this:

# Run the mask through YOLO
results = model(masked_img)

# Assuming 'car' is class index 2, adjust if necessary based on your model.classes
car_class_index = 2

# Filter results to just cars
car_detections = [det for det in results[0].boxes if det.cls == car_class_index]

# If you need to convert it to a pandas DataFrame
import pandas as pd
if car_detections:
    df = pd.DataFrame([det.numpy() for det in car_detections], columns=['x', 'y', 'w', 'h', 'conf', 'cls'])
    print(df)

This approach manually filters the detections for cars and then, if needed, converts them into a pandas DataFrame. Make sure to adjust car_class_index to match the index of cars in your model's class list. Happy coding! 🛠️

[elliosam]

This code works, thank you so much! Once last question: the default YOLOv8 model is great but it's having problems detecting cars from certain angles, mainly from straight above but also the farther away a car is, the less often it successfully finds it. Are there any other free models out there to look into that specialize in cars? Never used anything like this before, maybe there are models with education licenses?

[pderrenger]

@elliosam hey there! 🚗 It's awesome to hear the code is working well for you! Regarding detecting cars from challenging angles like directly above or at a distance, it's indeed a common issue due to the variation in appearance and scale.

While the default YOLOv8 models are quite robust, for specialized tasks like car detection from aerial or distant perspectives, you might want to explore models specifically trained on such datasets. Although we don't have a specialized "car from above" model, you can improve detection by:

1. Fine-tuning the YOLOv8 model on a dataset that includes more of these challenging angles and distances. This approach can significantly improve detection performance for your specific use case.
2. Exploring datasets like the Stanford Cars Dataset or UAVDT dataset for aerial vehicle detection, which might have pre-trained models available for use or fine-tuning.

Remember, fine-tuning requires a bit of effort but can yield great results tailored to your needs. If you're new to this, there are plenty of tutorials out there on how to fine-tune models with custom datasets. Good luck, and feel free to reach out if you have more questions! 🛠️

[nickguim]

Is there any recommendations or tutorials for object tracking with YOLOv5? I have been searching for awhile and struggling to get anything to work with DEEPSORT in OpenCV. Additionally, I tried this how-to-guide for yolov5: https://roboflow.com/how-to-track/yolov5 but the code used tells me it requires yolov8 weight files and will not accept v5.

Thank you

[glenn-jocher]

Hey there! 😄

I understand you're looking for guidance on object tracking with YOLOv5 and integration with DEEPSORT, especially when you've come across some hurdles adapting guides intended for YOLOv8. While it's true that YOLOv8 brings some enhancements and new capabilities that might not be directly transferable back to YOLOv5, you can still achieve effective tracking with YOLOv5.

For YOLOv5 with DEEPSORT, I recommend focusing on a few key areas:

1. Ensure DEEPSORT and OpenCV compatibility. You may need to adapt data formats between YOLOv5 detections and what DEEPSORT expects. Typically, DEEPSORT needs detections in the format [x1, y1, x2, y2, confidence].

2. Double-check paths and file names when loading your YOLOv5 weights. DEEPSORT itself should not mandate YOLOv8 weights specifically, but the error message you're seeing could be due to the script or command line expecting a particular format or path.

3. For the code examples you've found that are specific to YOLOv8, you might need to adapt them slightly for YOLOv5. The core concept remains similar; you're feeding the detection results from YOLO to a tracker such as DEEPSORT. Ensure that your input formats align with what DEEPSORT expects.

If you're still facing issues or have specific errors you're encountering, feel free to share some snippets or more detailed error messages. This way, it'll be easier to pinpoint the problem and guide you more effectively.

Here's a basic adaptation for integrating YOLOv5 detections with DEEPSORT you can start with:

import cv2
# Assume you have YOLOv5 and DEEPSORT instances setup

while True:
    ret, frame = cap.read()
    if not ret:
        break

    results = yolov5(frame)  # Get detections from YOLOv5
    detections = format_for_deepsort(results.xyxy[0])  # Convert to DEEPSORT format

    tracker.update(detections)
    # Further processing and visualization code here

    cv2.imshow('tracked', frame)
    if cv2.waitKey(1) & 0xFF == ord('q'):
        break

This pseudocode assumes you have a function format_for_deepsort() that formats YOLOv5 detections into the form DEEPSORT expects. The rest of your setup for YOLOv5 and DEEPSORT will dictate the specifics.

Lastly, I encourage exploring the Ultralytics GitHub Discussions for more community answers and insights related to YOLOv5 tracking. Many users share their experiences and solutions that might directly address your case.

Hope this helps! If you have further questions, feel free to ask. Happy coding! 🚀

[OUSSAMARC]

Hi, how can I train the YOLOv8 model on my own dataset?

[pderrenger]

@OUSSAMARC hi there!

To train the YOLOv8 model on your own dataset, you'll need to follow a few steps to prepare your data and configure the training process. Here's a concise guide to get you started:

1. Prepare Your Dataset:

   • Ensure your dataset is in a format compatible with YOLOv8. Typically, this involves having images and corresponding annotation files (in YOLO format, COCO format, etc.).
   • Create a dataset configuration file (e.g., data.yaml) that specifies the paths to your training and validation data, as well as the class names.

2. Install the Latest Versions:

   • Make sure you have the latest versions of torch and ultralytics installed. You can upgrade them using:

     pip install --upgrade torch ultralytics

3. Train the Model:

   • You can train the model using the Command Line Interface (CLI) or Python. Here are examples for both:

   CLI:

   yolo train data=path/to/data.yaml model=yolov8n.pt epochs=100 imgsz=640

   Python:

   from ultralytics import YOLO
   
   # Load a model
   model = YOLO('yolov8n.pt')  # load a pretrained model
   
   # Train the model
   model.train(data='path/to/data.yaml', epochs=100, imgsz=640)

4. Monitor Training:

   • During training, you can monitor the progress through the logs and visualizations provided by the Ultralytics framework.

For more detailed information on instance segmentation and tracking, you can refer to our Instance Segmentation and Tracking Guide.

If you encounter any issues or have further questions, feel free to ask here. Happy training! 🚀

[vinura]

Hi, When i use yolo track cli with rtsp there is no startup delay.

but when i do the same with opencv and yolo track there is a 10-20 seconds delay . I am running everything on jetsion orion nx 16gb.

Is there anything different between cli method and python api ?

[glenn-jocher]

Hi @vinura,

Thank you for the detailed information and for sharing your minimal reproducible code example. It's very helpful for diagnosing the issue! 😊

Given that you've confirmed the issue persists across different hardware setups and that the RTSP stream works flawlessly with OpenCV alone, it seems the delay is introduced when the YOLO model processes the frames.

Here are a few additional steps and considerations to help address the delay:

1. Model Initialization: Ensure that the model is fully initialized before starting the frame processing loop. Sometimes, the initial setup of the model can cause delays.

   import time
   import cv2
   from ultralytics import YOLO
   
   # Load the YOLOv8 model
   model = YOLO("/home/ra_ai/Desktop/ra_gimbal_ai/pre_trained_models/yolov8n.engine")
   
   # Warm-up the model
   dummy_frame = cv2.imread("path/to/dummy/image.jpg")
   model.track(dummy_frame, persist=True)
   
   # Open the video file
   video_path = "rtsp://192.168.8.178:8555/cam"
   cap = cv2.VideoCapture(video_path)
   
   # Loop through the video frames
   while cap.isOpened():
       success, frame = cap.read()
       if success:
           start_time = time.time()
           # Run YOLOv8 tracking on the frame, persisting tracks between frames
           results = model.track(frame, persist=True)
           end_time = time.time()
   
           # Visualize the results on the frame
           annotated_frame = results[0].plot()
   
           # Display the annotated frame
           cv2.imshow("YOLOv8 Tracking", annotated_frame)
           print(f"Frame processed in {end_time - start_time:.2f} seconds")
   
           # Break the loop if 'q' is pressed
           if cv2.waitKey(1) & 0xFF == ord("q"):
               break
       else:
           # Break the loop if the end of the video is reached
           break
   
   # Release the video capture object and close the display window
   cap.release()
   cv2.destroyAllWindows()

2. Profile the Code: Use Python's cProfile or similar tools to profile your code and identify any bottlenecks.

   import cProfile
   import pstats
   
   def main():
       # Your existing code here
   
   if __name__ == "__main__":
       profiler = cProfile.Profile()
       profiler.enable()
       main()
       profiler.disable()
       stats = pstats.Stats(profiler).sort_stats('cumtime')
       stats.print_stats()

3. TensorRT Optimization: Ensure that the TensorRT engine is optimized for your specific hardware. You might want to re-export the model with specific optimizations for your Jetson NX. Refer to the TensorRT Integration Guide for more details.

4. Check for Memory Leaks: Ensure there are no memory leaks in your code that could be causing the delay over time. Monitor the memory usage of your application using tools like htop or nvidia-smi.

5. Use Asynchronous Processing: Consider using asynchronous processing to handle the RTSP stream and model inference separately. This can help in reducing the delay by decoupling the frame capture and processing steps.

If the issue persists, please let us know, and we can further investigate. For more detailed guidance on instance segmentation and tracking, you can refer to our Instance Segmentation and Tracking Guide.

Thank you for your patience and cooperation!

[vinura]

Hi @glenn-jocher,

I get following error when i add the dummy image and it goes away as soon as i comment out the warm up step with dummy image

Traceback (most recent call last):
File "/home/ra_ai/Desktop/ra_gimbal_ai/test.py", line 22, in
results = model.track(frame, persist=True)
File "/home/ra_ai/.local/lib/python3.10/site-packages/ultralytics/engine/model.py", line 482, in track
return self.predict(source=source, stream=stream, **kwargs)
File "/home/ra_ai/.local/lib/python3.10/site-packages/ultralytics/engine/model.py", line 442, in predict
return self.predictor.predict_cli(source=source) if is_cli else self.predictor(source=source, stream=stream)
File "/home/ra_ai/.local/lib/python3.10/site-packages/ultralytics/engine/predictor.py", line 168, in call
return list(self.stream_inference(source, model, *args, **kwargs)) # merge list of Result into one
File "/home/ra_ai/.local/lib/python3.10/site-packages/torch/utils/_contextlib.py", line 35, in generator_context
response = gen.send(None)
File "/home/ra_ai/.local/lib/python3.10/site-packages/ultralytics/engine/predictor.py", line 262, in stream_inference
self.run_callbacks("on_predict_postprocess_end")
File "/home/ra_ai/.local/lib/python3.10/site-packages/ultralytics/engine/predictor.py", line 399, in run_callbacks
callback(self)
File "/home/ra_ai/.local/lib/python3.10/site-packages/ultralytics/trackers/track.py", line 70, in on_predict_postprocess_end
tracks = tracker.update(det, im0s[i])
File "/home/ra_ai/.local/lib/python3.10/site-packages/ultralytics/trackers/byte_tracker.py", line 293, in update
warp = self.gmc.apply(img, dets)
File "/home/ra_ai/.local/lib/python3.10/site-packages/ultralytics/trackers/utils/gmc.py", line 102, in apply
return self.applySparseOptFlow(raw_frame)
File "/home/ra_ai/.local/lib/python3.10/site-packages/ultralytics/trackers/utils/gmc.py", line 329, in applySparseOptFlow
matchedKeypoints, status, _ = cv2.calcOpticalFlowPyrLK(self.prevFrame, frame, self.prevKeyPoints, None)
cv2.error: OpenCV(4.10.0) /io/opencv/modules/video/src/lkpyramid.cpp:1394: error: (-215:Assertion failed) prevPyr[level * lvlStep1].size() == nextPyr[level * lvlStep2].size() in function 'calc'.

[vinura]

I was able to fix the video lag by using RTSP OpenCV implementation mentioned here

https://medium.com/@vikas.c20/optimizing-rtsp-video-processing-in-opencv-overcoming-fps-discrepancies-and-buffering-issues-463e204c7b86

[vinura]

Update - It is better than what it used to, but still i get behind after a while. this never happens on CLI version.

[glenn-jocher]

Hi @vinura,

Thank you for sharing the solution you found for the video lag issue using the RTSP OpenCV implementation. It's great to hear that it resolved the problem! 🎉

For others who might face similar issues, the article you referenced provides valuable insights into optimizing RTSP video processing with OpenCV, which can help overcome FPS discrepancies and buffering issues.

For those interested in instance segmentation and tracking with Ultralytics YOLOv8, you can find detailed guidance and examples in our Instance Segmentation and Tracking Guide. This resource covers various aspects of instance segmentation, including how to uniquely label and track objects in video streams.

If you have any further questions or run into other issues, feel free to reach out. We're here to help! 😊

[felixho789]

I have a question, when I annotate the item, do I only annotae it when it is clear?
When there is a obsticle blocked part of it, should I annotate it?
should I just annotate the the item(exclude obstacle area) or I just annotate the whole item (include the obstacle)?

If the purpose is tacking an moving item

[glenn-jocher]

@felixho789 hi there!

Great question! When annotating items for instance segmentation, especially for tracking moving objects, it's important to consider how the model will interpret the annotations.

1. Clear Visibility: If the item is fully visible, annotate the entire item as accurately as possible.

2. Partial Occlusion: If the item is partially blocked by an obstacle, you should still annotate the entire item, including the occluded parts. This helps the model learn to recognize and track objects even when they are partially obscured. Annotating only the visible part might lead to inconsistent tracking when the object moves and becomes fully visible again.

Here's a quick example to illustrate this:

import cv2
from ultralytics import YOLO
from ultralytics.utils.plotting import Annotator, colors

model = YOLO("yolov8n-seg.pt")  # segmentation model
cap = cv2.VideoCapture("path/to/video/file.mp4")
w, h, fps = (int(cap.get(x)) for x in (cv2.CAP_PROP_FRAME_WIDTH, cv2.CAP_PROP_FRAME_HEIGHT, cv2.CAP_PROP_FPS))

out = cv2.VideoWriter("instance-segmentation.avi", cv2.VideoWriter_fourcc(*"MJPG"), fps, (w, h))

while True:
    ret, im0 = cap.read()
    if not ret:
        print("Video frame is empty or video processing has been successfully completed.")
        break

    results = model.predict(im0)
    annotator = Annotator(im0, line_width=2)

    if results[0].masks is not None:
        clss = results[0].boxes.cls.cpu().tolist()
        masks = results[0].masks.xy
        for mask, cls in zip(masks, clss):
            annotator.seg_bbox(mask=mask, mask_color=colors(int(cls), True), det_label=names[int(cls)])

    out.write(im0)
    cv2.imshow("instance-segmentation", im0)

    if cv2.waitKey(1) & 0xFF == ord("q"):
        break

out.release()
cap.release()
cv2.destroyAllWindows()

This approach ensures that the model can handle various scenarios, including occlusions, which are common in real-world applications.

For more detailed guidance, you can refer to our Instance Segmentation and Tracking Guide.

Feel free to ask if you have any more questions! 😊

[svvmiiir]

Hello, i had trained Yolov8 on my custom dataset for object detection and i want to know if i can do tracking using my new model ? Thanks.

[pderrenger]

@svvmiiir yes, you can use your custom-trained YOLOv8 model for tracking. Simply load your model and use the model.track method to perform object tracking. This method will assign consistent IDs to objects across frames, allowing you to track them over time. For more details on instance segmentation and tracking, you can refer to the Ultralytics documentation.

[svvmiiir]

yes, I tried using my custom-trained YOLOv8 model for tracking on Google Colab. I loaded my model and used the model.track method to perform object tracking. However, I'm encountering issues with saving and displaying the tracking results.

[svvmiiir]

Hi @pderrenger yes, I tried using my custom-trained YOLOv8 model for tracking on Google Colab. I loaded my model and used the model.track method to perform object tracking. However, I'm encountering issues with saving and displaying the tracking results.

[pderrenger]

It sounds like you're on the right track with using your custom-trained YOLOv8 model for object tracking. For saving and displaying the tracking results, ensure that your video writer and display logic are correctly implemented. If issues persist, please verify that you are using the latest version of the Ultralytics package. For more detailed guidance, you can refer to the Instance Segmentation and Tracking guide. If you need further assistance, feel free to ask here.

[MarserHK]

Hello, may I inquire if it is possible to determine if there is an intersection between an object segmented by instance segmentation and an object detected by object detection? If it is possible, what would be the specific steps to do so?

[glenn-jocher]

@MarserHK yes, you can determine intersections between segmented and detected objects by comparing their bounding boxes or masks. Use the intersects method from libraries like OpenCV or Shapely to check for overlaps. For more details, refer to the Ultralytics documentation.

[MarserHK]

I've tried some methods but still can't achieve it. I want to use a segmentation model to obtain an irregular mask of an object and use a bounding box obtained from an object detection model to calculate their IOU. Could you write a piece of code that can accomplish this functionality? Thank you very much!

[glenn-jocher]

To calculate the IoU between a segmentation mask and a detection bounding box, you can use libraries like OpenCV or Shapely to find intersections. Here's a minimal code snippet to get you started:

from shapely.geometry import Polygon

# Example mask and box coordinates
mask_coords = [(x1, y1), (x2, y2), ...]
box_coords = [(x1, y1), (x2, y2), ...]

# Create polygons
mask_polygon = Polygon(mask_coords)
box_polygon = Polygon(box_coords)

# Calculate IoU
intersection = mask_polygon.intersection(box_polygon).area
union = mask_polygon.union(box_polygon).area
iou = intersection / union

This should help you compute the IoU for your use case.

[MarserHK]

Thank you for your answer. How can I obtain the coordinates of the mask? I am using the following code to get the mask：
`
cabinet_model = YOLO('runs/segment/train2/weights/best.pt')

def segment_cabinet(frame):
results = cabinet_model.predict(frame)
if results and len(results) > 0:
result = results[0]
if result.masks is not None:
masks = result.masks.xy
return masks

cabinet_masks = segment_cabinet(frame)
but I encountered an error, and the error message is as follows:
File "test_lean_pose_seg.py", line 278, in main
cabinet_masks = segment_cabinet(frame)
File "test_lean_pose_seg.py", line 197, in segment_cabinet
results = cabinet_model.predict(frame)

AttributeError: 'Segment' object has no attribute 'detect'
`

[glenn-jocher]

@MarserHK to obtain the coordinates of a segmented mask and calculate the Intersection over Union (IoU) with a detected bounding box, you can use the Shapely library to handle geometric operations. Here's a simple code snippet to help you achieve this:

from shapely.geometry import Polygon

def calculate_iou(mask_coords, box_coords):
    # Create polygons from the coordinates
    mask_polygon = Polygon(mask_coords)
    box_polygon = Polygon(box_coords)

    # Calculate intersection and union
    intersection = mask_polygon.intersection(box_polygon).area
    union = mask_polygon.union(box_polygon).area

    # Calculate IoU
    iou = intersection / union
    return iou

# Example usage
mask_coords = [(x1, y1), (x2, y2), ...]  # Replace with actual mask coordinates
box_coords = [(x1, y1), (x2, y2), (x3, y3), (x4, y4)]  # Replace with actual box coordinates

iou = calculate_iou(mask_coords, box_coords)
print(f"IoU: {iou}")

Make sure to replace mask_coords and box_coords with the actual coordinates from your results. If you have any more questions, feel free to ask!