modes/val/

[giscus[bot]]

modes/val/

Guide for Validating YOLOv8 Models. Learn how to evaluate the performance of your YOLO models using validation settings and metrics with Python and CLI examples.

https://docs.ultralytics.com/modes/val/

[shubhamjanu]

This line of code "metrics.box.map50" gives error saying that "'NoneType' object has no attribute 'box' ".

My code is described below:

Load a YOLOv8 model

model = YOLO('yolov8n.pt')

Train the model

results_train = model.train(data='japan5.yaml', epochs=1, imgsz=600)
model = YOLO('runs/detect/train11/weights/best.pt') # load a custom model

Validate the model

metrics = model.val() # no arguments needed, dataset and settings remembered
map50=metrics.box.map50
print(map50)

Please suggest.

[glenn-jocher]

@shubhamjanu hello! 😊 Let's dive right into your queries:

1. For training with high-resolution images while maintaining the aspect ratio, YOLOv8's training mechanism automatically handles resizing and aspect ratios elegantly. However, dealing with very high-resolution images does require a balance. For your case, using imgsz=640 (or slightly higher if your GPU supports it) is a sensible choice. It scales down images, maintaining their aspect ratio. YOLOv8 interprets imgsz as [height, width]. The worse results with [1920,1024] could be due to several factors, including overfitting or not fine-tuning other hyperparameters accordingly.

2. Yes, resizing with the imgsz parameter handles annotations accordingly. YOLOv8 takes care of scaling the annotations, so you don't have to manually adjust your annotation files.

3. Whether downsampling or upsampling is better can depend on various factors, including the nature of your dataset and what you're trying to detect. Generally, slight upsampling (e.g., 600x600 to 608x608) can be beneficial as it introduces minimal distortion while adhering to the model's architectural constraints. However, extensive experimentation with your specific dataset is the best way to determine the ideal approach.

4. The IoU threshold differences during training (0.7) and validation (0.6) are predefined settings based on empirical performance metrics. If your goal is to calculate mAP at IoU=0.5, setting iou=0.5 during both training and validation is a reasonable approach. This aligns the model's detection criteria closely with your evaluation metric and can provide more relevant performance insights.

Remember, experimenting with these settings in the context of your specific dataset and objectives is key to achieving the best results. Happy experimenting! 🚀

[DoFulangChen]

hi, when i use the pycharm, the code is (not cmd line)
model.train(data=r"datasets\data.yaml", epochs=5, batch=1, device='cpu', task='segment', imgsz=[1920, 1200])

one warning comes out
WARNING ⚠️ updating to 'imgsz=1920'. 'train' and 'val' imgsz must be an integer, while 'predict' and 'export' imgsz may be a [h, w] list or an integer, i.e. 'yolo export imgsz=640,480' or 'yolo export imgsz=640'

how can i train my own dataset ? img_width = 1920, img_height = 1200, many tks

[glenn-jocher]

Hey there! 👋 It looks like you've encountered a warning due to a slight misunderstanding in specifying the image size for training in PyCharm.

For training, YOLOv8 expects the imgsz argument to be a single integer, representing both the height and width since it scales the images keeping their aspect ratio intact. This means when you provide a single integer (like imgsz=640), it maintains the aspect ratio for you automatically. If you're seeing worse results at a specific size, consider different dimensions close to your target but as a single value, or experiment with the rect parameter as you mentioned.

The warning you encountered suggests a syntax more aligned with export or predict tasks, which can accept [height, width] lists. For training your dataset that has images of resolution 1920x1200, start with something manageable for your GPU like imgsz=640 and see if your hardware can handle higher resolutions gradually.

To answer your question, for training, you should adjust the imgsz to a single value that ensures your hardware supports it, without explicit width and height distinction. The annotation adjustments are taken care of by YOLOv8 based on the resizing.

Regarding whether to downsample or upsample, both have benefits and drawbacks, and the choice heavily depends on your specific problem and dataset. Downsampling retains more original content, but might miss fine details, whereas upsampling might introduce artifacts. Given the two options, I'd slightly lean towards downsampling for less distortion, but encourage testing both to see what works better for your case.

The IoU parameter differences during training and validation reflect typical use cases, but for mAP calculation at IoU=0.5, you should indeed adjust the IoU threshold to 0.5 during both training and validation for consistency with your evaluation criteria.

Hope this helps clarify things! Let me know if there's anything else I can do for you. Happy training! 😊

[DoFulangChen]

i see, thanks for your answer, regards

[glenn-jocher]

Hey there! 😊 Glad you found the answer helpful. For training your dataset with varying resolutions while preserving aspect ratios and avoiding out-of-memory issues, you might consider trying adaptive resizing with smaller imgsz values that fit your GPU capabilities, like imgsz=640. YOLOv8 indeed auto-scales annotations, so you're all set on that front!

Regarding image resolutions, the decision between downsampling and upsampling (e.g., 576 vs. 608) generally depends on your specific dataset and task. Experimenting is key, but starting with downsampling might be slightly more efficient to retain original content without artificially introducing pixels.

For calculating mAP at IoU=0.5, setting iou=0.5 for both training and validation phases would align your training closer with your evaluation metric.

Remember, YOLOv8 will take care of aspect ratios and annotations for you during training. Just specify your desired imgsz (as a single integer for training) and experiment to find what works best for your setup. Happy training! 🚀

[devandress]

i was trying to train my model but, i dont find de predict file after training either best.pt

[pderrenger]

Hey there! 👋 It sounds like you're having trouble locating the predict file and the best.pt model after training. No worries, let's sort this out!

After training with YOLOv8, the best.pt model (which is the best performing model on your validation set during training) should be saved in your runs/train/exp*/weights/ directory. The directory might have a different name if you've set a custom --project or --name during training.

As for the predict functionality, it's not a separate file but a mode you can use with the YOLO CLI or Python API. To make predictions using your trained model, you can use the following command if you're using the CLI:

yolo detect predict model=runs/train/exp*/weights/best.pt source='path/to/your/image/or/video'

Or, if you're using Python:

from ultralytics import YOLO

# Load your trained model
model = YOLO('runs/train/exp*/weights/best.pt')

# Make predictions on an image or video
results = model(['path/to/your/image/or/video'])

# To view results
results.show()

Make sure to replace 'path/to/your/image/or/video' with the actual path to your image or video file. Also, adjust the path to best.pt based on where your trained model is saved.

If you're still having issues or if there's anything else I can help you with, feel free to ask. Happy coding! 😊

[madasuvenky]

When I convert the YOLO v8 model weights to int16 and validate, I'm getting 0 accuracy, but with float32 model weights, I'm getting 0.87 accuracy.

[pderrenger]

Hi there! 👋

Thank you for reaching out. The discrepancy you're observing between the int16 and float32 model weights is quite significant. Here are a few steps and considerations to help diagnose and potentially resolve the issue:

1. Reproducible Example: To better understand and reproduce the issue, could you please provide a minimal reproducible example? This will help us pinpoint the problem more effectively. You can refer to our guide on creating a minimum reproducible example.

2. Latest Versions: Ensure you are using the latest versions of all relevant packages, including Ultralytics YOLOv8. Sometimes, issues are resolved in newer releases.

3. Quantization Impact: Int16 quantization can significantly impact model accuracy, especially if the model was not specifically trained or fine-tuned for lower precision. You might want to try FP16 (half-precision) quantization instead, which often maintains better accuracy compared to int16.

4. Validation Metrics: When validating, make sure to use appropriate metrics and settings. Here’s a quick example of how you can validate your model using Python:

   from ultralytics import YOLO
   
   # Load your model
   model = YOLO("path/to/your/model.pt")
   
   # Validate the model
   metrics = model.val()
   print(f"mAP50-95: {metrics.box.map}")
   print(f"mAP50: {metrics.box.map50}")
   print(f"mAP75: {metrics.box.map75}")

   And using CLI:

   yolo val model=path/to/your/model.pt

5. Quantization-Aware Training: If you haven't already, consider using quantization-aware training (QAT) to better prepare your model for lower precision inference. This can help mitigate the accuracy drop when converting to int16.

6. Alternative Quantization: If int16 is a requirement, you might need to explore other quantization techniques or frameworks that better support int16 without significant accuracy loss.

If you continue to experience issues, please share more details about your setup and any error messages you encounter. This will help us assist you more effectively.

Looking forward to your response! 😊

[madasuvenky]

I need int16 accuracy

[pderrenger]

Hi there! 👋

Thank you for reaching out. Achieving high accuracy with int16 quantization can be challenging, as it often introduces more significant precision loss compared to float32. Here are a few steps to help you improve your int16 model accuracy:

1. Quantization-Aware Training (QAT): If you haven't already, consider using quantization-aware training. QAT simulates the effects of quantization during the training process, helping the model adapt to lower precision and potentially improving accuracy.

2. Validation Settings: Ensure your validation settings are optimized. Here’s a quick example of how to validate your model using Python:

   from ultralytics import YOLO
   
   # Load your model
   model = YOLO("path/to/your/int16_model.pt")
   
   # Validate the model
   metrics = model.val()
   print(f"mAP50-95: {metrics.box.map}")
   print(f"mAP50: {metrics.box.map50}")
   print(f"mAP75: {metrics.box.map75}")

   And using CLI:

   yolo val model=path/to/your/int16_model.pt

3. Check Latest Versions: Make sure you are using the latest versions of all relevant packages, including Ultralytics YOLOv8. Sometimes, updates include fixes and improvements that can help with quantization issues.

4. Alternative Quantization: If int16 is a strict requirement, you might need to explore other quantization techniques or frameworks that better support int16 without significant accuracy loss.

5. Hyperparameter Tuning: Fine-tune your model’s hyperparameters to see if you can achieve better performance with int16 weights. Adjusting parameters like learning rate, batch size, and others can sometimes yield better results.

If you can provide a minimal reproducible example, it would greatly help in diagnosing the issue more effectively. You can refer to our guide on creating a minimum reproducible example.

Feel free to reach out if you have any more questions or need further assistance. Happy coding! 😊

[eddmar1993]

After the training of YOLOv8, I got my metrics results in csv file. However, I did validation. But I cannot get my results in csv format. Only images (png & jpeg). How to get validation results in csv?

[glenn-jocher]

Hey there! 😊 To get your validation results in CSV format, you can utilize the save_json argument during validation, which will save the results in a JSON file. From there, you can easily convert the JSON file to CSV using Python. Here's a quick example:

from ultralytics import YOLO
import json
import pandas as pd

# Validate the model and save results to JSON
model = YOLO('path/to/your/model.pt')
model.val(save_json=True)

# Assuming your JSON results are saved as 'results.json'
with open('results.json') as f:
    data = json.load(f)

# Convert JSON to DataFrame then save as CSV
df = pd.DataFrame(data)
df.to_csv('validation_results.csv', index=False)

This will give you a CSV file with your validation results. Hope this helps! Let me know if you have any more questions.

[RudeNinja]

Hey, I am trying to get metrics such as recall and also trying to save some images in validation. I am using coco.yaml. Any idea how I might do it?

[glenn-jocher]

Hey there! 👋

For calculating recall and saving validation images using coco.yaml, you can simply run the validation process with the save_json and plots arguments set to True. This will save the results in JSON format (which includes recall among other metrics) and also save plots of predictions vs. ground truth for visual evaluation.

Here's how you can do it:

yolo detect val model=yolov8n.pt data=coco.yaml save_json=True plots=True

This command will validate your model and save the detailed metrics and images in the runs directory. You can find recall in the saved JSON file and the validation images in the plots directory.

Hope this helps! Let me know if you have any other questions. 😊

[madasuvenky]

When I convert the YOLO v8 model weights to int16 and validate, I'm getting 0 accuracy, but with float32 model weights, I'm getting 0.87 accuracy.

[glenn-jocher]

Hi there! 👋

Thank you for reaching out. The issue you're experiencing with the int16 model weights resulting in 0 accuracy is likely due to the quantization process. Quantizing to int16 can sometimes lead to significant loss in model precision, especially if not done carefully.

Here are a few steps you can take to troubleshoot and potentially resolve this issue:

1. Verify Quantization Process: Ensure that the quantization process is correctly implemented. Sometimes, additional steps such as calibration with representative data are required to maintain model accuracy.

2. Check Compatibility: Verify that the int16 quantization is supported and optimized for the hardware and framework you are using. Some frameworks may not fully support int16 quantization, leading to poor performance.

3. Use FP16 Instead: If int16 quantization is causing issues, consider using FP16 (half-precision) instead. FP16 often provides a good balance between model size reduction and maintaining accuracy. You can enable FP16 by setting the half argument to True during validation:

   yolo detect val model=yolov8n.pt data=coco.yaml half=True

4. Update Packages: Ensure you are using the latest versions of all relevant packages. Sometimes, bugs and issues are resolved in newer releases.

5. Provide a Reproducible Example: If the issue persists, providing a minimum reproducible example can help us diagnose the problem more effectively. You can find guidelines on creating a reproducible example here.

Here's an example of how you might validate your model with FP16 precision:

from ultralytics import YOLO

# Load a model
model = YOLO("yolov8n.pt")  # or your custom model

# Validate the model with FP16 precision
metrics = model.val(data="coco.yaml", half=True)
print(metrics.box.map)  # mAP50-95

Feel free to explore more about validation settings and metrics in the Ultralytics documentation.

I hope this helps! Let me know if you have any further questions. 😊

[madasuvenky]

Thankyou but i need int16 accuracy

[glenn-jocher]

Hi there! 👋

Thank you for your patience. Achieving high accuracy with int16 quantization can be challenging due to the precision loss that occurs during the quantization process. Here are a few steps you can take to improve the accuracy of your int16 quantized model:

1. Calibration with Representative Data: Ensure that you are using a representative dataset for calibration during the quantization process. This helps the model maintain accuracy by better approximating the range of values it will encounter during inference.

2. Quantization-Aware Training (QAT): If possible, consider using quantization-aware training. This technique simulates quantization during the training process, allowing the model to adjust its parameters to better handle the reduced precision.

3. Verify Quantization Implementation: Double-check the implementation of your int16 quantization to ensure it is correctly applied. Sometimes, small errors in the quantization process can lead to significant drops in accuracy.

4. Use FP16 as an Alternative: If int16 quantization continues to cause issues, you might want to consider using FP16 (half-precision) quantization instead. FP16 often provides a good balance between model size reduction and maintaining accuracy. You can enable FP16 by setting the half argument to True during validation:

   yolo detect val model=yolov8n.pt data=coco.yaml half=True

5. Update Packages: Ensure you are using the latest versions of all relevant packages. Sometimes, bugs and issues are resolved in newer releases.

6. Provide a Reproducible Example: If the issue persists, providing a minimum reproducible example can help us diagnose the problem more effectively. You can find guidelines on creating a reproducible example here.

Here's an example of how you might validate your model with FP16 precision:

from ultralytics import YOLO

# Load a model
model = YOLO("yolov8n.pt")  # or your custom model

# Validate the model with FP16 precision
metrics = model.val(data="coco.yaml", half=True)
print(metrics.box.map)  # mAP50-95

Feel free to explore more about validation settings and metrics in the Ultralytics documentation.

I hope this helps! Let me know if you have any further questions. 😊

[vanquang203546]

I want to get parameters of a class in metrics and save it to csv. Thank you very much.

[pderrenger]

Hi there! 👋 To save class-specific metrics like mAP, precision, and recall to a CSV file after validation, follow these steps:

Python API Example

After running validation, you can access class-specific metrics from the metrics object and save them using pandas:

import pandas as pd
from ultralytics import YOLO

# Validate your model
model = YOLO('yolov8n.pt')  # or your custom model
metrics = model.val()

# Assuming you want mAP for each class
map50_95_per_class = metrics.box.maps.tolist()

# Create a DataFrame
df = pd.DataFrame({
    'class': model.names,
    'mAP50-95': map50_95_per_class
})

# Save to CSV
df.to_csv('class_metrics.csv', index=False)

CLI Approach

If you're looking to perform post-processing after using the CLI for validation, you'd need to parse the output logs or JSON file (--save_json for generating a JSON file) and then manually write to a CSV using a similar approach or any preferred scripting language.

Remember, if you choose to save results directly from validation with --save_json, the resulting file can be easily parsed and converted into a CSV with your preferred data handling tools like pandas or even Excel.

Let me know if this helps or if you have any more questions! 😊

[HafizRaheem]

hey

i want to do the evaluate my model i already get the detection images using ignition gazebo and ros2 but now i want to evaluate my model like accuracy or MAP or recall how to do it i did not have any dataset or any annotations i am using pretrained yolov8 model and using coco8.yaml

[HafizRaheem]

for option 2 Run Predictions on Real-Time Images. what it mean for this sentence (could be a ROS2 subscriber saving images to disk temporarily) before start the detection i have to save images after the ros2 but before the detection start using yolo and then pass those images to the yolov8 for detection

and for predict does it give me the precision or recall
that is the main thing i have to do it other than that i have images with scores
file:///home/hafiz/ros2_ws/processed_images/processed_10.jpg

[HafizRaheem]

or first i have to train my model and then do everything download the dataset from the kaggle and trian it and do everything

[glenn-jocher]

Hey there! 👋

Absolutely, if you've already got a pretrained model and have newly annotated images in COCO format, you’re on a fantastic track. Since the images are already detected using Ignition Gazebo and ROS2, and now that you've annotated some of them for evaluation, you’re ready to move on to the validation part using Ultralytics YOLOv8.

For validation, you don't need to re-train your model unless you're aiming to fine-tune it on very specific data from your environment. Given you're using a pretrained model, here’s how you can proceed:

1. Prepare your dataset YAML file, ensuring it points to your annotated images and COCO-style annotations. Your YAML might look something like this (adjust paths as necessary):

val: path/to/your/images
annotations: path/to/your/annotations.json

2. Run validation to evaluate your model against your manually annotated dataset. This will give you precision, recall, and mAP metrics, providing insight into how well your model is performing.

yolo detect val model=path/to/your/pretrained/model.pt data=path/to/your/dataset.yaml

This step uses the annotations you've created to compare against the model's predictions, allowing you to assess the model's accuracy.

Since you're handling real-time images and have already annotated them, this validation process will enable you to evaluate the YOLO model's effectiveness in your specific application scenario.

For real-time detection, there's typically no need for a separate dataset aside from monitoring the model's general performance or unless you're using it for continuous learning. So, you're correct; for real-time detection, you proceed with the model prediction directly on incoming images from Ignition Gazebo and ROS2.

Please let me know if you need further guidance or have more questions. Best of luck with your project and your upcoming deadline! 😊

[HafizRaheem]

Thank You but confused how to do it this part i share my code do i have to do this in sperate or do it in same file that i share i have some images with bounding box and scores. now i don't need to rerun the detection or my node so i have annotaton json file and images that i use and draw bounding box using cvat and save as coco format just used those for precison and recall.
sorry i know i am asking stupid question but i have no choice

[glenn-jocher]

Hey there! 😄 It looks like you're on the right track with creating annotations for your images in COCO format using CVAT. Now, since you have both images and annotations, you don't need to rerun detections for validation purposes. You can use your annotations and images directly to calculate precision, recall, and mAP metrics.

Here’s the simplified process:

1. Ensure your annotated dataset (images + COCO format annotations) is organized correctly. Your .yaml file for the dataset should correctly point to the location of your images and annotations.

2. Validate your model using the dataset to get precision, recall, and mAP. Here's how you can do it:

   from ultralytics import YOLO
   
   # Load your model
   model = YOLO('path/to/your/pretrained/model.pt')
   
   # Validate the model
   metrics = model.val(data='path/to/your/dataset.yaml')
   print(metrics.box.map, metrics.box.map50, metrics.box.map75)

Or via CLI,

yolo detect val model=path/to/your/pretrained/model.pt data=path/to/your/dataset.yaml

3. No need for separate detection runs for validation. The val process will internally use your model to predict on the images specified in your dataset YAML and compare those predictions against your annotations to compute the evaluation metrics.

4. Prediction and Validation are separate steps. Predictions are for visual inspection or application integration (using detected bounding boxes and scores). Validation (with .val()) computes how well your model is performing statistically (precision, recall, mAP).

For real-time detections coming from Ignition Gazebo and ROS2 to YOLOv8, you're essentially running predictions. If you wish to evaluate these real-time detections, you would need a way to capture the inferred annotations and have a ground truth to compare against, which might require building a custom process.

No questions are stupid; we all start somewhere! 👍 If you're under a tight deadline and need more detailed help or file reviews, consider reaching out directly on GitHub issues or discussions specific to the Ultralytics YOLO repository. The community and developers are pretty responsive there. Best of luck with your project and deadline!

[UsmanRashid12]

Hello i have two questions.
First one: i have trained model on custom dataset for object detection. After testing the model is giving false positives i want to calculate the false positives? How to calculate the false positives that how many times it is giving me false positive.
Second one: I want to what save_hybrid does? and how to implement it what are the benefits of it.

[UsmanRashid12]

Hello thankyou for your reply. To the first answer i have this already but wanted to see the FP directly. Secondly what if i want to run evaluations on my whole model and then make a confusion matrix on it. Like run the model on all my dataset including all the images of train test valid and then create a confusion matrix of it. The issue i face is when running inference on each jpeg file it saves the image just by save=True but the evaluations such as mask, bbox, keypoints, names and orig img. when i try to save them it gives me error even i cant convert them to JSON because of object error in list. so how do you recommend i do this.

Issue with my model is it is detecting false positives with a high percentage so i want to see what the problem really is between those two wrongly detected classes.

[UsmanRashid12]

Also there is not much information available about the save_hybrid=True argument ive searched it online that how does it shows us the predictions with the model against actual labels. Because i get results in batches of images , confusion matrix, and others like P-curve, R_curve. after val

[pderrenger]

Hello! Thanks for reaching out. 🙌

For the save_hybrid functionality, by setting save_hybrid=True during the validation process, the model will save a hybrid version of labels. This means it will combine your model predictions with the original labels in your validation dataset. This can immensely help in analyzing the model's performance by visually comparing its predictions against the ground truth directly on the images. You can see these comparisons in the resulting batch of images, confusion matrices, P and R curves, etc., provided after running yolo detect val with save_hybrid=True. It's a powerful tool for diagnosing model behavior, especially in identifying false positives or understanding class-wise performance.

To address the issue of saving evaluations like masks, bboxes, keypoints, names, and original images, it sounds like you're looking for a structured output. Typically, these details are part of the Python object returned by validation or prediction methods and might not be directly serializable to JSON due to their complex structures (e.g., tensors or custom objects).

For a comprehensive evaluation, including generating a confusion matrix that visually represents model performance across all your dataset images (train, test, validate), you'd likely benefit from aggregating predictions across your dataset in a structured form like a CSV or a database, then using a separate script or tool to calculate and visualize the confusion matrix.

If you're encountering serialization issues, consider converting these objects to more JSON-friendly structures (like converting tensors to lists) before attempting to convert to JSON. Here’s a quick snippet for converting a prediction result to a list (Pythonic way), which might give you an idea:

# Assuming 'results' is the output from model validation or prediction
predictions = [result.xyxy[0].tolist() for result in results]

For identifying why your model is giving false positives between two classes, diving into the save_hybrid results can provide visual clues. Furthermore, systematically reviewing the confusion matrix and the specific images associated with those false positives can reveal patterns or specific features the model might be misinterpreting.

I hope this helps! If you have any more questions or need further assistance, feel free to ask. Happy coding! 😊

[UsmanRashid12]

Hello thankyou so much for this detailed answer i used save_hybrid feature. i have one more question

If i want to see the wrongly predicted image in the model how can i do that. For example my model is wrongly predicting an image giving me a false positive can i see through validation or any argument that the model predicted which image wrong?

[pderrenger]

@UsmanRashid12 hey there! 😀 Really glad the save_hybrid feature was helpful for you! To identify and inspect wrongly predicted images or false positives more directly, you might consider running validation with detailed logging enabled or utilizing the save_hybrid=True feature to visually inspect outputs.

For validation, running the command below generates a detailed report, including a confusion matrix that can highlight where your model tends to predict wrong classes:

yolo detect val model=path/to/your/model.pt data=your_dataset.yaml

For save_hybrid=True, unfortunately, the YOLOv8 documentation is sparse on this. Generally, it combines model predictions with ground-truth annotations on saved output images, allowing you to visually compare predictions against truth directly for each image.

To further drill down on specific false positives or to build a custom confusion matrix from your entire dataset, consider collating predictions programmatically:

from ultralytics import YOLO

model = YOLO('path/to/your/model.pt')

# For each image in your dataset...
for path in dataset_paths:
    results = model.predict(path)
    # Process each result

You can structure predictions alongside labels for all images to analyze/model performance comprehensively, export them to JSON or another format, and then use that data to specifically identify and examine false positives more closely.

Hope this helps! Let me know if there's anything else you're curious about. 🌟

[UsmanRashid12]

Hello i have two questions kindly help me out.

when i have custom trained model. if i load the model and then to detect use" !yolo detect val model=/weights/best.pt data=/data.yaml save=True save_json=True conf=0.85 iou=0.5 split=val " can i give it a new yaml file other than the one which model was trained. For example i used an old dataset to train the model. Now i have new data on which i want to validation of the model i upload data in valid folder and give its yaml file to the model to load those images and validate against the model. (THOSE ARE NEW IMAGES ADDED AFTER TRAINING IN VALID) will it give me accurate results on the new dataset against the model trained? I added 10k images to my Valid folder and want to detection on those new images and see the validation of the model.

Second question is that does conf threshold matters while doing validation? and can you please explain how is the confusion matrix is build after validation because its prediction against classes so it goes and compares with the ground truth? Please explain thankyou

[pderrenger]

@UsmanRashid12 hello! I'm happy to help with your questions. 😊

1. Validating on a New Dataset: Yes, you can specify a new YAML file that points to a different dataset for validation, such as your new data.yaml with the updated valid folder images. YOLOv8 models adapt to the dataset specified during the validation phase. So, if your model was trained on an older dataset and you now have a new set of images, simply adjust your data.yaml to point to these new images for validation. This approach will give you insights into how your model performs on the new dataset. However, keep in mind, if the new dataset is significantly different or more challenging, the accuracy might vary.

Example command adjustment:

!yolo detect val model=/weights/best.pt data=/new_data.yaml save=True save_json=True conf=0.85 iou=0.5 split=val

2. Confidence Threshold and Confusion Matrix: Yes, the confidence (conf) threshold matters during validation because it filters out detections below the specified confidence level, affecting the final metrics like precision, recall, and mAP. Lowering the threshold might increase the number of detected objects but can also introduce more false positives. Raising it can reduce false positives but might also miss some true positives.

The confusion matrix is constructed by comparing the predicted class labels against the true class labels for all the validations or test examples. It helps in understanding the performance of the model for each class, indicating how often it confuses two classes. For building it after validation, each prediction is matched with ground truth based on IoU criteria. True Positives (TP), False Positives (FP), and False Negatives (FN) are then counted for each class which goes into making the confusion matrix.

Hope this clarifies your queries! Let me know if there's anything else you want to know.

[TersteegC]

Hi, I want to use the validate function to finetune my trained model and see how it performs against real life pictures. There is however a difference in parameters available between "predict" mode and val "mode". For instance, I would like to use specific settings for Augment and retina_mask as part of the validation. Is this possible? And out of curiosity, why are the default settings for Val different from Predict?

[glenn-jocher]

Hey there! ✨ Thanks for reaching out with your query.

To tailor your validation settings for Augment and retina_mask, you can directly modify these parameters in your dataset's YAML file or provide them as additional arguments in your custom script. These options, however, are not directly available through the CLI for the val mode. Changes to parameters like these are usually done within the training script or the dataset configuration to affect the validation phase indirectly.

Regarding defaults differences between val and predict modes – these modes serve different purposes and thus have distinct default settings optimized for their specific use-cases. Val mode focuses on model evaluation against a known dataset with ground truth for performance metrics computation, hence might prefer higher accuracy over inference speed. In contrast, predict mode is tailored towards real-time inference, possibly preferring speed and real-world applicability.

Here's a quick Python snippet to kickstart customization for validation (ensure your dataset YAML reflects your augmentation desires):

from ultralytics import YOLO

# Load your custom model
model = YOLO('path/to/your/model.pt')

# Validate the model with custom settings
validation_results = model.val(data='your_dataset.yaml', imgsz=640, batch=16, conf=0.25, iou=0.6)

Remember, Augment and custom settings may require adjustments within your training/validation Python script or dataset YAML rather than through CLI commands. Hope this helps clear things up! 🚀

[SimCromarty]

Hi, I am completing a vision project for an automated industrial packing cell. Currently I have trained 3 custom models for the different tasks. I want to test the performance of each with different scenarios like bright, dim, natural, artificial lighting and varied backgrounds or similar objects. What would be the best method to do so? Would it be best to create these standard test datasets, link with yaml then use the val function? Or use predict and count correct / incorrect detections and work out metrics from there? Once I have done this I also want to investigate performance with different model architecture sizes and training times so has to be easily repeatable. How is best to implement basically, thanks in advance!

[pderrenger]

@SimCromarty hello! 🌟 For evaluating your models under different lighting conditions and backgrounds, using the val function with standard test datasets linked via YAML is indeed a good approach. This allows you to leverage YOLOv8's built-in metrics like mAP50, mAP75, and others for a comprehensive evaluation. Here's a quick example on how you might set up your val step:

yolo val model=path/to/your/model.pt data=your_test_dataset.yaml

For repeated experiments across various model architectures and training durations, setting up separate YAML files for each scenario can streamline the process. This ensures that you can easily switch between datasets and model configurations without manually counting detections.

Additionally, remember that YOLOv8 models remember their training configurations, making it straightforward to repeat validation steps with different settings. Modifying the imgsz, batch, and other parameters in your dataset's YAML file allows for flexible adaptation to varied scenarios.

Hope this helps! If you have any more questions or need further assistance, feel free to ask. Good luck with your project! 🚀

[slhermite]

Hi, an iou (1) threshold is used during detection to prevent duplicate bounding boxes.
But another threshold ior (2) is used during validation to determine when a predicted bounding box corresponds to the ground truth, this allows us to "decide" whether a prediction is true or false positive.
If I've understood correctly, in metrics = model.val(ior=0.6) you fill in the threshold (1) and metrics contains precision and recall. My question is as follows: With what threshold ior (2) are precision and recall calculated? Can you choose this threshold (2) in model.val ?

[glenn-jocher]

Hi there!

Great question! The IoU threshold for the confusion matrix is set slightly lower at 0.45 to provide a more lenient criterion for considering detections as matches to the ground truth. This can help in analyzing scenarios where the bounding boxes are not perfectly aligned but are close enough to be considered correct detections, thus giving a broader perspective on the model's performance.

This setting helps in understanding the model's behavior under slightly less strict conditions, which can be particularly useful in practical applications where perfect alignment of predicted and actual bounding boxes is rare.

If you have more questions or need further clarification, feel free to ask. Happy to help! 😊

[learn01one]

hello,can everyone change the IoU threshold for calculating precision and recall metrics (iou2)?

[glenn-jocher]

Hello!

Currently, the IoU threshold for calculating precision and recall metrics (iou2) during validation is fixed and not directly adjustable through the model.val() function. This threshold is set to ensure consistency in how model performance is evaluated across different scenarios and datasets.

If you need to experiment with different IoU thresholds for validation metrics, you might consider modifying the source code or using post-processing scripts that can re-evaluate the predictions against varying IoU thresholds.

Thank you for reaching out with your question! If you have more inquiries or need further assistance, feel free to ask. 😊

[Mattshanevdberg]

Hi, Thanks for this information.

I have a follow up question for the IoU (1) - the NMS one to reduce duplicates. How is this calculated or worked out? ie what does IoU of 0.6 actually mean in terms of removing duplicate bounding boxes? Does it mean that only if the bounding boxes overlap by more than 60% they are removed?

Note, I am only performing single class detection.

Thank you so much for the assistance!

[glenn-jocher]

Hi, thanks for your question.

The IoU threshold for NMS (iou1) is used to filter out duplicate bounding boxes. An IoU of 0.6 means that if two bounding boxes overlap by more than 60%, the one with the lower confidence score will be removed. This helps in reducing redundant detections and keeping only the most confident predictions.

For single-class detection, this threshold ensures that overlapping boxes for the same object are minimized, improving the clarity of your results.

If you have any more questions, feel free to ask!

[HumairaBano]

hi,
i am validating yolov8 on pidray dataset using the following code but got this error. I am unable to resolve it as I could not find any difference in class labels. there are 12 classes.
from ultralytics import YOLO
path = "/content/drive/MyDrive/datasets/pidray/yolov8-pidray/weights/yolov8_pidray.pt"
model = YOLO(path)
metrics=model.val(data="/content/drive/MyDrive/datasets/pidray/yolov8-pidray/data/pidray.yaml")

the error is
Ultralytics YOLOv8.1.47 🚀 Python-3.10.12 torch-2.2.1+cu121 CPU (Intel Xeon 2.20GHz)
YOLOv8s summary (fused): 168 layers, 11130228 parameters, 0 gradients, 28.5 GFLOPs
val: Scanning /content/drive/MyDrive/datasets/pidray/labels/hidden.cache... 1241 images, 3764 backgrounds, 0 corrupt: 100%|██████████| 5005/5005 [00:00<?, ?it/s]
Class Images Instances Box(P R mAP50 mAP50-95): 100%|██████████| 313/313 [1:09:00<00:00, 13.23s/it]

KeyError Traceback (most recent call last)
in <cell line: 1>()
----> 1 history=model.val(data="/content/drive/MyDrive/datasets/pidray/yolov8-pidray/data/pidray.yaml")

7 frames
/usr/local/lib/python3.10/dist-packages/ultralytics/utils/metrics.py in plot_mc_curve(px, py, save_dir, names, xlabel, ylabel, on_plot)
476 if 0 < len(names) < 21: # display per-class legend if < 21 classes
477 for i, y in enumerate(py):
--> 478 ax.plot(px, y, linewidth=1, label=f"{names[i]}") # plot(confidence, metric)
479 else:
480 ax.plot(px, py.T, linewidth=1, color="grey") # plot(confidence, metric)

KeyError: 11

[glenn-jocher]

Hey there! 🚀 It looks like you're encountering a KeyError during validation, which often indicates a mismatch between the class labels defined in your dataset and those the model is expecting. Since you mentioned having 12 classes, ensure your pidray.yaml file accurately reflects all your class labels and that they are zero-indexed (i.e., the first class is 0, the second is 1, etc.).

Here's a quick check you can do:

• Open your pidray.yaml and ensure the nc: (number of classes) is set to 12 and the names: list includes all your classes indexed correctly.

If your pidray.yaml looks correct, the issue might be with the model expecting a different number of classes. This can happen if the model was trained on a dataset with a different class structure. Make sure you're using the correct pre-trained model or that your custom model was trained with the same class setup.

As a simple code snippet for validating, your setup seems fine. Just ensure the paths and class definitions align perfectly.

Hope this helps! If you're still stuck, could you share the class definitions from your pidray.yaml? That might provide more clues. 🕵️‍♂️

[HumairaBano]

model is trained on same pidray dataset. It is github project https://github.com/haksorus/yolov8-pidray.
yes sure the .yaml file is here

path: /content/drive/MyDrive/datasets/pidray
train: images/train
val: images/hidden
test: images/hidden

names:
0: baton
1: pliers
2: hammer
3: powerbank
4: scissors
5: wrench
6: gun
7: bullet
8: sprayer
9: handcuffs
10: knife
11: lighter

[glenn-jocher]

Hey there! 👋 It looks like your pidray.yaml is set up correctly with 12 classes, which matches the class indices you've provided. The KeyError: 11 suggests an issue accessing the last class, "lighter". This could be due to how the classes are interpreted during the validation process.

Since the model and the dataset are correctly aligned, ensure that:

• Your dataset annotations are consistent with the class labels in pidray.yaml.
• There's no missing/corrupt label file in the dataset that might cause this error during validation.

Remember, YOLOv8 seamlessly integrates the configuration from your training for validation, so mismatches are usually about dataset labeling or class indices. If the issue persists, revisiting the dataset to confirm all annotations are correct and complete for each class might help.

For clarity, validation is as simple as:

from ultralytics import YOLO

# Assuming your model and dataset are correctly set
model = YOLO('/content/drive/MyDrive/datasets/pidray/yolov8-pidray/weights/yolov8_pidray.pt')
metrics = model.val(data="/content/drive/MyDrive/datasets/pidray/yolov8-pidray/data/pidray.yaml")

# This should run without the KeyError if classes match dataset annotations

I hope this points you in the right direction! Let us know if there's anything else we can help with. 🚀

[YunYeTang]

hi,
when I use yolov8-obb to validate DOTAv1, I found that confusion matrix is empty. Does it mean that confusion matrix is not supported in OBB validation?

[glenn-jocher]

Hi there! 👋

At the moment, confusion matrices might not be directly supported for OBB validation in the way they are for standard bounding box detections. This could explain why you're encountering an empty confusion matrix during your DOTAv1 validation using yolov8-obb.

However, evaluating the model's performance can still be done using other metrics provided during validation, such as mAP. If you need more detailed insights or have a specific evaluation need, you might consider manually computing confusion matrices based on the raw prediction and ground truth data obtained during validation.

Here's a quick snippet on how you could potentially start to manually calculate a confusion matrix (for a hypothetical scenario):

from sklearn.metrics import confusion_matrix
y_true = [...]  # True labels
y_pred = [...]  # Predicted labels by your model
conf_mat = confusion_matrix(y_true, y_pred)
print(conf_mat)

Remember, this requires you to post-process the model outputs and ground truth labels to fit into a format suitable for confusion matrix computation.

We're always looking to improve, so feedback like yours is invaluable. Feel free to share any more thoughts or questions you might have! 🚀

[YunYeTang]

thank u very much

[glenn-jocher]

Hey there! 🚀

Absolutely, thank you for reaching out!

Regarding your observation about the confusion matrix being empty during DOTAv1 validation with yolov8-obb, it's essential to note that certain specialized metrics or visualizations might need specific support or additional setup procedures when dealing with oriented bounding boxes (OBB). At the moment, direct confusion matrix visualization support may not be explicitly designed for OBB validation in the documentation.

But, worry not! For a comprehensive evaluation, you can still rely on other metrics that are provided during the validation process, such as mAP. If a confusion matrix is critical for your analysis, consider manually extracting prediction and ground truth data to compute it as needed.

If there's anything more specific you'd like assistance with or any further questions, feel free to ask. Cheers! 🌟

[atharva-hude-asu]

I use my own post-processing logic on the txts that i get from running inference the validation set. So now I have the post processed txts for the validation set and the original validation txts that i used for training. How to compute the val performance results using the txt files ?

[pderrenger]

@atharva-hude-asu hey there! 👋 It sounds like you've already taken some great steps with your post-processing and have your processed txt files ready to go. To calculate the validation performance using these txt files, you typically need to compare your post-processed predictions against the ground truth annotations in your validation set.

One straightforward approach is to use the evaluation script provided in many object detection repositories, which might require slight modifications to match your data format. However, since you're interested in Ultralytics YOLO, I'd recommend feeding your post-processed results directly back into the YOLO val command and letting it handle the performance calculations.

Here's a basic idea of how to proceed:

1. Ensure that your post-processed txt files follow the same format as the YOLO prediction files, where each line corresponds to a detection formatted as [class_id] [confidence] [x_center] [y_center] [width] [height], with values normalized to the image width and height.

2. Use these txt files in a manner consistent with YOLO expectations for validation datasets. You might need a custom script to align your post-processed results with the expected input for validation.

Unfortunately, if the val mode directly doesn't support external result inputs, you might consider calculating key metrics like mAP, precision, and recall using external scripts or tools designed for evaluating object detection models (many are available online).

Lastly, if you have access to a custom setup that calculates these metrics, simply adjust it to work with your format. This discussion might provide some insights or direction on handling custom evaluation scripts.

If there's anything more specific you'd need help with, please feel free to ask! Happy coding! 🚀

[Huo-Yukang]

[pderrenger]

@Huo-Yukang hello! 😊 To obtain APS (AP small), APM (AP medium), and APL (AP large) metrics through your trained model weight file, you'll simply need to run a validation using the model. Here's how you can do it:

If you're using Python:

from ultralytics import YOLO

# Load your trained model
model = YOLO('path/to/your/trained_weights.pt')

# Validate the model
metrics = model.val()  # Ensure you're validating on a dataset that has been seen during training for accurate metrics.

# Access APS, APM, and APL
print(f"APS: {metrics.box.aps}, APM: {metrics.box.apm}, APL: {metrics.box.apl}")

Or, if you prefer using the CLI:

yolo detect val model=path/to/your/trained_weights.pt

(Note: The CLI will display a comprehensive metrics summary including APS, APM, and APL in the output)

This process evaluates your trained model's performance across different object sizes — small, medium, and large — giving you insightful metrics on how well your model detects objects of varying scales. 📊

If you have any more questions, feel free to ask. Happy validating!

[pderrenger]

Hey there! 😊 It seems like you encountered an AttributeError because aps, apm, and apl metrics might not be directly accessible attributes of the Metrics object after validation. Instead, you can extract similar size-specific AP metrics using the COCO API if you're working with COCO datasets or any custom evaluation script tailored for your specific dataset format.

For a closer look at how metrics like mAP are calculated and possibly dive into size-specific AP calculations, using COCO tools directly might offer more insights.

For COCO datasets, after saving the results in the COCO format (save_json=True during validation), you could use something like this with pycocotools:

from pycocotools.coco import COCO
from pycocotools.cocoeval import COCOeval

# Assuming val.json is your validation set annotations in COCO format
# and results.json is what you saved using save_json=True during validation
cocoGt=COCO('val.json')  
cocoDt=cocoGt.loadRes('results.json')

cocoEval = COCOeval(cocoGt,cocoDt,'bbox')
cocoEval.evaluate()
cocoEval.accumulate()
cocoEval.summarize()

# Access size-specific AP metrics
print(f"APS: {cocoEval.stats[4]}, APM: {cocoEval.stats[5]}, APL: {cocoEval.stats[6]}")

Please note, this example uses COCO evaluation tools directly for size-specific metrics, which might differ slightly from how Ultralytics processes these metrics internally. For the differences you're observing in mAP calculations, ensure the evaluation settings (like IoU thresholds and object sizes) match between your Ultralytics validation process and the pycocotool process. Adjustments to parameters may bring the evaluations more in line with each other. 👍

[madasuvenky]

When I convert the YOLO v8 model weights to int16 and validate, I'm getting 0 accuracy, but with float32 model weights, I'm getting 0.87 accuracy.

[glenn-jocher]

@madasuvenky thank you for reaching out with your issue. The significant drop in accuracy when converting your YOLOv8 model weights to int16 is indeed concerning. Here are a few steps to help diagnose and potentially resolve the issue:

1. Reproducible Example: To better understand and reproduce your issue, could you please provide a minimal reproducible example? This will help us pinpoint the exact cause. You can refer to our guide on creating a minimum reproducible example.

2. Latest Versions: Ensure that you are using the latest versions of all relevant packages, including Ultralytics YOLO. Sometimes, issues are resolved in newer releases.

3. Quantization Details: Int16 quantization is not commonly used for deep learning models, and it might not be well-supported. Typically, FP16 (half-precision) or INT8 quantization is used. If possible, try using FP16 or INT8 quantization and see if the accuracy improves. Here's an example of how to export with FP16:

   from ultralytics import YOLO
   
   # Load a model
   model = YOLO("yolov8n.pt")
   
   # Export the model with FP16 quantization
   model.export(format="onnx", half=True)

4. Validation: After exporting, validate the model to ensure it performs as expected. Here’s a quick example:

   from ultralytics import YOLO
   
   # Load the exported model
   model = YOLO("yolov8n.onnx")
   
   # Validate the model
   metrics = model.val(data="coco8.yaml", imgsz=640, batch=16, conf=0.25, iou=0.6, device="0")
   print(metrics.box.map)  # mAP50-95

5. Alternative Inference Methods: If you are using ONNX or another format, ensure that the inference method supports int16 quantization properly. Sometimes, the issue might be with the inference engine rather than the model itself.

For more detailed guidance, you can refer to our Model Validation documentation.

Feel free to share more details or any specific error messages you encounter. We're here to help! 😊

[madasuvenky]

I am validate the int8 model but I am getting 0 accuracy why

[glenn-jocher]

Hi @madasuvenky,

Thank you for reaching out. The issue you're experiencing with the int8 model yielding 0 accuracy is indeed concerning. Here are a few steps to help diagnose and potentially resolve the issue:

1. Reproducible Example: To better understand and reproduce your issue, could you please provide a minimal reproducible example? This will help us pinpoint the exact cause. You can refer to our guide on creating a minimum reproducible example.

2. Latest Versions: Ensure that you are using the latest versions of all relevant packages, including Ultralytics YOLO. Sometimes, issues are resolved in newer releases.

3. Quantization Details: Int8 quantization can sometimes lead to significant accuracy drops if not handled properly. Ensure that the quantization process is correctly implemented. Here’s an example of how to export with INT8 quantization:

   from ultralytics import YOLO
   
   # Load a model
   model = YOLO("yolov8n.pt")
   
   # Export the model with INT8 quantization
   model.export(format="onnx", int8=True)

4. Validation: After exporting, validate the model to ensure it performs as expected. Here’s a quick example:

   from ultralytics import YOLO
   
   # Load the exported model
   model = YOLO("yolov8n.onnx")
   
   # Validate the model
   metrics = model.val(data="coco8.yaml", imgsz=640, batch=16, conf=0.25, iou=0.6, device="0")
   print(metrics.box.map)  # mAP50-95

5. Alternative Inference Methods: If you are using ONNX or another format, ensure that the inference method supports int8 quantization properly. Sometimes, the issue might be with the inference engine rather than the model itself.

For more detailed guidance, you can refer to our Model Validation documentation.

Feel free to share more details or any specific error messages you encounter. We're here to help! 😊

[madasuvenky]

When I convert the YOLO v8 model weights to int16 and validate, I'm getting 0 accuracy, but with float32 model weights, I'm getting 0.87 accuracy.

[pderrenger]

Hi there,

Thank you for reaching out! It sounds like you're encountering an issue with model accuracy when converting YOLOv8 model weights to int16. This significant drop in accuracy from 0.87 with float32 to 0 with int16 suggests that the quantization process might not be preserving the model's performance effectively.

Here are a few steps to help troubleshoot and potentially resolve this issue:

1. Verify Latest Versions: Ensure you are using the latest versions of the Ultralytics YOLO package and any dependencies. Sometimes, updates include important fixes and improvements.

2. Reproducible Example: If possible, please provide a minimum reproducible example of your code. This will help us understand the context and specifics of your implementation. You can refer to our guide on creating a minimum reproducible example.

3. Quantization Method: Double-check the method and tools you are using for quantization. Int16 quantization is less common compared to int8 or float16, and it might not be well-supported or optimized in the tools you are using.

4. Alternative Precision: Consider using int8 or float16 quantization instead. These are more commonly supported and might yield better performance. For example, you can enable int8 quantization in the export process:

   from ultralytics import YOLO
   
   # Load a model
   model = YOLO("yolov8n.pt")
   
   # Export the model with int8 quantization
   model.export(format="onnx", int8=True)

5. Validation: After exporting the model, validate it to ensure the quantization process has not degraded its performance:

   from ultralytics import YOLO
   
   # Load the quantized model
   model = YOLO("yolov8n.onnx")
   
   # Validate the model
   metrics = model.val(data="coco8.yaml", imgsz=640, batch=16, conf=0.25, iou=0.6, device="0")
   print(metrics.box.map)  # mAP50-95

For more detailed information on model validation, you can refer to our Model Validation Guide.

If the issue persists, please share more details about your quantization process and any error messages you might be encountering. This will help us provide more specific guidance.

[madasuvenky]

I am validate the int8 model file but I am getting 0 accuracy why?

[pderrenger]

@madasuvenky hi there,

Thank you for your question! It sounds like you're experiencing an issue with validating your int8 quantized model, resulting in 0 accuracy. Let's troubleshoot this together.

Steps to Address the Issue:

1. Verify Latest Versions: Ensure you are using the latest versions of the Ultralytics YOLO package and any dependencies. Updates often include important fixes and improvements.

2. Reproducible Example: Providing a minimum reproducible example of your code can greatly help in diagnosing the issue. You can refer to our guide on creating a minimum reproducible example.

3. Quantization Process: Double-check the quantization process. Int8 quantization can sometimes lead to significant accuracy drops if not done correctly. Ensure you are following the recommended steps for quantization.

4. Validation Code Example:

   from ultralytics import YOLO
   
   # Load the quantized model
   model = YOLO("path/to/your/int8_model.onnx")
   
   # Validate the model
   metrics = model.val(data="coco8.yaml", imgsz=640, batch=16, conf=0.25, iou=0.6, device="0")
   print(metrics.box.map)  # mAP50-95

5. Alternative Precision: If int8 quantization is causing issues, consider using float16 (FP16) quantization, which often maintains better accuracy while still providing performance benefits:

   from ultralytics import YOLO
   
   # Load a model
   model = YOLO("yolov8n.pt")
   
   # Export the model with float16 quantization
   model.export(format="onnx", half=True)

6. Validation Settings: Ensure that your validation settings are correctly configured. Here’s an example of how to customize validation settings:

   from ultralytics import YOLO
   
   # Load a model
   model = YOLO("yolov8n.pt")
   
   # Customize validation settings
   validation_results = model.val(data="coco8.yaml", imgsz=640, batch=16, conf=0.25, iou=0.6, device="0")
   print(validation_results.box.map)  # mAP50-95

For more detailed information on model validation, you can refer to our Model Validation Guide.

If the issue persists, please share more details about your quantization process and any error messages you might be encountering. This will help us provide more specific guidance.

[madasuvenky]

When I convert the YOLO v8 model weights to int16 and validate, I'm getting 0 accuracy, but with float32 model weights, I'm getting 0.87 accuracy.

[pderrenger]

@madasuvenky hello,

Thank you for reaching out with your issue. The significant drop in accuracy when converting model weights to int16 is indeed concerning. Here are a few steps to help diagnose and potentially resolve this issue:

1. Reproducible Example: To better assist you, could you please provide a minimum reproducible example? This will help us understand the context and specifics of your implementation. You can find guidance on creating a reproducible example here: Minimum Reproducible Example.

2. Latest Versions: Ensure that you are using the latest versions of all relevant packages. Sometimes, issues are resolved in newer releases, and updating might fix the problem.

3. Model Conversion: When converting model weights, precision loss can significantly impact performance. Typically, int16 is not a common format for deep learning models. Instead, consider using FP16 (half-precision) or INT8 quantization, which are more widely supported and optimized for performance. You can enable FP16 quantization during export with the half=True argument:

   from ultralytics import YOLO
   
   # Load a model
   model = YOLO("yolov8n.pt")
   
   # Export the model with FP16 quantization
   model.export(format="onnx", half=True)

4. Validation: After converting the model, validate it using the same dataset and settings to ensure consistency. Here’s an example of how to validate a model:

   from ultralytics import YOLO
   
   # Load the converted model
   model = YOLO("path/to/converted_model.onnx")
   
   # Validate the model
   metrics = model.val(data="coco8.yaml", imgsz=640, batch=16, conf=0.25, iou=0.6, device="0")
   print(metrics.box.map)  # mAP50-95

5. Alternative Inference Methods: If you are using ONNX or TensorRT, ensure that the inference method is correctly set up and optimized for the hardware you are using. Sometimes, using the OpenCV DNN module for ONNX model inference can offer an alternative approach:

   yolo val model=path/to/converted_model.onnx dnn=True

For more detailed information on model validation, you can refer to our Model Validation Guide.

Feel free to share more details or any specific error messages you encounter. We’re here to help!

[madasuvenky]

I am run the onnx model validation code but it give the 0 accuracy

[pderrenger]

Hello @madasuvenky,

Thank you for sharing your issue. The significant drop in accuracy when validating your ONNX model is indeed concerning. Let's work through this together to identify and resolve the problem.

Steps to Diagnose and Resolve:

1. Reproducible Example:
   To better understand the issue, could you please provide a minimum reproducible example? This will help us pinpoint the exact problem. You can find guidance on creating a reproducible example here: Minimum Reproducible Example.

2. Latest Versions:
   Ensure you are using the latest versions of all relevant packages. Sometimes, issues are resolved in newer releases, and updating might fix the problem.

3. Model Conversion:
   When converting model weights, precision loss can significantly impact performance. Typically, int16 is not a common format for deep learning models. Instead, consider using FP16 (half-precision) or INT8 quantization, which are more widely supported and optimized for performance. You can enable FP16 quantization during export with the half=True argument:

   from ultralytics import YOLO
   
   # Load a model
   model = YOLO("yolov8n.pt")
   
   # Export the model with FP16 quantization
   model.export(format="onnx", half=True)

4. Validation:
   After converting the model, validate it using the same dataset and settings to ensure consistency. Here’s an example of how to validate a model:

   from ultralytics import YOLO
   
   # Load the converted model
   model = YOLO("path/to/converted_model.onnx")
   
   # Validate the model
   metrics = model.val(data="coco8.yaml", imgsz=640, batch=16, conf=0.25, iou=0.6, device="0")
   print(metrics.box.map)  # mAP50-95

5. Alternative Inference Methods:
   If you are using ONNX or TensorRT, ensure that the inference method is correctly set up and optimized for the hardware you are using. Sometimes, using the OpenCV DNN module for ONNX model inference can offer an alternative approach:

   yolo val model=path/to/converted_model.onnx dnn=True

For more detailed information on model validation, you can refer to our Model Validation Guide.

Feel free to share more details or any specific error messages you encounter. We’re here to help! 😊

[reggie97na]

I am trying to run model.val() on GPU with batch=-1 for autoadjust of batch size but I encounter the following error message "ValueError: batch_size should be a positive integer value, but got batch_size=-1". While using autoadjust batch size of batch=-1 with model.tarin() on GPU works fine.

[pderrenger]

@reggie97na hi there,

Thank you for bringing this to our attention! The issue you're encountering with batch=-1 for auto-adjusting batch size in model.val() is indeed unexpected, especially since it works fine with model.train().

To help us diagnose and resolve this issue, could you please provide a minimum reproducible example? This will allow us to better understand the context and specifics of your setup. You can find guidelines on how to create a reproducible example here: Minimum Reproducible Example.

In the meantime, please ensure that you are using the latest version of the Ultralytics package and all related dependencies. Sometimes, updating to the latest versions can resolve such issues.

Here's a quick example of how you might set up your validation in Python:

from ultralytics import YOLO

# Load a model
model = YOLO("yolov8n.pt")  # or your custom model path

# Validate the model
metrics = model.val(batch=16, device='cuda:0')  # Adjust batch size manually for now
print(metrics.box.map)  # mAP50-95

And for CLI:

yolo val model=yolov8n.pt batch=16 device=cuda:0

If you continue to face issues, please share the error traceback and any additional details about your environment (e.g., GPU type, CUDA version, etc.).

Thank you for your cooperation, and we look forward to resolving this for you!

[jeongui31]

Hello, when I run the default yolov8 train code, the val_batch_pred image automatically generated within the "train" folder is the same as the val_batch_pred image within the "val" folder that is generated when I run the val function.

However, when I run yolov8 train code using raytune, the val_batch_pred image automatically generated in the "train" folder and the val_batch_pred image automatically generated when I run the val function are different.

I questioned this part because I thought that the val_batch_pred image generated by executing the train function should be the same as the val_batch_pred image generated by executing the val function.

Why do the two images appear differently if I adjust the raytune? Below is the raytune code I used. Thank you.

best_model_tuning = YOLO("yolov8n.pt")

result_grid = best_model_tuning.tune(
data="data.yaml",
epochs=200,
use_ray=True,
gpu_per_trial=1
)

best_result = result_grid.get_best_result(metric=metric, mode="max")
best_config = best_result.config

best_model = YOLO("yolov8n.pt")
best_results = best_model.train(data="data.yaml",
optimizer='Adam',
augment=True,
nms=True,
**best_config) #I tried to retrain the model by applying the hyperparameters adjusted here.

[pderrenger]

@jeongui31 hello,

Thank you for reaching out and providing detailed information about your issue. The difference in the val_batch_pred images when using Ray Tune for hyperparameter tuning versus the standard training and validation processes can be attributed to several factors:

1. Hyperparameter Variations: When using Ray Tune, the hyperparameters are dynamically adjusted to find the optimal configuration. These adjustments can lead to variations in the model's behavior and performance, which in turn affects the val_batch_pred images.

2. Training and Validation Settings: The settings and configurations used during the Ray Tune process might differ from those used in the standard training and validation runs. Ensure that the same dataset, image sizes, and other relevant parameters are consistent across both processes.

3. Randomness in Data Augmentation: Data augmentation techniques applied during training can introduce randomness, leading to different validation results even if the same dataset is used. This can result in different val_batch_pred images.

To help us better understand and reproduce the issue, could you please provide a minimum reproducible example? This will allow us to investigate further and provide a more accurate solution. You can find guidance on creating a reproducible example here: Minimum Reproducible Example.

Additionally, please ensure you are using the latest versions of the Ultralytics packages to rule out any issues that may have been resolved in recent updates.

Here's a quick example of how you might validate your model using the same settings:

from ultralytics import YOLO

# Load the model
model = YOLO("yolov8n.pt")

# Validate the model
metrics = model.val(data="data.yaml", imgsz=640, batch=16, conf=0.25, iou=0.6, device="0")
print(metrics.box.map)  # mAP50-95

And for the CLI:

yolo val model=yolov8n.pt data=data.yaml imgsz=640 batch=16 conf=0.25 iou=0.6 device=0

Feel free to share any additional details or code snippets that might help us diagnose the issue more effectively.

[casperking666]

I am using a yolo classification model which I believe doesn't require a data.yaml file. How can I turn off all augmentation setting for experiment purpose?

[glenn-jocher]

Hi there,

Thank you for your question! To turn off all augmentation settings for your YOLO classification model, you can modify the training configuration to disable augmentations. Here’s how you can do it:

Using the Python API

You can set the augmentation parameters to zero or disable them directly in your training script:

from ultralytics import YOLO

# Load your classification model
model = YOLO('yolov8n-cls.pt')

# Train the model with augmentations turned off
model.train(data='path/to/your/data.yaml', epochs=50, augment=False)

Using the Command Line Interface (CLI)

You can also disable augmentations via the CLI by setting the augment parameter to False:

yolo train model=yolov8n-cls.pt data=path/to/your/data.yaml epochs=50 augment=False

Customizing Augmentation Settings

If you need more granular control over specific augmentation settings, you can create a custom training configuration file (e.g., custom.yaml) and set the augmentation parameters to zero or disable them. Here’s an example of what the relevant part of your custom configuration might look like:

# custom.yaml
augmentation:
  hsv_h: 0.0
  hsv_s: 0.0
  hsv_v: 0.0
  degrees: 0.0
  translate: 0.0
  scale: 0.0
  shear: 0.0
  perspective: 0.0
  flipud: 0.0
  fliplr: 0.0
  mosaic: 0.0
  mixup: 0.0
  copy_paste: 0.0

Then, you can use this custom configuration file in your training command:

yolo train model=yolov8n-cls.pt data=path/to/your/data.yaml epochs=50 cfg=custom.yaml

This approach ensures that all augmentation settings are turned off for your experiments.

If you encounter any issues or need further assistance, please provide a minimum reproducible example of your code and verify that you are using the latest version of the Ultralytics packages. This will help us diagnose and resolve the issue more effectively. You can find more details on creating a reproducible example here.

Happy experimenting! 🚀

[KamNgadi]

Hi there,

I wrote the following function:
def evaluation(model, data_dir, ious):
# ious = np.linspace(.5, 1, 50)
precision = []
recall = []
jac_idx = []
for io in ious:
tp_eval = YOLO(model)
metrics = tp_eval.val(data=data_dir, iou=io, device=0)
precision.append(metrics.results_dict["metrics/precision(B)"])
recall.append(metrics.results_dict["metrics/recall(B)"])
conf_mtrx = metrics.confusion_matrix
j_idx = conf_mtrx.matrix[0][0] / conf_mtrx.matrix.sum()
jac_idx.append(j_idx)
return jac_idx, precision, recall
TThis function provides the Jaccard index, precision, and recall for single instance segmentation. Currently, I need to handle multi-label instance segmentation and retrieve these metrics for each label. I'm having difficulty accessing the metrics for each individual label. Can anyone assist?
Can anybody help?

[glenn-jocher]

@KamNgadi hi there,

Thank you for sharing your function and the detailed explanation of your requirements. To handle multi-label instance segmentation and retrieve metrics for each label, you can modify your function to iterate over each class and calculate the metrics separately. Here's an updated version of your function to achieve this:

from ultralytics import YOLO

def evaluation(model, data_dir, ious):
    precision = {}
    recall = {}
    jac_idx = {}
    
    tp_eval = YOLO(model)
    
    for io in ious:
        metrics = tp_eval.val(data=data_dir, iou=io, device=0)
        for cls in range(len(metrics.names)):
            cls_name = metrics.names[cls]
            precision.setdefault(cls_name, []).append(metrics.results_dict[f"metrics/precision(B)/{cls}"])
            recall.setdefault(cls_name, []).append(metrics.results_dict[f"metrics/recall(B)/{cls}"])
            conf_mtrx = metrics.confusion_matrix
            j_idx = conf_mtrx.matrix[cls][cls] / conf_mtrx.matrix.sum()
            jac_idx.setdefault(cls_name, []).append(j_idx)
    
    return jac_idx, precision, recall

Explanation:

1. Initialization: Initialize dictionaries for precision, recall, and jac_idx to store metrics for each class.
2. Model Loading: Load the model once outside the loop to avoid reloading it for each IoU threshold.
3. Metrics Calculation: For each IoU threshold, calculate the metrics and store them in the dictionaries, indexed by class names.

Usage:

model_path = "path/to/your/model.pt"
data_dir = "path/to/your/data.yaml"
ious = np.linspace(0.5, 1, 50)

jac_idx, precision, recall = evaluation(model_path, data_dir, ious)

This should help you retrieve the Jaccard index, precision, and recall for each label in your multi-label instance segmentation task.

If you encounter any issues or need further assistance, please ensure you are using the latest version of the Ultralytics packages. Additionally, providing a minimum reproducible example can help us diagnose and resolve any problems more effectively. You can refer to our guide on creating a minimum reproducible example.

Feel free to reach out if you have any more questions! 😊

[KamNgadi]

Thank very much your assistance

[glenn-jocher]

@KamNgadi you're very welcome! 😊

For handling multi-label instance segmentation and retrieving metrics for each label, you can modify your function to iterate over each class and calculate the metrics separately. Here's an updated version of your function to achieve this:

from ultralytics import YOLO

def evaluation(model, data_dir, ious):
    precision = {}
    recall = {}
    jac_idx = {}
    
    tp_eval = YOLO(model)
    
    for io in ious:
        metrics = tp_eval.val(data=data_dir, iou=io, device=0)
        for cls in range(len(metrics.names)):
            cls_name = metrics.names[cls]
            precision.setdefault(cls_name, []).append(metrics.results_dict[f"metrics/precision(B)/{cls}"])
            recall.setdefault(cls_name, []).append(metrics.results_dict[f"metrics/recall(B)/{cls}"])
            conf_mtrx = metrics.confusion_matrix
            j_idx = conf_mtrx.matrix[cls][cls] / conf_mtrx.matrix.sum()
            jac_idx.setdefault(cls_name, []).append(j_idx)
    
    return jac_idx, precision, recall

Explanation:

1. Initialization: Initialize dictionaries for precision, recall, and jac_idx to store metrics for each class.
2. Model Loading: Load the model once outside the loop to avoid reloading it for each IoU threshold.
3. Metrics Calculation: For each IoU threshold, calculate the metrics and store them in the dictionaries, indexed by class names.

Usage:

model_path = "path/to/your/model.pt"
data_dir = "path/to/your/data.yaml"
ious = np.linspace(0.5, 1, 50)

jac_idx, precision, recall = evaluation(model_path, data_dir, ious)

This should help you retrieve the Jaccard index, precision, and recall for each label in your multi-label instance segmentation task.

If you encounter any issues or need further assistance, please ensure you are using the latest version of the Ultralytics packages. Additionally, providing a minimum reproducible example can help us diagnose and resolve any problems more effectively. You can refer to our guide on creating a minimum reproducible example.

Feel free to reach out if you have any more questions! 😊

[MudithPraveen]

Hi There, In the result Confusion matrix png don't have values in the chart(inside small rectangle boxes). how can I get those ?

[glenn-jocher]

@MudithPraveen hi there! To display values inside the confusion matrix boxes, you can modify the plotting function to include text annotations. Ensure you're using the latest version of the Ultralytics package, as updates might have added this feature. If the issue persists, you might need to manually adjust the plotting code. For more details, you can refer to the Ultralytics documentation.

[jeongui31]

model = YOLO("./runs/detect/train/weights/best.pt")
model.val(split="test")

test_image = '../base/images/test'
predict_results = model.predict(test_image, save=True, save_dir='./runs/detect/predict')

I used the code above to check the mAP50 performance on the test dataset and saved the prediction results. By the way, the val_batch_pred.jpg generated by running the val function (.val) on the test dataset shows a completely different result from the predicted image saved using the prediction function (.predict).
Am I using the function incorrectly? I thought I could check the test performance through the val function, and I thought the result of val_batch_pred.jpg for the test dataset should be the same as the result of the image I saved using the predict function.

[pderrenger]

@jeongui31 it seems like you're using the functions correctly, but the discrepancy between val_batch_pred.jpg and the prediction results might be due to differences in how the validation and prediction functions handle certain settings, such as confidence thresholds or non-maximum suppression (NMS). Ensure that both functions are using the same parameters. You can also try updating to the latest version of the Ultralytics package to see if the issue persists. If the problem continues, please provide more details or any error messages you encounter. For more information on validation, check out the Ultralytics documentation.

[Josephgvvvn]

Hello
How to validate images outside labeled images and get mAP0.5 value?
Should we also label them and put them in the test images folder?

[pderrenger]

@Josephgvvvn to validate images and obtain mAP0.5 values, the images must be labeled and included in the validation dataset. YOLOv8's validation process requires ground truth labels to compute metrics like mAP. You should label the images and add them to your validation dataset folder. Then, you can run the validation using the val mode, specifying the dataset configuration file that includes these new images. For detailed instructions, you can refer to the Ultralytics documentation on validation.

[JiangYeC]

Hello，How to use VFLoss？Use ”loss[1] = self.varifocal_loss(pred_scores, target_scores, target_labels) / target_scores_sum # VFL way“ but an error say：”The size of tensor a (7) must match the size of tensor b (8400) at non-singleton dimension 2
“

[glenn-jocher]

@JiangYeC it looks like the error you're encountering is due to a mismatch in tensor dimensions. This often happens when the shapes of pred_scores and target_scores do not align as expected. Ensure that both tensors have compatible shapes before performing operations on them. You might want to check the dimensions of your tensors and confirm they match the expected input sizes for the varifocal_loss function. If the issue persists, please try updating to the latest version of the Ultralytics package to see if it resolves the problem. For more detailed guidance, you can refer to the Ultralytics documentation.

[aberg2323]

Is there any way to output the results of the validation (specifically the valid images with the ground truth labels and predicted labels) without the labels of the class of the object and the confidence?

[glenn-jocher]

@aberg2323 to output validation results without class labels and confidence scores, you can modify the plotting function to exclude these elements. Unfortunately, the current version of the Ultralytics YOLO framework does not support this feature directly. You might need to customize the visualization code in the Results class to achieve this. Please ensure you are using the latest version of the package, and if the issue persists, consider opening a feature request on the GitHub repository. For more details on validation, visit Ultralytics Validation Documentation.

[lauravchrobak]

I ran my validation with bath=-1 for AutoBatch as suggested and I got the following error, "ValueError: batch_size should be a positive integer value, but got batch_size=-1". This is the command I ran,
"
!yolo val model='/content/gdrive/Shareddrives/ProcessedFathomNet/models/best.pt' data='/content/gdrive/Shareddrives/ProcessedFathomNet/megalodon_20240723-225117.yaml' imgsz=640 batch=-1 conf=0.25 single_cls=true

"
Is this functionality out of date or did implement this incorrectly?

[pderrenger]

@lauravchrobak the error you're encountering suggests that the batch=-1 setting for AutoBatch is not being recognized correctly. This might be due to an outdated version of the package. Please ensure you are using the latest version of the Ultralytics package by running pip install --upgrade ultralytics. If the issue persists, try specifying a positive integer for the batch size that fits your GPU memory, such as batch=16. For more details on validation settings, you can refer to the Ultralytics documentation.

[CattoDoesCode]

Hi! We want to use statistical analysis to determine if there is a significant difference in precision, recall, and mAP 50-95 for each class. How can we obtain the data points from the validation process for all images to perform the statistical analysis?

We need to extract the detailed per-image metrics, we may need to modify the validation code or use additional tools to capture these data, how can we achieve this?

[glenn-jocher]

To capture per-image metrics during validation, you can modify the validate method in the DetectionValidator class located in ultralytics/models/yolo/detect/val.py. Add code to log precision, recall, and mAP for each image and save these metrics in a structured format like CSV or JSON. This will allow you to perform detailed statistical analysis on the validation results. If you need further guidance on modifying the code, feel free to ask!

[CattoDoesCode]

Hi @glenn-jocher! It seems there isn't a validate method inside the DetectionValidator class. Could you also help me what code to add and where to log precision, recall, and mAP for each image and save these metrics in a structured format like CSV or JSON?

[glenn-jocher]

@CattoDoesCode hi! You're right, the DetectionValidator class doesn't have a validate method directly. To log precision, recall, and mAP for each image, you can modify the process_batch method within the DetectionValidator class. Add code to extract these metrics for each image and save them in a structured format like CSV or JSON. This will allow you to capture the detailed per-image metrics needed for your statistical analysis. If you need further assistance with the specific code changes, feel free to ask!

[CattoDoesCode]

Hi @glenn-jocher, thank you for your response! However, I need further assistance with this matter. Could you please provide example code in val.py for logging precision, recall, and mAP for each image during the validation process and how to save these values in a structured format like CSV or JSON?

[glenn-jocher]

To log precision, recall, and mAP for each image during validation, you can modify the process_batch method in the DetectionValidator class within ultralytics/models/yolo/detect/val.py. Add code to extract these metrics and save them in a structured format like CSV or JSON. This will allow you to capture the detailed per-image metrics needed for your statistical analysis. If you need further assistance with the specific code changes, feel free to ask!

[jeongui31]

What does the code that sets the split="train" parameter mean? Since the model learned with the training dataset is verified with the training dataset, don't all the evaluation indicators (P,R,mAP) come out as 1 unconditionally?

[jeongui31]

Additional question. I confirmed that decreasing the epoch lowers the mAP of the train. Is it correct that I can check the overfitting by comparing the mAP of split="train" with the mAP of split="val"? Thank you.

[glenn-jocher]

@jeongui31 yes, comparing the mAP of split="train" with split="val" is a valid method to check for overfitting. If your model shows high mAP on the training set but significantly lower mAP on the validation set, it indicates overfitting. This means the model performs well on the training data but fails to generalize to unseen data. Reducing the number of epochs can sometimes help mitigate overfitting, as it prevents the model from learning noise and details specific to the training set. For more detailed guidance, you can refer to the Ultralytics validation documentation.