val.py csv file outputting feature for comparison of a model performance along different test datasets

[CemEntok]

Search before asking

• I have searched the YOLOv5 issues and found no similar feature requests.

Description

Is it possible to add a feature to output metric values as continuous data when val.py code is run?

Use case

By having this, it would be possible to compare metric outputs of a model for different test datasets, but in current metric outputting, only images are outputted which makes comparison of performance of a model for different datasets harder.

Additional

Thank you

Are you willing to submit a PR?

• Yes I'd like to help by submitting a PR!

[github-actions]

👋 Hello @CemEntok, thank you for your interest in YOLOv5 🚀! Please visit our ⭐️ Tutorials to get started, where you can find quickstart guides for simple tasks like Custom Data Training all the way to advanced concepts like Hyperparameter Evolution.

If this is a 🐛 Bug Report, please provide a minimum reproducible example to help us debug it.

If this is a custom training ❓ Question, please provide as much information as possible, including dataset image examples and training logs, and verify you are following our Tips for Best Training Results.

Requirements

Python>=3.7.0 with all requirements.txt installed including PyTorch>=1.7. To get started:

git clone https://github.com/ultralytics/yolov5  # clone
cd yolov5
pip install -r requirements.txt  # install

Environments

YOLOv5 may be run in any of the following up-to-date verified environments (with all dependencies including CUDA/CUDNN, Python and PyTorch preinstalled):

• Notebooks with free GPU:
• Google Cloud Deep Learning VM. See GCP Quickstart Guide
• Amazon Deep Learning AMI. See AWS Quickstart Guide
• Docker Image. See Docker Quickstart Guide

Status

If this badge is green, all YOLOv5 GitHub Actions Continuous Integration (CI) tests are currently passing. CI tests verify correct operation of YOLOv5 training, validation, inference, export and benchmarks on macOS, Windows, and Ubuntu every 24 hours and on every commit.

Introducing YOLOv8 🚀

We're excited to announce the launch of our latest state-of-the-art (SOTA) object detection model for 2023 - YOLOv8 🚀!

Designed to be fast, accurate, and easy to use, YOLOv8 is an ideal choice for a wide range of object detection, image segmentation and image classification tasks. With YOLOv8, you'll be able to quickly and accurately detect objects in real-time, streamline your workflows, and achieve new levels of accuracy in your projects.

Check out our YOLOv8 Docs for details and get started with:

pip install ultralytics

[glenn-jocher]

Hello @CemEntok,

Thank you for your request. We appreciate your interest in improving YOLOv8.

We can definitely consider implementing the feature you have requested, however, we need more information from you regarding the "continuous data" output you are suggesting. Could you please provide us with more details? Examples or diagrams would be really helpful.

Looking forward to hearing back from you!

[CemEntok]

Hello and thank you for your fast response!

Actually I am not familiar with yolov8, since I am using Yolov5 but I assume they have similar features.

Let me make the question clearer with giving an example. Lets say I have trained a custom yolov5s model for my custom data.

It produces PR curves, F1-score etc. as metrics. Also, those metrics are saved for every epochs while training the model. However, resulted PR curves, F1-scores are plotted as .png file only, so they are not comparable when it comes to compare to trained model performance in the same figure.

It is really good to have this figure, but if we could have its datapoints as .csv file for example, we can compare PR curves of different trained models in only one figure.

This feature is even more important for validating a model, since at any point, there is a comparison of two models solving the same problem by evaluating their performance over the same test dataset. In this case, comparon of their PR curves in only one figure would be super useful.

I know that I could also add this feature by myself by finding the plotting line of these curves in metrics.py possibly and saving the parameters, a.k.a, precision and recall datapoints as csv, but I think it might be good and easy feature to add in the model, or maybe there is a feature doing that already and I am missing to find it, that's why I also asking this question.

Thank you for your time and help!

[CemEntok]

Looks like there is a function in plots.py file which can help compare metrics of different training schedules:

 def plot_results(file='path/to/results.csv', dir=''):
    # Plot training results.csv. Usage: from utils.plots import *; plot_results('path/to/results.csv')
    save_dir = Path(file).parent if file else Path(dir)
    fig, ax = plt.subplots(2, 5, figsize=(12, 6), tight_layout=True)
    ax = ax.ravel()
    files = list(save_dir.glob('results*.csv'))
    assert len(files), f'No results.csv files found in {save_dir.resolve()}, nothing to plot.'
    for f in files:
        try:
            data = pd.read_csv(f)
            s = [x.strip() for x in data.columns]
            x = data.values[:, 0]
            for i, j in enumerate([1, 2, 3, 4, 5, 8, 9, 10, 6, 7]):
                y = data.values[:, j].astype('float')
                # y[y == 0] = np.nan  # don't show zero values
                ax[i].plot(x, y, marker='.', label=f.stem, linewidth=2, markersize=8)
                ax[i].set_title(s[j], fontsize=12)
                # if j in [8, 9, 10]:  # share train and val loss y axes
                #     ax[i].get_shared_y_axes().join(ax[i], ax[i - 5])
        except Exception as e:
            LOGGER.info(f'Warning: Plotting error for {f}: {e}')
    ax[1].legend()
    fig.savefig(save_dir / 'results.png', dpi=200)
    plt.close()

[glenn-jocher]

Hello @CemEntok,

Have you tried using the plot_results() function in the plots.py file? This function is already implemented in YOLOv5 and produces a single figure which includes multiple metrics for different variations of a given model like precision, recall, mAP, loss and other performance metrics over epochs. By using this function, you may plot your results on a .png file and compare model performances.

However, if you still wish to get the datapoints of the plots captured instead of a .png file output, you may want to consider modifying the plot_results() function accordingly to save the data in a file format you need such as .csv or .txt.

Hope this helps! Let us know if you have further questions.

[abdalluhahmed]

Hello @glenn-jocher,

I hope this message finds you well. I am currently working with YOLOv8 for object detection tasks, and I've come across your discussions related to saving validation and testing results as .csv files. I find this approach quite useful for post-processing and analysis.

I would greatly appreciate it if you could provide some guidance on how to implement this feature. Specifically, I'm interested in understanding how I can modify the code to capture prediction results during validation and testing and then save them into a .csv file format. I've gone through the codebase, but I'm unsure about the best way to extract the necessary information and structure it for CSV output.

If you could share some code snippets, pointers, or steps to follow, I would be extremely grateful. Your insights would undoubtedly help me and the community better understand and utilize this functionality within YOLOv8.

Thank you for your time and expertise. Looking forward to your response.

Best regards,

[abdalluhahmed]

here just for training as .csv

@threaded
def plot_images(images,
batch_idx,
cls,
bboxes=np.zeros(0, dtype=np.float32),
masks=np.zeros(0, dtype=np.uint8),
kpts=np.zeros((0, 51), dtype=np.float32),
paths=None,
fname='images.jpg',
names=None,
on_plot=None):
"""Plot image grid with labels."""
if isinstance(images, torch.Tensor):
images = images.cpu().float().numpy()
if isinstance(cls, torch.Tensor):
cls = cls.cpu().numpy()
if isinstance(bboxes, torch.Tensor):
bboxes = bboxes.cpu().numpy()
if isinstance(masks, torch.Tensor):
masks = masks.cpu().numpy().astype(int)
if isinstance(kpts, torch.Tensor):
kpts = kpts.cpu().numpy()
if isinstance(batch_idx, torch.Tensor):
batch_idx = batch_idx.cpu().numpy()

max_size = 1920  # max image size
max_subplots = 16  # max image subplots, i.e. 4x4
bs, _, h, w = images.shape  # batch size, _, height, width
bs = min(bs, max_subplots)  # limit plot images
ns = np.ceil(bs ** 0.5)  # number of subplots (square)
if np.max(images[0]) <= 1:
    images *= 255  # de-normalise (optional)

# Build Image
mosaic = np.full((int(ns * h), int(ns * w), 3), 255, dtype=np.uint8)  # init
for i, im in enumerate(images):
    if i == max_subplots:  # if last batch has fewer images than we expect
        break
    x, y = int(w * (i // ns)), int(h * (i % ns))  # block origin
    im = im.transpose(1, 2, 0)
    mosaic[y:y + h, x:x + w, :] = im

# Resize (optional)
scale = max_size / ns / max(h, w)
if scale < 1:
    h = math.ceil(scale * h)
    w = math.ceil(scale * w)
    mosaic = cv2.resize(mosaic, tuple(int(x * ns) for x in (w, h)))

# Annotate
fs = int((h + w) * ns * 0.01)  # font size
annotator = Annotator(mosaic, line_width=round(fs / 10), font_size=fs, pil=True, example=names)
for i in range(i + 1):
    x, y = int(w * (i // ns)), int(h * (i % ns))  # block origin
    annotator.rectangle([x, y, x + w, y + h], None, (255, 255, 255), width=2)  # borders
    if paths:
        annotator.text((x + 5, y + 5), text=Path(paths[i]).name[:40], txt_color=(220, 220, 220))  # filenames
    if len(cls) > 0:
        idx = batch_idx == i
        classes = cls[idx].astype('int')

        if len(bboxes):
            boxes = xywh2xyxy(bboxes[idx, :4]).T
            labels = bboxes.shape[1] == 4  # labels if no conf column
            conf = None if labels else bboxes[idx, 4]  # check for confidence presence (label vs pred)

            if boxes.shape[1]:
                if boxes.max() <= 1.01:  # if normalized with tolerance 0.01
                    boxes[[0, 2]] *= w  # scale to pixels
                    boxes[[1, 3]] *= h
                elif scale < 1:  # absolute coords need scale if image scales
                    boxes *= scale
            boxes[[0, 2]] += x
            boxes[[1, 3]] += y
            for j, box in enumerate(boxes.T.tolist()):
                c = classes[j]
                color = colors(c)
                c = names.get(c, c) if names else c
                if labels or conf[j] > 0.25:  # 0.25 conf thresh
                    label = f'{c}' if labels else f'{c} {conf[j]:.1f}'
                    annotator.box_label(box, label, color=color)
        elif len(classes):
            for c in classes:
                color = colors(c)
                c = names.get(c, c) if names else c
                annotator.text((x, y), f'{c}', txt_color=color, box_style=True)

        # Plot keypoints
        if len(kpts):
            kpts_ = kpts[idx].copy()
            if len(kpts_):
                if kpts_[..., 0].max() <= 1.01 or kpts_[..., 1].max() <= 1.01:  # if normalized with tolerance .01
                    kpts_[..., 0] *= w  # scale to pixels
                    kpts_[..., 1] *= h
                elif scale < 1:  # absolute coords need scale if image scales
                    kpts_ *= scale
            kpts_[..., 0] += x
            kpts_[..., 1] += y
            for j in range(len(kpts_)):
                if labels or conf[j] > 0.25:  # 0.25 conf thresh
                    annotator.kpts(kpts_[j])

        # Plot masks
        if len(masks):
            if idx.shape[0] == masks.shape[0]:  # overlap_masks=False
                image_masks = masks[idx]
            else:  # overlap_masks=True
                image_masks = masks[[i]]  # (1, 640, 640)
                nl = idx.sum()
                index = np.arange(nl).reshape((nl, 1, 1)) + 1
                image_masks = np.repeat(image_masks, nl, axis=0)
                image_masks = np.where(image_masks == index, 1.0, 0.0)

            im = np.asarray(annotator.im).copy()
            for j, box in enumerate(boxes.T.tolist()):
                if labels or conf[j] > 0.25:  # 0.25 conf thresh
                    color = colors(classes[j])
                    mh, mw = image_masks[j].shape
                    if mh != h or mw != w:
                        mask = image_masks[j].astype(np.uint8)
                        mask = cv2.resize(mask, (w, h))
                        mask = mask.astype(bool)
                    else:
                        mask = image_masks[j].astype(bool)
                    with contextlib.suppress(Exception):
                        im[y:y + h, x:x + w, :][mask] = im[y:y + h, x:x + w, :][mask] * 0.4 + np.array(color) * 0.6
            annotator.fromarray(im)
annotator.im.save(fname)  # save
if on_plot:
    on_plot(fname)

@plt_settings()
def plot_results(file='path/to/results.csv', dir='', segment=False, pose=False, classify=False, on_plot=None):
"""
Plot training results from results CSV file.

Example:
    ```python
    from ultralytics.utils.plotting import plot_results

    plot_results('path/to/results.csv')
    ```
"""
import pandas as pd
from scipy.ndimage import gaussian_filter1d
save_dir = Path(file).parent if file else Path(dir)
if classify:
    fig, ax = plt.subplots(2, 2, figsize=(6, 6), tight_layout=True)
    index = [1, 4, 2, 3]
elif segment:
    fig, ax = plt.subplots(2, 8, figsize=(18, 6), tight_layout=True)
    index = [1, 2, 3, 4, 5, 6, 9, 10, 13, 14, 15, 16, 7, 8, 11, 12]
elif pose:
    fig, ax = plt.subplots(2, 9, figsize=(21, 6), tight_layout=True)
    index = [1, 2, 3, 4, 5, 6, 7, 10, 11, 14, 15, 16, 17, 18, 8, 9, 12, 13]
else:
    fig, ax = plt.subplots(2, 5, figsize=(12, 6), tight_layout=True)
    index = [1, 2, 3, 4, 5, 8, 9, 10, 6, 7]
ax = ax.ravel()
files = list(save_dir.glob('results*.csv'))
assert len(files), f'No results.csv files found in {save_dir.resolve()}, nothing to plot.'
for f in files:
    try:
        data = pd.read_csv(f)
        s = [x.strip() for x in data.columns]
        x = data.values[:, 0]
        for i, j in enumerate(index):
            y = data.values[:, j].astype('float')
            # y[y == 0] = np.nan  # don't show zero values
            ax[i].plot(x, y, marker='.', label=f.stem, linewidth=2, markersize=8)  # actual results
            ax[i].plot(x, gaussian_filter1d(y, sigma=3), ':', label='smooth', linewidth=2)  # smoothing line
            ax[i].set_title(s[j], fontsize=12)
            # if j in [8, 9, 10]:  # share train and val loss y axes
            #     ax[i].get_shared_y_axes().join(ax[i], ax[i - 5])
    except Exception as e:
        LOGGER.warning(f'WARNING: Plotting error for {f}: {e}')
ax[1].legend()
fname = save_dir / 'results.png'
fig.savefig(fname, dpi=200)
plt.close()
if on_plot:
    on_plot(fname)

[glenn-jocher]

@abdalluhahmed hi there,

I see that you are working on a function called plot_images in the YOLOv5 repository. The function takes in various arguments and plots a grid of images with labels, bounding boxes, masks, and keypoints.

From the code, it seems like the function first converts the input tensors to numpy arrays and then builds the image grid. It resizes the images if the size exceeds the maximum limit and annotates the images with labels, boxes, masks, and keypoints. Finally, it saves the annotated grid as an image file.

The code looks well-implemented, and it seems to cover various use cases such as plotting bounding boxes, masks, and keypoints. However, if you have any specific issue or question regarding this code or if you need any further assistance, please let me know.

Thank you!