请问NMS及其改进？

[Mengyao-Zhang]

Search before asking

• I have searched the YOLOv5 issues and discussions and found no similar questions.

Question

您好！jkocherhans 。最近我尝试在YOLOv5中改进soft-NMS及其变形，但是效果很不好，希望得到你的帮助
1、请问NMS在v5训练阶段有使用么？
2、改进后的模型，训练很慢且效果不好，是我的代码有bug么？
3、我可以将模型正常训练但是使用改进后的增强NMS进行测试么？您认为这会是有效的改进么？

Additional

def bbox_iou_for_nms(box1, box2, xywh=True, GIoU=False, DIoU=False, CIoU=False, SIoU=False, EIoU=False, WIoU=False,
                     Focal=False, alpha=1, gamma=0.5, scale=False, `eps=1e-7):
    # Returns Intersection over Union (IoU) of box1(1,4) to box2(n,4)

    # Get the coordinates of bounding boxes
    if xywh:  # transform from xywh to xyxy
        (x1, y1, w1, h1), (x2, y2, w2, h2) = box1.chunk(4, -1), box2.chunk(4, -1)
        w1_, h1_, w2_, h2_ = w1 / 2, h1 / 2, w2 / 2, h2 / 2
        b1_x1, b1_x2, b1_y1, b1_y2 = x1 - w1_, x1 + w1_, y1 - h1_, y1 + h1_
        b2_x1, b2_x2, b2_y1, b2_y2 = x2 - w2_, x2 + w2_, y2 - h2_, y2 + h2_
    else:  # x1, y1, x2, y2 = box1
        b1_x1, b1_y1, b1_x2, b1_y2 = box1.chunk(4, -1)
        b2_x1, b2_y1, b2_x2, b2_y2 = box2.chunk(4, -1)
        w1, h1 = b1_x2 - b1_x1, (b1_y2 - b1_y1).clamp(eps)
        w2, h2 = b2_x2 - b2_x1, (b2_y2 - b2_y1).clamp(eps)

    # Intersection area
    inter = (b1_x2.minimum(b2_x2) - b1_x1.maximum(b2_x1)).clamp(0) * \
            (b1_y2.minimum(b2_y2) - b1_y1.maximum(b2_y1)).clamp(0)

    # Union Area
    union = w1 * h1 + w2 * h2 - inter + eps
    if scale:
        self = WIoU_Scale(1 - (inter / union))

    # IoU
    # iou = inter / union # ori iou
    iou = torch.pow(inter / (union + eps), alpha)  # alpha iou
    if CIoU or DIoU or GIoU or EIoU or SIoU or WIoU:
        cw = b1_x2.maximum(b2_x2) - b1_x1.minimum(b2_x1)  # convex (smallest enclosing box) width
        ch = b1_y2.maximum(b2_y2) - b1_y1.minimum(b2_y1)  # convex height
        if CIoU or DIoU or EIoU or SIoU or WIoU:  # Distance or Complete IoU https://arxiv.org/abs/1911.08287v1
            c2 = (cw ** 2 + ch ** 2) ** alpha + eps  # convex diagonal squared
            rho2 = (((b2_x1 + b2_x2 - b1_x1 - b1_x2) ** 2 + (
                        b2_y1 + b2_y2 - b1_y1 - b1_y2) ** 2) / 4) ** alpha  # center dist ** 2
            if CIoU:  # https://github.com/Zzh-tju/DIoU-SSD-pytorch/blob/master/utils/box/box_utils.py#L47
                v = (4 / math.pi ** 2) * (torch.atan(w2 / h2) - torch.atan(w1 / h1)).pow(2)
                with torch.no_grad():
                    alpha_ciou = v / (v - iou + (1 + eps))
                if Focal:
                    return iou - (rho2 / c2 + torch.pow(v * alpha_ciou + eps, alpha)), torch.pow(inter / (union + eps),
                                                                                                 gamma)  # Focal_CIoU
                else:
                    return iou - (rho2 / c2 + torch.pow(v * alpha_ciou + eps, alpha))  # CIoU
            elif EIoU:
                rho_w2 = ((b2_x2 - b2_x1) - (b1_x2 - b1_x1)) ** 2
                rho_h2 = ((b2_y2 - b2_y1) - (b1_y2 - b1_y1)) ** 2
                cw2 = torch.pow(cw ** 2 + eps, alpha)
                ch2 = torch.pow(ch ** 2 + eps, alpha)
                if Focal:
                    return iou - (rho2 / c2 + rho_w2 / cw2 + rho_h2 / ch2), torch.pow(inter / (union + eps),
                                                                                      gamma)  # Focal_EIou
                else:
                    return iou - (rho2 / c2 + rho_w2 / cw2 + rho_h2 / ch2)  # EIou
            elif SIoU:
                # SIoU Loss https://arxiv.org/pdf/2205.12740.pdf
                s_cw = (b2_x1 + b2_x2 - b1_x1 - b1_x2) * 0.5 + eps
                s_ch = (b2_y1 + b2_y2 - b1_y1 - b1_y2) * 0.5 + eps
                sigma = torch.pow(s_cw ** 2 + s_ch ** 2, 0.5)
                sin_alpha_1 = torch.abs(s_cw) / sigma
                sin_alpha_2 = torch.abs(s_ch) / sigma
                threshold = pow(2, 0.5) / 2
                sin_alpha = torch.where(sin_alpha_1 > threshold, sin_alpha_2, sin_alpha_1)
                angle_cost = torch.cos(torch.arcsin(sin_alpha) * 2 - math.pi / 2)
                rho_x = (s_cw / cw) ** 2
                rho_y = (s_ch / ch) ** 2
                gamma = angle_cost - 2
                distance_cost = 2 - torch.exp(gamma * rho_x) - torch.exp(gamma * rho_y)
                omiga_w = torch.abs(w1 - w2) / torch.max(w1, w2)
                omiga_h = torch.abs(h1 - h2) / torch.max(h1, h2)
                shape_cost = torch.pow(1 - torch.exp(-1 * omiga_w), 4) + torch.pow(1 - torch.exp(-1 * omiga_h), 4)
                if Focal:
                    return iou - torch.pow(0.5 * (distance_cost + shape_cost) + eps, alpha), torch.pow(
                        inter / (union + eps), gamma)  # Focal_SIou
                else:
                    return iou - torch.pow(0.5 * (distance_cost + shape_cost) + eps, alpha)  # SIou
            elif WIoU:
                if Focal:
                    raise RuntimeError("WIoU do not support Focal.")
                elif scale:
                    return getattr(WIoU_Scale, '_scaled_loss')(self), (1 - iou) * torch.exp(
                        (rho2 / c2)), iou  # WIoU https://arxiv.org/abs/2301.10051
                else:
                    return iou, torch.exp((rho2 / c2))  # WIoU v1
            if Focal:
                return iou - rho2 / c2, torch.pow(inter / (union + eps), gamma)  # Focal_DIoU
            else:
                return iou - rho2 / c2  # DIoU
        c_area = cw * ch + eps  # convex area
        if Focal:
            return iou - torch.pow((c_area - union) / c_area + eps, alpha), torch.pow(inter / (union + eps),
                                                                                      gamma)  # Focal_GIoU https://arxiv.org/pdf/1902.09630.pdf
        else:
            return iou - torch.pow((c_area - union) / c_area + eps, alpha)  # GIoU https://arxiv.org/pdf/1902.09630.pdf
    if Focal:
        return iou, torch.pow(inter / (union + eps), gamma)  # Focal_IoU
    else:
        return iou  # IoU

def soft_nms(bboxes, scores, iou_thresh=0.5, sigma=0.5, score_threshold=0.25):
    order = torch.arange(0, scores.size(0)).to(bboxes.device)
    keep = []
    while order.numel() > 1:
        if order.numel() == 1:
            keep.append(order[0])
            break
        else:
            i = order[0]
            keep.append(i)
        iou = bbox_iou_for_nms(bboxes[i], bboxes[order[1:]]).squeeze()
        idx = (iou > iou_thresh).nonzero().squeeze()
        if idx.numel() > 0:
            iou = iou[idx]
            newScores = torch.exp(-torch.pow(iou, 2) / sigma)
            scores[order[idx + 1]] *= newScores
        newOrder = (scores[order[1:]] > score_threshold).nonzero().squeeze()
        if newOrder.numel() == 0:
            break
        else:
            maxScoreIndex = torch.argmax(scores[order[newOrder + 1]])
            if maxScoreIndex != 0:
                newOrder[[0, maxScoreIndex],] = newOrder[[maxScoreIndex, 0],]
            order = order[newOrder + 1]

    return torch.LongTensor(keep)

[Mengyao-Zhang]

@jkocherhans 感谢，盼复！

[glenn-jocher]

@18846169373 hi there,

Thank you for bringing up your questions and sharing your code improvement attempts for soft-NMS in YOLOv5. I'll be happy to assist you.

To answer your questions:

1. In YOLOv5, NMS (Non-Maximum Suppression) is not used during the training phase. It is typically applied during the post-processing stage to remove duplicate and overlapping bounding boxes.

2. If the performance of your model is slow and the results are not satisfactory after implementing your code improvements, there could be a bug in your code. I suggest double-checking your code implementation and verifying that you are using the correct variables and parameters.

3. Yes, you can use your improved NMS method for testing the trained model. However, whether it will be an effective improvement depends on various factors, such as the dataset, the specific problem you are trying to solve, and the characteristics of the bounding boxes. It would be advisable to thoroughly evaluate and compare the results with the original NMS method to determine the effectiveness of your improvement.

I hope this helps! If you have any further questions or need additional assistance, please feel free to ask.

Best regards.

[Mengyao-Zhang]

感谢您如此迅速的回复！
据您说，YOLOv5在训练阶段没有使用NMS操作。正确的操作是模型训练好后，使用改进的NMS技术测试。
那么，我有几点疑问：
1、在训练阶段没有采用NMS是如何筛选出预测框，计算损失的？
2、为什么我在训练阶段使用改进的NMS技术，会影响结果（如上图绿色：soft-NMS；红色NMS）
上述两个问题，希望得到您的解答，谢谢！

[glenn-jocher]

@18846169373 感谢您的追问！

关于您的问题：

1、在YOLOv5的训练阶段，预测框的筛选和计算损失是通过IoU（交并比）阈值来实现的。在训练过程中，如果一个预测框与标注框之间的IoU大于设定的阈值，则该预测框被视为正样本，用于计算损失。这种方式可以在训练过程中有效地筛选出与目标物体重叠较好的预测框。

2、关于您在训练阶段使用改进的NMS技术后出现结果差异的问题，这可能是由于几个原因引起的。首先，改进的NMS技术可能与模型的训练目标和损失函数不兼容，导致训练过程中的优化效果降低。其次，改进的NMS技术可能与训练数据集的特点不匹配，导致模型在测试阶段无法准确预测。最后，可能存在代码错误或参数设置问题。建议您仔细检查代码实现和参数设置，确保正确性。

希望以上回答能解决您的疑问。如果您还有其他问题或需要进一步帮助，请随时提问。

祝您工作顺利！

[Mengyao-Zhang]

Thank you！祝好！

[glenn-jocher]

@18846169373 thanks for reporting the issue! We appreciate your contribution and apologize for any inconvenience caused.

To better assist you, could you please provide us with more details about the problem you are experiencing? It would be helpful if you could share the specific error message or traceback, along with any relevant code snippets or steps to reproduce the issue.

Once we have more information, we will be able to investigate the problem further and provide you with a suitable solution.

Thank you again for bringing this to our attention, and we look forward to resolving the issue for you. Have a great day!

[github-actions]

👋 Hello there! We wanted to give you a friendly reminder that this issue has not had any recent activity and may be closed soon, but don't worry - you can always reopen it if needed. If you still have any questions or concerns, please feel free to let us know how we can help.

For additional resources and information, please see the links below:

• Docs: https://docs.ultralytics.com
• HUB: https://hub.ultralytics.com
• Community: https://community.ultralytics.com

Feel free to inform us of any other issues you discover or feature requests that come to mind in the future. Pull Requests (PRs) are also always welcomed!

Thank you for your contributions to YOLO 🚀 and Vision AI ⭐