Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

YOLOv5 v7.0 release updates #10245

Merged
merged 17 commits into from
Nov 22, 2022
Merged

YOLOv5 v7.0 release updates #10245

merged 17 commits into from
Nov 22, 2022

Conversation

glenn-jocher
Copy link
Member

@glenn-jocher glenn-jocher commented Nov 21, 2022

πŸ› οΈ PR Summary

Made with ❀️ by Ultralytics Actions

🌟 Summary

Updated READMEs, tutorials, and weight download scripts to align with the new YOLOv5 v7.0 release.

πŸ“Š Key Changes

  • Updated YOLOv5 splash image links in READMEs and tutorial notebooks to reflect the new v7.0 artwork.
  • Introduced instance segmentation models, with benchmarks and usage examples in the README.
  • Provided training, validation, and prediction snippets for the new segmentation models.
  • Added segmentation model checkpoints and details to the README.
  • Changed default download script to fetch the latest v7.0 weights.

🎯 Purpose & Impact

  • Improves documentation with up-to-date information and helps users to utilize the new YOLOv5 v7.0 features effectively.
  • Enhances ease of access to new instance segmentation models, promoting state-of-the-art performance in real-world applications. πŸš€
  • Simplifies the process for users to work with the latest models by updating download scripts to grab the newest weights. πŸ”„

@glenn-jocher glenn-jocher self-assigned this Nov 21, 2022
Signed-off-by: Glenn Jocher <glenn.jocher@ultralytics.com>
Signed-off-by: Glenn Jocher <glenn.jocher@ultralytics.com>
Signed-off-by: Glenn Jocher <glenn.jocher@ultralytics.com>
@glenn-jocher
Copy link
Member Author

@AyushExel @Laughing-q @taliabender @AlanDimmer @pderrenger new YOLOv5 v7.0 splash images for release tomorrow Tuesday the 22nd.

@glenn-jocher glenn-jocher changed the title YOLOv5 v7.0 splash image update YOLOv5 v7.0 splash updates Nov 21, 2022
@AyushExel
Copy link
Contributor

@glenn-jocher If our models are current SOTA, should we have this kind of chart for segmentation also in the collapsible section?
Screenshot 2022-11-22 at 12 12 36 AM

@glenn-jocher glenn-jocher changed the title YOLOv5 v7.0 splash updates YOLOv5 v7.0 release updates Nov 21, 2022
@glenn-jocher
Copy link
Member Author

glenn-jocher commented Nov 21, 2022

@AyushExel would be nice but putting the comparison plot together is a significant undertaking. The EfficientDet speeds and accuracies I gathered from their repo rather than actually running the tests myself.

@AyushExel
Copy link
Contributor

@glenn-jocher ok got it!
2 more things:

  • How about adding a "open in colab" badge for segmentation and classification notebooks under their sections?
  • Adding official example inference image in the "usage" section of segmentation

@glenn-jocher
Copy link
Member Author

glenn-jocher commented Nov 21, 2022

@Laughing-q do you think you could help create a comparison graph that what @AyushExel is showing above? You could do this on any GPU, or if you have a script I could run this on our A100 instance also.

EDIT: If you send me the data I can also plot it. The current plot is generated by:

yolov5/utils/plots.py

Lines 356 to 357 in be348cc

def plot_val_study(file='', dir='', x=None): # from utils.plots import *; plot_val_study()
# Plot file=study.txt generated by val.py (or plot all study*.txt in dir)

@glenn-jocher If our models are current SOTA, should we have this kind of chart for segmentation also in the collapsible section? Screenshot 2022-11-22 at 12 12 36 AM

@AyushExel
Copy link
Contributor

@glenn-jocher @Laughing-q
These are the official models in the SOTA benchmark on paperswithcode https://paperswithcode.com/sota/real-time-instance-segmentation-on-mscoco

Signed-off-by: Glenn Jocher <glenn.jocher@ultralytics.com>
@glenn-jocher
Copy link
Member Author

@AyushExel you think we could just edit the leaderboard? Screenshot 2022-11-21 at 23 59 25

@AyushExel
Copy link
Contributor

@glenn-jocher I don't know πŸ€·β€β™‚οΈ , Creating an account now to see.

@AyushExel
Copy link
Contributor

@glenn-jocher Ok! looks like I can edit it. We should add L-seg or X-seg model?
Screenshot 2022-11-22 at 4 34 25 AM

Signed-off-by: Glenn Jocher <glenn.jocher@ultralytics.com>
@glenn-jocher
Copy link
Member Author

@AyushExel wow, I didn't realize it was that easy to edit the leaderboard. Maybe there's a review process. I updated README segmentation section with your two changes, can you take a look?

Technically our x model should also be realtime, i.e. it's 4.5ms inference + about 1ms NMS for TRT FP16 on A100 at batch-size 1 --img 640.

Signed-off-by: Glenn Jocher <glenn.jocher@ultralytics.com>
@glenn-jocher glenn-jocher added the enhancement New feature or request label Nov 21, 2022
@AyushExel
Copy link
Contributor

AyushExel commented Nov 21, 2022

@glenn-jocher okay looks perfect πŸ‘

Btw what should be the inference speed in FPS for X-seg model? Converting from ms/frame to FPS, It should be 1000/4.5.. Seems too high right?
EDIT:
It is mandatory to choose a published paper :(
Screenshot 2022-11-22 at 4 50 01 AM

@glenn-jocher
Copy link
Member Author

@AyushExel ya seems pretty fast, but we are using a fast GPU. I'd add the 1ms NMS just to play it safe, 1000/5.5 = 182 FPS

@AyushExel
Copy link
Contributor

@glenn-jocher I think so. The field self-populates with options and you need to choose one from the drop-down. I guess it's reading from arxiv

@glenn-jocher
Copy link
Member Author

TODO: update download scripts to point to v7.0

@Laughing-q
Copy link
Member

@Laughing-q do you think you could help create a comparison graph that what @AyushExel is showing above? You could do this on any GPU, or if you have a script I could run this on our A100 instance also.

EDIT: If you send me the data I can also plot it. The current plot is generated by:

yolov5/utils/plots.py

Lines 356 to 357 in be348cc

def plot_val_study(file='', dir='', x=None): # from utils.plots import *; plot_val_study()
# Plot file=study.txt generated by val.py (or plot all study*.txt in dir)

@glenn-jocher If our models are current SOTA, should we have this kind of chart for segmentation also in the collapsible section? Screenshot 2022-11-22 at 12 12 36 AM

ok, got it. So we probably want to show other one or two models in our chart, like SparseInst-608 and maYOLACT-700 in the leaderboard. I'm not sure if we just copy the speed from their paper or we test it by ourselves, also if we want to test it by ourselves, should we use tensorrt-fp16 or just torch-fp16?
The yolov5 detection speed chart is generated by tensorrt or torch?

@Laughing-q
Copy link
Member

@glenn-jocher @AyushExel okay I believe we can easily get the tensorrt speed and torch-speed of yolov5-seg, I'll try to export SparseInst and maYOLACT to tensorrt and compare the speed.

@AyushExel
Copy link
Contributor

@Laughing-q do you think you could help create a comparison graph that what @AyushExel is showing above? You could do this on any GPU, or if you have a script I could run this on our A100 instance also.
EDIT: If you send me the data I can also plot it. The current plot is generated by:

yolov5/utils/plots.py

Lines 356 to 357 in be348cc

def plot_val_study(file='', dir='', x=None): # from utils.plots import *; plot_val_study()
# Plot file=study.txt generated by val.py (or plot all study*.txt in dir)

@glenn-jocher If our models are current SOTA, should we have this kind of chart for segmentation also in the collapsible section? Screenshot 2022-11-22 at 12 12 36 AM

ok, got it. So we probably want to show other one or two models in our chart, like SparseInst-608 and maYOLACT-700 in the leaderboard. I'm not sure if we just copy the speed from their paper or we test it by ourselves, also if we want to test it by ourselves, should we use tensorrt-fp16 or just torch-fp16? The yolov5 detection speed chart is generated by tensorrt or torch?

@Laughing-q I think if we can't calculate the speeds ourselves then it's fine to use it from the chart. Not sure about the detection speed but the segmentation chart mentions TRT so I think Glenn used tensorrt for that.

@AyushExel
Copy link
Contributor

@glenn-jocher @AyushExel okay I believe we can easily get the tensorrt speed and torch-speed of yolov5-seg, I'll try to export SparseInst and maYOLACT to tensorrt and compare the speed.

Ok perfect. If we can't get speeds for some, we can just take them from the charts

@Laughing-q
Copy link
Member

Laughing-q commented Nov 22, 2022

@glenn-jocher @AyushExel Actually I can't smoothly export sparseInst models to tensorrt and I found their repo got the script to evaluate cocoval and test speed, so I tested all the speeds by torch-fp16.
Figure_1

  • All the speeds are tested with torch-fp16 and include inference time and nms time.
  • 640min means the minimum of input-size is 640, which means the height or width can exceed 640.
  • 640max means the maximum of input-size is 640, which means the height or width can't exceed 640.
  • the speeds of yolov5-seg are tested by following command, btw yolov5-seg can be regarded as 640max case.
    # modified val.py to only test 640 case.
    python segment/val.py --data coco.yaml --iou-thres 0.7 --batch-size 1 --half \
    --weights weights/yolov5n-seg.pt weights/yolov5s-seg.pt weights/yolov5m-seg.pt weights/yolov5l-seg.pt weights/yolov5x-seg.pt\
    --task study
  • the speeds of SparseInst are tested by using hustvl/SparseInst, which I modified a little bit to use fp16.
    python tools/test_net.py \
    --config-file config.yaml \
    --fp16\
    MODEL.WEIGHTS sparseinst.pth \
    INPUT.MIN_SIZE_TEST 640\
    # INPUT.MAX_SIZE_TEST 640\   # uncomment this line if test the 640max
  • the speeds of SOLOv2 are tested by using aim-uofa/AdelaiDet, which I modified a little bit to use fp16.
    python tools/train_net.py \
    --config-file configs/SOLOv2/R50_3x.yaml \
    --eval-only \
    --num-gpus 1 \
    OUTPUT_DIR output \
    MODEL.WEIGHTS ./weights/SOLOv2_R50_3x.pth\
    INPUT.MIN_SIZE_TEST 640\
    # INPUT.MAX_SIZE_TEST 640\  # uncomment this line if test the 640max
  • The maYOLACT repo seems not release the weights of their best model, so I didn't test it but test SOLOv2.
  • The SOLOv2 are trained with input-size=800 so it's normal that the mAP I evaluated with input-size=640 is lower than official repo.

The plot script is here, also the speed and mAP are in the following code:

import matplotlib.pyplot as plt

yolov5_seg = dict(
    N=dict(speed=4.0, mAP=27.6),
    S=dict(speed=4.9, mAP=31.7),
    M=dict(speed=9.4, mAP=37.1),
    L=dict(speed=14.0, mAP=39.9),
    X=dict(speed=24.1, mAP=41.4),
)

sparseinst_640max = dict(
    r50vd_dcn_giam=dict(speed=16.5, mAP=35.5),
    r50vd_giam=dict(speed=15.9, mAP=34.0),
    r50vd=dict(speed=15.5, mAP=32.58),
)

sparseinst_640min = dict(
    r50vd_dcn_giam=dict(speed=26.4, mAP=37.7),
    r50vd_giam=dict(speed=25.3, mAP=35.6),
    r50vd=dict(speed=24.7, mAP=34.1),
)

solov2_640max = dict(
    r50=dict(speed=31.7, mAP=28.9),
    r101=dict(speed=37.5, mAP=31.14),
)

solov2_640min = dict(
    r50=dict(speed=44.7, mAP=32.5),
    r101=dict(speed=54.5, mAP=34.8),
)

colors = ["#1f77b4", "#ff7f0e", "#2ca02c", "#7E2F8E", "#A2142F"]
names = ["yolov5-seg", "sparseinst_640min", "sparseinst_640max", "solov2_640min", "solov2_640max"]
results = [yolov5_seg, sparseinst_640min, sparseinst_640max, solov2_640min, solov2_640max]
for i, result in enumerate(results):
    plt.plot(
        [value["speed"] for _, value in result.items()],
        [value["mAP"] for _, value in result.items()],
        "*-",
        linewidth=2,
        markersize=8,
        label=f"{names[i]}",
        color=f"{colors[i]}",
    )
    for k, v in result.items():
        plt.text(
            v["speed"],
            v["mAP"],
            k,
            fontsize=15,
            # style="italic",
            weight="light",
            # verticalalignment="center",
            horizontalalignment="right",
            # rotation=45,
            color=f"{colors[i]}",
        )


plt.grid(alpha=0.5)
plt.xlabel("Torch-FP16 Latency(ms) on RTX3060")
plt.ylabel("COCO AP val(%)")
plt.legend(loc="lower right")

plt.show()
# plt.savefig("result_new.png")

@glenn-jocher
Copy link
Member Author

@Laughing-q wow, super awesome!! Ok I'm taking a look at this now and I'll update the Release notes with it before I do the release later today.

@glenn-jocher
Copy link
Member Author

glenn-jocher commented Nov 22, 2022

Ok I've updated it a bit, looking amazing! I'm going to include this version in our release notes and in the readme.

study

import matplotlib.pyplot as plt
import numpy as np

yolov5_seg = dict(
    YOLOv5n=dict(speed=4.0, mAP=27.6),
    YOLOv5s=dict(speed=4.9, mAP=31.7),
    YOLOv5m=dict(speed=9.4, mAP=37.1),
    YOLOv5l=dict(speed=14.0, mAP=39.9),
    YOLOv5x=dict(speed=24.1, mAP=41.4),
)

sparseinst_640max = dict(
    r50vd_dcn_giam=dict(speed=16.5, mAP=35.5),
    r50vd_giam=dict(speed=15.9, mAP=34.0),
    r50vd=dict(speed=15.5, mAP=32.58),
)

sparseinst_640min = dict(
    r50vd_dcn_giam=dict(speed=26.4, mAP=37.7),
    r50vd_giam=dict(speed=25.3, mAP=35.6),
    r50vd=dict(speed=24.7, mAP=34.1),
)

solov2_640max = dict(
    r50=dict(speed=31.7, mAP=28.9),
    r101=dict(speed=37.5, mAP=31.14),
)

solov2_640min = dict(
    r50=dict(speed=44.7, mAP=32.5),
    r101=dict(speed=54.5, mAP=34.8),
)

colors = ["#1f77b4", "#ff7f0e", "#2ca02c", "#7E2F8E", "#A2142F"]
names = ["YOLOv5-seg", "sparseinst_640min", "sparseinst_640max", "solov2_640min", "solov2_640max"]
results = [yolov5_seg, sparseinst_640min, sparseinst_640max, solov2_640min, solov2_640max]
fig2, ax2 = plt.subplots(1, 1, figsize=(8, 4.25), tight_layout=True)
for i, result in enumerate(results):
    ax2.plot(
        [value["speed"] for _, value in result.items()],
        [value["mAP"] for _, value in result.items()],
        ".-",
        linewidth=2,
        markersize=10,
        label=f"{names[i]}",
        color=f"{colors[i]}",
    )
    for k, v in result.items():
        print(k)
        plt.text(
            v["speed"] - 0.5,
            v["mAP"],
            k,
            fontsize=12,
            # style="italic",
            weight="light",
            # verticalalignment="center",
            horizontalalignment="right",
            # rotation=45,
            color=f"{colors[i]}",
        )

ax2.grid(alpha=0.2)
ax2.set_yticks(np.arange(20, 60, 5))
ax2.set_xlim(0, 60)
ax2.set_ylim(25, 43)
ax2.set_xlabel('PyTorch FP16 RTX3060 Latency (ms/img)')
ax2.set_ylabel('COCO Mask AP val')
ax2.set_title('YOLOv5 v7.0: SOTA Realtime Instance Segmentation')
ax2.legend(loc='lower right')
f = 'study.png'
print(f'Saving {f}...')
plt.savefig(f, dpi=300)
plt.show()

glenn-jocher and others added 5 commits November 22, 2022 16:08
Signed-off-by: Glenn Jocher <glenn.jocher@ultralytics.com>
Signed-off-by: Glenn Jocher <glenn.jocher@ultralytics.com>
Signed-off-by: Glenn Jocher <glenn.jocher@ultralytics.com>
@glenn-jocher glenn-jocher merged commit 915bbf2 into master Nov 22, 2022
@glenn-jocher glenn-jocher deleted the update_splash branch November 22, 2022 15:23
@MSutt
Copy link
Contributor

MSutt commented Nov 23, 2022

Hello @glenn-jocher,

Congratulation for releasing the v7.0.
I saw on docker hub that the docker image with the v7.0 tag is not published yet.
Could you please push it ?

Thanks

@glenn-jocher
Copy link
Member Author

@MSutt oh of course, thanks for the reminder!

@glenn-jocher
Copy link
Member Author

glenn-jocher commented Nov 24, 2022

@MSutt all 3 release images pushed to DockerHub now :)

Screenshot 2022-11-24 at 16 03 53

@Laughing-q
Copy link
Member

Laughing-q commented Nov 28, 2022

@glenn-jocher @AyushExel updated the chart.
Actually I found that the scripts of SpareInst and SOLOv2 include mask processing in their inference time. So I also added our mask processing time and update the chart. It takes more 1~2ms.

with dt[2]:
    preds = non_max_suppression(preds,
                                conf_thres,
                                iou_thres,
                                labels=lb,
                                multi_label=True,
                                agnostic=single_cls,
                                max_det=max_det,
                                nm=nm)
    # add this to include processing time
    for i, det in enumerate(preds):  # per image
        if len(det):
            _ = process_mask(protos[i], det[:, 6:], det[:, :4], im[i].shape[1:], upsample=True)  # HWC

And I found their max_det equals to 100, so I tested our models with --max-det 100.(our default max_det is 300 but more max_det, more time of mask processing will cost)

# modified val.py to only test 640 case.
python segment/val.py --data coco.yaml --iou-thres 0.7 --batch-size 1 --half \
--weights weights/yolov5n-seg.pt weights/yolov5s-seg.pt weights/yolov5m-seg.pt weights/yolov5l-seg.pt weights/yolov5x-seg.pt\
--task study\
--max-det 100
import matplotlib.pyplot as plt
import numpy as np

yolov5_seg = dict(
    YOLOv5n=dict(speed=5.75, mAP=27.6),
    YOLOv5s=dict(speed=6.5, mAP=31.7),
    YOLOv5m=dict(speed=11.0, mAP=37.1),
    YOLOv5l=dict(speed=15.5, mAP=39.9),
    YOLOv5x=dict(speed=25.6, mAP=41.4),
)

sparseinst_640max = dict(
    r50vd_dcn_giam=dict(speed=16.5, mAP=35.5),
    r50vd_giam=dict(speed=15.9, mAP=34.0),
    r50vd=dict(speed=15.5, mAP=32.58),
)

sparseinst_640min = dict(
    r50vd_dcn_giam=dict(speed=26.4, mAP=37.7),
    r50vd_giam=dict(speed=25.3, mAP=35.6),
    r50vd=dict(speed=24.7, mAP=34.1),
)

solov2_640max = dict(
    r50=dict(speed=31.7, mAP=28.9),
    r101=dict(speed=37.5, mAP=31.14),
)

solov2_640min = dict(
    r50=dict(speed=44.7, mAP=32.5),
    r101=dict(speed=54.5, mAP=34.8),
)

colors = ["#1f77b4", "#ff7f0e", "#2ca02c", "#7E2F8E", "#A2142F"]
names = ["YOLOv5-seg", "sparseinst_640min", "sparseinst_640max", "solov2_640min", "solov2_640max"]
results = [yolov5_seg, sparseinst_640min, sparseinst_640max, solov2_640min, solov2_640max]
fig2, ax2 = plt.subplots(1, 1, figsize=(8, 4.25), tight_layout=True)
for i, result in enumerate(results):
    ax2.plot(
        [value["speed"] for _, value in result.items()],
        [value["mAP"] for _, value in result.items()],
        ".-",
        linewidth=2,
        markersize=10,
        label=f"{names[i]}",
        color=f"{colors[i]}",
    )
    for k, v in result.items():
        print(k)
        plt.text(
            v["speed"] - 0.5,
            v["mAP"],
            k,
            fontsize=12,
            # style="italic",
            weight="light",
            # verticalalignment="center",
            horizontalalignment="right",
            # rotation=45,
            color=f"{colors[i]}",
        )

ax2.grid(alpha=0.2)
ax2.set_yticks(np.arange(20, 60, 5))
ax2.set_xlim(0, 60)
ax2.set_ylim(25, 43)
ax2.set_xlabel('PyTorch FP16 RTX3060 Latency (ms/img)')
ax2.set_ylabel('COCO Mask AP val')
ax2.set_title('YOLOv5 v7.0: SOTA Realtime Instance Segmentation')
ax2.legend(loc='lower right')
f = 'study.png'
print(f'Saving {f}...')
plt.savefig(f, dpi=300)
plt.show()

This chart includes the time of mask processing.
study

Actually I don't think this 1~2 ms matters, but in case users will ask about it so I commented the speed results here.

@AyushExel
Copy link
Contributor

@Laughing-q got it. Thanks!

@AyushExel
Copy link
Contributor

@glenn-jocher remember to update the image on readme with this ^^

glenn-jocher added a commit that referenced this pull request Nov 30, 2022
@AyushExel @Laughing-q updated plot here in README

Addresses #10245 (comment)

Signed-off-by: Glenn Jocher <glenn.jocher@ultralytics.com>
glenn-jocher added a commit that referenced this pull request Nov 30, 2022
@AyushExel @Laughing-q updated plot here in README

Addresses #10245 (comment)

Signed-off-by: Glenn Jocher <glenn.jocher@ultralytics.com>

Signed-off-by: Glenn Jocher <glenn.jocher@ultralytics.com>
@glenn-jocher
Copy link
Member Author

@Laughing-q thanks! I've fixed our plot in #10344

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

4 participants