New model support RTDETR #29077

SangbumChoi · 2024-02-17T08:34:49Z

What does this PR do?

This is the new model for RTDETR that is complete version from #27247

There are several TO DOs

reslove conflicts
weight files for other 7 RTDETR
Edit testing script
(optional) enable training

Before submitting

This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
Did you read the contributor guideline,
Pull Request section?
Was this discussed/approved via a Github issue or the forum? Please add a link
to it if that's the case.
Did you make sure to update the documentation with your changes? Here are the
documentation guidelines, and
here are tips on formatting docstrings.
Did you write any new necessary tests?

Who can review?

Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors who may be interested in your PR.

@amyeroberts @NielsRogge

src/transformers/models/timm_backbone/modeling_timm_backbone.py

src/transformers/models/deformable_detr/modeling_deformable_detr.py

amyeroberts · 2024-02-19T18:49:28Z

Looking good @SangbumChoi! Let us know when the PR is ready for review 🤗

amyeroberts

Thanks for all the work adding this model!

Overall looking good - main comment is about the image processor preparing labels and the ordering of object definitions in the modeling file.

I've done a fairly high-level review this time and will do more in-depth over the next pass

src/transformers/models/rt_detr/__init__.py

docs/source/en/model_doc/rt_detr.md

src/transformers/__init__.py

tests/models/rt_detr/test_modeling_rt_detr.py

src/transformers/models/rt_detr/modeling_rt_detr.py

tests/models/rt_detr/test_modeling_rt_detr.py

…into rtdetr

https://github.com/huggingface/transformers/pull/29077/files/75dcd3a0e82cca36f12178b65bbd071ab7b25088#r1506391856

…factor

SangbumChoi · 2024-03-03T12:46:38Z

@amyeroberts Hi amy! I think I can pass all the test in this week. Similar to other last PRs that I opened, I resolved conversation which is very simple or fixed as your comment. However, I did not resolved conversation which might need you to confirm or TO DOs.

SangbumChoi · 2024-03-07T14:00:57Z

@amyeroberts Hi amy, Finally I have passed all the mandatory pass for this model. I think it is a good timing for to request 2nd review for the PR! Personally I think there are 3 things to check.

deepcopy issue
postprocessing logic in image test
modeloutput, objectdetection output order.

Also is it possible to have difference local pytest and ci test ?
Below is my test for local pytest
==================================== 55 passed, 52 skipped, 8 warnings in 39.38s =====================================

SangbumChoi · 2024-06-14T13:29:22Z

@amyeroberts Hi amy I added the test script you suggested! Maybe can you rerun the CI?

amyeroberts

Awesome work adding this model!

amyeroberts · 2024-06-17T15:20:12Z

@SangbumChoi Could you rebase to include the upstream changes on main? This should resolve the CI failues

SangbumChoi · 2024-06-18T00:36:30Z

@amyeroberts Hi, amy. Thanks for the final review, it turns out that there was some issue beside rebase the current main branch.

b2d37a3

test_image_processor required appropriate dictionary in order to pass the test. So I changed at the upper commit!

amyeroberts · 2024-06-18T15:40:15Z

@SangbumChoi Great, thanks for handling that! Could you do a final empty commit for the slow tests?

git commit --allow-empty -m "[run-slow] rt_detr, rt_detr_resnet"

tests/models/rt_detr/test_modeling_rt_detr.py

SangbumChoi · 2024-06-19T00:38:51Z

@amyeroberts I have handled the typo + appropriate testing in
62d4b70
and also empty commit for run slow
50727ab

…into rtdetr

SangbumChoi · 2024-06-21T01:10:05Z

@amyeroberts Hi amy, Could you do a final review? (I think for the rt_detr_resnet has no slow test so it is good to merge even though the CI fails?)

amyeroberts

@SangbumChoi Thanks for all work on this!

amyeroberts · 2024-06-21T16:50:04Z

cc @ydshieh This higlights a case where we might want to update the [run_slow] logic (or maybe not :) ). Here, there's more than one model under the model folder - rt_detr and rt_detr_resnet. How to select which model isn't completely obvious - should we just do rt_detr and let both model's tests be run?

SangbumChoi · 2024-06-22T02:20:16Z

FYI) @ydshieh, @amyeroberts
There was a similar comment from @NielsRogge suggesting that seperating rt_detr_resnet like on-going PR vitpose. Since this rt_detr_resnet architecture is not a standard resnet layer and also only used in this rt_detr I made in same folder like mask2former-swin. (For the purpose of not making minor architecture as a seperate folder.)

amyeroberts · 2024-06-23T18:59:55Z

@SangbumChoi I think it's find to have the rt detr resnet under this model's folder, as @NielsRogge suggested. We just need to make sure our other tools can adapt to these kinds of patterns :)

* fill out docs string in configuration https://github.com/huggingface/transformers/pull/29077/files/75dcd3a0e82cca36f12178b65bbd071ab7b25088#r1506391856 * reduce the input image size for the tests * remove the unappropriate tests * only 5 failes exists * make style * fill up missed architecture for object detection in docs * fix auto modeling * simple fix in missing import * major change including backbone refactor and objectdetectionoutput refactor * minor fix only 4 fails left * intermediate fix * revert __init__.py * revert __init__.py * make style * fixes in pr_docs * intermediate fix * make style * two fixes * pass doctest * only one fix left * intermediate commit * all fixed * Update src/transformers/models/rt_detr/image_processing_rt_detr.py Co-authored-by: amyeroberts <22614925+amyeroberts@users.noreply.github.com> * Update src/transformers/models/rt_detr/convert_rt_detr_original_pytorch_checkpoint_to_pytorch.py Co-authored-by: amyeroberts <22614925+amyeroberts@users.noreply.github.com> * Update src/transformers/models/rt_detr/configuration_rt_detr.py Co-authored-by: amyeroberts <22614925+amyeroberts@users.noreply.github.com> * Update tests/models/rt_detr/test_modeling_rt_detr.py Co-authored-by: amyeroberts <22614925+amyeroberts@users.noreply.github.com> * function class above the model definition in dice_loss * Update src/transformers/models/rt_detr/modeling_rt_detr.py Co-authored-by: amyeroberts <22614925+amyeroberts@users.noreply.github.com> * simple fix * layernorm add config.layer_norm_eps * fix inputs_docstring * make style * simple fix * add custom coco loading test in image_processor * fix error in BaseModelOutput huggingface#29077 (comment) * simple typo * Update src/transformers/models/rt_detr/modeling_rt_detr.py Co-authored-by: amyeroberts <22614925+amyeroberts@users.noreply.github.com> * intermediate fix * fix with load_backbone format * remove unused configuration * 3 fix test left * make style * Update src/transformers/models/rt_detr/image_processing_rt_detr.py Co-authored-by: Sounak Dey <dey.sounak@gmail.com> * change last_hidden_state to first index * all pass fix TO DO: minor update in comments * make fix-copies * remove deepcopy * pr_document fix * revert deepcopy due to the issue of unexpceted behavior in decoderlayer * add atol in final * add no_split_module * _no_split_modules = None * device transfer for model parallelism * minor fix * make fix-copies * fix typo * add test_image_processor with post_processing * Update src/transformers/models/rt_detr/configuration_rt_detr.py Co-authored-by: amyeroberts <22614925+amyeroberts@users.noreply.github.com> * add config in RTDETRPredictionHead * Update src/transformers/models/rt_detr/modeling_rt_detr.py Co-authored-by: amyeroberts <22614925+amyeroberts@users.noreply.github.com> * set lru_cache with max_size 32 * Update src/transformers/models/rt_detr/configuration_rt_detr.py Co-authored-by: amyeroberts <22614925+amyeroberts@users.noreply.github.com> * add lru_cache import and configuration change * change the order of definition * make fix-copies * add docs and change config error * revert strange make-fix * Update src/transformers/models/rt_detr/modeling_rt_detr.py Co-authored-by: amyeroberts <22614925+amyeroberts@users.noreply.github.com> * test pass * fix get_clones related and remove deepcopy * Update src/transformers/models/rt_detr/configuration_rt_detr.py Co-authored-by: NielsRogge <48327001+NielsRogge@users.noreply.github.com> * Update src/transformers/models/rt_detr/configuration_rt_detr.py Co-authored-by: NielsRogge <48327001+NielsRogge@users.noreply.github.com> * Update src/transformers/models/rt_detr/image_processing_rt_detr.py Co-authored-by: NielsRogge <48327001+NielsRogge@users.noreply.github.com> * Update src/transformers/models/rt_detr/image_processing_rt_detr.py Co-authored-by: NielsRogge <48327001+NielsRogge@users.noreply.github.com> * Update src/transformers/models/rt_detr/modeling_rt_detr.py Co-authored-by: NielsRogge <48327001+NielsRogge@users.noreply.github.com> * Update src/transformers/models/rt_detr/modeling_rt_detr.py Co-authored-by: NielsRogge <48327001+NielsRogge@users.noreply.github.com> * Update src/transformers/models/rt_detr/image_processing_rt_detr.py Co-authored-by: NielsRogge <48327001+NielsRogge@users.noreply.github.com> * Update src/transformers/models/rt_detr/modeling_rt_detr.py Co-authored-by: NielsRogge <48327001+NielsRogge@users.noreply.github.com> * Update src/transformers/models/rt_detr/image_processing_rt_detr.py Co-authored-by: NielsRogge <48327001+NielsRogge@users.noreply.github.com> * nit for paper section * Update src/transformers/models/rt_detr/configuration_rt_detr.py Co-authored-by: NielsRogge <48327001+NielsRogge@users.noreply.github.com> * rename denoising related parameters * Update src/transformers/models/rt_detr/image_processing_rt_detr.py Co-authored-by: NielsRogge <48327001+NielsRogge@users.noreply.github.com> * check the image transformation logic * make style * make style * Update src/transformers/models/rt_detr/configuration_rt_detr.py Co-authored-by: NielsRogge <48327001+NielsRogge@users.noreply.github.com> * Update src/transformers/models/rt_detr/modeling_rt_detr.py Co-authored-by: NielsRogge <48327001+NielsRogge@users.noreply.github.com> * Update src/transformers/models/rt_detr/modeling_rt_detr.py Co-authored-by: NielsRogge <48327001+NielsRogge@users.noreply.github.com> * Update src/transformers/models/rt_detr/modeling_rt_detr.py Co-authored-by: NielsRogge <48327001+NielsRogge@users.noreply.github.com> * Update src/transformers/models/rt_detr/modeling_rt_detr.py Co-authored-by: NielsRogge <48327001+NielsRogge@users.noreply.github.com> * Update src/transformers/models/rt_detr/modeling_rt_detr.py Co-authored-by: NielsRogge <48327001+NielsRogge@users.noreply.github.com> * pe_encoding -> positional_encoding_temperature * remove TODO * Update src/transformers/models/rt_detr/image_processing_rt_detr.py Co-authored-by: NielsRogge <48327001+NielsRogge@users.noreply.github.com> * remove eval_idx since transformer DETR is giving all decoder output * Update src/transformers/models/rt_detr/configuration_rt_detr.py Co-authored-by: NielsRogge <48327001+NielsRogge@users.noreply.github.com> * Update src/transformers/models/rt_detr/configuration_rt_detr.py Co-authored-by: NielsRogge <48327001+NielsRogge@users.noreply.github.com> * change variable name * make style and docs import update * Revert "Update src/transformers/models/rt_detr/image_processing_rt_detr.py" This reverts commit 74aa3e1. * fix typo * add postprocessing in docs * move import scipy to top * change varaible name * make fix-copies * remove eval_idx in test * move to after first sentence * update image_processor since box loss requires normalized one * change appropriate name to auxiliary_outputs * Update src/transformers/models/rt_detr/__init__.py Co-authored-by: NielsRogge <48327001+NielsRogge@users.noreply.github.com> * Update src/transformers/models/rt_detr/__init__.py Co-authored-by: NielsRogge <48327001+NielsRogge@users.noreply.github.com> * Update docs/source/en/model_doc/rt_detr.md Co-authored-by: NielsRogge <48327001+NielsRogge@users.noreply.github.com> * Update docs/source/en/model_doc/rt_detr.md Co-authored-by: NielsRogge <48327001+NielsRogge@users.noreply.github.com> * make style * remove panoptic related comments * make style * revert valid_processor_keys * fix aux related test * make style * change origination from config to backbone API * enable the dn_loss * fix test and conversion * renewal weight initialization * change initializer_range * make fix-up * fix the loss issue in the auxiliary output and denoising part * change weight loss to original RTDETR * fix in initialization * sync shape format of dn and aux * make style * stable fine-tuning and compatible conversion for resnet101 * make style * skip input_embed * change encoder related variable * enable converting rtdetr_r101 * add r101 related conversion code * Update src/transformers/models/rt_detr/modeling_rt_detr.py Co-authored-by: amyeroberts <22614925+amyeroberts@users.noreply.github.com> * Update src/transformers/models/rt_detr/modeling_rt_detr.py Co-authored-by: amyeroberts <22614925+amyeroberts@users.noreply.github.com> * Update docs/source/en/model_doc/rt_detr.md Co-authored-by: amyeroberts <22614925+amyeroberts@users.noreply.github.com> * Update src/transformers/models/rt_detr/configuration_rt_detr.py Co-authored-by: amyeroberts <22614925+amyeroberts@users.noreply.github.com> * Update src/transformers/__init__.py Co-authored-by: amyeroberts <22614925+amyeroberts@users.noreply.github.com> * Update src/transformers/__init__.py Co-authored-by: amyeroberts <22614925+amyeroberts@users.noreply.github.com> * Update src/transformers/models/rt_detr/image_processing_rt_detr.py Co-authored-by: amyeroberts <22614925+amyeroberts@users.noreply.github.com> * Update src/transformers/models/rt_detr/image_processing_rt_detr.py Co-authored-by: amyeroberts <22614925+amyeroberts@users.noreply.github.com> * Update src/transformers/models/rt_detr/modeling_rt_detr.py Co-authored-by: amyeroberts <22614925+amyeroberts@users.noreply.github.com> * change name _shape to _reshape * Update src/transformers/__init__.py Co-authored-by: amyeroberts <22614925+amyeroberts@users.noreply.github.com> * Update src/transformers/__init__.py Co-authored-by: amyeroberts <22614925+amyeroberts@users.noreply.github.com> * maket style * make fix-copies * remove deprecated import * more fix * remove last_hidden_state for task-specific model * Revert "remove last_hidden_state for task-specific model" This reverts commit ccb7a34. * minore change in convert * remove print * make style and fix-copies * add custom rtdetr backbone for r18, r34 * remove print * change copied * add pad_size * make style * change layertype to optional to pass the CI * make style * add test in modeling_resnet_rt_detr * make fix-copies * skip tmp file test * fix comment * add docs * change to modeling_resnet file format * enabling resnet50 above * Update src/transformers/models/rt_detr/modeling_rt_detr.py Co-authored-by: Jason Wu <jasonkit@users.noreply.github.com> * enable all the rtdetr model :) * finish except CI * add RTDetrResNetBackbone * make fix-copies * fix TO DO: CI enable * make style * rename test * add docs * add special fix * revert resnet * Update src/transformers/models/rt_detr/modeling_rt_detr_resnet.py Co-authored-by: NielsRogge <48327001+NielsRogge@users.noreply.github.com> * add more comment * remove swin comment * Update src/transformers/models/rt_detr/configuration_rt_detr.py Co-authored-by: NielsRogge <48327001+NielsRogge@users.noreply.github.com> * rename convert and add verify backbone * Update docs/source/en/_toctree.yml Co-authored-by: NielsRogge <48327001+NielsRogge@users.noreply.github.com> * Update docs/source/en/model_doc/rt_detr.md Co-authored-by: NielsRogge <48327001+NielsRogge@users.noreply.github.com> * Update docs/source/en/model_doc/rt_detr.md Co-authored-by: NielsRogge <48327001+NielsRogge@users.noreply.github.com> * make style * requests for docs * more general test docs * general script docs * make fix-copies * final commit * Revert "Update src/transformers/models/rt_detr/configuration_rt_detr.py" This reverts commit d136225. * skip test_model_get_set_embeddings * remove target * add changes * make fix-copies * remove decoder_attention_mask * add load_backbone function for auto_backbone * remove comment * fix repo name * Update src/transformers/models/rt_detr/configuration_rt_detr.py Co-authored-by: amyeroberts <22614925+amyeroberts@users.noreply.github.com> * final commit * remove unused downsample_in_bottleneck * new test for autobackbone * change to appropriate indices * test fix * fix dict in test_image_processor * fix test * [run-slow] rt_detr, rt_detr_resnet * change the slow test * [run-slow] rt_detr * [run-slow] rt_detr, rt_detr_resnet * make in to same cuda in CSPRepLayer * [run-slow] rt_detr, rt_detr_resnet --------- Co-authored-by: amyeroberts <22614925+amyeroberts@users.noreply.github.com> Co-authored-by: Sounak Dey <dey.sounak@gmail.com> Co-authored-by: NielsRogge <48327001+NielsRogge@users.noreply.github.com> Co-authored-by: Jason Wu <jasonkit@users.noreply.github.com> Co-authored-by: ChoiSangBum <choisangbum@ChoiSangBumui-MacBookPro.local>

ydshieh · 2024-06-24T13:55:58Z

How to select which model isn't completely obvious - should we just do rt_detr and let both model's tests be run?

Hi @amyeroberts. IMO, it's fine to just specify the folder name and run everything inside it.
(But if you have use cases where we really need to select part of the files to run the tests, we can discuss)

SangbumChoi commented Feb 17, 2024

View reviewed changes

src/transformers/models/timm_backbone/modeling_timm_backbone.py Show resolved Hide resolved

SangbumChoi commented Feb 17, 2024

View reviewed changes

src/transformers/models/deformable_detr/modeling_deformable_detr.py Show resolved Hide resolved

amyeroberts reviewed Feb 28, 2024

View reviewed changes

SangbumChoi added 12 commits March 1, 2024 10:31

Merge branch 'rtdetr' of https://github.com/SangbumChoi/transformers …

abe0342

…into rtdetr

fill out docs string in configuration

6cf7bf0

https://github.com/huggingface/transformers/pull/29077/files/75dcd3a0e82cca36f12178b65bbd071ab7b25088#r1506391856

reduce the input image size for the tests

3a76611

remove the unappropriate tests

bceeefc

only 5 failes exists

45f4906

make style

9c7a744

fill up missed architecture for object detection in docs

ce5be15

fix auto modeling

029f2cb

simple fix in missing import

faaf9c3

major change including backbone refactor and objectdetectionoutput re…

b063502

…factor

minor fix only 4 fails left

ba241ce

intermediate fix

6a7a74a

SangbumChoi and others added 12 commits March 4, 2024 08:55

Merge branch 'huggingface:main' into rtdetr

41d8bc2

revert __init__.py

c859766

revert __init__.py

34b73a4

make style

02899ab

fixes in pr_docs

fd4af13

intermediate fix

e6727c2

make style

3a2cde6

two fixes

5feb365

pass doctest

f3bf10d

only one fix left

f45a776

intermediate commit

9aa1312

all fixed

7feeed0

ChoiSangBum added 2 commits June 14, 2024 20:42

new test for autobackbone

cd8b70a

change to appropriate indices

a19429b

SangbumChoi requested a review from amyeroberts June 16, 2024 23:41

amyeroberts approved these changes Jun 17, 2024

View reviewed changes

SangbumChoi and others added 3 commits June 18, 2024 07:33

Merge branch 'huggingface:main' into rtdetr

99d1f01

test fix

1dd3a6e

fix dict in test_image_processor

b2d37a3

amyeroberts reviewed Jun 18, 2024

View reviewed changes

tests/models/rt_detr/test_modeling_rt_detr.py Outdated Show resolved Hide resolved

SangbumChoi added 2 commits June 19, 2024 00:37

fix test

62d4b70

[run-slow] rt_detr, rt_detr_resnet

50727ab

SangbumChoi added 6 commits June 20, 2024 20:45

change the slow test

8ba08dc

[run-slow] rt_detr

228e82c

[run-slow] rt_detr, rt_detr_resnet

a8e5888

make in to same cuda in CSPRepLayer

14ef411

Merge branch 'rtdetr' of https://github.com/SangbumChoi/transformers …

72362a5

…into rtdetr

[run-slow] rt_detr, rt_detr_resnet

4b39b12

SangbumChoi requested a review from amyeroberts June 21, 2024 01:07

amyeroberts approved these changes Jun 21, 2024

View reviewed changes

amyeroberts merged commit 74a2074 into huggingface:main Jun 21, 2024
25 of 27 checks passed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

New model support RTDETR #29077

New model support RTDETR #29077

SangbumChoi commented Feb 17, 2024 •

edited

Loading

amyeroberts commented Feb 19, 2024

amyeroberts left a comment

SangbumChoi commented Mar 3, 2024

SangbumChoi commented Mar 7, 2024

SangbumChoi commented Jun 14, 2024

amyeroberts left a comment

amyeroberts commented Jun 17, 2024

SangbumChoi commented Jun 18, 2024

amyeroberts commented Jun 18, 2024

SangbumChoi commented Jun 19, 2024

SangbumChoi commented Jun 21, 2024

amyeroberts left a comment

amyeroberts commented Jun 21, 2024

SangbumChoi commented Jun 22, 2024 •

edited

Loading

amyeroberts commented Jun 23, 2024

ydshieh commented Jun 24, 2024

New model support RTDETR #29077

New model support RTDETR #29077

Conversation

SangbumChoi commented Feb 17, 2024 • edited Loading

What does this PR do?

Before submitting

Who can review?

amyeroberts commented Feb 19, 2024

amyeroberts left a comment

Choose a reason for hiding this comment

SangbumChoi commented Mar 3, 2024

SangbumChoi commented Mar 7, 2024

SangbumChoi commented Jun 14, 2024

amyeroberts left a comment

Choose a reason for hiding this comment

amyeroberts commented Jun 17, 2024

SangbumChoi commented Jun 18, 2024

amyeroberts commented Jun 18, 2024

SangbumChoi commented Jun 19, 2024

SangbumChoi commented Jun 21, 2024

amyeroberts left a comment

Choose a reason for hiding this comment

amyeroberts commented Jun 21, 2024

SangbumChoi commented Jun 22, 2024 • edited Loading

amyeroberts commented Jun 23, 2024

ydshieh commented Jun 24, 2024

SangbumChoi commented Feb 17, 2024 •

edited

Loading

SangbumChoi commented Jun 22, 2024 •

edited

Loading