Enable HF pretrained backbones #31145

amyeroberts · 2024-05-30T14:28:45Z

What does this PR do?

Enables loading HF pretrained model weights for backbones.

from transformers import MaskFormerConfig, MaskFormerForInstanceSegmentation

config = MaskFormerConfig(backbone="microsoft/resnet-50", use_pretrained_backbone=True)
model = MaskFormerForInstanceSegmentation(config)

Updates documentation to show correct initialization
Removes check in verify_backbone_config_arguments
Adds test for loading HF checkpoints

Before submitting

This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
Did you read the contributor guideline,
Pull Request section?
Was this discussed/approved via a Github issue or the forum? Please add a link
to it if that's the case.
Did you make sure to update the documentation with your changes? Here are the
documentation guidelines, and
here are tips on formatting docstrings.
Did you write any new necessary tests?

HuggingFaceDocBuilderDev · 2024-05-30T14:50:43Z

The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

docs/source/en/create_a_model.md

amyeroberts · 2024-06-04T17:45:30Z

docs/source/en/create_a_model.md

@@ -333,16 +333,6 @@ config = MaskFormerConfig(backbone="microsoft/resnet50", use_pretrained_backbone
 model = MaskFormerForInstanceSegmentation(config) # head
 ```

-You could also load the backbone config separately and then pass it to the model config.


Removed as it's repeated and included in the section about loading pretrained backbones, but the example loads randomly initialized weights

amyeroberts · 2024-06-04T17:47:54Z

src/transformers/models/timm_backbone/modeling_timm_backbone.py

@@ -50,7 +50,7 @@ def __init__(self, config, **kwargs):
        if config.backbone is None:
            raise ValueError("backbone is not set in the config. Please set it to a timm model name.")

-        if config.backbone not in timm.list_models():
+        if config.backbone.split(".")[0] not in timm.list_models():


This change is because certain timm checkpoints will have the base model + a bunch of specific model specifications e.g. vit_large_patch14_dinov2.lvd142m

Ok, maybe worth adding this comment to the code or doing it outside if statement with a clear variable name?

amyeroberts · 2024-06-04T17:53:17Z

docs/source/en/create_a_model.md


-config = MaskFormerConfig(backbone="microsoft/resnet50", use_pretrained_backbone=True) # backbone and neck config
+config = MaskFormerConfig(backbone="microsoft/resnet-50", use_pretrained_backbone=True) # backbone and neck config


Changed to the correct checkpoint

amyeroberts · 2024-06-04T17:55:25Z

src/transformers/models/depth_anything/modeling_depth_anything.py

-        self.backbone = AutoBackbone.from_config(
-            config.backbone_config, attn_implementation=config._attn_implementation
-        )
+        self.backbone = load_backbone(config)


All backbones should be loaded through load_backbone. Being able to propogate attn_implementation should be done in a follow up (and wasn't being used for depth anything)

qubvel

Great, thanks for working on this! Just a few comments, nothing critical

qubvel · 2024-06-06T09:50:29Z

src/transformers/models/conditional_detr/modeling_conditional_detr.py

+        backbone_model_type = None
+        if config.backbone is not None:
+            backbone_model_type = config.backbone
+        elif config.backbone_config is not None:
+            backbone_model_type = config.backbone_config.model_type


Is there a chance that backbone and backbone_config are both None by misconfiguration? Should we throw an error here or check in verify_backbone_config_arguments?

If backbone_model_type will remain None, then if "resnet" in backbone_model_type: will raise an error:

TypeError: argument of type 'NoneType' is not iterable

Is there a chance that backbone and backbone_config are both None by misconfiguration?

It shouldn't happen from a fresh config, as verify_backbone_config_arguments will check that.

It's possible something like this could happen:

from transformers import ConditionalDetrConfig, ConditionalDetrModel config = ConditionalDetrConfig() config.backbone = None config.backbone_config = None model = ConditionalDetrModel(config)

as it's not uncommon to modify configs post-creation.

If backbone_model_type will remain None, then if "resnet" in backbone_model_type: will raise an error:

Good point. I'll add in an exception is neither are set here and for the other, similar bits of logic

qubvel · 2024-06-06T10:07:18Z

src/transformers/models/dpt/configuration_dpt.py

@@ -182,7 +182,7 @@ def __init__(

        use_autobackbone = False
        if self.is_hybrid:
-            if backbone_config is None and backbone is None:
+            if backbone_config is None:


nit, in case we use if-if instead of if-elif we can remove logging and config initialization for the first if. This will be handled by the second if:

if backbone_config is None: - logger.info("Initializing the config with a `BiT` backbone.") backbone_config = { "global_padding": "same", "layer_type": "bottleneck", "depths": [3, 4, 9], "out_features": ["stage1", "stage2", "stage3"], "embedding_dynamic_padding": True, } - backbone_config = BitConfig(**backbone_config) if isinstance(backbone_config, dict): logger.info("Initializing the config with a `BiT` backbone.") backbone_config = BitConfig(**backbone_config)

qubvel · 2024-06-06T10:21:36Z

src/transformers/models/timm_backbone/modeling_timm_backbone.py

@@ -50,7 +50,7 @@ def __init__(self, config, **kwargs):
        if config.backbone is None:
            raise ValueError("backbone is not set in the config. Please set it to a timm model name.")

-        if config.backbone not in timm.list_models():
+        if config.backbone.split(".")[0] not in timm.list_models():


Ok, maybe worth adding this comment to the code or doing it outside if statement with a clear variable name?

src/transformers/models/tvp/modeling_tvp.py

qubvel · 2024-06-06T10:50:02Z

docs/source/en/create_a_model.md

+Use `use_timm_backbone=True` and `use_pretrained_backbone=True` to load pretrained timm weights for the backbone.
+
+```python
+from transformers import MaskFormerConfig, MaskFormerForInstanceSegmentation
+
+config = MaskFormerConfig(backbone="resnet50", use_pretrained_backbone=True, use_timm_backbone=True) # backbone and neck config
+model = MaskFormerForInstanceSegmentation(config) # head
+```
+
+Set `use_timm_backbone=True` and `use_pretrained_backbone=False` to load a randomly initialized timm backbone.
+
+```python
+from transformers import MaskFormerConfig, MaskFormerForInstanceSegmentation
+
+config = MaskFormerConfig(backbone="resnet50", use_pretrained_backbone=False, use_timm_backbone=True) # backbone and neck config
+model = MaskFormerForInstanceSegmentation(config) # head
+```
+
+You could also load the backbone config and use it to create a `TimmBackbone` or pass it to the model config. Timm backbones will load pretrained weights by default. Set `use_pretrained_backbone=False` to load randomly initialized weights.


amyeroberts

Accidentally left comments as review comments and I'm too lazy to store / delete / readd

amyeroberts · 2024-06-05T12:40:40Z

src/transformers/models/dpt/configuration_dpt.py


        self.backbone = backbone
        self.use_pretrained_backbone = use_pretrained_backbone
        self.use_timm_backbone = use_timm_backbone
        self.backbone_kwargs = backbone_kwargs
-        self.num_hidden_layers = None if use_autobackbone else num_hidden_layers


We don't default to setting these to None if use_autobackbone is True as:

Even if is_hybrid is False, some of these values are needed e.g. image_size and patch_size are needed in patch embeddings which are used in DPTViTEmbeddings initialized here.

It can lead to surprising behaviour e.g. I pass in DPTConfig(is_hybrid=False, num_hidden_layers=5) and see the num_hidden_layers is then set to None.

amyeroberts · 2024-06-05T14:46:02Z

src/transformers/models/dpt/configuration_dpt.py

@@ -208,9 +208,8 @@ def __init__(
            if readout_type != "project":
                raise ValueError("Readout type must be 'project' when using `DPT-hybrid` mode.")

-        elif backbone_config is not None:
+        elif backbone is not None or backbone_config is not None:


To specify which backbone to load from a config, the user can do one of two things, either:

Specify the checkpoint e.g. backbone=microsoft/resnet-10

Specify a config e.g. backbone_config=BitConfg()

We need to be able to support both to enable load_backbone i.e. loading timm or HF pretrained and randomly initialized architectures.

amyeroberts · 2024-06-06T14:52:35Z

src/transformers/models/conditional_detr/modeling_conditional_detr.py

+        backbone_model_type = None
+        if config.backbone is not None:
+            backbone_model_type = config.backbone
+        elif config.backbone_config is not None:
+            backbone_model_type = config.backbone_config.model_type


Is there a chance that backbone and backbone_config are both None by misconfiguration?

It shouldn't happen from a fresh config, as verify_backbone_config_arguments will check that.

It's possible something like this could happen:

from transformers import ConditionalDetrConfig, ConditionalDetrModel config = ConditionalDetrConfig() config.backbone = None config.backbone_config = None model = ConditionalDetrModel(config)

as it's not uncommon to modify configs post-creation.

If backbone_model_type will remain None, then if "resnet" in backbone_model_type: will raise an error:

Good point. I'll add in an exception is neither are set here and for the other, similar bits of logic

* Enable load HF or tim backbone checkpoints * Fix up * Fix test - pass in proper out_indices * Update docs * Fix tvp tests * Fix doc examples * Fix doc examples * Try to resolve DPT backbone param init * Don't conditionally set to None * Add condition based on whether backbone is defined * Address review comments

amyeroberts changed the title ~~Hf pretrained backbones~~ Enable HF pretrained backbones May 30, 2024

NielsRogge mentioned this pull request Jun 1, 2024

DPTForDepthEstimation with Dinov2 does not use pretrained weights #30069

Closed

2 tasks

amyeroberts force-pushed the hf-pretrained-backbones branch from 450e7c1 to 2d4e302 Compare June 4, 2024 15:48

amyeroberts mentioned this pull request Jun 4, 2024

Move out common backbone config param validation #31144

Merged

amyeroberts commented Jun 4, 2024

View reviewed changes

docs/source/en/create_a_model.md Outdated Show resolved Hide resolved

amyeroberts commented Jun 4, 2024

View reviewed changes

docs/source/en/create_a_model.md Outdated Show resolved Hide resolved

amyeroberts commented Jun 4, 2024

View reviewed changes

amyeroberts force-pushed the hf-pretrained-backbones branch from fe0a50a to 3763618 Compare June 4, 2024 17:46

amyeroberts commented Jun 4, 2024

View reviewed changes

amyeroberts mentioned this pull request Jun 5, 2024

Intel/dpt-swinv2-tiny-256: TypeError: unsupported operand type(s) for //: 'NoneType' and 'NoneType' #31249

Closed

4 tasks

amyeroberts force-pushed the hf-pretrained-backbones branch from 969fb8c to 2c84a2b Compare June 5, 2024 13:47

amyeroberts requested a review from qubvel June 5, 2024 14:47

qubvel approved these changes Jun 6, 2024

View reviewed changes

amyeroberts commented Jun 6, 2024

View reviewed changes

amyeroberts added 11 commits June 6, 2024 15:08

Enable load HF or tim backbone checkpoints

afc0313

Fix up

e94e955

Fix test - pass in proper out_indices

3b55ccb

Update docs

9fd282f

Fix tvp tests

226021f

Fix doc examples

7016fb2

Fix doc examples

2733864

Try to resolve DPT backbone param init

d983ad7

Don't conditionally set to None

6adb09d

Add condition based on whether backbone is defined

59a071c

Address review comments

d40fcd2

amyeroberts force-pushed the hf-pretrained-backbones branch from d9c7d77 to d40fcd2 Compare June 6, 2024 15:09

amyeroberts merged commit bdf36dc into huggingface:main Jun 6, 2024
23 checks passed

amyeroberts deleted the hf-pretrained-backbones branch June 6, 2024 21:02

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Enable HF pretrained backbones #31145

Enable HF pretrained backbones #31145

amyeroberts commented May 30, 2024 •

edited

Loading

HuggingFaceDocBuilderDev commented May 30, 2024

amyeroberts Jun 4, 2024

amyeroberts Jun 4, 2024

qubvel Jun 6, 2024

amyeroberts Jun 4, 2024

amyeroberts Jun 4, 2024

qubvel left a comment

qubvel Jun 6, 2024

amyeroberts Jun 6, 2024

qubvel Jun 6, 2024

qubvel Jun 6, 2024

qubvel Jun 6, 2024

amyeroberts left a comment

amyeroberts Jun 5, 2024 •

edited

Loading

amyeroberts Jun 5, 2024

amyeroberts Jun 6, 2024


		config = MaskFormerConfig(backbone="microsoft/resnet50", use_pretrained_backbone=True) # backbone and neck config
		config = MaskFormerConfig(backbone="microsoft/resnet-50", use_pretrained_backbone=True) # backbone and neck config

Enable HF pretrained backbones #31145

Enable HF pretrained backbones #31145

Conversation

amyeroberts commented May 30, 2024 • edited Loading

What does this PR do?

Before submitting

HuggingFaceDocBuilderDev commented May 30, 2024

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

qubvel left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

amyeroberts left a comment

Choose a reason for hiding this comment

amyeroberts Jun 5, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

amyeroberts commented May 30, 2024 •

edited

Loading

amyeroberts Jun 5, 2024 •

edited

Loading