While running pipeline.py - IndexError

[richasempire]

"WhitePaint" cluster, q=39.57%
512x512 kept 0 patches -> /content/gdrive/MyDrive/MIT/SEM02/ComputationDesignLab/MaterialPalette/my_images/crops/WhitePaint
256x256 kept 0 patches -> /content/gdrive/MyDrive/MIT/SEM02/ComputationDesignLab/MaterialPalette/my_images/crops/WhitePaint
192x192 kept 0 patches -> /content/gdrive/MyDrive/MIT/SEM02/ComputationDesignLab/MaterialPalette/my_images/crops/WhitePaint
128x128 kept 4 patches -> /content/gdrive/MyDrive/MIT/SEM02/ComputationDesignLab/MaterialPalette/my_images/crops/WhitePaint
---- kept 2/3 crops.
04/18/2024 03:24:04 - INFO - concept.utils - Distributed environment: NO
Num processes: 1
Process index: 0
Local process index: 0
Device: cuda

Mixed precision type: fp16

trainable params: 589,824 || all params: 123,650,304 || trainable%: 0.4770097451600281
{'scaling_factor', 'force_upcast'} was not found in config. Values will be initialized to default values.
{'num_attention_heads', 'mid_block_only_cross_attention', 'dual_cross_attention', 'resnet_skip_time_act', 'addition_time_embed_dim', 'time_embedding_type', 'cross_attention_norm', 'class_embed_type', 'only_cross_attention', 'conv_out_kernel', 'time_embedding_dim', 'resnet_time_scale_shift', 'conv_in_kernel', 'transformer_layers_per_block', 'encoder_hid_dim_type', 'addition_embed_type', 'resnet_out_scale_factor', 'projection_class_embeddings_input_dim', 'timestep_post_act', 'time_cond_proj_dim', 'num_class_embeds', 'mid_block_type', 'encoder_hid_dim', 'class_embeddings_concat', 'time_embedding_act_fn', 'addition_embed_type_num_heads', 'upcast_attention', 'use_linear_projection'} was not found in config. Values will be initialized to default values.
trainable params: 1,594,368 || all params: 861,115,332 || trainable%: 0.18515150535027286
04/18/2024 03:24:08 - INFO - concept.utils - ***** Running training *****
04/18/2024 03:24:08 - INFO - concept.utils - Num examples = 12
04/18/2024 03:24:08 - INFO - concept.utils - Num batches each epoch = 12
04/18/2024 03:24:08 - INFO - concept.utils - Instantaneous batch size per device = 1
04/18/2024 03:24:08 - INFO - concept.utils - Total train batch size (w. parallel, distributed) = 1
04/18/2024 03:24:08 - INFO - concept.utils - Total optimization steps = 800
Steps: 100% 800/800 [05:04<00:00, 2.62it/s, loss=0.685, lr=0.0001]
loading LoRA with token azertyuiop
{'requires_safety_checker'} was not found in config. Values will be initialized to default values.
Loading pipeline components...: 0% 0/6 [00:00<?, ?it/s]Loaded feature_extractor as CLIPImageProcessor from feature_extractor subfolder of runwayml/stable-diffusion-v1-5.
Loaded text_encoder as CLIPTextModel from text_encoder subfolder of runwayml/stable-diffusion-v1-5.
Loading pipeline components...: 33% 2/6 [00:00<00:00, 7.29it/s]{'timestep_spacing', 'prediction_type'} was not found in config. Values will be initialized to default values.
Loaded scheduler as PNDMScheduler from scheduler subfolder of runwayml/stable-diffusion-v1-5.
{'scaling_factor', 'force_upcast'} was not found in config. Values will be initialized to default values.
Loaded vae as AutoencoderKL from vae subfolder of runwayml/stable-diffusion-v1-5.
Loading pipeline components...: 67% 4/6 [00:00<00:00, 8.02it/s]Loaded tokenizer as CLIPTokenizer from tokenizer subfolder of runwayml/stable-diffusion-v1-5.
{'num_attention_heads', 'mid_block_only_cross_attention', 'dual_cross_attention', 'resnet_skip_time_act', 'addition_time_embed_dim', 'time_embedding_type', 'cross_attention_norm', 'class_embed_type', 'only_cross_attention', 'conv_out_kernel', 'time_embedding_dim', 'resnet_time_scale_shift', 'conv_in_kernel', 'transformer_layers_per_block', 'encoder_hid_dim_type', 'addition_embed_type', 'resnet_out_scale_factor', 'projection_class_embeddings_input_dim', 'timestep_post_act', 'time_cond_proj_dim', 'num_class_embeds', 'mid_block_type', 'encoder_hid_dim', 'class_embeddings_concat', 'time_embedding_act_fn', 'addition_embed_type_num_heads', 'upcast_attention', 'use_linear_projection'} was not found in config. Values will be initialized to default values.
Loaded unet as UNet2DConditionModel from unet subfolder of runwayml/stable-diffusion-v1-5.
Loading pipeline components...: 100% 6/6 [00:01<00:00, 4.90it/s]
You have disabled the safety checker for <class 'diffusers.pipelines.stable_diffusion.pipeline_stable_diffusion.StableDiffusionPipeline'> by passing safety_checker=None. Ensure that you abide to the conditions of the Stable Diffusion license and do not expose unfiltered results in services or applications open to the public. Both the diffusers team and Hugging Face strongly recommend to keep the safety filter enabled in all public facing circumstances, disabling it only for use-cases that involve analyzing network behavior or auditing its results. For more information, please have a look at huggingface/diffusers#254 .
p1 => top view realistic texture of {}
ignoring args.outdir and using path /content/gdrive/MyDrive/MIT/SEM02/ComputationDesignLab/MaterialPalette/my_images/weights/Pebbles/an_object_with_azertyuiop_texture/checkpoint-800/outputs
preparing for /content/gdrive/MyDrive/MIT/SEM02/ComputationDesignLab/MaterialPalette/my_images/weights/Pebbles/an_object_with_azertyuiop_texture/checkpoint-800/outputs/azertyuiop_1K_t50_wmean_top-view-realistic-texture-of-o_1.png
100% 50/50 [00:10<00:00, 4.82it/s]
Traceback (most recent call last):
File "/content/gdrive/MyDrive/MIT/SEM02/ComputationDesignLab/MaterialPalette/pipeline.py", line 21, in
concept.infer(lora, renorm=True)
File "/content/gdrive/MyDrive/MIT/SEM02/ComputationDesignLab/MaterialPalette/concept/infer.py", line 398, in infer
return main(Namespace(
File "/usr/local/envs/matpal/lib/python3.10/site-packages/torch/autograd/grad_mode.py", line 27, in decorate_context
return func(*args, **kwargs)
File "/content/gdrive/MyDrive/MIT/SEM02/ComputationDesignLab/MaterialPalette/concept/infer.py", line 393, in main
renorm(fname)
File "/content/gdrive/MyDrive/MIT/SEM02/ComputationDesignLab/MaterialPalette/concept/renorm.py", line 40, in renorm
low_threshold = sorted_pixels[exclude_count]
IndexError: index 0 is out of bounds for dimension 0 with size 0

[wonjunior]

Hi, it seems sorted_pixels is empty, can you check that? Also, have you tried running without renorm on?

[richasempire]

I ran the process without renorm and it functioned, but it didn't utilize all the masks—it only produced two materials instead of three. I'm curious about how to resolve the issue with sorted_pixels.

Also, thank you for your prompt response; I’m really enjoying this work. I’m part of the Design and Computation group at MIT, where I’m exploring AI-generated renders for buildings and extracting materials to reapply to 3D models. I'm particularly interested in how materials can be characterized by their architectural type. Any suggestions you have would be greatly appreciated.

[wonjunior]

I am happy to hear you are enjoying our work!

• It is possible that the mask not returning any material is too small. When you extract the crops in the first stage of the method, you will get logs regarding the size and number of crops kept. In your example above, you have 2 out of 3 crops used for "WhitePaint". There is a minimum crop size under which the method won't be able to work, that is probably what happened to your missing material.
• As for the renorm problem, it seems to be an issue with the mask. So you know, it is not a mandatory step, it is a post-processing to normalize the generated texture with the input image. Could you share with me the "WhitePaint" mask?

I'd be happy to discuss your project by email.
Best

[richasempire]

1. Understood thanks!

This is the whitepaint mask.

I am glad! I couldn't find the email. My email is richag@mit.edu

[wonjunior]

Thanks for sharing, as I said I think it's an issue with the masking of the values. If you have the opportunity to go into the code, you may check whether sorted_pixels is empty or not, the error message you got suggests so.