Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Feature] Support Delving into High-Quality Synthetic Face Occlusion Segmentation Datasets #2194

Merged
merged 17 commits into from
Nov 11, 2022

Conversation

jinwonkim93
Copy link

Thanks for your contribution and we appreciate it a lot. The following instructions would make your pull request more healthy and more easily get feedback. If you do not understand some items, don't worry, just make the pull request and seek help from maintainers.

Motivation

Support for Delving into High-Quality Synthetic Face Occlusion Segmentation Datasets

Modification

Please briefly describe what modification is made in this PR.
add dataset
add dataset config
add model config
add data_prepare.md

BC-breaking (Optional)

Does the modification introduce changes that break the backward-compatibility of the downstream repos?
If so, please describe how it breaks the compatibility and how the downstream projects should modify their code to keep compatibility with this PR.

Use cases (Optional)

If this PR introduces a new feature, it is better to list some use cases here, and update the documentation.

Checklist

  1. Pre-commit or other linting tools are used to fix the potential lint issues.
  2. The modification is covered by complete unit tests. If not, please add more unit test to ensure the correctness.
  3. If the modification has potential influence on downstream projects, this PR should be tested with downstream projects, like MMDet or MMDet3D.
  4. The documentation has been modified accordingly, like docstring or example tutorials.

@codecov
Copy link

codecov bot commented Oct 17, 2022

Codecov Report

Base: 88.97% // Head: 88.97% // Increases project coverage by +0.00% 🎉

Coverage data is based on head (70b2853) compared to base (7b09967).
Patch coverage: 81.81% of modified lines in pull request are covered.

Additional details and impacted files
@@           Coverage Diff           @@
##           master    #2194   +/-   ##
=======================================
  Coverage   88.97%   88.97%           
=======================================
  Files         145      146    +1     
  Lines        8735     8746   +11     
  Branches     1473     1474    +1     
=======================================
+ Hits         7772     7782   +10     
- Misses        720      722    +2     
+ Partials      243      242    -1     
Flag Coverage Δ
unittests 88.97% <81.81%> (+<0.01%) ⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

Impacted Files Coverage Δ
mmseg/datasets/face.py 80.00% <80.00%> (ø)
mmseg/datasets/__init__.py 100.00% <100.00%> (ø)
mmseg/datasets/pipelines/transforms.py 98.83% <0.00%> (+0.19%) ⬆️

Help us with your feedback. Take ten seconds to tell us how you rate us. Have a feature suggestion? Share it here.

☔ View full report at Codecov.
📢 Do you have feedback about the report comment? Let us know in this issue.

Copy link
Collaborator

@MeowZheng MeowZheng left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Many thanks for your contribution, and we are working on do some experiments to test the pr.


The extracted and upsampled COCO objects images and masks can be found in this [drive](https://drive.google.com/drive/folders/15nZETWlGMdcKY6aHbchRsWkUI42KTNs5?usp=sharing).

Please extract CelebAMask-HQ and 11k Hands images based on the splits found in [drive](https://drive.google.com/drive/folders/15nZETWlGMdcKY6aHbchRsWkUI42KTNs5?usp=sharing).
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think it's better to provide a script to help other users extract and split these images.

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

done

Comment on lines 421 to 434
## Data Generation

Example script to generate NatOcc dataset

bash NatOcc.sh

Example script to generate RandOcc dataset

bash RandOcc.sh
<!-- #endregion -->

```python

```
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is it possible to use codes from the original repo as a reference and redevelop a script?

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

done

Comment on lines 45 to 72
img_dir='CelebAMask-HQ-original/image',
ann_dir='CelebAMask-HQ-original/mask_edited',
split='CelebAMask-HQ-original/split/train.txt',
pipeline=train_pipeline)

dataset_train_B = dict(
type=dataset_type,
data_root=data_root,
img_dir='NatOcc-SOT/image',
ann_dir='NatOcc-SOT/mask',
split='NatOcc-SOT/split/train.txt',
pipeline=train_pipeline)


dataset_valid = dict(
type=dataset_type,
data_root=data_root,
img_dir='RealOcc/image',
ann_dir='RealOcc/mask',
split='RealOcc/split/val.txt',
pipeline=test_pipeline)

dataset_test = dict(
type=dataset_type,
data_root=data_root,
img_dir='RealOcc/image',
ann_dir='RealOcc/mask',
split='RealOcc/test.txt',
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The structure of the folder is not consistent with what the readme writes, could you also write the directory structure after conversion in the README?

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

done

Comment on lines 64 to 65
work_dir = './work_dirs/deeplabv3plus_r101_512x512_C-CM+C-WO-NatOcc-SOT'
gpu_ids = range(0, 2)
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

In general, we do not need to set these two configs.

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

okay i will delete this

@xiexinch
Copy link
Collaborator

Hi @jinwonkim93,
Many thanks for your contribution, please follow our contribution guide to fix the lint problem.

@jinwonkim93
Copy link
Author

Hi @jinwonkim93, Many thanks for your contribution, please follow our contribution guide to fix the lint problem.

fixed lint problem :)
I have re do all the process and it worked well.

@xiexinch
Copy link
Collaborator

Thanks for updating, we'll do some tests and feedback to you asap :)

@MeowZheng MeowZheng added this to the 2.0.0rc2 milestone Oct 31, 2022
Copy link
Collaborator

@xiexinch xiexinch left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Sorry for the late reply, I'm running the generation scripts, after the data generation, we'll test your config then feedback to you.

RealOcc.7z
RealOcc-Wild.7z
11k-hands_mask.7z
11k-hands_image.7z
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is it the Hands.zip?

Copy link
Author

@jinwonkim93 jinwonkim93 Nov 1, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Perhaps I missed some information, I only found this link to download hand images at https://sites.google.com/view/11khands.
It is Hands.zip but not 11k-hands_image.7z, am I right?

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

yes. https://sites.google.com/view/11khands visit this site and download Hand images

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

i will redefine the steps to download materials.

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

i rewrote the procedure of downloading materials

7za x CelebAMask-HQ-masks_corrected.7z -o./CelebAMask-HQ
#suggest better code if you have
rsync -a ./CelebAMask-HQ/CelebA-HQ-img/ --files-from=./CelebAMask-HQ-WO-train.txt ./CelebAMask-HQ-WO-Train_img
basename -s .jpg ./CelebAMask-HQ-train/* > train.txt
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Does ./CelebAMask-HQ-train/* correspond to ./CelebAMask-HQ-WO-Train_img?

Copy link
Author

@jinwonkim93 jinwonkim93 Nov 1, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

yes you are right. sorry for the typo error. i will fix it

basename -s .jpg ./CelebAMask-HQ-train/* > train.txt
xargs -n 1 -i echo {}.png < train.txt > mask_train.txt
rsync -a ./CelebAMask-HQ/CelebAMask-HQ-masks_corrected/ --files-from=./mask_train.txt ./CelebAMask-HQ-WO-Train_mask
mv train.txt ../data/occlusion-aware-face-dataset
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I suggest creating the folder occlusion-aware-face-dataset first.

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

okay

Comment on lines +523 to +525
SOURCE_DATASET.MASK_DIR "path/to/mmsegmentation/data_materials/CelebAMask-HQ-WO-Train_mask" \
OCCLUDER_DATASET.IMG_DIR "path/to/mmsegmentation/data_materials/11k-hands_img" \
OCCLUDER_DATASET.MASK_DIR "path/to/mmsegmentation/data_materials/11k-hands_masks"
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The '/' should be added to the end of the address, otherwise, the mask image will not be found.
Did you meet this problem?

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, and i have fix the problem and PR it to the author. Try git pull the latest version

Comment on lines 481 to 505
├── data
│ ├── occlusion-aware-face-dataset
│ │ ├── train.txt
│ │ ├── NatOcc_hand_sot
│ │ │ ├── img
│ │ │ │ ├── {image}.jpg
│ │ │ ├── mask
│ │ │ │ ├── {mask}.png
│ │ ├── NatOcc_object
│ │ │ ├── img
│ │ │ │ ├── {image}.jpg
│ │ │ ├── mask
│ │ │ │ ├── {mask}.png
│ │ ├── RandOcc
│ │ │ ├── img
│ │ │ │ ├── {image}.jpg
│ │ │ ├── mask
│ │ │ │ ├── {mask}.png
│ │ ├── RealOcc
│ │ │ ├── img
│ │ │ │ ├── {image}.jpg
│ │ │ ├── mask
│ │ │ │ ├── {mask}.png
│ │ │ ├── split
│ │ │ │ ├── val.txt
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think this directory structure should be moved to the end, after the generation scripts.

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

done.

@xiexinch
Copy link
Collaborator

xiexinch commented Nov 8, 2022

Sorry for the late reply, I'm running the generation scripts, after the data generation, we'll test your config then feedback to you.

The data processing script can be run successfully and the experiment can be run normally.

Method mIoU
DeepLabV3+ 95.60

@jinwonkim93
Copy link
Author

Sorry for the late reply, I'm running the generation scripts, after the data generation, we'll test your config then feedback to you.

The data processing script can be run successfully and the experiment can be run normally.

Method mIoU
DeepLabV3+ 95.60

oh thank you for testing!

PR fix version to original repository. change to original repository.
@jinwonkim93
Copy link
Author

@xiexinch what do you think of merging this PR? Is there more things to do?

@xiexinch
Copy link
Collaborator

@xiexinch what do you think of merging this PR? Is there more things to do?

Sorry for the late reply. Did you try to train models with this dataset? The data processing guidelines look good to me, but the training result of DeepLabV3+ is not as good as the official paper, do you have any suggestions?

@jinwonkim93
Copy link
Author

jinwonkim93 commented Nov 11, 2022

@xiexinch what do you think of merging this PR? Is there more things to do?

Sorry for the late reply. Did you try to train models with this dataset? The data processing guidelines look good to me, but the training result of DeepLabV3+ is not as good as the official paper, do you have any suggestions?

There is ablation study on the paper about combination of the dataset. I have done some of the experiments and in my case CelebAMask-HQ-WO with corrected masks (this is original data which does not have occlusion) (C-CM) and one set of hand-occluded face (NatOcc-SOT) and one set of COCO-object occluded dataset (NatOcc) was the best with my additional datasets.

@xiexinch
Copy link
Collaborator

@xiexinch what do you think of merging this PR? Is there more things to do?

Sorry for the late reply. Did you try to train models with this dataset? The data processing guidelines look good to me, but the training result of DeepLabV3+ is not as good as the official paper, do you have any suggestions?

There is ablation study on the paper about combination of the dataset. I have done some of the experiments and in my case CelebAMask-HQ-WO with corrected masks (this is original data which does not have occlusion) (C-CM) and one set of hand-occluded face (NatOcc-SOT) and one set of COCO-object occluded dataset (NatOcc) was the best with my additional datasets.

Could you provide more configs to us? Then we can do the experiments and publish new models.

@MeowZheng MeowZheng merged commit 6b4c7ff into open-mmlab:master Nov 11, 2022
@jinwonkim93
Copy link
Author

@xiexinch what do you think of merging this PR? Is there more things to do?

Sorry for the late reply. Did you try to train models with this dataset? The data processing guidelines look good to me, but the training result of DeepLabV3+ is not as good as the official paper, do you have any suggestions?

There is ablation study on the paper about combination of the dataset. I have done some of the experiments and in my case CelebAMask-HQ-WO with corrected masks (this is original data which does not have occlusion) (C-CM) and one set of hand-occluded face (NatOcc-SOT) and one set of COCO-object occluded dataset (NatOcc) was the best with my additional datasets.

Could you provide more configs to us? Then we can do the experiments and publish new models.

sure.

@xiexinch
Copy link
Collaborator

@xiexinch what do you think of merging this PR? Is there more things to do?

Sorry for the late reply. Did you try to train models with this dataset? The data processing guidelines look good to me, but the training result of DeepLabV3+ is not as good as the official paper, do you have any suggestions?

There is ablation study on the paper about combination of the dataset. I have done some of the experiments and in my case CelebAMask-HQ-WO with corrected masks (this is original data which does not have occlusion) (C-CM) and one set of hand-occluded face (NatOcc-SOT) and one set of COCO-object occluded dataset (NatOcc) was the best with my additional datasets.

Could you provide more configs to us? Then we can do the experiments and publish new models.

sure.

Many thanks for your contribution, this PR is merged :)
You could create a new PR to provide configs to us.

MeowZheng pushed a commit to MeowZheng/mmsegmentation that referenced this pull request Jan 9, 2023
…Segmentation Datasets open-mmlab#2194

* add custom dataset

* add face occlusion dataset

* add config file for occlusion face

* fix format

* update prepare.md

* formatting

* formatting

* fix typo error for doc

* update downloading process

* Update dataset_prepare.md

PR fix version to original repository. change to original repository.
MeowZheng pushed a commit that referenced this pull request Jan 11, 2023
…Segmentation Datasets (#2194)

add custom dataset

add face occlusion dataset

add config file for occlusion face

fix format

update prepare.md

formatting

formatting

fix typo error for doc

update downloading process

Update dataset_prepare.md

PR fix version to original repository. change to original repository.
huajiangjiangLi added a commit to pytorchuser/HDB-Seg that referenced this pull request Apr 12, 2023
…Segmentation Datasets (open-mmlab#2194)

add custom dataset

add face occlusion dataset

add config file for occlusion face

fix format

update prepare.md

formatting

formatting

fix typo error for doc

update downloading process

Update dataset_prepare.md

PR fix version to original repository. change to original repository.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants